merge(r2): r2-plan03

docs/requirements/Component-SiteRuntime.md

@@ -96,7 +96,7 @@ flowchart TD
 
 > **Startup config load is retried (S5)**: step 1's deployed-config read is best-effort against local SQLite, which can transiently fail (locked/busy). A failed load must **not** leave the site as a silent zero-instance node — it is re-attempted on a fixed interval until it succeeds, then the retry self-cancels.
 
-> **Per-instance compilation during staggered startup (P6, follow-up)**: the Roslyn compile of each instance's scripts still runs inside Instance Actor start during staggered startup; moving it off-thread is a deferred optimization (it affects failover time-to-recover only, not correctness).
+> **Per-instance compilation during staggered startup (P6, follow-up)**: the Roslyn compile of each instance's scripts still runs inside Instance Actor start during staggered startup; moving it off-thread is a deferred optimization (it affects failover time-to-recover only, not correctness). As of the round-2 hardening, a process-wide compile cache dedupes identical script bodies within a node's process lifetime (the deploy gate's compile is reused by Instance Actor start), shrinking the recompile cost; the *first* compile after process start still runs inside Instance Actor start, so the deferral stands.
 
 ### Deployment Handling
 - Receives flattened instance configurations from central via the Communication Layer.
@@ -238,6 +238,7 @@ When the Instance Actor is stopped (due to disable, delete, or redeployment), Ak
   - **Range Violation**: Value is outside the allowed min/max range.
   - **Rate of Change**: Value change rate exceeds the defined threshold over a configurable time window. Direction filter (rising / falling / either) restricts which side of the rate triggers.
   - **HiLo**: Multi-setpoint level alarm with up to four configurable setpoints (LoLo, Lo, Hi, HiHi). Any subset may be configured. Each setpoint may carry its own priority that overrides the alarm-level priority for that band.
+- **Expression** trigger evaluation runs on the bounded script-execution thread pool shared with script bodies (and Expression *script* triggers): scheduler saturation — e.g. stuck scripts occupying all threads — delays Expression alarm transitions site-wide until a thread frees. This is a deliberate trade-off (evaluation no longer blocks dispatcher threads; per-actor coalescing bounds queue growth) and the scheduler gauges + stuck-script watchdog make the saturated state visible.
 - For binary trigger types (ValueMatch / RangeViolation / RateOfChange), when the condition is met and the alarm is currently in **normal** state, the alarm transitions to **active**:
   - Updates the alarm state on the parent Instance Actor (which publishes to the Akka stream).
   - If an on-trigger script is defined, spawns an Alarm Execution Actor to execute it.
@@ -486,7 +487,7 @@ Per Akka.NET best practices, internal actor communication uses **Tell** (fire-an
 ### Script Compilation Errors
 - If script compilation fails when a deployment is received, the entire deployment for that instance is **rejected**. No partial state is applied.
 - The failure is reported back to central as a failed deployment.
-- **Enforced by a site-side pre-compile validation gate (S3)**: before the Instance Actor is created or the config persisted, the Deployment Manager compiles **all** of the instance's scripts, alarm on-trigger scripts, and trigger expressions **off-thread**; any compile failure rejects the deployment with the collected compile errors. This makes the deployment result honest even if central's pre-deployment validation was bypassed or skew let a bad artifact through.
+- **Enforced by a site-side pre-compile validation gate (S3)**: before the Instance Actor is created or the config persisted, the Deployment Manager compiles **all** of the instance's scripts, alarm on-trigger scripts, and trigger expressions **synchronously, as a pure prefix step of the deploy handler on the singleton's actor thread** — deliberately not off-thread, because piping the verdict back to self would reorder concurrent deploys against delete/disable and break the redeploy-supersede mailbox-FIFO ordering (see the in-code rationale in `DeploymentManagerActor.HandleDeploy`); unchanged script bodies hit the process-wide compile cache, so the gate recompiles only genuinely new code. Any compile failure rejects the deployment with the collected compile errors. This makes the deployment result honest even if central's pre-deployment validation was bypassed or skew let a bad artifact through.
 - Note: Pre-deployment validation at central should still catch compilation errors before they reach the site; the site gate is the authoritative backstop.
 
 ---

src/ZB.MOM.WW.ScadaBridge.DataConnectionLayer/Adapters/MxGatewayDataConnection.cs

@@ -105,14 +105,16 @@ public class MxGatewayDataConnection : IDataConnection, IBrowsableDataConnection
 
         // Background event loop: route each value change to the matching subscription callback.
         _eventLoopCts = new CancellationTokenSource();
-        _ = Task.Run(() => RunEventLoopAsync(_eventLoopCts.Token));
+        var loopToken = _eventLoopCts.Token;   // capture NOW: the field may be replaced by a
+        var loopClient = _client;              // subsequent reconnect before Task.Run executes (N1)
+        _ = Task.Run(() => RunEventLoopAsync(loopClient, loopToken));
     }
 
-    private async Task RunEventLoopAsync(CancellationToken ct)
+    private async Task RunEventLoopAsync(IMxGatewayClient client, CancellationToken ct)
     {
         try
         {
-            await _client!.RunEventLoopAsync(update =>
+            await client.RunEventLoopAsync(update =>
             {
                 if (_tagToSub.TryGetValue(update.TagPath, out var subId) && _subs.TryGetValue(subId, out var s))
                     s.Callback(update.TagPath, new TagValue(update.Value, update.Quality, update.Timestamp));
@@ -122,6 +124,18 @@ public class MxGatewayDataConnection : IDataConnection, IBrowsableDataConnection
         {
             // Normal shutdown (DisconnectAsync / DisposeAsync cancelled the loop).
         }
+        catch (Exception ex) when (ct.IsCancellationRequested)
+        {
+            // Teardown fault of a superseded loop (N1): Cancel() is not synchronous with
+            // the loop's exit — a cancelled/disposed gRPC stream commonly surfaces as
+            // RpcException(StatusCode.Cancelled) (Grpc.Net default without
+            // ThrowOperationCanceledOnCancellation) or ObjectDisposedException from the
+            // concurrent DisposeAsync, milliseconds AFTER ConnectAsync has already reset
+            // _disconnectFired. Any fault on a cancelled token is normal shutdown;
+            // raising Disconnected here would flap the NEW healthy session.
+            _logger.LogDebug(ex,
+                "MxGateway event loop faulted after cancellation — superseded loop teardown; not signalling disconnect");
+        }
         catch (Exception ex)
         {
             _logger.LogWarning(ex, "MxGateway event stream faulted; signalling disconnect");

src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/AlarmActor.cs

@@ -186,6 +186,10 @@ public class AlarmActor : ReceiveActor
 
         // Handle the off-dispatcher trigger-expression evaluation result (P1).
         Receive<ExpressionEvalResult>(HandleExpressionEvalResult);
+
+        // Handle a faulted off-dispatcher evaluation task (N2) — clears in-flight
+        // instead of stranding the alarm trigger with an unhandled Status.Failure.
+        Receive<ExpressionEvalFailed>(HandleExpressionEvalFailed);
     }
 
     /// <inheritdoc />
@@ -551,7 +555,8 @@ public class AlarmActor : ReceiveActor
             }
         }, CancellationToken.None, TaskCreationOptions.DenyChildAttach,
             ScriptExecutionScheduler.Shared(_options)).Unwrap().PipeTo(self,
-            success: r => new ExpressionEvalResult(r));
+            success: r => new ExpressionEvalResult(r),
+            failure: ex => new ExpressionEvalFailed(ex));
     }
 
     /// <summary>
@@ -576,6 +581,22 @@ public class AlarmActor : ReceiveActor
         if (_evalPending) { _evalPending = false; StartExpressionEvaluation(); }
     }
 
+    /// <summary>
+    /// Applies a faulted off-dispatcher evaluation task (N2): logs the fault, then
+    /// delegates to the false result (per the class contract — a throwing expression
+    /// is treated as false so the state machine still runs, an Active alarm clears)
+    /// which clears in-flight and drains any pending evaluation. Without this the
+    /// alarm trigger would be permanently parked with _evalInFlight stuck true.
+    /// </summary>
+    private void HandleExpressionEvalFailed(ExpressionEvalFailed msg)
+    {
+        _healthCollector?.IncrementAlarmError();
+        _logger.LogError(msg.Cause,
+            "Alarm {Alarm} trigger-expression evaluation task faulted on {Instance}; treated as false",
+            _alarmName, _instanceName);
+        HandleExpressionEvalResult(new ExpressionEvalResult(false));
+    }
+
     /// <summary>
     /// HiLo level evaluator: returns the most-severe matching band for the
     /// given value. Severity order checked from highest to lowest so that a
@@ -795,6 +816,15 @@ public class AlarmActor : ReceiveActor
     /// actor thread.
     /// </summary>
     private sealed record ExpressionEvalResult(bool Result);
+
+    /// <summary>
+    /// Piped back to self when the off-dispatcher evaluation TASK itself faults
+    /// (e.g. ObjectDisposedException from a disposed ScriptExecutionScheduler
+    /// during shutdown) — the inner body's catch never sees that. Without this
+    /// mapping the actor receives an unhandled Status.Failure and _evalInFlight
+    /// stays true forever, permanently parking the expression trigger (N2).
+    /// </summary>
+    internal sealed record ExpressionEvalFailed(Exception Cause);
 }
 
 internal enum RateOfChangeDirection { Either, Rising, Falling }

src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/DeploymentManagerActor.cs

@@ -518,7 +518,11 @@ public class DeploymentManagerActor : ReceiveActor, IWithTimers
         // delete/disable commands, breaking that ordering. A deploy is an
         // infrequent admin command, so briefly holding the singleton for a pure
         // Roslyn compile is acceptable; the central deployer already Asks and
-        // waits for the DeploymentStatusResponse.
+        // waits for the DeploymentStatusResponse. Redeploys and multi-instance
+        // deploys of unchanged scripts hit the process-wide compile cache
+        // (SiteScriptCompileCache), so the synchronous gate recompiles only
+        // genuinely new code and the Instance Actor's PreStart reuses the gate's
+        // compile (N4).
         var errors = _deployCompileValidator.Validate(command.FlattenedConfigurationJson);
         if (errors.Count > 0)
         {

src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Actors/ScriptActor.cs

@@ -159,6 +159,10 @@ public class ScriptActor : ReceiveActor, IWithTimers
 
         // Handle the off-dispatcher trigger-expression evaluation result (P1).
         Receive<ExpressionEvalResult>(HandleExpressionEvalResult);
+
+        // Handle a faulted off-dispatcher evaluation task (N2) — clears in-flight
+        // instead of stranding the trigger with an unhandled Status.Failure.
+        Receive<ExpressionEvalFailed>(HandleExpressionEvalFailed);
     }
 
     /// <inheritdoc />
@@ -310,7 +314,8 @@ public class ScriptActor : ReceiveActor, IWithTimers
             }
         }, CancellationToken.None, TaskCreationOptions.DenyChildAttach,
             ScriptExecutionScheduler.Shared(_options)).Unwrap().PipeTo(self,
-            success: r => new ExpressionEvalResult(r));
+            success: r => new ExpressionEvalResult(r),
+            failure: ex => new ExpressionEvalFailed(ex));
     }
 
     /// <summary>
@@ -333,6 +338,18 @@ public class ScriptActor : ReceiveActor, IWithTimers
         if (_evalPending) { _evalPending = false; StartExpressionEvaluation(); }
     }
 
+    /// <summary>
+    /// Applies a faulted off-dispatcher evaluation task (N2): logs the fault via the
+    /// existing error path (which also increments the health counter), then reuses the
+    /// false-result path — clearing in-flight, applying the false edge, and draining any
+    /// pending evaluation — so a transient scheduler fault cannot permanently park the trigger.
+    /// </summary>
+    private void HandleExpressionEvalFailed(ExpressionEvalFailed msg)
+    {
+        LogExpressionError(msg.Cause);
+        HandleExpressionEvalResult(new ExpressionEvalResult(false));
+    }
+
     /// <summary>
     /// Applies a WhileTrue trigger's condition-state transition: on the
     /// false→true edge, fire once and start the re-fire timer; on the
@@ -621,6 +638,15 @@ public class ScriptActor : ReceiveActor, IWithTimers
     /// the actor thread.
     /// </summary>
     private sealed record ExpressionEvalResult(bool Result);
+
+    /// <summary>
+    /// Piped back to self when the off-dispatcher evaluation TASK itself faults
+    /// (e.g. ObjectDisposedException from a disposed ScriptExecutionScheduler
+    /// during shutdown) — the inner body's catch never sees that. Without this
+    /// mapping the actor receives an unhandled Status.Failure and _evalInFlight
+    /// stays true forever, permanently parking the expression trigger (N2).
+    /// </summary>
+    internal sealed record ExpressionEvalFailed(Exception Cause);
 }
 
 // ── Trigger config types ──

src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Deployment/DeployCompileValidator.cs

@@ -14,8 +14,11 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Deployment;
 /// deployment; no partial state is applied).
 ///
 /// Because it holds no actor state and only calls the (stateless)
-/// <see cref="ScriptCompilationService"/>, it is safe to run off the singleton
-/// mailbox on a <c>Task.Run</c> thread — the deploy path never blocks on Roslyn.
+/// <see cref="ScriptCompilationService"/>, it is pure and thread-safe; the Deployment
+/// Manager nevertheless invokes it <em>synchronously on the singleton's actor thread</em>
+/// as a pure prefix step, preserving mailbox-FIFO deploy ordering (see
+/// <see cref="Actors.DeploymentManagerActor.HandleDeploy"/>). Compiles of unchanged
+/// script bodies are deduped by the process-wide compile cache.
 /// </summary>
 public sealed class DeployCompileValidator
 {

src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/ScriptCompilationService.cs

@@ -13,6 +13,13 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
 /// <see cref="ScriptTrustValidator"/>; this service keeps the real
 /// execution-path compile of the script against <see cref="ScriptGlobals"/> /
 /// <see cref="TriggerExpressionGlobals"/>.
+///
+/// Compile results — success AND failure — are memoised process-wide in
+/// <see cref="SiteScriptCompileCache"/> (keyed on code hash + globals type), so an
+/// unchanged script body compiles once per process lifetime and the deploy gate,
+/// the Instance Actor's PreStart recompile, and SharedScriptLibrary all reuse one
+/// compiled <c>Script&lt;object?&gt;</c>. Error text is name-free by construction, so a
+/// cached failure renders correctly for every caller.
 /// </summary>
 public class ScriptCompilationService
 {
@@ -92,10 +99,33 @@ public class ScriptCompilationService
         => CompileCore(name, expression, typeof(TriggerExpressionGlobals));
 
     /// <summary>
-    /// Shared compilation path: validates the trust model, builds the script
-    /// against the given globals type, and returns the compiled result.
+    /// Shared compilation funnel for every site-side compile (the deploy gate, the
+    /// Instance Actor's PreStart recompile, and SharedScriptLibrary). Results — success
+    /// AND failure — are memoised process-wide in <see cref="SiteScriptCompileCache"/>,
+    /// keyed on code hash + globals type: an unchanged script body compiles once per
+    /// process lifetime and the compiled <c>Script&lt;object?&gt;</c> is shared across all
+    /// call sites (N4). Cached error text is name-free by construction (Roslyn diagnostics /
+    /// the compile-exception message never embed the caller's script name), so a cached
+    /// failure renders correctly for every caller.
     /// </summary>
     private ScriptCompilationResult CompileCore(string name, string code, Type globalsType)
+    {
+        var result = SiteScriptCompileCache.GetOrAdd(code, globalsType,
+            () => CompileUncached(name, code, globalsType));
+        if (!result.IsSuccess)
+            _logger.LogDebug(
+                "Script {Script}: returning cached compile failure ({ErrorCount} error(s))",
+                name, result.Errors.Count); // miss-path logs name the FIRST caller; keep this caller visible
+        return result;
+    }
+
+    /// <summary>
+    /// The uncached compile: validates the trust model, builds the script against the
+    /// given globals type, and returns the compiled result. The verdict is deterministic
+    /// on the code (plus the compile-time-static trust policy and globals surface), so it
+    /// is safe to cache alongside the compiled script.
+    /// </summary>
+    private ScriptCompilationResult CompileUncached(string name, string code, Type globalsType)
     {
         // Validate trust model
         var violations = ValidateTrustModel(code);

src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Scripts/SiteScriptCompileCache.cs

@@ -0,0 +1,84 @@
+using System.Collections.Concurrent;
+using System.Security.Cryptography;
+using System.Text;
+
+namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
+
+/// <summary>
+/// Process-wide cache of site-side script <see cref="ScriptCompilationResult"/>s, keyed on
+/// the globals type plus a SHA-256 hash of the script code. Unlike the TemplateEngine's
+/// <c>ScriptCompileVerdictCache</c> (which caches central-validation <em>verdicts</em>), this
+/// caches the full compiled result — the immutable Roslyn <c>Script&lt;object?&gt;</c> itself —
+/// so the synchronous deploy gate and the <c>InstanceActor.PreStart</c> recompile share one
+/// Roslyn compile per script body per process lifetime (N4).
+///
+/// <para>Soundness rests on three invariants:</para>
+/// <list type="number">
+///   <item>A compile result is a pure function of (code, globals type, compile-time-static
+///   trust policy) — none of which vary at runtime for a given key.</item>
+///   <item>The stored error text is name-free (Roslyn <c>Diagnostic.GetMessage()</c> /
+///   <c>"Compilation exception: …"</c>); callers prefix their own script-name context, so a
+///   shared cached failure renders correctly for every caller. <em>Failures are cached too</em> —
+///   a bad script body fails identically for every caller.</item>
+///   <item>Roslyn <c>Script&lt;T&gt;</c> is immutable and thread-safe (<c>RunAsync</c> creates
+///   per-run state), so one compiled instance may be shared across actors and threads.</item>
+/// </list>
+///
+/// <para>
+/// Bounded at <see cref="MaxEntries"/> entries — deliberately smaller than the verdict cache's
+/// 4096 because entries pin compiled assemblies, not just verdict strings. On overflow the cache
+/// is cleared wholesale (results are recomputable, so a coarse reset avoids eviction bookkeeping).
+/// <see cref="Hits"/>/<see cref="Count"/>/<see cref="Clear"/> are exposed for tests and diagnostics.
+/// </para>
+/// </summary>
+internal static class SiteScriptCompileCache
+{
+    /// <summary>Upper bound on cached entries; the cache is cleared wholesale on overflow.</summary>
+    internal const int MaxEntries = 1024;
+
+    private static readonly ConcurrentDictionary<string, ScriptCompilationResult> Cache = new();
+    private static long _hits;
+
+    /// <summary>Number of cache hits observed since the last <see cref="Clear"/>.</summary>
+    public static long Hits => Interlocked.Read(ref _hits);
+
+    /// <summary>Current number of cached compile results.</summary>
+    public static int Count => Cache.Count;
+
+    /// <summary>
+    /// Returns the cached compile result for <paramref name="code"/> against
+    /// <paramref name="globalsType"/>, or computes it via <paramref name="factory"/> and caches
+    /// it on a miss. A hit increments <see cref="Hits"/>. Both success and failure results are
+    /// cached — the error text is name-free by construction.
+    /// </summary>
+    /// <param name="code">The script source code to look up (hashed to form the cache key).</param>
+    /// <param name="globalsType">The Roslyn globals surface the script compiles against; part of the key so identical source under different globals stays distinct.</param>
+    /// <param name="factory">Computes the compile result on a cache miss.</param>
+    /// <returns>The cached or newly computed compile result.</returns>
+    public static ScriptCompilationResult GetOrAdd(string code, Type globalsType, Func<ScriptCompilationResult> factory)
+    {
+        var key = globalsType.FullName + ":" + Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(code)));
+
+        if (Cache.TryGetValue(key, out var result))
+        {
+            Interlocked.Increment(ref _hits);
+            return result;
+        }
+
+        result = factory();
+
+        // Coarse bound: on overflow drop everything rather than track evictions.
+        if (Cache.Count >= MaxEntries)
+            Cache.Clear();
+
+        Cache[key] = result;
+        return result;
+    }
+
+    /// <summary>Clears all cached compile results and resets the hit counter.</summary>
+    public static void Clear()
+    {
+        Cache.Clear();
+        Interlocked.Exchange(ref _hits, 0);
+    }
+}

src/ZB.MOM.WW.ScadaBridge.SiteRuntime/SiteRuntimeOptionsValidator.cs

@@ -56,5 +56,14 @@ public sealed class SiteRuntimeOptionsValidator : OptionsValidatorBase<SiteRunti
         builder.RequireThat(options.ConfigFetchRetryCount >= 0,
             $"ScadaBridge:SiteRuntime:ConfigFetchRetryCount must be >= 0 " +
             $"(was {options.ConfigFetchRetryCount}).");
+
+        builder.RequireThat(options.TagSubscribeRetryIntervalMs > 0,
+            $"ScadaBridge:SiteRuntime:TagSubscribeRetryIntervalMs must be greater than 0 " +
+            $"(was {options.TagSubscribeRetryIntervalMs}); a zero interval hot-loops the tag-subscribe " +
+            "retry and a negative one throws inside the Instance Actor's retry scheduling.");
+
+        builder.RequireThat(options.StuckScriptGraceMs >= 0,
+            $"ScadaBridge:SiteRuntime:StuckScriptGraceMs must be >= 0 " +
+            $"(was {options.StuckScriptGraceMs}); a negative grace throws inside the stuck-script watchdog's delay.");
     }
 }

tests/ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Tests/Adapters/MxGatewayDataConnectionReconnectTests.cs

@@ -1,3 +1,4 @@
+using Grpc.Core;
 using Microsoft.Extensions.Logging.Abstractions;
 using NSubstitute;
 using ZB.MOM.WW.ScadaBridge.DataConnectionLayer.Adapters;
@@ -47,6 +48,73 @@ public class MxGatewayDataConnectionReconnectTests
         Assert.False(loopTokens[1].IsCancellationRequested);  // new loop live
     }
 
+    [Fact]
+    public async Task StaleEventLoopRpcFault_AfterReconnect_DoesNotSignalDisconnected()
+    {
+        var loops = new List<TaskCompletionSource>();
+        var loopTokens = new List<CancellationToken>();
+        var adapter = CreateAdapterWithControllableLoops(loops, loopTokens, out var disconnectsRef);
+
+        var details = new Dictionary<string, string> { ["Endpoint"] = "http://gw:5000", ["ApiKey"] = "k" };
+
+        await adapter.ConnectAsync(details);
+        await WaitUntilAsync(() => loopTokens.Count == 1);
+        await adapter.ConnectAsync(details);          // reconnect: cancels loop #0, resets _disconnectFired
+        await WaitUntilAsync(() => loopTokens.Count == 2);
+
+        // The OLD loop observes its cancellation as a gRPC fault, not an OCE —
+        // the Grpc.Net default without ThrowOperationCanceledOnCancellation.
+        loops[0].SetException(new RpcException(new Status(StatusCode.Cancelled, "call cancelled")));
+        await Task.Delay(200);
+        Assert.Equal(0, Volatile.Read(ref disconnectsRef.Value));  // stale fault absorbed — the fresh connection must not flap
+
+        // The CURRENT loop's genuine fault must still signal, exactly once.
+        loops[1].SetException(new RpcException(new Status(StatusCode.Unavailable, "gateway gone")));
+        await WaitUntilAsync(() => Volatile.Read(ref disconnectsRef.Value) == 1);
+    }
+
+    [Fact]
+    public async Task StaleEventLoopObjectDisposedFault_AfterReconnect_DoesNotSignalDisconnected()
+    {
+        var loops = new List<TaskCompletionSource>();
+        var loopTokens = new List<CancellationToken>();
+        var adapter = CreateAdapterWithControllableLoops(loops, loopTokens, out var disconnectsRef);
+
+        var details = new Dictionary<string, string> { ["Endpoint"] = "http://gw:5000", ["ApiKey"] = "k" };
+
+        await adapter.ConnectAsync(details);
+        await WaitUntilAsync(() => loopTokens.Count == 1);
+        await adapter.ConnectAsync(details);          // reconnect
+        await WaitUntilAsync(() => loopTokens.Count == 2);
+
+        // The old loop faults with the concurrent-DisposeAsync shape.
+        loops[0].SetException(new ObjectDisposedException("MxGatewayClient"));
+        await Task.Delay(200);
+        Assert.Equal(0, Volatile.Read(ref disconnectsRef.Value));
+    }
+
+    private sealed class IntRef { public int Value; }
+
+    private static MxGatewayDataConnection CreateAdapterWithControllableLoops(
+        List<TaskCompletionSource> loops, List<CancellationToken> loopTokens, out IntRef disconnectsRef)
+    {
+        var factory = Substitute.For<IMxGatewayClientFactory>();
+        factory.Create().Returns(_ =>
+        {
+            var c = Substitute.For<IMxGatewayClient>();
+            var tcs = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
+            loops.Add(tcs);
+            c.RunEventLoopAsync(Arg.Any<Action<MxValueUpdate>>(), Arg.Do<CancellationToken>(t => loopTokens.Add(t)))
+                .Returns(tcs.Task);
+            return c;
+        });
+        var adapter = new MxGatewayDataConnection(factory, NullLogger<MxGatewayDataConnection>.Instance);
+        var refHolder = new IntRef();
+        adapter.Disconnected += () => Interlocked.Increment(ref refHolder.Value);
+        disconnectsRef = refHolder;
+        return adapter;
+    }
+
     private static async Task WaitUntilAsync(Func<bool> condition)
     {
         for (var i = 0; i < 100 && !condition(); i++)

tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/AlarmActorTests.cs

@@ -1054,4 +1054,31 @@ public class AlarmActorTests : TestKit, IDisposable
         var change = instanceProbe.ExpectMsg<AlarmStateChanged>(TimeSpan.FromSeconds(10));
         Assert.Equal(AlarmState.Active, change.State);
     }
+
+    [Fact]
+    public void ExpressionEvalTaskFault_DoesNotKillTheAlarmTrigger()
+    {
+        var expr = CompileRawTriggerExpression("true");
+        var alarmConfig = new ResolvedAlarm
+        {
+            CanonicalName = "ExprFaultAlarm",
+            TriggerType = "Expression",
+            TriggerConfiguration = "{\"expression\":\"true\"}",
+            PriorityLevel = 1
+        };
+        var instanceProbe = CreateTestProbe();
+        var alarm = ActorOf(Props.Create(() => new AlarmActor(
+            "ExprFaultAlarm", "Pump1", instanceProbe.Ref, alarmConfig,
+            null, _sharedLibrary, _options,
+            NullLogger<AlarmActor>.Instance, expr)));
+
+        // Simulate the faulted scheduler task: must be handled (logged, in-flight
+        // cleared, false applied) — NOT an unhandled Status.Failure.
+        alarm.Tell(new AlarmActor.ExpressionEvalFailed(new ObjectDisposedException("scheduler")));
+
+        // The trigger must still be alive: the next change evaluates and activates.
+        alarm.Tell(new AttributeValueChanged("Pump1", "A", "A", 1, "Good", DateTimeOffset.UtcNow));
+        var change = instanceProbe.ExpectMsg<AlarmStateChanged>(TimeSpan.FromSeconds(10));
+        Assert.Equal(AlarmState.Active, change.State);
+    }
 }

tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Actors/ScriptActorTests.cs

@@ -343,13 +343,41 @@ public class ScriptActorTests : TestKit, IDisposable
     private static Script<object?> CompileRawTriggerExpression(string expression)
     {
         var opts = ScriptOptions.Default
-            .WithReferences(typeof(object).Assembly, typeof(Enumerable).Assembly)
+            // The test assembly is referenced so a raw-compiled expression may reference
+            // test-only hooks like EvalGate (used by the faulted-task N2 test).
+            .WithReferences(typeof(object).Assembly, typeof(Enumerable).Assembly, typeof(ScriptActorTests).Assembly)
             .WithImports("System", "System.Collections.Generic", "System.Linq", "System.Threading.Tasks");
         var s = CSharpScript.Create<object?>(expression, opts, typeof(TriggerExpressionGlobals));
         s.Compile();
         return s;
     }
 
+    [Fact]
+    public void ExpressionEvalTaskFault_ClearsInFlight_AndDrainsPendingEvaluation()
+    {
+        var expr = CompileRawTriggerExpression(
+            "ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Actors.EvalGate.Block()");
+        var (actor, _) = CreateTriggeredActor(
+            "ExprFault", "Expression", "{\"expression\":\"true\",\"mode\":\"OnTrue\"}", null, expr);
+        try
+        {
+            actor.Tell(Change("A", "1"));                              // eval starts on the scheduler and BLOCKS → _evalInFlight = true
+            AwaitAssert(() => Assert.Equal(1, EvalGate.Entries), TimeSpan.FromSeconds(10));
+            actor.Tell(Change("A", "2"));                              // coalesces → _evalPending = true
+
+            // Simulate the faulted scheduler task the PipeTo failure mapping now surfaces
+            // (e.g. ObjectDisposedException from a disposed ScriptExecutionScheduler).
+            actor.Tell(new ScriptActor.ExpressionEvalFailed(new ObjectDisposedException("scheduler")));
+
+            // In-flight cleared + pending drained ⇒ a SECOND evaluation starts and blocks too.
+            AwaitAssert(() => Assert.Equal(2, EvalGate.Entries), TimeSpan.FromSeconds(10));
+        }
+        finally
+        {
+            EvalGate.Gate.Release(EvalGate.Entries); // ALWAYS free the shared scheduler threads
+        }
+    }
+
     [Fact]
     public void ExpressionTrigger_EvaluatesOnScriptSchedulerThread_AndStillFires()
     {
@@ -504,3 +532,17 @@ public class ScriptActorTests : TestKit, IDisposable
         instance.ExpectMsg<SetStaticAttributeCommand>(TimeSpan.FromSeconds(2));
     }
 }
+
+/// <summary>
+/// Test-only hook referenced by a raw-compiled trigger expression: <see cref="Block"/>
+/// blocks the calling script-scheduler thread on <see cref="Gate"/> so a test can hold an
+/// evaluation "in flight" and count how many evaluations have entered. Used by
+/// <c>ExpressionEvalTaskFault_ClearsInFlight_AndDrainsPendingEvaluation</c>.
+/// </summary>
+public static class EvalGate
+{
+    public static readonly SemaphoreSlim Gate = new(0);
+    private static int _entries;
+    public static int Entries => Volatile.Read(ref _entries);
+    public static bool Block() { Interlocked.Increment(ref _entries); Gate.Wait(); return false; }
+}

tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/ScriptCompilationServiceTests.cs

@@ -16,6 +16,7 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Scripts;
 /// <c>Stopwatch</c> stays allowed. The real execution-path compile against
 /// <c>ScriptGlobals</c> / <c>TriggerExpressionGlobals</c> is unchanged.
 /// </summary>
+[Collection("SiteScriptCompileCache")]
 public class ScriptCompilationServiceTests
 {
     private readonly ScriptCompilationService _service;
@@ -34,6 +35,28 @@ public class ScriptCompilationServiceTests
         Assert.Empty(result.Errors);
     }
 
+    [Fact]
+    public void Compile_SameCodeTwice_SharesOneRoslynCompile()
+    {
+        SiteScriptCompileCache.Clear();
+        var r1 = _service.Compile("deploy-gate-copy", "return 1 + 1;");
+        var r2 = _service.Compile("prestart-copy", "return 1 + 1;");
+
+        Assert.True(r1.IsSuccess);
+        Assert.Same(r1.CompiledScript, r2.CompiledScript);   // one compile, shared Script<T> (N4)
+        Assert.Equal(1, SiteScriptCompileCache.Hits);
+    }
+
+    [Fact]
+    public void Compile_ScriptAndTriggerExpression_DoNotCrossContaminate()
+    {
+        SiteScriptCompileCache.Clear();
+        var script = _service.Compile("s", "1 > 0");
+        var trigger = _service.CompileTriggerExpression("t", "1 > 0");
+
+        Assert.NotSame(script.CompiledScript, trigger.CompiledScript); // different globals surfaces
+    }
+
     [Fact]
     public void Compile_InvalidSyntax_ReturnsErrors()
     {

tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Scripts/SiteScriptCompileCacheTests.cs

@@ -0,0 +1,54 @@
+using Microsoft.CodeAnalysis.CSharp.Scripting;
+using Microsoft.CodeAnalysis.Scripting;
+using ZB.MOM.WW.ScadaBridge.SiteRuntime.Scripts;
+
+namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests.Scripts;
+
+/// <summary>
+/// Tests for the process-wide <see cref="SiteScriptCompileCache"/>. The cache is
+/// static/process-wide state, so this class shares the "SiteScriptCompileCache" xunit
+/// collection with <c>ScriptCompilationServiceTests</c> (no cross-class parallelism) and
+/// calls <see cref="SiteScriptCompileCache.Clear"/> at the top of each test.
+/// </summary>
+[Collection("SiteScriptCompileCache")]
+public class SiteScriptCompileCacheTests
+{
+    private static ScriptCompilationResult Ok() =>
+        ScriptCompilationResult.Succeeded(CSharpScript.Create<object?>("1", ScriptOptions.Default, typeof(ScriptGlobals)));
+
+    [Fact]
+    public void GetOrAdd_SameCodeAndGlobals_ComputesOnce()
+    {
+        SiteScriptCompileCache.Clear();
+        var factoryCalls = 0;
+        ScriptCompilationResult Factory() { factoryCalls++; return Ok(); }
+
+        var r1 = SiteScriptCompileCache.GetOrAdd("return 1;", typeof(ScriptGlobals), Factory);
+        var r2 = SiteScriptCompileCache.GetOrAdd("return 1;", typeof(ScriptGlobals), Factory);
+
+        Assert.Equal(1, factoryCalls);
+        Assert.Same(r1, r2);
+        Assert.Equal(1, SiteScriptCompileCache.Hits);
+    }
+
+    [Fact]
+    public void GetOrAdd_SameCode_DifferentGlobals_AreSeparateEntries()
+    {
+        SiteScriptCompileCache.Clear();
+        SiteScriptCompileCache.GetOrAdd("1 > 0", typeof(ScriptGlobals), Ok);
+        SiteScriptCompileCache.GetOrAdd("1 > 0", typeof(TriggerExpressionGlobals), Ok);
+
+        Assert.Equal(2, SiteScriptCompileCache.Count);
+        Assert.Equal(0, SiteScriptCompileCache.Hits); // no cross-globals hit
+    }
+
+    [Fact]
+    public void Overflow_ClearsWholesale()
+    {
+        SiteScriptCompileCache.Clear();
+        for (var i = 0; i <= SiteScriptCompileCache.MaxEntries; i++)
+            SiteScriptCompileCache.GetOrAdd($"return {i};", typeof(ScriptGlobals), Ok);
+
+        Assert.True(SiteScriptCompileCache.Count <= SiteScriptCompileCache.MaxEntries);
+    }
+}

tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/SiteRuntimeOptionsValidatorTests.cs

@@ -40,6 +40,24 @@ public class SiteRuntimeOptionsValidatorTests
         Assert.Contains("ScriptExecutionThreadCount", result.FailureMessage);
     }
 
+    [Fact]
+    public void ZeroTagSubscribeRetryIntervalMs_IsRejected()
+    {
+        var result = Validate(new SiteRuntimeOptions { TagSubscribeRetryIntervalMs = 0 });
+
+        Assert.True(result.Failed);
+        Assert.Contains("TagSubscribeRetryIntervalMs", result.FailureMessage);
+    }
+
+    [Fact]
+    public void NegativeStuckScriptGraceMs_IsRejected()
+    {
+        var result = Validate(new SiteRuntimeOptions { StuckScriptGraceMs = -1 });
+
+        Assert.True(result.Failed);
+        Assert.Contains("StuckScriptGraceMs", result.FailureMessage);
+    }
+
     [Fact]
     public void ZeroStreamBufferSize_IsRejected()
     {