fix(scriptanalysis): compile gate reads diagnostics only and never caches a faulted verdict (review 2026-10-21 04/M1, 04/C1)

RoslynScriptCompiler.Compile now uses Script.GetCompilation().GetDiagnostics()
with one shared, never-loading InteractiveAssemblyLoader instead of
Script.Compile(), which emitted and loaded a non-collectible assembly (and a
fresh AssemblyLoadContext) per distinct script. The catch-all that turned any
exception into a 'Compilation exception' diagnostic is gone; TryCompile and
CheckExpressionSyntax report a fault for that call only, uncached.

docs/requirements/Component-ScriptAnalysis.md

@@ -167,13 +167,31 @@ generations of 2048 entries. New verdicts land in the *hot* generation; a hit in
 *cold* is promoted back into hot; on overflow hot becomes the new cold and only the
 old cold — the half nothing has touched for a full generation — is dropped.
 
-The bound must never be enforced by clearing the cache outright. That reads as
-harmless ("a verdict is cheap to recompute") but is not: a recompute is a fresh
-Roslyn compile, and every script compile loads an assembly through a
-**non-collectible `InteractiveAssemblyLoader`**. Bounding that leak is the entire
-reason this cache exists, so dropping every hot entry at the 4096th distinct script
-would re-open it for the whole working set at once. Segmenting retains anything in
-active use across an eviction while keeping the same ceiling.
+The bound must never be enforced by clearing the cache outright: a recompute is a
+fresh trust analysis plus Roslyn compile against the full reference set (tens of ms
+each), so dropping every hot entry at the 4096th distinct script would re-pay that
+for the whole working set at once. Segmenting retains anything in active use across
+an eviction while keeping the same ceiling.
+
+**The gate never emits or loads an assembly (review 2026-10-21 04/M1).**
+`RoslynScriptCompiler.Compile` reads `Script.GetCompilation().GetDiagnostics()` —
+declaration and method-body errors, no PE emit — and passes every
+`CSharpScript.Create` one shared `InteractiveAssemblyLoader` that never loads
+anything. (It used to call `Script.Compile()`, which builds the executor: emit, then a
+load through a fresh **non-collectible** `AssemblyLoadContext` per *distinct* script —
+and `CSharpScript.Create` with no loader argument registers a new ALC even without the
+load.) Pinned by a test that N distinct compiles leave `AssemblyLoadContext.All` and
+the loaded-assembly count flat. The verdict cache therefore bounds CPU, not memory.
+The site runtime's `ScriptCompilationService` is unaffected — it needs an executable
+`Script` and keeps its own compile.
+
+**Only diagnostic verdicts are cached (review 2026-10-21 04/C1).** `Compile` no longer
+catches exceptions and returns them as a `"Compilation exception: …"` message; an
+unexpected fault (metadata I/O, out-of-memory, …) propagates. The cache never stores a
+verdict whose factory threw, and `ScriptCompiler.TryCompile` / `CheckExpressionSyntax`
+turn the fault into an uncached failure for that call alone ("could not be checked —
+the compile gate faulted … retry"), so a one-off fault can no longer become a sticky,
+process-wide "failed to compile" on one central node until restart.
 
 #### Parity guard
 

src/ZB.MOM.WW.ScadaBridge.ScriptAnalysis/RoslynScriptCompiler.cs

@@ -2,6 +2,7 @@ using Microsoft.CodeAnalysis;
 using Microsoft.CodeAnalysis.CSharp;
 using Microsoft.CodeAnalysis.CSharp.Scripting;
 using Microsoft.CodeAnalysis.Scripting;
+using Microsoft.CodeAnalysis.Scripting.Hosting;
 
 namespace ZB.MOM.WW.ScadaBridge.ScriptAnalysis;
 
@@ -16,6 +17,16 @@ namespace ZB.MOM.WW.ScadaBridge.ScriptAnalysis;
 /// </summary>
 public static class RoslynScriptCompiler
 {
+    /// <summary>
+    /// One loader for every design-time compile. <c>CSharpScript.Create</c> otherwise
+    /// constructs a fresh <see cref="InteractiveAssemblyLoader"/> per call, and on .NET
+    /// each loader registers its own non-collectible <c>AssemblyLoadContext</c> — even
+    /// when nothing is ever emitted or loaded through it. The gate never executes a
+    /// script, so this loader never loads anything; sharing it keeps the per-compile
+    /// ALC count at zero (review 2026-10-21 04/M1).
+    /// </summary>
+    private static readonly InteractiveAssemblyLoader SharedNonLoadingLoader = new();
+
     /// <summary>
     /// Parses the script as C# script source and returns syntax-error diagnostic
     /// messages (severity Error only). Empty list means the script parses.
@@ -47,38 +58,50 @@ public static class RoslynScriptCompiler
     /// <param name="extraReferences">Optional additional metadata references.</param>
     /// <param name="extraImports">Optional additional namespace imports.</param>
     /// <returns>Error-severity compile diagnostic messages; empty if the script compiles.</returns>
+    /// <exception cref="Exception">
+    /// Any non-diagnostic failure (metadata I/O, out-of-memory, an invalid import) is
+    /// thrown, never folded into the returned list — a returned message is always a
+    /// genuine compile error and therefore safe to memoise.
+    /// </exception>
     public static IReadOnlyList<string> Compile(
         string code,
         Type? globalsType = null,
         IEnumerable<MetadataReference>? extraReferences = null,
         IEnumerable<string>? extraImports = null)
     {
-        try
-        {
-            var references = ScriptTrustPolicy.DefaultReferences.ToList();
-            if (extraReferences != null)
-                references.AddRange(extraReferences);
+        var references = ScriptTrustPolicy.DefaultReferences.ToList();
+        if (extraReferences != null)
+            references.AddRange(extraReferences);
 
-            var imports = ScriptTrustPolicy.DefaultImports.AsEnumerable();
-            if (extraImports != null)
-                imports = imports.Concat(extraImports);
+        var imports = ScriptTrustPolicy.DefaultImports.AsEnumerable();
+        if (extraImports != null)
+            imports = imports.Concat(extraImports);
 
-            var options = ScriptOptions.Default
-                .WithReferences(references)
-                .WithMetadataResolver(CachingScriptMetadataResolver.Instance)
-                .WithImports(imports);
+        var options = ScriptOptions.Default
+            .WithReferences(references)
+            .WithMetadataResolver(CachingScriptMetadataResolver.Instance)
+            .WithImports(imports);
 
-            var script = CSharpScript.Create<object?>(code, options, globalsType);
-            var diagnostics = script.Compile();
+        var script = CSharpScript.Create<object?>(code, options, globalsType, SharedNonLoadingLoader);
 
-            return diagnostics
-                .Where(d => d.Severity == DiagnosticSeverity.Error)
-                .Select(d => d.GetMessage())
-                .ToList();
-        }
-        catch (Exception ex)
-        {
-            return [$"Compilation exception: {ex.Message}"];
-        }
+        // Diagnostics only — NO emit, NO load. `Script.Compile()` builds the
+        // executor, which emits the PE image and loads it through
+        // InteractiveAssemblyLoader into a fresh, non-collectible
+        // AssemblyLoadContext: one permanent assembly per DISTINCT script for a gate
+        // that never executes anything (review 2026-10-21 04/M1). The compilation's
+        // own diagnostics are the declaration + method-body errors that decide
+        // whether the script compiles.
+        //
+        // No catch-all here (review 2026-10-21 04/C1): an unexpected exception
+        // (I/O reading metadata, OOM, ...) used to come back as a
+        // "Compilation exception: ..." diagnostic, which the design-time verdict
+        // cache then memoised as a sticky, process-wide "failed to compile". It now
+        // propagates, so the caller can report it without caching it.
+        var diagnostics = script.GetCompilation().GetDiagnostics();
+
+        return diagnostics
+            .Where(d => d.Severity == DiagnosticSeverity.Error)
+            .Select(d => d.GetMessage())
+            .ToList();
     }
 }

src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompileVerdictCache.cs

@@ -11,7 +11,14 @@ namespace ZB.MOM.WW.ScadaBridge.TemplateEngine.Validation;
 /// compile-time-static <c>ScriptTrustPolicy</c> AND the globals surface it was
 /// compiled against, so it is sound to memoise it across the whole process: an
 /// unchanged script runs the forbidden-API analysis and the Roslyn compile exactly
-/// once, removing both the repeat CPU cost and the repeat compiled-assembly load.
+/// once, removing the repeat CPU cost. (The compile itself no longer emits or loads an
+/// assembly — <c>RoslynScriptCompiler.Compile</c> reads compilation diagnostics only.)
+///
+/// <para>
+/// <b>Only completed verdicts are stored.</b> A factory that throws propagates out of
+/// <see cref="GetOrAdd"/> and leaves the slot empty, so the next call re-runs it — a
+/// transient fault must never become a sticky, process-wide failure verdict.
+/// </para>
 ///
 /// <para>
 /// The globals surface is part of the key because a verdict is NOT interchangeable
@@ -32,11 +39,10 @@ namespace ZB.MOM.WW.ScadaBridge.TemplateEngine.Validation;
 /// <para>
 /// <b>Eviction is segmented, never wholesale.</b> Overflow used to
 /// <c>Clear()</c> the whole cache, which reads as harmless ("the verdict is cheap
-/// to recompute") but is not: a recompute is a fresh Roslyn compile, and every
-/// script compile loads an assembly through a NON-COLLECTIBLE
-/// <c>InteractiveAssemblyLoader</c>. That leak is the entire reason this cache
-/// exists, so dropping every hot entry at the 4096th distinct script re-opens it
-/// for the whole working set at once. Instead the cache runs two generations: new
+/// to recompute") but is not: a recompute is a fresh trust analysis plus Roslyn
+/// compile against the full reference set, so dropping every hot entry at the
+/// 4096th distinct script re-pays that for the whole working set at once. Instead
+/// the cache runs two generations: new
 /// verdicts land in <c>hot</c>, a hit in <c>cold</c> is promoted back into
 /// <c>hot</c>, and on overflow <c>hot</c> becomes the new <c>cold</c> and the old
 /// <c>cold</c> — the half nothing has touched for a full generation — is the only

src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ScriptCompiler.cs

@@ -22,6 +22,27 @@ namespace ZB.MOM.WW.ScadaBridge.TemplateEngine.Validation;
 /// </summary>
 public class ScriptCompiler
 {
+    private readonly Func<string, IReadOnlyList<string>> _compile;
+
+    /// <summary>
+    /// Creates the gate over the real Roslyn compile against
+    /// <see cref="ScriptCompileSurface"/>.
+    /// </summary>
+    public ScriptCompiler()
+        : this(code => RoslynScriptCompiler.Compile(code, typeof(ScriptCompileSurface)))
+    {
+    }
+
+    /// <summary>
+    /// Test seam: substitutes the compile step (the forbidden-API verdict still runs
+    /// for real). Used to prove a faulting compile is never memoised.
+    /// </summary>
+    /// <param name="compile">Returns error-severity diagnostics for the code, or throws.</param>
+    internal ScriptCompiler(Func<string, IReadOnlyList<string>> compile)
+    {
+        _compile = compile;
+    }
+
     /// <summary>
     /// Attempts to compile a script and returns success or a compilation error.
     /// The security check runs first: a script that uses a forbidden API is
@@ -39,23 +60,40 @@ public class ScriptCompiler
         // globals surface are compile-time static), so memoise it process-wide: an
         // unchanged script compiles exactly once per process. The cached error is
         // name-free — the script name is formatted in below at read time.
-        var (ok, error) = ScriptCompileVerdictCache.GetOrAdd(nameof(ScriptCompileSurface), code, () =>
+        //
+        // Only a DIAGNOSTIC verdict is cached. If the analysis or compile throws
+        // (metadata I/O, out-of-memory, ...) the factory propagates, the cache slot
+        // stays empty, and this call alone reports the fault — the next attempt
+        // re-runs the gate instead of serving a sticky, process-wide "failed to
+        // compile" that only a restart clears (review 2026-10-21 04/C1).
+        bool ok;
+        string? error;
+        try
         {
-            // Authoritative forbidden-API verdict first — a security violation must
-            // gate the script regardless of whether it otherwise compiles. Report
-            // ALL violations, not just the first, so an operator fixing
-            // one forbidden API isn't surprised by a second on the next attempt.
-            var violations = ScriptTrustValidator.FindViolations(code);
-            if (violations.Count > 0)
-                return (false, $"uses forbidden API: {string.Join("; ", violations)}");
+            (ok, error) = ScriptCompileVerdictCache.GetOrAdd(nameof(ScriptCompileSurface), code, () =>
+            {
+                // Authoritative forbidden-API verdict first — a security violation must
+                // gate the script regardless of whether it otherwise compiles. Report
+                // ALL violations, not just the first, so an operator fixing
+                // one forbidden API isn't surprised by a second on the next attempt.
+                var violations = ScriptTrustValidator.FindViolations(code);
+                if (violations.Count > 0)
+                    return (false, $"uses forbidden API: {string.Join("; ", violations)}");
 
-            // Real CSharpScript compile against the runtime-mirroring globals surface.
-            var errors = RoslynScriptCompiler.Compile(code, typeof(ScriptCompileSurface));
-            if (errors.Count > 0)
-                return (false, $"failed to compile: {string.Join("; ", errors)}");
+                // Real CSharpScript compile against the runtime-mirroring globals surface.
+                var errors = _compile(code);
+                if (errors.Count > 0)
+                    return (false, $"failed to compile: {string.Join("; ", errors)}");
 
-            return (true, (string?)null);
-        });
+                return (true, (string?)null);
+            });
+        }
+        catch (Exception ex)
+        {
+            return Result<bool>.Failure(
+                $"Script '{scriptName}' could not be checked — the compile gate faulted " +
+                $"({ex.GetType().Name}: {ex.Message}). This is not a script error; retry.");
+        }
 
         return ok
             ? Result<bool>.Success(true)

src/ZB.MOM.WW.ScadaBridge.TemplateEngine/Validation/ValidationService.cs

@@ -97,7 +97,7 @@ public class ValidationService
     /// </param>
     /// <param name="validateScriptCompilation">
     /// <c>true</c> (default) runs the authoritative Roslyn <see cref="ValidateScriptCompilation"/>
-    /// stage — the single most expensive check (a non-collectible assembly load per script) —
+    /// stage — the single most expensive check (a full trust analysis + Roslyn compile per script) —
     /// AND the Expression-trigger syntax/compile check
     /// (<see cref="CheckExpressionSyntax"/>, a forbidden-API verdict plus a real compile of every
     /// Expression-triggered script/alarm body against <see cref="TriggerCompileSurface"/>).
@@ -125,7 +125,7 @@ public class ValidationService
             ValidateFlatteningSuccess(configuration),
             ValidateNamingCollisions(configuration),
             // The Roslyn compile is the single most expensive validation stage (a
-            // non-collectible InteractiveAssemblyLoader assembly load per script).
+            // full trust analysis + compile per uncached script).
             // Read-only staleness/comparison callers pass validateScriptCompilation:
             // false to skip ONLY this stage; every other structural/semantic check
             // still runs. The deploy gate keeps the default (true) — it is the
@@ -556,22 +556,34 @@ public class ValidationService
         // the verdict is a pure function of expression + policy + TriggerCompileSurface, and
         // it is the exact hot-path cost N1 flagged — every staleness sweep / import probe of
         // an Expression-triggered config re-ran a full trust compilation + script compile.
-        var (_, error) = ScriptCompileVerdictCache.GetOrAdd(nameof(TriggerCompileSurface), expression, () =>
+        //
+        // A faulting gate (exception, not diagnostic) is reported for this call only and
+        // never memoised — see ScriptCompiler.TryCompile (review 2026-10-21 04/C1).
+        string? error;
+        try
         {
-            // Authoritative forbidden-API verdict first. Report ALL violations, not
-            // just the first, so an operator fixing one forbidden API isn't surprised
-            // by a second on the next deploy attempt (mirrors ScriptCompiler.TryCompile).
-            var violations = ScriptTrustValidator.FindViolations(expression);
-            if (violations.Count > 0)
-                return (false, $"uses forbidden API: {string.Join("; ", violations)}");
+            (_, error) = ScriptCompileVerdictCache.GetOrAdd(nameof(TriggerCompileSurface), expression, () =>
+            {
+                // Authoritative forbidden-API verdict first. Report ALL violations, not
+                // just the first, so an operator fixing one forbidden API isn't surprised
+                // by a second on the next deploy attempt (mirrors ScriptCompiler.TryCompile).
+                var violations = ScriptTrustValidator.FindViolations(expression);
+                if (violations.Count > 0)
+                    return (false, $"uses forbidden API: {string.Join("; ", violations)}");
 
-            // Real compile of the bare boolean expression against the trigger globals.
-            var errors = RoslynScriptCompiler.Compile(expression, typeof(TriggerCompileSurface));
-            if (errors.Count > 0)
-                return (false, $"is not a valid expression: {string.Join("; ", errors)}");
+                // Real compile of the bare boolean expression against the trigger globals.
+                var errors = RoslynScriptCompiler.Compile(expression, typeof(TriggerCompileSurface));
+                if (errors.Count > 0)
+                    return (false, $"is not a valid expression: {string.Join("; ", errors)}");
 
-            return (true, (string?)null);
-        });
+                return (true, (string?)null);
+            });
+        }
+        catch (Exception ex)
+        {
+            return $"could not be checked — the compile gate faulted ({ex.GetType().Name}: {ex.Message}). " +
+                   "This is not an expression error; retry.";
+        }
         return error;
     }
 

tests/ZB.MOM.WW.ScadaBridge.ScriptAnalysis.Tests/RoslynScriptCompilerTests.cs

@@ -206,4 +206,65 @@ public class RoslynScriptCompilerTests
         Assert.Empty(RoslynScriptCompiler.Compile("return 1 + 1;"));
         Assert.Empty(RoslynScriptCompiler.Compile("return 2 + 2;"));
     }
+
+    /// <summary>
+    /// Review 2026-10-21 04/M1: the design-time gate only reads diagnostics, so it must
+    /// never emit + load an assembly. Each distinct script used to add a permanent,
+    /// non-collectible <see cref="System.Runtime.Loader.AssemblyLoadContext"/> via
+    /// <c>Script.Compile()</c> → <c>InteractiveAssemblyLoader</c>.
+    /// </summary>
+    [Fact]
+    public void Compile_DistinctScripts_DoesNotLoadAnAssemblyPerScript()
+    {
+        const int n = 12;
+        // Warm up any one-off loads (Roslyn itself, the surface assembly).
+        RoslynScriptCompiler.Compile("return 0;", typeof(ScriptCompileSurface));
+
+        var contextsBefore = System.Runtime.Loader.AssemblyLoadContext.All.Count();
+        var assembliesBefore = AppDomain.CurrentDomain.GetAssemblies().Length;
+
+        for (var i = 0; i < n; i++)
+            Assert.Empty(RoslynScriptCompiler.Compile($"return {i} + {Guid.NewGuid().GetHashCode()};",
+                typeof(ScriptCompileSurface)));
+
+        // Tolerance for unrelated loads made by tests running in parallel; a per-script
+        // load would grow by at least n.
+        Assert.True(System.Runtime.Loader.AssemblyLoadContext.All.Count() - contextsBefore < n / 2,
+            "a distinct-script compile created an AssemblyLoadContext");
+        Assert.True(AppDomain.CurrentDomain.GetAssemblies().Length - assembliesBefore < n / 2,
+            "a distinct-script compile loaded an assembly");
+    }
+
+    /// <summary>
+    /// Review 2026-10-21 04/C1: an unexpected exception inside the compile gate must
+    /// propagate rather than come back as a "Compilation exception" diagnostic — the
+    /// diagnostic would be memoised as a sticky, process-wide "failed to compile"
+    /// verdict by the design-time verdict cache.
+    /// </summary>
+    [Fact]
+    public void Compile_UnexpectedException_Propagates_NotReturnedAsDiagnostic()
+    {
+        // A reference whose metadata read faults — a stand-in for any non-diagnostic
+        // failure (I/O reading metadata, OOM, a loader fault).
+        Assert.Throws<InvalidOperationException>(() =>
+            RoslynScriptCompiler.Compile("return 1;", extraReferences: [new FaultingReference()]));
+    }
+
+    /// <summary>A metadata reference whose metadata cannot be produced.</summary>
+    private sealed class FaultingReference : Microsoft.CodeAnalysis.PortableExecutableReference
+    {
+        public FaultingReference()
+            : base(Microsoft.CodeAnalysis.MetadataReferenceProperties.Assembly, "faulting.dll")
+        {
+        }
+
+        protected override Microsoft.CodeAnalysis.Metadata GetMetadataImpl()
+            => throw new InvalidOperationException("transient metadata fault");
+
+        protected override Microsoft.CodeAnalysis.DocumentationProvider CreateDocumentationProvider()
+            => Microsoft.CodeAnalysis.DocumentationProvider.Default;
+
+        protected override Microsoft.CodeAnalysis.PortableExecutableReference WithPropertiesImpl(
+            Microsoft.CodeAnalysis.MetadataReferenceProperties properties) => new FaultingReference();
+    }
 }

tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ScriptCompileVerdictCacheEvictionTests.cs

@@ -4,9 +4,8 @@ namespace ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests.Validation;
 
 /// <summary>
 /// The verdict cache's eviction policy. Overflow used to <c>Clear()</c> the
-/// whole cache, which re-opened the non-collectible <c>InteractiveAssemblyLoader</c>
-/// leak the cache exists to bound — every hot script had to be recompiled, and every
-/// recompile loads another assembly that can never be unloaded. These tests pin the
+/// whole cache, so every hot script had to be re-analysed and recompiled at once.
+/// These tests pin the
 /// replacement: eviction is segmented, so entries in active use survive it.
 ///
 /// <para>

tests/ZB.MOM.WW.ScadaBridge.TemplateEngine.Tests/Validation/ScriptCompilerTests.cs

@@ -259,4 +259,53 @@ public class ScriptCompilerTests
         Assert.Contains("'Beta'", second.Error);
         Assert.Contains("compile", second.Error, StringComparison.OrdinalIgnoreCase);
     }
+
+    // --- Review 2026-10-21 04/C1: transient faults must not become sticky verdicts ---
+
+    /// <summary>Verifies that a factory that throws leaves the cache slot empty, so the next call re-runs it.</summary>
+    [Fact]
+    public void GetOrAdd_FactoryThrows_NextCallReRunsFactory()
+    {
+        ScriptCompileVerdictCache.Clear();
+        Assert.Throws<InvalidOperationException>(() =>
+            ScriptCompileVerdictCache.GetOrAdd("SurfaceC1", "return 7;",
+                () => throw new InvalidOperationException("transient")));
+
+        var calls = 0;
+        var verdict = ScriptCompileVerdictCache.GetOrAdd("SurfaceC1", "return 7;", () =>
+        {
+            calls++;
+            return (true, null);
+        });
+
+        Assert.Equal(1, calls);
+        Assert.True(verdict.Ok);
+        Assert.Equal(0, ScriptCompileVerdictCache.Hits);
+    }
+
+    /// <summary>
+    /// Verifies that a compile gate that faults once reports a failure for that call
+    /// only — the next call with the same code re-runs the compile and succeeds.
+    /// </summary>
+    [Fact]
+    public void TryCompile_CompileFaultsOnce_FailureIsNotCached()
+    {
+        ScriptCompileVerdictCache.Clear();
+        var faults = 1;
+        var c = new ScriptCompiler(_ =>
+        {
+            if (faults-- > 0)
+                throw new System.IO.IOException("metadata read failed");
+            return [];
+        });
+        const string code = "return 8;";
+
+        var first = c.TryCompile(code, "Flaky");
+        Assert.True(first.IsFailure);
+        Assert.Contains("'Flaky'", first.Error);
+        Assert.Contains("metadata read failed", first.Error);
+        Assert.Equal(0, ScriptCompileVerdictCache.Count);   // nothing memoised
+
+        Assert.True(c.TryCompile(code, "Flaky").IsSuccess);  // pre-fix: sticky "failed to compile"
+    }
 }