feat(scripting): PE cache, quarantine-race fix, recovery, readiness for the out-of-process runner (lmxopcua#780 P1)

Second P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md; still opt-in behind
Scripting:Runner=OutOfProcess (InProcess unchanged).

- Compiled-PE cache (design §7, §14 #8): Scripting:Worker:PeCacheDir holds <dir>/pe/<id>.pe entries authenticated
  with HMAC-SHA256 under a per-node random key (<dir>/pe-cache.key, created by the host, 0600/0700 on Unix), handed to
  the worker in Hello. Entry id = SHA-256(compile fingerprint + kind + source); the fingerprint covers the Roslyn
  version, the Core.Scripting build (analyzer, wrapper, stack guard), the sandbox reference set and the runtime. Edited,
  truncated, other-key or renamed entries are rejected, deleted and recompiled; the worker never loads unverified bytes.
  Pruned by recency at startup (30 d / 50 000). ScriptEvaluator splits into CompileToImage + FromImage. Measured on
  40 scripts, restart to every first result: macOS 820 ms no cache / 840 ms empty / 85 ms warm;
  windev (Windows) 9.5 s / 9.4 s / 1.0 s.
- Quarantine race: WorkerConnection runs exit attribution synchronously BEFORE completing the evaluations/loads it held
  (and before Exited), and the shard passes it straight to the runner — a kill strike is in place before the killed
  key's slot is released.
- Supervision: a dead worker restarts by itself after its backoff; Unavailable outcomes carry ScriptRunOutcome.Recovery
  (-> VirtualTagEvalResult.RetryWhen) and the VT actor / Calculation driver re-evaluate once against current inputs when
  it completes. Isolation worker stops after Scripting:Worker:IsolationIdleTimeout (10 min). Keys compiling when a
  worker dies are suspects too (isolation finds a poison compile and quarantines it). Startup orphan sweep by launch
  marker: kills only a process whose marker's host is gone and whose pid + start time + image match.
- Crash-loop circuit (Scripting:Worker:CircuitOpenAfter, 2 min): runner Unavailable, one Error, evaluations answered
  at once while shards keep retrying at the cap.
- Observability (§10): otopcua.script.runner.state, worker.up/inflight/queue_depth/heap_bytes (shard),
  ipc.roundtrip + eval.duration (kind), compile{cache}, isolation.suspects, exits reason=idle. `script-runner` health
  check on the ready tier, at worst Degraded (§14 #12: degraded but still ready).

Left on #780: self-contained worker publish, installer (account, pipe ACL, firewall), docker-dev live drill,
scripted-alarm seam, the §4.5 per-seam batch submit, the §8 perf harness.

Tests: SupervisionTests, PeCacheStoreTests/PeCacheWorkerTests (real workers), ScriptRunnerHealthCheckTests, VT + Calc
recovery seam tests; 14 mutations each turned their test red.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV

src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/IScriptRunner.cs

@@ -97,6 +97,13 @@ public sealed record ScriptRunOutcome(ScriptRunStatus Status, object? Value, str
     /// <summary>Reason when no worker became ready within <see cref="ScriptWorkerOptions.UnavailableGrace"/>.</summary>
     public const string RunnerUnavailableReason = "script runner unavailable";
 
+    /// <summary>
+    ///     For <see cref="ScriptRunStatus.Unavailable"/>: completes when the runner can run this script again (its worker
+    ///     is back), so the seam re-evaluates it then with its latest inputs (design §9, "re-evaluated on the shard's
+    ///     return"). Null when there is nothing to wait for.
+    /// </summary>
+    public Task? Recovery { get; init; }
+
     /// <summary>A successful run.</summary>
     /// <param name="value">The script's result.</param>
     /// <returns>The outcome.</returns>
@@ -130,7 +137,7 @@ public static class ScriptRunOutcomeExtensions
         {
             ScriptRunStatus.Ok => Commons.Engines.VirtualTagEvalResult.Ok(outcome.Value),
             ScriptRunStatus.Unavailable => Commons.Engines.VirtualTagEvalResult.Unavailable(
-                outcome.Reason ?? ScriptRunOutcome.RunnerUnavailableReason),
+                outcome.Reason ?? ScriptRunOutcome.RunnerUnavailableReason) with { RetryWhen = outcome.Recovery },
             _ => Commons.Engines.VirtualTagEvalResult.Failure(outcome.Reason ?? "evaluator failure"),
         };
     }

src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ScriptEvaluator.cs

@@ -70,7 +70,18 @@ public sealed class ScriptEvaluator<TContext, TResult> : IDisposable
     /// <summary>Compiles user script source into an evaluator.</summary>
     /// <param name="scriptSource">The user script source code to compile.</param>
     /// <returns>A ready-to-run <see cref="ScriptEvaluator{TContext, TResult}"/> wrapping the compiled script.</returns>
-    public static ScriptEvaluator<TContext, TResult> Compile(string scriptSource)
+    public static ScriptEvaluator<TContext, TResult> Compile(string scriptSource) => FromImage(CompileToImage(scriptSource));
+
+    /// <summary>
+    ///     The compile half of <see cref="Compile"/>: parse, sandbox compile, <see cref="ForbiddenTypeAnalyzer"/>, the
+    ///     stack-guard rewrite and the emit — returning the PE bytes without loading them. The out-of-process worker
+    ///     caches these bytes on disk (lmxopcua#780 design §7) so a restart does not re-enter Roslyn.
+    /// </summary>
+    /// <param name="scriptSource">The user script source code to compile.</param>
+    /// <returns>The emitted PE image.</returns>
+    /// <exception cref="CompilationErrorException">The source does not compile.</exception>
+    /// <exception cref="ScriptSandboxViolationException">The source uses a forbidden API.</exception>
+    public static byte[] CompileToImage(string scriptSource)
     {
         if (scriptSource is null) throw new ArgumentNullException(nameof(scriptSource));
 
@@ -142,8 +153,22 @@ public sealed class ScriptEvaluator<TContext, TResult> : IDisposable
             throw new CompilationErrorException(emitErrors);
         }
 
-        peStream.Position = 0;
-        var alc = new ScriptAssemblyLoadContext(assemblyName);
+        return peStream.ToArray();
+    }
+
+    /// <summary>
+    ///     The load half of <see cref="Compile"/>: loads a PE image produced by <see cref="CompileToImage"/> into a fresh
+    ///     collectible ALC and binds its <c>Run</c> method. The caller vouches for the image — only bytes this pipeline
+    ///     emitted (the worker's HMAC-verified PE cache) may be passed, because loading arbitrary PE bytes would run
+    ///     code the analyzer never saw.
+    /// </summary>
+    /// <param name="peImage">The image.</param>
+    /// <returns>A ready-to-run evaluator.</returns>
+    public static ScriptEvaluator<TContext, TResult> FromImage(byte[] peImage)
+    {
+        ArgumentNullException.ThrowIfNull(peImage);
+        using var peStream = new MemoryStream(peImage, writable: false);
+        var alc = new ScriptAssemblyLoadContext("ZB.MOM.WW.OtOpcUa.Core.Scripting.Compiled." + Guid.NewGuid().ToString("N"));
         Assembly assembly;
         try
         {
@@ -182,6 +207,33 @@ public sealed class ScriptEvaluator<TContext, TResult> : IDisposable
         return new ScriptEvaluator<TContext, TResult>(alc, func);
     }
 
+    /// <summary>
+    ///     A digest of everything that decides what <see cref="CompileToImage"/> emits for a given source, other than
+    ///     the source itself (lmxopcua#780 design §7): the Roslyn version, this assembly's build identity (the
+    ///     analyzer's deny-lists, the wrapper template, the stack-guard rewrite), the sandbox reference set, the
+    ///     runtime version and <typeparamref name="TContext"/> / <typeparamref name="TResult"/>. A compiled-PE cache
+    ///     keys on it, so an analyzer or compiler change invalidates every cached image and a cached PE can never skip
+    ///     a newer sandbox rule.
+    /// </summary>
+    public static byte[] CacheFingerprint => FingerprintHolder.Value;
+
+    private static readonly Lazy<byte[]> FingerprintHolder = new(ComputeFingerprint, LazyThreadSafetyMode.ExecutionAndPublication);
+
+    private static byte[] ComputeFingerprint()
+    {
+        var sb = new StringBuilder("otopcua-script-pe/1\n");
+        static void AppendAssembly(StringBuilder b, System.Reflection.Assembly a)
+            => b.Append(a.FullName).Append('|').Append(a.ManifestModule.ModuleVersionId.ToString("N")).Append('\n');
+        AppendAssembly(sb, typeof(CSharpCompilation).Assembly);
+        AppendAssembly(sb, typeof(Compilation).Assembly);
+        AppendAssembly(sb, typeof(ScriptEvaluator<,>).Assembly);
+        sb.Append(Environment.Version).Append('\n');
+        sb.Append(typeof(TContext).AssemblyQualifiedName).Append('\n');
+        sb.Append(typeof(TResult).AssemblyQualifiedName).Append('\n');
+        foreach (var identity in ScriptSandbox.ReferenceIdentities(typeof(TContext))) sb.Append(identity).Append('\n');
+        return System.Security.Cryptography.SHA256.HashData(Encoding.UTF8.GetBytes(sb.ToString()));
+    }
+
     /// <summary>Runs the script against an already-constructed context.</summary>
     /// <param name="context">The script context.</param>
     /// <param name="ct">Cancellation token for the operation.</param>

src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ScriptRunnerOptions.cs

@@ -74,6 +74,12 @@ public sealed class ScriptWorkerOptions
     /// <summary>Default <see cref="IsolationProbationRuns"/>: 20.</summary>
     public const int DefaultIsolationProbationRuns = 20;
 
+    /// <summary>Default <see cref="IsolationIdleTimeout"/>: 10 min (design §3.3).</summary>
+    public static readonly TimeSpan DefaultIsolationIdleTimeout = TimeSpan.FromMinutes(10);
+
+    /// <summary>Default <see cref="CircuitOpenAfter"/>: 2 min (design §5.3.6).</summary>
+    public static readonly TimeSpan DefaultCircuitOpenAfter = TimeSpan.FromMinutes(2);
+
     /// <summary>Shard workers per node; a script is placed by a stable hash of its key. Clamped to 1–8.</summary>
     public int Shards { get; set; } = DefaultShards;
 
@@ -102,9 +108,22 @@ public sealed class ScriptWorkerOptions
     public int IsolationProbationRuns { get; set; } = DefaultIsolationProbationRuns;
 
     /// <summary>
-    ///     Directory of the node-local compiled-PE cache (design §7). Must be absolute and outside the content root
-    ///     when set; startup fails otherwise (the family's fail-closed path rule). Reserved: the cache itself has
-    ///     not landed yet (lmxopcua#780 P1 remainder), so a worker restart recompiles.
+    ///     How long the isolation worker may sit with nothing to run before it is stopped (it is started again by the
+    ///     next suspect). Default 10 min; 1 s – 24 h.
+    /// </summary>
+    public TimeSpan IsolationIdleTimeout { get; set; } = DefaultIsolationIdleTimeout;
+
+    /// <summary>
+    ///     The crash-loop circuit (design §5.3.6): when every shard worker has been down for this long, the runner
+    ///     reports <c>Unavailable</c>, logs one Error and answers evaluations at once (no <see cref="UnavailableGrace"/>
+    ///     wait) while it keeps retrying at the backoff cap. Default 2 min; 1 s – 1 h.
+    /// </summary>
+    public TimeSpan CircuitOpenAfter { get; set; } = DefaultCircuitOpenAfter;
+
+    /// <summary>
+    ///     Directory of the node-local compiled-PE cache (design §7): HMAC-authenticated PE images, so a worker restart
+    ///     does not re-enter Roslyn. Must be absolute and outside the content root when set; startup fails otherwise
+    ///     (the family's fail-closed path rule). Null (the default) runs without a cache: every worker start recompiles.
     /// </summary>
     public string? PeCacheDir { get; set; }
 
@@ -175,9 +194,27 @@ public sealed class ScriptWorkerOptions
             probation = clamped;
         }
 
+        var idle = IsolationIdleTimeout;
+        if (idle < TimeSpan.FromSeconds(1) || idle > TimeSpan.FromHours(24))
+        {
+            var clamped = idle < TimeSpan.FromSeconds(1) ? TimeSpan.FromSeconds(1) : TimeSpan.FromHours(24);
+            notes.Add($"{prefix}{nameof(IsolationIdleTimeout)} {idle} is outside 00:00:01–1.00:00:00; clamped to {clamped}.");
+            idle = clamped;
+        }
+
+        var circuit = CircuitOpenAfter;
+        if (circuit < TimeSpan.FromSeconds(1) || circuit > TimeSpan.FromHours(1))
+        {
+            var clamped = circuit < TimeSpan.FromSeconds(1) ? TimeSpan.FromSeconds(1) : TimeSpan.FromHours(1);
+            notes.Add($"{prefix}{nameof(CircuitOpenAfter)} {circuit} is outside 00:00:01–01:00:00; clamped to {clamped}.");
+            circuit = clamped;
+        }
+
         adjustments = notes;
         return new ScriptWorkerOptions
         {
+            IsolationIdleTimeout = idle,
+            CircuitOpenAfter = circuit,
             Shards = shards,
             HardGrace = grace,
             HeapLimitBytes = heap,

src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ScriptSandbox.cs

@@ -99,6 +99,23 @@ public static class ScriptSandbox
         return new SandboxConfig(references, imports);
     }
 
+    /// <summary>
+    ///     A stable textual identity of the reference set <see cref="Build"/> compiles against for
+    ///     <paramref name="contextType"/> — each pinned assembly's name and module version id, then each BCL path — without
+    ///     opening any of them. Part of <see cref="ScriptEvaluator{TContext, TResult}.CacheFingerprint"/> (lmxopcua#780 §7).
+    /// </summary>
+    /// <param name="contextType">The concrete script context type.</param>
+    /// <returns>One line per reference, in a stable order.</returns>
+    public static IEnumerable<string> ReferenceIdentities(Type contextType)
+    {
+        ArgumentNullException.ThrowIfNull(contextType);
+        var pinned = new[] { typeof(DataValueSnapshot).Assembly, typeof(ScriptContext).Assembly, typeof(Serilog.ILogger).Assembly, contextType.Assembly }
+            .Distinct()
+            .Select(static a => $"{a.FullName}|{a.ManifestModule.ModuleVersionId:N}")
+            .Order(StringComparer.Ordinal);
+        return pinned.Concat(EnumerateBclAssemblyPaths().Order(StringComparer.Ordinal)).ToArray();
+    }
+
     private static readonly System.Collections.Concurrent.ConcurrentDictionary<System.Reflection.Assembly, MetadataReference> PinnedReferences = new();
 
     // The trusted-platform set is process-constant, so the BCL reference list is context-independent.