feat(scripting): PE cache, quarantine-race fix, recovery, readiness for the out-of-process runner (lmxopcua#780 P1)

Second P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md; still opt-in behind
Scripting:Runner=OutOfProcess (InProcess unchanged).

- Compiled-PE cache (design §7, §14 #8): Scripting:Worker:PeCacheDir holds <dir>/pe/<id>.pe entries authenticated
  with HMAC-SHA256 under a per-node random key (<dir>/pe-cache.key, created by the host, 0600/0700 on Unix), handed to
  the worker in Hello. Entry id = SHA-256(compile fingerprint + kind + source); the fingerprint covers the Roslyn
  version, the Core.Scripting build (analyzer, wrapper, stack guard), the sandbox reference set and the runtime. Edited,
  truncated, other-key or renamed entries are rejected, deleted and recompiled; the worker never loads unverified bytes.
  Pruned by recency at startup (30 d / 50 000). ScriptEvaluator splits into CompileToImage + FromImage. Measured on
  40 scripts, restart to every first result: macOS 820 ms no cache / 840 ms empty / 85 ms warm;
  windev (Windows) 9.5 s / 9.4 s / 1.0 s.
- Quarantine race: WorkerConnection runs exit attribution synchronously BEFORE completing the evaluations/loads it held
  (and before Exited), and the shard passes it straight to the runner — a kill strike is in place before the killed
  key's slot is released.
- Supervision: a dead worker restarts by itself after its backoff; Unavailable outcomes carry ScriptRunOutcome.Recovery
  (-> VirtualTagEvalResult.RetryWhen) and the VT actor / Calculation driver re-evaluate once against current inputs when
  it completes. Isolation worker stops after Scripting:Worker:IsolationIdleTimeout (10 min). Keys compiling when a
  worker dies are suspects too (isolation finds a poison compile and quarantines it). Startup orphan sweep by launch
  marker: kills only a process whose marker's host is gone and whose pid + start time + image match.
- Crash-loop circuit (Scripting:Worker:CircuitOpenAfter, 2 min): runner Unavailable, one Error, evaluations answered
  at once while shards keep retrying at the cap.
- Observability (§10): otopcua.script.runner.state, worker.up/inflight/queue_depth/heap_bytes (shard),
  ipc.roundtrip + eval.duration (kind), compile{cache}, isolation.suspects, exits reason=idle. `script-runner` health
  check on the ready tier, at worst Degraded (§14 #12: degraded but still ready).

Left on #780: self-contained worker publish, installer (account, pipe ACL, firewall), docker-dev live drill,
scripted-alarm seam, the §4.5 per-seam batch submit, the §8 perf harness.

Tests: SupervisionTests, PeCacheStoreTests/PeCacheWorkerTests (real workers), ScriptRunnerHealthCheckTests, VT + Calc
recovery seam tests; 14 mutations each turned their test red.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV

src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ScriptRunnerSelection.cs

@@ -29,6 +29,10 @@ public sealed class ScriptRunnerSelection : IAsyncDisposable, IDisposable
     /// <summary>The out-of-process runner, when one is configured.</summary>
     public IScriptRunner? OutOfProcessRunner => _runner;
 
+    /// <summary>The out-of-process runner's status (the readiness check), or null when every seam runs in process.</summary>
+    /// <returns>The status, or null.</returns>
+    public ScriptRunnerStatus? Status() => _runner?.Status();
+
     /// <summary>The runner <paramref name="seam"/> must use, or null for its in-process path.</summary>
     /// <param name="seam">The seam.</param>
     /// <returns>The runner, or null.</returns>

src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/HealthEndpoints.cs

@@ -62,6 +62,16 @@ public static class HealthEndpoints
                 sp => new OpcUaEndpointHealthCheck(sp.GetRequiredService<IOpcUaEndpointState>()),
                 failureStatus: null,
                 tags: tags));
+
+            // lmxopcua#780 design §10 / owner decision §14 #12: the out-of-process script runner is a READY-tier
+            // component that is at worst Degraded (still 200): OPC UA keeps serving driver data while scripts are
+            // Bad, so a sick runner must never fail readiness (on a pair that would move ServiceLevel to the peer).
+            // Healthy when every seam runs in process. Not on the active tier.
+            checks.Add(new HealthCheckRegistration(
+                ScriptRunnerHealthCheck.Name,
+                sp => new ScriptRunnerHealthCheck(sp.GetService<ScriptRunnerSelection>()),
+                failureStatus: null,
+                tags: new[] { ZbHealthTags.Ready }));
         }
 
         // configdb probe is admin-only (per-cluster mesh Phase 4). A driver-only node gets its config

src/Server/ZB.MOM.WW.OtOpcUa.Host/Health/ScriptRunnerHealthCheck.cs

@@ -0,0 +1,68 @@
+using Microsoft.Extensions.Diagnostics.HealthChecks;
+using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
+using ZB.MOM.WW.OtOpcUa.Host.Configuration;
+
+namespace ZB.MOM.WW.OtOpcUa.Host.Health;
+
+/// <summary>
+///     Pure decision function for the script-runner probe (lmxopcua#780 design §10, owner decision §14 #12), factored
+///     out of <see cref="ScriptRunnerHealthCheck"/> so the matrix is table-testable without worker processes.
+/// </summary>
+public static class ScriptRunnerHealthDecision
+{
+    /// <summary>
+    ///     Maps the runner's status to a health status. <b>Never Unhealthy:</b> a sick script runner does not stop the
+    ///     node serving driver data over OPC UA, so it must not fail readiness — on a redundant pair that would shift
+    ///     ServiceLevel to the peer over a script problem. Degraded is still ready (200) but visible.
+    /// </summary>
+    /// <param name="status">The out-of-process runner's status, or null when every seam runs in process.</param>
+    /// <returns>The status, a description and the per-shard data.</returns>
+    public static (HealthStatus Status, string Description, IReadOnlyDictionary<string, object> Data) Evaluate(ScriptRunnerStatus? status)
+    {
+        if (status is null)
+            return (HealthStatus.Healthy, "Scripts run in process (Scripting:Runner=InProcess).", new Dictionary<string, object>());
+
+        var data = new Dictionary<string, object>(StringComparer.Ordinal)
+        {
+            ["state"] = status.State.ToString(),
+            ["suspects"] = status.SuspectCount,
+            ["quarantined"] = status.QuarantinedCount,
+            ["peCache"] = status.PeCacheEnabled,
+        };
+        foreach (var shard in status.Shards)
+        {
+            data[$"shard.{shard.Label}"] = shard.Up
+                ? $"up pid {shard.Pid}, {shard.InFlight} in flight"
+                : shard.DownFor is { } downFor ? $"down {downFor.TotalSeconds:F0} s ({shard.ConsecutiveFailures} failure(s))" : "not started";
+        }
+
+        var down = status.Shards.Where(s => s.DownFor is not null).Select(s => s.Label).ToArray();
+        return status.State switch
+        {
+            ScriptRunnerState.Ready => (HealthStatus.Healthy, "Out-of-process script runner ready.", data),
+            ScriptRunnerState.Degraded => (HealthStatus.Degraded,
+                $"Out-of-process script runner degraded: ScriptWorker {string.Join(", ", down)} down; scripts placed there report BadResourceUnavailable until it restarts.",
+                data),
+            _ => (HealthStatus.Degraded,
+                "Out-of-process script runner unavailable (circuit open): every ScriptWorker shard is down; every compiled script reports BadResourceUnavailable. OPC UA keeps serving driver data.",
+                data),
+        };
+    }
+}
+
+/// <summary>The script-runner readiness component: Healthy, or Degraded-but-ready (see <see cref="ScriptRunnerHealthDecision"/>).</summary>
+/// <param name="selection">The node's runner selection; null on nodes that never register one.</param>
+public sealed class ScriptRunnerHealthCheck(ScriptRunnerSelection? selection) : IHealthCheck
+{
+    /// <summary>The check's registration name.</summary>
+    public const string Name = "script-runner";
+
+    /// <inheritdoc />
+    public Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default)
+    {
+        var (status, description, data) = ScriptRunnerHealthDecision.Evaluate(selection?.Status());
+        return Task.FromResult(status == HealthStatus.Healthy
+            ? HealthCheckResult.Healthy(description, data)
+            : HealthCheckResult.Degraded(description, data: data));
+    }
+}

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagActor.cs

@@ -51,6 +51,12 @@ public sealed class VirtualTagActor : ReceiveActor, IWithUnboundedStash
     /// when the tag has a <c>TimerIntervalMs</c>.</summary>
     private sealed class TimerTick { public static readonly TimerTick Instance = new(); private TimerTick() { } }
 
+    /// <summary>
+    ///     The out-of-process script runner that could not run this tag's script is serving again (lmxopcua#780 design
+    ///     §9): re-evaluate once against the latest inputs. Private to the actor.
+    /// </summary>
+    private sealed class RunnerRecovered { public static readonly RunnerRecovered Instance = new(); private RunnerRecovered() { } }
+
     /// <summary>The asynchronous evaluation started by <see cref="Evaluate"/> finished. Private to the actor.</summary>
     private sealed record EvaluationCompleted(
         Task<VirtualTagEvalResult> Evaluation, uint WorstInputStatus, DateTime TimestampUtc);
@@ -108,6 +114,8 @@ public sealed class VirtualTagActor : ReceiveActor, IWithUnboundedStash
     private readonly bool _changeTriggered;
     private readonly int? _timerIntervalMs;
     private ICancelable? _timer;
+    // One outstanding RunnerRecovered wait at most: a burst of Unavailable results re-evaluates once (latest-wins).
+    private bool _recoveryArmed;
     // R02:P6 — a script failing on every trigger of a 1 Hz input published one cluster-wide DPS
     // ScriptLogEntry per second per tag; 200 such tags shed everyone's entries out of the bounded
     // telemetry channels. The per-failure entry is now rate-limited per tag (20/min + one summary).
@@ -212,6 +220,11 @@ public sealed class VirtualTagActor : ReceiveActor, IWithUnboundedStash
         Receive<DependencyValueBatch>(OnDependencyBatch);
         Receive<ReassertValue>(_ => OnReassertValue());
         Receive<TimerTick>(_ => Evaluate(DateTime.UtcNow));
+        Receive<RunnerRecovered>(_ =>
+        {
+            _recoveryArmed = false;
+            Evaluate(DateTime.UtcNow);
+        });
         // Only reachable for a completion that outlived its behaviour (a supervision restart mid-evaluation
         // resets the behaviour stack): the incarnation it belonged to is gone, so it is dropped.
         Receive<EvaluationCompleted>(_ => { });
@@ -368,6 +381,21 @@ public sealed class VirtualTagActor : ReceiveActor, IWithUnboundedStash
         ReceiveAny(_ => Stash.Stash());
     }
 
+    /// <summary>
+    ///     A runner failure (BadResourceUnavailable) carries the task that completes when the runner is back: evaluate
+    ///     again then, with whatever inputs are current, instead of holding the Bad until the next input change — which,
+    ///     for a static input, would never come (lmxopcua#780 design §9, "re-evaluated on the shard's return").
+    /// </summary>
+    /// <param name="retryWhen">The recovery task, or null.</param>
+    private void ArmRecovery(Task? retryWhen)
+    {
+        if (retryWhen is null || _recoveryArmed) return;
+        _recoveryArmed = true;
+        var self = Self;
+        retryWhen.ContinueWith(_ => self.Tell(RunnerRecovered.Instance),
+            CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
+    }
+
     private void OnEvaluatorThrew(Exception ex, DateTime timestampUtc)
     {
         _log.Warning(ex, "VirtualTag {Id}: evaluator threw", _virtualTagId);
@@ -393,6 +421,7 @@ public sealed class VirtualTagActor : ReceiveActor, IWithUnboundedStash
         {
             OnEvaluationFailed(result.Reason ?? "evaluator failure", "Warning", timestampUtc,
                 result.StatusCode ?? OpcUaStatus.BadInternalError);
+            ArmRecovery(result.RetryWhen);
             return;
         }
 

src/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker/WorkerHost.cs

@@ -44,6 +44,7 @@ internal sealed class WorkerHost
     private int _queueCapacity = 4096;
     private TimeSpan _hardGrace = TimeSpan.FromSeconds(1);
     private TimeSpan _heartbeatInterval = TimeSpan.FromSeconds(1);
+    private ScriptPeCacheStore? _peCache;
     private long _seq;
 
     public WorkerHost(NamedPipeClientStream pipe, string nonce, int parentPid)
@@ -102,6 +103,11 @@ internal sealed class WorkerHost
         if (hello.QueueCapacity > 0) _queueCapacity = (int)hello.QueueCapacity;
         if (hello.HardGraceMs > 0) _hardGrace = TimeSpan.FromMilliseconds(hello.HardGraceMs);
         if (hello.HeartbeatIntervalMs > 0) _heartbeatInterval = TimeSpan.FromMilliseconds(hello.HeartbeatIntervalMs);
+        if (!string.IsNullOrEmpty(hello.PeCacheDir) && hello.PeCacheKey.Length == ScriptPeCacheStore.KeyLength)
+        {
+            // Design §7: the host owns the key; it reaches us only here, never through the environment or a file.
+            _peCache = new ScriptPeCacheStore(hello.PeCacheDir, hello.PeCacheKey.ToByteArray());
+        }
     }
 
     private void StartBackground(int evalThreads)
@@ -181,7 +187,8 @@ internal sealed class WorkerHost
             var outcome = new LoadOutcome { Key = script.Key, Ok = true };
             try
             {
-                var loaded = LoadedScript.Compile(script);
+                var loaded = LoadedScript.Load(script, _peCache, out var cache);
+                outcome.Cache = cache;
                 if (_scripts.TryRemove(script.Key, out var previous)) previous.Dispose();
                 _scripts[script.Key] = loaded;
             }
@@ -499,15 +506,52 @@ internal sealed class WorkerHost
 
         public string[] InputPaths { get; }
 
-        public static LoadedScript Compile(ScriptSource source) => source.Kind switch
+        /// <summary>
+        ///     Loads a script: from a verified PE cache entry when there is one (Roslyn is not entered), otherwise by
+        ///     compiling it and writing the entry (design §7). A compile error is never cached — the host's negative
+        ///     cache covers that — and an entry that fails verification is deleted by the store and recompiled.
+        /// </summary>
+        public static LoadedScript Load(ScriptSource source, ScriptPeCacheStore? cache, out PeCache how) => source.Kind switch
         {
             ContextKind.VirtualTag or ContextKind.Calculation => new LoadedScript(
-                [.. source.InputPaths], ScriptEvaluator<VirtualTagContext, object?>.Compile(source.Source), null),
+                [.. source.InputPaths], Load<VirtualTagContext, object?>(source, cache, out how), null),
             ContextKind.AlarmPredicate => new LoadedScript(
-                [.. source.InputPaths], null, ScriptEvaluator<AlarmPredicateContext, bool>.Compile(source.Source)),
+                [.. source.InputPaths], null, Load<AlarmPredicateContext, bool>(source, cache, out how)),
             _ => throw new ScriptWorkerProtocolException($"unknown script kind {source.Kind}"),
         };
 
+        private static ScriptEvaluator<TContext, TResult> Load<TContext, TResult>(ScriptSource source, ScriptPeCacheStore? cache,
+            out PeCache how) where TContext : ScriptContext
+        {
+            if (cache is null)
+            {
+                how = PeCache.None;
+                return ScriptEvaluator<TContext, TResult>.Compile(source.Source);
+            }
+
+            var id = ScriptPeCacheStore.EntryId(ScriptEvaluator<TContext, TResult>.CacheFingerprint, (int)source.Kind, source.Source);
+            var lookup = cache.TryRead(id, out var image);
+            if (lookup == PeCacheLookup.Hit)
+            {
+                try
+                {
+                    how = PeCache.Hit;
+                    return ScriptEvaluator<TContext, TResult>.FromImage(image!);
+                }
+                catch (Exception ex) when (ex is BadImageFormatException or InvalidOperationException or TypeLoadException or MissingMethodException)
+                {
+                    // Authenticated but unloadable: only possible across a runtime change the fingerprint missed.
+                    try { File.Delete(cache.EntryPath(id)); } catch (IOException) { }
+                    lookup = PeCacheLookup.Rejected;
+                }
+            }
+
+            var compiled = ScriptEvaluator<TContext, TResult>.CompileToImage(source.Source);
+            cache.TryWrite(id, compiled);
+            how = lookup == PeCacheLookup.Rejected ? PeCache.Rejected : PeCache.Miss;
+            return ScriptEvaluator<TContext, TResult>.FromImage(compiled);
+        }
+
         public object? Run(IReadOnlyDictionary<string, DataValueSnapshot> inputs, ILogger logger, Func<DateTime> clock,
             CancellationToken ct, out bool isPredicate)
         {