feat(scripting): PE cache, quarantine-race fix, recovery, readiness for the out-of-process runner (lmxopcua#780 P1)

Second P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md; still opt-in behind
Scripting:Runner=OutOfProcess (InProcess unchanged).

- Compiled-PE cache (design §7, §14 #8): Scripting:Worker:PeCacheDir holds <dir>/pe/<id>.pe entries authenticated
  with HMAC-SHA256 under a per-node random key (<dir>/pe-cache.key, created by the host, 0600/0700 on Unix), handed to
  the worker in Hello. Entry id = SHA-256(compile fingerprint + kind + source); the fingerprint covers the Roslyn
  version, the Core.Scripting build (analyzer, wrapper, stack guard), the sandbox reference set and the runtime. Edited,
  truncated, other-key or renamed entries are rejected, deleted and recompiled; the worker never loads unverified bytes.
  Pruned by recency at startup (30 d / 50 000). ScriptEvaluator splits into CompileToImage + FromImage. Measured on
  40 scripts, restart to every first result: macOS 820 ms no cache / 840 ms empty / 85 ms warm;
  windev (Windows) 9.5 s / 9.4 s / 1.0 s.
- Quarantine race: WorkerConnection runs exit attribution synchronously BEFORE completing the evaluations/loads it held
  (and before Exited), and the shard passes it straight to the runner — a kill strike is in place before the killed
  key's slot is released.
- Supervision: a dead worker restarts by itself after its backoff; Unavailable outcomes carry ScriptRunOutcome.Recovery
  (-> VirtualTagEvalResult.RetryWhen) and the VT actor / Calculation driver re-evaluate once against current inputs when
  it completes. Isolation worker stops after Scripting:Worker:IsolationIdleTimeout (10 min). Keys compiling when a
  worker dies are suspects too (isolation finds a poison compile and quarantines it). Startup orphan sweep by launch
  marker: kills only a process whose marker's host is gone and whose pid + start time + image match.
- Crash-loop circuit (Scripting:Worker:CircuitOpenAfter, 2 min): runner Unavailable, one Error, evaluations answered
  at once while shards keep retrying at the cap.
- Observability (§10): otopcua.script.runner.state, worker.up/inflight/queue_depth/heap_bytes (shard),
  ipc.roundtrip + eval.duration (kind), compile{cache}, isolation.suspects, exits reason=idle. `script-runner` health
  check on the ready tier, at worst Degraded (§14 #12: degraded but still ready).

Left on #780: self-contained worker publish, installer (account, pipe ACL, firewall), docker-dev live drill,
scripted-alarm seam, the §4.5 per-seam batch submit, the §8 perf harness.

Tests: SupervisionTests, PeCacheStoreTests/PeCacheWorkerTests (real workers), ScriptRunnerHealthCheckTests, VT + Calc
recovery seam tests; 14 mutations each turned their test red.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV

src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/OutOfProcessScriptRunner.cs

@@ -1,17 +1,45 @@
 using System.Collections.Concurrent;
+using System.Diagnostics;
+using System.Diagnostics.Metrics;
 using System.Security.Cryptography;
 using System.Text;
 using Serilog;
-using Serilog.Core;
 using Serilog.Events;
 using ZB.MOM.WW.OtOpcUa.Commons.Engines;
+using ZB.MOM.WW.OtOpcUa.Commons.Observability;
 using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
 using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
 
 namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
 
+/// <summary>The out-of-process runner's health (the <c>otopcua.script.runner.state</c> gauge: 0 / 1 / 2).</summary>
+public enum ScriptRunnerState
+{
+    /// <summary>Every shard that has been needed is up.</summary>
+    Ready = 0,
+
+    /// <summary>At least one shard (or the isolation worker) is down after a failure; the rest serve.</summary>
+    Degraded = 1,
+
+    /// <summary>The crash-loop circuit is open: every shard has been down for <c>CircuitOpenAfter</c> (design §5.3.6).</summary>
+    Unavailable = 2,
+}
+
+/// <summary>A point-in-time view of the runner (the readiness check and diagnostics).</summary>
+/// <param name="State">The overall state.</param>
+/// <param name="Shards">Every shard, the isolation worker last.</param>
+/// <param name="SuspectCount">Keys currently routed to the isolation worker.</param>
+/// <param name="QuarantinedCount">Keys currently quarantined (all seams).</param>
+/// <param name="PeCacheEnabled">Whether the compiled-PE cache is in use.</param>
+public sealed record ScriptRunnerStatus(
+    ScriptRunnerState State,
+    IReadOnlyList<ScriptWorkerShardStatus> Shards,
+    int SuspectCount,
+    int QuarantinedCount,
+    bool PeCacheEnabled);
+
 /// <summary>
-///     The pre-emptible <see cref="IScriptRunner"/> (lmxopcua#780, design §3–§9): compiled scripts run in a pool of
+///     The pre-emptible <see cref="IScriptRunner"/> (lmxopcua#780, design §3–§10): compiled scripts run in a pool of
 ///     <c>ScriptWorker</c> processes — K shards placed by a stable hash of the script key, plus an isolation worker
 ///     for suspects — so a script that overruns its hard deadline is stopped by killing its worker, and an OOM, a
 ///     native crash or a fail-fast takes down a worker instead of the server.
@@ -21,16 +49,22 @@ namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
 ///         <item><b>Two deadlines.</b> At the soft deadline <c>T</c> the caller gets today's timeout failure (and the
 ///         run token is cancelled inside the worker). At <c>T + G</c> (<see cref="ScriptWorkerOptions.EffectiveHardGrace"/>)
 ///         the worker is killed and the key takes a kill strike.</item>
-///         <item><b>Attribution.</b> A deadline kill blames exactly one key; the other evaluations in flight on that
-///         worker are victims and report <see cref="ScriptRunOutcome.WorkerRestartedReason"/> (no strike). A crash with
-///         no deadline crossed makes every key in flight a suspect, and suspects run in the isolation worker one at a
-///         time, where a crash names exactly one key. A suspect returns to its shard after
+///         <item><b>Attribution</b> runs synchronously in the worker's exit path, before any evaluation it held is
+///         completed — so a quarantine earned by a kill is in place before the key's slot is released and the next
+///         trigger is admitted. A deadline kill blames exactly one key; the other evaluations in flight on that worker
+///         are victims and report <see cref="ScriptRunOutcome.WorkerRestartedReason"/> (no strike). A crash with no
+///         deadline crossed makes every key in flight — evaluating or compiling — a suspect, and suspects run (and
+///         compile) in the isolation worker one at a time, where a crash names exactly one key; that is also how a
+///         script whose compile keeps crashing its worker is found. A suspect returns to its shard after
 ///         <see cref="ScriptWorkerOptions.IsolationProbationRuns"/> clean runs.</item>
 ///         <item><b>Quarantine</b> lives here, in the host, so it survives worker restarts: kill strikes (within an
 ///         hour), consecutive soft timeouts, or one allocation-budget breach. Reasons keep the
 ///         <see cref="ScriptRunGuard.QuarantinedReasonPrefix"/>, and a source change or <see cref="Reset"/> lifts them.</item>
 ///         <item><b>Fail closed.</b> When no worker is ready within <see cref="ScriptWorkerOptions.UnavailableGrace"/>
-///         the result is <see cref="ScriptRunStatus.Unavailable"/> — never an in-process run.</item>
+///         the result is <see cref="ScriptRunStatus.Unavailable"/> — never an in-process run — carrying a
+///         <see cref="ScriptRunOutcome.Recovery"/> task the seam re-evaluates on.</item>
+///         <item><b>PE cache</b> (design §7): with <see cref="ScriptWorkerOptions.PeCacheDir"/> set, workers load
+///         HMAC-verified compiled images instead of re-entering Roslyn after a restart.</item>
 ///     </list>
 /// </remarks>
 public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable, IDisposable
@@ -44,8 +78,52 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
     /// <summary>Failure reason when the worker could not take more work.</summary>
     public const string WorkerBusyReason = "script worker busy";
 
+    /// <summary>PE cache entries unused for this long are pruned at startup.</summary>
+    internal static readonly TimeSpan PeCacheMaxAge = TimeSpan.FromDays(30);
+
+    /// <summary>At most this many PE cache entries are kept.</summary>
+    internal const int PeCacheMaxEntries = 50_000;
+
     private static readonly TimeSpan KillStrikeWindow = TimeSpan.FromHours(1);
 
+    private static readonly ConcurrentDictionary<OutOfProcessScriptRunner, byte> LiveRunners = new();
+
+    private static readonly Counter<long> CompileCounter = OtOpcUaTelemetry.Meter.CreateCounter<long>(
+        "otopcua.script.compile", unit: "{compile}",
+        description: "Out-of-process script loads (tag: cache = hit | miss | rejected | uncached | negative).");
+
+    private static readonly Histogram<double> RoundTripHistogram = OtOpcUaTelemetry.Meter.CreateHistogram<double>(
+        "otopcua.script.ipc.roundtrip", unit: "us",
+        description: "Host-measured submit-to-result time of one out-of-process evaluation (tag: kind).");
+
+    private static readonly Histogram<double> EvalDurationHistogram = OtOpcUaTelemetry.Meter.CreateHistogram<double>(
+        "otopcua.script.eval.duration", unit: "us",
+        description: "Worker-measured run time of one out-of-process evaluation (tag: kind).");
+
+    static OutOfProcessScriptRunner()
+    {
+        // Gauges observe every live runner (one per node in production; tests make several).
+        OtOpcUaTelemetry.Meter.CreateObservableGauge("otopcua.script.runner.state",
+            static () => LiveRunners.Keys.Select(r => new Measurement<int>((int)r.State)),
+            unit: "{state}", description: "Out-of-process script runner state: 0 ready, 1 degraded, 2 unavailable (circuit open).");
+        OtOpcUaTelemetry.Meter.CreateObservableGauge("otopcua.script.worker.up",
+            static () => ShardMeasurements(s => s.Up ? 1 : 0),
+            unit: "{process}", description: "1 when the shard's ScriptWorker is live (tag: shard).");
+        OtOpcUaTelemetry.Meter.CreateObservableGauge("otopcua.script.worker.inflight",
+            static () => ShardMeasurements(s => s.InFlight),
+            unit: "{eval}", description: "Evaluations outstanding on the shard's ScriptWorker (tag: shard).");
+        OtOpcUaTelemetry.Meter.CreateObservableGauge("otopcua.script.worker.queue_depth",
+            static () => ShardMeasurements(s => s.Queued),
+            unit: "{eval}", description: "Evaluations queued inside the shard's ScriptWorker, from its heartbeat (tag: shard).");
+        OtOpcUaTelemetry.Meter.CreateObservableGauge("otopcua.script.worker.heap_bytes",
+            static () => LiveRunners.Keys.SelectMany(r => r.AllShards.Select(s => s.Status()))
+                .Select(s => new Measurement<long>(s.HeapBytes, new KeyValuePair<string, object?>("shard", s.Label))),
+            unit: "By", description: "The shard's ScriptWorker managed heap, from its heartbeat (tag: shard).");
+        OtOpcUaTelemetry.Meter.CreateObservableGauge("otopcua.script.isolation.suspects",
+            static () => LiveRunners.Keys.Select(r => new Measurement<int>(r.SuspectKeys.Count)),
+            unit: "{script}", description: "Scripts currently routed to the isolation worker.");
+    }
+
     private readonly ScriptingOptions _options;
     private readonly ScriptWorkerOptions _worker;
     private readonly ILogger _log;
@@ -54,6 +132,15 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
     private readonly WorkerShard _isolation;
     private readonly ConcurrentDictionary<string, KeyState> _keys = new(StringComparer.Ordinal);
     private readonly ConcurrentDictionary<string, string> _compileFailures = new(StringComparer.Ordinal);
+    private readonly Timer _maintenance;
+    private readonly string _markerDirectory;
+    private readonly string? _peCacheDirectory;
+    private readonly byte[]? _peCacheKey;
+    private long _peCacheHits;
+    private long _peCacheMisses;
+    private long _peCacheRejected;
+    private int _maintaining;
+    private int _circuitLogged;
     private int _disposed;
 
     /// <summary>Initializes the runner. No process starts until the first evaluation.</summary>
@@ -66,7 +153,7 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
     }
 
     internal OutOfProcessScriptRunner(ScriptingOptions options, ILogger? log, ILogger? scriptRoot,
-        TimeSpan heartbeatInterval, TimeSpan loadTimeout)
+        TimeSpan heartbeatInterval, TimeSpan loadTimeout, TimeSpan? maintenanceInterval = null, string? markerDirectory = null)
     {
         ArgumentNullException.ThrowIfNull(options);
         _options = options.Sanitized(out _);
@@ -76,6 +163,32 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
         LoadTimeout = loadTimeout;
         WorkerPath = ResolveWorkerPath(_worker);
 
+        // The PE cache is best effort: a directory or key that cannot be used is a Warning and a cache-less runner,
+        // never a startup failure (the path RULE is enforced at startup by ScriptRunnerSelection.Validate).
+        if (!string.IsNullOrWhiteSpace(_worker.PeCacheDir))
+        {
+            try
+            {
+                _peCacheKey = ScriptPeCacheStore.LoadOrCreateKey(_worker.PeCacheDir);
+                _peCacheDirectory = _worker.PeCacheDir;
+                var pruned = new ScriptPeCacheStore(_peCacheDirectory, _peCacheKey).Prune(PeCacheMaxAge, PeCacheMaxEntries);
+                if (pruned > 0) _log.Information("Script PE cache {Directory}: pruned {Count} stale entr(ies)", _peCacheDirectory, pruned);
+            }
+            catch (Exception ex) when (ex is IOException or UnauthorizedAccessException)
+            {
+                _peCacheKey = null;
+                _peCacheDirectory = null;
+                _log.Warning(ex, "Script PE cache {Directory} is unusable; ScriptWorkers will recompile every script on each start",
+                    _worker.PeCacheDir);
+            }
+        }
+
+        // Design §3.3: workers a dead predecessor of THIS host instance left behind, identified by their launch
+        // markers — never by process name.
+        _markerDirectory = markerDirectory ?? WorkerLaunchMarker.DirectoryFor(_worker.PeCacheDir);
+        var swept = WorkerLaunchMarker.Sweep(_markerDirectory, _log);
+        if (swept.Count > 0) _log.Warning("Killed {Count} orphaned ScriptWorker process(es) at startup: {Pids}", swept.Count, swept);
+
         var evalThreads = Math.Max(2, Environment.ProcessorCount / _worker.Shards);
         _shards = new WorkerShard[_worker.Shards];
         for (var i = 0; i < _shards.Length; i++)
@@ -84,11 +197,27 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
             _shards[i] = NewShard(label, isolation: false, evalThreads, heartbeatInterval);
         }
         _isolation = NewShard("iso", isolation: true, evalThreads: 1, heartbeatInterval);
+
+        var tick = maintenanceInterval ?? TimeSpan.FromSeconds(1);
+        _maintenance = new Timer(_ => _ = MaintainAsync(), null, tick, tick);
+        LiveRunners.TryAdd(this, 0);
     }
 
     /// <summary>The worker entry assembly this runner launches.</summary>
     public string WorkerPath { get; }
 
+    /// <summary>The overall state (the readiness check, the state gauge).</summary>
+    public ScriptRunnerState State => ComputeState(_shards.Select(s => s.DownFor).ToArray(), _isolation.DownFor, _worker.CircuitOpenAfter);
+
+    /// <summary>A snapshot of the runner for diagnostics and health.</summary>
+    /// <returns>The status.</returns>
+    public ScriptRunnerStatus Status() => new(
+        State,
+        AllShards.Select(s => s.Status()).ToArray(),
+        SuspectKeys.Count,
+        _keys.Count(kv => kv.Value.QuarantineReason is not null),
+        _peCacheDirectory is not null);
+
     /// <summary>How long a compile in the worker may take before the worker is killed.</summary>
     internal TimeSpan LoadTimeout { get; }
 
@@ -98,10 +227,22 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
     /// <summary>The isolation worker (tests).</summary>
     internal WorkerShard Isolation => _isolation;
 
+    /// <summary>The marker directory this runner writes launch markers to and sweeps at startup.</summary>
+    internal string MarkerDirectory => _markerDirectory;
+
+    /// <summary>PE cache outcomes reported by this runner's workers (tests / diagnostics).</summary>
+    internal (long Hits, long Misses, long Rejected) PeCacheCounts =>
+        (Interlocked.Read(ref _peCacheHits), Interlocked.Read(ref _peCacheMisses), Interlocked.Read(ref _peCacheRejected));
+
+    /// <summary>Test hook: runs inside the continuation that releases a key's single-flight slot, just before it does.</summary>
+    internal Action<string>? SlotReleasingForTest { get; set; }
+
     /// <summary>Scoped keys currently treated as suspects (tests / diagnostics).</summary>
     internal IReadOnlyCollection<string> SuspectKeys =>
         _keys.Where(kv => kv.Value.Suspect).Select(kv => kv.Key).ToArray();
 
+    private IEnumerable<WorkerShard> AllShards => _shards.Append(_isolation);
+
     /// <summary>The shard a key is placed on when it is not a suspect: a stable FNV-1a hash, so placement survives restarts.</summary>
     /// <param name="scopedKey">The scoped key.</param>
     /// <param name="shardCount">K.</param>
@@ -117,6 +258,23 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
         return (int)(hash % (uint)shardCount);
     }
 
+    /// <summary>
+    ///     The state rule: Unavailable when every shard has been down for at least <paramref name="circuitOpenAfter"/>,
+    ///     Degraded when any shard or the isolation worker is down, Ready otherwise.
+    /// </summary>
+    /// <param name="shardsDownFor">Each regular shard's down time (null = up / never needed).</param>
+    /// <param name="isolationDownFor">The isolation worker's.</param>
+    /// <param name="circuitOpenAfter">The circuit threshold.</param>
+    /// <returns>The state.</returns>
+    internal static ScriptRunnerState ComputeState(IReadOnlyList<TimeSpan?> shardsDownFor, TimeSpan? isolationDownFor, TimeSpan circuitOpenAfter)
+    {
+        if (shardsDownFor.Count > 0 && shardsDownFor.All(d => d is { } down && down >= circuitOpenAfter))
+            return ScriptRunnerState.Unavailable;
+        return shardsDownFor.Any(d => d is not null) || isolationDownFor is not null
+            ? ScriptRunnerState.Degraded
+            : ScriptRunnerState.Ready;
+    }
+
     /// <summary>The default worker path, or the configured one.</summary>
     /// <param name="worker">The worker options.</param>
     /// <returns>The absolute worker path.</returns>
@@ -137,7 +295,11 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
 
         var sha = SHA256.HashData(Encoding.UTF8.GetBytes(invocation.Source));
         var failureKey = InProcessScriptRunner.ScopedKey(invocation.Kind, Convert.ToHexString(sha));
-        if (_compileFailures.TryGetValue(failureKey, out var knownFailure)) return ScriptRunOutcome.Failed(knownFailure);
+        if (_compileFailures.TryGetValue(failureKey, out var knownFailure))
+        {
+            CompileCounter.Add(1, new KeyValuePair<string, object?>("cache", "negative"));
+            return ScriptRunOutcome.Failed(knownFailure);
+        }
 
         if (Interlocked.CompareExchange(ref state.InFlight, 1, 0) != 0)
         {
@@ -147,19 +309,23 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
 
         var handedOff = false;
         SemaphoreSlim? gate = null;
+        var shard = state.Suspect ? _isolation : _shards[ShardIndex(key, _shards.Length)];
+        WorkerConnection? connection = null;
         try
         {
             state.ScriptLogger = invocation.ScriptLogger;
-            var shard = state.Suspect ? _isolation : _shards[ShardIndex(key, _shards.Length)];
+            // Circuit open (design §5.3.6): answer at once instead of making every trigger wait out the grace. A zero
+            // grace still kicks a start that is due, so the shards keep retrying at the backoff cap.
+            var grace = State == ScriptRunnerState.Unavailable ? TimeSpan.Zero : _worker.UnavailableGrace;
             if (shard.IsolationGate is { } isolationGate)
             {
-                if (!await isolationGate.WaitAsync(_worker.UnavailableGrace, ct).ConfigureAwait(false))
-                    return ScriptRunOutcome.Unavailable(ScriptRunOutcome.RunnerUnavailableReason);
+                if (!await isolationGate.WaitAsync(grace, ct).ConfigureAwait(false))
+                    return Unavailable(ScriptRunOutcome.RunnerUnavailableReason, shard, null);
                 gate = isolationGate;
             }
 
-            var connection = await shard.GetReadyAsync(_worker.UnavailableGrace, ct).ConfigureAwait(false);
-            if (connection is null) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.RunnerUnavailableReason);
+            connection = await shard.GetReadyAsync(grace, ct).ConfigureAwait(false);
+            if (connection is null) return Unavailable(ScriptRunOutcome.RunnerUnavailableReason, shard, null);
 
             var paths = invocation.Inputs.Keys.Order(StringComparer.Ordinal).ToArray();
             WorkerConnection.LoadEntry? entry;
@@ -178,7 +344,8 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
                 connection.Kill(WorkerExitReason.KilledLoadTimeout, key);
                 return ScriptRunOutcome.Failed($"compile failure: compile did not finish within {LoadTimeout.TotalSeconds:F0} s");
             }
-            if (entry?.Outcome is not { } outcome) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason);
+            if (entry?.Outcome is not { } outcome) return Unavailable(ScriptRunOutcome.WorkerRestartedReason, RouteFor(key, state), connection);
+            NoteLoad(entry);
             if (!outcome.Ok)
             {
                 var reason = outcome.Error?.Kind switch
@@ -192,16 +359,19 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
                 return ScriptRunOutcome.Failed(reason);
             }
 
+            var submitted = Stopwatch.GetTimestamp();
             var pending = connection.Submit(key, BuildRequest(key, entry, invocation));
             ArmHardDeadline(connection, pending, invocation.Timeout + _worker.EffectiveHardGrace(invocation.Timeout));
 
             // From here the in-flight slot (and the isolation gate) belong to the evaluation itself: they are released
             // when the worker answers or dies — not when this caller stops waiting at the soft deadline. That keeps a
-            // key single-flight between T and T + G, and the isolation worker strictly one-at-a-time.
+            // key single-flight between T and T + G, and the isolation worker strictly one-at-a-time. On a worker
+            // death, attribution (strike, quarantine, suspect) has already run when this continuation fires.
             handedOff = true;
             var heldGate = gate;
             _ = pending.Completion.ContinueWith(_ =>
             {
+                SlotReleasingForTest?.Invoke(key);
                 Interlocked.Exchange(ref state.InFlight, 0);
                 heldGate?.Release();
             }, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
@@ -221,8 +391,11 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
                 return ScriptRunOutcome.Failed($"{ScriptEvalFailureReasons.TimedOutPrefix}{invocation.Timeout.TotalSeconds:F1}s");
             }
 
-            if (result is null) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason);
-            return Complete(key, state, result);
+            if (result is null) return Unavailable(ScriptRunOutcome.WorkerRestartedReason, RouteFor(key, state), connection);
+            var kindTag = new KeyValuePair<string, object?>("kind", KindTag(invocation.Kind));
+            RoundTripHistogram.Record(Stopwatch.GetElapsedTime(submitted).TotalMicroseconds, kindTag);
+            EvalDurationHistogram.Record(result.RunMicros, kindTag);
+            return Complete(key, state, result, shard, connection);
         }
         catch (OperationCanceledException) when (ct.IsCancellationRequested)
         {
@@ -251,7 +424,7 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
         {
             if (key.StartsWith(prefix, StringComparison.Ordinal)) _compileFailures.TryRemove(key, out _);
         }
-        foreach (var shard in _shards.Append(_isolation))
+        foreach (var shard in AllShards)
             shard.Current?.Unload(dropped);
     }
 
@@ -270,12 +443,47 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
     public async ValueTask DisposeAsync()
     {
         if (Interlocked.Exchange(ref _disposed, 1) != 0) return;
-        await Task.WhenAll(_shards.Append(_isolation).Select(s => s.DisposeAsync().AsTask())).ConfigureAwait(false);
+        LiveRunners.TryRemove(this, out _);
+        await _maintenance.DisposeAsync().ConfigureAwait(false);
+        await Task.WhenAll(AllShards.Select(s => s.DisposeAsync().AsTask())).ConfigureAwait(false);
     }
 
     /// <summary>Stops every worker (blocking).</summary>
     public void Dispose() => DisposeAsync().AsTask().GetAwaiter().GetResult();
 
+    /// <summary>
+    ///     The periodic pass: stop the isolation worker once it has been idle for
+    ///     <see cref="ScriptWorkerOptions.IsolationIdleTimeout"/>, and log the circuit opening / closing once each.
+    /// </summary>
+    /// <returns>A task for the pass.</returns>
+    internal async Task MaintainAsync()
+    {
+        if (Volatile.Read(ref _disposed) != 0 || Interlocked.Exchange(ref _maintaining, 1) != 0) return;
+        try
+        {
+            await _isolation.StopIfIdleAsync(_worker.IsolationIdleTimeout).ConfigureAwait(false);
+
+            var open = State == ScriptRunnerState.Unavailable;
+            if (open && Interlocked.Exchange(ref _circuitLogged, 1) == 0)
+            {
+                _log.Error("Script runner circuit OPEN: every ScriptWorker shard has been down for at least {After}; every compiled script on this node reports {Reason} until a worker starts (retrying at the backoff cap)",
+                    _worker.CircuitOpenAfter, ScriptRunOutcome.RunnerUnavailableReason);
+            }
+            else if (!open && Interlocked.Exchange(ref _circuitLogged, 0) == 1)
+            {
+                _log.Information("Script runner circuit closed: a ScriptWorker shard is serving again");
+            }
+        }
+        catch (Exception ex)
+        {
+            _log.Warning(ex, "Script runner maintenance pass failed");
+        }
+        finally
+        {
+            Interlocked.Exchange(ref _maintaining, 0);
+        }
+    }
+
     private WorkerShard NewShard(string label, bool isolation, int evalThreads, TimeSpan heartbeatInterval)
     {
         var spec = new WorkerLaunchSpec(
@@ -285,12 +493,45 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
             EvalThreads: evalThreads,
             HardGrace: _worker.HardGrace ?? TimeSpan.FromSeconds(Math.Max(1, _options.RunTimeout.TotalSeconds)),
             HeartbeatInterval: heartbeatInterval,
-            HandshakeTimeout: TimeSpan.FromSeconds(15));
+            HandshakeTimeout: TimeSpan.FromSeconds(15),
+            PeCacheDirectory: _peCacheDirectory,
+            PeCacheKey: _peCacheKey,
+            MarkerDirectory: _markerDirectory);
         return new WorkerShard(label, isolation,
-            ct => WorkerConnection.StartAsync(spec, _log, OnLogs, ct),
+            (onExiting, ct) => WorkerConnection.StartAsync(spec, _log, OnLogs, onExiting, ct),
             OnWorkerExit, _log);
     }
 
+    /// <summary>The shard a key's NEXT evaluation goes to (attribution may just have made it a suspect).</summary>
+    private WorkerShard RouteFor(string key, KeyState state)
+        => state.Suspect ? _isolation : _shards[ShardIndex(key, _shards.Length)];
+
+    private static ScriptRunOutcome Unavailable(string reason, WorkerShard shard, WorkerConnection? failed)
+        => ScriptRunOutcome.Unavailable(reason) with { Recovery = shard.WhenRecoveredAsync(failed) };
+
+    private void NoteLoad(WorkerConnection.LoadEntry entry)
+    {
+        // Counted once per compile, by the caller that waited for it (an already-loaded key is not a load).
+        if (Interlocked.Exchange(ref entry.Counted, 1) != 0 || entry.Outcome is not { Ok: true } outcome) return;
+        var tag = outcome.Cache switch
+        {
+            PeCache.Hit => "hit",
+            PeCache.Miss => "miss",
+            PeCache.Rejected => "rejected",
+            _ => "uncached",
+        };
+        CompileCounter.Add(1, new KeyValuePair<string, object?>("cache", tag));
+        switch (outcome.Cache)
+        {
+            case PeCache.Hit: Interlocked.Increment(ref _peCacheHits); break;
+            case PeCache.Miss: Interlocked.Increment(ref _peCacheMisses); break;
+            case PeCache.Rejected:
+                Interlocked.Increment(ref _peCacheRejected);
+                _log.Warning("Script {ScriptKey}: its PE cache entry failed verification and was deleted; recompiled", outcome.Key);
+                break;
+        }
+    }
+
     private KeyState Admit(string key, string source)
     {
         while (true)
@@ -355,7 +596,7 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
         }, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
     }
 
-    private ScriptRunOutcome Complete(string key, KeyState state, EvalResult result)
+    private ScriptRunOutcome Complete(string key, KeyState state, EvalResult result, WorkerShard shard, WorkerConnection connection)
     {
         // The script returned control (a value or a throw): the soft-timeout streak resets, as in process.
         Interlocked.Exchange(ref state.ConsecutiveTimeouts, 0);
@@ -396,7 +637,7 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
                     ScriptError.Types.Kind.Threw => ScriptRunOutcome.Failed($"script threw: {result.Error.Message}"),
                     ScriptError.Types.Kind.UnsupportedType => ScriptRunOutcome.Failed($"unsupported type: {result.Error.Message}"),
                     ScriptError.Types.Kind.Busy => ScriptRunOutcome.Failed(WorkerBusyReason),
-                    ScriptError.Types.Kind.NotLoaded => ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason),
+                    ScriptError.Types.Kind.NotLoaded => Unavailable(ScriptRunOutcome.WorkerRestartedReason, shard, connection),
                     _ => ScriptRunOutcome.Failed($"script failed: {result.Error.Message}"),
                 };
             default:
@@ -415,7 +656,14 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
         return true;
     }
 
-    private void OnWorkerExit(WorkerShard shard, WorkerExit exit)
+    /// <summary>
+    ///     Attribution for one worker exit. Called synchronously from the worker's exit path BEFORE any evaluation or
+    ///     compile it held is completed (see <see cref="WorkerConnection.StartAsync"/>), so its verdict is in place
+    ///     before the affected keys' single-flight slots are released.
+    /// </summary>
+    /// <param name="shard">The shard whose worker ended.</param>
+    /// <param name="exit">How it ended.</param>
+    internal void OnWorkerExit(WorkerShard shard, WorkerExit exit)
     {
         switch (exit.Reason)
         {
@@ -426,22 +674,25 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
                 KillStrike(culprit, "its compile did not finish and its worker was killed", "killed");
                 break;
             case WorkerExitReason.Crash or WorkerExitReason.KilledHeartbeat or WorkerExitReason.ProtocolFault:
+                // Compiles count as well as evaluations: a source that crashes Roslyn (or the load) must be found too.
+                var implicated = exit.InFlightKeys.Concat(exit.LoadingKeys).Distinct(StringComparer.Ordinal).ToArray();
                 if (shard.IsIsolation)
                 {
-                    // One evaluation at a time here: the crash is this key's (design §5.3.3).
-                    foreach (var key in exit.InFlightKeys)
-                        KillStrike(key, "it crashed the isolation worker", "isolated_crash");
+                    // One evaluation (or compile) at a time here: the crash is this key's (design §5.3.3).
+                    foreach (var key in implicated)
+                        KillStrike(key, exit.InFlightKeys.Contains(key) ? "it crashed the isolation worker" : "its compile crashed the isolation worker",
+                            "isolated_crash");
                 }
-                else if (exit.InFlightKeys.Count > 0)
+                else if (implicated.Length > 0)
                 {
-                    foreach (var key in exit.InFlightKeys)
+                    foreach (var key in implicated)
                     {
                         if (!_keys.TryGetValue(key, out var state)) continue;
                         state.Suspect = true;
                         Interlocked.Exchange(ref state.ProbationRuns, 0);
                     }
                     _log.Warning("ScriptWorker[{Shard}] ended ({Reason}) with no deadline crossed; {Count} script(s) in flight move to the isolation worker: {Suspects}",
-                        shard.Label, exit.Reason, exit.InFlightKeys.Count, string.Join(", ", exit.InFlightKeys));
+                        shard.Label, exit.Reason, implicated.Length, string.Join(", ", implicated));
                 }
                 break;
         }
@@ -491,6 +742,18 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
             _log.Warning("ScriptWorker dropped {Dropped} script log line(s): a script is logging faster than the pipe drains", batch.Dropped);
     }
 
+    private static string KindTag(ScriptContextKind kind) => kind switch
+    {
+        ScriptContextKind.VirtualTag => "virtual_tag",
+        ScriptContextKind.Calculation => "calculation",
+        ScriptContextKind.AlarmPredicate => "alarm",
+        _ => "other",
+    };
+
+    private static IEnumerable<Measurement<int>> ShardMeasurements(Func<ScriptWorkerShardStatus, int> value)
+        => LiveRunners.Keys.SelectMany(r => r.AllShards.Select(s => s.Status()))
+            .Select(s => new Measurement<int>(value(s), new KeyValuePair<string, object?>("shard", s.Label)));
+
     private sealed class KeyState(string source)
     {
         public readonly string Source = source;