feat(scripting): PE cache, quarantine-race fix, recovery, readiness for the out-of-process runner (lmxopcua#780 P1)

Second P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md; still opt-in behind
Scripting:Runner=OutOfProcess (InProcess unchanged).

- Compiled-PE cache (design §7, §14 #8): Scripting:Worker:PeCacheDir holds <dir>/pe/<id>.pe entries authenticated
  with HMAC-SHA256 under a per-node random key (<dir>/pe-cache.key, created by the host, 0600/0700 on Unix), handed to
  the worker in Hello. Entry id = SHA-256(compile fingerprint + kind + source); the fingerprint covers the Roslyn
  version, the Core.Scripting build (analyzer, wrapper, stack guard), the sandbox reference set and the runtime. Edited,
  truncated, other-key or renamed entries are rejected, deleted and recompiled; the worker never loads unverified bytes.
  Pruned by recency at startup (30 d / 50 000). ScriptEvaluator splits into CompileToImage + FromImage. Measured on
  40 scripts, restart to every first result: macOS 820 ms no cache / 840 ms empty / 85 ms warm;
  windev (Windows) 9.5 s / 9.4 s / 1.0 s.
- Quarantine race: WorkerConnection runs exit attribution synchronously BEFORE completing the evaluations/loads it held
  (and before Exited), and the shard passes it straight to the runner — a kill strike is in place before the killed
  key's slot is released.
- Supervision: a dead worker restarts by itself after its backoff; Unavailable outcomes carry ScriptRunOutcome.Recovery
  (-> VirtualTagEvalResult.RetryWhen) and the VT actor / Calculation driver re-evaluate once against current inputs when
  it completes. Isolation worker stops after Scripting:Worker:IsolationIdleTimeout (10 min). Keys compiling when a
  worker dies are suspects too (isolation finds a poison compile and quarantines it). Startup orphan sweep by launch
  marker: kills only a process whose marker's host is gone and whose pid + start time + image match.
- Crash-loop circuit (Scripting:Worker:CircuitOpenAfter, 2 min): runner Unavailable, one Error, evaluations answered
  at once while shards keep retrying at the cap.
- Observability (§10): otopcua.script.runner.state, worker.up/inflight/queue_depth/heap_bytes (shard),
  ipc.roundtrip + eval.duration (kind), compile{cache}, isolation.suspects, exits reason=idle. `script-runner` health
  check on the ready tier, at worst Degraded (§14 #12: degraded but still ready).

Left on #780: self-contained worker publish, installer (account, pipe ACL, firewall), docker-dev live drill,
scripted-alarm seam, the §4.5 per-seam batch submit, the §8 perf harness.

Tests: SupervisionTests, PeCacheStoreTests/PeCacheWorkerTests (real workers), ScriptRunnerHealthCheckTests, VT + Calc
recovery seam tests; 14 mutations each turned their test red.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV

src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/WorkerShard.cs

@@ -6,13 +6,39 @@ using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
 
 namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
 
+/// <summary>A point-in-time view of one shard (diagnostics, the readiness check, the runner-state gauge).</summary>
+/// <param name="Label">The shard label (<c>0</c>…<c>K-1</c>, or <c>iso</c>).</param>
+/// <param name="IsIsolation">Whether it is the isolation worker.</param>
+/// <param name="Up">A worker is live and serving.</param>
+/// <param name="DownFor">How long it has been down after a failure; null when up or never needed.</param>
+/// <param name="Pid">The live worker's pid.</param>
+/// <param name="ConsecutiveFailures">Start failures and crashes since the last healthy period.</param>
+/// <param name="InFlight">Evaluations outstanding on the live worker.</param>
+/// <param name="Queued">Evaluations queued in the worker (its last heartbeat).</param>
+/// <param name="HeapBytes">The worker's managed heap (its last heartbeat).</param>
+public sealed record ScriptWorkerShardStatus(
+    string Label,
+    bool IsIsolation,
+    bool Up,
+    TimeSpan? DownFor,
+    int? Pid,
+    int ConsecutiveFailures,
+    int InFlight,
+    int Queued,
+    long HeapBytes);
+
 /// <summary>
 ///     Supervises one shard's worker process (design §3.3, §5.3.5): starts it lazily on first use, reports its end to
-///     the runner (for attribution), and restarts it after an exponential backoff — 250 ms doubling to a 30 s cap with
-///     ±20 % jitter, reset once a worker has stayed up for 60 s. It never runs a script in the host when the worker
-///     cannot start (fail closed, §14 #4): <see cref="GetReadyAsync"/> returns null and the caller reports
-///     <c>BadResourceUnavailable</c>.
+///     the runner (for attribution, synchronously — see <see cref="WorkerConnection.StartAsync"/>), and restarts it after
+///     an exponential backoff — 250 ms doubling to a 30 s cap with ±20 % jitter, reset once a worker has stayed up for
+///     60 s. It never runs a script in the host when the worker cannot start (fail closed, §14 #4):
+///     <see cref="GetReadyAsync"/> returns null and the caller reports <c>BadResourceUnavailable</c>.
 /// </summary>
+/// <remarks>
+///     A caller that got <c>Unavailable</c> waits on <see cref="WhenRecoveredAsync"/>; that both tells it when to
+///     re-evaluate and makes the shard restart proactively (with backoff) instead of waiting for the next trigger, so
+///     victims of a crash are re-evaluated on the shard's return (design §9).
+/// </remarks>
 internal sealed class WorkerShard : IAsyncDisposable
 {
     internal static readonly TimeSpan InitialBackoff = TimeSpan.FromMilliseconds(250);
@@ -24,20 +50,24 @@ internal sealed class WorkerShard : IAsyncDisposable
 
     private static readonly Counter<long> ExitsCounter = OtOpcUaTelemetry.Meter.CreateCounter<long>(
         "otopcua.script.worker.exits", unit: "{process}",
-        description: "ScriptWorker processes ended (tags: shard, reason = clean | killed_deadline | killed_heartbeat | killed_load | crash | protocol | disposed).");
+        description: "ScriptWorker processes ended (tags: shard, reason = clean | killed_deadline | killed_heartbeat | killed_load | crash | protocol | idle | disposed).");
 
-    private readonly Func<CancellationToken, Task<WorkerConnection>> _launch;
+    private readonly Func<Action<WorkerExit>, CancellationToken, Task<WorkerConnection>> _launch;
     private readonly Action<WorkerShard, WorkerExit> _onExit;
     private readonly ILogger _log;
     private readonly object _gate = new();
     private readonly Random _jitter = new();
+    private readonly CancellationTokenSource _disposing = new();
     private WorkerConnection? _current;
     private Task<WorkerConnection?>? _starting;
+    private TaskCompletionSource? _recovered;
+    private bool _recovering;
     private long _backoffUntil;
+    private long _downSince;
     private int _consecutiveFailures;
     private bool _disposed;
 
-    public WorkerShard(string label, bool isolation, Func<CancellationToken, Task<WorkerConnection>> launch,
+    public WorkerShard(string label, bool isolation, Func<Action<WorkerExit>, CancellationToken, Task<WorkerConnection>> launch,
         Action<WorkerShard, WorkerExit> onExit, ILogger log)
     {
         Label = label;
@@ -70,9 +100,40 @@ internal sealed class WorkerShard : IAsyncDisposable
         get { lock (_gate) return _consecutiveFailures; }
     }
 
+    /// <summary>How long the shard has been down after a failure; null when it is up or was never needed.</summary>
+    public TimeSpan? DownFor
+    {
+        get
+        {
+            lock (_gate)
+            {
+                if (_current is { IsAlive: true } || _downSince == 0) return null;
+                return Stopwatch.GetElapsedTime(_downSince);
+            }
+        }
+    }
+
+    /// <summary>A snapshot for diagnostics and health.</summary>
+    /// <returns>The status.</returns>
+    public ScriptWorkerShardStatus Status()
+    {
+        WorkerConnection? live;
+        int failures;
+        TimeSpan? downFor;
+        lock (_gate)
+        {
+            live = _current is { IsAlive: true } c ? c : null;
+            failures = _consecutiveFailures;
+            downFor = live is null && _downSince != 0 ? Stopwatch.GetElapsedTime(_downSince) : null;
+        }
+        var heartbeat = live?.LastHeartbeat;
+        return new ScriptWorkerShardStatus(Label, IsIsolation, live is not null, downFor, live?.Pid, failures,
+            live?.InFlightCount ?? 0, (int)(heartbeat?.Queued ?? 0), heartbeat?.HeapBytes ?? 0);
+    }
+
     /// <summary>
     ///     The ready worker, starting one if needed. Waits at most <paramref name="grace"/> — through a start in progress
-    ///     or a backoff — and returns null if no worker is ready by then.
+    ///     or a backoff — and returns null if no worker is ready by then. A zero grace still kicks a start that is due.
     /// </summary>
     /// <param name="grace">How long to wait.</param>
     /// <param name="ct">Cancellation.</param>
@@ -125,18 +186,123 @@ internal sealed class WorkerShard : IAsyncDisposable
         }
     }
 
+    /// <summary>
+    ///     Completes when this shard has a live worker other than <paramref name="failed"/>. Starts a background
+    ///     recovery (start with backoff, retried at the cap) if none is live, so the caller does not depend on a later
+    ///     trigger to bring the shard back.
+    /// </summary>
+    /// <param name="failed">The worker the caller's evaluation died on, if any.</param>
+    /// <returns>A task that completes on recovery (or when the shard is disposed).</returns>
+    public Task WhenRecoveredAsync(WorkerConnection? failed)
+    {
+        lock (_gate)
+        {
+            if (_disposed) return Task.CompletedTask;
+            if (_current is { IsAlive: true } live)
+            {
+                // Still the same worker (it answered NOT_LOADED, or the caller lost a race with a restart that has
+                // already happened): a short pause, so a seam cannot spin on a key the worker keeps refusing.
+                return ReferenceEquals(live, failed) ? Task.Delay(InitialBackoff) : Task.CompletedTask;
+            }
+            _recovered ??= new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
+            var task = _recovered.Task;
+            EnsureRecoveringLocked();
+            return task;
+        }
+    }
+
+    private void EnsureRecoveringLocked()
+    {
+        if (_recovering || _disposed) return;
+        _recovering = true;
+        _ = Task.Run(RecoverAsync);
+    }
+
+    /// <summary>Stops the live worker if it has had nothing to do for <paramref name="idle"/> (the isolation worker).</summary>
+    /// <param name="idle">The idle threshold.</param>
+    /// <returns>True when a worker was stopped.</returns>
+    public async Task<bool> StopIfIdleAsync(TimeSpan idle)
+    {
+        WorkerConnection? victim;
+        // Holding the isolation gate means no evaluation can pick this worker up while it is being stopped.
+        var gate = IsolationGate;
+        try
+        {
+            if (gate is not null && !gate.Wait(0)) return false;
+        }
+        catch (ObjectDisposedException)
+        {
+            return false;
+        }
+        try
+        {
+            lock (_gate)
+            {
+                if (_disposed || _current is not { IsAlive: true } live) return false;
+                if (live.InFlightCount > 0 || live.LoadingCount > 0) return false;
+                if (Stopwatch.GetElapsedTime(live.LastUsedTimestamp) < idle) return false;
+                victim = live;
+                _current = null;
+            }
+            _log.Information("ScriptWorker[{Shard}] pid {Pid} idle for {Idle}; stopping it (it restarts on demand)", Label, victim.Pid, idle);
+            await victim.StopAsync(WorkerExitReason.Idle).ConfigureAwait(false);
+            return true;
+        }
+        finally
+        {
+            gate?.Release();
+        }
+    }
+
+    private async Task RecoverAsync()
+    {
+        try
+        {
+            while (true)
+            {
+                // A long grace: GetReadyAsync waits out the backoff itself and returns null only on a failed start.
+                var live = await GetReadyAsync(MaxBackoff * 2, _disposing.Token).ConfigureAwait(false);
+                lock (_gate)
+                {
+                    if (_disposed || live is not null)
+                    {
+                        _recovering = false;
+                        TakeRecoveredLocked()?.TrySetResult();
+                        return;
+                    }
+                }
+            }
+        }
+        catch (OperationCanceledException)
+        {
+            lock (_gate)
+            {
+                _recovering = false;
+                TakeRecoveredLocked()?.TrySetResult();
+            }
+        }
+    }
+
+    private TaskCompletionSource? TakeRecoveredLocked()
+    {
+        var tcs = _recovered;
+        _recovered = null;
+        return tcs;
+    }
+
     private async Task<WorkerConnection?> StartCoreAsync()
     {
         await Task.Yield();
         WorkerConnection? connection = null;
         try
         {
-            connection = await _launch(CancellationToken.None).ConfigureAwait(false);
+            connection = await _launch(Attribute, CancellationToken.None).ConfigureAwait(false);
             StartsCounter.Add(1, new KeyValuePair<string, object?>("shard", Label));
             _log.Information("ScriptWorker[{Shard}] started, pid {Pid}", Label, connection.Pid);
             var started = connection;
             _ = started.Exited.ContinueWith(t => OnExited(started, t.Result), CancellationToken.None,
                 TaskContinuationOptions.ExecuteSynchronously | TaskContinuationOptions.OnlyOnRanToCompletion, TaskScheduler.Default);
+            TaskCompletionSource? recovered;
             lock (_gate)
             {
                 _starting = null;
@@ -146,7 +312,10 @@ internal sealed class WorkerShard : IAsyncDisposable
                     return null;
                 }
                 _current = started;
+                _downSince = 0;
+                recovered = TakeRecoveredLocked();
             }
+            recovered?.TrySetResult();
             return started;
         }
         catch (Exception ex)
@@ -155,6 +324,7 @@ internal sealed class WorkerShard : IAsyncDisposable
             lock (_gate)
             {
                 _starting = null;
+                MarkDownLocked();
                 ScheduleBackoffLocked();
             }
             if (connection is not null) await connection.DisposeAsync().ConfigureAwait(false);
@@ -162,29 +332,44 @@ internal sealed class WorkerShard : IAsyncDisposable
         }
     }
 
+    /// <summary>Synchronous attribution, from <see cref="WorkerConnection"/>'s exit path, before any evaluation completes.</summary>
+    private void Attribute(WorkerExit exit)
+    {
+        try { _onExit(this, exit); }
+        catch (Exception ex) { _log.Error(ex, "ScriptWorker[{Shard}] exit handler threw", Label); }
+    }
+
     private void OnExited(WorkerConnection connection, WorkerExit exit)
     {
         ExitsCounter.Add(1, new KeyValuePair<string, object?>("shard", Label),
             new KeyValuePair<string, object?>("reason", MetricReason(exit.Reason)));
+        var expected = exit.Reason is WorkerExitReason.Disposed or WorkerExitReason.Idle;
         lock (_gate)
         {
             if (ReferenceEquals(_current, connection)) _current = null;
-            if (!_disposed)
+            if (!_disposed && !expected)
             {
                 if (Stopwatch.GetElapsedTime(connection.ReadyTimestamp) >= HealthyResetAfter) _consecutiveFailures = 0;
+                MarkDownLocked();
                 ScheduleBackoffLocked();
+                // A worker that was serving and died comes back by itself (after the backoff), so the scripts placed
+                // on it find it warm and its victims are re-evaluated on its return (design §9).
+                EnsureRecoveringLocked();
             }
         }
-        if (exit.Reason != WorkerExitReason.Disposed)
+        if (!expected)
         {
-            _log.Warning("ScriptWorker[{Shard}] pid {Pid} ended: {Reason} (exit code {ExitCode}, culprit {Culprit}, {InFlight} evaluation(s) in flight)",
-                Label, connection.Pid, exit.Reason, exit.ExitCode, exit.CulpritKey ?? "none", exit.InFlightKeys.Count);
+            _log.Warning("ScriptWorker[{Shard}] pid {Pid} ended: {Reason} (exit code {ExitCode}, culprit {Culprit}, {InFlight} evaluation(s) and {Loading} compile(s) in flight)",
+                Label, connection.Pid, exit.Reason, exit.ExitCode, exit.CulpritKey ?? "none", exit.InFlightKeys.Count, exit.LoadingKeys.Count);
         }
-        try { _onExit(this, exit); }
-        catch (Exception ex) { _log.Error(ex, "ScriptWorker[{Shard}] exit handler threw", Label); }
         _ = connection.DisposeAsync().AsTask();
     }
 
+    private void MarkDownLocked()
+    {
+        if (_downSince == 0) _downSince = Stopwatch.GetTimestamp();
+    }
+
     private void ScheduleBackoffLocked()
     {
         _consecutiveFailures++;
@@ -212,6 +397,7 @@ internal sealed class WorkerShard : IAsyncDisposable
         WorkerExitReason.KilledLoadTimeout => "killed_load",
         WorkerExitReason.ProtocolFault => "protocol",
         WorkerExitReason.Disposed => "disposed",
+        WorkerExitReason.Idle => "idle",
         _ => "crash",
     };
 
@@ -221,13 +407,18 @@ internal sealed class WorkerShard : IAsyncDisposable
     {
         WorkerConnection? current;
         Task<WorkerConnection?>? starting;
+        TaskCompletionSource? recovered;
         lock (_gate)
         {
+            if (_disposed) return;
             _disposed = true;
             current = _current;
             starting = _starting;
             _current = null;
+            recovered = TakeRecoveredLocked();
         }
+        recovered?.TrySetResult();
+        _disposing.Cancel();
         if (starting is not null)
         {
             try { current ??= await starting.WaitAsync(TimeSpan.FromSeconds(15)).ConfigureAwait(false); } catch { /* shutting down */ }