feat(scripting): PE cache, quarantine-race fix, recovery, readiness for the out-of-process runner (lmxopcua#780 P1)
v2-ci / unit-tests (push) Failing after 1h12m10s

Second P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md; still opt-in behind
Scripting:Runner=OutOfProcess (InProcess unchanged).

- Compiled-PE cache (design §7, §14 #8): Scripting:Worker:PeCacheDir holds <dir>/pe/<id>.pe entries authenticated
  with HMAC-SHA256 under a per-node random key (<dir>/pe-cache.key, created by the host, 0600/0700 on Unix), handed to
  the worker in Hello. Entry id = SHA-256(compile fingerprint + kind + source); the fingerprint covers the Roslyn
  version, the Core.Scripting build (analyzer, wrapper, stack guard), the sandbox reference set and the runtime. Edited,
  truncated, other-key or renamed entries are rejected, deleted and recompiled; the worker never loads unverified bytes.
  Pruned by recency at startup (30 d / 50 000). ScriptEvaluator splits into CompileToImage + FromImage. Measured on
  40 scripts, restart to every first result: macOS 820 ms no cache / 840 ms empty / 85 ms warm;
  windev (Windows) 9.5 s / 9.4 s / 1.0 s.
- Quarantine race: WorkerConnection runs exit attribution synchronously BEFORE completing the evaluations/loads it held
  (and before Exited), and the shard passes it straight to the runner — a kill strike is in place before the killed
  key's slot is released.
- Supervision: a dead worker restarts by itself after its backoff; Unavailable outcomes carry ScriptRunOutcome.Recovery
  (-> VirtualTagEvalResult.RetryWhen) and the VT actor / Calculation driver re-evaluate once against current inputs when
  it completes. Isolation worker stops after Scripting:Worker:IsolationIdleTimeout (10 min). Keys compiling when a
  worker dies are suspects too (isolation finds a poison compile and quarantines it). Startup orphan sweep by launch
  marker: kills only a process whose marker's host is gone and whose pid + start time + image match.
- Crash-loop circuit (Scripting:Worker:CircuitOpenAfter, 2 min): runner Unavailable, one Error, evaluations answered
  at once while shards keep retrying at the cap.
- Observability (§10): otopcua.script.runner.state, worker.up/inflight/queue_depth/heap_bytes (shard),
  ipc.roundtrip + eval.duration (kind), compile{cache}, isolation.suspects, exits reason=idle. `script-runner` health
  check on the ready tier, at worst Degraded (§14 #12: degraded but still ready).

Left on #780: self-contained worker publish, installer (account, pipe ACL, firewall), docker-dev live drill,
scripted-alarm seam, the §4.5 per-seam batch submit, the §8 perf harness.

Tests: SupervisionTests, PeCacheStoreTests/PeCacheWorkerTests (real workers), ScriptRunnerHealthCheckTests, VT + Calc
recovery seam tests; 14 mutations each turned their test red.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-10 00:20:24 -05:00
parent 69a6e0b4c8
commit f50b29abae
27 changed files with 2291 additions and 78 deletions
@@ -647,6 +647,13 @@ public sealed class CalculationDriver : IDriver, IDependencyConsumer, ISubscriba
if (toPublish is not null || !string.Equals(st.LastFailureReason, reason, StringComparison.Ordinal))
failureToLog = reason;
st.LastFailureReason = reason;
if (result.RetryWhen is { } retryWhen && !st.RecoveryArmed)
{
st.RecoveryArmed = true;
var rawPath = def.RawPath;
_ = retryWhen.ContinueWith(_ => OnRunnerRecovered(rawPath), CancellationToken.None,
TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
}
}
else
{
@@ -659,6 +666,24 @@ public sealed class CalculationDriver : IDriver, IDependencyConsumer, ISubscriba
if (toPublish is { } p) Publish(def.RawPath, p.Value, p.StatusCode, pending.TimestampUtc);
}
/// <summary>
/// The out-of-process runner that could not run <paramref name="rawPath"/>'s script is serving again
/// (lmxopcua#780 design §9): queue one evaluation against the CURRENT inputs, so a calc tag with static inputs
/// does not stay BadResourceUnavailable until an input happens to change.
/// </summary>
/// <param name="rawPath">The calc tag.</param>
private void OnRunnerRecovered(string rawPath)
{
lock (_lock)
{
if (_stateByRawPath.TryGetValue(rawPath, out var st)) st.RecoveryArmed = false;
if (!_running || _disposed) return;
if (PrepareEvaluation(rawPath, DateTime.UtcNow) is not { } prepared) return;
_pending.Enqueue(prepared);
SchedulePump();
}
}
/// <summary>Record a successful evaluation. MUST be called under <see cref="_lock"/>; returns what to
/// publish, or null when the result is deduped.
/// <para><b>The dedup key is (value, status), not value alone</b> — generalising the
@@ -852,6 +877,9 @@ public sealed class CalculationDriver : IDriver, IDependencyConsumer, ISubscriba
/// <summary>The reason of the last failed evaluation (null after a success / inputs-Bad hold) — the
/// gate that keeps the script-log entry to the transition and to a CHANGE of reason.</summary>
public string? LastFailureReason;
/// <summary>A runner-recovery re-evaluation is armed (lmxopcua#780): at most one per tag (latest-wins).</summary>
public bool RecoveryArmed;
}
private sealed record CalculationSubscriptionHandle(string DiagnosticId) : ISubscriptionHandle;