feat(scripting): out-of-process pre-emptible script runner behind Scripting:Runner (lmxopcua#780 P1)
v2-ci / unit-tests (push) Waiting to run

Opt-in (Scripting:Runner=OutOfProcess; InProcess stays the default and its code path is unchanged). Implements
the P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md with the owner's §14 decisions:

- Options: Scripting:Runner (InProcess | Shadow | OutOfProcess), Scripting:Runner:Seams, Scripting:Worker:*
  (Shards 2, HardGrace max(1 s,T), HeapLimitBytes 512 MiB, QuarantineAfterKills 2, UnavailableGrace 5 s,
  IsolationProbationRuns 20, PeCacheDir validated absolute + outside the content root (fail closed),
  ExecutablePath). Shadow is not implemented yet: runs InProcess with a startup Warning.
- Core.Scripting: IScriptRunner seam + InProcessScriptRunner (today's guard, the parity baseline).
- Core.Scripting.Protocol: protobuf envelopes, 4-byte LE framing (zero/oversize/truncated/garbage refused),
  Variant carrying the exact CLR type; unsupported types are explicit failures.
- ScriptWorker app: compiles in the worker, P-thread scheduler, host-driven soft cancel, fail-fast
  watchdog, heartbeats, ctx.Logger forwarding, exits on pipe EOF / parent gone, forbidden-assembly
  self-check; Akka and Grpc.Core.Api excluded from its folder. Shipped in script-worker/ (Build + Publish).
- Core.Scripting.OutOfProcess: K shards + isolation worker, soft T / hard T+G kill, attribution
  (deadline kill = one key; crash = suspects -> isolation; isolation crash = strike), kill-strike
  quarantine, backoff 250 ms -> 30 s, fail closed, cleared allow-listed env, nonce + pid handshake on a
  current-user pipe, Windows Job Object.
- VirtualTag + Calculation seams use the runner when selected; a runner-side failure (worker restarted /
  runner unavailable) publishes BadResourceUnavailable (0x80040000) — ScadaBridge#112. Script faults keep
  BadInternalError. The scripted-alarm seam stays in process (alarms last).

Tests: new ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests (Category=ScriptWorker, real processes; green on macOS and
on windev/Windows), seam tests in Host/Runtime/Calculation suites; mutation-checked.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 23:31:51 -05:00
parent f5c30ed0ef
commit 69a6e0b4c8
49 changed files with 5107 additions and 25 deletions
@@ -0,0 +1,137 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting;
/// <summary>
/// The execution seam every host script owner can run a compiled script through (lmxopcua#780, design §11): a
/// script key, its source, its input snapshots and its budget go in; one <see cref="ScriptRunOutcome"/> comes
/// out. Implementations never throw for script faults.
/// </summary>
/// <remarks>
/// <para>
/// <see cref="InProcessScriptRunner"/> is today's <see cref="ScriptRunGuard"/> + <see cref="CompiledScriptCache{TContext, TResult}"/>
/// behind this interface (the parity baseline). The out-of-process runner
/// (<c>ZB.MOM.WW.OtOpcUa.Core.Scripting.Runner</c>) runs the same compile in a pre-emptible worker process.
/// The passthrough fast path (<c>PassthroughScript</c>) never reaches a runner: it runs no user code.
/// </para>
/// <para>
/// Keys are namespaced by <see cref="ScriptInvocation.Kind"/>, so a virtual-tag id and a calc RawPath that
/// happen to be equal never share quarantine state, and <see cref="Reset"/> for one seam leaves the other
/// seams' quarantines alone.
/// </para>
/// </remarks>
public interface IScriptRunner
{
/// <summary>Runs one compiled evaluation.</summary>
/// <param name="invocation">What to run.</param>
/// <param name="ct">Caller cancellation (shutdown); wins over the budget.</param>
/// <returns>The outcome; never faults for a script failure.</returns>
Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default);
/// <summary>
/// The deploy boundary for one seam: lifts every quarantine and forgets every remembered compile failure of
/// <paramref name="kind"/>, and drops its compiled scripts.
/// </summary>
/// <param name="kind">The seam's context kind.</param>
void Reset(ScriptContextKind kind);
/// <summary>The keys of <paramref name="kind"/> currently quarantined.</summary>
/// <param name="kind">The seam's context kind.</param>
/// <returns>The quarantined keys (un-namespaced).</returns>
IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind);
}
/// <summary>Which script context (and result type) a script compiles against.</summary>
public enum ScriptContextKind
{
/// <summary>A virtual tag: <c>VirtualTagContext</c> → <c>object?</c>.</summary>
VirtualTag = 1,
/// <summary>A Calculation tag: <c>VirtualTagContext</c> → <c>object?</c>.</summary>
Calculation = 2,
/// <summary>A scripted-alarm predicate: <c>AlarmPredicateContext</c> → <c>bool</c> (not run out of process yet).</summary>
AlarmPredicate = 3,
}
/// <summary>One evaluation handed to an <see cref="IScriptRunner"/>.</summary>
/// <param name="Key">The script instance key (virtual-tag id, calc RawPath, alarm id).</param>
/// <param name="Kind">The context kind.</param>
/// <param name="Source">The script source (also its identity: a change lifts the key's quarantine).</param>
/// <param name="Inputs">The read cache: dependency snapshots by path. Not mutated until the run completes.</param>
/// <param name="Timeout">The soft budget <c>T</c>.</param>
/// <param name="ScriptLogger">The per-evaluation script logger <c>ctx.Logger</c> writes through; null ⇒ a silent logger.</param>
public sealed record ScriptInvocation(
string Key,
ScriptContextKind Kind,
string Source,
IReadOnlyDictionary<string, DataValueSnapshot> Inputs,
TimeSpan Timeout,
Serilog.ILogger? ScriptLogger = null);
/// <summary>How an evaluation ended.</summary>
public enum ScriptRunStatus
{
/// <summary>The script returned <see cref="ScriptRunOutcome.Value"/>.</summary>
Ok = 0,
/// <summary>A script-attributable failure (threw, timed out, refused, quarantined, did not compile) — Bad as today.</summary>
Failed = 1,
/// <summary>
/// The runner could not run the script, and it is not the script's fault: its worker restarted under it, or no
/// worker became ready in time. Published as <c>BadResourceUnavailable</c> (design §9 / §14 #3).
/// </summary>
Unavailable = 2,
}
/// <summary>The result of one <see cref="IScriptRunner.RunAsync"/>.</summary>
/// <param name="Status">How it ended.</param>
/// <param name="Value">The script's result when <see cref="ScriptRunStatus.Ok"/>.</param>
/// <param name="Reason">The failure reason otherwise; the same strings the evaluators have always published.</param>
public sealed record ScriptRunOutcome(ScriptRunStatus Status, object? Value, string? Reason)
{
/// <summary>Reason when the worker running the script died under it (another script's kill, a crash).</summary>
public const string WorkerRestartedReason = "script worker restarted";
/// <summary>Reason when no worker became ready within <see cref="ScriptWorkerOptions.UnavailableGrace"/>.</summary>
public const string RunnerUnavailableReason = "script runner unavailable";
/// <summary>A successful run.</summary>
/// <param name="value">The script's result.</param>
/// <returns>The outcome.</returns>
public static ScriptRunOutcome Ok(object? value) => new(ScriptRunStatus.Ok, value, null);
/// <summary>A script-attributable failure.</summary>
/// <param name="reason">The reason.</param>
/// <returns>The outcome.</returns>
public static ScriptRunOutcome Failed(string reason) => new(ScriptRunStatus.Failed, null, reason);
/// <summary>A runner-attributable failure.</summary>
/// <param name="reason">The reason.</param>
/// <returns>The outcome.</returns>
public static ScriptRunOutcome Unavailable(string reason) => new(ScriptRunStatus.Unavailable, null, reason);
}
/// <summary>Maps an <see cref="IScriptRunner"/> outcome onto the value seams' result type.</summary>
public static class ScriptRunOutcomeExtensions
{
/// <summary>
/// <see cref="ScriptRunStatus.Ok"/> → <c>Ok(value)</c>; <see cref="ScriptRunStatus.Failed"/> → <c>Failure(reason)</c>
/// (published <c>BadInternalError</c>, as today); <see cref="ScriptRunStatus.Unavailable"/> →
/// <c>Unavailable(reason)</c> (published <c>BadResourceUnavailable</c>).
/// </summary>
/// <param name="outcome">The runner's outcome.</param>
/// <returns>The seam result.</returns>
public static Commons.Engines.VirtualTagEvalResult ToVirtualTagEvalResult(this ScriptRunOutcome outcome)
{
ArgumentNullException.ThrowIfNull(outcome);
return outcome.Status switch
{
ScriptRunStatus.Ok => Commons.Engines.VirtualTagEvalResult.Ok(outcome.Value),
ScriptRunStatus.Unavailable => Commons.Engines.VirtualTagEvalResult.Unavailable(
outcome.Reason ?? ScriptRunOutcome.RunnerUnavailableReason),
_ => Commons.Engines.VirtualTagEvalResult.Failure(outcome.Reason ?? "evaluator failure"),
};
}
}