feat(scripting): Scripting:Runner=Shadow — in-process results published, each compiled run mirrored to the ScriptWorker pool and compared (lmxopcua#780 P2)

- ShadowScriptRunner (Core.Scripting): IScriptShadow.Mirror copies the inputs and queues the shadow run on the
  pool; compares value (structural), CLR type, status and failure class (compile / budget / threw+message /
  unsupported_type); runner-fault shadow outcomes (Unavailable, worker busy) are not compared. Bounded:
  Scripting:Shadow:MaxInFlight (256) and one per script; drops counted (behind | key_busy).
- Metrics otopcua.script.shadow.{compared,mismatch{script},dropped,inflight}; one rate-limited Warning per
  script and reason with a diff summary (Scripting:Shadow:MismatchLogsPerMinute, re-armed at the seam's deploy).
- The VirtualTag, Calculation and scripted-alarm seams recognise the shadow and keep their in-process path
  byte-for-byte (no coalescer, one-at-a-time pump, in-process engine), mirroring after each compiled run.
- ScriptRunnerSelection builds the shadow over an OutOfProcessScriptRunner whose lines carry
  ScriptRunnerRole=Shadow; Status() is null under Shadow so readiness never sees the shadow pool; the
  script-runner health entry stays Healthy and reports the shadow counters as data.
- docker-dev/docker-compose.script-runner-shadow.yml; docs/VirtualTags.md "Shadow mode".
- Tests: comparison table, per-script metric + once-per-reason log, rate limit, hung/blocking/throwing shadow
  never reaches the caller, MaxInFlight drop accounting, input copy; real-worker divergent + unsupported-type
  scripts counted against exactly those scripts, a worker-only runaway killed/quarantined there with every
  published value intact, alarm transitions identical to in process across a kill -9 of the shadow worker.
  Mutation-proved (16 mutants, 15 red; the one survivor added an assertion).

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-10 07:18:40 -05:00
parent 0796928504
commit 03686eb5d7
17 changed files with 1477 additions and 52 deletions
@@ -122,6 +122,9 @@ public sealed record ScriptRunOutcome(ScriptRunStatus Status, object? Value, str
/// <summary>Reason when no worker became ready within <see cref="ScriptWorkerOptions.UnavailableGrace"/>.</summary>
public const string RunnerUnavailableReason = "script runner unavailable";
/// <summary>Reason when the worker could not take more work (the out-of-process runner's back-pressure answer).</summary>
public const string WorkerBusyReason = "script worker busy";
/// <summary>
/// For <see cref="ScriptRunStatus.Unavailable"/>: completes when the runner can run this script again (its worker
/// is back), so the seam re-evaluates it then with its latest inputs (design §9, "re-evaluated on the shard's
@@ -175,7 +178,11 @@ public static class ScriptRunOutcomeExtensions
/// </summary>
public interface IScriptRunnerProvider
{
/// <summary>The runner <paramref name="seam"/> must use, or null for its in-process path.</summary>
/// <summary>
/// The runner <paramref name="seam"/> must use, or null for its in-process path. Under
/// <c>Scripting:Runner=Shadow</c> this is a <see cref="ShadowScriptRunner"/>: the seam recognises it as an
/// <see cref="IScriptShadow"/>, keeps its in-process path and mirrors each evaluation to it.
/// </summary>
/// <param name="seam">The seam.</param>
/// <returns>The runner, or null.</returns>
IScriptRunner? ForSeam(ScriptSeam seam);