feat(scripting): ScriptRunGuard quarantines runaway scripts across every seam (E1)

A timed-out script is abandoned, not stopped, and every seam re-ran it on the
next trigger, leaking one pool thread per trigger (R03:1.1, R05:2, R07:L1).
A shared ScriptRunGuard, keyed per script instance, now:

- refuses a run while that script's previous orphan is still executing
  (Bad "previous evaluation still running" + otopcua.script.run_refused);
- quarantines after 3 consecutive non-completions or an allocation-budget
  breach (sampled with GC.GetAllocatedBytesForCurrentThread; default 64 MB):
  no further dispatch, Bad quality, one Error, otopcua.script.quarantined,
  cleared on source change / redeploy / reset;
- starts the budget clock when the pool picks the run up, so starvation is
  no longer charged to the script (R07:P5), and cancels the evaluator token
  on timeout.

Wired into RoslynVirtualTagEvaluator, CalculationEvaluator (new async
EvaluateAsync; the driver's pump and timer ticks now await the run behind a
SemaphoreSlim gate) and ScriptedAlarmEngine (a non-completing predicate
holds state and annotates the condition BadInternalError). ScriptingOptions
(Scripting:MaxAllocBytesPerRun, Scripting:QuarantineAfterConsecutiveTimeouts)
carries the knobs; the Calculation driver config accepts the same two keys.

Tests: guard unit suite (incl. the 30-trigger runaway leaving one orphan with
a healthy sibling as positive control), per-seam refusal/quarantine/reset
tests. The S12 race test now uses one vtag id per loop: twelve concurrent
evaluations of ONE id is a shape production never produces and the guard
refuses by design.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
This commit is contained in:
Joseph Doherty
2026-10-22 06:39:43 -04:00
parent ad2b5c4044
commit eb9269c337
17 changed files with 1313 additions and 74 deletions
@@ -0,0 +1,42 @@
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting;
/// <summary>
/// Node-wide script runtime-safety knobs (the <c>Scripting</c> appsettings section), consumed by
/// <see cref="ScriptRunGuard"/>. One instance is shared by every script seam on the node — the
/// virtual-tag evaluator, the Calculation evaluator and the scripted-alarm engine — so a single setting
/// bounds every authored script the same way.
/// </summary>
/// <remarks>
/// <para>
/// The in-process sandbox cannot pre-empt a script: a <c>while(true)</c> is abandoned on its pool
/// thread at the timeout, and an allocation is observed only after the run returns. These knobs bound
/// the <em>damage</em> — one orphan per script, no further runs until the source changes — not the
/// first occurrence. Pre-emptive CPU/memory budgeting needs an out-of-process script runner.
/// </para>
/// </remarks>
public sealed class ScriptingOptions
{
/// <summary>The appsettings section this binds from.</summary>
public const string SectionName = "Scripting";
/// <summary>Default <see cref="MaxAllocBytesPerRun"/>: 64 MB.</summary>
public const long DefaultMaxAllocBytesPerRun = 64L * 1024 * 1024;
/// <summary>Default <see cref="QuarantineAfterConsecutiveTimeouts"/>: 3.</summary>
public const int DefaultQuarantineAfterConsecutiveTimeouts = 3;
/// <summary>
/// Managed bytes one script run may allocate on its own thread (sampled with
/// <see cref="GC.GetAllocatedBytesForCurrentThread"/> around the run) before the script is
/// quarantined. Zero or negative disables the budget. Default 64 MB.
/// </summary>
public long MaxAllocBytesPerRun { get; set; } = DefaultMaxAllocBytesPerRun;
/// <summary>
/// Consecutive non-completions (a run that times out, or a run refused because the previous one is
/// still executing) after which the script is quarantined: no further runs are dispatched, every
/// evaluation reports Bad, and the quarantine lifts only on a source change, a redeploy or an explicit
/// reset. Default 3; values below 1 are treated as 1.
/// </summary>
public int QuarantineAfterConsecutiveTimeouts { get; set; } = DefaultQuarantineAfterConsecutiveTimeouts;
}