feat(scripting): ScriptRunGuard quarantines runaway scripts across every seam (E1)

A timed-out script is abandoned, not stopped, and every seam re-ran it on the
next trigger, leaking one pool thread per trigger (R03:1.1, R05:2, R07:L1).
A shared ScriptRunGuard, keyed per script instance, now:

- refuses a run while that script's previous orphan is still executing
  (Bad "previous evaluation still running" + otopcua.script.run_refused);
- quarantines after 3 consecutive non-completions or an allocation-budget
  breach (sampled with GC.GetAllocatedBytesForCurrentThread; default 64 MB):
  no further dispatch, Bad quality, one Error, otopcua.script.quarantined,
  cleared on source change / redeploy / reset;
- starts the budget clock when the pool picks the run up, so starvation is
  no longer charged to the script (R07:P5), and cancels the evaluator token
  on timeout.

Wired into RoslynVirtualTagEvaluator, CalculationEvaluator (new async
EvaluateAsync; the driver's pump and timer ticks now await the run behind a
SemaphoreSlim gate) and ScriptedAlarmEngine (a non-completing predicate
holds state and annotates the condition BadInternalError). ScriptingOptions
(Scripting:MaxAllocBytesPerRun, Scripting:QuarantineAfterConsecutiveTimeouts)
carries the knobs; the Calculation driver config accepts the same two keys.

Tests: guard unit suite (incl. the 30-trigger runaway leaving one orphan with
a healthy sibling as positive control), per-seam refusal/quarantine/reset
tests. The S12 race test now uses one vtag id per loop: twelve concurrent
evaluations of ONE id is a shape production never produces and the guard
refuses by design.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
This commit is contained in:
Joseph Doherty
2026-10-22 06:39:43 -04:00
parent ad2b5c4044
commit eb9269c337
17 changed files with 1313 additions and 74 deletions
@@ -7,11 +7,13 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Calculation;
/// <see cref="System.Threading.Timer"/> per distinct interval group (so the wire count stays low
/// regardless of tag count) with a per-group in-flight flag that skips a tick when the prior tick for
/// the same group is still running. Each tick invokes the driver's own evaluate
/// callback per tag in the group.
/// callback per tag in the group and AWAITS it — the in-flight flag is held across the awaited script
/// run, not just across the synchronous part of the callback, so a slow script still bounds its group
/// to one outstanding tick.
/// </summary>
internal sealed class CalculationTimerScheduler : IDisposable
{
private readonly Action<string> _evaluate;
private readonly Func<string, Task> _evaluate;
private readonly ILogger _logger;
private readonly List<Timer> _timers = [];
private readonly List<TickGroup> _groups = [];
@@ -20,9 +22,9 @@ internal sealed class CalculationTimerScheduler : IDisposable
private bool _disposed;
/// <summary>Initializes a new instance of the <see cref="CalculationTimerScheduler"/> class.</summary>
/// <param name="evaluate">Callback invoked with a calc tag's RawPath when its timer group ticks.</param>
/// <param name="evaluate">Callback invoked with a calc tag's RawPath when its timer group ticks; awaited.</param>
/// <param name="logger">Logger for diagnostics.</param>
public CalculationTimerScheduler(Action<string> evaluate, ILogger logger)
public CalculationTimerScheduler(Func<string, Task> evaluate, ILogger logger)
{
_evaluate = evaluate ?? throw new ArgumentNullException(nameof(evaluate));
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
@@ -69,12 +71,20 @@ internal sealed class CalculationTimerScheduler : IDisposable
return;
}
// The Timer callback is void; the tick itself is async so the awaited script runs do not pin this
// pool thread. TickAsync owns the in-flight flag and swallows everything — nothing can escape to the
// Timer (which would be an unobserved fault on a timer thread).
_ = TickAsync(ctx);
}
private async Task TickAsync(TickGroup ctx)
{
try
{
foreach (var path in ctx.Paths)
{
if (_cts.IsCancellationRequested) return;
try { _evaluate(path); }
try { await _evaluate(path).ConfigureAwait(false); }
catch (Exception ex) { _logger.LogError(ex, "CalculationTimerScheduler evaluate failed for {Path}", path); }
}
}