dohertj2/lmxopcua
1
0
Fork 0
Code Issues 43 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(scripting): quarantine a refusal behind a stuck orphan (fix round 2)

Browse Source 
Ruling refinement on I1. Only timeouts are strikes, so a never-returning
while(true), which times out once and is then only refused, was never
quarantined, and that defeated E1. A refusal now quarantines immediately
when the orphan it was refused behind has been executing for longer than
ScriptingOptions.StuckOrphanQuarantineFactor x the script's timeout
(default 10, so 20 s at the 2 s default; 0 or less disables it). It uses
the same quarantine path: one Error, otopcua.script.quarantined with
reason=stuck, Bad, lifted on source change or redeploy. A young orphan
(a short overrun) is still a plain refusal.

The guard stamps the run's start (Stopwatch timestamp) when the pool
picks it up and clears it when the run really finishes.

Tests: the 30-trigger while(true) regression (paced) now ends
quarantined with reason "stuck", still with exactly one orphan and a
healthy sibling. Positive control: a 2x overrun is refused, not
quarantined, and evaluates again once the orphan completes. A factor of
0 disables the rule. Docs (ScriptEditor "Runtime safety", VirtualTags)
and the seam remarks are updated.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
This commit is contained in:
Joseph Doherty
2026-10-22 13:59:10 -04:00
parent ef2eebe6b6
commit a85d1c069a
8 changed files with 133 additions and 23 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Tests/ScriptRunGuardTests.cs
+76 -12
View File
@@ -151,10 +151,13 @@ public sealed class ScriptRunGuardTests
}
/// <summary>
/// The thread-leak regression. Pre-guard, a <c>while(true)</c> script on a 1 Hz trigger leaked one
/// pool thread per trigger — 30 orphans after 30 triggers, trending toward pool starvation. Now the
/// first trigger's orphan is the only one, every later trigger is refused, and a healthy
/// sibling keeps evaluating throughout (the positive control an absence assertion needs).
/// The thread-leak regression, and the stuck-orphan quarantine (fix round 2). Pre-guard, a
/// <c>while(true)</c> script on a 1 Hz trigger leaked one pool thread per trigger — 30 orphans after 30
/// triggers, trending toward pool starvation. Now the first trigger's orphan is the only one; later
/// triggers are refused while it is young, and once it has run longer than
/// <see cref="ScriptingOptions.StuckOrphanQuarantineFactor"/> × the timeout (10 × 60 ms here — the
/// trigger cadence is scaled down from 1 Hz to keep the test short) the next refusal quarantines it.
/// A healthy sibling keeps evaluating throughout (the positive control an absence assertion needs).
/// </summary>
/// <remarks>
/// The load-bearing assertions are on the guard's own bookkeeping (<see cref="ScriptRunGuard.InFlightCount"/>
@@ -163,30 +166,37 @@ public sealed class ScriptRunGuardTests
/// it must stay far below the 30 the leak would have produced.
/// </remarks>
[Fact]
public async Task Runaway_script_leaves_at_most_one_orphan_after_30_triggers_while_a_sibling_keeps_evaluating()
public async Task Runaway_script_leaves_one_orphan_is_quarantined_once_stuck_and_a_sibling_keeps_evaluating()
{
var guard = new ScriptRunGuard();
using var spin = Compile(SpinScript);
using var fast = Compile(FastScript);
var ct = TestContext.Current.CancellationToken;
var threadsBefore = ThreadPool.ThreadCount;
var stuckAfter = ShortTimeout * ScriptingOptions.DefaultStuckOrphanQuarantineFactor; // 600 ms
var healthyResults = new List<int>();
var outcomes = new List<string>();
var sw = System.Diagnostics.Stopwatch.StartNew();
for (var trigger = 0; trigger < 30; trigger++)
{
try { await Run(guard, "runaway", SpinScript, spin, ShortTimeout, ct); }
catch (ScriptTimeoutException) { }
catch (ScriptRunRefusedException) { }
catch (ScriptQuarantinedException) { }
try { await Run(guard, "runaway", SpinScript, spin, ShortTimeout, ct); outcomes.Add("value"); }
catch (ScriptTimeoutException) { outcomes.Add("timeout"); }
catch (ScriptRunRefusedException) { outcomes.Add(sw.Elapsed < stuckAfter ? "refused-young" : "refused"); }
catch (ScriptQuarantinedException) { outcomes.Add("quarantined"); }
healthyResults.Add(await Run(guard, "healthy", FastScript, fast, ct: ct));
await Task.Delay(50, ct);
}
guard.InFlightCount.ShouldBe(1, "exactly one orphaned run — the first trigger's — may be executing");
guard.StartedRunCount.ShouldBe(1 + 30, "one dispatch for the runaway script, thirty for the healthy sibling");
// One timeout + 29 refusals: not quarantined (a refusal is not a strike), but never re-dispatched.
guard.IsQuarantined("runaway", out _).ShouldBeFalse();
guard.RefusedWhileRunningCount.ShouldBe(29);
outcomes[0].ShouldBe("timeout");
outcomes.ShouldContain("refused-young", "a young orphan is only refused — a short overrun is not quarantined");
guard.IsQuarantined("runaway", out var reason).ShouldBeTrue("the orphan outlived 10x its budget: it is stuck");
reason!.ShouldContain("stuck");
outcomes[^1].ShouldBe("quarantined");
guard.QuarantinedCount.ShouldBe(1);
healthyResults.ShouldAllBe(v => v == 42);
healthyResults.Count.ShouldBe(30);
@@ -194,6 +204,60 @@ public sealed class ScriptRunGuardTests
threadDelta.ShouldBeLessThan(15, "thirty leaked orphans would have grown the pool by ~30 threads");
}
/// <summary>
/// Positive control for the stuck rule: a 2x overrun (one timeout, refusals while the orphan is young)
/// is NOT quarantined, and once the orphan completes the script evaluates normally again.
/// </summary>
[Fact]
public async Task A_2x_overrun_is_not_quarantined_and_resumes_evaluating_once_the_orphan_completes()
{
var guard = new ScriptRunGuard();
const string source =
"""
var ms = (int)(ctx.GetTag("spinMs").Value ?? 0);
var end = System.DateTime.UtcNow.AddMilliseconds(ms);
while (System.DateTime.UtcNow < end) { }
return 7;
""";
using var eval = Compile(source);
var ct = TestContext.Current.CancellationToken;
var timeout = TimeSpan.FromMilliseconds(100);
FakeScriptContext Ctx(int spinMs)
{
var c = new FakeScriptContext();
c.Tags["spinMs"] = new Core.Abstractions.DataValueSnapshot(spinMs, 0u, c.Now, c.Now);
return c;
}
// Warm the JIT so the first measured run's overrun is the script's, not the compiler's.
(await guard.RunAsync("k", source, eval, Ctx(0), TimeSpan.FromSeconds(5), ct)).ShouldBe(7);
// 2x overrun: 200 ms against a 100 ms budget — times out, then is refused while it finishes.
await Should.ThrowAsync<ScriptTimeoutException>(() => guard.RunAsync("k", source, eval, Ctx(200), timeout, ct));
await Should.ThrowAsync<ScriptRunRefusedException>(() => guard.RunAsync("k", source, eval, Ctx(0), timeout, ct));
await WaitForIdle(guard);
guard.IsQuarantined("k", out _).ShouldBeFalse("a 2x overrun is far below the 10x stuck threshold");
(await guard.RunAsync("k", source, eval, Ctx(0), timeout, ct)).ShouldBe(7, "the script resumes evaluating");
guard.QuarantinedCount.ShouldBe(0);
}
/// <summary>The stuck rule is a knob: with <see cref="ScriptingOptions.StuckOrphanQuarantineFactor"/> ≤ 0 a
/// stuck orphan is only ever refused.</summary>
[Fact]
public async Task StuckOrphanQuarantineFactor_zero_disables_the_stuck_rule()
{
var guard = new ScriptRunGuard(new ScriptingOptions { StuckOrphanQuarantineFactor = 0 });
using var spin = Compile(SpinScript);
var ct = TestContext.Current.CancellationToken;
await Should.ThrowAsync<ScriptTimeoutException>(() => Run(guard, "k", SpinScript, spin, ShortTimeout, ct));
await Task.Delay(ShortTimeout * 12, ct);
await Should.ThrowAsync<ScriptRunRefusedException>(() => Run(guard, "k", SpinScript, spin, ShortTimeout, ct));
guard.IsQuarantined("k", out _).ShouldBeFalse();
}
[Fact]
public async Task A_source_change_clears_the_quarantine_and_abandons_the_old_orphan_slot()
{