fix(scripting): the run guard frees a script's slot before its run completes, not in a continuation (#843)

RoslynVirtualTagEvaluatorTests' race test failed 1 of 3 full Host
integration runs under 28 CPU burners (load ~300) with 24 of 4,800
evaluations refused 'previous evaluation still running' - impossible for one
sequential loop per id unless a finished run still held its slot after the
caller resumed. It did: ScriptRunGuard released the slot from a ContinueWith
on the run task, and a task is observably complete before its continuations
run, so the next run of the key (the next trigger, or the evaluator's
immediate retry after a raced ObjectDisposedException) could be refused - a
Bad publish for a script that had finished. Same class #789 fixed for the
out-of-process runner.

The release (and the late allocation-breach quarantine, now ahead of it) runs
in the run's own finally, before the task completes; the continuation only
disposes the CTS and observes the fault. New
ScriptRunGuardTests.A_caller_that_awaited_its_run_is_never_refused_by_it
widens the release with a test hook (SlotReleasingForTest) - mutation-checked:
releasing from a continuation again refuses 49 of 50.

The race test now prints elapsed / evaluations / clears / pool state / failure
histogram on every assertion and on its hang detector. After: 3/3 full Host
integration runs pass under the same load. Core.Scripting 298, ScriptedAlarms
109, Calculation 84 green.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Joseph Doherty
2026-11-11 05:09:36 -05:00
parent 4cd648b029
commit e89ac2df33
3 changed files with 100 additions and 16 deletions
@@ -61,6 +61,40 @@ public sealed class ScriptRunGuardTests
guard.IsQuarantined("vt-1", out _).ShouldBeFalse();
}
/// <summary>
/// lmxopcua#843: a caller that awaited its run is never refused by it. The slot used to be released by a
/// ContinueWith on the run task — and a task is observably complete before its continuations run, so the
/// caller (or the evaluator's immediate ODE retry) could start the key's next run while the finished one still
/// held the slot: "previous evaluation still running", 24 of 4,800 sequential evaluations in the Host
/// integration suite under load. The second phase makes every release slow (the hook sleeps inside it), so
/// the ordering, not luck, is what keeps the caller from being refused. Mutation check: release from a
/// continuation again and the second phase goes red.
/// </summary>
[Fact]
public async Task A_caller_that_awaited_its_run_is_never_refused_by_it()
{
var guard = new ScriptRunGuard();
using var fast = Compile(FastScript);
var ct = TestContext.Current.CancellationToken;
var refused = new List<string>();
for (var i = 0; i < 500; i++)
{
try { (await Run(guard, "seq", FastScript, fast, ct: ct)).ShouldBe(42); }
catch (ScriptRunRefusedException) { refused.Add($"#{i}"); }
}
refused.ShouldBeEmpty();
guard.SlotReleasingForTest = _ => Thread.Sleep(5);
for (var i = 0; i < 50; i++)
{
try { (await Run(guard, "seq", FastScript, fast, ct: ct)).ShouldBe(42); }
catch (ScriptRunRefusedException) { refused.Add($"slow release #{i}"); }
}
refused.ShouldBeEmpty();
guard.RefusedWhileRunningCount.ShouldBe(0);
guard.InFlightCount.ShouldBe(0);
}
[Fact]
public async Task A_timed_out_run_throws_ScriptTimeoutException_and_keeps_the_slot_held_by_the_orphan()
{