dohertj2/lmxopcua
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(scripting): the run guard frees a script's slot before its run completes, not in a continuation (#843)

Browse Source 
RoslynVirtualTagEvaluatorTests' race test failed 1 of 3 full Host
integration runs under 28 CPU burners (load ~300) with 24 of 4,800
evaluations refused 'previous evaluation still running' - impossible for one
sequential loop per id unless a finished run still held its slot after the
caller resumed. It did: ScriptRunGuard released the slot from a ContinueWith
on the run task, and a task is observably complete before its continuations
run, so the next run of the key (the next trigger, or the evaluator's
immediate retry after a raced ObjectDisposedException) could be refused - a
Bad publish for a script that had finished. Same class #789 fixed for the
out-of-process runner.

The release (and the late allocation-breach quarantine, now ahead of it) runs
in the run's own finally, before the task completes; the continuation only
disposes the CTS and observes the fault. New
ScriptRunGuardTests.A_caller_that_awaited_its_run_is_never_refused_by_it
widens the release with a test hook (SlotReleasingForTest) - mutation-checked:
releasing from a continuation again refuses 49 of 50.

The race test now prints elapsed / evaluations / clears / pool state / failure
histogram on every assertion and on its hang detector. After: 3/3 full Host
integration runs pass under the same load. Core.Scripting 298, ScriptedAlarms
109, Calculation 84 green.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Joseph Doherty
2026-11-11 05:09:36 -05:00
parent 4cd648b029
commit e89ac2df33
3 changed files with 100 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/RoslynVirtualTagEvaluatorTests.cs
+36 -5
View File
@@ -466,6 +466,19 @@ public sealed class RoslynVirtualTagEvaluatorTests
/// for the same reason: a handful of queued-too-long evaluations is the environment, but 10% of them
/// missing a 5 s budget means the evaluator stopped evaluating. If a loaded box ever does trip EITHER
/// bound, the rate itself is the finding: record it before widening the number.</para></summary>
/// <remarks>
/// <b>lmxopcua#843 — this test found a product bug, not a timing assumption.</b> Under the full Host
/// integration suite plus 28 CPU burners (load ~300) it failed 1 run in 3 with 24 of 4,800 evaluations
/// refused <c>"previous evaluation still running"</c> — impossible for one sequential loop per id unless the
/// id's previous run still held its <see cref="ScriptRunGuard"/> slot after its caller had resumed. It did:
/// the slot was released by a continuation on the run task, and a task is observably complete before its
/// continuations run (worst for the evaluator's immediate retry after a raced
/// <see cref="ObjectDisposedException"/>). The release now happens inside the run, before it completes
/// (<c>ScriptRunGuardTests.A_caller_that_awaited_its_run_is_never_refused_by_it</c> pins the ordering). A
/// refusal stays in the "leaked" bucket on purpose. The doc above predates P5: the guard's budget clock
/// starts when the pool starts the run, so a queued evaluation no longer lands in the timed-out bucket.
/// Every assertion now prints the run's shape (elapsed, evaluations, clears, pool state, failure histogram).
/// </remarks>
[Fact]
public async Task Evaluate_racing_ClearCompiledScripts_never_leaks_a_raw_disposed_failure()
{
@@ -511,19 +524,37 @@ public sealed class RoslynVirtualTagEvaluatorTests
Interlocked.Decrement(ref remaining);
}, TestContext.Current.CancellationToken)).ToArray();
var clears = 0L;
var clearLoop = Task.Run(() =>
{
while (Volatile.Read(ref remaining) > 0)
{
sut.ClearCompiledScripts();
Interlocked.Increment(ref clears);
Thread.SpinWait(4000); // ~tens of µs between clears — frequent, but not back-to-back
}
}, TestContext.Current.CancellationToken);
// lmxopcua#843: every assertion below carries the run's shape, so a failure under load says which bucket
// moved and how starved the box was (it passed 5/5 on rerun with nothing recorded).
var elapsed = System.Diagnostics.Stopwatch.StartNew();
string Diagnostics() =>
$" [elapsed {elapsed.ElapsedMilliseconds} ms, {Volatile.Read(ref evaluated)} evaluations, " +
$"{Interlocked.Read(ref clears)} clears, loops still running {Volatile.Read(ref remaining)}, " +
$"pool threads {ThreadPool.ThreadCount} / pending {ThreadPool.PendingWorkItemCount}; failures: " +
string.Join(", ", failures.GroupBy(r => r.Length > 80 ? r[..80] : r).Select(g => $"{g.Count()}× \"{g.Key}\"")) + "]";
// Hang detector only: the loops self-terminate at RaceWindow, so firing this means something
// wedged, not that the box was slow.
await Task.WhenAll(evalLoops.Append(clearLoop))
.WaitAsync(TimeSpan.FromSeconds(45), TestContext.Current.CancellationToken);
try
{
await Task.WhenAll(evalLoops.Append(clearLoop))
.WaitAsync(TimeSpan.FromSeconds(45), TestContext.Current.CancellationToken);
}
catch (TimeoutException ex)
{
throw new TimeoutException("the racing loops did not finish within 45 s" + Diagnostics(), ex);
}
var totalEvaluations = Volatile.Read(ref evaluated);
// Loose by design, with an absolute floor so a starved run (few evaluations landed) cannot trip on
@@ -544,20 +575,20 @@ public sealed class RoslynVirtualTagEvaluatorTests
leaked.ShouldBeEmpty(
"a racing ClearCompiledScripts must never surface a failure other than " +
$"\"{ScriptEvalFailureReasons.EvaluatorDisposedDuringEvaluation}\" or " +
$"\"{ScriptEvalFailureReasons.TimedOutPrefix}…\"; observed: {string.Join(" | ", leaked)}");
$"\"{ScriptEvalFailureReasons.TimedOutPrefix}…\"; observed: {string.Join(" | ", leaked)}" + Diagnostics());
// (2) Non-vacuity: the retry must actually be absorbing the race, not merely relabelling it.
exhausted.ShouldBeLessThan(
rateBound,
"the one-shot retry must absorb the overwhelming majority of raced clears; " +
$"{exhausted}/{totalEvaluations} evaluations exhausted it");
$"{exhausted}/{totalEvaluations} evaluations exhausted it" + Diagnostics());
// (3) The starvation bucket is tolerated but not unbounded — at 10% the evaluator has stopped
// evaluating rather than the box being busy.
starved.ShouldBeLessThan(
rateBound,
"queue-starvation timeouts are expected under this stress but must stay marginal; " +
$"{starved}/{totalEvaluations} evaluations missed the 5s budget");
$"{starved}/{totalEvaluations} evaluations missed the 5s budget" + Diagnostics());
}
// ── input quality reaches the script (archreview 02/S14) ────────────────────────────────