dohertj2/lmxopcua
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(scripting): the run guard frees a script's slot before its run completes, not in a continuation (#843)

Browse Source 
RoslynVirtualTagEvaluatorTests' race test failed 1 of 3 full Host
integration runs under 28 CPU burners (load ~300) with 24 of 4,800
evaluations refused 'previous evaluation still running' - impossible for one
sequential loop per id unless a finished run still held its slot after the
caller resumed. It did: ScriptRunGuard released the slot from a ContinueWith
on the run task, and a task is observably complete before its continuations
run, so the next run of the key (the next trigger, or the evaluator's
immediate retry after a raced ObjectDisposedException) could be refused - a
Bad publish for a script that had finished. Same class #789 fixed for the
out-of-process runner.

The release (and the late allocation-breach quarantine, now ahead of it) runs
in the run's own finally, before the task completes; the continuation only
disposes the CTS and observes the fault. New
ScriptRunGuardTests.A_caller_that_awaited_its_run_is_never_refused_by_it
widens the release with a test hook (SlotReleasingForTest) - mutation-checked:
releasing from a continuation again refuses 49 of 50.

The race test now prints elapsed / evaluations / clears / pool state / failure
histogram on every assertion and on its hang detector. After: 3/3 full Host
integration runs pass under the same load. Core.Scripting 298, ScriptedAlarms
109, Calculation 84 green.

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Joseph Doherty
2026-11-11 05:09:36 -05:00
parent 4cd648b029
commit e89ac2df33
3 changed files with 100 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ScriptRunGuard.cs
+30 -11
View File
@@ -105,6 +105,13 @@ public sealed class ScriptRunGuard
_logger = logger ?? Log.Logger;
}
/// <summary>
/// Test seam (lmxopcua#843): invoked on the run's own thread just before its slot is released, with the key.
/// A sleep here widens the release window, so a test can prove the release is ordered before the caller
/// resumes rather than winning a race.
/// </summary>
internal Action<string>? SlotReleasingForTest { get; set; }
/// <summary>Raised (on the thread that detected it) when a script is quarantined.</summary>
public event EventHandler<ScriptQuarantinedEventArgs>? Quarantined;
@@ -233,34 +240,46 @@ public sealed class ScriptRunGuard
Volatile.Write(ref state.RunStartedTimestamp, Stopwatch.GetTimestamp());
started.TrySetResult();
var before = GC.GetAllocatedBytesForCurrentThread();
var succeeded = false;
try
{
// ScriptEvaluator.RunAsync executes the compiled delegate synchronously and returns a completed
// task, so the after-sample is taken on the same thread, right after the script returned.
var pending = evaluator.RunAsync(context, runToken);
slot.AllocatedBytes = GC.GetAllocatedBytesForCurrentThread() - before;
return await pending.ConfigureAwait(false);
var value = await pending.ConfigureAwait(false);
succeeded = true;
return value;
}
catch
{
slot.AllocatedBytes = GC.GetAllocatedBytesForCurrentThread() - before;
throw;
}
finally
{
// Release the slot when the run REALLY finishes — which for an abandoned run is long after
// RunAsync has thrown ScriptTimeoutException — and BEFORE runTask completes (lmxopcua#843). This
// used to be a ContinueWith on runTask: a task is observably complete before its continuations
// run, so a caller that saw the completion (e.g. the virtual-tag evaluator's immediate retry after a
// raced ObjectDisposedException, or simply the next trigger) could
// start the key's NEXT run while this one still held the slot and be refused "previous
// evaluation still running" — a Bad publish for a script that had finished. Seen under load:
// 24 of 4,800 sequential same-key evaluations. A late-finishing orphan that breached the
// allocation budget quarantines its key first, so no re-trigger slips in between.
if (succeeded && BreachedBudget(slot.AllocatedBytes))
Quarantine(key, state, AllocationReason(slot.AllocatedBytes), "allocation");
SlotReleasingForTest?.Invoke(key);
Volatile.Write(ref state.RunStartedTimestamp, 0);
Interlocked.Exchange(ref state.InFlight, 0);
Interlocked.Decrement(ref _inFlight);
}
});
// Release the slot when the run REALLY finishes — which for an abandoned run is long after this
// method has thrown ScriptTimeoutException. Registered before the wait so it fires regardless of
// which way the wait below comes out. A late-finishing orphan that breached the allocation budget
// still quarantines its key here.
_ = runTask.ContinueWith(t =>
{
Volatile.Write(ref state.RunStartedTimestamp, 0);
Interlocked.Exchange(ref state.InFlight, 0);
Interlocked.Decrement(ref _inFlight);
runCts.Dispose();
if (t.IsFaulted) _ = t.Exception; // observed below by the awaiter when it is still listening
if (t.IsCompletedSuccessfully && BreachedBudget(slot.AllocatedBytes))
Quarantine(key, state, AllocationReason(slot.AllocatedBytes), "allocation");
}, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
// The budget clock starts when the pool starts the run, not when we queued it (P5): pool starvation
@@ -313,7 +332,7 @@ public sealed class ScriptRunGuard
&& !string.Equals(state.SourceIdentity, sourceIdentity, StringComparison.Ordinal))
{
// The script changed under this key: start from a clean slate. The old state's orphan (if any)
// releases the GLOBAL in-flight count through its own continuation; nothing else references it.
// releases the GLOBAL in-flight count when it finishes; nothing else references it.
_keys.TryUpdate(key, new KeyState(sourceIdentity), state);
continue;
}