feat(scripting): batch submit, exact per-item deadlines, scripted-alarm seam on the out-of-process runner (lmxopcua#780)
v2-ci / unit-tests (push) Waiting to run
v2-ci / unit-tests (push) Waiting to run
- IScriptRunner.RunBatch: one EvalBatch frame per worker for every evaluation ready to run; per-item, streamed results. The default (InProcessScriptRunner) is RunAsync per item. Used by the Calculation pump (dequeue all, split at a repeated tag, publish in dequeue order), the VirtualTag seam (ScriptBatchCoalescer, no timer) and the scripted-alarm engine (whole dirty set). In process the pump and the engine stay one at a time, unchanged. - Exact per-item deadlines: the worker hands work to an idle evaluation thread or defers it (EvalResults deferred_eval_ids); the host stops that run's clock and restarts T then G when it starts (started_eval_ids). A run queued behind a runaway is neither timed out nor struck and is a victim, not the culprit, at the kill. - Scripted-alarm seam behind Scripting:Runner:Seams; unset Seams now means VirtualTag + Calculation, so alarms run out of process only when listed. Load still compiles each predicate (to PE bytes only) so a broken one fails the load; passes batch and apply in the serial order under the gate. Script faults freeze the condition BadInternalError; runner faults freeze it BadResourceUnavailable and re-evaluate on recovery; a dead worker never clears an alarm. IScriptRunnerProvider carries the runner to DriverHostActor. - Perf harness: batch throughput (RunBatch vs RunAsync x n) and alarm-pass latency (in process / batched / serial); design §12.3 records macOS + windev. On windev the serial out-of-process alarm pass misses 250 ms from ~500 alarms; batched p99 <= 53 ms at 1,000. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -28,6 +28,31 @@ public interface IScriptRunner
|
||||
/// <returns>The outcome; never faults for a script failure.</returns>
|
||||
Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// Submits several evaluations at once (design §4.5): the out-of-process runner sends every one that is ready to
|
||||
/// run as ONE request frame per worker, instead of one frame and one submission per evaluation. Results are
|
||||
/// per item and streamed — each task completes when its own evaluation does, so a slow item never holds a fast
|
||||
/// sibling's result — and every item keeps exactly the semantics of <see cref="RunAsync"/>: its own budget, its
|
||||
/// own single-flight slot, and its own blame (a runaway item is killed and struck alone; an item queued behind
|
||||
/// it is neither timed out nor struck for its time).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The default implementation — what <see cref="InProcessScriptRunner"/> uses — is <see cref="RunAsync"/> per
|
||||
/// item, started together. A key that appears twice in one batch is refused for its second item exactly as two
|
||||
/// concurrent <see cref="RunAsync"/> calls would be; callers that may repeat a key split the batch there.
|
||||
/// Never throws for script faults; the returned tasks never fault.
|
||||
/// </remarks>
|
||||
/// <param name="invocations">What to run, in order.</param>
|
||||
/// <param name="ct">Caller cancellation (shutdown); wins over every item's budget.</param>
|
||||
/// <returns>One task per invocation, in the same order.</returns>
|
||||
IReadOnlyList<Task<ScriptRunOutcome>> RunBatch(IReadOnlyList<ScriptInvocation> invocations, CancellationToken ct = default)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(invocations);
|
||||
var outcomes = new Task<ScriptRunOutcome>[invocations.Count];
|
||||
for (var i = 0; i < outcomes.Length; i++) outcomes[i] = RunAsync(invocations[i], ct);
|
||||
return outcomes;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The deploy boundary for one seam: lifts every quarantine and forgets every remembered compile failure of
|
||||
/// <paramref name="kind"/>, and drops its compiled scripts.
|
||||
@@ -50,7 +75,7 @@ public enum ScriptContextKind
|
||||
/// <summary>A Calculation tag: <c>VirtualTagContext</c> → <c>object?</c>.</summary>
|
||||
Calculation = 2,
|
||||
|
||||
/// <summary>A scripted-alarm predicate: <c>AlarmPredicateContext</c> → <c>bool</c> (not run out of process yet).</summary>
|
||||
/// <summary>A scripted-alarm predicate: <c>AlarmPredicateContext</c> → <c>bool</c>.</summary>
|
||||
AlarmPredicate = 3,
|
||||
}
|
||||
|
||||
@@ -142,3 +167,16 @@ public static class ScriptRunOutcomeExtensions
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Which <see cref="IScriptRunner"/> each script seam uses on this node (lmxopcua#780), for components that cannot
|
||||
/// reference the Host's selection directly (the Runtime's scripted-alarm host). Null for a seam means its in-process
|
||||
/// guard path.
|
||||
/// </summary>
|
||||
public interface IScriptRunnerProvider
|
||||
{
|
||||
/// <summary>The runner <paramref name="seam"/> must use, or null for its in-process path.</summary>
|
||||
/// <param name="seam">The seam.</param>
|
||||
/// <returns>The runner, or null.</returns>
|
||||
IScriptRunner? ForSeam(ScriptSeam seam);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
||||
|
||||
/// <summary>
|
||||
/// Opportunistic batching for a seam whose evaluations are submitted independently — one virtual-tag actor each —
|
||||
/// but often become ready together (one upstream burst fans out to many tags): evaluations submitted while another
|
||||
/// submission is in progress are handed to <see cref="IScriptRunner.RunBatch"/> together, in arrival order, by the
|
||||
/// submitter already holding the flush (lmxopcua#780, design §4.5).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>There is no timer: an evaluation on an idle seam is submitted at once by its own caller, so batching never
|
||||
/// adds latency. Per-item semantics are <see cref="IScriptRunner.RunBatch"/>'s — each item keeps its own budget, slot
|
||||
/// and blame.</para>
|
||||
/// <para>Each caller's task completes synchronously on the thread that completes its run (the runner's result
|
||||
/// dispatcher), exactly as awaiting <see cref="IScriptRunner.RunAsync"/> directly does, so there is no extra thread
|
||||
/// hop.</para>
|
||||
/// <para>A key submitted twice concurrently is refused for the second exactly as two concurrent
|
||||
/// <see cref="IScriptRunner.RunAsync"/> calls would be; the virtual-tag actor never has two evaluations in flight.</para>
|
||||
/// </remarks>
|
||||
public sealed class ScriptBatchCoalescer
|
||||
{
|
||||
private readonly IScriptRunner _runner;
|
||||
private readonly object _lock = new();
|
||||
private List<(ScriptInvocation Invocation, TaskCompletionSource<ScriptRunOutcome> Done)> _queue = [];
|
||||
private bool _flushing;
|
||||
|
||||
/// <summary>Initializes the coalescer.</summary>
|
||||
/// <param name="runner">The runner every batch goes to.</param>
|
||||
public ScriptBatchCoalescer(IScriptRunner runner)
|
||||
{
|
||||
_runner = runner ?? throw new ArgumentNullException(nameof(runner));
|
||||
}
|
||||
|
||||
/// <summary>Test hook: runs once per <see cref="IScriptRunner.RunBatch"/> call, with the batch size.</summary>
|
||||
internal Action<int>? BatchSubmittedForTest { get; set; }
|
||||
|
||||
/// <summary>Submits one evaluation, together with any submitted concurrently.</summary>
|
||||
/// <param name="invocation">What to run.</param>
|
||||
/// <returns>The outcome; never faults.</returns>
|
||||
public Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(invocation);
|
||||
// Synchronous continuations: the caller resumes where its run completed, as with a direct await of RunAsync.
|
||||
var done = new TaskCompletionSource<ScriptRunOutcome>(TaskCreationOptions.None);
|
||||
lock (_lock)
|
||||
{
|
||||
_queue.Add((invocation, done));
|
||||
if (_flushing) return done.Task;
|
||||
_flushing = true;
|
||||
}
|
||||
Flush();
|
||||
return done.Task;
|
||||
}
|
||||
|
||||
private void Flush()
|
||||
{
|
||||
while (true)
|
||||
{
|
||||
List<(ScriptInvocation Invocation, TaskCompletionSource<ScriptRunOutcome> Done)> batch;
|
||||
lock (_lock)
|
||||
{
|
||||
if (_queue.Count == 0)
|
||||
{
|
||||
_flushing = false;
|
||||
return;
|
||||
}
|
||||
batch = _queue;
|
||||
_queue = [];
|
||||
}
|
||||
|
||||
IReadOnlyList<Task<ScriptRunOutcome>> outcomes;
|
||||
try
|
||||
{
|
||||
var invocations = new ScriptInvocation[batch.Count];
|
||||
for (var i = 0; i < invocations.Length; i++) invocations[i] = batch[i].Invocation;
|
||||
BatchSubmittedForTest?.Invoke(invocations.Length);
|
||||
outcomes = _runner.RunBatch(invocations);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// A defect in the runner must not strand the callers (nor the flush flag).
|
||||
foreach (var (_, done) in batch) done.TrySetResult(ScriptRunOutcome.Failed($"script runner threw: {ex.Message}"));
|
||||
continue;
|
||||
}
|
||||
|
||||
for (var i = 0; i < batch.Count; i++) Link(outcomes[i], batch[i].Done);
|
||||
}
|
||||
}
|
||||
|
||||
private static void Link(Task<ScriptRunOutcome> outcome, TaskCompletionSource<ScriptRunOutcome> done)
|
||||
{
|
||||
if (outcome.IsCompleted)
|
||||
{
|
||||
Settle(outcome, done);
|
||||
return;
|
||||
}
|
||||
outcome.ContinueWith(static (t, s) => Settle(t, (TaskCompletionSource<ScriptRunOutcome>)s!), done,
|
||||
CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
|
||||
}
|
||||
|
||||
private static void Settle(Task<ScriptRunOutcome> outcome, TaskCompletionSource<ScriptRunOutcome> done)
|
||||
{
|
||||
if (outcome.IsCompletedSuccessfully) done.TrySetResult(outcome.Result);
|
||||
else done.TrySetResult(ScriptRunOutcome.Failed(
|
||||
$"script runner threw: {outcome.Exception?.GetBaseException().Message ?? "cancelled"}"));
|
||||
}
|
||||
}
|
||||
@@ -32,7 +32,10 @@ public enum ScriptSeam
|
||||
/// <summary>Calculation tags (<c>CalculationEvaluator</c>).</summary>
|
||||
Calculation = 1,
|
||||
|
||||
/// <summary>Scripted-alarm predicates (<c>ScriptedAlarmEngine</c>). Last in the rollout (design §14 #10).</summary>
|
||||
/// <summary>
|
||||
/// Scripted-alarm predicates (<c>ScriptedAlarmEngine</c>). Last in the rollout (design §14 #10): out of process only
|
||||
/// when <c>Scripting:Runner:Seams</c> lists it — an unset list does not select it.
|
||||
/// </summary>
|
||||
ScriptedAlarm = 2,
|
||||
}
|
||||
|
||||
|
||||
@@ -80,7 +80,8 @@ public sealed class ScriptingOptions
|
||||
|
||||
/// <summary>
|
||||
/// The seams <see cref="Runner"/> applies to when it is not <see cref="ScriptRunnerMode.InProcess"/>
|
||||
/// (<c>Scripting:Runner:Seams</c>, e.g. <c>["Calculation"]</c>). Null or empty means all three. Bound by
|
||||
/// (<c>Scripting:Runner:Seams</c>, e.g. <c>["Calculation"]</c>). Null or empty means VirtualTag and Calculation:
|
||||
/// scripted alarms run out of process only when listed explicitly (design §14 #10, alarms last). Bound by
|
||||
/// <c>ScriptingOptionsBinding</c> from the documented key, because <c>Scripting:Runner</c> is itself a value.
|
||||
/// </summary>
|
||||
public IReadOnlyList<ScriptSeam>? RunnerSeams { get; set; }
|
||||
@@ -91,10 +92,10 @@ public sealed class ScriptingOptions
|
||||
/// <summary>Whether <paramref name="seam"/> is selected for out-of-process execution by these options.</summary>
|
||||
/// <param name="seam">The seam.</param>
|
||||
/// <returns><c>true</c> when <see cref="Runner"/> is <see cref="ScriptRunnerMode.OutOfProcess"/> and the seam is in
|
||||
/// <see cref="RunnerSeams"/> (or that list is empty).</returns>
|
||||
/// <see cref="RunnerSeams"/> (or that list is empty and the seam is not <see cref="ScriptSeam.ScriptedAlarm"/>).</returns>
|
||||
public bool IsOutOfProcess(ScriptSeam seam)
|
||||
=> Runner == ScriptRunnerMode.OutOfProcess
|
||||
&& (RunnerSeams is not { Count: > 0 } seams || seams.Contains(seam));
|
||||
&& (RunnerSeams is { Count: > 0 } seams ? seams.Contains(seam) : seam != ScriptSeam.ScriptedAlarm);
|
||||
|
||||
/// <summary>Upper bound <see cref="Sanitized"/> applies to <see cref="StuckOrphanQuarantineFactor"/>.</summary>
|
||||
public const double MaxStuckOrphanQuarantineFactor = 1000;
|
||||
|
||||
@@ -26,6 +26,8 @@
|
||||
|
||||
<ItemGroup>
|
||||
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.Core.Scripting.Tests"/>
|
||||
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests"/>
|
||||
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.Host.Tests"/>
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
||||
Reference in New Issue
Block a user