feat(scripting): batch submit, exact per-item deadlines, scripted-alarm seam on the out-of-process runner (lmxopcua#780)
v2-ci / unit-tests (push) Waiting to run

- IScriptRunner.RunBatch: one EvalBatch frame per worker for every evaluation ready to run; per-item, streamed results.
  The default (InProcessScriptRunner) is RunAsync per item. Used by the Calculation pump (dequeue all, split at a
  repeated tag, publish in dequeue order), the VirtualTag seam (ScriptBatchCoalescer, no timer) and the scripted-alarm
  engine (whole dirty set). In process the pump and the engine stay one at a time, unchanged.
- Exact per-item deadlines: the worker hands work to an idle evaluation thread or defers it (EvalResults
  deferred_eval_ids); the host stops that run's clock and restarts T then G when it starts (started_eval_ids). A run
  queued behind a runaway is neither timed out nor struck and is a victim, not the culprit, at the kill.
- Scripted-alarm seam behind Scripting:Runner:Seams; unset Seams now means VirtualTag + Calculation, so alarms run out of
  process only when listed. Load still compiles each predicate (to PE bytes only) so a broken one fails the load; passes
  batch and apply in the serial order under the gate. Script faults freeze the condition BadInternalError; runner faults
  freeze it BadResourceUnavailable and re-evaluate on recovery; a dead worker never clears an alarm. IScriptRunnerProvider
  carries the runner to DriverHostActor.
- Perf harness: batch throughput (RunBatch vs RunAsync x n) and alarm-pass latency (in process / batched / serial);
  design §12.3 records macOS + windev. On windev the serial out-of-process alarm pass misses 250 ms from ~500 alarms;
  batched p99 <= 53 ms at 1,000.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-10 06:37:36 -05:00
parent 11a5c1d4e2
commit 0796928504
33 changed files with 2168 additions and 117 deletions
@@ -28,6 +28,31 @@ public interface IScriptRunner
/// <returns>The outcome; never faults for a script failure.</returns>
Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default);
/// <summary>
/// Submits several evaluations at once (design §4.5): the out-of-process runner sends every one that is ready to
/// run as ONE request frame per worker, instead of one frame and one submission per evaluation. Results are
/// per item and streamed — each task completes when its own evaluation does, so a slow item never holds a fast
/// sibling's result — and every item keeps exactly the semantics of <see cref="RunAsync"/>: its own budget, its
/// own single-flight slot, and its own blame (a runaway item is killed and struck alone; an item queued behind
/// it is neither timed out nor struck for its time).
/// </summary>
/// <remarks>
/// The default implementation — what <see cref="InProcessScriptRunner"/> uses — is <see cref="RunAsync"/> per
/// item, started together. A key that appears twice in one batch is refused for its second item exactly as two
/// concurrent <see cref="RunAsync"/> calls would be; callers that may repeat a key split the batch there.
/// Never throws for script faults; the returned tasks never fault.
/// </remarks>
/// <param name="invocations">What to run, in order.</param>
/// <param name="ct">Caller cancellation (shutdown); wins over every item's budget.</param>
/// <returns>One task per invocation, in the same order.</returns>
IReadOnlyList<Task<ScriptRunOutcome>> RunBatch(IReadOnlyList<ScriptInvocation> invocations, CancellationToken ct = default)
{
ArgumentNullException.ThrowIfNull(invocations);
var outcomes = new Task<ScriptRunOutcome>[invocations.Count];
for (var i = 0; i < outcomes.Length; i++) outcomes[i] = RunAsync(invocations[i], ct);
return outcomes;
}
/// <summary>
/// The deploy boundary for one seam: lifts every quarantine and forgets every remembered compile failure of
/// <paramref name="kind"/>, and drops its compiled scripts.
@@ -50,7 +75,7 @@ public enum ScriptContextKind
/// <summary>A Calculation tag: <c>VirtualTagContext</c> → <c>object?</c>.</summary>
Calculation = 2,
/// <summary>A scripted-alarm predicate: <c>AlarmPredicateContext</c> → <c>bool</c> (not run out of process yet).</summary>
/// <summary>A scripted-alarm predicate: <c>AlarmPredicateContext</c> → <c>bool</c>.</summary>
AlarmPredicate = 3,
}
@@ -142,3 +167,16 @@ public static class ScriptRunOutcomeExtensions
};
}
}
/// <summary>
/// Which <see cref="IScriptRunner"/> each script seam uses on this node (lmxopcua#780), for components that cannot
/// reference the Host's selection directly (the Runtime's scripted-alarm host). Null for a seam means its in-process
/// guard path.
/// </summary>
public interface IScriptRunnerProvider
{
/// <summary>The runner <paramref name="seam"/> must use, or null for its in-process path.</summary>
/// <param name="seam">The seam.</param>
/// <returns>The runner, or null.</returns>
IScriptRunner? ForSeam(ScriptSeam seam);
}
@@ -0,0 +1,106 @@
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting;
/// <summary>
/// Opportunistic batching for a seam whose evaluations are submitted independently — one virtual-tag actor each —
/// but often become ready together (one upstream burst fans out to many tags): evaluations submitted while another
/// submission is in progress are handed to <see cref="IScriptRunner.RunBatch"/> together, in arrival order, by the
/// submitter already holding the flush (lmxopcua#780, design §4.5).
/// </summary>
/// <remarks>
/// <para>There is no timer: an evaluation on an idle seam is submitted at once by its own caller, so batching never
/// adds latency. Per-item semantics are <see cref="IScriptRunner.RunBatch"/>'s — each item keeps its own budget, slot
/// and blame.</para>
/// <para>Each caller's task completes synchronously on the thread that completes its run (the runner's result
/// dispatcher), exactly as awaiting <see cref="IScriptRunner.RunAsync"/> directly does, so there is no extra thread
/// hop.</para>
/// <para>A key submitted twice concurrently is refused for the second exactly as two concurrent
/// <see cref="IScriptRunner.RunAsync"/> calls would be; the virtual-tag actor never has two evaluations in flight.</para>
/// </remarks>
public sealed class ScriptBatchCoalescer
{
private readonly IScriptRunner _runner;
private readonly object _lock = new();
private List<(ScriptInvocation Invocation, TaskCompletionSource<ScriptRunOutcome> Done)> _queue = [];
private bool _flushing;
/// <summary>Initializes the coalescer.</summary>
/// <param name="runner">The runner every batch goes to.</param>
public ScriptBatchCoalescer(IScriptRunner runner)
{
_runner = runner ?? throw new ArgumentNullException(nameof(runner));
}
/// <summary>Test hook: runs once per <see cref="IScriptRunner.RunBatch"/> call, with the batch size.</summary>
internal Action<int>? BatchSubmittedForTest { get; set; }
/// <summary>Submits one evaluation, together with any submitted concurrently.</summary>
/// <param name="invocation">What to run.</param>
/// <returns>The outcome; never faults.</returns>
public Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation)
{
ArgumentNullException.ThrowIfNull(invocation);
// Synchronous continuations: the caller resumes where its run completed, as with a direct await of RunAsync.
var done = new TaskCompletionSource<ScriptRunOutcome>(TaskCreationOptions.None);
lock (_lock)
{
_queue.Add((invocation, done));
if (_flushing) return done.Task;
_flushing = true;
}
Flush();
return done.Task;
}
private void Flush()
{
while (true)
{
List<(ScriptInvocation Invocation, TaskCompletionSource<ScriptRunOutcome> Done)> batch;
lock (_lock)
{
if (_queue.Count == 0)
{
_flushing = false;
return;
}
batch = _queue;
_queue = [];
}
IReadOnlyList<Task<ScriptRunOutcome>> outcomes;
try
{
var invocations = new ScriptInvocation[batch.Count];
for (var i = 0; i < invocations.Length; i++) invocations[i] = batch[i].Invocation;
BatchSubmittedForTest?.Invoke(invocations.Length);
outcomes = _runner.RunBatch(invocations);
}
catch (Exception ex)
{
// A defect in the runner must not strand the callers (nor the flush flag).
foreach (var (_, done) in batch) done.TrySetResult(ScriptRunOutcome.Failed($"script runner threw: {ex.Message}"));
continue;
}
for (var i = 0; i < batch.Count; i++) Link(outcomes[i], batch[i].Done);
}
}
private static void Link(Task<ScriptRunOutcome> outcome, TaskCompletionSource<ScriptRunOutcome> done)
{
if (outcome.IsCompleted)
{
Settle(outcome, done);
return;
}
outcome.ContinueWith(static (t, s) => Settle(t, (TaskCompletionSource<ScriptRunOutcome>)s!), done,
CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
}
private static void Settle(Task<ScriptRunOutcome> outcome, TaskCompletionSource<ScriptRunOutcome> done)
{
if (outcome.IsCompletedSuccessfully) done.TrySetResult(outcome.Result);
else done.TrySetResult(ScriptRunOutcome.Failed(
$"script runner threw: {outcome.Exception?.GetBaseException().Message ?? "cancelled"}"));
}
}
@@ -32,7 +32,10 @@ public enum ScriptSeam
/// <summary>Calculation tags (<c>CalculationEvaluator</c>).</summary>
Calculation = 1,
/// <summary>Scripted-alarm predicates (<c>ScriptedAlarmEngine</c>). Last in the rollout (design §14 #10).</summary>
/// <summary>
/// Scripted-alarm predicates (<c>ScriptedAlarmEngine</c>). Last in the rollout (design §14 #10): out of process only
/// when <c>Scripting:Runner:Seams</c> lists it — an unset list does not select it.
/// </summary>
ScriptedAlarm = 2,
}
@@ -80,7 +80,8 @@ public sealed class ScriptingOptions
/// <summary>
/// The seams <see cref="Runner"/> applies to when it is not <see cref="ScriptRunnerMode.InProcess"/>
/// (<c>Scripting:Runner:Seams</c>, e.g. <c>["Calculation"]</c>). Null or empty means all three. Bound by
/// (<c>Scripting:Runner:Seams</c>, e.g. <c>["Calculation"]</c>). Null or empty means VirtualTag and Calculation:
/// scripted alarms run out of process only when listed explicitly (design §14 #10, alarms last). Bound by
/// <c>ScriptingOptionsBinding</c> from the documented key, because <c>Scripting:Runner</c> is itself a value.
/// </summary>
public IReadOnlyList<ScriptSeam>? RunnerSeams { get; set; }
@@ -91,10 +92,10 @@ public sealed class ScriptingOptions
/// <summary>Whether <paramref name="seam"/> is selected for out-of-process execution by these options.</summary>
/// <param name="seam">The seam.</param>
/// <returns><c>true</c> when <see cref="Runner"/> is <see cref="ScriptRunnerMode.OutOfProcess"/> and the seam is in
/// <see cref="RunnerSeams"/> (or that list is empty).</returns>
/// <see cref="RunnerSeams"/> (or that list is empty and the seam is not <see cref="ScriptSeam.ScriptedAlarm"/>).</returns>
public bool IsOutOfProcess(ScriptSeam seam)
=> Runner == ScriptRunnerMode.OutOfProcess
&& (RunnerSeams is not { Count: > 0 } seams || seams.Contains(seam));
&& (RunnerSeams is { Count: > 0 } seams ? seams.Contains(seam) : seam != ScriptSeam.ScriptedAlarm);
/// <summary>Upper bound <see cref="Sanitized"/> applies to <see cref="StuckOrphanQuarantineFactor"/>.</summary>
public const double MaxStuckOrphanQuarantineFactor = 1000;
@@ -26,6 +26,8 @@
<ItemGroup>
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.Core.Scripting.Tests"/>
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests"/>
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.Host.Tests"/>
</ItemGroup>
</Project>