Joseph Doherty
f7ca2c5826
The soft deadline T was armed on the host at submit, so the pipe round trip
and the time a starved worker waits to be scheduled counted against the
script: under load a trivial script reported 'script timed out', in Shadow
mode that was a script mismatch, and at T + G the host killed the worker and
struck the key.
- Protocol (additive, negotiated): Hello/Ready.worker_soft_clock and
EvalResults.soft_expired_eval_ids. protocol_version stays 1; a worker that
does not echo the flag keeps the pre-#827 host clock.
- Worker: the watchdog runs each run's soft clock from the moment an
evaluation thread starts it and reports the crossing (woken early for
budgets shorter than its 50 ms poll). It still never cancels by itself;
its fail-fast is anchored at the soft notice + G + margin.
- Host (PendingEval): the soft notice answers the caller and arms the kill G
later. The host's own timer becomes a transport deadline, T + G from submit
(restarted at a deferred run's start); passing it with no result and no
notice yields TransportTimedOut -> Unavailable 'script worker unresponsive'
(BadResourceUnavailable, recovery = the run's completion), no strike, no
kill, no Cancel. A re-trigger refused by that still-unanswered run is
Unavailable too, not 'previous run still running'. Counted as
otopcua.script.ipc.timeouts{kind}.
- Frame writer: coalescing now carries soft_expired_eval_ids (it silently
dropped them); the host processes soft notices before results in a frame.
- Shadow: unresponsive is shadow_unavailable (Unavailable), never a mismatch.
Tests: WorkerSoftClockTests suspends the real worker (SIGSTOP via libc kill /
NtSuspendProcess) - a stall past T still returns 7; a stall past T + G is a
runner fault with nothing struck; a late-starting runaway is still timed out
at T from its start and killed G after; Shadow stays match /
shadow_unavailable; both clocks enforce T and G; PendingEval state machine.
196 lines
10 KiB
C#
196 lines
10 KiB
C#
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
|
|
|
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
|
|
|
/// <summary>
|
|
/// The execution seam every host script owner can run a compiled script through (lmxopcua#780, design §11): a
|
|
/// script key, its source, its input snapshots and its budget go in; one <see cref="ScriptRunOutcome"/> comes
|
|
/// out. Implementations never throw for script faults.
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// <para>
|
|
/// <see cref="InProcessScriptRunner"/> is today's <see cref="ScriptRunGuard"/> + <see cref="CompiledScriptCache{TContext, TResult}"/>
|
|
/// behind this interface (the parity baseline). The out-of-process runner
|
|
/// (<c>ZB.MOM.WW.OtOpcUa.Core.Scripting.Runner</c>) runs the same compile in a pre-emptible worker process.
|
|
/// The passthrough fast path (<c>PassthroughScript</c>) never reaches a runner: it runs no user code.
|
|
/// </para>
|
|
/// <para>
|
|
/// Keys are namespaced by <see cref="ScriptInvocation.Kind"/>, so a virtual-tag id and a calc RawPath that
|
|
/// happen to be equal never share quarantine state, and <see cref="Reset"/> for one seam leaves the other
|
|
/// seams' quarantines alone.
|
|
/// </para>
|
|
/// </remarks>
|
|
public interface IScriptRunner
|
|
{
|
|
/// <summary>Runs one compiled evaluation.</summary>
|
|
/// <param name="invocation">What to run.</param>
|
|
/// <param name="ct">Caller cancellation (shutdown); wins over the budget.</param>
|
|
/// <returns>The outcome; never faults for a script failure.</returns>
|
|
Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default);
|
|
|
|
/// <summary>
|
|
/// Submits several evaluations at once (design §4.5): the out-of-process runner sends every one that is ready to
|
|
/// run as ONE request frame per worker, instead of one frame and one submission per evaluation. Results are
|
|
/// per item and streamed — each task completes when its own evaluation does, so a slow item never holds a fast
|
|
/// sibling's result — and every item keeps exactly the semantics of <see cref="RunAsync"/>: its own budget, its
|
|
/// own single-flight slot, and its own blame (a runaway item is killed and struck alone; an item queued behind
|
|
/// it is neither timed out nor struck for its time).
|
|
/// </summary>
|
|
/// <remarks>
|
|
/// The default implementation — what <see cref="InProcessScriptRunner"/> uses — is <see cref="RunAsync"/> per
|
|
/// item, started together. A key that appears twice in one batch is refused for its second item exactly as two
|
|
/// concurrent <see cref="RunAsync"/> calls would be; callers that may repeat a key split the batch there.
|
|
/// Never throws for script faults; the returned tasks never fault.
|
|
/// </remarks>
|
|
/// <param name="invocations">What to run, in order.</param>
|
|
/// <param name="ct">Caller cancellation (shutdown); wins over every item's budget.</param>
|
|
/// <returns>One task per invocation, in the same order.</returns>
|
|
IReadOnlyList<Task<ScriptRunOutcome>> RunBatch(IReadOnlyList<ScriptInvocation> invocations, CancellationToken ct = default)
|
|
{
|
|
ArgumentNullException.ThrowIfNull(invocations);
|
|
var outcomes = new Task<ScriptRunOutcome>[invocations.Count];
|
|
for (var i = 0; i < outcomes.Length; i++) outcomes[i] = RunAsync(invocations[i], ct);
|
|
return outcomes;
|
|
}
|
|
|
|
/// <summary>
|
|
/// The deploy boundary for one seam: lifts every quarantine and forgets every remembered compile failure of
|
|
/// <paramref name="kind"/>, and drops its compiled scripts.
|
|
/// </summary>
|
|
/// <param name="kind">The seam's context kind.</param>
|
|
void Reset(ScriptContextKind kind);
|
|
|
|
/// <summary>The keys of <paramref name="kind"/> currently quarantined.</summary>
|
|
/// <param name="kind">The seam's context kind.</param>
|
|
/// <returns>The quarantined keys (un-namespaced).</returns>
|
|
IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind);
|
|
}
|
|
|
|
/// <summary>Which script context (and result type) a script compiles against.</summary>
|
|
public enum ScriptContextKind
|
|
{
|
|
/// <summary>A virtual tag: <c>VirtualTagContext</c> → <c>object?</c>.</summary>
|
|
VirtualTag = 1,
|
|
|
|
/// <summary>A Calculation tag: <c>VirtualTagContext</c> → <c>object?</c>.</summary>
|
|
Calculation = 2,
|
|
|
|
/// <summary>A scripted-alarm predicate: <c>AlarmPredicateContext</c> → <c>bool</c>.</summary>
|
|
AlarmPredicate = 3,
|
|
}
|
|
|
|
/// <summary>One evaluation handed to an <see cref="IScriptRunner"/>.</summary>
|
|
/// <param name="Key">The script instance key (virtual-tag id, calc RawPath, alarm id).</param>
|
|
/// <param name="Kind">The context kind.</param>
|
|
/// <param name="Source">The script source (also its identity: a change lifts the key's quarantine).</param>
|
|
/// <param name="Inputs">The read cache: dependency snapshots by path. Not mutated until the run completes.</param>
|
|
/// <param name="Timeout">The soft budget <c>T</c>.</param>
|
|
/// <param name="ScriptLogger">The per-evaluation script logger <c>ctx.Logger</c> writes through; null ⇒ a silent logger.</param>
|
|
public sealed record ScriptInvocation(
|
|
string Key,
|
|
ScriptContextKind Kind,
|
|
string Source,
|
|
IReadOnlyDictionary<string, DataValueSnapshot> Inputs,
|
|
TimeSpan Timeout,
|
|
Serilog.ILogger? ScriptLogger = null);
|
|
|
|
/// <summary>How an evaluation ended.</summary>
|
|
public enum ScriptRunStatus
|
|
{
|
|
/// <summary>The script returned <see cref="ScriptRunOutcome.Value"/>.</summary>
|
|
Ok = 0,
|
|
|
|
/// <summary>A script-attributable failure (threw, timed out, refused, quarantined, did not compile) — Bad as today.</summary>
|
|
Failed = 1,
|
|
|
|
/// <summary>
|
|
/// The runner could not run the script, and it is not the script's fault: its worker restarted under it, or no
|
|
/// worker became ready in time. Published as <c>BadResourceUnavailable</c> (design §9 / §14 #3).
|
|
/// </summary>
|
|
Unavailable = 2,
|
|
}
|
|
|
|
/// <summary>The result of one <see cref="IScriptRunner.RunAsync"/>.</summary>
|
|
/// <param name="Status">How it ended.</param>
|
|
/// <param name="Value">The script's result when <see cref="ScriptRunStatus.Ok"/>.</param>
|
|
/// <param name="Reason">The failure reason otherwise; the same strings the evaluators have always published.</param>
|
|
public sealed record ScriptRunOutcome(ScriptRunStatus Status, object? Value, string? Reason)
|
|
{
|
|
/// <summary>Reason when the worker running the script died under it (another script's kill, a crash).</summary>
|
|
public const string WorkerRestartedReason = "script worker restarted";
|
|
|
|
/// <summary>Reason when no worker became ready within <see cref="ScriptWorkerOptions.UnavailableGrace"/>.</summary>
|
|
public const string RunnerUnavailableReason = "script runner unavailable";
|
|
|
|
/// <summary>Reason when the worker could not take more work (the out-of-process runner's back-pressure answer).</summary>
|
|
public const string WorkerBusyReason = "script worker busy";
|
|
|
|
/// <summary>
|
|
/// Reason when the worker neither answered nor reported the script over its budget within T + G of submit — the
|
|
/// pipe or a starved / stalled worker process, not the script (lmxopcua#827).
|
|
/// </summary>
|
|
public const string WorkerUnresponsiveReason = "script worker unresponsive";
|
|
|
|
/// <summary>
|
|
/// For <see cref="ScriptRunStatus.Unavailable"/>: completes when the runner can run this script again (its worker
|
|
/// is back), so the seam re-evaluates it then with its latest inputs (design §9, "re-evaluated on the shard's
|
|
/// return"). Null when there is nothing to wait for.
|
|
/// </summary>
|
|
public Task? Recovery { get; init; }
|
|
|
|
/// <summary>A successful run.</summary>
|
|
/// <param name="value">The script's result.</param>
|
|
/// <returns>The outcome.</returns>
|
|
public static ScriptRunOutcome Ok(object? value) => new(ScriptRunStatus.Ok, value, null);
|
|
|
|
/// <summary>A script-attributable failure.</summary>
|
|
/// <param name="reason">The reason.</param>
|
|
/// <returns>The outcome.</returns>
|
|
public static ScriptRunOutcome Failed(string reason) => new(ScriptRunStatus.Failed, null, reason);
|
|
|
|
/// <summary>A runner-attributable failure.</summary>
|
|
/// <param name="reason">The reason.</param>
|
|
/// <returns>The outcome.</returns>
|
|
public static ScriptRunOutcome Unavailable(string reason) => new(ScriptRunStatus.Unavailable, null, reason);
|
|
}
|
|
|
|
/// <summary>Maps an <see cref="IScriptRunner"/> outcome onto the value seams' result type.</summary>
|
|
public static class ScriptRunOutcomeExtensions
|
|
{
|
|
/// <summary>
|
|
/// <see cref="ScriptRunStatus.Ok"/> → <c>Ok(value)</c>; <see cref="ScriptRunStatus.Failed"/> → <c>Failure(reason)</c>
|
|
/// (published <c>BadInternalError</c>, as today); <see cref="ScriptRunStatus.Unavailable"/> →
|
|
/// <c>Unavailable(reason)</c> (published <c>BadResourceUnavailable</c>).
|
|
/// </summary>
|
|
/// <param name="outcome">The runner's outcome.</param>
|
|
/// <returns>The seam result.</returns>
|
|
public static Commons.Engines.VirtualTagEvalResult ToVirtualTagEvalResult(this ScriptRunOutcome outcome)
|
|
{
|
|
ArgumentNullException.ThrowIfNull(outcome);
|
|
return outcome.Status switch
|
|
{
|
|
ScriptRunStatus.Ok => Commons.Engines.VirtualTagEvalResult.Ok(outcome.Value),
|
|
ScriptRunStatus.Unavailable => Commons.Engines.VirtualTagEvalResult.Unavailable(
|
|
outcome.Reason ?? ScriptRunOutcome.RunnerUnavailableReason) with { RetryWhen = outcome.Recovery },
|
|
_ => Commons.Engines.VirtualTagEvalResult.Failure(outcome.Reason ?? "evaluator failure"),
|
|
};
|
|
}
|
|
}
|
|
|
|
/// <summary>
|
|
/// Which <see cref="IScriptRunner"/> each script seam uses on this node (lmxopcua#780), for components that cannot
|
|
/// reference the Host's selection directly (the Runtime's scripted-alarm host). Null for a seam means its in-process
|
|
/// guard path.
|
|
/// </summary>
|
|
public interface IScriptRunnerProvider
|
|
{
|
|
/// <summary>
|
|
/// The runner <paramref name="seam"/> must use, or null for its in-process path. Under
|
|
/// <c>Scripting:Runner=Shadow</c> this is a <see cref="ShadowScriptRunner"/>: the seam recognises it as an
|
|
/// <see cref="IScriptShadow"/>, keeps its in-process path and mirrors each evaluation to it.
|
|
/// </summary>
|
|
/// <param name="seam">The seam.</param>
|
|
/// <returns>The runner, or null.</returns>
|
|
IScriptRunner? ForSeam(ScriptSeam seam);
|
|
}
|