feat(scripting): out-of-process pre-emptible script runner behind Scripting:Runner (lmxopcua#780 P1)
v2-ci / unit-tests (push) Waiting to run

Opt-in (Scripting:Runner=OutOfProcess; InProcess stays the default and its code path is unchanged). Implements
the P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md with the owner's §14 decisions:

- Options: Scripting:Runner (InProcess | Shadow | OutOfProcess), Scripting:Runner:Seams, Scripting:Worker:*
  (Shards 2, HardGrace max(1 s,T), HeapLimitBytes 512 MiB, QuarantineAfterKills 2, UnavailableGrace 5 s,
  IsolationProbationRuns 20, PeCacheDir validated absolute + outside the content root (fail closed),
  ExecutablePath). Shadow is not implemented yet: runs InProcess with a startup Warning.
- Core.Scripting: IScriptRunner seam + InProcessScriptRunner (today's guard, the parity baseline).
- Core.Scripting.Protocol: protobuf envelopes, 4-byte LE framing (zero/oversize/truncated/garbage refused),
  Variant carrying the exact CLR type; unsupported types are explicit failures.
- ScriptWorker app: compiles in the worker, P-thread scheduler, host-driven soft cancel, fail-fast
  watchdog, heartbeats, ctx.Logger forwarding, exits on pipe EOF / parent gone, forbidden-assembly
  self-check; Akka and Grpc.Core.Api excluded from its folder. Shipped in script-worker/ (Build + Publish).
- Core.Scripting.OutOfProcess: K shards + isolation worker, soft T / hard T+G kill, attribution
  (deadline kill = one key; crash = suspects -> isolation; isolation crash = strike), kill-strike
  quarantine, backoff 250 ms -> 30 s, fail closed, cleared allow-listed env, nonce + pid handshake on a
  current-user pipe, Windows Job Object.
- VirtualTag + Calculation seams use the runner when selected; a runner-side failure (worker restarted /
  runner unavailable) publishes BadResourceUnavailable (0x80040000) — ScadaBridge#112. Script faults keep
  BadInternalError. The scripted-alarm seam stays in process (alarms last).

Tests: new ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests (Category=ScriptWorker, real processes; green on macOS and
on windev/Windows), seam tests in Host/Runtime/Calculation suites; mutation-checked.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 23:31:51 -05:00
parent f5c30ed0ef
commit 69a6e0b4c8
49 changed files with 5107 additions and 25 deletions
@@ -87,6 +87,19 @@ public sealed record VirtualTagEvalResult(bool Success, object? Value, string? R
/// <param name="reason">Description of the failure cause.</param>
/// <returns>A failed evaluation result.</returns>
public static VirtualTagEvalResult Failure(string reason) => new(false, null, reason);
/// <summary>
/// The OPC UA status a <see cref="Failure"/> publishes. Null (every failure the in-process path produces) means
/// the seam's long-standing <c>BadInternalError</c>; the out-of-process runner sets
/// <see cref="OpcUa.OpcUaStatus.BadResourceUnavailable"/> when the runner, not the script, failed (lmxopcua#780).
/// </summary>
public uint? StatusCode { get; init; }
/// <summary>A failure that is the runner's, not the script's: published <c>BadResourceUnavailable</c>.</summary>
/// <param name="reason">Description of the failure cause.</param>
/// <returns>A failed evaluation result carrying <see cref="OpcUa.OpcUaStatus.BadResourceUnavailable"/>.</returns>
public static VirtualTagEvalResult Unavailable(string reason) =>
new(false, null, reason) { StatusCode = OpcUa.OpcUaStatus.BadResourceUnavailable };
}
/// <summary>
@@ -47,6 +47,12 @@ public static class OpcUaStatus
/// see docs/VirtualTags.md §"Quarantine: what a client sees").</summary>
public const uint BadInternalError = 0x80020000;
/// <summary>The operating system / runtime could not provide a resource. The out-of-process script runner's code
/// for "the script did not run and it is not the script's fault" — its worker restarted under it, or no worker
/// was ready in time (lmxopcua#780, design §9). Distinct from <see cref="BadInternalError"/> so an operator, and
/// ScadaBridge, can tell a sick node from a broken script. Only produced under <c>Scripting:Runner=OutOfProcess</c>.</summary>
public const uint BadResourceUnavailable = 0x80040000;
/// <summary>Part 4 severity bucket of <paramref name="statusCode"/>: the top two bits.</summary>
/// <param name="statusCode">The status code to classify.</param>
/// <returns>0 for Good, 1 for Uncertain, 2 or 3 for Bad.</returns>
@@ -0,0 +1,505 @@
using System.Collections.Concurrent;
using System.Security.Cryptography;
using System.Text;
using Serilog;
using Serilog.Core;
using Serilog.Events;
using ZB.MOM.WW.OtOpcUa.Commons.Engines;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
/// <summary>
/// The pre-emptible <see cref="IScriptRunner"/> (lmxopcua#780, design §3–§9): compiled scripts run in a pool of
/// <c>ScriptWorker</c> processes — K shards placed by a stable hash of the script key, plus an isolation worker
/// for suspects — so a script that overruns its hard deadline is stopped by killing its worker, and an OOM, a
/// native crash or a fail-fast takes down a worker instead of the server.
/// </summary>
/// <remarks>
/// <list type="bullet">
/// <item><b>Two deadlines.</b> At the soft deadline <c>T</c> the caller gets today's timeout failure (and the
/// run token is cancelled inside the worker). At <c>T + G</c> (<see cref="ScriptWorkerOptions.EffectiveHardGrace"/>)
/// the worker is killed and the key takes a kill strike.</item>
/// <item><b>Attribution.</b> A deadline kill blames exactly one key; the other evaluations in flight on that
/// worker are victims and report <see cref="ScriptRunOutcome.WorkerRestartedReason"/> (no strike). A crash with
/// no deadline crossed makes every key in flight a suspect, and suspects run in the isolation worker one at a
/// time, where a crash names exactly one key. A suspect returns to its shard after
/// <see cref="ScriptWorkerOptions.IsolationProbationRuns"/> clean runs.</item>
/// <item><b>Quarantine</b> lives here, in the host, so it survives worker restarts: kill strikes (within an
/// hour), consecutive soft timeouts, or one allocation-budget breach. Reasons keep the
/// <see cref="ScriptRunGuard.QuarantinedReasonPrefix"/>, and a source change or <see cref="Reset"/> lifts them.</item>
/// <item><b>Fail closed.</b> When no worker is ready within <see cref="ScriptWorkerOptions.UnavailableGrace"/>
/// the result is <see cref="ScriptRunStatus.Unavailable"/> — never an in-process run.</item>
/// </list>
/// </remarks>
public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable, IDisposable
{
/// <summary>The worker entry assembly's file name.</summary>
public const string WorkerAssemblyFileName = "ZB.MOM.WW.OtOpcUa.ScriptWorker.dll";
/// <summary>The folder under the host's base directory the Host build places the worker in.</summary>
public const string WorkerFolderName = "script-worker";
/// <summary>Failure reason when the worker could not take more work.</summary>
public const string WorkerBusyReason = "script worker busy";
private static readonly TimeSpan KillStrikeWindow = TimeSpan.FromHours(1);
private readonly ScriptingOptions _options;
private readonly ScriptWorkerOptions _worker;
private readonly ILogger _log;
private readonly ILogger _scriptRoot;
private readonly WorkerShard[] _shards;
private readonly WorkerShard _isolation;
private readonly ConcurrentDictionary<string, KeyState> _keys = new(StringComparer.Ordinal);
private readonly ConcurrentDictionary<string, string> _compileFailures = new(StringComparer.Ordinal);
private int _disposed;
/// <summary>Initializes the runner. No process starts until the first evaluation.</summary>
/// <param name="options">The node's <c>Scripting</c> options (sanitized again here).</param>
/// <param name="log">Host diagnostics: starts, kills, quarantines; null ⇒ Serilog's global logger.</param>
/// <param name="scriptRoot">Fallback logger for forwarded <c>ctx.Logger</c> lines; null ⇒ <paramref name="log"/>.</param>
public OutOfProcessScriptRunner(ScriptingOptions options, ILogger? log = null, ILogger? scriptRoot = null)
: this(options, log, scriptRoot, heartbeatInterval: TimeSpan.FromSeconds(1), loadTimeout: TimeSpan.FromSeconds(60))
{
}
internal OutOfProcessScriptRunner(ScriptingOptions options, ILogger? log, ILogger? scriptRoot,
TimeSpan heartbeatInterval, TimeSpan loadTimeout)
{
ArgumentNullException.ThrowIfNull(options);
_options = options.Sanitized(out _);
_worker = _options.Worker;
_log = log ?? Log.Logger;
_scriptRoot = scriptRoot ?? _log;
LoadTimeout = loadTimeout;
WorkerPath = ResolveWorkerPath(_worker);
var evalThreads = Math.Max(2, Environment.ProcessorCount / _worker.Shards);
_shards = new WorkerShard[_worker.Shards];
for (var i = 0; i < _shards.Length; i++)
{
var label = i.ToString(System.Globalization.CultureInfo.InvariantCulture);
_shards[i] = NewShard(label, isolation: false, evalThreads, heartbeatInterval);
}
_isolation = NewShard("iso", isolation: true, evalThreads: 1, heartbeatInterval);
}
/// <summary>The worker entry assembly this runner launches.</summary>
public string WorkerPath { get; }
/// <summary>How long a compile in the worker may take before the worker is killed.</summary>
internal TimeSpan LoadTimeout { get; }
/// <summary>The shard workers (tests).</summary>
internal IReadOnlyList<WorkerShard> Shards => _shards;
/// <summary>The isolation worker (tests).</summary>
internal WorkerShard Isolation => _isolation;
/// <summary>Scoped keys currently treated as suspects (tests / diagnostics).</summary>
internal IReadOnlyCollection<string> SuspectKeys =>
_keys.Where(kv => kv.Value.Suspect).Select(kv => kv.Key).ToArray();
/// <summary>The shard a key is placed on when it is not a suspect: a stable FNV-1a hash, so placement survives restarts.</summary>
/// <param name="scopedKey">The scoped key.</param>
/// <param name="shardCount">K.</param>
/// <returns>The shard index.</returns>
internal static int ShardIndex(string scopedKey, int shardCount)
{
var hash = 2166136261u;
foreach (var b in Encoding.UTF8.GetBytes(scopedKey))
{
hash ^= b;
hash *= 16777619u;
}
return (int)(hash % (uint)shardCount);
}
/// <summary>The default worker path, or the configured one.</summary>
/// <param name="worker">The worker options.</param>
/// <returns>The absolute worker path.</returns>
public static string ResolveWorkerPath(ScriptWorkerOptions worker)
=> string.IsNullOrWhiteSpace(worker.ExecutablePath)
? Path.Combine(AppContext.BaseDirectory, WorkerFolderName, WorkerAssemblyFileName)
: Path.GetFullPath(worker.ExecutablePath);
/// <inheritdoc />
public async Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default)
{
ArgumentNullException.ThrowIfNull(invocation);
if (Volatile.Read(ref _disposed) != 0) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.RunnerUnavailableReason);
var key = InProcessScriptRunner.ScopedKey(invocation.Kind, invocation.Key);
var state = Admit(key, invocation.Source);
if (state.QuarantineReason is { } quarantined) return ScriptRunOutcome.Failed(quarantined);
var sha = SHA256.HashData(Encoding.UTF8.GetBytes(invocation.Source));
var failureKey = InProcessScriptRunner.ScopedKey(invocation.Kind, Convert.ToHexString(sha));
if (_compileFailures.TryGetValue(failureKey, out var knownFailure)) return ScriptRunOutcome.Failed(knownFailure);
if (Interlocked.CompareExchange(ref state.InFlight, 1, 0) != 0)
{
ScriptRunGuard.RefusedCounter.Add(1, new KeyValuePair<string, object?>("script", key));
return ScriptRunOutcome.Failed(ScriptRunGuard.PreviousRunStillRunningReason);
}
var handedOff = false;
SemaphoreSlim? gate = null;
try
{
state.ScriptLogger = invocation.ScriptLogger;
var shard = state.Suspect ? _isolation : _shards[ShardIndex(key, _shards.Length)];
if (shard.IsolationGate is { } isolationGate)
{
if (!await isolationGate.WaitAsync(_worker.UnavailableGrace, ct).ConfigureAwait(false))
return ScriptRunOutcome.Unavailable(ScriptRunOutcome.RunnerUnavailableReason);
gate = isolationGate;
}
var connection = await shard.GetReadyAsync(_worker.UnavailableGrace, ct).ConfigureAwait(false);
if (connection is null) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.RunnerUnavailableReason);
var paths = invocation.Inputs.Keys.Order(StringComparer.Ordinal).ToArray();
WorkerConnection.LoadEntry? entry;
try
{
entry = await connection
.EnsureLoadedAsync(key, WorkerShard.ToWire(invocation.Kind), invocation.Source, sha, paths)
.WaitAsync(LoadTimeout, ct).ConfigureAwait(false);
}
catch (TimeoutException)
{
// A compile that never finishes is the script's fault (a pathological source): the worker dies, the
// key takes a kill strike through the exit handler.
_log.Warning("Script {ScriptKey}: compile in ScriptWorker[{Shard}] exceeded {Timeout} s; killing the worker",
key, shard.Label, LoadTimeout.TotalSeconds);
connection.Kill(WorkerExitReason.KilledLoadTimeout, key);
return ScriptRunOutcome.Failed($"compile failure: compile did not finish within {LoadTimeout.TotalSeconds:F0} s");
}
if (entry?.Outcome is not { } outcome) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason);
if (!outcome.Ok)
{
var reason = outcome.Error?.Kind switch
{
ScriptError.Types.Kind.Compile => $"compile error: {outcome.Error.Message}",
ScriptError.Types.Kind.Sandbox => $"sandbox violation: {outcome.Error.Message}",
_ => $"compile failure: {outcome.Error?.Message}",
};
// Remembered until the next deploy boundary, so a broken script never re-enters Roslyn (design §7).
_compileFailures.TryAdd(failureKey, reason);
return ScriptRunOutcome.Failed(reason);
}
var pending = connection.Submit(key, BuildRequest(key, entry, invocation));
ArmHardDeadline(connection, pending, invocation.Timeout + _worker.EffectiveHardGrace(invocation.Timeout));
// From here the in-flight slot (and the isolation gate) belong to the evaluation itself: they are released
// when the worker answers or dies — not when this caller stops waiting at the soft deadline. That keeps a
// key single-flight between T and T + G, and the isolation worker strictly one-at-a-time.
handedOff = true;
var heldGate = gate;
_ = pending.Completion.ContinueWith(_ =>
{
Interlocked.Exchange(ref state.InFlight, 0);
heldGate?.Release();
}, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
EvalResult? result;
try
{
result = await pending.Completion.WaitAsync(invocation.Timeout, ct).ConfigureAwait(false);
}
catch (TimeoutException)
{
// As in process: the run token is cancelled only once the caller has its timeout, so a polling script
// ends itself (and frees its slot) without the worker being killed.
connection.Cancel(pending);
if (ct.IsCancellationRequested) return ScriptRunOutcome.Failed("evaluation cancelled");
if (NoteTimeout(key, state, invocation.Timeout)) return ScriptRunOutcome.Failed(state.QuarantineReason!);
return ScriptRunOutcome.Failed($"{ScriptEvalFailureReasons.TimedOutPrefix}{invocation.Timeout.TotalSeconds:F1}s");
}
if (result is null) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason);
return Complete(key, state, result);
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
return ScriptRunOutcome.Failed("evaluation cancelled");
}
finally
{
if (!handedOff)
{
Interlocked.Exchange(ref state.InFlight, 0);
gate?.Release();
}
}
}
/// <inheritdoc />
public void Reset(ScriptContextKind kind)
{
var prefix = InProcessScriptRunner.ScopedKey(kind, string.Empty);
var dropped = new List<string>();
foreach (var key in _keys.Keys)
{
if (key.StartsWith(prefix, StringComparison.Ordinal) && _keys.TryRemove(key, out _)) dropped.Add(key);
}
foreach (var key in _compileFailures.Keys)
{
if (key.StartsWith(prefix, StringComparison.Ordinal)) _compileFailures.TryRemove(key, out _);
}
foreach (var shard in _shards.Append(_isolation))
shard.Current?.Unload(dropped);
}
/// <inheritdoc />
public IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind)
{
var prefix = InProcessScriptRunner.ScopedKey(kind, string.Empty);
return _keys
.Where(kv => kv.Value.QuarantineReason is not null && kv.Key.StartsWith(prefix, StringComparison.Ordinal))
.Select(kv => kv.Key[prefix.Length..])
.ToArray();
}
/// <summary>Stops every worker.</summary>
/// <returns>A task that completes when they have ended.</returns>
public async ValueTask DisposeAsync()
{
if (Interlocked.Exchange(ref _disposed, 1) != 0) return;
await Task.WhenAll(_shards.Append(_isolation).Select(s => s.DisposeAsync().AsTask())).ConfigureAwait(false);
}
/// <summary>Stops every worker (blocking).</summary>
public void Dispose() => DisposeAsync().AsTask().GetAwaiter().GetResult();
private WorkerShard NewShard(string label, bool isolation, int evalThreads, TimeSpan heartbeatInterval)
{
var spec = new WorkerLaunchSpec(
Shard: label,
WorkerPath: WorkerPath,
HeapLimitBytes: _worker.HeapLimitBytes,
EvalThreads: evalThreads,
HardGrace: _worker.HardGrace ?? TimeSpan.FromSeconds(Math.Max(1, _options.RunTimeout.TotalSeconds)),
HeartbeatInterval: heartbeatInterval,
HandshakeTimeout: TimeSpan.FromSeconds(15));
return new WorkerShard(label, isolation,
ct => WorkerConnection.StartAsync(spec, _log, OnLogs, ct),
OnWorkerExit, _log);
}
private KeyState Admit(string key, string source)
{
while (true)
{
var state = _keys.GetOrAdd(key, static (_, src) => new KeyState(src), source);
if (string.Equals(state.Source, source, StringComparison.Ordinal)) return state;
// The script changed under this key: a clean slate, exactly as the in-process guard does. An evaluation
// of the old source still in flight releases the OLD state's slot when it ends.
if (_keys.TryUpdate(key, new KeyState(source), state)) continue;
}
}
private EvalRequest BuildRequest(string key, WorkerConnection.LoadEntry entry, ScriptInvocation invocation)
{
var request = new EvalRequest
{
Key = key,
NowUtcTicks = DateTime.UtcNow.Ticks,
SoftTimeoutMs = (uint)Math.Clamp(invocation.Timeout.TotalMilliseconds, 1, uint.MaxValue),
HardGraceMs = (uint)Math.Clamp(_worker.EffectiveHardGrace(invocation.Timeout).TotalMilliseconds, 1, uint.MaxValue),
};
foreach (var (path, snapshot) in invocation.Inputs)
{
if (snapshot is null) continue;
var input = new Input
{
StatusCode = snapshot.StatusCode,
SourceTsTicks = snapshot.SourceTimestampUtc is { } source ? VariantCodec.ToUtc(source).Ticks : 0,
ServerTsTicks = VariantCodec.ToUtc(snapshot.ServerTimestampUtc).Ticks,
};
if (entry.Slots.TryGetValue(path, out var slot)) input.Slot = slot;
else input.Path = path;
if (VariantCodec.TryEncode(snapshot.Value, out var variant, out var unsupported))
{
input.Value = variant;
}
else
{
// Never silently converted (design §14 #11): the worker answers UNSUPPORTED_TYPE for this input.
input.Unsupported = true;
input.UnsupportedType = unsupported ?? "?";
}
request.Inputs.Add(input);
}
return request;
}
private void ArmHardDeadline(WorkerConnection connection, PendingEval pending, TimeSpan after)
{
var cts = new CancellationTokenSource(after);
var registration = cts.Token.Register(() =>
{
if (pending.Completion.IsCompleted) return;
_log.Warning("Script {ScriptKey} exceeded its hard deadline ({Deadline} ms); killing ScriptWorker[{Shard}] pid {Pid}",
pending.Key, (long)after.TotalMilliseconds, connection.Shard, connection.Pid);
connection.Kill(WorkerExitReason.KilledDeadline, pending.Key);
});
_ = pending.Completion.ContinueWith(_ =>
{
registration.Dispose();
cts.Dispose();
}, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
}
private ScriptRunOutcome Complete(string key, KeyState state, EvalResult result)
{
// The script returned control (a value or a throw): the soft-timeout streak resets, as in process.
Interlocked.Exchange(ref state.ConsecutiveTimeouts, 0);
if (_options.MaxAllocBytesPerRun > 0 && (long)result.AllocatedBytes > _options.MaxAllocBytesPerRun)
{
Quarantine(key, state,
$"one evaluation allocated {result.AllocatedBytes / (1024.0 * 1024.0):F1} MB, over the " +
$"{_options.MaxAllocBytesPerRun / (1024.0 * 1024.0):F0} MB budget (Scripting:MaxAllocBytesPerRun); " +
"the script is not evaluated again until its source changes or it is redeployed",
"allocation");
return ScriptRunOutcome.Failed(state.QuarantineReason!);
}
if (state.Suspect && Interlocked.Increment(ref state.ProbationRuns) >= _worker.IsolationProbationRuns)
{
state.Suspect = false;
_log.Information("Script {ScriptKey} completed {Runs} clean run(s) in the isolation worker; returning it to its shard",
key, _worker.IsolationProbationRuns);
}
switch (result.OutcomeCase)
{
case EvalResult.OutcomeOneofCase.Value:
try
{
return ScriptRunOutcome.Ok(VariantCodec.Decode(result.Value));
}
catch (ScriptWorkerProtocolException ex)
{
return ScriptRunOutcome.Failed($"script worker returned an undecodable value: {ex.Message}");
}
case EvalResult.OutcomeOneofCase.Predicate:
return ScriptRunOutcome.Ok(result.Predicate);
case EvalResult.OutcomeOneofCase.Error:
return result.Error.Kind switch
{
ScriptError.Types.Kind.Threw => ScriptRunOutcome.Failed($"script threw: {result.Error.Message}"),
ScriptError.Types.Kind.UnsupportedType => ScriptRunOutcome.Failed($"unsupported type: {result.Error.Message}"),
ScriptError.Types.Kind.Busy => ScriptRunOutcome.Failed(WorkerBusyReason),
ScriptError.Types.Kind.NotLoaded => ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason),
_ => ScriptRunOutcome.Failed($"script failed: {result.Error.Message}"),
};
default:
return ScriptRunOutcome.Ok(null);
}
}
private bool NoteTimeout(string key, KeyState state, TimeSpan timeout)
{
var count = Interlocked.Increment(ref state.ConsecutiveTimeouts);
if (count < _options.QuarantineAfterConsecutiveTimeouts) return false;
Quarantine(key, state,
$"{count} consecutive evaluation(s) timed out (budget {timeout.TotalMilliseconds:F0} ms); " +
"the script is not evaluated again until its source changes or it is redeployed",
"timeouts");
return true;
}
private void OnWorkerExit(WorkerShard shard, WorkerExit exit)
{
switch (exit.Reason)
{
case WorkerExitReason.KilledDeadline when exit.CulpritKey is { } culprit:
KillStrike(culprit, "it exceeded its hard deadline and its worker was killed", "killed");
break;
case WorkerExitReason.KilledLoadTimeout when exit.CulpritKey is { } culprit:
KillStrike(culprit, "its compile did not finish and its worker was killed", "killed");
break;
case WorkerExitReason.Crash or WorkerExitReason.KilledHeartbeat or WorkerExitReason.ProtocolFault:
if (shard.IsIsolation)
{
// One evaluation at a time here: the crash is this key's (design §5.3.3).
foreach (var key in exit.InFlightKeys)
KillStrike(key, "it crashed the isolation worker", "isolated_crash");
}
else if (exit.InFlightKeys.Count > 0)
{
foreach (var key in exit.InFlightKeys)
{
if (!_keys.TryGetValue(key, out var state)) continue;
state.Suspect = true;
Interlocked.Exchange(ref state.ProbationRuns, 0);
}
_log.Warning("ScriptWorker[{Shard}] ended ({Reason}) with no deadline crossed; {Count} script(s) in flight move to the isolation worker: {Suspects}",
shard.Label, exit.Reason, exit.InFlightKeys.Count, string.Join(", ", exit.InFlightKeys));
}
break;
}
}
private void KillStrike(string key, string why, string metricReason)
{
if (!_keys.TryGetValue(key, out var state)) return;
int strikes;
lock (state.KillStrikes)
{
var now = DateTime.UtcNow;
state.KillStrikes.Add(now);
state.KillStrikes.RemoveAll(t => now - t > KillStrikeWindow);
strikes = state.KillStrikes.Count;
}
if (strikes >= _worker.QuarantineAfterKills)
{
Quarantine(key, state,
$"{why} ({strikes} time(s) in the last hour); the script is not evaluated again until its source changes or it is redeployed",
metricReason);
}
}
private void Quarantine(string key, KeyState state, string why, string metricReason)
{
var reason = ScriptRunGuard.QuarantinedReasonPrefix + why;
if (Interlocked.CompareExchange(ref state.QuarantineReason, reason, null) is not null) return;
ScriptRunGuard.QuarantinedCounter.Add(1,
new KeyValuePair<string, object?>("script", key),
new KeyValuePair<string, object?>("reason", metricReason));
_scriptRoot.ForContext(ScriptLoggerFactory.ScriptNameProperty, key).Error("Script {ScriptKey} quarantined: {Reason}", key, why);
}
private void OnLogs(LogBatch batch)
{
foreach (var line in batch.Lines)
{
var logger = _keys.TryGetValue(line.Key, out var state) && state.ScriptLogger is { } scoped ? scoped : _scriptRoot;
var level = line.Level <= (uint)LogEventLevel.Fatal ? (LogEventLevel)line.Level : LogEventLevel.Information;
if (string.IsNullOrEmpty(line.Exception))
logger.Write(level, "{ScriptMessage:l}", line.Rendered);
else
logger.Write(level, "{ScriptMessage:l}{NewLine}{ScriptException:l}", line.Rendered, Environment.NewLine, line.Exception);
}
if (batch.Dropped > 0)
_log.Warning("ScriptWorker dropped {Dropped} script log line(s): a script is logging faster than the pipe drains", batch.Dropped);
}
private sealed class KeyState(string source)
{
public readonly string Source = source;
public readonly List<DateTime> KillStrikes = [];
public int InFlight;
public int ConsecutiveTimeouts;
public int ProbationRuns;
public volatile bool Suspect;
public string? QuarantineReason;
public ILogger? ScriptLogger;
}
}
@@ -0,0 +1,114 @@
using System.ComponentModel;
using System.Diagnostics;
using System.Runtime.InteropServices;
using System.Runtime.Versioning;
using Microsoft.Win32.SafeHandles;
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
/// <summary>
/// A Windows Job Object holding one ScriptWorker (design §3.3, §5.2): <c>KILL_ON_JOB_CLOSE</c> so the kernel kills
/// the worker when the host dies (its handle closes with it), <c>ACTIVE_PROCESS = 1</c> so the worker can start no
/// child process, and a process-memory backstop slightly above the managed heap limit for native and JIT memory.
/// </summary>
[SupportedOSPlatform("windows")]
internal sealed class WindowsJobObject : IDisposable
{
private const int JobObjectExtendedLimitInformation = 9;
private const uint JOB_OBJECT_LIMIT_ACTIVE_PROCESS = 0x00000008;
private const uint JOB_OBJECT_LIMIT_PROCESS_MEMORY = 0x00000100;
private const uint JOB_OBJECT_LIMIT_DIE_ON_UNHANDLED_EXCEPTION = 0x00000400;
private const uint JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE = 0x00002000;
private readonly SafeFileHandle _handle;
private WindowsJobObject(SafeFileHandle handle) => _handle = handle;
/// <summary>Creates a job with the worker limits and assigns <paramref name="process"/> to it.</summary>
/// <param name="process">The freshly started worker.</param>
/// <param name="processMemoryLimitBytes">The per-process commit limit.</param>
/// <returns>The job; dispose it to kill the worker.</returns>
public static WindowsJobObject Assign(Process process, long processMemoryLimitBytes)
{
var handle = CreateJobObjectW(IntPtr.Zero, null);
if (handle.IsInvalid) throw new Win32Exception(Marshal.GetLastWin32Error(), "CreateJobObject failed");
var job = new WindowsJobObject(handle);
try
{
var info = new JOBOBJECT_EXTENDED_LIMIT_INFORMATION
{
BasicLimitInformation = new JOBOBJECT_BASIC_LIMIT_INFORMATION
{
LimitFlags = JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE | JOB_OBJECT_LIMIT_ACTIVE_PROCESS
| JOB_OBJECT_LIMIT_PROCESS_MEMORY | JOB_OBJECT_LIMIT_DIE_ON_UNHANDLED_EXCEPTION,
ActiveProcessLimit = 1,
},
ProcessMemoryLimit = (UIntPtr)(ulong)processMemoryLimitBytes,
};
if (!SetInformationJobObject(handle, JobObjectExtendedLimitInformation, ref info,
(uint)Marshal.SizeOf<JOBOBJECT_EXTENDED_LIMIT_INFORMATION>()))
throw new Win32Exception(Marshal.GetLastWin32Error(), "SetInformationJobObject failed");
if (!AssignProcessToJobObject(handle, process.Handle))
throw new Win32Exception(Marshal.GetLastWin32Error(), "AssignProcessToJobObject failed");
return job;
}
catch
{
job.Dispose();
throw;
}
}
/// <summary>Closes the job handle, which kills every process still in it.</summary>
public void Dispose() => _handle.Dispose();
[StructLayout(LayoutKind.Sequential)]
private struct JOBOBJECT_BASIC_LIMIT_INFORMATION
{
public long PerProcessUserTimeLimit;
public long PerJobUserTimeLimit;
public uint LimitFlags;
public UIntPtr MinimumWorkingSetSize;
public UIntPtr MaximumWorkingSetSize;
public uint ActiveProcessLimit;
public UIntPtr Affinity;
public uint PriorityClass;
public uint SchedulingClass;
}
[StructLayout(LayoutKind.Sequential)]
private struct IO_COUNTERS
{
public ulong ReadOperationCount;
public ulong WriteOperationCount;
public ulong OtherOperationCount;
public ulong ReadTransferCount;
public ulong WriteTransferCount;
public ulong OtherTransferCount;
}
[StructLayout(LayoutKind.Sequential)]
private struct JOBOBJECT_EXTENDED_LIMIT_INFORMATION
{
public JOBOBJECT_BASIC_LIMIT_INFORMATION BasicLimitInformation;
public IO_COUNTERS IoInfo;
public UIntPtr ProcessMemoryLimit;
public UIntPtr JobMemoryLimit;
public UIntPtr PeakProcessMemoryUsed;
public UIntPtr PeakJobMemoryUsed;
}
#pragma warning disable SYSLIB1054 // DllImport kept: LibraryImport would need AllowUnsafeBlocks for three calls.
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
private static extern SafeFileHandle CreateJobObjectW(IntPtr securityAttributes, string? name);
[DllImport("kernel32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool SetInformationJobObject(SafeFileHandle job, int infoClass,
ref JOBOBJECT_EXTENDED_LIMIT_INFORMATION info, uint length);
[DllImport("kernel32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool AssignProcessToJobObject(SafeFileHandle job, IntPtr process);
#pragma warning restore SYSLIB1054
}
@@ -0,0 +1,699 @@
using System.Collections.Concurrent;
using System.Diagnostics;
using System.IO.Pipes;
using System.Security.Cryptography;
using System.Text;
using System.Threading.Channels;
using Serilog;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
/// <summary>How a worker process ended.</summary>
internal enum WorkerExitReason
{
/// <summary>It exited by itself with code 0 (a Shutdown).</summary>
Clean,
/// <summary>The host killed it: a script crossed its hard deadline (<see cref="WorkerExit.CulpritKey"/>).</summary>
KilledDeadline,
/// <summary>The host killed it: no frame for three heartbeat intervals.</summary>
KilledHeartbeat,
/// <summary>The host killed it: a compile did not finish in time (<see cref="WorkerExit.CulpritKey"/>).</summary>
KilledLoadTimeout,
/// <summary>It died by itself (OOM, heap limit, fail-fast, external kill).</summary>
Crash,
/// <summary>A framing or handshake violation.</summary>
ProtocolFault,
/// <summary>The runner is shutting down.</summary>
Disposed,
}
/// <summary>The end of one worker process.</summary>
/// <param name="Reason">Why it ended.</param>
/// <param name="CulpritKey">The scoped key the host blamed, for a deadline or load kill.</param>
/// <param name="ExitCode">The OS exit code, when known.</param>
/// <param name="InFlightKeys">The scoped keys that had an evaluation outstanding when it ended.</param>
internal sealed record WorkerExit(WorkerExitReason Reason, string? CulpritKey, int? ExitCode, IReadOnlyList<string> InFlightKeys);
/// <summary>What the host needs to launch one worker.</summary>
/// <param name="Shard">The shard label (<c>0</c>, <c>1</c>, … or <c>iso</c>).</param>
/// <param name="WorkerPath">The worker entry assembly (<c>.dll</c>, run through the dotnet muxer) or apphost.</param>
/// <param name="HeapLimitBytes">The worker's <c>DOTNET_GCHeapHardLimit</c>.</param>
/// <param name="EvalThreads">The worker scheduler's width (1 for the isolation worker).</param>
/// <param name="HardGrace">The worker watchdog's grace past the soft deadline.</param>
/// <param name="HeartbeatInterval">How often the worker sends a heartbeat.</param>
/// <param name="HandshakeTimeout">How long a launch may take to reach Ready.</param>
internal sealed record WorkerLaunchSpec(
string Shard,
string WorkerPath,
long HeapLimitBytes,
int EvalThreads,
TimeSpan HardGrace,
TimeSpan HeartbeatInterval,
TimeSpan HandshakeTimeout);
/// <summary>
/// One live ScriptWorker process and its pipe (lmxopcua#780, design §3–§6): launch with a cleared, allow-listed
/// environment, a random pipe that accepts exactly one client, a nonce handshake, a frame reader and a coalescing
/// writer, heartbeat supervision, and the kill. It never restarts itself; <see cref="WorkerShard"/> does that.
/// </summary>
internal sealed class WorkerConnection : IAsyncDisposable
{
/// <summary>
/// Environment variables a worker inherits from the host (design §6): the runtime's own knobs, the temp
/// directories and the minimum a Windows process needs. Everything else — notably <c>ServerHistorian__ApiKey</c>,
/// ConfigDb connection strings and every other <c>Section__Key</c> override — is dropped.
/// </summary>
internal static readonly IReadOnlyList<string> InheritedEnvironment =
[
"PATH", "TEMP", "TMP", "TMPDIR", "HOME", "SystemRoot", "SYSTEMROOT", "windir", "SystemDrive", "ComSpec",
"PATHEXT", "NUMBER_OF_PROCESSORS", "PROCESSOR_ARCHITECTURE", "LOCALAPPDATA",
];
private readonly Process _process;
private readonly NamedPipeServerStream _pipe;
private readonly IDisposable? _job;
private readonly string _workDir;
private readonly ILogger _log;
private readonly Action<LogBatch> _onLogs;
private readonly Channel<Envelope> _outbound = Channel.CreateUnbounded<Envelope>(new UnboundedChannelOptions { SingleReader = true });
private readonly ConcurrentDictionary<ulong, PendingEval> _pending = new();
private readonly ConcurrentDictionary<ulong, TaskCompletionSource<LoadResult>> _loads = new();
private readonly ConcurrentDictionary<string, LoadEntry> _loaded = new(StringComparer.Ordinal);
private readonly TaskCompletionSource<WorkerExit> _exited = new(TaskCreationOptions.RunContinuationsAsynchronously);
private readonly CancellationTokenSource _stopping = new();
private readonly TimeSpan _heartbeatInterval;
private TaskCompletionSource<DiagnosticsReply>? _diag;
private (WorkerExitReason Reason, string? Culprit)? _killedFor;
private long _lastFrameTimestamp;
private long _seq;
private long _nextEvalId;
private long _nextLoadId;
private int _maxFrameBytes = ScriptWorkerFrameCodec.DefaultMaxFrameBytes;
private int _disposed;
private WorkerConnection(string shard, Process process, NamedPipeServerStream pipe, IDisposable? job, string workDir,
ILogger log, Action<LogBatch> onLogs, TimeSpan heartbeatInterval)
{
Shard = shard;
_process = process;
_pipe = pipe;
_job = job;
_workDir = workDir;
_log = log;
_onLogs = onLogs;
_heartbeatInterval = heartbeatInterval;
Pid = process.Id;
}
/// <summary>The shard label.</summary>
public string Shard { get; }
/// <summary>The worker's process id.</summary>
public int Pid { get; }
/// <summary>Completes when the worker has ended, with why.</summary>
public Task<WorkerExit> Exited => _exited.Task;
/// <summary>Whether the worker is still serving.</summary>
public bool IsAlive => !_exited.Task.IsCompleted && !_stopping.IsCancellationRequested;
/// <summary><see cref="Stopwatch.GetTimestamp"/> when the handshake completed.</summary>
public long ReadyTimestamp { get; private set; }
/// <summary>Evaluations currently outstanding on this worker.</summary>
public int InFlightCount => _pending.Count;
/// <summary>Launches a worker and completes the handshake.</summary>
/// <param name="spec">What to launch.</param>
/// <param name="log">Host diagnostics (stderr lines, exits).</param>
/// <param name="onLogs">Receives the worker's forwarded <c>ctx.Logger</c> batches.</param>
/// <param name="ct">Cancellation.</param>
/// <returns>A ready connection.</returns>
public static async Task<WorkerConnection> StartAsync(WorkerLaunchSpec spec, ILogger log, Action<LogBatch> onLogs, CancellationToken ct)
{
var pipeName = "otopcua-sw-" + Convert.ToHexString(RandomNumberGenerator.GetBytes(12)).ToLowerInvariant();
var nonce = Convert.ToHexString(RandomNumberGenerator.GetBytes(32));
var workDir = Path.Combine(Path.GetTempPath(), "otopcua-scriptworker", pipeName);
Directory.CreateDirectory(workDir);
// One instance, current user only: the pipe accepts exactly one connection, from this account.
var pipe = new NamedPipeServerStream(pipeName, PipeDirection.InOut, maxNumberOfServerInstances: 1,
PipeTransmissionMode.Byte, PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly);
Process? process = null;
IDisposable? job = null;
try
{
var psi = BuildStartInfo(spec, pipeName, nonce, workDir);
process = new Process { StartInfo = psi, EnableRaisingEvents = true };
if (!process.Start()) throw new InvalidOperationException("the worker process did not start");
if (OperatingSystem.IsWindows())
{
// Native and JIT memory backstop above the managed heap limit.
job = WindowsJobObject.Assign(process, spec.HeapLimitBytes + 256L * 1024 * 1024);
}
var shardLabel = spec.Shard;
process.ErrorDataReceived += (_, e) =>
{
if (e.Data is { Length: > 0 } line) log.Warning("ScriptWorker[{Shard}] {WorkerStderr}", shardLabel, line);
};
process.OutputDataReceived += (_, e) =>
{
if (e.Data is { Length: > 0 } line) log.Debug("ScriptWorker[{Shard}] stdout {WorkerStdout}", shardLabel, line);
};
process.BeginErrorReadLine();
process.BeginOutputReadLine();
using var handshakeCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
handshakeCts.CancelAfter(spec.HandshakeTimeout);
var exitedEarly = process.WaitForExitAsync(handshakeCts.Token);
var accept = pipe.WaitForConnectionAsync(handshakeCts.Token);
if (await Task.WhenAny(accept, exitedEarly).ConfigureAwait(false) != accept)
throw new InvalidOperationException(
$"the worker exited (code {SafeExitCode(process)}) before connecting to its pipe");
await accept.ConfigureAwait(false);
var hello = new Hello
{
Nonce = nonce,
MaxFrameBytes = ScriptWorkerFrameCodec.DefaultMaxFrameBytes,
EvalThreads = (uint)spec.EvalThreads,
QueueCapacity = 4096,
HardGraceMs = (uint)Math.Clamp(spec.HardGrace.TotalMilliseconds, 1, uint.MaxValue),
HeartbeatIntervalMs = (uint)spec.HeartbeatInterval.TotalMilliseconds,
MaxLogLines = 1000,
};
await HandshakeAsync(pipe, hello, nonce, process.Id, handshakeCts.Token).ConfigureAwait(false);
var connection = new WorkerConnection(spec.Shard, process, pipe, job, workDir, log, onLogs, spec.HeartbeatInterval);
connection.Begin();
return connection;
}
catch
{
try { if (process is { HasExited: false }) process.Kill(entireProcessTree: true); } catch { /* best effort */ }
job?.Dispose();
process?.Dispose();
await pipe.DisposeAsync().ConfigureAwait(false);
TryDeleteDirectory(workDir);
throw;
}
}
/// <summary>
/// The host half of the handshake: send <paramref name="hello"/>, require a <c>Ready</c> as the first frame,
/// echoing <paramref name="expectedNonce"/> (constant-time compare) from <paramref name="expectedPid"/>.
/// </summary>
/// <param name="stream">The accepted pipe.</param>
/// <param name="hello">The Hello to send.</param>
/// <param name="expectedNonce">The nonce put in the worker's environment.</param>
/// <param name="expectedPid">The launched process id; null skips the check.</param>
/// <param name="ct">Cancellation (the handshake timeout).</param>
/// <returns>The worker's Ready.</returns>
/// <exception cref="ScriptWorkerProtocolException">Anything but a matching Ready.</exception>
internal static async Task<Ready> HandshakeAsync(Stream stream, Hello hello, string expectedNonce, int? expectedPid, CancellationToken ct)
{
await ScriptWorkerFrameCodec.WriteAsync(stream, new Envelope
{
ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion,
Seq = 1,
Hello = hello,
}, ScriptWorkerFrameCodec.DefaultMaxFrameBytes, ct).ConfigureAwait(false);
await stream.FlushAsync(ct).ConfigureAwait(false);
var reply = await ScriptWorkerFrameCodec.ReadAsync(stream, ScriptWorkerFrameCodec.DefaultMaxFrameBytes, ct).ConfigureAwait(false)
?? throw new ScriptWorkerProtocolException("the worker closed the pipe during the handshake");
if (reply.BodyCase != Envelope.BodyOneofCase.Ready)
throw new ScriptWorkerProtocolException($"expected Ready, got {reply.BodyCase}");
var expected = Encoding.UTF8.GetBytes(expectedNonce);
var actual = Encoding.UTF8.GetBytes(reply.Ready.Nonce ?? string.Empty);
if (!CryptographicOperations.FixedTimeEquals(expected, actual))
throw new ScriptWorkerProtocolException("the worker's nonce does not match the launch nonce");
if (expectedPid is { } pid && reply.Ready.Pid != pid)
throw new ScriptWorkerProtocolException($"the pipe client reports pid {reply.Ready.Pid}, not the launched {pid}");
return reply.Ready;
}
private static ProcessStartInfo BuildStartInfo(WorkerLaunchSpec spec, string pipeName, string nonce, string workDir)
{
var psi = new ProcessStartInfo
{
UseShellExecute = false,
CreateNoWindow = true,
RedirectStandardError = true,
RedirectStandardOutput = true,
RedirectStandardInput = false,
WorkingDirectory = workDir,
};
if (spec.WorkerPath.EndsWith(".dll", StringComparison.OrdinalIgnoreCase))
{
psi.FileName = ResolveMuxer();
psi.ArgumentList.Add("exec");
psi.ArgumentList.Add(spec.WorkerPath);
}
else
{
psi.FileName = spec.WorkerPath;
}
// Cleared, then repopulated from an allow-list (design §6). ProcessStartInfo.Environment starts as a copy of
// the host's environment; ScadaBridge's sandbox inherits it whole, which this design rejects.
var inherited = psi.Environment.ToDictionary(kv => kv.Key, kv => kv.Value, StringComparer.Ordinal);
psi.Environment.Clear();
foreach (var (name, value) in inherited)
{
if (value is null) continue;
if (InheritedEnvironment.Contains(name, StringComparer.OrdinalIgnoreCase)
|| (name.StartsWith("DOTNET_", StringComparison.OrdinalIgnoreCase)
&& !name.StartsWith("DOTNET_GCHeap", StringComparison.OrdinalIgnoreCase)))
psi.Environment[name] = value;
}
psi.Environment["DOTNET_GCHeapHardLimit"] = "0x" + spec.HeapLimitBytes.ToString("X", System.Globalization.CultureInfo.InvariantCulture);
psi.Environment["DOTNET_gcServer"] = "0";
psi.Environment["DOTNET_CLI_TELEMETRY_OPTOUT"] = "1";
psi.Environment[ScriptWorkerEnvironment.PipeName] = pipeName;
psi.Environment[ScriptWorkerEnvironment.Nonce] = nonce;
psi.Environment[ScriptWorkerEnvironment.ParentPid] = Environment.ProcessId.ToString(System.Globalization.CultureInfo.InvariantCulture);
psi.Environment[ScriptWorkerEnvironment.Shard] = spec.Shard;
return psi;
}
/// <summary>The dotnet muxer that runs a framework-dependent worker <c>.dll</c>.</summary>
/// <returns>The muxer path, or plain <c>dotnet</c> (resolved on PATH) as a last resort.</returns>
internal static string ResolveMuxer()
{
var hostPath = Environment.GetEnvironmentVariable("DOTNET_HOST_PATH");
if (!string.IsNullOrEmpty(hostPath) && File.Exists(hostPath)) return hostPath;
var processPath = Environment.ProcessPath;
if (processPath is not null
&& string.Equals(Path.GetFileNameWithoutExtension(processPath), "dotnet", StringComparison.OrdinalIgnoreCase))
return processPath;
var root = Environment.GetEnvironmentVariable("DOTNET_ROOT");
if (!string.IsNullOrEmpty(root))
{
var candidate = Path.Combine(root, OperatingSystem.IsWindows() ? "dotnet.exe" : "dotnet");
if (File.Exists(candidate)) return candidate;
}
return "dotnet";
}
private void Begin()
{
ReadyTimestamp = Stopwatch.GetTimestamp();
Volatile.Write(ref _lastFrameTimestamp, ReadyTimestamp);
_ = Task.Run(ReadLoopAsync);
_ = Task.Run(WriteLoopAsync);
_ = Task.Run(HeartbeatMonitorAsync);
_ = Task.Run(WatchProcessAsync);
}
/// <summary>Ensures <paramref name="key"/> is compiled in this worker at <paramref name="sourceHash"/>.</summary>
/// <param name="key">The scoped key.</param>
/// <param name="kind">The context kind.</param>
/// <param name="source">The source.</param>
/// <param name="sourceHash">SHA-256 of the source.</param>
/// <param name="inputPaths">The slot order.</param>
/// <returns>The load outcome and the slot map; null when the worker ended first.</returns>
public Task<LoadEntry?> EnsureLoadedAsync(string key, ContextKind kind, string source, byte[] sourceHash, IReadOnlyList<string> inputPaths)
{
var hashHex = Convert.ToHexString(sourceHash);
if (_loaded.TryGetValue(key, out var existing) && existing.SourceHash == hashHex && existing.SameSlots(inputPaths))
return existing.Ready;
var entry = new LoadEntry(hashHex, inputPaths);
_loaded[key] = entry;
var loadId = (ulong)Interlocked.Increment(ref _nextLoadId);
var tcs = new TaskCompletionSource<LoadResult>(TaskCreationOptions.RunContinuationsAsynchronously);
_loads[loadId] = tcs;
var load = new LoadScripts { LoadId = loadId };
var script = new ScriptSource { Key = key, Kind = kind, Source = source, SourceSha256 = Google.Protobuf.ByteString.CopyFrom(sourceHash) };
script.InputPaths.AddRange(inputPaths);
load.Scripts.Add(script);
if (!Send(new Envelope { Load = load })) { _loads.TryRemove(loadId, out _); entry.Complete(null); return entry.Ready; }
_ = tcs.Task.ContinueWith(t =>
{
if (t.IsCompletedSuccessfully)
{
var outcome = t.Result.Outcomes.FirstOrDefault(o => o.Key == key);
entry.Complete(outcome ?? new LoadOutcome { Key = key, Ok = false, Error = new ScriptError { Kind = ScriptError.Types.Kind.Threw, Message = "no load outcome" } });
if (outcome is { Ok: false }) _loaded.TryRemove(new KeyValuePair<string, LoadEntry>(key, entry));
}
else
{
entry.Complete(null);
_loaded.TryRemove(new KeyValuePair<string, LoadEntry>(key, entry));
}
}, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
return entry.Ready;
}
/// <summary>Forgets a key's compiled form here and in the worker.</summary>
/// <param name="keys">The scoped keys.</param>
public void Unload(IReadOnlyCollection<string> keys)
{
var unload = new Unload();
foreach (var key in keys)
if (_loaded.TryRemove(key, out _)) unload.Keys.Add(key);
if (unload.Keys.Count > 0) Send(new Envelope { Unload = unload });
}
/// <summary>Submits one evaluation. Its <see cref="PendingEval.Completion"/> yields null when the worker ends first.</summary>
/// <param name="key">The scoped key.</param>
/// <param name="request">The request (its id is assigned here).</param>
/// <returns>The pending evaluation.</returns>
public PendingEval Submit(string key, EvalRequest request)
{
request.EvalId = (ulong)Interlocked.Increment(ref _nextEvalId);
var pending = new PendingEval(key, request.EvalId);
_pending[request.EvalId] = pending;
var batch = new EvalBatch();
batch.Items.Add(request);
if (!Send(new Envelope { Eval = batch }) || _exited.Task.IsCompleted)
{
if (_pending.TryRemove(request.EvalId, out _)) pending.Complete(null);
}
return pending;
}
/// <summary>Cancels a run's <c>ctx.CancellationToken</c> in the worker (sent at the soft deadline).</summary>
/// <param name="pending">The evaluation.</param>
public void Cancel(PendingEval pending)
{
if (!pending.Completion.IsCompleted) Send(new Envelope { Cancel = new Protocol.V1.Cancel { EvalId = pending.EvalId } });
}
/// <summary>Asks the worker for its process diagnostics (loaded assemblies, environment names).</summary>
/// <param name="timeout">How long to wait.</param>
/// <returns>The reply.</returns>
public async Task<DiagnosticsReply> DiagnosticsAsync(TimeSpan timeout)
{
var tcs = new TaskCompletionSource<DiagnosticsReply>(TaskCreationOptions.RunContinuationsAsynchronously);
Interlocked.Exchange(ref _diag, tcs);
Send(new Envelope { DiagRequest = new DiagnosticsRequest() });
return await tcs.Task.WaitAsync(timeout).ConfigureAwait(false);
}
/// <summary>Kills the worker (process tree). The first reason given is the one recorded.</summary>
/// <param name="reason">Why.</param>
/// <param name="culpritKey">The scoped key blamed, if any.</param>
public void Kill(WorkerExitReason reason, string? culpritKey)
{
lock (_pending)
{
_killedFor ??= (reason, culpritKey);
}
try
{
if (!_process.HasExited) _process.Kill(entireProcessTree: true);
}
catch (InvalidOperationException) { /* already gone */ }
catch (System.ComponentModel.Win32Exception ex)
{
_log.Warning(ex, "ScriptWorker[{Shard}] kill of pid {Pid} failed", Shard, Pid);
}
}
/// <summary>Asks the worker to stop, waits briefly, then kills it.</summary>
/// <returns>A task that completes when the worker has ended.</returns>
public async ValueTask DisposeAsync()
{
// Both the shard (on exit) and the runner (on shutdown) dispose a connection; only the first does the work.
if (Interlocked.Exchange(ref _disposed, 1) != 0) return;
if (!_exited.Task.IsCompleted)
{
lock (_pending) { _killedFor ??= (WorkerExitReason.Disposed, null); }
Send(new Envelope { Shutdown = new Shutdown() });
try
{
await _process.WaitForExitAsync().WaitAsync(TimeSpan.FromSeconds(1)).ConfigureAwait(false);
}
catch (TimeoutException)
{
Kill(WorkerExitReason.Disposed, null);
}
try { await _exited.Task.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false); } catch (TimeoutException) { }
}
_stopping.Cancel();
_outbound.Writer.TryComplete();
await _pipe.DisposeAsync().ConfigureAwait(false);
_job?.Dispose();
_process.Dispose();
TryDeleteDirectory(_workDir);
}
/// <summary>Test hook: closes the host end of the pipe without touching the process, as a dying host would.</summary>
internal void SeverPipeForTest() => _pipe.Dispose();
private bool Send(Envelope envelope)
{
envelope.ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion;
return _outbound.Writer.TryWrite(envelope);
}
private async Task WriteLoopAsync()
{
var reader = _outbound.Reader;
try
{
while (await reader.WaitToReadAsync(_stopping.Token).ConfigureAwait(false))
{
// Opportunistic batching (design §4.5): every evaluation queued right now goes out as ONE EvalBatch.
EvalBatch? batch = null;
while (reader.TryRead(out var envelope))
{
if (envelope.BodyCase == Envelope.BodyOneofCase.Eval)
{
(batch ??= new EvalBatch()).Items.AddRange(envelope.Eval.Items);
continue;
}
envelope.Seq = (ulong)Interlocked.Increment(ref _seq);
await ScriptWorkerFrameCodec.WriteAsync(_pipe, envelope, _maxFrameBytes, _stopping.Token).ConfigureAwait(false);
}
if (batch is not null)
{
await ScriptWorkerFrameCodec.WriteAsync(_pipe, new Envelope
{
ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion,
Seq = (ulong)Interlocked.Increment(ref _seq),
Eval = batch,
}, _maxFrameBytes, _stopping.Token).ConfigureAwait(false);
}
await _pipe.FlushAsync(_stopping.Token).ConfigureAwait(false);
}
}
catch (OperationCanceledException) { }
catch (Exception ex) when (ex is IOException or ObjectDisposedException or ScriptWorkerProtocolException)
{
if (!_stopping.IsCancellationRequested)
_log.Debug(ex, "ScriptWorker[{Shard}] pipe write ended", Shard);
Kill(WorkerExitReason.Crash, null);
}
}
private async Task ReadLoopAsync()
{
WorkerExitReason? fault = null;
try
{
while (true)
{
var frame = await ScriptWorkerFrameCodec.ReadAsync(_pipe, _maxFrameBytes, _stopping.Token).ConfigureAwait(false);
if (frame is null) break;
Volatile.Write(ref _lastFrameTimestamp, Stopwatch.GetTimestamp());
switch (frame.BodyCase)
{
case Envelope.BodyOneofCase.Results:
foreach (var result in frame.Results.Items)
{
// An eval id with no outstanding request is a protocol violation (design §6 "what the host
// trusts"), except for one that was already completed by a kill race — ignored then.
if (_pending.TryRemove(result.EvalId, out var pending)) pending.Complete(result);
else if (result.EvalId == 0 || result.EvalId > (ulong)Interlocked.Read(ref _nextEvalId))
throw new ScriptWorkerProtocolException($"result for unknown eval id {result.EvalId}");
}
break;
case Envelope.BodyOneofCase.LoadResult:
if (_loads.TryRemove(frame.LoadResult.LoadId, out var load)) load.TrySetResult(frame.LoadResult);
else throw new ScriptWorkerProtocolException($"load result for unknown load id {frame.LoadResult.LoadId}");
break;
case Envelope.BodyOneofCase.Logs:
try { _onLogs(frame.Logs); }
catch (Exception ex) { _log.Warning(ex, "ScriptWorker[{Shard}] log forwarding threw", Shard); }
break;
case Envelope.BodyOneofCase.Heartbeat:
break;
case Envelope.BodyOneofCase.DiagReply:
Volatile.Read(ref _diag)?.TrySetResult(frame.DiagReply);
break;
case Envelope.BodyOneofCase.Fault:
_log.Warning("ScriptWorker[{Shard}] reported a fault: {Fault}", Shard, frame.Fault.Message);
break;
default:
throw new ScriptWorkerProtocolException($"unexpected frame {frame.BodyCase} from the worker");
}
}
}
catch (OperationCanceledException) { }
catch (ScriptWorkerProtocolException ex)
{
_log.Warning("ScriptWorker[{Shard}] protocol violation: {Message}; killing pid {Pid}", Shard, ex.Message, Pid);
fault = WorkerExitReason.ProtocolFault;
}
catch (Exception ex) when (ex is IOException or ObjectDisposedException)
{
}
// The pipe is gone: the worker is dead or dying. Make sure, then report.
try
{
if (fault is { } f) Kill(f, null);
try
{
await _process.WaitForExitAsync().WaitAsync(TimeSpan.FromSeconds(2)).ConfigureAwait(false);
}
catch (TimeoutException)
{
Kill(WorkerExitReason.Crash, null);
try { await _process.WaitForExitAsync().WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false); } catch (TimeoutException) { }
}
}
catch (InvalidOperationException)
{
// The process object was disposed by a concurrent shutdown: the worker is already gone.
}
Finish();
}
private async Task WatchProcessAsync()
{
try { await _process.WaitForExitAsync(_stopping.Token).ConfigureAwait(false); }
catch (OperationCanceledException) { return; }
catch (InvalidOperationException) { return; }
// The reader normally finishes first on EOF; this covers a worker that died while the pipe stayed open.
await Task.Delay(TimeSpan.FromSeconds(2)).ConfigureAwait(false);
Finish();
}
private async Task HeartbeatMonitorAsync()
{
using var timer = new PeriodicTimer(_heartbeatInterval);
try
{
while (await timer.WaitForNextTickAsync(_stopping.Token).ConfigureAwait(false))
{
if (_exited.Task.IsCompleted) return;
var silent = Stopwatch.GetElapsedTime(Volatile.Read(ref _lastFrameTimestamp));
if (silent > _heartbeatInterval * 3)
{
_log.Warning("ScriptWorker[{Shard}] pid {Pid} sent nothing for {Silent} ms; killing it", Shard, Pid, (long)silent.TotalMilliseconds);
Kill(WorkerExitReason.KilledHeartbeat, null);
return;
}
}
}
catch (OperationCanceledException) { }
}
private void Finish()
{
int? code = null;
try { if (_process.HasExited) code = _process.ExitCode; } catch (InvalidOperationException) { }
(WorkerExitReason Reason, string? Culprit)? killed;
lock (_pending) { killed = _killedFor; }
var reason = killed?.Reason ?? (code == 0 ? WorkerExitReason.Clean : WorkerExitReason.Crash);
var inFlight = _pending.Values.Select(p => p.Key).Distinct(StringComparer.Ordinal).ToArray();
if (!_exited.TrySetResult(new WorkerExit(reason, killed?.Culprit, code, inFlight))) return;
// Every outstanding evaluation and load ends now; null tells the waiter "the worker went away".
foreach (var id in _pending.Keys.ToArray())
if (_pending.TryRemove(id, out var pending)) pending.Complete(null);
foreach (var id in _loads.Keys.ToArray())
if (_loads.TryRemove(id, out var load)) load.TrySetCanceled();
Volatile.Read(ref _diag)?.TrySetCanceled();
_outbound.Writer.TryComplete();
}
private static string SafeExitCode(Process process)
{
try { return process.HasExited ? process.ExitCode.ToString(System.Globalization.CultureInfo.InvariantCulture) : "?"; }
catch (InvalidOperationException) { return "?"; }
}
private static void TryDeleteDirectory(string dir)
{
try { if (Directory.Exists(dir)) Directory.Delete(dir, recursive: true); } catch { /* temp dir; best effort */ }
}
/// <summary>A key's compiled form in this worker.</summary>
internal sealed class LoadEntry
{
private readonly TaskCompletionSource<LoadEntry?> _ready = new(TaskCreationOptions.RunContinuationsAsynchronously);
public LoadEntry(string sourceHash, IReadOnlyList<string> inputPaths)
{
SourceHash = sourceHash;
InputPaths = inputPaths;
var slots = new Dictionary<string, uint>(inputPaths.Count, StringComparer.Ordinal);
for (var i = 0; i < inputPaths.Count; i++) slots[inputPaths[i]] = (uint)i;
Slots = slots;
}
public string SourceHash { get; }
public IReadOnlyList<string> InputPaths { get; }
/// <summary>Path → slot in the registered order.</summary>
public IReadOnlyDictionary<string, uint> Slots { get; }
/// <summary>The worker's verdict (null until it arrives).</summary>
public LoadOutcome? Outcome { get; private set; }
public Task<LoadEntry?> Ready => _ready.Task;
public bool SameSlots(IReadOnlyList<string> inputPaths)
{
if (inputPaths.Count != InputPaths.Count) return false;
for (var i = 0; i < inputPaths.Count; i++)
if (!string.Equals(inputPaths[i], InputPaths[i], StringComparison.Ordinal)) return false;
return true;
}
public void Complete(LoadOutcome? outcome)
{
Outcome = outcome;
_ready.TrySetResult(outcome is null ? null : this);
}
}
}
/// <summary>One evaluation outstanding on a worker.</summary>
internal sealed class PendingEval
{
private readonly TaskCompletionSource<EvalResult?> _completion = new(TaskCreationOptions.RunContinuationsAsynchronously);
public PendingEval(string key, ulong evalId)
{
Key = key;
EvalId = evalId;
}
/// <summary>The scoped key.</summary>
public string Key { get; }
/// <summary>The eval id on the wire.</summary>
public ulong EvalId { get; }
/// <summary>The worker's result, or null when the worker ended first.</summary>
public Task<EvalResult?> Completion => _completion.Task;
public void Complete(EvalResult? result) => _completion.TrySetResult(result);
}
@@ -0,0 +1,249 @@
using System.Diagnostics;
using System.Diagnostics.Metrics;
using Serilog;
using ZB.MOM.WW.OtOpcUa.Commons.Observability;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
/// <summary>
/// Supervises one shard's worker process (design §3.3, §5.3.5): starts it lazily on first use, reports its end to
/// the runner (for attribution), and restarts it after an exponential backoff — 250 ms doubling to a 30 s cap with
/// ±20 % jitter, reset once a worker has stayed up for 60 s. It never runs a script in the host when the worker
/// cannot start (fail closed, §14 #4): <see cref="GetReadyAsync"/> returns null and the caller reports
/// <c>BadResourceUnavailable</c>.
/// </summary>
internal sealed class WorkerShard : IAsyncDisposable
{
internal static readonly TimeSpan InitialBackoff = TimeSpan.FromMilliseconds(250);
internal static readonly TimeSpan MaxBackoff = TimeSpan.FromSeconds(30);
internal static readonly TimeSpan HealthyResetAfter = TimeSpan.FromSeconds(60);
private static readonly Counter<long> StartsCounter = OtOpcUaTelemetry.Meter.CreateCounter<long>(
"otopcua.script.worker.starts", unit: "{process}", description: "ScriptWorker processes started (tag: shard).");
private static readonly Counter<long> ExitsCounter = OtOpcUaTelemetry.Meter.CreateCounter<long>(
"otopcua.script.worker.exits", unit: "{process}",
description: "ScriptWorker processes ended (tags: shard, reason = clean | killed_deadline | killed_heartbeat | killed_load | crash | protocol | disposed).");
private readonly Func<CancellationToken, Task<WorkerConnection>> _launch;
private readonly Action<WorkerShard, WorkerExit> _onExit;
private readonly ILogger _log;
private readonly object _gate = new();
private readonly Random _jitter = new();
private WorkerConnection? _current;
private Task<WorkerConnection?>? _starting;
private long _backoffUntil;
private int _consecutiveFailures;
private bool _disposed;
public WorkerShard(string label, bool isolation, Func<CancellationToken, Task<WorkerConnection>> launch,
Action<WorkerShard, WorkerExit> onExit, ILogger log)
{
Label = label;
IsIsolation = isolation;
_launch = launch;
_onExit = onExit;
_log = log;
// The isolation worker runs one evaluation at a time, so a crash there names exactly one key (design §5.3.3).
IsolationGate = isolation ? new SemaphoreSlim(1, 1) : null;
}
/// <summary>The shard label (<c>0</c>…<c>K-1</c>, or <c>iso</c>).</summary>
public string Label { get; }
/// <summary>Whether this is the isolation ("penalty box") worker.</summary>
public bool IsIsolation { get; }
/// <summary>Serialises evaluations on the isolation worker; null on a normal shard.</summary>
public SemaphoreSlim? IsolationGate { get; }
/// <summary>The live worker, if any (diagnostics / tests).</summary>
public WorkerConnection? Current
{
get { lock (_gate) return _current is { IsAlive: true } c ? c : null; }
}
/// <summary>Consecutive start failures and crashes since the last healthy period (diagnostics / tests).</summary>
public int ConsecutiveFailures
{
get { lock (_gate) return _consecutiveFailures; }
}
/// <summary>
/// The ready worker, starting one if needed. Waits at most <paramref name="grace"/> — through a start in progress
/// or a backoff — and returns null if no worker is ready by then.
/// </summary>
/// <param name="grace">How long to wait.</param>
/// <param name="ct">Cancellation.</param>
/// <returns>The worker, or null.</returns>
public async Task<WorkerConnection?> GetReadyAsync(TimeSpan grace, CancellationToken ct)
{
var deadline = Stopwatch.GetTimestamp() + (long)(grace.TotalSeconds * Stopwatch.Frequency);
while (true)
{
Task<WorkerConnection?> starting;
TimeSpan wait;
lock (_gate)
{
if (_disposed) return null;
if (_current is { IsAlive: true } live) return live;
var now = Stopwatch.GetTimestamp();
if (_starting is null && now >= _backoffUntil)
{
_starting = StartCoreAsync();
}
if (_starting is not null)
{
starting = _starting;
wait = TimeSpan.Zero;
}
else
{
starting = null!;
wait = Stopwatch.GetElapsedTime(now, _backoffUntil);
}
}
var remaining = Stopwatch.GetElapsedTime(Stopwatch.GetTimestamp(), deadline);
if (remaining <= TimeSpan.Zero) return null;
if (starting is not null)
{
try
{
var started = await starting.WaitAsync(remaining, ct).ConfigureAwait(false);
if (started is { IsAlive: true }) return started;
}
catch (TimeoutException)
{
return null;
}
continue;
}
if (wait > remaining) return null;
await Task.Delay(wait, ct).ConfigureAwait(false);
}
}
private async Task<WorkerConnection?> StartCoreAsync()
{
await Task.Yield();
WorkerConnection? connection = null;
try
{
connection = await _launch(CancellationToken.None).ConfigureAwait(false);
StartsCounter.Add(1, new KeyValuePair<string, object?>("shard", Label));
_log.Information("ScriptWorker[{Shard}] started, pid {Pid}", Label, connection.Pid);
var started = connection;
_ = started.Exited.ContinueWith(t => OnExited(started, t.Result), CancellationToken.None,
TaskContinuationOptions.ExecuteSynchronously | TaskContinuationOptions.OnlyOnRanToCompletion, TaskScheduler.Default);
lock (_gate)
{
_starting = null;
if (_disposed)
{
_ = started.DisposeAsync().AsTask();
return null;
}
_current = started;
}
return started;
}
catch (Exception ex)
{
_log.Error(ex, "ScriptWorker[{Shard}] failed to start; scripts on it report runner unavailable until it does", Label);
lock (_gate)
{
_starting = null;
ScheduleBackoffLocked();
}
if (connection is not null) await connection.DisposeAsync().ConfigureAwait(false);
return null;
}
}
private void OnExited(WorkerConnection connection, WorkerExit exit)
{
ExitsCounter.Add(1, new KeyValuePair<string, object?>("shard", Label),
new KeyValuePair<string, object?>("reason", MetricReason(exit.Reason)));
lock (_gate)
{
if (ReferenceEquals(_current, connection)) _current = null;
if (!_disposed)
{
if (Stopwatch.GetElapsedTime(connection.ReadyTimestamp) >= HealthyResetAfter) _consecutiveFailures = 0;
ScheduleBackoffLocked();
}
}
if (exit.Reason != WorkerExitReason.Disposed)
{
_log.Warning("ScriptWorker[{Shard}] pid {Pid} ended: {Reason} (exit code {ExitCode}, culprit {Culprit}, {InFlight} evaluation(s) in flight)",
Label, connection.Pid, exit.Reason, exit.ExitCode, exit.CulpritKey ?? "none", exit.InFlightKeys.Count);
}
try { _onExit(this, exit); }
catch (Exception ex) { _log.Error(ex, "ScriptWorker[{Shard}] exit handler threw", Label); }
_ = connection.DisposeAsync().AsTask();
}
private void ScheduleBackoffLocked()
{
_consecutiveFailures++;
var delay = NextBackoff(_consecutiveFailures, _jitter.NextDouble());
_backoffUntil = Stopwatch.GetTimestamp() + (long)(delay.TotalSeconds * Stopwatch.Frequency);
}
/// <summary>The restart delay after <paramref name="failures"/> consecutive failures (design §5.3.5).</summary>
/// <param name="failures">1 for the first failure.</param>
/// <param name="unitJitter">A sample in [0, 1) mapped to ±20 %.</param>
/// <returns>The delay.</returns>
internal static TimeSpan NextBackoff(int failures, double unitJitter)
{
var exponent = Math.Clamp(failures - 1, 0, 30);
var baseMs = Math.Min(MaxBackoff.TotalMilliseconds, InitialBackoff.TotalMilliseconds * Math.Pow(2, exponent));
var factor = 0.8 + (0.4 * Math.Clamp(unitJitter, 0, 1));
return TimeSpan.FromMilliseconds(Math.Min(MaxBackoff.TotalMilliseconds * 1.2, baseMs * factor));
}
private static string MetricReason(WorkerExitReason reason) => reason switch
{
WorkerExitReason.Clean => "clean",
WorkerExitReason.KilledDeadline => "killed_deadline",
WorkerExitReason.KilledHeartbeat => "killed_heartbeat",
WorkerExitReason.KilledLoadTimeout => "killed_load",
WorkerExitReason.ProtocolFault => "protocol",
WorkerExitReason.Disposed => "disposed",
_ => "crash",
};
/// <summary>Stops the shard's worker.</summary>
/// <returns>A task that completes when the worker has ended.</returns>
public async ValueTask DisposeAsync()
{
WorkerConnection? current;
Task<WorkerConnection?>? starting;
lock (_gate)
{
_disposed = true;
current = _current;
starting = _starting;
_current = null;
}
if (starting is not null)
{
try { current ??= await starting.WaitAsync(TimeSpan.FromSeconds(15)).ConfigureAwait(false); } catch { /* shutting down */ }
}
if (current is not null) await current.DisposeAsync().ConfigureAwait(false);
IsolationGate?.Dispose();
}
/// <summary>Maps a context kind to the wire enum.</summary>
/// <param name="kind">The kind.</param>
/// <returns>The wire kind.</returns>
internal static ContextKind ToWire(ScriptContextKind kind) => kind switch
{
ScriptContextKind.VirtualTag => ContextKind.VirtualTag,
ScriptContextKind.Calculation => ContextKind.Calculation,
ScriptContextKind.AlarmPredicate => ContextKind.AlarmPredicate,
_ => ContextKind.Unspecified,
};
}
@@ -0,0 +1,20 @@
<Project Sdk="Microsoft.NET.Sdk">
<!-- lmxopcua#780 — the host side of the out-of-process, pre-emptible script runner (design §3, §5, §9): worker
launch + pipe handshake, the per-shard supervisor, attribution and quarantine. Selected by
Scripting:Runner=OutOfProcess; nothing here runs under the InProcess default. -->
<PropertyGroup>
<RootNamespace>ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess</RootNamespace>
<GenerateDocumentationFile>true</GenerateDocumentationFile>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.Core.Scripting\ZB.MOM.WW.OtOpcUa.Core.Scripting.csproj"/>
<ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol\ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.csproj"/>
</ItemGroup>
<ItemGroup>
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests"/>
</ItemGroup>
</Project>
@@ -0,0 +1,214 @@
// lmxopcua#780 — host <-> ScriptWorker IPC (design docs/plans/2026-11-09-out-of-process-script-runner-design.md §4).
// Framing: 4-byte little-endian length prefix + one serialized Envelope. Zero-length and oversize frames are protocol
// faults. One request frame and one reply frame per evaluation; the only other worker->host traffic is logs and
// heartbeats. Bump protocol_version on any incompatible change.
syntax = "proto3";
package otopcua.scriptworker.v1;
option csharp_namespace = "ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1";
message Envelope {
uint32 protocol_version = 1;
uint64 seq = 2;
oneof body {
Hello hello = 10; // host -> worker, first frame
Ready ready = 11; // worker -> host, reply to Hello
LoadScripts load = 12; // host -> worker
LoadResult load_result = 13; // worker -> host
Unload unload = 14; // host -> worker
EvalBatch eval = 15; // host -> worker
EvalResults results = 16; // worker -> host, streamed as runs complete
LogBatch logs = 17; // worker -> host, fire-and-forget, bounded
Heartbeat heartbeat = 18; // worker -> host, every heartbeat_interval_ms
Shutdown shutdown = 19; // host -> worker
Fault fault = 20; // either way; the sender closes after it
DiagnosticsRequest diag_request = 21; // host -> worker (tests / ops)
DiagnosticsReply diag_reply = 22; // worker -> host
Cancel cancel = 23; // host -> worker, at the soft deadline
}
}
message Hello {
string nonce = 1; // must be echoed in Ready
uint32 max_frame_bytes = 2;
uint32 eval_threads = 3; // the worker's scheduler width; 1 for the isolation worker
uint32 queue_capacity = 4; // queued evaluations above this answer BUSY
uint32 hard_grace_ms = 5; // worker watchdog: fail fast past soft + grace (+ margin)
uint32 heartbeat_interval_ms = 6;
uint32 max_log_lines = 7; // outbound log buffer cap
}
message Ready {
string nonce = 1;
int32 pid = 2;
string runtime_version = 3;
}
enum ContextKind {
CONTEXT_KIND_UNSPECIFIED = 0;
VIRTUAL_TAG = 1;
CALCULATION = 2;
ALARM_PREDICATE = 3;
}
message ScriptSource {
string key = 1;
bytes source_sha256 = 2;
ContextKind kind = 3;
string source = 4;
repeated string input_paths = 5; // slot order for EvalRequest.inputs
}
message LoadScripts {
uint64 load_id = 1;
repeated ScriptSource scripts = 2;
}
message LoadOutcome {
string key = 1;
bool ok = 2;
ScriptError error = 3; // COMPILE or SANDBOX when !ok
}
message LoadResult {
uint64 load_id = 1;
repeated LoadOutcome outcomes = 2;
}
message Unload {
repeated string keys = 1;
}
message EvalBatch {
repeated EvalRequest items = 1;
}
message EvalRequest {
uint64 eval_id = 1;
string key = 2;
repeated Input inputs = 3;
int64 now_utc_ticks = 4; // ctx.Now (host clock)
uint32 soft_timeout_ms = 5;
uint32 hard_grace_ms = 6; // this run's G; 0 = Hello.hard_grace_ms
}
message Input {
uint32 slot = 1; // index into ScriptSource.input_paths; ignored when path is set
Variant value = 2;
uint32 status_code = 3;
int64 source_ts_ticks = 4; // 0 = null
int64 server_ts_ticks = 5;
string path = 6; // inline path for an input with no registered slot
bool unsupported = 7; // the host could not encode the value's CLR type (see unsupported_type)
string unsupported_type = 8;
}
message EvalResult {
uint64 eval_id = 1;
oneof outcome {
Variant value = 2;
bool predicate = 3;
ScriptError error = 4;
}
uint64 allocated_bytes = 5; // GC.GetAllocatedBytesForCurrentThread around the run
uint32 run_micros = 6;
}
message EvalResults {
repeated EvalResult items = 1;
}
message ScriptError {
enum Kind {
THREW = 0;
COMPILE = 1;
SANDBOX = 2;
SOFT_TIMEOUT = 3;
UNSUPPORTED_TYPE = 4;
BUSY = 5;
NOT_LOADED = 6;
}
Kind kind = 1;
string message = 2;
}
message Variant {
enum TypeTag {
NULL = 0;
BOOLEAN = 1;
SBYTE = 2;
BYTE = 3;
INT16 = 4;
UINT16 = 5;
INT32 = 6;
UINT32 = 7;
INT64 = 8;
UINT64 = 9;
FLOAT = 10;
DOUBLE = 11;
STRING = 12;
DATETIME_UTC = 13;
ARRAY = 14;
}
TypeTag type = 1; // the exact CLR type: (short)ctx.GetTag(x).Value must keep working
oneof v {
bool b = 2;
sint64 i = 3;
uint64 u = 4;
double d = 5;
float f = 6;
string s = 7;
int64 dt_ticks = 8;
ArrayValue array = 9;
}
}
message ArrayValue {
Variant.TypeTag element_type = 1;
repeated Variant items = 2;
}
message LogBatch {
repeated LogLine lines = 1;
uint32 dropped = 2;
}
message LogLine {
string key = 1;
uint32 level = 2; // Serilog LogEventLevel
string rendered = 3;
string exception = 4;
int64 ts_ticks = 5;
}
message Heartbeat {
uint32 inflight = 1;
uint32 queued = 2;
int64 heap_bytes = 3;
uint32 gen2_count = 4;
}
message Shutdown {
}
// Sent by the host when it has answered the caller with a soft timeout: cancels the run's ctx.CancellationToken, so a
// polling script ends itself. The worker never cancels on its own clock, which keeps the in-process ordering — the
// caller always has its timeout before the script can observe cancellation.
message Cancel {
uint64 eval_id = 1;
}
message Fault {
string message = 1;
}
message DiagnosticsRequest {
}
message DiagnosticsReply {
int32 pid = 1;
repeated string loaded_assemblies = 2;
repeated string environment_names = 3;
string working_directory = 4;
}
@@ -0,0 +1,54 @@
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
/// <summary>
/// The environment variables the host sets on a ScriptWorker (design §6). The nonce travels in the environment,
/// never on the command line (mxaccessgw <c>WorkerBootstrap.md</c>): a command line is world-readable on most
/// systems.
/// </summary>
public static class ScriptWorkerEnvironment
{
/// <summary>The pipe name the worker connects to.</summary>
public const string PipeName = "OTOPCUA_SCRIPTWORKER_PIPE";
/// <summary>The one-time launch nonce the worker must echo in <c>Ready</c>.</summary>
public const string Nonce = "OTOPCUA_SCRIPTWORKER_NONCE";
/// <summary>The host's process id; the worker exits when it disappears.</summary>
public const string ParentPid = "OTOPCUA_SCRIPTWORKER_PARENT_PID";
/// <summary>The worker's shard label, for its own diagnostics.</summary>
public const string Shard = "OTOPCUA_SCRIPTWORKER_SHARD";
/// <summary>
/// Exit code of a worker whose pipe reached end of stream (the host went away). Distinct from a crash so the
/// host's exit reason metric can tell them apart.
/// </summary>
public const int ExitPipeClosed = 3;
/// <summary>Exit code of a worker whose handshake or framing failed.</summary>
public const int ExitProtocolFault = 4;
/// <summary>Exit code of a worker whose parent process disappeared.</summary>
public const int ExitParentGone = 5;
/// <summary>Exit code of a worker that found a forbidden assembly loaded in itself.</summary>
public const int ExitForbiddenAssembly = 6;
/// <summary>
/// Assembly-name prefixes that must never be loaded in a worker (design §6): the host's cluster, database,
/// OPC UA and secrets stacks. The worker checks itself against this list and exits if it finds one.
/// </summary>
public static readonly IReadOnlyList<string> ForbiddenAssemblyPrefixes =
[
"Akka",
"Microsoft.Data.SqlClient",
"System.Data.SqlClient",
"Microsoft.EntityFrameworkCore",
"Opc.Ua",
"ZB.MOM.WW.Secrets",
"ZB.MOM.WW.OtOpcUa.Runtime",
"ZB.MOM.WW.OtOpcUa.Host",
"ZB.MOM.WW.OtOpcUa.Driver.",
"Grpc.",
];
}
@@ -0,0 +1,116 @@
using System.Buffers;
using System.Buffers.Binary;
using Google.Protobuf;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
/// <summary>
/// Length-prefixed protobuf framing for the ScriptWorker pipe (lmxopcua#780, design §4.2): a 4-byte little-endian
/// length, then one serialized <see cref="Envelope"/>. Copies mxaccessgw's <c>WorkerFrameProtocol</c> discipline:
/// zero-length, oversize, truncated and unparseable frames are all a <see cref="ScriptWorkerProtocolException"/>,
/// after which the stream is unusable and the peer is dropped.
/// </summary>
public static class ScriptWorkerFrameCodec
{
/// <summary>The protocol version both ends put on every envelope.</summary>
public const uint ProtocolVersion = 1;
/// <summary>Default frame cap: 4 MiB.</summary>
public const int DefaultMaxFrameBytes = 4 * 1024 * 1024;
/// <summary>Writes one frame. Not thread-safe: callers serialise writes.</summary>
/// <param name="stream">The pipe.</param>
/// <param name="envelope">The envelope.</param>
/// <param name="maxFrameBytes">The negotiated cap; a larger envelope is refused before anything is written.</param>
/// <param name="ct">Cancellation.</param>
/// <returns>A task that completes when the frame is written (not flushed).</returns>
public static async ValueTask WriteAsync(Stream stream, Envelope envelope, int maxFrameBytes, CancellationToken ct = default)
{
ArgumentNullException.ThrowIfNull(stream);
ArgumentNullException.ThrowIfNull(envelope);
var size = envelope.CalculateSize();
if (size == 0) throw new ScriptWorkerProtocolException("refusing to write an empty frame");
if (size > maxFrameBytes)
throw new ScriptWorkerProtocolException($"frame of {size} bytes exceeds the {maxFrameBytes}-byte cap");
var buffer = ArrayPool<byte>.Shared.Rent(size + 4);
try
{
BinaryPrimitives.WriteInt32LittleEndian(buffer, size);
envelope.WriteTo(new Span<byte>(buffer, 4, size));
await stream.WriteAsync(buffer.AsMemory(0, size + 4), ct).ConfigureAwait(false);
}
finally
{
ArrayPool<byte>.Shared.Return(buffer);
}
}
/// <summary>Reads one frame.</summary>
/// <param name="stream">The pipe.</param>
/// <param name="maxFrameBytes">The negotiated cap.</param>
/// <param name="ct">Cancellation.</param>
/// <returns>The envelope, or null on a clean end of stream at a frame boundary.</returns>
/// <exception cref="ScriptWorkerProtocolException">A zero-length, oversize, truncated or unparseable frame.</exception>
public static async ValueTask<Envelope?> ReadAsync(Stream stream, int maxFrameBytes, CancellationToken ct = default)
{
ArgumentNullException.ThrowIfNull(stream);
var header = new byte[4];
var got = await ReadFullyAsync(stream, header, ct).ConfigureAwait(false);
if (got == 0) return null;
if (got < 4) throw new ScriptWorkerProtocolException("truncated frame header");
var size = BinaryPrimitives.ReadInt32LittleEndian(header);
if (size == 0) throw new ScriptWorkerProtocolException("zero-length frame");
if (size < 0 || size > maxFrameBytes)
throw new ScriptWorkerProtocolException($"frame length {size} is outside 1..{maxFrameBytes}");
var body = ArrayPool<byte>.Shared.Rent(size);
try
{
var read = await ReadFullyAsync(stream, body.AsMemory(0, size), ct).ConfigureAwait(false);
if (read < size) throw new ScriptWorkerProtocolException($"truncated frame: {read} of {size} bytes");
Envelope envelope;
try
{
envelope = Envelope.Parser.ParseFrom(body, 0, size);
}
catch (InvalidProtocolBufferException ex)
{
throw new ScriptWorkerProtocolException($"unparseable frame: {ex.Message}", ex);
}
if (envelope.ProtocolVersion != ProtocolVersion)
throw new ScriptWorkerProtocolException(
$"protocol version {envelope.ProtocolVersion} is not the supported {ProtocolVersion}");
if (envelope.BodyCase == Envelope.BodyOneofCase.None)
throw new ScriptWorkerProtocolException("frame carries no body");
return envelope;
}
finally
{
ArrayPool<byte>.Shared.Return(body);
}
}
private static async ValueTask<int> ReadFullyAsync(Stream stream, Memory<byte> buffer, CancellationToken ct)
{
var total = 0;
while (total < buffer.Length)
{
var n = await stream.ReadAsync(buffer[total..], ct).ConfigureAwait(false);
if (n == 0) break;
total += n;
}
return total;
}
}
/// <summary>A framing or schema violation on the ScriptWorker pipe; the connection is dropped.</summary>
public sealed class ScriptWorkerProtocolException : Exception
{
/// <summary>Initializes the exception.</summary>
/// <param name="message">What was wrong.</param>
/// <param name="inner">The underlying parser error, if any.</param>
public ScriptWorkerProtocolException(string message, Exception? inner = null) : base(message, inner) { }
}
@@ -0,0 +1,159 @@
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
/// <summary>
/// CLR value ↔ <see cref="Variant"/> (design §4.4). The exact boxed CLR type survives the round trip — a script
/// doing <c>(short)ctx.GetTag(x).Value</c> throws <see cref="InvalidCastException"/> if a <c>short</c> arrives as an
/// <c>int</c> — so the type tag names the CLR type, not just an OPC UA one. A type with no tag is never converted:
/// <see cref="TryEncode"/> refuses it, and the caller reports <c>UNSUPPORTED_TYPE</c> (design §14 #11).
/// </summary>
public static class VariantCodec
{
/// <summary>Encodes <paramref name="value"/>.</summary>
/// <param name="value">The boxed CLR value (null allowed).</param>
/// <param name="variant">The encoded variant when supported.</param>
/// <param name="unsupportedType">The CLR type name when not supported.</param>
/// <returns><c>true</c> when the type has a tag.</returns>
public static bool TryEncode(object? value, out Variant variant, out string? unsupportedType)
{
unsupportedType = null;
variant = new Variant();
switch (value)
{
case null: variant.Type = Variant.Types.TypeTag.Null; return true;
case bool b: variant.Type = Variant.Types.TypeTag.Boolean; variant.B = b; return true;
case sbyte v: variant.Type = Variant.Types.TypeTag.Sbyte; variant.I = v; return true;
case byte v: variant.Type = Variant.Types.TypeTag.Byte; variant.U = v; return true;
case short v: variant.Type = Variant.Types.TypeTag.Int16; variant.I = v; return true;
case ushort v: variant.Type = Variant.Types.TypeTag.Uint16; variant.U = v; return true;
case int v: variant.Type = Variant.Types.TypeTag.Int32; variant.I = v; return true;
case uint v: variant.Type = Variant.Types.TypeTag.Uint32; variant.U = v; return true;
case long v: variant.Type = Variant.Types.TypeTag.Int64; variant.I = v; return true;
case ulong v: variant.Type = Variant.Types.TypeTag.Uint64; variant.U = v; return true;
case float v: variant.Type = Variant.Types.TypeTag.Float; variant.F = v; return true;
case double v: variant.Type = Variant.Types.TypeTag.Double; variant.D = v; return true;
case string v: variant.Type = Variant.Types.TypeTag.String; variant.S = v; return true;
case DateTime v:
variant.Type = Variant.Types.TypeTag.DatetimeUtc;
variant.DtTicks = ToUtc(v).Ticks;
return true;
case Array array when array.Rank == 1 && TryElementTag(array.GetType().GetElementType()!, out var elementTag):
{
var av = new ArrayValue { ElementType = elementTag };
foreach (var item in array)
{
if (!TryEncode(item, out var element, out unsupportedType)) return false;
av.Items.Add(element);
}
variant.Type = Variant.Types.TypeTag.Array;
variant.Array = av;
return true;
}
default:
unsupportedType = value.GetType().FullName ?? value.GetType().Name;
return false;
}
}
/// <summary>Decodes <paramref name="variant"/> back to the boxed CLR value it was encoded from.</summary>
/// <param name="variant">The variant.</param>
/// <returns>The boxed value.</returns>
/// <exception cref="ScriptWorkerProtocolException">An unknown tag, or a payload that does not match the tag.</exception>
public static object? Decode(Variant? variant)
{
if (variant is null) return null;
return variant.Type switch
{
Variant.Types.TypeTag.Null => null,
Variant.Types.TypeTag.Boolean => Expect(variant, Variant.VOneofCase.B).B,
Variant.Types.TypeTag.Sbyte => checked((sbyte)Expect(variant, Variant.VOneofCase.I).I),
Variant.Types.TypeTag.Byte => checked((byte)Expect(variant, Variant.VOneofCase.U).U),
Variant.Types.TypeTag.Int16 => checked((short)Expect(variant, Variant.VOneofCase.I).I),
Variant.Types.TypeTag.Uint16 => checked((ushort)Expect(variant, Variant.VOneofCase.U).U),
Variant.Types.TypeTag.Int32 => checked((int)Expect(variant, Variant.VOneofCase.I).I),
Variant.Types.TypeTag.Uint32 => checked((uint)Expect(variant, Variant.VOneofCase.U).U),
Variant.Types.TypeTag.Int64 => Expect(variant, Variant.VOneofCase.I).I,
Variant.Types.TypeTag.Uint64 => Expect(variant, Variant.VOneofCase.U).U,
Variant.Types.TypeTag.Float => Expect(variant, Variant.VOneofCase.F).F,
Variant.Types.TypeTag.Double => Expect(variant, Variant.VOneofCase.D).D,
Variant.Types.TypeTag.String => Expect(variant, Variant.VOneofCase.S).S,
Variant.Types.TypeTag.DatetimeUtc => new DateTime(Expect(variant, Variant.VOneofCase.DtTicks).DtTicks, DateTimeKind.Utc),
Variant.Types.TypeTag.Array => DecodeArray(Expect(variant, Variant.VOneofCase.Array).Array),
_ => throw new ScriptWorkerProtocolException($"unknown variant type tag {(int)variant.Type}"),
};
}
/// <summary>A <see cref="DateTime"/> as UTC: a <c>Local</c> value is converted; an <c>Unspecified</c> one is taken as UTC.</summary>
/// <param name="value">The value.</param>
/// <returns>The UTC value.</returns>
public static DateTime ToUtc(DateTime value) => value.Kind switch
{
DateTimeKind.Utc => value,
DateTimeKind.Local => value.ToUniversalTime(),
_ => DateTime.SpecifyKind(value, DateTimeKind.Utc),
};
private static Variant Expect(Variant variant, Variant.VOneofCase expected)
{
// proto3 drops a default scalar (0, false, "") from the oneof, so an absent payload is the default value.
if (variant.VCase != expected && variant.VCase != Variant.VOneofCase.None)
throw new ScriptWorkerProtocolException($"variant tagged {variant.Type} carries a {variant.VCase} payload");
return variant;
}
private static Array DecodeArray(ArrayValue? array)
{
if (array is null) throw new ScriptWorkerProtocolException("array variant without an array payload");
var elementType = ClrType(array.ElementType);
var result = Array.CreateInstance(elementType, array.Items.Count);
for (var i = 0; i < array.Items.Count; i++)
{
var item = array.Items[i];
if (item.Type != array.ElementType && item.Type != Variant.Types.TypeTag.Null)
throw new ScriptWorkerProtocolException($"array of {array.ElementType} holds a {item.Type} element");
result.SetValue(Decode(item), i);
}
return result;
}
private static bool TryElementTag(Type elementType, out Variant.Types.TypeTag tag)
{
tag = elementType switch
{
_ when elementType == typeof(bool) => Variant.Types.TypeTag.Boolean,
_ when elementType == typeof(sbyte) => Variant.Types.TypeTag.Sbyte,
_ when elementType == typeof(byte) => Variant.Types.TypeTag.Byte,
_ when elementType == typeof(short) => Variant.Types.TypeTag.Int16,
_ when elementType == typeof(ushort) => Variant.Types.TypeTag.Uint16,
_ when elementType == typeof(int) => Variant.Types.TypeTag.Int32,
_ when elementType == typeof(uint) => Variant.Types.TypeTag.Uint32,
_ when elementType == typeof(long) => Variant.Types.TypeTag.Int64,
_ when elementType == typeof(ulong) => Variant.Types.TypeTag.Uint64,
_ when elementType == typeof(float) => Variant.Types.TypeTag.Float,
_ when elementType == typeof(double) => Variant.Types.TypeTag.Double,
_ when elementType == typeof(string) => Variant.Types.TypeTag.String,
_ when elementType == typeof(DateTime) => Variant.Types.TypeTag.DatetimeUtc,
_ => Variant.Types.TypeTag.Null,
};
return tag != Variant.Types.TypeTag.Null;
}
private static Type ClrType(Variant.Types.TypeTag tag) => tag switch
{
Variant.Types.TypeTag.Boolean => typeof(bool),
Variant.Types.TypeTag.Sbyte => typeof(sbyte),
Variant.Types.TypeTag.Byte => typeof(byte),
Variant.Types.TypeTag.Int16 => typeof(short),
Variant.Types.TypeTag.Uint16 => typeof(ushort),
Variant.Types.TypeTag.Int32 => typeof(int),
Variant.Types.TypeTag.Uint32 => typeof(uint),
Variant.Types.TypeTag.Int64 => typeof(long),
Variant.Types.TypeTag.Uint64 => typeof(ulong),
Variant.Types.TypeTag.Float => typeof(float),
Variant.Types.TypeTag.Double => typeof(double),
Variant.Types.TypeTag.String => typeof(string),
Variant.Types.TypeTag.DatetimeUtc => typeof(DateTime),
_ => throw new ScriptWorkerProtocolException($"array element type {tag} is not supported"),
};
}
@@ -0,0 +1,28 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<RootNamespace>ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol</RootNamespace>
<GenerateDocumentationFile>true</GenerateDocumentationFile>
<!-- protoc-generated members carry no XML docs. -->
<NoWarn>$(NoWarn);CS1591</NoWarn>
</PropertyGroup>
<!-- lmxopcua#780: the host <-> ScriptWorker frame protocol (design §4). Shared by the host-side runner and the
worker, so it references nothing but Google.Protobuf: the worker's assembly set stays small (design §6). -->
<ItemGroup>
<PackageReference Include="Google.Protobuf"/>
<PackageReference Include="Grpc.Tools">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
</ItemGroup>
<ItemGroup>
<Protobuf Include="Protos\script_worker.proto" GrpcServices="None"/>
</ItemGroup>
<ItemGroup>
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests"/>
</ItemGroup>
</Project>
@@ -0,0 +1,137 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting;
/// <summary>
/// The execution seam every host script owner can run a compiled script through (lmxopcua#780, design §11): a
/// script key, its source, its input snapshots and its budget go in; one <see cref="ScriptRunOutcome"/> comes
/// out. Implementations never throw for script faults.
/// </summary>
/// <remarks>
/// <para>
/// <see cref="InProcessScriptRunner"/> is today's <see cref="ScriptRunGuard"/> + <see cref="CompiledScriptCache{TContext, TResult}"/>
/// behind this interface (the parity baseline). The out-of-process runner
/// (<c>ZB.MOM.WW.OtOpcUa.Core.Scripting.Runner</c>) runs the same compile in a pre-emptible worker process.
/// The passthrough fast path (<c>PassthroughScript</c>) never reaches a runner: it runs no user code.
/// </para>
/// <para>
/// Keys are namespaced by <see cref="ScriptInvocation.Kind"/>, so a virtual-tag id and a calc RawPath that
/// happen to be equal never share quarantine state, and <see cref="Reset"/> for one seam leaves the other
/// seams' quarantines alone.
/// </para>
/// </remarks>
public interface IScriptRunner
{
/// <summary>Runs one compiled evaluation.</summary>
/// <param name="invocation">What to run.</param>
/// <param name="ct">Caller cancellation (shutdown); wins over the budget.</param>
/// <returns>The outcome; never faults for a script failure.</returns>
Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default);
/// <summary>
/// The deploy boundary for one seam: lifts every quarantine and forgets every remembered compile failure of
/// <paramref name="kind"/>, and drops its compiled scripts.
/// </summary>
/// <param name="kind">The seam's context kind.</param>
void Reset(ScriptContextKind kind);
/// <summary>The keys of <paramref name="kind"/> currently quarantined.</summary>
/// <param name="kind">The seam's context kind.</param>
/// <returns>The quarantined keys (un-namespaced).</returns>
IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind);
}
/// <summary>Which script context (and result type) a script compiles against.</summary>
public enum ScriptContextKind
{
/// <summary>A virtual tag: <c>VirtualTagContext</c> → <c>object?</c>.</summary>
VirtualTag = 1,
/// <summary>A Calculation tag: <c>VirtualTagContext</c> → <c>object?</c>.</summary>
Calculation = 2,
/// <summary>A scripted-alarm predicate: <c>AlarmPredicateContext</c> → <c>bool</c> (not run out of process yet).</summary>
AlarmPredicate = 3,
}
/// <summary>One evaluation handed to an <see cref="IScriptRunner"/>.</summary>
/// <param name="Key">The script instance key (virtual-tag id, calc RawPath, alarm id).</param>
/// <param name="Kind">The context kind.</param>
/// <param name="Source">The script source (also its identity: a change lifts the key's quarantine).</param>
/// <param name="Inputs">The read cache: dependency snapshots by path. Not mutated until the run completes.</param>
/// <param name="Timeout">The soft budget <c>T</c>.</param>
/// <param name="ScriptLogger">The per-evaluation script logger <c>ctx.Logger</c> writes through; null ⇒ a silent logger.</param>
public sealed record ScriptInvocation(
string Key,
ScriptContextKind Kind,
string Source,
IReadOnlyDictionary<string, DataValueSnapshot> Inputs,
TimeSpan Timeout,
Serilog.ILogger? ScriptLogger = null);
/// <summary>How an evaluation ended.</summary>
public enum ScriptRunStatus
{
/// <summary>The script returned <see cref="ScriptRunOutcome.Value"/>.</summary>
Ok = 0,
/// <summary>A script-attributable failure (threw, timed out, refused, quarantined, did not compile) — Bad as today.</summary>
Failed = 1,
/// <summary>
/// The runner could not run the script, and it is not the script's fault: its worker restarted under it, or no
/// worker became ready in time. Published as <c>BadResourceUnavailable</c> (design §9 / §14 #3).
/// </summary>
Unavailable = 2,
}
/// <summary>The result of one <see cref="IScriptRunner.RunAsync"/>.</summary>
/// <param name="Status">How it ended.</param>
/// <param name="Value">The script's result when <see cref="ScriptRunStatus.Ok"/>.</param>
/// <param name="Reason">The failure reason otherwise; the same strings the evaluators have always published.</param>
public sealed record ScriptRunOutcome(ScriptRunStatus Status, object? Value, string? Reason)
{
/// <summary>Reason when the worker running the script died under it (another script's kill, a crash).</summary>
public const string WorkerRestartedReason = "script worker restarted";
/// <summary>Reason when no worker became ready within <see cref="ScriptWorkerOptions.UnavailableGrace"/>.</summary>
public const string RunnerUnavailableReason = "script runner unavailable";
/// <summary>A successful run.</summary>
/// <param name="value">The script's result.</param>
/// <returns>The outcome.</returns>
public static ScriptRunOutcome Ok(object? value) => new(ScriptRunStatus.Ok, value, null);
/// <summary>A script-attributable failure.</summary>
/// <param name="reason">The reason.</param>
/// <returns>The outcome.</returns>
public static ScriptRunOutcome Failed(string reason) => new(ScriptRunStatus.Failed, null, reason);
/// <summary>A runner-attributable failure.</summary>
/// <param name="reason">The reason.</param>
/// <returns>The outcome.</returns>
public static ScriptRunOutcome Unavailable(string reason) => new(ScriptRunStatus.Unavailable, null, reason);
}
/// <summary>Maps an <see cref="IScriptRunner"/> outcome onto the value seams' result type.</summary>
public static class ScriptRunOutcomeExtensions
{
/// <summary>
/// <see cref="ScriptRunStatus.Ok"/> → <c>Ok(value)</c>; <see cref="ScriptRunStatus.Failed"/> → <c>Failure(reason)</c>
/// (published <c>BadInternalError</c>, as today); <see cref="ScriptRunStatus.Unavailable"/> →
/// <c>Unavailable(reason)</c> (published <c>BadResourceUnavailable</c>).
/// </summary>
/// <param name="outcome">The runner's outcome.</param>
/// <returns>The seam result.</returns>
public static Commons.Engines.VirtualTagEvalResult ToVirtualTagEvalResult(this ScriptRunOutcome outcome)
{
ArgumentNullException.ThrowIfNull(outcome);
return outcome.Status switch
{
ScriptRunStatus.Ok => Commons.Engines.VirtualTagEvalResult.Ok(outcome.Value),
ScriptRunStatus.Unavailable => Commons.Engines.VirtualTagEvalResult.Unavailable(
outcome.Reason ?? ScriptRunOutcome.RunnerUnavailableReason),
_ => Commons.Engines.VirtualTagEvalResult.Failure(outcome.Reason ?? "evaluator failure"),
};
}
}
@@ -0,0 +1,141 @@
using System.Collections.Concurrent;
using Serilog;
using Serilog.Core;
using ZB.MOM.WW.OtOpcUa.Commons.Engines;
using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
using ZB.MOM.WW.OtOpcUa.Core.VirtualTags;
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting;
/// <summary>
/// <see cref="IScriptRunner"/> over today's in-process machinery: a <see cref="CompiledScriptCache{TContext, TResult}"/>
/// per context kind and one <see cref="ScriptRunGuard"/> (lmxopcua#780, design §11). It produces the outcomes and
/// reason strings the seams' own in-process paths produce, which makes it the parity baseline the out-of-process
/// runner is tested against, and the in-process half of the future shadow mode (P2).
/// </summary>
/// <remarks>
/// The seams do not route their <see cref="ScriptRunnerMode.InProcess"/> path through this class: they keep their
/// existing guard code, so the default mode is unchanged byte for byte. Unlike those paths, this runner remembers
/// compile failures by source for every kind (Calculation already did, R07:P4).
/// </remarks>
public sealed class InProcessScriptRunner : IScriptRunner, IDisposable
{
private readonly CompiledScriptCache<VirtualTagContext, object?> _valueCache = new();
private readonly CompiledScriptCache<AlarmPredicateContext, bool> _predicateCache = new();
private readonly ConcurrentDictionary<string, string> _compileFailures = new(StringComparer.Ordinal);
private readonly ScriptRunGuard _guard;
/// <summary>Initializes the runner.</summary>
/// <param name="scripting">Run-guard knobs; null ⇒ defaults.</param>
/// <param name="logger">Where quarantine lines go; null ⇒ Serilog's global logger.</param>
public InProcessScriptRunner(ScriptingOptions? scripting = null, ILogger? logger = null)
{
_guard = new ScriptRunGuard(scripting, logger);
}
/// <summary>The run guard (diagnostics / tests).</summary>
public ScriptRunGuard RunGuard => _guard;
/// <inheritdoc />
public async Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default)
{
ArgumentNullException.ThrowIfNull(invocation);
var key = ScopedKey(invocation.Kind, invocation.Key);
var failureKey = ScopedKey(invocation.Kind, invocation.Source);
if (_compileFailures.TryGetValue(failureKey, out var known)) return ScriptRunOutcome.Failed(known);
var logger = invocation.ScriptLogger ?? Logger.None;
try
{
switch (invocation.Kind)
{
case ScriptContextKind.VirtualTag:
case ScriptContextKind.Calculation:
{
var evaluator = await Task.Run(() => _valueCache.GetOrCompile(invocation.Source), ct).ConfigureAwait(false);
var context = new VirtualTagContext(invocation.Inputs, static (_, _) => { }, logger);
var value = await _guard.RunAsync(key, invocation.Source, evaluator, context, invocation.Timeout, ct)
.ConfigureAwait(false);
return ScriptRunOutcome.Ok(value);
}
case ScriptContextKind.AlarmPredicate:
{
var evaluator = await Task.Run(() => _predicateCache.GetOrCompile(invocation.Source), ct).ConfigureAwait(false);
var context = new AlarmPredicateContext(invocation.Inputs, logger);
var value = await _guard.RunAsync(key, invocation.Source, evaluator, context, invocation.Timeout, ct)
.ConfigureAwait(false);
return ScriptRunOutcome.Ok(value);
}
default:
return ScriptRunOutcome.Failed($"unsupported script kind {invocation.Kind}");
}
}
catch (CompilationErrorException ex)
{
return RememberCompileFailure(failureKey, $"compile error: {ex.Message}");
}
catch (ScriptSandboxViolationException ex)
{
return RememberCompileFailure(failureKey, $"sandbox violation: {ex.Message}");
}
catch (ScriptTimeoutException)
{
return ScriptRunOutcome.Failed($"{ScriptEvalFailureReasons.TimedOutPrefix}{invocation.Timeout.TotalSeconds:F1}s");
}
catch (ScriptRunRefusedException ex)
{
return ScriptRunOutcome.Failed(ex.Message);
}
catch (ScriptQuarantinedException ex)
{
return ScriptRunOutcome.Failed(ex.Message);
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
return ScriptRunOutcome.Failed("evaluation cancelled");
}
catch (Exception ex)
{
return ScriptRunOutcome.Failed($"script threw: {ex.Message}");
}
}
/// <inheritdoc />
public void Reset(ScriptContextKind kind)
{
var prefix = ScopedKey(kind, string.Empty);
foreach (var k in _compileFailures.Keys)
if (k.StartsWith(prefix, StringComparison.Ordinal)) _compileFailures.TryRemove(k, out _);
foreach (var k in _guard.QuarantinedKeys)
if (k.StartsWith(prefix, StringComparison.Ordinal)) _guard.Reset(k);
}
/// <inheritdoc />
public IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind)
{
var prefix = ScopedKey(kind, string.Empty);
return _guard.QuarantinedKeys
.Where(k => k.StartsWith(prefix, StringComparison.Ordinal))
.Select(k => k[prefix.Length..])
.ToArray();
}
/// <inheritdoc />
public void Dispose()
{
_valueCache.Dispose();
_predicateCache.Dispose();
}
/// <summary>The runner-internal key: <c>{kind}:{key}</c>.</summary>
/// <param name="kind">The context kind.</param>
/// <param name="key">The seam's key.</param>
/// <returns>The namespaced key.</returns>
public static string ScopedKey(ScriptContextKind kind, string key) => $"{(int)kind}:{key}";
private ScriptRunOutcome RememberCompileFailure(string failureKey, string reason)
{
_compileFailures.TryAdd(failureKey, reason);
return ScriptRunOutcome.Failed(reason);
}
}
@@ -0,0 +1,213 @@
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting;
/// <summary>
/// Where compiled scripts run (<c>Scripting:Runner</c>, lmxopcua#780). <see cref="InProcess"/> is the default
/// and the documented rollback: it is today's <see cref="ScriptRunGuard"/> path, unchanged.
/// </summary>
public enum ScriptRunnerMode
{
/// <summary>Compiled scripts run in the host process under <see cref="ScriptRunGuard"/> (the default).</summary>
InProcess = 0,
/// <summary>
/// Run in process AND submit to the worker, comparing the two (design P2). Not implemented yet: a node
/// configured for it runs <see cref="InProcess"/> and logs a startup Warning.
/// </summary>
Shadow = 1,
/// <summary>
/// Compiled scripts of the selected seams run in the pre-emptible <c>ScriptWorker</c> process pool. The
/// runner fails closed: a worker that cannot start makes the script Bad
/// (<c>BadResourceUnavailable</c>), never an in-process run.
/// </summary>
OutOfProcess = 2,
}
/// <summary>The three host seams that run operator-authored C# (<c>Scripting:Runner:Seams</c>).</summary>
public enum ScriptSeam
{
/// <summary>Virtual tags (<c>RoslynVirtualTagEvaluator</c>).</summary>
VirtualTag = 0,
/// <summary>Calculation tags (<c>CalculationEvaluator</c>).</summary>
Calculation = 1,
/// <summary>Scripted-alarm predicates (<c>ScriptedAlarmEngine</c>). Last in the rollout (design §14 #10).</summary>
ScriptedAlarm = 2,
}
/// <summary>
/// The out-of-process worker pool's knobs (the <c>Scripting:Worker</c> section). Every value is clamped by
/// <see cref="Sanitized"/>, which reports each out-of-range key as a startup Warning, like the rest of the
/// <c>Scripting</c> section. Only consulted when <see cref="ScriptingOptions.Runner"/> is not
/// <see cref="ScriptRunnerMode.InProcess"/>.
/// </summary>
public sealed class ScriptWorkerOptions
{
/// <summary>Default <see cref="Shards"/>: 2 (design §14 #1).</summary>
public const int DefaultShards = 2;
/// <summary>Upper bound of <see cref="Shards"/>.</summary>
public const int MaxShards = 8;
/// <summary>Default <see cref="HeapLimitBytes"/>: 512 MiB.</summary>
public const long DefaultHeapLimitBytes = 512L * 1024 * 1024;
/// <summary>Lower bound of <see cref="HeapLimitBytes"/>: 64 MiB (a worker below this cannot host Roslyn).</summary>
public const long MinHeapLimitBytes = 64L * 1024 * 1024;
/// <summary>Default <see cref="QuarantineAfterKills"/>: 2.</summary>
public const int DefaultQuarantineAfterKills = 2;
/// <summary>Default <see cref="UnavailableGrace"/>: 5 s.</summary>
public static readonly TimeSpan DefaultUnavailableGrace = TimeSpan.FromSeconds(5);
/// <summary>Lower bound of an explicit <see cref="HardGrace"/>: 250 ms.</summary>
public static readonly TimeSpan MinHardGrace = TimeSpan.FromMilliseconds(250);
/// <summary>Upper bound of an explicit <see cref="HardGrace"/>: 30 s.</summary>
public static readonly TimeSpan MaxHardGrace = TimeSpan.FromSeconds(30);
/// <summary>Upper bound of <see cref="UnavailableGrace"/>: 60 s.</summary>
public static readonly TimeSpan MaxUnavailableGrace = TimeSpan.FromSeconds(60);
/// <summary>Default <see cref="IsolationProbationRuns"/>: 20.</summary>
public const int DefaultIsolationProbationRuns = 20;
/// <summary>Shard workers per node; a script is placed by a stable hash of its key. Clamped to 1–8.</summary>
public int Shards { get; set; } = DefaultShards;
/// <summary>
/// Grace after the soft deadline <c>T</c> before the worker running the script is killed. Null (the default)
/// means <c>max(1 s, T)</c> per run (design §14 #2); an explicit value is clamped to 250 ms – 30 s.
/// </summary>
public TimeSpan? HardGrace { get; set; }
/// <summary>The worker's <c>DOTNET_GCHeapHardLimit</c>. Default 512 MiB, minimum 64 MiB.</summary>
public long HeapLimitBytes { get; set; } = DefaultHeapLimitBytes;
/// <summary>Hard-deadline kills (within one hour) after which a script is quarantined. Default 2; 1–100.</summary>
public int QuarantineAfterKills { get; set; } = DefaultQuarantineAfterKills;
/// <summary>
/// How long an evaluation waits for its shard to become ready (a restart, a backoff) before it reports
/// <c>script runner unavailable</c>. Default 5 s; 0–60 s.
/// </summary>
public TimeSpan UnavailableGrace { get; set; } = DefaultUnavailableGrace;
/// <summary>
/// Successful runs a suspect script must complete in the isolation worker before it returns to its shard.
/// Default 20; 1–10 000.
/// </summary>
public int IsolationProbationRuns { get; set; } = DefaultIsolationProbationRuns;
/// <summary>
/// Directory of the node-local compiled-PE cache (design §7). Must be absolute and outside the content root
/// when set; startup fails otherwise (the family's fail-closed path rule). Reserved: the cache itself has
/// not landed yet (lmxopcua#780 P1 remainder), so a worker restart recompiles.
/// </summary>
public string? PeCacheDir { get; set; }
/// <summary>
/// Path of the worker entry assembly (<c>ZB.MOM.WW.OtOpcUa.ScriptWorker.dll</c>). Null (the default) means
/// <c>script-worker/</c> under the host's base directory, where the Host build places it.
/// </summary>
public string? ExecutablePath { get; set; }
/// <summary>The hard grace that applies to a run with soft budget <paramref name="softTimeout"/>.</summary>
/// <param name="softTimeout">The run's soft deadline <c>T</c>.</param>
/// <returns><see cref="HardGrace"/> when set, otherwise <c>max(1 s, T)</c>.</returns>
public TimeSpan EffectiveHardGrace(TimeSpan softTimeout)
=> HardGrace ?? (softTimeout > TimeSpan.FromSeconds(1) ? softTimeout : TimeSpan.FromSeconds(1));
/// <summary>Returns a clamped copy; one line per out-of-range key in <paramref name="adjustments"/>.</summary>
/// <param name="adjustments">The clamps applied, naming each key.</param>
/// <returns>A new, clamped instance.</returns>
public ScriptWorkerOptions Sanitized(out IReadOnlyList<string> adjustments)
{
const string prefix = ScriptingOptions.SectionName + ":Worker:";
var notes = new List<string>();
var shards = Shards;
if (shards < 1 || shards > MaxShards)
{
var clamped = Math.Clamp(shards, 1, MaxShards);
notes.Add($"{prefix}{nameof(Shards)} {shards} is outside 1–{MaxShards}; clamped to {clamped}.");
shards = clamped;
}
var grace = HardGrace;
if (grace is { } g && (g < MinHardGrace || g > MaxHardGrace))
{
var clamped = g < MinHardGrace ? MinHardGrace : MaxHardGrace;
notes.Add($"{prefix}{nameof(HardGrace)} {g} is outside {MinHardGrace}–{MaxHardGrace}; clamped to {clamped}.");
grace = clamped;
}
var heap = HeapLimitBytes;
if (heap < MinHeapLimitBytes)
{
notes.Add($"{prefix}{nameof(HeapLimitBytes)} {heap} is below the minimum; clamped to {MinHeapLimitBytes}.");
heap = MinHeapLimitBytes;
}
var kills = QuarantineAfterKills;
if (kills < 1 || kills > 100)
{
var clamped = Math.Clamp(kills, 1, 100);
notes.Add($"{prefix}{nameof(QuarantineAfterKills)} {kills} is outside 1–100; clamped to {clamped}.");
kills = clamped;
}
var unavailable = UnavailableGrace;
if (unavailable < TimeSpan.Zero || unavailable > MaxUnavailableGrace)
{
var clamped = unavailable < TimeSpan.Zero ? TimeSpan.Zero : MaxUnavailableGrace;
notes.Add($"{prefix}{nameof(UnavailableGrace)} {unavailable} is outside 0–{MaxUnavailableGrace}; clamped to {clamped}.");
unavailable = clamped;
}
var probation = IsolationProbationRuns;
if (probation < 1 || probation > 10_000)
{
var clamped = Math.Clamp(probation, 1, 10_000);
notes.Add($"{prefix}{nameof(IsolationProbationRuns)} {probation} is outside 1–10000; clamped to {clamped}.");
probation = clamped;
}
adjustments = notes;
return new ScriptWorkerOptions
{
Shards = shards,
HardGrace = grace,
HeapLimitBytes = heap,
QuarantineAfterKills = kills,
UnavailableGrace = unavailable,
IsolationProbationRuns = probation,
PeCacheDir = PeCacheDir,
ExecutablePath = ExecutablePath,
};
}
/// <summary>
/// Fail-closed path validation (design §7): <see cref="PeCacheDir"/>, when set, must be absolute and must not
/// sit inside <paramref name="contentRootPath"/>. Same rule as the family's <c>Secrets:SqlitePath</c>.
/// </summary>
/// <param name="contentRootPath">The host's content root.</param>
/// <returns>The violation, or null when the options are acceptable.</returns>
public string? ValidatePaths(string contentRootPath)
{
if (string.IsNullOrWhiteSpace(PeCacheDir)) return null;
const string key = ScriptingOptions.SectionName + ":Worker:" + nameof(PeCacheDir);
if (!Path.IsPathFullyQualified(PeCacheDir))
return $"{key} '{PeCacheDir}' must be an absolute path.";
var dir = Path.TrimEndingDirectorySeparator(Path.GetFullPath(PeCacheDir));
var root = Path.TrimEndingDirectorySeparator(Path.GetFullPath(contentRootPath));
var comparison = OperatingSystem.IsWindows() ? StringComparison.OrdinalIgnoreCase : StringComparison.Ordinal;
if (string.Equals(dir, root, comparison)
|| dir.StartsWith(root + Path.DirectorySeparatorChar, comparison)
|| dir.StartsWith(root + Path.AltDirectorySeparatorChar, comparison))
return $"{key} '{PeCacheDir}' is inside the content root '{contentRootPath}'; it must live outside it.";
return null;
}
}
@@ -72,6 +72,30 @@ public sealed class ScriptingOptions
/// </summary>
public double StuckOrphanQuarantineFactor { get; set; } = DefaultStuckOrphanQuarantineFactor;
/// <summary>
/// Where compiled scripts run (<c>Scripting:Runner</c>, lmxopcua#780). Default
/// <see cref="ScriptRunnerMode.InProcess"/> — today's path, unchanged, and the rollback for the other modes.
/// </summary>
public ScriptRunnerMode Runner { get; set; } = ScriptRunnerMode.InProcess;
/// <summary>
/// The seams <see cref="Runner"/> applies to when it is not <see cref="ScriptRunnerMode.InProcess"/>
/// (<c>Scripting:Runner:Seams</c>, e.g. <c>["Calculation"]</c>). Null or empty means all three. Bound by
/// <c>ScriptingOptionsBinding</c> from the documented key, because <c>Scripting:Runner</c> is itself a value.
/// </summary>
public IReadOnlyList<ScriptSeam>? RunnerSeams { get; set; }
/// <summary>The out-of-process worker pool's knobs (<c>Scripting:Worker</c>).</summary>
public ScriptWorkerOptions Worker { get; set; } = new();
/// <summary>Whether <paramref name="seam"/> is selected for out-of-process execution by these options.</summary>
/// <param name="seam">The seam.</param>
/// <returns><c>true</c> when <see cref="Runner"/> is <see cref="ScriptRunnerMode.OutOfProcess"/> and the seam is in
/// <see cref="RunnerSeams"/> (or that list is empty).</returns>
public bool IsOutOfProcess(ScriptSeam seam)
=> Runner == ScriptRunnerMode.OutOfProcess
&& (RunnerSeams is not { Count: > 0 } seams || seams.Contains(seam));
/// <summary>Upper bound <see cref="Sanitized"/> applies to <see cref="StuckOrphanQuarantineFactor"/>.</summary>
public const double MaxStuckOrphanQuarantineFactor = 1000;
@@ -158,6 +182,9 @@ public sealed class ScriptingOptions
runTimeout = MaxRunTimeout;
}
var worker = (Worker ?? new ScriptWorkerOptions()).Sanitized(out var workerNotes);
notes.AddRange(workerNotes);
adjustments = notes;
return new ScriptingOptions
{
@@ -165,6 +192,9 @@ public sealed class ScriptingOptions
MaxAllocBytesPerRun = alloc,
QuarantineAfterConsecutiveTimeouts = threshold,
StuckOrphanQuarantineFactor = factor,
Runner = Runner,
RunnerSeams = RunnerSeams is null ? null : [.. RunnerSeams],
Worker = worker,
};
}
}