feat(scripting): out-of-process pre-emptible script runner behind Scripting:Runner (lmxopcua#780 P1)
v2-ci / unit-tests (push) Waiting to run
v2-ci / unit-tests (push) Waiting to run
Opt-in (Scripting:Runner=OutOfProcess; InProcess stays the default and its code path is unchanged). Implements the P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md with the owner's §14 decisions: - Options: Scripting:Runner (InProcess | Shadow | OutOfProcess), Scripting:Runner:Seams, Scripting:Worker:* (Shards 2, HardGrace max(1 s,T), HeapLimitBytes 512 MiB, QuarantineAfterKills 2, UnavailableGrace 5 s, IsolationProbationRuns 20, PeCacheDir validated absolute + outside the content root (fail closed), ExecutablePath). Shadow is not implemented yet: runs InProcess with a startup Warning. - Core.Scripting: IScriptRunner seam + InProcessScriptRunner (today's guard, the parity baseline). - Core.Scripting.Protocol: protobuf envelopes, 4-byte LE framing (zero/oversize/truncated/garbage refused), Variant carrying the exact CLR type; unsupported types are explicit failures. - ScriptWorker app: compiles in the worker, P-thread scheduler, host-driven soft cancel, fail-fast watchdog, heartbeats, ctx.Logger forwarding, exits on pipe EOF / parent gone, forbidden-assembly self-check; Akka and Grpc.Core.Api excluded from its folder. Shipped in script-worker/ (Build + Publish). - Core.Scripting.OutOfProcess: K shards + isolation worker, soft T / hard T+G kill, attribution (deadline kill = one key; crash = suspects -> isolation; isolation crash = strike), kill-strike quarantine, backoff 250 ms -> 30 s, fail closed, cleared allow-listed env, nonce + pid handshake on a current-user pipe, Windows Job Object. - VirtualTag + Calculation seams use the runner when selected; a runner-side failure (worker restarted / runner unavailable) publishes BadResourceUnavailable (0x80040000) — ScadaBridge#112. Script faults keep BadInternalError. The scripted-alarm seam stays in process (alarms last). Tests: new ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests (Category=ScriptWorker, real processes; green on macOS and on windev/Windows), seam tests in Host/Runtime/Calculation suites; mutation-checked. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,505 @@
|
||||
using System.Collections.Concurrent;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
using Serilog;
|
||||
using Serilog.Core;
|
||||
using Serilog.Events;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Engines;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
|
||||
|
||||
/// <summary>
|
||||
/// The pre-emptible <see cref="IScriptRunner"/> (lmxopcua#780, design §3–§9): compiled scripts run in a pool of
|
||||
/// <c>ScriptWorker</c> processes — K shards placed by a stable hash of the script key, plus an isolation worker
|
||||
/// for suspects — so a script that overruns its hard deadline is stopped by killing its worker, and an OOM, a
|
||||
/// native crash or a fail-fast takes down a worker instead of the server.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <list type="bullet">
|
||||
/// <item><b>Two deadlines.</b> At the soft deadline <c>T</c> the caller gets today's timeout failure (and the
|
||||
/// run token is cancelled inside the worker). At <c>T + G</c> (<see cref="ScriptWorkerOptions.EffectiveHardGrace"/>)
|
||||
/// the worker is killed and the key takes a kill strike.</item>
|
||||
/// <item><b>Attribution.</b> A deadline kill blames exactly one key; the other evaluations in flight on that
|
||||
/// worker are victims and report <see cref="ScriptRunOutcome.WorkerRestartedReason"/> (no strike). A crash with
|
||||
/// no deadline crossed makes every key in flight a suspect, and suspects run in the isolation worker one at a
|
||||
/// time, where a crash names exactly one key. A suspect returns to its shard after
|
||||
/// <see cref="ScriptWorkerOptions.IsolationProbationRuns"/> clean runs.</item>
|
||||
/// <item><b>Quarantine</b> lives here, in the host, so it survives worker restarts: kill strikes (within an
|
||||
/// hour), consecutive soft timeouts, or one allocation-budget breach. Reasons keep the
|
||||
/// <see cref="ScriptRunGuard.QuarantinedReasonPrefix"/>, and a source change or <see cref="Reset"/> lifts them.</item>
|
||||
/// <item><b>Fail closed.</b> When no worker is ready within <see cref="ScriptWorkerOptions.UnavailableGrace"/>
|
||||
/// the result is <see cref="ScriptRunStatus.Unavailable"/> — never an in-process run.</item>
|
||||
/// </list>
|
||||
/// </remarks>
|
||||
public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable, IDisposable
|
||||
{
|
||||
/// <summary>The worker entry assembly's file name.</summary>
|
||||
public const string WorkerAssemblyFileName = "ZB.MOM.WW.OtOpcUa.ScriptWorker.dll";
|
||||
|
||||
/// <summary>The folder under the host's base directory the Host build places the worker in.</summary>
|
||||
public const string WorkerFolderName = "script-worker";
|
||||
|
||||
/// <summary>Failure reason when the worker could not take more work.</summary>
|
||||
public const string WorkerBusyReason = "script worker busy";
|
||||
|
||||
private static readonly TimeSpan KillStrikeWindow = TimeSpan.FromHours(1);
|
||||
|
||||
private readonly ScriptingOptions _options;
|
||||
private readonly ScriptWorkerOptions _worker;
|
||||
private readonly ILogger _log;
|
||||
private readonly ILogger _scriptRoot;
|
||||
private readonly WorkerShard[] _shards;
|
||||
private readonly WorkerShard _isolation;
|
||||
private readonly ConcurrentDictionary<string, KeyState> _keys = new(StringComparer.Ordinal);
|
||||
private readonly ConcurrentDictionary<string, string> _compileFailures = new(StringComparer.Ordinal);
|
||||
private int _disposed;
|
||||
|
||||
/// <summary>Initializes the runner. No process starts until the first evaluation.</summary>
|
||||
/// <param name="options">The node's <c>Scripting</c> options (sanitized again here).</param>
|
||||
/// <param name="log">Host diagnostics: starts, kills, quarantines; null ⇒ Serilog's global logger.</param>
|
||||
/// <param name="scriptRoot">Fallback logger for forwarded <c>ctx.Logger</c> lines; null ⇒ <paramref name="log"/>.</param>
|
||||
public OutOfProcessScriptRunner(ScriptingOptions options, ILogger? log = null, ILogger? scriptRoot = null)
|
||||
: this(options, log, scriptRoot, heartbeatInterval: TimeSpan.FromSeconds(1), loadTimeout: TimeSpan.FromSeconds(60))
|
||||
{
|
||||
}
|
||||
|
||||
internal OutOfProcessScriptRunner(ScriptingOptions options, ILogger? log, ILogger? scriptRoot,
|
||||
TimeSpan heartbeatInterval, TimeSpan loadTimeout)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
_options = options.Sanitized(out _);
|
||||
_worker = _options.Worker;
|
||||
_log = log ?? Log.Logger;
|
||||
_scriptRoot = scriptRoot ?? _log;
|
||||
LoadTimeout = loadTimeout;
|
||||
WorkerPath = ResolveWorkerPath(_worker);
|
||||
|
||||
var evalThreads = Math.Max(2, Environment.ProcessorCount / _worker.Shards);
|
||||
_shards = new WorkerShard[_worker.Shards];
|
||||
for (var i = 0; i < _shards.Length; i++)
|
||||
{
|
||||
var label = i.ToString(System.Globalization.CultureInfo.InvariantCulture);
|
||||
_shards[i] = NewShard(label, isolation: false, evalThreads, heartbeatInterval);
|
||||
}
|
||||
_isolation = NewShard("iso", isolation: true, evalThreads: 1, heartbeatInterval);
|
||||
}
|
||||
|
||||
/// <summary>The worker entry assembly this runner launches.</summary>
|
||||
public string WorkerPath { get; }
|
||||
|
||||
/// <summary>How long a compile in the worker may take before the worker is killed.</summary>
|
||||
internal TimeSpan LoadTimeout { get; }
|
||||
|
||||
/// <summary>The shard workers (tests).</summary>
|
||||
internal IReadOnlyList<WorkerShard> Shards => _shards;
|
||||
|
||||
/// <summary>The isolation worker (tests).</summary>
|
||||
internal WorkerShard Isolation => _isolation;
|
||||
|
||||
/// <summary>Scoped keys currently treated as suspects (tests / diagnostics).</summary>
|
||||
internal IReadOnlyCollection<string> SuspectKeys =>
|
||||
_keys.Where(kv => kv.Value.Suspect).Select(kv => kv.Key).ToArray();
|
||||
|
||||
/// <summary>The shard a key is placed on when it is not a suspect: a stable FNV-1a hash, so placement survives restarts.</summary>
|
||||
/// <param name="scopedKey">The scoped key.</param>
|
||||
/// <param name="shardCount">K.</param>
|
||||
/// <returns>The shard index.</returns>
|
||||
internal static int ShardIndex(string scopedKey, int shardCount)
|
||||
{
|
||||
var hash = 2166136261u;
|
||||
foreach (var b in Encoding.UTF8.GetBytes(scopedKey))
|
||||
{
|
||||
hash ^= b;
|
||||
hash *= 16777619u;
|
||||
}
|
||||
return (int)(hash % (uint)shardCount);
|
||||
}
|
||||
|
||||
/// <summary>The default worker path, or the configured one.</summary>
|
||||
/// <param name="worker">The worker options.</param>
|
||||
/// <returns>The absolute worker path.</returns>
|
||||
public static string ResolveWorkerPath(ScriptWorkerOptions worker)
|
||||
=> string.IsNullOrWhiteSpace(worker.ExecutablePath)
|
||||
? Path.Combine(AppContext.BaseDirectory, WorkerFolderName, WorkerAssemblyFileName)
|
||||
: Path.GetFullPath(worker.ExecutablePath);
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(invocation);
|
||||
if (Volatile.Read(ref _disposed) != 0) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.RunnerUnavailableReason);
|
||||
|
||||
var key = InProcessScriptRunner.ScopedKey(invocation.Kind, invocation.Key);
|
||||
var state = Admit(key, invocation.Source);
|
||||
if (state.QuarantineReason is { } quarantined) return ScriptRunOutcome.Failed(quarantined);
|
||||
|
||||
var sha = SHA256.HashData(Encoding.UTF8.GetBytes(invocation.Source));
|
||||
var failureKey = InProcessScriptRunner.ScopedKey(invocation.Kind, Convert.ToHexString(sha));
|
||||
if (_compileFailures.TryGetValue(failureKey, out var knownFailure)) return ScriptRunOutcome.Failed(knownFailure);
|
||||
|
||||
if (Interlocked.CompareExchange(ref state.InFlight, 1, 0) != 0)
|
||||
{
|
||||
ScriptRunGuard.RefusedCounter.Add(1, new KeyValuePair<string, object?>("script", key));
|
||||
return ScriptRunOutcome.Failed(ScriptRunGuard.PreviousRunStillRunningReason);
|
||||
}
|
||||
|
||||
var handedOff = false;
|
||||
SemaphoreSlim? gate = null;
|
||||
try
|
||||
{
|
||||
state.ScriptLogger = invocation.ScriptLogger;
|
||||
var shard = state.Suspect ? _isolation : _shards[ShardIndex(key, _shards.Length)];
|
||||
if (shard.IsolationGate is { } isolationGate)
|
||||
{
|
||||
if (!await isolationGate.WaitAsync(_worker.UnavailableGrace, ct).ConfigureAwait(false))
|
||||
return ScriptRunOutcome.Unavailable(ScriptRunOutcome.RunnerUnavailableReason);
|
||||
gate = isolationGate;
|
||||
}
|
||||
|
||||
var connection = await shard.GetReadyAsync(_worker.UnavailableGrace, ct).ConfigureAwait(false);
|
||||
if (connection is null) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.RunnerUnavailableReason);
|
||||
|
||||
var paths = invocation.Inputs.Keys.Order(StringComparer.Ordinal).ToArray();
|
||||
WorkerConnection.LoadEntry? entry;
|
||||
try
|
||||
{
|
||||
entry = await connection
|
||||
.EnsureLoadedAsync(key, WorkerShard.ToWire(invocation.Kind), invocation.Source, sha, paths)
|
||||
.WaitAsync(LoadTimeout, ct).ConfigureAwait(false);
|
||||
}
|
||||
catch (TimeoutException)
|
||||
{
|
||||
// A compile that never finishes is the script's fault (a pathological source): the worker dies, the
|
||||
// key takes a kill strike through the exit handler.
|
||||
_log.Warning("Script {ScriptKey}: compile in ScriptWorker[{Shard}] exceeded {Timeout} s; killing the worker",
|
||||
key, shard.Label, LoadTimeout.TotalSeconds);
|
||||
connection.Kill(WorkerExitReason.KilledLoadTimeout, key);
|
||||
return ScriptRunOutcome.Failed($"compile failure: compile did not finish within {LoadTimeout.TotalSeconds:F0} s");
|
||||
}
|
||||
if (entry?.Outcome is not { } outcome) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason);
|
||||
if (!outcome.Ok)
|
||||
{
|
||||
var reason = outcome.Error?.Kind switch
|
||||
{
|
||||
ScriptError.Types.Kind.Compile => $"compile error: {outcome.Error.Message}",
|
||||
ScriptError.Types.Kind.Sandbox => $"sandbox violation: {outcome.Error.Message}",
|
||||
_ => $"compile failure: {outcome.Error?.Message}",
|
||||
};
|
||||
// Remembered until the next deploy boundary, so a broken script never re-enters Roslyn (design §7).
|
||||
_compileFailures.TryAdd(failureKey, reason);
|
||||
return ScriptRunOutcome.Failed(reason);
|
||||
}
|
||||
|
||||
var pending = connection.Submit(key, BuildRequest(key, entry, invocation));
|
||||
ArmHardDeadline(connection, pending, invocation.Timeout + _worker.EffectiveHardGrace(invocation.Timeout));
|
||||
|
||||
// From here the in-flight slot (and the isolation gate) belong to the evaluation itself: they are released
|
||||
// when the worker answers or dies — not when this caller stops waiting at the soft deadline. That keeps a
|
||||
// key single-flight between T and T + G, and the isolation worker strictly one-at-a-time.
|
||||
handedOff = true;
|
||||
var heldGate = gate;
|
||||
_ = pending.Completion.ContinueWith(_ =>
|
||||
{
|
||||
Interlocked.Exchange(ref state.InFlight, 0);
|
||||
heldGate?.Release();
|
||||
}, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
|
||||
|
||||
EvalResult? result;
|
||||
try
|
||||
{
|
||||
result = await pending.Completion.WaitAsync(invocation.Timeout, ct).ConfigureAwait(false);
|
||||
}
|
||||
catch (TimeoutException)
|
||||
{
|
||||
// As in process: the run token is cancelled only once the caller has its timeout, so a polling script
|
||||
// ends itself (and frees its slot) without the worker being killed.
|
||||
connection.Cancel(pending);
|
||||
if (ct.IsCancellationRequested) return ScriptRunOutcome.Failed("evaluation cancelled");
|
||||
if (NoteTimeout(key, state, invocation.Timeout)) return ScriptRunOutcome.Failed(state.QuarantineReason!);
|
||||
return ScriptRunOutcome.Failed($"{ScriptEvalFailureReasons.TimedOutPrefix}{invocation.Timeout.TotalSeconds:F1}s");
|
||||
}
|
||||
|
||||
if (result is null) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason);
|
||||
return Complete(key, state, result);
|
||||
}
|
||||
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
||||
{
|
||||
return ScriptRunOutcome.Failed("evaluation cancelled");
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (!handedOff)
|
||||
{
|
||||
Interlocked.Exchange(ref state.InFlight, 0);
|
||||
gate?.Release();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Reset(ScriptContextKind kind)
|
||||
{
|
||||
var prefix = InProcessScriptRunner.ScopedKey(kind, string.Empty);
|
||||
var dropped = new List<string>();
|
||||
foreach (var key in _keys.Keys)
|
||||
{
|
||||
if (key.StartsWith(prefix, StringComparison.Ordinal) && _keys.TryRemove(key, out _)) dropped.Add(key);
|
||||
}
|
||||
foreach (var key in _compileFailures.Keys)
|
||||
{
|
||||
if (key.StartsWith(prefix, StringComparison.Ordinal)) _compileFailures.TryRemove(key, out _);
|
||||
}
|
||||
foreach (var shard in _shards.Append(_isolation))
|
||||
shard.Current?.Unload(dropped);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind)
|
||||
{
|
||||
var prefix = InProcessScriptRunner.ScopedKey(kind, string.Empty);
|
||||
return _keys
|
||||
.Where(kv => kv.Value.QuarantineReason is not null && kv.Key.StartsWith(prefix, StringComparison.Ordinal))
|
||||
.Select(kv => kv.Key[prefix.Length..])
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
/// <summary>Stops every worker.</summary>
|
||||
/// <returns>A task that completes when they have ended.</returns>
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
if (Interlocked.Exchange(ref _disposed, 1) != 0) return;
|
||||
await Task.WhenAll(_shards.Append(_isolation).Select(s => s.DisposeAsync().AsTask())).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>Stops every worker (blocking).</summary>
|
||||
public void Dispose() => DisposeAsync().AsTask().GetAwaiter().GetResult();
|
||||
|
||||
private WorkerShard NewShard(string label, bool isolation, int evalThreads, TimeSpan heartbeatInterval)
|
||||
{
|
||||
var spec = new WorkerLaunchSpec(
|
||||
Shard: label,
|
||||
WorkerPath: WorkerPath,
|
||||
HeapLimitBytes: _worker.HeapLimitBytes,
|
||||
EvalThreads: evalThreads,
|
||||
HardGrace: _worker.HardGrace ?? TimeSpan.FromSeconds(Math.Max(1, _options.RunTimeout.TotalSeconds)),
|
||||
HeartbeatInterval: heartbeatInterval,
|
||||
HandshakeTimeout: TimeSpan.FromSeconds(15));
|
||||
return new WorkerShard(label, isolation,
|
||||
ct => WorkerConnection.StartAsync(spec, _log, OnLogs, ct),
|
||||
OnWorkerExit, _log);
|
||||
}
|
||||
|
||||
private KeyState Admit(string key, string source)
|
||||
{
|
||||
while (true)
|
||||
{
|
||||
var state = _keys.GetOrAdd(key, static (_, src) => new KeyState(src), source);
|
||||
if (string.Equals(state.Source, source, StringComparison.Ordinal)) return state;
|
||||
// The script changed under this key: a clean slate, exactly as the in-process guard does. An evaluation
|
||||
// of the old source still in flight releases the OLD state's slot when it ends.
|
||||
if (_keys.TryUpdate(key, new KeyState(source), state)) continue;
|
||||
}
|
||||
}
|
||||
|
||||
private EvalRequest BuildRequest(string key, WorkerConnection.LoadEntry entry, ScriptInvocation invocation)
|
||||
{
|
||||
var request = new EvalRequest
|
||||
{
|
||||
Key = key,
|
||||
NowUtcTicks = DateTime.UtcNow.Ticks,
|
||||
SoftTimeoutMs = (uint)Math.Clamp(invocation.Timeout.TotalMilliseconds, 1, uint.MaxValue),
|
||||
HardGraceMs = (uint)Math.Clamp(_worker.EffectiveHardGrace(invocation.Timeout).TotalMilliseconds, 1, uint.MaxValue),
|
||||
};
|
||||
foreach (var (path, snapshot) in invocation.Inputs)
|
||||
{
|
||||
if (snapshot is null) continue;
|
||||
var input = new Input
|
||||
{
|
||||
StatusCode = snapshot.StatusCode,
|
||||
SourceTsTicks = snapshot.SourceTimestampUtc is { } source ? VariantCodec.ToUtc(source).Ticks : 0,
|
||||
ServerTsTicks = VariantCodec.ToUtc(snapshot.ServerTimestampUtc).Ticks,
|
||||
};
|
||||
if (entry.Slots.TryGetValue(path, out var slot)) input.Slot = slot;
|
||||
else input.Path = path;
|
||||
if (VariantCodec.TryEncode(snapshot.Value, out var variant, out var unsupported))
|
||||
{
|
||||
input.Value = variant;
|
||||
}
|
||||
else
|
||||
{
|
||||
// Never silently converted (design §14 #11): the worker answers UNSUPPORTED_TYPE for this input.
|
||||
input.Unsupported = true;
|
||||
input.UnsupportedType = unsupported ?? "?";
|
||||
}
|
||||
request.Inputs.Add(input);
|
||||
}
|
||||
return request;
|
||||
}
|
||||
|
||||
private void ArmHardDeadline(WorkerConnection connection, PendingEval pending, TimeSpan after)
|
||||
{
|
||||
var cts = new CancellationTokenSource(after);
|
||||
var registration = cts.Token.Register(() =>
|
||||
{
|
||||
if (pending.Completion.IsCompleted) return;
|
||||
_log.Warning("Script {ScriptKey} exceeded its hard deadline ({Deadline} ms); killing ScriptWorker[{Shard}] pid {Pid}",
|
||||
pending.Key, (long)after.TotalMilliseconds, connection.Shard, connection.Pid);
|
||||
connection.Kill(WorkerExitReason.KilledDeadline, pending.Key);
|
||||
});
|
||||
_ = pending.Completion.ContinueWith(_ =>
|
||||
{
|
||||
registration.Dispose();
|
||||
cts.Dispose();
|
||||
}, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
|
||||
}
|
||||
|
||||
private ScriptRunOutcome Complete(string key, KeyState state, EvalResult result)
|
||||
{
|
||||
// The script returned control (a value or a throw): the soft-timeout streak resets, as in process.
|
||||
Interlocked.Exchange(ref state.ConsecutiveTimeouts, 0);
|
||||
|
||||
if (_options.MaxAllocBytesPerRun > 0 && (long)result.AllocatedBytes > _options.MaxAllocBytesPerRun)
|
||||
{
|
||||
Quarantine(key, state,
|
||||
$"one evaluation allocated {result.AllocatedBytes / (1024.0 * 1024.0):F1} MB, over the " +
|
||||
$"{_options.MaxAllocBytesPerRun / (1024.0 * 1024.0):F0} MB budget (Scripting:MaxAllocBytesPerRun); " +
|
||||
"the script is not evaluated again until its source changes or it is redeployed",
|
||||
"allocation");
|
||||
return ScriptRunOutcome.Failed(state.QuarantineReason!);
|
||||
}
|
||||
|
||||
if (state.Suspect && Interlocked.Increment(ref state.ProbationRuns) >= _worker.IsolationProbationRuns)
|
||||
{
|
||||
state.Suspect = false;
|
||||
_log.Information("Script {ScriptKey} completed {Runs} clean run(s) in the isolation worker; returning it to its shard",
|
||||
key, _worker.IsolationProbationRuns);
|
||||
}
|
||||
|
||||
switch (result.OutcomeCase)
|
||||
{
|
||||
case EvalResult.OutcomeOneofCase.Value:
|
||||
try
|
||||
{
|
||||
return ScriptRunOutcome.Ok(VariantCodec.Decode(result.Value));
|
||||
}
|
||||
catch (ScriptWorkerProtocolException ex)
|
||||
{
|
||||
return ScriptRunOutcome.Failed($"script worker returned an undecodable value: {ex.Message}");
|
||||
}
|
||||
case EvalResult.OutcomeOneofCase.Predicate:
|
||||
return ScriptRunOutcome.Ok(result.Predicate);
|
||||
case EvalResult.OutcomeOneofCase.Error:
|
||||
return result.Error.Kind switch
|
||||
{
|
||||
ScriptError.Types.Kind.Threw => ScriptRunOutcome.Failed($"script threw: {result.Error.Message}"),
|
||||
ScriptError.Types.Kind.UnsupportedType => ScriptRunOutcome.Failed($"unsupported type: {result.Error.Message}"),
|
||||
ScriptError.Types.Kind.Busy => ScriptRunOutcome.Failed(WorkerBusyReason),
|
||||
ScriptError.Types.Kind.NotLoaded => ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason),
|
||||
_ => ScriptRunOutcome.Failed($"script failed: {result.Error.Message}"),
|
||||
};
|
||||
default:
|
||||
return ScriptRunOutcome.Ok(null);
|
||||
}
|
||||
}
|
||||
|
||||
private bool NoteTimeout(string key, KeyState state, TimeSpan timeout)
|
||||
{
|
||||
var count = Interlocked.Increment(ref state.ConsecutiveTimeouts);
|
||||
if (count < _options.QuarantineAfterConsecutiveTimeouts) return false;
|
||||
Quarantine(key, state,
|
||||
$"{count} consecutive evaluation(s) timed out (budget {timeout.TotalMilliseconds:F0} ms); " +
|
||||
"the script is not evaluated again until its source changes or it is redeployed",
|
||||
"timeouts");
|
||||
return true;
|
||||
}
|
||||
|
||||
private void OnWorkerExit(WorkerShard shard, WorkerExit exit)
|
||||
{
|
||||
switch (exit.Reason)
|
||||
{
|
||||
case WorkerExitReason.KilledDeadline when exit.CulpritKey is { } culprit:
|
||||
KillStrike(culprit, "it exceeded its hard deadline and its worker was killed", "killed");
|
||||
break;
|
||||
case WorkerExitReason.KilledLoadTimeout when exit.CulpritKey is { } culprit:
|
||||
KillStrike(culprit, "its compile did not finish and its worker was killed", "killed");
|
||||
break;
|
||||
case WorkerExitReason.Crash or WorkerExitReason.KilledHeartbeat or WorkerExitReason.ProtocolFault:
|
||||
if (shard.IsIsolation)
|
||||
{
|
||||
// One evaluation at a time here: the crash is this key's (design §5.3.3).
|
||||
foreach (var key in exit.InFlightKeys)
|
||||
KillStrike(key, "it crashed the isolation worker", "isolated_crash");
|
||||
}
|
||||
else if (exit.InFlightKeys.Count > 0)
|
||||
{
|
||||
foreach (var key in exit.InFlightKeys)
|
||||
{
|
||||
if (!_keys.TryGetValue(key, out var state)) continue;
|
||||
state.Suspect = true;
|
||||
Interlocked.Exchange(ref state.ProbationRuns, 0);
|
||||
}
|
||||
_log.Warning("ScriptWorker[{Shard}] ended ({Reason}) with no deadline crossed; {Count} script(s) in flight move to the isolation worker: {Suspects}",
|
||||
shard.Label, exit.Reason, exit.InFlightKeys.Count, string.Join(", ", exit.InFlightKeys));
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
private void KillStrike(string key, string why, string metricReason)
|
||||
{
|
||||
if (!_keys.TryGetValue(key, out var state)) return;
|
||||
int strikes;
|
||||
lock (state.KillStrikes)
|
||||
{
|
||||
var now = DateTime.UtcNow;
|
||||
state.KillStrikes.Add(now);
|
||||
state.KillStrikes.RemoveAll(t => now - t > KillStrikeWindow);
|
||||
strikes = state.KillStrikes.Count;
|
||||
}
|
||||
if (strikes >= _worker.QuarantineAfterKills)
|
||||
{
|
||||
Quarantine(key, state,
|
||||
$"{why} ({strikes} time(s) in the last hour); the script is not evaluated again until its source changes or it is redeployed",
|
||||
metricReason);
|
||||
}
|
||||
}
|
||||
|
||||
private void Quarantine(string key, KeyState state, string why, string metricReason)
|
||||
{
|
||||
var reason = ScriptRunGuard.QuarantinedReasonPrefix + why;
|
||||
if (Interlocked.CompareExchange(ref state.QuarantineReason, reason, null) is not null) return;
|
||||
ScriptRunGuard.QuarantinedCounter.Add(1,
|
||||
new KeyValuePair<string, object?>("script", key),
|
||||
new KeyValuePair<string, object?>("reason", metricReason));
|
||||
_scriptRoot.ForContext(ScriptLoggerFactory.ScriptNameProperty, key).Error("Script {ScriptKey} quarantined: {Reason}", key, why);
|
||||
}
|
||||
|
||||
private void OnLogs(LogBatch batch)
|
||||
{
|
||||
foreach (var line in batch.Lines)
|
||||
{
|
||||
var logger = _keys.TryGetValue(line.Key, out var state) && state.ScriptLogger is { } scoped ? scoped : _scriptRoot;
|
||||
var level = line.Level <= (uint)LogEventLevel.Fatal ? (LogEventLevel)line.Level : LogEventLevel.Information;
|
||||
if (string.IsNullOrEmpty(line.Exception))
|
||||
logger.Write(level, "{ScriptMessage:l}", line.Rendered);
|
||||
else
|
||||
logger.Write(level, "{ScriptMessage:l}{NewLine}{ScriptException:l}", line.Rendered, Environment.NewLine, line.Exception);
|
||||
}
|
||||
if (batch.Dropped > 0)
|
||||
_log.Warning("ScriptWorker dropped {Dropped} script log line(s): a script is logging faster than the pipe drains", batch.Dropped);
|
||||
}
|
||||
|
||||
private sealed class KeyState(string source)
|
||||
{
|
||||
public readonly string Source = source;
|
||||
public readonly List<DateTime> KillStrikes = [];
|
||||
public int InFlight;
|
||||
public int ConsecutiveTimeouts;
|
||||
public int ProbationRuns;
|
||||
public volatile bool Suspect;
|
||||
public string? QuarantineReason;
|
||||
public ILogger? ScriptLogger;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
using System.ComponentModel;
|
||||
using System.Diagnostics;
|
||||
using System.Runtime.InteropServices;
|
||||
using System.Runtime.Versioning;
|
||||
using Microsoft.Win32.SafeHandles;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
|
||||
|
||||
/// <summary>
|
||||
/// A Windows Job Object holding one ScriptWorker (design §3.3, §5.2): <c>KILL_ON_JOB_CLOSE</c> so the kernel kills
|
||||
/// the worker when the host dies (its handle closes with it), <c>ACTIVE_PROCESS = 1</c> so the worker can start no
|
||||
/// child process, and a process-memory backstop slightly above the managed heap limit for native and JIT memory.
|
||||
/// </summary>
|
||||
[SupportedOSPlatform("windows")]
|
||||
internal sealed class WindowsJobObject : IDisposable
|
||||
{
|
||||
private const int JobObjectExtendedLimitInformation = 9;
|
||||
private const uint JOB_OBJECT_LIMIT_ACTIVE_PROCESS = 0x00000008;
|
||||
private const uint JOB_OBJECT_LIMIT_PROCESS_MEMORY = 0x00000100;
|
||||
private const uint JOB_OBJECT_LIMIT_DIE_ON_UNHANDLED_EXCEPTION = 0x00000400;
|
||||
private const uint JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE = 0x00002000;
|
||||
|
||||
private readonly SafeFileHandle _handle;
|
||||
|
||||
private WindowsJobObject(SafeFileHandle handle) => _handle = handle;
|
||||
|
||||
/// <summary>Creates a job with the worker limits and assigns <paramref name="process"/> to it.</summary>
|
||||
/// <param name="process">The freshly started worker.</param>
|
||||
/// <param name="processMemoryLimitBytes">The per-process commit limit.</param>
|
||||
/// <returns>The job; dispose it to kill the worker.</returns>
|
||||
public static WindowsJobObject Assign(Process process, long processMemoryLimitBytes)
|
||||
{
|
||||
var handle = CreateJobObjectW(IntPtr.Zero, null);
|
||||
if (handle.IsInvalid) throw new Win32Exception(Marshal.GetLastWin32Error(), "CreateJobObject failed");
|
||||
var job = new WindowsJobObject(handle);
|
||||
try
|
||||
{
|
||||
var info = new JOBOBJECT_EXTENDED_LIMIT_INFORMATION
|
||||
{
|
||||
BasicLimitInformation = new JOBOBJECT_BASIC_LIMIT_INFORMATION
|
||||
{
|
||||
LimitFlags = JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE | JOB_OBJECT_LIMIT_ACTIVE_PROCESS
|
||||
| JOB_OBJECT_LIMIT_PROCESS_MEMORY | JOB_OBJECT_LIMIT_DIE_ON_UNHANDLED_EXCEPTION,
|
||||
ActiveProcessLimit = 1,
|
||||
},
|
||||
ProcessMemoryLimit = (UIntPtr)(ulong)processMemoryLimitBytes,
|
||||
};
|
||||
if (!SetInformationJobObject(handle, JobObjectExtendedLimitInformation, ref info,
|
||||
(uint)Marshal.SizeOf<JOBOBJECT_EXTENDED_LIMIT_INFORMATION>()))
|
||||
throw new Win32Exception(Marshal.GetLastWin32Error(), "SetInformationJobObject failed");
|
||||
if (!AssignProcessToJobObject(handle, process.Handle))
|
||||
throw new Win32Exception(Marshal.GetLastWin32Error(), "AssignProcessToJobObject failed");
|
||||
return job;
|
||||
}
|
||||
catch
|
||||
{
|
||||
job.Dispose();
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Closes the job handle, which kills every process still in it.</summary>
|
||||
public void Dispose() => _handle.Dispose();
|
||||
|
||||
[StructLayout(LayoutKind.Sequential)]
|
||||
private struct JOBOBJECT_BASIC_LIMIT_INFORMATION
|
||||
{
|
||||
public long PerProcessUserTimeLimit;
|
||||
public long PerJobUserTimeLimit;
|
||||
public uint LimitFlags;
|
||||
public UIntPtr MinimumWorkingSetSize;
|
||||
public UIntPtr MaximumWorkingSetSize;
|
||||
public uint ActiveProcessLimit;
|
||||
public UIntPtr Affinity;
|
||||
public uint PriorityClass;
|
||||
public uint SchedulingClass;
|
||||
}
|
||||
|
||||
[StructLayout(LayoutKind.Sequential)]
|
||||
private struct IO_COUNTERS
|
||||
{
|
||||
public ulong ReadOperationCount;
|
||||
public ulong WriteOperationCount;
|
||||
public ulong OtherOperationCount;
|
||||
public ulong ReadTransferCount;
|
||||
public ulong WriteTransferCount;
|
||||
public ulong OtherTransferCount;
|
||||
}
|
||||
|
||||
[StructLayout(LayoutKind.Sequential)]
|
||||
private struct JOBOBJECT_EXTENDED_LIMIT_INFORMATION
|
||||
{
|
||||
public JOBOBJECT_BASIC_LIMIT_INFORMATION BasicLimitInformation;
|
||||
public IO_COUNTERS IoInfo;
|
||||
public UIntPtr ProcessMemoryLimit;
|
||||
public UIntPtr JobMemoryLimit;
|
||||
public UIntPtr PeakProcessMemoryUsed;
|
||||
public UIntPtr PeakJobMemoryUsed;
|
||||
}
|
||||
|
||||
#pragma warning disable SYSLIB1054 // DllImport kept: LibraryImport would need AllowUnsafeBlocks for three calls.
|
||||
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
|
||||
private static extern SafeFileHandle CreateJobObjectW(IntPtr securityAttributes, string? name);
|
||||
|
||||
[DllImport("kernel32.dll", SetLastError = true)]
|
||||
[return: MarshalAs(UnmanagedType.Bool)]
|
||||
private static extern bool SetInformationJobObject(SafeFileHandle job, int infoClass,
|
||||
ref JOBOBJECT_EXTENDED_LIMIT_INFORMATION info, uint length);
|
||||
|
||||
[DllImport("kernel32.dll", SetLastError = true)]
|
||||
[return: MarshalAs(UnmanagedType.Bool)]
|
||||
private static extern bool AssignProcessToJobObject(SafeFileHandle job, IntPtr process);
|
||||
#pragma warning restore SYSLIB1054
|
||||
}
|
||||
@@ -0,0 +1,699 @@
|
||||
using System.Collections.Concurrent;
|
||||
using System.Diagnostics;
|
||||
using System.IO.Pipes;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
using System.Threading.Channels;
|
||||
using Serilog;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
|
||||
|
||||
/// <summary>How a worker process ended.</summary>
|
||||
internal enum WorkerExitReason
|
||||
{
|
||||
/// <summary>It exited by itself with code 0 (a Shutdown).</summary>
|
||||
Clean,
|
||||
|
||||
/// <summary>The host killed it: a script crossed its hard deadline (<see cref="WorkerExit.CulpritKey"/>).</summary>
|
||||
KilledDeadline,
|
||||
|
||||
/// <summary>The host killed it: no frame for three heartbeat intervals.</summary>
|
||||
KilledHeartbeat,
|
||||
|
||||
/// <summary>The host killed it: a compile did not finish in time (<see cref="WorkerExit.CulpritKey"/>).</summary>
|
||||
KilledLoadTimeout,
|
||||
|
||||
/// <summary>It died by itself (OOM, heap limit, fail-fast, external kill).</summary>
|
||||
Crash,
|
||||
|
||||
/// <summary>A framing or handshake violation.</summary>
|
||||
ProtocolFault,
|
||||
|
||||
/// <summary>The runner is shutting down.</summary>
|
||||
Disposed,
|
||||
}
|
||||
|
||||
/// <summary>The end of one worker process.</summary>
|
||||
/// <param name="Reason">Why it ended.</param>
|
||||
/// <param name="CulpritKey">The scoped key the host blamed, for a deadline or load kill.</param>
|
||||
/// <param name="ExitCode">The OS exit code, when known.</param>
|
||||
/// <param name="InFlightKeys">The scoped keys that had an evaluation outstanding when it ended.</param>
|
||||
internal sealed record WorkerExit(WorkerExitReason Reason, string? CulpritKey, int? ExitCode, IReadOnlyList<string> InFlightKeys);
|
||||
|
||||
/// <summary>What the host needs to launch one worker.</summary>
|
||||
/// <param name="Shard">The shard label (<c>0</c>, <c>1</c>, … or <c>iso</c>).</param>
|
||||
/// <param name="WorkerPath">The worker entry assembly (<c>.dll</c>, run through the dotnet muxer) or apphost.</param>
|
||||
/// <param name="HeapLimitBytes">The worker's <c>DOTNET_GCHeapHardLimit</c>.</param>
|
||||
/// <param name="EvalThreads">The worker scheduler's width (1 for the isolation worker).</param>
|
||||
/// <param name="HardGrace">The worker watchdog's grace past the soft deadline.</param>
|
||||
/// <param name="HeartbeatInterval">How often the worker sends a heartbeat.</param>
|
||||
/// <param name="HandshakeTimeout">How long a launch may take to reach Ready.</param>
|
||||
internal sealed record WorkerLaunchSpec(
|
||||
string Shard,
|
||||
string WorkerPath,
|
||||
long HeapLimitBytes,
|
||||
int EvalThreads,
|
||||
TimeSpan HardGrace,
|
||||
TimeSpan HeartbeatInterval,
|
||||
TimeSpan HandshakeTimeout);
|
||||
|
||||
/// <summary>
|
||||
/// One live ScriptWorker process and its pipe (lmxopcua#780, design §3–§6): launch with a cleared, allow-listed
|
||||
/// environment, a random pipe that accepts exactly one client, a nonce handshake, a frame reader and a coalescing
|
||||
/// writer, heartbeat supervision, and the kill. It never restarts itself; <see cref="WorkerShard"/> does that.
|
||||
/// </summary>
|
||||
internal sealed class WorkerConnection : IAsyncDisposable
|
||||
{
|
||||
/// <summary>
|
||||
/// Environment variables a worker inherits from the host (design §6): the runtime's own knobs, the temp
|
||||
/// directories and the minimum a Windows process needs. Everything else — notably <c>ServerHistorian__ApiKey</c>,
|
||||
/// ConfigDb connection strings and every other <c>Section__Key</c> override — is dropped.
|
||||
/// </summary>
|
||||
internal static readonly IReadOnlyList<string> InheritedEnvironment =
|
||||
[
|
||||
"PATH", "TEMP", "TMP", "TMPDIR", "HOME", "SystemRoot", "SYSTEMROOT", "windir", "SystemDrive", "ComSpec",
|
||||
"PATHEXT", "NUMBER_OF_PROCESSORS", "PROCESSOR_ARCHITECTURE", "LOCALAPPDATA",
|
||||
];
|
||||
|
||||
private readonly Process _process;
|
||||
private readonly NamedPipeServerStream _pipe;
|
||||
private readonly IDisposable? _job;
|
||||
private readonly string _workDir;
|
||||
private readonly ILogger _log;
|
||||
private readonly Action<LogBatch> _onLogs;
|
||||
private readonly Channel<Envelope> _outbound = Channel.CreateUnbounded<Envelope>(new UnboundedChannelOptions { SingleReader = true });
|
||||
private readonly ConcurrentDictionary<ulong, PendingEval> _pending = new();
|
||||
private readonly ConcurrentDictionary<ulong, TaskCompletionSource<LoadResult>> _loads = new();
|
||||
private readonly ConcurrentDictionary<string, LoadEntry> _loaded = new(StringComparer.Ordinal);
|
||||
private readonly TaskCompletionSource<WorkerExit> _exited = new(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
private readonly CancellationTokenSource _stopping = new();
|
||||
private readonly TimeSpan _heartbeatInterval;
|
||||
private TaskCompletionSource<DiagnosticsReply>? _diag;
|
||||
private (WorkerExitReason Reason, string? Culprit)? _killedFor;
|
||||
private long _lastFrameTimestamp;
|
||||
private long _seq;
|
||||
private long _nextEvalId;
|
||||
private long _nextLoadId;
|
||||
private int _maxFrameBytes = ScriptWorkerFrameCodec.DefaultMaxFrameBytes;
|
||||
private int _disposed;
|
||||
|
||||
private WorkerConnection(string shard, Process process, NamedPipeServerStream pipe, IDisposable? job, string workDir,
|
||||
ILogger log, Action<LogBatch> onLogs, TimeSpan heartbeatInterval)
|
||||
{
|
||||
Shard = shard;
|
||||
_process = process;
|
||||
_pipe = pipe;
|
||||
_job = job;
|
||||
_workDir = workDir;
|
||||
_log = log;
|
||||
_onLogs = onLogs;
|
||||
_heartbeatInterval = heartbeatInterval;
|
||||
Pid = process.Id;
|
||||
}
|
||||
|
||||
/// <summary>The shard label.</summary>
|
||||
public string Shard { get; }
|
||||
|
||||
/// <summary>The worker's process id.</summary>
|
||||
public int Pid { get; }
|
||||
|
||||
/// <summary>Completes when the worker has ended, with why.</summary>
|
||||
public Task<WorkerExit> Exited => _exited.Task;
|
||||
|
||||
/// <summary>Whether the worker is still serving.</summary>
|
||||
public bool IsAlive => !_exited.Task.IsCompleted && !_stopping.IsCancellationRequested;
|
||||
|
||||
/// <summary><see cref="Stopwatch.GetTimestamp"/> when the handshake completed.</summary>
|
||||
public long ReadyTimestamp { get; private set; }
|
||||
|
||||
/// <summary>Evaluations currently outstanding on this worker.</summary>
|
||||
public int InFlightCount => _pending.Count;
|
||||
|
||||
/// <summary>Launches a worker and completes the handshake.</summary>
|
||||
/// <param name="spec">What to launch.</param>
|
||||
/// <param name="log">Host diagnostics (stderr lines, exits).</param>
|
||||
/// <param name="onLogs">Receives the worker's forwarded <c>ctx.Logger</c> batches.</param>
|
||||
/// <param name="ct">Cancellation.</param>
|
||||
/// <returns>A ready connection.</returns>
|
||||
public static async Task<WorkerConnection> StartAsync(WorkerLaunchSpec spec, ILogger log, Action<LogBatch> onLogs, CancellationToken ct)
|
||||
{
|
||||
var pipeName = "otopcua-sw-" + Convert.ToHexString(RandomNumberGenerator.GetBytes(12)).ToLowerInvariant();
|
||||
var nonce = Convert.ToHexString(RandomNumberGenerator.GetBytes(32));
|
||||
var workDir = Path.Combine(Path.GetTempPath(), "otopcua-scriptworker", pipeName);
|
||||
Directory.CreateDirectory(workDir);
|
||||
|
||||
// One instance, current user only: the pipe accepts exactly one connection, from this account.
|
||||
var pipe = new NamedPipeServerStream(pipeName, PipeDirection.InOut, maxNumberOfServerInstances: 1,
|
||||
PipeTransmissionMode.Byte, PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly);
|
||||
Process? process = null;
|
||||
IDisposable? job = null;
|
||||
try
|
||||
{
|
||||
var psi = BuildStartInfo(spec, pipeName, nonce, workDir);
|
||||
process = new Process { StartInfo = psi, EnableRaisingEvents = true };
|
||||
if (!process.Start()) throw new InvalidOperationException("the worker process did not start");
|
||||
if (OperatingSystem.IsWindows())
|
||||
{
|
||||
// Native and JIT memory backstop above the managed heap limit.
|
||||
job = WindowsJobObject.Assign(process, spec.HeapLimitBytes + 256L * 1024 * 1024);
|
||||
}
|
||||
var shardLabel = spec.Shard;
|
||||
process.ErrorDataReceived += (_, e) =>
|
||||
{
|
||||
if (e.Data is { Length: > 0 } line) log.Warning("ScriptWorker[{Shard}] {WorkerStderr}", shardLabel, line);
|
||||
};
|
||||
process.OutputDataReceived += (_, e) =>
|
||||
{
|
||||
if (e.Data is { Length: > 0 } line) log.Debug("ScriptWorker[{Shard}] stdout {WorkerStdout}", shardLabel, line);
|
||||
};
|
||||
process.BeginErrorReadLine();
|
||||
process.BeginOutputReadLine();
|
||||
|
||||
using var handshakeCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
|
||||
handshakeCts.CancelAfter(spec.HandshakeTimeout);
|
||||
var exitedEarly = process.WaitForExitAsync(handshakeCts.Token);
|
||||
var accept = pipe.WaitForConnectionAsync(handshakeCts.Token);
|
||||
if (await Task.WhenAny(accept, exitedEarly).ConfigureAwait(false) != accept)
|
||||
throw new InvalidOperationException(
|
||||
$"the worker exited (code {SafeExitCode(process)}) before connecting to its pipe");
|
||||
await accept.ConfigureAwait(false);
|
||||
|
||||
var hello = new Hello
|
||||
{
|
||||
Nonce = nonce,
|
||||
MaxFrameBytes = ScriptWorkerFrameCodec.DefaultMaxFrameBytes,
|
||||
EvalThreads = (uint)spec.EvalThreads,
|
||||
QueueCapacity = 4096,
|
||||
HardGraceMs = (uint)Math.Clamp(spec.HardGrace.TotalMilliseconds, 1, uint.MaxValue),
|
||||
HeartbeatIntervalMs = (uint)spec.HeartbeatInterval.TotalMilliseconds,
|
||||
MaxLogLines = 1000,
|
||||
};
|
||||
await HandshakeAsync(pipe, hello, nonce, process.Id, handshakeCts.Token).ConfigureAwait(false);
|
||||
|
||||
var connection = new WorkerConnection(spec.Shard, process, pipe, job, workDir, log, onLogs, spec.HeartbeatInterval);
|
||||
connection.Begin();
|
||||
return connection;
|
||||
}
|
||||
catch
|
||||
{
|
||||
try { if (process is { HasExited: false }) process.Kill(entireProcessTree: true); } catch { /* best effort */ }
|
||||
job?.Dispose();
|
||||
process?.Dispose();
|
||||
await pipe.DisposeAsync().ConfigureAwait(false);
|
||||
TryDeleteDirectory(workDir);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The host half of the handshake: send <paramref name="hello"/>, require a <c>Ready</c> as the first frame,
|
||||
/// echoing <paramref name="expectedNonce"/> (constant-time compare) from <paramref name="expectedPid"/>.
|
||||
/// </summary>
|
||||
/// <param name="stream">The accepted pipe.</param>
|
||||
/// <param name="hello">The Hello to send.</param>
|
||||
/// <param name="expectedNonce">The nonce put in the worker's environment.</param>
|
||||
/// <param name="expectedPid">The launched process id; null skips the check.</param>
|
||||
/// <param name="ct">Cancellation (the handshake timeout).</param>
|
||||
/// <returns>The worker's Ready.</returns>
|
||||
/// <exception cref="ScriptWorkerProtocolException">Anything but a matching Ready.</exception>
|
||||
internal static async Task<Ready> HandshakeAsync(Stream stream, Hello hello, string expectedNonce, int? expectedPid, CancellationToken ct)
|
||||
{
|
||||
await ScriptWorkerFrameCodec.WriteAsync(stream, new Envelope
|
||||
{
|
||||
ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion,
|
||||
Seq = 1,
|
||||
Hello = hello,
|
||||
}, ScriptWorkerFrameCodec.DefaultMaxFrameBytes, ct).ConfigureAwait(false);
|
||||
await stream.FlushAsync(ct).ConfigureAwait(false);
|
||||
|
||||
var reply = await ScriptWorkerFrameCodec.ReadAsync(stream, ScriptWorkerFrameCodec.DefaultMaxFrameBytes, ct).ConfigureAwait(false)
|
||||
?? throw new ScriptWorkerProtocolException("the worker closed the pipe during the handshake");
|
||||
if (reply.BodyCase != Envelope.BodyOneofCase.Ready)
|
||||
throw new ScriptWorkerProtocolException($"expected Ready, got {reply.BodyCase}");
|
||||
var expected = Encoding.UTF8.GetBytes(expectedNonce);
|
||||
var actual = Encoding.UTF8.GetBytes(reply.Ready.Nonce ?? string.Empty);
|
||||
if (!CryptographicOperations.FixedTimeEquals(expected, actual))
|
||||
throw new ScriptWorkerProtocolException("the worker's nonce does not match the launch nonce");
|
||||
if (expectedPid is { } pid && reply.Ready.Pid != pid)
|
||||
throw new ScriptWorkerProtocolException($"the pipe client reports pid {reply.Ready.Pid}, not the launched {pid}");
|
||||
return reply.Ready;
|
||||
}
|
||||
|
||||
private static ProcessStartInfo BuildStartInfo(WorkerLaunchSpec spec, string pipeName, string nonce, string workDir)
|
||||
{
|
||||
var psi = new ProcessStartInfo
|
||||
{
|
||||
UseShellExecute = false,
|
||||
CreateNoWindow = true,
|
||||
RedirectStandardError = true,
|
||||
RedirectStandardOutput = true,
|
||||
RedirectStandardInput = false,
|
||||
WorkingDirectory = workDir,
|
||||
};
|
||||
if (spec.WorkerPath.EndsWith(".dll", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
psi.FileName = ResolveMuxer();
|
||||
psi.ArgumentList.Add("exec");
|
||||
psi.ArgumentList.Add(spec.WorkerPath);
|
||||
}
|
||||
else
|
||||
{
|
||||
psi.FileName = spec.WorkerPath;
|
||||
}
|
||||
|
||||
// Cleared, then repopulated from an allow-list (design §6). ProcessStartInfo.Environment starts as a copy of
|
||||
// the host's environment; ScadaBridge's sandbox inherits it whole, which this design rejects.
|
||||
var inherited = psi.Environment.ToDictionary(kv => kv.Key, kv => kv.Value, StringComparer.Ordinal);
|
||||
psi.Environment.Clear();
|
||||
foreach (var (name, value) in inherited)
|
||||
{
|
||||
if (value is null) continue;
|
||||
if (InheritedEnvironment.Contains(name, StringComparer.OrdinalIgnoreCase)
|
||||
|| (name.StartsWith("DOTNET_", StringComparison.OrdinalIgnoreCase)
|
||||
&& !name.StartsWith("DOTNET_GCHeap", StringComparison.OrdinalIgnoreCase)))
|
||||
psi.Environment[name] = value;
|
||||
}
|
||||
psi.Environment["DOTNET_GCHeapHardLimit"] = "0x" + spec.HeapLimitBytes.ToString("X", System.Globalization.CultureInfo.InvariantCulture);
|
||||
psi.Environment["DOTNET_gcServer"] = "0";
|
||||
psi.Environment["DOTNET_CLI_TELEMETRY_OPTOUT"] = "1";
|
||||
psi.Environment[ScriptWorkerEnvironment.PipeName] = pipeName;
|
||||
psi.Environment[ScriptWorkerEnvironment.Nonce] = nonce;
|
||||
psi.Environment[ScriptWorkerEnvironment.ParentPid] = Environment.ProcessId.ToString(System.Globalization.CultureInfo.InvariantCulture);
|
||||
psi.Environment[ScriptWorkerEnvironment.Shard] = spec.Shard;
|
||||
return psi;
|
||||
}
|
||||
|
||||
/// <summary>The dotnet muxer that runs a framework-dependent worker <c>.dll</c>.</summary>
|
||||
/// <returns>The muxer path, or plain <c>dotnet</c> (resolved on PATH) as a last resort.</returns>
|
||||
internal static string ResolveMuxer()
|
||||
{
|
||||
var hostPath = Environment.GetEnvironmentVariable("DOTNET_HOST_PATH");
|
||||
if (!string.IsNullOrEmpty(hostPath) && File.Exists(hostPath)) return hostPath;
|
||||
var processPath = Environment.ProcessPath;
|
||||
if (processPath is not null
|
||||
&& string.Equals(Path.GetFileNameWithoutExtension(processPath), "dotnet", StringComparison.OrdinalIgnoreCase))
|
||||
return processPath;
|
||||
var root = Environment.GetEnvironmentVariable("DOTNET_ROOT");
|
||||
if (!string.IsNullOrEmpty(root))
|
||||
{
|
||||
var candidate = Path.Combine(root, OperatingSystem.IsWindows() ? "dotnet.exe" : "dotnet");
|
||||
if (File.Exists(candidate)) return candidate;
|
||||
}
|
||||
return "dotnet";
|
||||
}
|
||||
|
||||
private void Begin()
|
||||
{
|
||||
ReadyTimestamp = Stopwatch.GetTimestamp();
|
||||
Volatile.Write(ref _lastFrameTimestamp, ReadyTimestamp);
|
||||
_ = Task.Run(ReadLoopAsync);
|
||||
_ = Task.Run(WriteLoopAsync);
|
||||
_ = Task.Run(HeartbeatMonitorAsync);
|
||||
_ = Task.Run(WatchProcessAsync);
|
||||
}
|
||||
|
||||
/// <summary>Ensures <paramref name="key"/> is compiled in this worker at <paramref name="sourceHash"/>.</summary>
|
||||
/// <param name="key">The scoped key.</param>
|
||||
/// <param name="kind">The context kind.</param>
|
||||
/// <param name="source">The source.</param>
|
||||
/// <param name="sourceHash">SHA-256 of the source.</param>
|
||||
/// <param name="inputPaths">The slot order.</param>
|
||||
/// <returns>The load outcome and the slot map; null when the worker ended first.</returns>
|
||||
public Task<LoadEntry?> EnsureLoadedAsync(string key, ContextKind kind, string source, byte[] sourceHash, IReadOnlyList<string> inputPaths)
|
||||
{
|
||||
var hashHex = Convert.ToHexString(sourceHash);
|
||||
if (_loaded.TryGetValue(key, out var existing) && existing.SourceHash == hashHex && existing.SameSlots(inputPaths))
|
||||
return existing.Ready;
|
||||
|
||||
var entry = new LoadEntry(hashHex, inputPaths);
|
||||
_loaded[key] = entry;
|
||||
var loadId = (ulong)Interlocked.Increment(ref _nextLoadId);
|
||||
var tcs = new TaskCompletionSource<LoadResult>(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
_loads[loadId] = tcs;
|
||||
var load = new LoadScripts { LoadId = loadId };
|
||||
var script = new ScriptSource { Key = key, Kind = kind, Source = source, SourceSha256 = Google.Protobuf.ByteString.CopyFrom(sourceHash) };
|
||||
script.InputPaths.AddRange(inputPaths);
|
||||
load.Scripts.Add(script);
|
||||
if (!Send(new Envelope { Load = load })) { _loads.TryRemove(loadId, out _); entry.Complete(null); return entry.Ready; }
|
||||
|
||||
_ = tcs.Task.ContinueWith(t =>
|
||||
{
|
||||
if (t.IsCompletedSuccessfully)
|
||||
{
|
||||
var outcome = t.Result.Outcomes.FirstOrDefault(o => o.Key == key);
|
||||
entry.Complete(outcome ?? new LoadOutcome { Key = key, Ok = false, Error = new ScriptError { Kind = ScriptError.Types.Kind.Threw, Message = "no load outcome" } });
|
||||
if (outcome is { Ok: false }) _loaded.TryRemove(new KeyValuePair<string, LoadEntry>(key, entry));
|
||||
}
|
||||
else
|
||||
{
|
||||
entry.Complete(null);
|
||||
_loaded.TryRemove(new KeyValuePair<string, LoadEntry>(key, entry));
|
||||
}
|
||||
}, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
|
||||
return entry.Ready;
|
||||
}
|
||||
|
||||
/// <summary>Forgets a key's compiled form here and in the worker.</summary>
|
||||
/// <param name="keys">The scoped keys.</param>
|
||||
public void Unload(IReadOnlyCollection<string> keys)
|
||||
{
|
||||
var unload = new Unload();
|
||||
foreach (var key in keys)
|
||||
if (_loaded.TryRemove(key, out _)) unload.Keys.Add(key);
|
||||
if (unload.Keys.Count > 0) Send(new Envelope { Unload = unload });
|
||||
}
|
||||
|
||||
/// <summary>Submits one evaluation. Its <see cref="PendingEval.Completion"/> yields null when the worker ends first.</summary>
|
||||
/// <param name="key">The scoped key.</param>
|
||||
/// <param name="request">The request (its id is assigned here).</param>
|
||||
/// <returns>The pending evaluation.</returns>
|
||||
public PendingEval Submit(string key, EvalRequest request)
|
||||
{
|
||||
request.EvalId = (ulong)Interlocked.Increment(ref _nextEvalId);
|
||||
var pending = new PendingEval(key, request.EvalId);
|
||||
_pending[request.EvalId] = pending;
|
||||
var batch = new EvalBatch();
|
||||
batch.Items.Add(request);
|
||||
if (!Send(new Envelope { Eval = batch }) || _exited.Task.IsCompleted)
|
||||
{
|
||||
if (_pending.TryRemove(request.EvalId, out _)) pending.Complete(null);
|
||||
}
|
||||
return pending;
|
||||
}
|
||||
|
||||
/// <summary>Cancels a run's <c>ctx.CancellationToken</c> in the worker (sent at the soft deadline).</summary>
|
||||
/// <param name="pending">The evaluation.</param>
|
||||
public void Cancel(PendingEval pending)
|
||||
{
|
||||
if (!pending.Completion.IsCompleted) Send(new Envelope { Cancel = new Protocol.V1.Cancel { EvalId = pending.EvalId } });
|
||||
}
|
||||
|
||||
/// <summary>Asks the worker for its process diagnostics (loaded assemblies, environment names).</summary>
|
||||
/// <param name="timeout">How long to wait.</param>
|
||||
/// <returns>The reply.</returns>
|
||||
public async Task<DiagnosticsReply> DiagnosticsAsync(TimeSpan timeout)
|
||||
{
|
||||
var tcs = new TaskCompletionSource<DiagnosticsReply>(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
Interlocked.Exchange(ref _diag, tcs);
|
||||
Send(new Envelope { DiagRequest = new DiagnosticsRequest() });
|
||||
return await tcs.Task.WaitAsync(timeout).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>Kills the worker (process tree). The first reason given is the one recorded.</summary>
|
||||
/// <param name="reason">Why.</param>
|
||||
/// <param name="culpritKey">The scoped key blamed, if any.</param>
|
||||
public void Kill(WorkerExitReason reason, string? culpritKey)
|
||||
{
|
||||
lock (_pending)
|
||||
{
|
||||
_killedFor ??= (reason, culpritKey);
|
||||
}
|
||||
try
|
||||
{
|
||||
if (!_process.HasExited) _process.Kill(entireProcessTree: true);
|
||||
}
|
||||
catch (InvalidOperationException) { /* already gone */ }
|
||||
catch (System.ComponentModel.Win32Exception ex)
|
||||
{
|
||||
_log.Warning(ex, "ScriptWorker[{Shard}] kill of pid {Pid} failed", Shard, Pid);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Asks the worker to stop, waits briefly, then kills it.</summary>
|
||||
/// <returns>A task that completes when the worker has ended.</returns>
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
// Both the shard (on exit) and the runner (on shutdown) dispose a connection; only the first does the work.
|
||||
if (Interlocked.Exchange(ref _disposed, 1) != 0) return;
|
||||
if (!_exited.Task.IsCompleted)
|
||||
{
|
||||
lock (_pending) { _killedFor ??= (WorkerExitReason.Disposed, null); }
|
||||
Send(new Envelope { Shutdown = new Shutdown() });
|
||||
try
|
||||
{
|
||||
await _process.WaitForExitAsync().WaitAsync(TimeSpan.FromSeconds(1)).ConfigureAwait(false);
|
||||
}
|
||||
catch (TimeoutException)
|
||||
{
|
||||
Kill(WorkerExitReason.Disposed, null);
|
||||
}
|
||||
try { await _exited.Task.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false); } catch (TimeoutException) { }
|
||||
}
|
||||
_stopping.Cancel();
|
||||
_outbound.Writer.TryComplete();
|
||||
await _pipe.DisposeAsync().ConfigureAwait(false);
|
||||
_job?.Dispose();
|
||||
_process.Dispose();
|
||||
TryDeleteDirectory(_workDir);
|
||||
}
|
||||
|
||||
/// <summary>Test hook: closes the host end of the pipe without touching the process, as a dying host would.</summary>
|
||||
internal void SeverPipeForTest() => _pipe.Dispose();
|
||||
|
||||
private bool Send(Envelope envelope)
|
||||
{
|
||||
envelope.ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion;
|
||||
return _outbound.Writer.TryWrite(envelope);
|
||||
}
|
||||
|
||||
private async Task WriteLoopAsync()
|
||||
{
|
||||
var reader = _outbound.Reader;
|
||||
try
|
||||
{
|
||||
while (await reader.WaitToReadAsync(_stopping.Token).ConfigureAwait(false))
|
||||
{
|
||||
// Opportunistic batching (design §4.5): every evaluation queued right now goes out as ONE EvalBatch.
|
||||
EvalBatch? batch = null;
|
||||
while (reader.TryRead(out var envelope))
|
||||
{
|
||||
if (envelope.BodyCase == Envelope.BodyOneofCase.Eval)
|
||||
{
|
||||
(batch ??= new EvalBatch()).Items.AddRange(envelope.Eval.Items);
|
||||
continue;
|
||||
}
|
||||
envelope.Seq = (ulong)Interlocked.Increment(ref _seq);
|
||||
await ScriptWorkerFrameCodec.WriteAsync(_pipe, envelope, _maxFrameBytes, _stopping.Token).ConfigureAwait(false);
|
||||
}
|
||||
if (batch is not null)
|
||||
{
|
||||
await ScriptWorkerFrameCodec.WriteAsync(_pipe, new Envelope
|
||||
{
|
||||
ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion,
|
||||
Seq = (ulong)Interlocked.Increment(ref _seq),
|
||||
Eval = batch,
|
||||
}, _maxFrameBytes, _stopping.Token).ConfigureAwait(false);
|
||||
}
|
||||
await _pipe.FlushAsync(_stopping.Token).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) { }
|
||||
catch (Exception ex) when (ex is IOException or ObjectDisposedException or ScriptWorkerProtocolException)
|
||||
{
|
||||
if (!_stopping.IsCancellationRequested)
|
||||
_log.Debug(ex, "ScriptWorker[{Shard}] pipe write ended", Shard);
|
||||
Kill(WorkerExitReason.Crash, null);
|
||||
}
|
||||
}
|
||||
|
||||
private async Task ReadLoopAsync()
|
||||
{
|
||||
WorkerExitReason? fault = null;
|
||||
try
|
||||
{
|
||||
while (true)
|
||||
{
|
||||
var frame = await ScriptWorkerFrameCodec.ReadAsync(_pipe, _maxFrameBytes, _stopping.Token).ConfigureAwait(false);
|
||||
if (frame is null) break;
|
||||
Volatile.Write(ref _lastFrameTimestamp, Stopwatch.GetTimestamp());
|
||||
switch (frame.BodyCase)
|
||||
{
|
||||
case Envelope.BodyOneofCase.Results:
|
||||
foreach (var result in frame.Results.Items)
|
||||
{
|
||||
// An eval id with no outstanding request is a protocol violation (design §6 "what the host
|
||||
// trusts"), except for one that was already completed by a kill race — ignored then.
|
||||
if (_pending.TryRemove(result.EvalId, out var pending)) pending.Complete(result);
|
||||
else if (result.EvalId == 0 || result.EvalId > (ulong)Interlocked.Read(ref _nextEvalId))
|
||||
throw new ScriptWorkerProtocolException($"result for unknown eval id {result.EvalId}");
|
||||
}
|
||||
break;
|
||||
case Envelope.BodyOneofCase.LoadResult:
|
||||
if (_loads.TryRemove(frame.LoadResult.LoadId, out var load)) load.TrySetResult(frame.LoadResult);
|
||||
else throw new ScriptWorkerProtocolException($"load result for unknown load id {frame.LoadResult.LoadId}");
|
||||
break;
|
||||
case Envelope.BodyOneofCase.Logs:
|
||||
try { _onLogs(frame.Logs); }
|
||||
catch (Exception ex) { _log.Warning(ex, "ScriptWorker[{Shard}] log forwarding threw", Shard); }
|
||||
break;
|
||||
case Envelope.BodyOneofCase.Heartbeat:
|
||||
break;
|
||||
case Envelope.BodyOneofCase.DiagReply:
|
||||
Volatile.Read(ref _diag)?.TrySetResult(frame.DiagReply);
|
||||
break;
|
||||
case Envelope.BodyOneofCase.Fault:
|
||||
_log.Warning("ScriptWorker[{Shard}] reported a fault: {Fault}", Shard, frame.Fault.Message);
|
||||
break;
|
||||
default:
|
||||
throw new ScriptWorkerProtocolException($"unexpected frame {frame.BodyCase} from the worker");
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) { }
|
||||
catch (ScriptWorkerProtocolException ex)
|
||||
{
|
||||
_log.Warning("ScriptWorker[{Shard}] protocol violation: {Message}; killing pid {Pid}", Shard, ex.Message, Pid);
|
||||
fault = WorkerExitReason.ProtocolFault;
|
||||
}
|
||||
catch (Exception ex) when (ex is IOException or ObjectDisposedException)
|
||||
{
|
||||
}
|
||||
|
||||
// The pipe is gone: the worker is dead or dying. Make sure, then report.
|
||||
try
|
||||
{
|
||||
if (fault is { } f) Kill(f, null);
|
||||
try
|
||||
{
|
||||
await _process.WaitForExitAsync().WaitAsync(TimeSpan.FromSeconds(2)).ConfigureAwait(false);
|
||||
}
|
||||
catch (TimeoutException)
|
||||
{
|
||||
Kill(WorkerExitReason.Crash, null);
|
||||
try { await _process.WaitForExitAsync().WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false); } catch (TimeoutException) { }
|
||||
}
|
||||
}
|
||||
catch (InvalidOperationException)
|
||||
{
|
||||
// The process object was disposed by a concurrent shutdown: the worker is already gone.
|
||||
}
|
||||
Finish();
|
||||
}
|
||||
|
||||
private async Task WatchProcessAsync()
|
||||
{
|
||||
try { await _process.WaitForExitAsync(_stopping.Token).ConfigureAwait(false); }
|
||||
catch (OperationCanceledException) { return; }
|
||||
catch (InvalidOperationException) { return; }
|
||||
// The reader normally finishes first on EOF; this covers a worker that died while the pipe stayed open.
|
||||
await Task.Delay(TimeSpan.FromSeconds(2)).ConfigureAwait(false);
|
||||
Finish();
|
||||
}
|
||||
|
||||
private async Task HeartbeatMonitorAsync()
|
||||
{
|
||||
using var timer = new PeriodicTimer(_heartbeatInterval);
|
||||
try
|
||||
{
|
||||
while (await timer.WaitForNextTickAsync(_stopping.Token).ConfigureAwait(false))
|
||||
{
|
||||
if (_exited.Task.IsCompleted) return;
|
||||
var silent = Stopwatch.GetElapsedTime(Volatile.Read(ref _lastFrameTimestamp));
|
||||
if (silent > _heartbeatInterval * 3)
|
||||
{
|
||||
_log.Warning("ScriptWorker[{Shard}] pid {Pid} sent nothing for {Silent} ms; killing it", Shard, Pid, (long)silent.TotalMilliseconds);
|
||||
Kill(WorkerExitReason.KilledHeartbeat, null);
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) { }
|
||||
}
|
||||
|
||||
private void Finish()
|
||||
{
|
||||
int? code = null;
|
||||
try { if (_process.HasExited) code = _process.ExitCode; } catch (InvalidOperationException) { }
|
||||
|
||||
(WorkerExitReason Reason, string? Culprit)? killed;
|
||||
lock (_pending) { killed = _killedFor; }
|
||||
var reason = killed?.Reason ?? (code == 0 ? WorkerExitReason.Clean : WorkerExitReason.Crash);
|
||||
var inFlight = _pending.Values.Select(p => p.Key).Distinct(StringComparer.Ordinal).ToArray();
|
||||
if (!_exited.TrySetResult(new WorkerExit(reason, killed?.Culprit, code, inFlight))) return;
|
||||
|
||||
// Every outstanding evaluation and load ends now; null tells the waiter "the worker went away".
|
||||
foreach (var id in _pending.Keys.ToArray())
|
||||
if (_pending.TryRemove(id, out var pending)) pending.Complete(null);
|
||||
foreach (var id in _loads.Keys.ToArray())
|
||||
if (_loads.TryRemove(id, out var load)) load.TrySetCanceled();
|
||||
Volatile.Read(ref _diag)?.TrySetCanceled();
|
||||
_outbound.Writer.TryComplete();
|
||||
}
|
||||
|
||||
private static string SafeExitCode(Process process)
|
||||
{
|
||||
try { return process.HasExited ? process.ExitCode.ToString(System.Globalization.CultureInfo.InvariantCulture) : "?"; }
|
||||
catch (InvalidOperationException) { return "?"; }
|
||||
}
|
||||
|
||||
private static void TryDeleteDirectory(string dir)
|
||||
{
|
||||
try { if (Directory.Exists(dir)) Directory.Delete(dir, recursive: true); } catch { /* temp dir; best effort */ }
|
||||
}
|
||||
|
||||
/// <summary>A key's compiled form in this worker.</summary>
|
||||
internal sealed class LoadEntry
|
||||
{
|
||||
private readonly TaskCompletionSource<LoadEntry?> _ready = new(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
|
||||
public LoadEntry(string sourceHash, IReadOnlyList<string> inputPaths)
|
||||
{
|
||||
SourceHash = sourceHash;
|
||||
InputPaths = inputPaths;
|
||||
var slots = new Dictionary<string, uint>(inputPaths.Count, StringComparer.Ordinal);
|
||||
for (var i = 0; i < inputPaths.Count; i++) slots[inputPaths[i]] = (uint)i;
|
||||
Slots = slots;
|
||||
}
|
||||
|
||||
public string SourceHash { get; }
|
||||
|
||||
public IReadOnlyList<string> InputPaths { get; }
|
||||
|
||||
/// <summary>Path → slot in the registered order.</summary>
|
||||
public IReadOnlyDictionary<string, uint> Slots { get; }
|
||||
|
||||
/// <summary>The worker's verdict (null until it arrives).</summary>
|
||||
public LoadOutcome? Outcome { get; private set; }
|
||||
|
||||
public Task<LoadEntry?> Ready => _ready.Task;
|
||||
|
||||
public bool SameSlots(IReadOnlyList<string> inputPaths)
|
||||
{
|
||||
if (inputPaths.Count != InputPaths.Count) return false;
|
||||
for (var i = 0; i < inputPaths.Count; i++)
|
||||
if (!string.Equals(inputPaths[i], InputPaths[i], StringComparison.Ordinal)) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
public void Complete(LoadOutcome? outcome)
|
||||
{
|
||||
Outcome = outcome;
|
||||
_ready.TrySetResult(outcome is null ? null : this);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>One evaluation outstanding on a worker.</summary>
|
||||
internal sealed class PendingEval
|
||||
{
|
||||
private readonly TaskCompletionSource<EvalResult?> _completion = new(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
|
||||
public PendingEval(string key, ulong evalId)
|
||||
{
|
||||
Key = key;
|
||||
EvalId = evalId;
|
||||
}
|
||||
|
||||
/// <summary>The scoped key.</summary>
|
||||
public string Key { get; }
|
||||
|
||||
/// <summary>The eval id on the wire.</summary>
|
||||
public ulong EvalId { get; }
|
||||
|
||||
/// <summary>The worker's result, or null when the worker ended first.</summary>
|
||||
public Task<EvalResult?> Completion => _completion.Task;
|
||||
|
||||
public void Complete(EvalResult? result) => _completion.TrySetResult(result);
|
||||
}
|
||||
@@ -0,0 +1,249 @@
|
||||
using System.Diagnostics;
|
||||
using System.Diagnostics.Metrics;
|
||||
using Serilog;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Observability;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
|
||||
|
||||
/// <summary>
|
||||
/// Supervises one shard's worker process (design §3.3, §5.3.5): starts it lazily on first use, reports its end to
|
||||
/// the runner (for attribution), and restarts it after an exponential backoff — 250 ms doubling to a 30 s cap with
|
||||
/// ±20 % jitter, reset once a worker has stayed up for 60 s. It never runs a script in the host when the worker
|
||||
/// cannot start (fail closed, §14 #4): <see cref="GetReadyAsync"/> returns null and the caller reports
|
||||
/// <c>BadResourceUnavailable</c>.
|
||||
/// </summary>
|
||||
internal sealed class WorkerShard : IAsyncDisposable
|
||||
{
|
||||
internal static readonly TimeSpan InitialBackoff = TimeSpan.FromMilliseconds(250);
|
||||
internal static readonly TimeSpan MaxBackoff = TimeSpan.FromSeconds(30);
|
||||
internal static readonly TimeSpan HealthyResetAfter = TimeSpan.FromSeconds(60);
|
||||
|
||||
private static readonly Counter<long> StartsCounter = OtOpcUaTelemetry.Meter.CreateCounter<long>(
|
||||
"otopcua.script.worker.starts", unit: "{process}", description: "ScriptWorker processes started (tag: shard).");
|
||||
|
||||
private static readonly Counter<long> ExitsCounter = OtOpcUaTelemetry.Meter.CreateCounter<long>(
|
||||
"otopcua.script.worker.exits", unit: "{process}",
|
||||
description: "ScriptWorker processes ended (tags: shard, reason = clean | killed_deadline | killed_heartbeat | killed_load | crash | protocol | disposed).");
|
||||
|
||||
private readonly Func<CancellationToken, Task<WorkerConnection>> _launch;
|
||||
private readonly Action<WorkerShard, WorkerExit> _onExit;
|
||||
private readonly ILogger _log;
|
||||
private readonly object _gate = new();
|
||||
private readonly Random _jitter = new();
|
||||
private WorkerConnection? _current;
|
||||
private Task<WorkerConnection?>? _starting;
|
||||
private long _backoffUntil;
|
||||
private int _consecutiveFailures;
|
||||
private bool _disposed;
|
||||
|
||||
public WorkerShard(string label, bool isolation, Func<CancellationToken, Task<WorkerConnection>> launch,
|
||||
Action<WorkerShard, WorkerExit> onExit, ILogger log)
|
||||
{
|
||||
Label = label;
|
||||
IsIsolation = isolation;
|
||||
_launch = launch;
|
||||
_onExit = onExit;
|
||||
_log = log;
|
||||
// The isolation worker runs one evaluation at a time, so a crash there names exactly one key (design §5.3.3).
|
||||
IsolationGate = isolation ? new SemaphoreSlim(1, 1) : null;
|
||||
}
|
||||
|
||||
/// <summary>The shard label (<c>0</c>…<c>K-1</c>, or <c>iso</c>).</summary>
|
||||
public string Label { get; }
|
||||
|
||||
/// <summary>Whether this is the isolation ("penalty box") worker.</summary>
|
||||
public bool IsIsolation { get; }
|
||||
|
||||
/// <summary>Serialises evaluations on the isolation worker; null on a normal shard.</summary>
|
||||
public SemaphoreSlim? IsolationGate { get; }
|
||||
|
||||
/// <summary>The live worker, if any (diagnostics / tests).</summary>
|
||||
public WorkerConnection? Current
|
||||
{
|
||||
get { lock (_gate) return _current is { IsAlive: true } c ? c : null; }
|
||||
}
|
||||
|
||||
/// <summary>Consecutive start failures and crashes since the last healthy period (diagnostics / tests).</summary>
|
||||
public int ConsecutiveFailures
|
||||
{
|
||||
get { lock (_gate) return _consecutiveFailures; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The ready worker, starting one if needed. Waits at most <paramref name="grace"/> — through a start in progress
|
||||
/// or a backoff — and returns null if no worker is ready by then.
|
||||
/// </summary>
|
||||
/// <param name="grace">How long to wait.</param>
|
||||
/// <param name="ct">Cancellation.</param>
|
||||
/// <returns>The worker, or null.</returns>
|
||||
public async Task<WorkerConnection?> GetReadyAsync(TimeSpan grace, CancellationToken ct)
|
||||
{
|
||||
var deadline = Stopwatch.GetTimestamp() + (long)(grace.TotalSeconds * Stopwatch.Frequency);
|
||||
while (true)
|
||||
{
|
||||
Task<WorkerConnection?> starting;
|
||||
TimeSpan wait;
|
||||
lock (_gate)
|
||||
{
|
||||
if (_disposed) return null;
|
||||
if (_current is { IsAlive: true } live) return live;
|
||||
var now = Stopwatch.GetTimestamp();
|
||||
if (_starting is null && now >= _backoffUntil)
|
||||
{
|
||||
_starting = StartCoreAsync();
|
||||
}
|
||||
if (_starting is not null)
|
||||
{
|
||||
starting = _starting;
|
||||
wait = TimeSpan.Zero;
|
||||
}
|
||||
else
|
||||
{
|
||||
starting = null!;
|
||||
wait = Stopwatch.GetElapsedTime(now, _backoffUntil);
|
||||
}
|
||||
}
|
||||
|
||||
var remaining = Stopwatch.GetElapsedTime(Stopwatch.GetTimestamp(), deadline);
|
||||
if (remaining <= TimeSpan.Zero) return null;
|
||||
if (starting is not null)
|
||||
{
|
||||
try
|
||||
{
|
||||
var started = await starting.WaitAsync(remaining, ct).ConfigureAwait(false);
|
||||
if (started is { IsAlive: true }) return started;
|
||||
}
|
||||
catch (TimeoutException)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (wait > remaining) return null;
|
||||
await Task.Delay(wait, ct).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
private async Task<WorkerConnection?> StartCoreAsync()
|
||||
{
|
||||
await Task.Yield();
|
||||
WorkerConnection? connection = null;
|
||||
try
|
||||
{
|
||||
connection = await _launch(CancellationToken.None).ConfigureAwait(false);
|
||||
StartsCounter.Add(1, new KeyValuePair<string, object?>("shard", Label));
|
||||
_log.Information("ScriptWorker[{Shard}] started, pid {Pid}", Label, connection.Pid);
|
||||
var started = connection;
|
||||
_ = started.Exited.ContinueWith(t => OnExited(started, t.Result), CancellationToken.None,
|
||||
TaskContinuationOptions.ExecuteSynchronously | TaskContinuationOptions.OnlyOnRanToCompletion, TaskScheduler.Default);
|
||||
lock (_gate)
|
||||
{
|
||||
_starting = null;
|
||||
if (_disposed)
|
||||
{
|
||||
_ = started.DisposeAsync().AsTask();
|
||||
return null;
|
||||
}
|
||||
_current = started;
|
||||
}
|
||||
return started;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.Error(ex, "ScriptWorker[{Shard}] failed to start; scripts on it report runner unavailable until it does", Label);
|
||||
lock (_gate)
|
||||
{
|
||||
_starting = null;
|
||||
ScheduleBackoffLocked();
|
||||
}
|
||||
if (connection is not null) await connection.DisposeAsync().ConfigureAwait(false);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private void OnExited(WorkerConnection connection, WorkerExit exit)
|
||||
{
|
||||
ExitsCounter.Add(1, new KeyValuePair<string, object?>("shard", Label),
|
||||
new KeyValuePair<string, object?>("reason", MetricReason(exit.Reason)));
|
||||
lock (_gate)
|
||||
{
|
||||
if (ReferenceEquals(_current, connection)) _current = null;
|
||||
if (!_disposed)
|
||||
{
|
||||
if (Stopwatch.GetElapsedTime(connection.ReadyTimestamp) >= HealthyResetAfter) _consecutiveFailures = 0;
|
||||
ScheduleBackoffLocked();
|
||||
}
|
||||
}
|
||||
if (exit.Reason != WorkerExitReason.Disposed)
|
||||
{
|
||||
_log.Warning("ScriptWorker[{Shard}] pid {Pid} ended: {Reason} (exit code {ExitCode}, culprit {Culprit}, {InFlight} evaluation(s) in flight)",
|
||||
Label, connection.Pid, exit.Reason, exit.ExitCode, exit.CulpritKey ?? "none", exit.InFlightKeys.Count);
|
||||
}
|
||||
try { _onExit(this, exit); }
|
||||
catch (Exception ex) { _log.Error(ex, "ScriptWorker[{Shard}] exit handler threw", Label); }
|
||||
_ = connection.DisposeAsync().AsTask();
|
||||
}
|
||||
|
||||
private void ScheduleBackoffLocked()
|
||||
{
|
||||
_consecutiveFailures++;
|
||||
var delay = NextBackoff(_consecutiveFailures, _jitter.NextDouble());
|
||||
_backoffUntil = Stopwatch.GetTimestamp() + (long)(delay.TotalSeconds * Stopwatch.Frequency);
|
||||
}
|
||||
|
||||
/// <summary>The restart delay after <paramref name="failures"/> consecutive failures (design §5.3.5).</summary>
|
||||
/// <param name="failures">1 for the first failure.</param>
|
||||
/// <param name="unitJitter">A sample in [0, 1) mapped to ±20 %.</param>
|
||||
/// <returns>The delay.</returns>
|
||||
internal static TimeSpan NextBackoff(int failures, double unitJitter)
|
||||
{
|
||||
var exponent = Math.Clamp(failures - 1, 0, 30);
|
||||
var baseMs = Math.Min(MaxBackoff.TotalMilliseconds, InitialBackoff.TotalMilliseconds * Math.Pow(2, exponent));
|
||||
var factor = 0.8 + (0.4 * Math.Clamp(unitJitter, 0, 1));
|
||||
return TimeSpan.FromMilliseconds(Math.Min(MaxBackoff.TotalMilliseconds * 1.2, baseMs * factor));
|
||||
}
|
||||
|
||||
private static string MetricReason(WorkerExitReason reason) => reason switch
|
||||
{
|
||||
WorkerExitReason.Clean => "clean",
|
||||
WorkerExitReason.KilledDeadline => "killed_deadline",
|
||||
WorkerExitReason.KilledHeartbeat => "killed_heartbeat",
|
||||
WorkerExitReason.KilledLoadTimeout => "killed_load",
|
||||
WorkerExitReason.ProtocolFault => "protocol",
|
||||
WorkerExitReason.Disposed => "disposed",
|
||||
_ => "crash",
|
||||
};
|
||||
|
||||
/// <summary>Stops the shard's worker.</summary>
|
||||
/// <returns>A task that completes when the worker has ended.</returns>
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
WorkerConnection? current;
|
||||
Task<WorkerConnection?>? starting;
|
||||
lock (_gate)
|
||||
{
|
||||
_disposed = true;
|
||||
current = _current;
|
||||
starting = _starting;
|
||||
_current = null;
|
||||
}
|
||||
if (starting is not null)
|
||||
{
|
||||
try { current ??= await starting.WaitAsync(TimeSpan.FromSeconds(15)).ConfigureAwait(false); } catch { /* shutting down */ }
|
||||
}
|
||||
if (current is not null) await current.DisposeAsync().ConfigureAwait(false);
|
||||
IsolationGate?.Dispose();
|
||||
}
|
||||
|
||||
/// <summary>Maps a context kind to the wire enum.</summary>
|
||||
/// <param name="kind">The kind.</param>
|
||||
/// <returns>The wire kind.</returns>
|
||||
internal static ContextKind ToWire(ScriptContextKind kind) => kind switch
|
||||
{
|
||||
ScriptContextKind.VirtualTag => ContextKind.VirtualTag,
|
||||
ScriptContextKind.Calculation => ContextKind.Calculation,
|
||||
ScriptContextKind.AlarmPredicate => ContextKind.AlarmPredicate,
|
||||
_ => ContextKind.Unspecified,
|
||||
};
|
||||
}
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<!-- lmxopcua#780 — the host side of the out-of-process, pre-emptible script runner (design §3, §5, §9): worker
|
||||
launch + pipe handshake, the per-shard supervisor, attribution and quarantine. Selected by
|
||||
Scripting:Runner=OutOfProcess; nothing here runs under the InProcess default. -->
|
||||
<PropertyGroup>
|
||||
<RootNamespace>ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess</RootNamespace>
|
||||
<GenerateDocumentationFile>true</GenerateDocumentationFile>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.Core.Scripting\ZB.MOM.WW.OtOpcUa.Core.Scripting.csproj"/>
|
||||
<ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol\ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.csproj"/>
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests"/>
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
Reference in New Issue
Block a user