dohertj2/lmxopcua
1
0
Fork 0
Code Issues 40 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(scripting): PE cache, quarantine-race fix, recovery, readiness for the out-of-process runner (lmxopcua#780 P1)
v2-ci / unit-tests (push) Failing after 1h12m10s
Details

Browse Source 
Second P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md; still opt-in behind
Scripting:Runner=OutOfProcess (InProcess unchanged).

- Compiled-PE cache (design §7, §14 #8): Scripting:Worker:PeCacheDir holds <dir>/pe/<id>.pe entries authenticated
  with HMAC-SHA256 under a per-node random key (<dir>/pe-cache.key, created by the host, 0600/0700 on Unix), handed to
  the worker in Hello. Entry id = SHA-256(compile fingerprint + kind + source); the fingerprint covers the Roslyn
  version, the Core.Scripting build (analyzer, wrapper, stack guard), the sandbox reference set and the runtime. Edited,
  truncated, other-key or renamed entries are rejected, deleted and recompiled; the worker never loads unverified bytes.
  Pruned by recency at startup (30 d / 50 000). ScriptEvaluator splits into CompileToImage + FromImage. Measured on
  40 scripts, restart to every first result: macOS 820 ms no cache / 840 ms empty / 85 ms warm;
  windev (Windows) 9.5 s / 9.4 s / 1.0 s.
- Quarantine race: WorkerConnection runs exit attribution synchronously BEFORE completing the evaluations/loads it held
  (and before Exited), and the shard passes it straight to the runner — a kill strike is in place before the killed
  key's slot is released.
- Supervision: a dead worker restarts by itself after its backoff; Unavailable outcomes carry ScriptRunOutcome.Recovery
  (-> VirtualTagEvalResult.RetryWhen) and the VT actor / Calculation driver re-evaluate once against current inputs when
  it completes. Isolation worker stops after Scripting:Worker:IsolationIdleTimeout (10 min). Keys compiling when a
  worker dies are suspects too (isolation finds a poison compile and quarantines it). Startup orphan sweep by launch
  marker: kills only a process whose marker's host is gone and whose pid + start time + image match.
- Crash-loop circuit (Scripting:Worker:CircuitOpenAfter, 2 min): runner Unavailable, one Error, evaluations answered
  at once while shards keep retrying at the cap.
- Observability (§10): otopcua.script.runner.state, worker.up/inflight/queue_depth/heap_bytes (shard),
  ipc.roundtrip + eval.duration (kind), compile{cache}, isolation.suspects, exits reason=idle. `script-runner` health
  check on the ready tier, at worst Degraded (§14 #12: degraded but still ready).

Left on #780: self-contained worker publish, installer (account, pipe ACL, firewall), docker-dev live drill,
scripted-alarm seam, the §4.5 per-seam batch submit, the §8 perf harness.

Tests: SupervisionTests, PeCacheStoreTests/PeCacheWorkerTests (real workers), ScriptRunnerHealthCheckTests, VT + Calc
recovery seam tests; 14 mutations each turned their test red.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-10 00:20:24 -05:00
parent 69a6e0b4c8
commit f50b29abae
27 changed files with 2291 additions and 78 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/WorkerConnection.cs
+90 -12
View File
@@ -33,6 +33,9 @@ internal enum WorkerExitReason
/// <summary>The runner is shutting down.</summary>
Disposed,
/// <summary>The host stopped an idle worker (the isolation worker after <c>IsolationIdleTimeout</c>); not a failure.</summary>
Idle,
}
/// <summary>The end of one worker process.</summary>
@@ -40,7 +43,16 @@ internal enum WorkerExitReason
/// <param name="CulpritKey">The scoped key the host blamed, for a deadline or load kill.</param>
/// <param name="ExitCode">The OS exit code, when known.</param>
/// <param name="InFlightKeys">The scoped keys that had an evaluation outstanding when it ended.</param>
internal sealed record WorkerExit(WorkerExitReason Reason, string? CulpritKey, int? ExitCode, IReadOnlyList<string> InFlightKeys);
/// <param name="LoadingKeys">The scoped keys whose compile (load) was outstanding when it ended.</param>
internal sealed record WorkerExit(WorkerExitReason Reason, string? CulpritKey, int? ExitCode, IReadOnlyList<string> InFlightKeys,
IReadOnlyList<string> LoadingKeys)
{
/// <summary>An exit with no load in flight.</summary>
public WorkerExit(WorkerExitReason reason, string? culpritKey, int? exitCode, IReadOnlyList<string> inFlightKeys)
: this(reason, culpritKey, exitCode, inFlightKeys, [])
{
}
}
/// <summary>What the host needs to launch one worker.</summary>
/// <param name="Shard">The shard label (<c>0</c>, <c>1</c>, … or <c>iso</c>).</param>
@@ -50,6 +62,9 @@ internal sealed record WorkerExit(WorkerExitReason Reason, string? CulpritKey, i
/// <param name="HardGrace">The worker watchdog's grace past the soft deadline.</param>
/// <param name="HeartbeatInterval">How often the worker sends a heartbeat.</param>
/// <param name="HandshakeTimeout">How long a launch may take to reach Ready.</param>
/// <param name="PeCacheDirectory">The compiled-PE cache directory handed to the worker; null = no cache.</param>
/// <param name="PeCacheKey">The cache's HMAC key (with <paramref name="PeCacheDirectory"/>).</param>
/// <param name="MarkerDirectory">Where the launch marker for the orphan sweep is written; null = none.</param>
internal sealed record WorkerLaunchSpec(
string Shard,
string WorkerPath,
@@ -57,7 +72,10 @@ internal sealed record WorkerLaunchSpec(
int EvalThreads,
TimeSpan HardGrace,
TimeSpan HeartbeatInterval,
TimeSpan HandshakeTimeout);
TimeSpan HandshakeTimeout,
string? PeCacheDirectory = null,
byte[]? PeCacheKey = null,
string? MarkerDirectory = null);
/// <summary>
/// One live ScriptWorker process and its pipe (lmxopcua#780, design §3–§6): launch with a cleared, allow-listed
@@ -85,12 +103,17 @@ internal sealed class WorkerConnection : IAsyncDisposable
private readonly Action<LogBatch> _onLogs;
private readonly Channel<Envelope> _outbound = Channel.CreateUnbounded<Envelope>(new UnboundedChannelOptions { SingleReader = true });
private readonly ConcurrentDictionary<ulong, PendingEval> _pending = new();
private readonly ConcurrentDictionary<ulong, TaskCompletionSource<LoadResult>> _loads = new();
private readonly ConcurrentDictionary<ulong, (string Key, TaskCompletionSource<LoadResult> Completion)> _loads = new();
private readonly ConcurrentDictionary<string, LoadEntry> _loaded = new(StringComparer.Ordinal);
private readonly TaskCompletionSource<WorkerExit> _exited = new(TaskCreationOptions.RunContinuationsAsynchronously);
private readonly CancellationTokenSource _stopping = new();
private readonly TimeSpan _heartbeatInterval;
private readonly Action<WorkerExit> _onExiting;
private readonly string? _markerPath;
private TaskCompletionSource<DiagnosticsReply>? _diag;
private Heartbeat? _lastHeartbeat;
private long _lastUsedTimestamp;
private int _finished;
private (WorkerExitReason Reason, string? Culprit)? _killedFor;
private long _lastFrameTimestamp;
private long _seq;
@@ -100,8 +123,10 @@ internal sealed class WorkerConnection : IAsyncDisposable
private int _disposed;
private WorkerConnection(string shard, Process process, NamedPipeServerStream pipe, IDisposable? job, string workDir,
ILogger log, Action<LogBatch> onLogs, TimeSpan heartbeatInterval)
ILogger log, Action<LogBatch> onLogs, TimeSpan heartbeatInterval, Action<WorkerExit> onExiting, string? markerPath)
{
_onExiting = onExiting;
_markerPath = markerPath;
Shard = shard;
_process = process;
_pipe = pipe;
@@ -131,13 +156,29 @@ internal sealed class WorkerConnection : IAsyncDisposable
/// <summary>Evaluations currently outstanding on this worker.</summary>
public int InFlightCount => _pending.Count;
/// <summary>Compiles currently outstanding on this worker.</summary>
public int LoadingCount => _loads.Count;
/// <summary>The worker's last heartbeat (its in-flight / queued counts and heap), or null before the first.</summary>
public Heartbeat? LastHeartbeat => Volatile.Read(ref _lastHeartbeat);
/// <summary><see cref="Stopwatch.GetTimestamp"/> of the last load or evaluation submitted (or of Ready).</summary>
public long LastUsedTimestamp => Interlocked.Read(ref _lastUsedTimestamp);
/// <summary>Launches a worker and completes the handshake.</summary>
/// <param name="spec">What to launch.</param>
/// <param name="log">Host diagnostics (stderr lines, exits).</param>
/// <param name="onLogs">Receives the worker's forwarded <c>ctx.Logger</c> batches.</param>
/// <param name="onExiting">
/// Attribution, called synchronously when the worker has ended and BEFORE any outstanding evaluation or load is
/// completed or <see cref="Exited"/> is signalled. A kill strike or suspect mark recorded here is therefore in
/// place before any caller can see its evaluation end and trigger the script again (the quarantine race,
/// lmxopcua#780).
/// </param>
/// <param name="ct">Cancellation.</param>
/// <returns>A ready connection.</returns>
public static async Task<WorkerConnection> StartAsync(WorkerLaunchSpec spec, ILogger log, Action<LogBatch> onLogs, CancellationToken ct)
public static async Task<WorkerConnection> StartAsync(WorkerLaunchSpec spec, ILogger log, Action<LogBatch> onLogs,
Action<WorkerExit> onExiting, CancellationToken ct)
{
var pipeName = "otopcua-sw-" + Convert.ToHexString(RandomNumberGenerator.GetBytes(12)).ToLowerInvariant();
var nonce = Convert.ToHexString(RandomNumberGenerator.GetBytes(32));
@@ -149,11 +190,15 @@ internal sealed class WorkerConnection : IAsyncDisposable
PipeTransmissionMode.Byte, PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly);
Process? process = null;
IDisposable? job = null;
string? markerPath = null;
try
{
var psi = BuildStartInfo(spec, pipeName, nonce, workDir);
process = new Process { StartInfo = psi, EnableRaisingEvents = true };
if (!process.Start()) throw new InvalidOperationException("the worker process did not start");
// Before anything else can fail: a host that dies from here on leaves a marker the next start's orphan
// sweep can match to exactly this process (design §3.3).
if (spec.MarkerDirectory is { } markerDir) markerPath = WorkerLaunchMarker.Write(markerDir, process, pipeName, log);
if (OperatingSystem.IsWindows())
{
// Native and JIT memory backstop above the managed heap limit.
@@ -190,9 +235,15 @@ internal sealed class WorkerConnection : IAsyncDisposable
HeartbeatIntervalMs = (uint)spec.HeartbeatInterval.TotalMilliseconds,
MaxLogLines = 1000,
};
if (spec.PeCacheDirectory is { } cacheDir && spec.PeCacheKey is { } cacheKey)
{
hello.PeCacheDir = cacheDir;
hello.PeCacheKey = Google.Protobuf.ByteString.CopyFrom(cacheKey);
}
await HandshakeAsync(pipe, hello, nonce, process.Id, handshakeCts.Token).ConfigureAwait(false);
var connection = new WorkerConnection(spec.Shard, process, pipe, job, workDir, log, onLogs, spec.HeartbeatInterval);
var connection = new WorkerConnection(spec.Shard, process, pipe, job, workDir, log, onLogs, spec.HeartbeatInterval,
onExiting, markerPath);
connection.Begin();
return connection;
}
@@ -203,6 +254,7 @@ internal sealed class WorkerConnection : IAsyncDisposable
process?.Dispose();
await pipe.DisposeAsync().ConfigureAwait(false);
TryDeleteDirectory(workDir);
WorkerLaunchMarker.TryDelete(markerPath);
throw;
}
}
@@ -308,6 +360,7 @@ internal sealed class WorkerConnection : IAsyncDisposable
{
ReadyTimestamp = Stopwatch.GetTimestamp();
Volatile.Write(ref _lastFrameTimestamp, ReadyTimestamp);
Interlocked.Exchange(ref _lastUsedTimestamp, ReadyTimestamp);
_ = Task.Run(ReadLoopAsync);
_ = Task.Run(WriteLoopAsync);
_ = Task.Run(HeartbeatMonitorAsync);
@@ -329,9 +382,10 @@ internal sealed class WorkerConnection : IAsyncDisposable
var entry = new LoadEntry(hashHex, inputPaths);
_loaded[key] = entry;
Interlocked.Exchange(ref _lastUsedTimestamp, Stopwatch.GetTimestamp());
var loadId = (ulong)Interlocked.Increment(ref _nextLoadId);
var tcs = new TaskCompletionSource<LoadResult>(TaskCreationOptions.RunContinuationsAsynchronously);
_loads[loadId] = tcs;
_loads[loadId] = (key, tcs);
var load = new LoadScripts { LoadId = loadId };
var script = new ScriptSource { Key = key, Kind = kind, Source = source, SourceSha256 = Google.Protobuf.ByteString.CopyFrom(sourceHash) };
script.InputPaths.AddRange(inputPaths);
@@ -372,6 +426,7 @@ internal sealed class WorkerConnection : IAsyncDisposable
public PendingEval Submit(string key, EvalRequest request)
{
request.EvalId = (ulong)Interlocked.Increment(ref _nextEvalId);
Interlocked.Exchange(ref _lastUsedTimestamp, Stopwatch.GetTimestamp());
var pending = new PendingEval(key, request.EvalId);
_pending[request.EvalId] = pending;
var batch = new EvalBatch();
@@ -423,13 +478,18 @@ internal sealed class WorkerConnection : IAsyncDisposable
/// <summary>Asks the worker to stop, waits briefly, then kills it.</summary>
/// <returns>A task that completes when the worker has ended.</returns>
public async ValueTask DisposeAsync()
public ValueTask DisposeAsync() => StopAsync(WorkerExitReason.Disposed);
/// <summary>As <see cref="DisposeAsync"/>, recording <paramref name="reason"/> as the exit reason.</summary>
/// <param name="reason">Why the host is stopping it (<see cref="WorkerExitReason.Disposed"/> or <see cref="WorkerExitReason.Idle"/>).</param>
/// <returns>A task that completes when the worker has ended.</returns>
public async ValueTask StopAsync(WorkerExitReason reason)
{
// Both the shard (on exit) and the runner (on shutdown) dispose a connection; only the first does the work.
if (Interlocked.Exchange(ref _disposed, 1) != 0) return;
if (!_exited.Task.IsCompleted)
{
lock (_pending) { _killedFor ??= (WorkerExitReason.Disposed, null); }
lock (_pending) { _killedFor ??= (reason, null); }
Send(new Envelope { Shutdown = new Shutdown() });
try
{
@@ -447,6 +507,7 @@ internal sealed class WorkerConnection : IAsyncDisposable
_job?.Dispose();
_process.Dispose();
TryDeleteDirectory(_workDir);
WorkerLaunchMarker.TryDelete(_markerPath);
}
/// <summary>Test hook: closes the host end of the pipe without touching the process, as a dying host would.</summary>
@@ -521,7 +582,7 @@ internal sealed class WorkerConnection : IAsyncDisposable
}
break;
case Envelope.BodyOneofCase.LoadResult:
if (_loads.TryRemove(frame.LoadResult.LoadId, out var load)) load.TrySetResult(frame.LoadResult);
if (_loads.TryRemove(frame.LoadResult.LoadId, out var load)) load.Completion.TrySetResult(frame.LoadResult);
else throw new ScriptWorkerProtocolException($"load result for unknown load id {frame.LoadResult.LoadId}");
break;
case Envelope.BodyOneofCase.Logs:
@@ -529,6 +590,7 @@ internal sealed class WorkerConnection : IAsyncDisposable
catch (Exception ex) { _log.Warning(ex, "ScriptWorker[{Shard}] log forwarding threw", Shard); }
break;
case Envelope.BodyOneofCase.Heartbeat:
Volatile.Write(ref _lastHeartbeat, frame.Heartbeat);
break;
case Envelope.BodyOneofCase.DiagReply:
Volatile.Read(ref _diag)?.TrySetResult(frame.DiagReply);
@@ -604,6 +666,11 @@ internal sealed class WorkerConnection : IAsyncDisposable
private void Finish()
{
// The reader (on EOF) and the process watcher both get here; only the first reports.
if (Interlocked.Exchange(ref _finished, 1) != 0) return;
// No new frame can be queued from here on, so every Submit / load either fails its Send (and completes itself)
// or was registered before this point and is completed below.
_outbound.Writer.TryComplete();
int? code = null;
try { if (_process.HasExited) code = _process.ExitCode; } catch (InvalidOperationException) { }
@@ -611,13 +678,21 @@ internal sealed class WorkerConnection : IAsyncDisposable
lock (_pending) { killed = _killedFor; }
var reason = killed?.Reason ?? (code == 0 ? WorkerExitReason.Clean : WorkerExitReason.Crash);
var inFlight = _pending.Values.Select(p => p.Key).Distinct(StringComparer.Ordinal).ToArray();
if (!_exited.TrySetResult(new WorkerExit(reason, killed?.Culprit, code, inFlight))) return;
var loading = _loads.Values.Select(l => l.Key).Distinct(StringComparer.Ordinal).ToArray();
var exit = new WorkerExit(reason, killed?.Culprit, code, inFlight, loading);
// Attribution FIRST, synchronously: a kill strike / quarantine / suspect mark must be in place before any
// evaluation below completes, because completing it releases the key's single-flight slot and the very next
// trigger is admitted against whatever the quarantine table says at that instant (lmxopcua#780 race fix).
try { _onExiting(exit); }
catch (Exception ex) { _log.Error(ex, "ScriptWorker[{Shard}] exit attribution threw", Shard); }
_exited.TrySetResult(exit);
// Every outstanding evaluation and load ends now; null tells the waiter "the worker went away".
foreach (var id in _pending.Keys.ToArray())
if (_pending.TryRemove(id, out var pending)) pending.Complete(null);
foreach (var id in _loads.Keys.ToArray())
if (_loads.TryRemove(id, out var load)) load.TrySetCanceled();
if (_loads.TryRemove(id, out var load)) load.Completion.TrySetCanceled();
Volatile.Read(ref _diag)?.TrySetCanceled();
_outbound.Writer.TryComplete();
}
@@ -638,6 +713,9 @@ internal sealed class WorkerConnection : IAsyncDisposable
{
private readonly TaskCompletionSource<LoadEntry?> _ready = new(TaskCreationOptions.RunContinuationsAsynchronously);
/// <summary>Set once the runner has counted this load in its metrics.</summary>
public int Counted;
public LoadEntry(string sourceHash, IReadOnlyList<string> inputPaths)
{
SourceHash = sourceHash;