feat(scripting): out-of-process pre-emptible script runner behind Scripting:Runner (lmxopcua#780 P1)
v2-ci / unit-tests (push) Waiting to run

Opt-in (Scripting:Runner=OutOfProcess; InProcess stays the default and its code path is unchanged). Implements
the P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md with the owner's §14 decisions:

- Options: Scripting:Runner (InProcess | Shadow | OutOfProcess), Scripting:Runner:Seams, Scripting:Worker:*
  (Shards 2, HardGrace max(1 s,T), HeapLimitBytes 512 MiB, QuarantineAfterKills 2, UnavailableGrace 5 s,
  IsolationProbationRuns 20, PeCacheDir validated absolute + outside the content root (fail closed),
  ExecutablePath). Shadow is not implemented yet: runs InProcess with a startup Warning.
- Core.Scripting: IScriptRunner seam + InProcessScriptRunner (today's guard, the parity baseline).
- Core.Scripting.Protocol: protobuf envelopes, 4-byte LE framing (zero/oversize/truncated/garbage refused),
  Variant carrying the exact CLR type; unsupported types are explicit failures.
- ScriptWorker app: compiles in the worker, P-thread scheduler, host-driven soft cancel, fail-fast
  watchdog, heartbeats, ctx.Logger forwarding, exits on pipe EOF / parent gone, forbidden-assembly
  self-check; Akka and Grpc.Core.Api excluded from its folder. Shipped in script-worker/ (Build + Publish).
- Core.Scripting.OutOfProcess: K shards + isolation worker, soft T / hard T+G kill, attribution
  (deadline kill = one key; crash = suspects -> isolation; isolation crash = strike), kill-strike
  quarantine, backoff 250 ms -> 30 s, fail closed, cleared allow-listed env, nonce + pid handshake on a
  current-user pipe, Windows Job Object.
- VirtualTag + Calculation seams use the runner when selected; a runner-side failure (worker restarted /
  runner unavailable) publishes BadResourceUnavailable (0x80040000) — ScadaBridge#112. Script faults keep
  BadInternalError. The scripted-alarm seam stays in process (alarms last).

Tests: new ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests (Category=ScriptWorker, real processes; green on macOS and
on windev/Windows), seam tests in Host/Runtime/Calculation suites; mutation-checked.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 23:31:51 -05:00
parent f5c30ed0ef
commit 69a6e0b4c8
49 changed files with 5107 additions and 25 deletions
@@ -0,0 +1,94 @@
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
using SerilogLogger = Serilog.ILogger;
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// <summary>
/// Which <see cref="IScriptRunner"/> each script seam uses on this node (lmxopcua#780, <c>Scripting:Runner</c>). Under
/// the <see cref="ScriptRunnerMode.InProcess"/> default every seam gets null and keeps its in-process guard path,
/// unchanged. Under <see cref="ScriptRunnerMode.OutOfProcess"/> the selected seams share one
/// <see cref="OutOfProcessScriptRunner"/>, which starts no worker until the first compiled evaluation, so a node with
/// no scripts (or an admin-only node) never launches one. Disposed with the host's service provider, which stops the
/// workers.
/// </summary>
public sealed class ScriptRunnerSelection : IAsyncDisposable, IDisposable
{
/// <summary>Seams the out-of-process runner can serve in this build. Scripted alarms go last (design §14 #10).</summary>
public static readonly IReadOnlyList<ScriptSeam> SupportedOutOfProcessSeams = [ScriptSeam.VirtualTag, ScriptSeam.Calculation];
private readonly OutOfProcessScriptRunner? _runner;
private readonly HashSet<ScriptSeam> _seams;
private ScriptRunnerSelection(OutOfProcessScriptRunner? runner, IEnumerable<ScriptSeam> seams)
{
_runner = runner;
_seams = [.. seams];
}
/// <summary>The out-of-process runner, when one is configured.</summary>
public IScriptRunner? OutOfProcessRunner => _runner;
/// <summary>The runner <paramref name="seam"/> must use, or null for its in-process path.</summary>
/// <param name="seam">The seam.</param>
/// <returns>The runner, or null.</returns>
public IScriptRunner? ForSeam(ScriptSeam seam) => _runner is not null && _seams.Contains(seam) ? _runner : null;
/// <summary>
/// Startup checks for the runner options: returns the Warnings to log, and throws when the configuration must
/// not start (a <c>PeCacheDir</c> that is relative or inside the content root — fail closed).
/// </summary>
/// <param name="options">The bound, sanitized <c>Scripting</c> options.</param>
/// <param name="contentRootPath">The host's content root.</param>
/// <returns>Warnings for the caller to log.</returns>
/// <exception cref="InvalidOperationException">The worker path options are unacceptable.</exception>
public static IReadOnlyList<string> Validate(ScriptingOptions options, string contentRootPath)
{
ArgumentNullException.ThrowIfNull(options);
var warnings = new List<string>();
if (options.Runner == ScriptRunnerMode.InProcess) return warnings;
if (options.Worker.ValidatePaths(contentRootPath) is { } violation)
throw new InvalidOperationException($"Scripting misconfiguration: {violation}");
if (options.Runner == ScriptRunnerMode.Shadow)
{
warnings.Add("Scripting:Runner=Shadow is not implemented yet (lmxopcua#780 P2); every seam runs InProcess.");
return warnings;
}
foreach (var seam in RequestedSeams(options))
{
if (!SupportedOutOfProcessSeams.Contains(seam))
warnings.Add($"Scripting:Runner=OutOfProcess: the {seam} seam cannot run out of process yet (lmxopcua#780); it stays InProcess.");
}
var workerPath = OutOfProcessScriptRunner.ResolveWorkerPath(options.Worker);
if (!File.Exists(workerPath))
warnings.Add($"Scripting:Runner=OutOfProcess: the script worker '{workerPath}' does not exist; every out-of-process " +
"script reports BadResourceUnavailable (the runner fails closed, it never falls back to in process).");
return warnings;
}
/// <summary>Builds the selection for <paramref name="options"/>.</summary>
/// <param name="options">The bound, sanitized <c>Scripting</c> options.</param>
/// <param name="log">Host diagnostics for the runner.</param>
/// <param name="scriptRoot">The root script logger forwarded <c>ctx.Logger</c> lines fall back to.</param>
/// <returns>The selection.</returns>
public static ScriptRunnerSelection Create(ScriptingOptions options, SerilogLogger log, SerilogLogger? scriptRoot)
{
ArgumentNullException.ThrowIfNull(options);
if (options.Runner != ScriptRunnerMode.OutOfProcess) return new ScriptRunnerSelection(null, []);
var seams = RequestedSeams(options).Where(SupportedOutOfProcessSeams.Contains).ToArray();
if (seams.Length == 0) return new ScriptRunnerSelection(null, []);
return new ScriptRunnerSelection(new OutOfProcessScriptRunner(options, log, scriptRoot), seams);
}
/// <inheritdoc />
public ValueTask DisposeAsync() => _runner?.DisposeAsync() ?? ValueTask.CompletedTask;
/// <inheritdoc />
public void Dispose() => _runner?.Dispose();
private static IEnumerable<ScriptSeam> RequestedSeams(ScriptingOptions options)
=> options.RunnerSeams is { Count: > 0 } seams ? seams.Distinct() : Enum.GetValues<ScriptSeam>();
}
@@ -18,6 +18,11 @@ public static class ScriptingOptionsBinding
{
ArgumentNullException.ThrowIfNull(configuration);
var bound = configuration.GetSection(ScriptingOptions.SectionName).Get<ScriptingOptions>() ?? new ScriptingOptions();
// lmxopcua#780: the seam subset lives under the runner key (Scripting:Runner:Seams), which the binder cannot
// reach because Scripting:Runner is itself a value. Bound explicitly; an unknown seam name is a startup error,
// like an unknown Scripting:Runner value.
var seams = configuration.GetSection($"{ScriptingOptions.SectionName}:Runner:Seams").Get<List<ScriptSeam>>();
if (seams is { Count: > 0 }) bound.RunnerSeams = seams;
return bound.Sanitized(out adjustments);
}
}
@@ -102,7 +102,10 @@ public static class DriverFactoryBootstrap
// failures; without this it writes to NullLogger in production.
ConfigureAbTransportLogging(sp);
Register(registry, loggerFactory, secretResolver, primaryRoleSource, ResolveAbCipDefaultTransport(sp),
sp.GetRequiredService<Driver.Galaxy.Runtime.IGalaxyAckIdentityDirectory>(), scriptRoot, scripting);
sp.GetRequiredService<Driver.Galaxy.Runtime.IGalaxyAckIdentityDirectory>(), scriptRoot, scripting,
// lmxopcua#780: null under the InProcess default (and in test graphs that register no selection).
sp.GetService<ZB.MOM.WW.OtOpcUa.Host.Configuration.ScriptRunnerSelection>()
?.ForSeam(ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptSeam.Calculation));
return registry;
});
services.AddSingleton<IDriverFactory>(sp =>
@@ -233,11 +236,12 @@ public static class DriverFactoryBootstrap
Driver.AbCip.AbCipTransport abCipDefaultTransport,
Driver.Galaxy.Runtime.IGalaxyAckIdentityDirectory galaxyAckIdentities,
ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptRootLogger? scriptRoot = null,
ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptingOptions? scripting = null)
ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptingOptions? scripting = null,
ZB.MOM.WW.OtOpcUa.Core.Scripting.IScriptRunner? scriptRunner = null)
{
Driver.AbCip.AbCipDriverFactoryExtensions.Register(registry, loggerFactory, abCipDefaultTransport);
Driver.AbLegacy.AbLegacyDriverFactoryExtensions.Register(registry, loggerFactory);
Driver.Calculation.CalculationDriverFactoryExtensions.Register(registry, loggerFactory, scriptRoot, scripting);
Driver.Calculation.CalculationDriverFactoryExtensions.Register(registry, loggerFactory, scriptRoot, scripting, scriptRunner);
Driver.FOCAS.FocasDriverFactoryExtensions.Register(registry, loggerFactory);
Driver.Galaxy.GalaxyDriverFactoryExtensions.Register(registry, secretResolver, loggerFactory, galaxyAckIdentities);
Driver.Modbus.ModbusDriverFactoryExtensions.Register(registry, loggerFactory);
@@ -53,6 +53,7 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac
private readonly SerilogLogger _scriptRoot;
private readonly TimeSpan _runTimeout;
private readonly ScriptRunGuard _guard;
private readonly IScriptRunner? _runner;
private bool _disposed;
/// <summary>Initializes a new RoslynVirtualTagEvaluator with the given loggers and optional timeout.</summary>
@@ -61,18 +62,28 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac
/// <param name="runTimeout">Maximum execution time for each script; null ⇒ <see cref="ScriptingOptions.RunTimeout"/>
/// of <paramref name="scripting"/> (2 s by default).</param>
/// <param name="scripting">Run-guard budget + quarantine threshold (the <c>Scripting</c> section); null ⇒ defaults.</param>
/// <param name="scriptRunner">
/// lmxopcua#780 — the out-of-process runner when <c>Scripting:Runner=OutOfProcess</c> selects the VirtualTag seam;
/// null (the default, <c>InProcess</c>) keeps the in-process guard path below, unchanged. The passthrough fast path
/// never reaches it.
/// </param>
public RoslynVirtualTagEvaluator(
ILogger<RoslynVirtualTagEvaluator> logger,
ScriptRootLogger scriptRoot,
TimeSpan? runTimeout = null,
ScriptingOptions? scripting = null)
ScriptingOptions? scripting = null,
IScriptRunner? scriptRunner = null)
{
_logger = logger;
_scriptRoot = (scriptRoot ?? throw new ArgumentNullException(nameof(scriptRoot))).Logger;
_runTimeout = runTimeout ?? (scripting ?? new ScriptingOptions()).Sanitized(out _).RunTimeout;
_guard = new ScriptRunGuard(scripting, _scriptRoot);
_runner = scriptRunner;
}
/// <summary>The out-of-process runner this evaluator delegates compiled runs to, or null (in process).</summary>
public IScriptRunner? ScriptRunner => _runner;
/// <summary>The run guard every compiled evaluation goes through (diagnostics / tests).</summary>
public ScriptRunGuard RunGuard => _guard;
@@ -147,6 +158,16 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac
virtualTagId, path),
logger: scriptLog);
// lmxopcua#780 — OutOfProcess: the compiled run goes to the pre-emptible worker pool, which compiles in the
// worker and owns timeouts, kills and quarantine for this seam. An Unavailable outcome (worker restarted, no
// worker ready) publishes BadResourceUnavailable; everything else keeps today's Bad contract.
if (_runner is not null)
{
var outcome = await _runner.RunAsync(new ScriptInvocation(
virtualTagId, ScriptContextKind.VirtualTag, expression, readCache, _runTimeout, scriptLog)).ConfigureAwait(false);
return outcome.ToVirtualTagEvalResult();
}
// 02/S12: fetch + run inside a bounded loop so an ObjectDisposedException caused by a concurrent
// apply-boundary ClearCompiledScripts (which disposed the evaluator between GetOrCompile and the
// run's disposed guard) re-fetches and retries exactly once. The loop runs at most twice — the
@@ -235,13 +256,20 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac
// The compile-cache clear IS the deploy boundary for this evaluator: a redeploy lifts every quarantine
// (an edited expression already lifts its own through the guard's source identity).
_guard.ResetAll();
_runner?.Reset(ScriptContextKind.VirtualTag);
}
/// <inheritdoc />
public void ResetQuarantines() => _guard.ResetAll();
public void ResetQuarantines()
{
_guard.ResetAll();
_runner?.Reset(ScriptContextKind.VirtualTag);
}
/// <inheritdoc />
public IReadOnlyCollection<string> QuarantinedScripts => _guard.QuarantinedKeys;
public IReadOnlyCollection<string> QuarantinedScripts => _runner is null
? _guard.QuarantinedKeys
: [.. _guard.QuarantinedKeys, .. _runner.QuarantinedKeys(ScriptContextKind.VirtualTag)];
/// <summary>Disposes the evaluator and cleans up all cached script evaluators.</summary>
public void Dispose()
+13 -1
View File
@@ -376,6 +376,16 @@ if (hasDriver)
Log.Warning("Scripting misconfiguration detected at startup: {Adjustment}", adjustment);
builder.Services.AddSingleton(scriptingOptions);
// lmxopcua#780: Scripting:Runner (InProcess default / Shadow / OutOfProcess). Under the default every seam keeps its
// in-process guard path unchanged; OutOfProcess hands the selected seams' compiled runs to the pre-emptible
// ScriptWorker pool. A PeCacheDir that is relative or inside the content root stops startup (fail closed).
foreach (var warning in ZB.MOM.WW.OtOpcUa.Host.Configuration.ScriptRunnerSelection.Validate(
scriptingOptions, builder.Environment.ContentRootPath))
Log.Warning("Scripting runner: {RunnerWarning}", warning);
Log.Information("Scripting runner: {RunnerMode}", scriptingOptions.Runner);
builder.Services.AddSingleton(sp => ZB.MOM.WW.OtOpcUa.Host.Configuration.ScriptRunnerSelection.Create(
sp.GetRequiredService<ScriptingOptions>(), Log.Logger, sp.GetService<ScriptRootLogger>()?.Logger));
// lmxopcua#777: the ONE operator-write timeout (DriverWrites:Timeout) every write bound is derived from — the
// driver child's cancel + abandon, its queue budget, DriverHostActor's Ask and the inbound-write gateway's Ask.
// Checked against the resilience tiers' Write timeouts here; an inconsistency is a startup Warning (a tier
@@ -392,7 +402,9 @@ if (hasDriver)
new RoslynVirtualTagEvaluator(
sp.GetRequiredService<ILoggerFactory>().CreateLogger<RoslynVirtualTagEvaluator>(),
sp.GetRequiredService<ScriptRootLogger>(),
scripting: sp.GetRequiredService<ScriptingOptions>()));
scripting: sp.GetRequiredService<ScriptingOptions>(),
scriptRunner: sp.GetRequiredService<ZB.MOM.WW.OtOpcUa.Host.Configuration.ScriptRunnerSelection>()
.ForSeam(ScriptSeam.VirtualTag)));
builder.Services.AddSingleton<IVirtualTagEvaluator>(sp => sp.GetRequiredService<RoslynVirtualTagEvaluator>());
// Script-log fan-out (Layer 0). The DPS publisher resolves the ActorSystem lazily so it never
@@ -57,6 +57,7 @@
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Configuration\ZB.MOM.WW.OtOpcUa.Configuration.csproj"/>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core\ZB.MOM.WW.OtOpcUa.Core.csproj"/>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.Scripting\ZB.MOM.WW.OtOpcUa.Core.Scripting.csproj"/>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess\ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess.csproj"/>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.VirtualTags\ZB.MOM.WW.OtOpcUa.Core.VirtualTags.csproj"/>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms\ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms.csproj"/>
<ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.Security\ZB.MOM.WW.OtOpcUa.Security.csproj"/>
@@ -90,4 +91,7 @@
<ProjectReference Include="..\..\Drivers\ZB.MOM.WW.OtOpcUa.Driver.TwinCAT\ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.csproj"/>
</ItemGroup>
<!-- lmxopcua#780: the out-of-process script runner's worker ships in script-worker/ beside the host. -->
<Import Project="..\ZB.MOM.WW.OtOpcUa.ScriptWorker\build\ScriptWorker.Deploy.targets"/>
</Project>
@@ -391,7 +391,8 @@ public sealed class VirtualTagActor : ReceiveActor, IWithUnboundedStash
var result = evaluation.Result;
if (!result.Success)
{
OnEvaluationFailed(result.Reason ?? "evaluator failure", "Warning", timestampUtc);
OnEvaluationFailed(result.Reason ?? "evaluator failure", "Warning", timestampUtc,
result.StatusCode ?? OpcUaStatus.BadInternalError);
return;
}
@@ -472,7 +473,10 @@ public sealed class VirtualTagActor : ReceiveActor, IWithUnboundedStash
/// <param name="reason">The failure reason for the script-log entry.</param>
/// <param name="level">The script-log level ("Warning" for a Failure result, "Error" for a throw).</param>
/// <param name="timestampUtc">The triggering dependency change's timestamp, stamped on the Bad result.</param>
private void OnEvaluationFailed(string reason, string level, DateTime timestampUtc)
/// <param name="badStatus">The Bad code to publish: <see cref="OpcUaStatus.BadInternalError"/> for every script
/// failure, <see cref="OpcUaStatus.BadResourceUnavailable"/> when the out-of-process runner could not run the
/// script (lmxopcua#780). The once-per-transition dedup is on this code, so a switch between the two publishes.</param>
private void OnEvaluationFailed(string reason, string level, DateTime timestampUtc, uint badStatus = OpcUaStatus.BadInternalError)
{
OtOpcUaTelemetry.VirtualTagEval.Add(1, new KeyValuePair<string, object?>("outcome", "fail"));
@@ -488,15 +492,15 @@ public sealed class VirtualTagActor : ReceiveActor, IWithUnboundedStash
}
if (admission.Admitted) PublishLog(level, reason);
if ((_hasPublished && _lastPublishedStatus == OpcUaStatus.BadInternalError) || !AllDependenciesArrived())
if ((_hasPublished && _lastPublishedStatus == badStatus) || !AllDependenciesArrived())
{
return;
}
_hasPublished = true;
_lastPublishedStatus = OpcUaStatus.BadInternalError;
_lastPublishedStatus = badStatus;
Context.Parent.Tell(new EvaluationResult(
_virtualTagId, _hasLastValue ? _lastValue : null, timestampUtc, CorrelationId.NewId(), OpcUaStatus.BadInternalError));
_virtualTagId, _hasLastValue ? _lastValue : null, timestampUtc, CorrelationId.NewId(), badStatus));
}
/// <summary>Inputs-ready gate (the retired cascade engine's rule): true once
@@ -0,0 +1,36 @@
using System.IO.Pipes;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker;
/// <summary>Entry point: connect to the host's pipe named in the environment and serve until told to stop.</summary>
internal static class Program
{
private static async Task<int> Main()
{
var pipeName = Environment.GetEnvironmentVariable(ScriptWorkerEnvironment.PipeName);
var nonce = Environment.GetEnvironmentVariable(ScriptWorkerEnvironment.Nonce);
if (string.IsNullOrEmpty(pipeName) || string.IsNullOrEmpty(nonce))
{
await Console.Error.WriteLineAsync("ScriptWorker: launched without a pipe name or nonce; exiting.");
return ScriptWorkerEnvironment.ExitProtocolFault;
}
_ = int.TryParse(Environment.GetEnvironmentVariable(ScriptWorkerEnvironment.ParentPid), out var parentPid);
// CurrentUserOnly: the client verifies the server end is owned by this account as well.
var pipe = new NamedPipeClientStream(".", pipeName, PipeDirection.InOut,
PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly);
try
{
await pipe.ConnectAsync(TimeSpan.FromSeconds(10), CancellationToken.None);
}
catch (Exception ex)
{
await Console.Error.WriteLineAsync($"ScriptWorker: could not connect to the host pipe: {ex.Message}");
return ScriptWorkerEnvironment.ExitProtocolFault;
}
var host = new WorkerHost(pipe, nonce, parentPid);
return await host.RunAsync();
}
}
@@ -0,0 +1,576 @@
using System.Collections;
using System.Collections.Concurrent;
using System.Diagnostics;
using System.IO.Pipes;
using System.Threading.Channels;
using Serilog;
using Serilog.Core;
using Serilog.Events;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
using ZB.MOM.WW.OtOpcUa.Core.VirtualTags;
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker;
/// <summary>
/// One worker's whole life (lmxopcua#780, design §3.2): handshake, compile-on-load, a bounded evaluation scheduler
/// on dedicated threads, a hard-deadline watchdog, heartbeats, and log forwarding. The host is authoritative for
/// deadlines (it kills this process); the watchdog is the belt to that brace and fails fast slightly later, so the
/// exit names the key that overran.
/// </summary>
/// <remarks>
/// Anything that means "the host is gone" — pipe end of stream, a protocol fault, the parent pid disappearing —
/// ends the process. The worker never outlives its host (design §3.3).
/// </remarks>
internal sealed class WorkerHost
{
/// <summary>The watchdog fails fast this long after the host's own hard deadline, so the host normally wins.</summary>
private static readonly TimeSpan WatchdogMargin = TimeSpan.FromMilliseconds(500);
private readonly NamedPipeClientStream _pipe;
private readonly string _nonce;
private readonly int _parentPid;
private readonly Channel<Envelope> _outbound = Channel.CreateUnbounded<Envelope>(
new UnboundedChannelOptions { SingleReader = true });
private readonly ConcurrentDictionary<string, LoadedScript> _scripts = new(StringComparer.Ordinal);
private readonly ConcurrentDictionary<long, RunningEval> _running = new();
private readonly BlockingCollection<EvalRequest> _queue = new(new ConcurrentQueue<EvalRequest>());
private readonly ForwardingLogSink _logSink;
private readonly ILogger _scriptLogger;
private int _maxFrameBytes = ScriptWorkerFrameCodec.DefaultMaxFrameBytes;
private int _queueCapacity = 4096;
private TimeSpan _hardGrace = TimeSpan.FromSeconds(1);
private TimeSpan _heartbeatInterval = TimeSpan.FromSeconds(1);
private long _seq;
public WorkerHost(NamedPipeClientStream pipe, string nonce, int parentPid)
{
_pipe = pipe;
_nonce = nonce;
_parentPid = parentPid;
_logSink = new ForwardingLogSink();
_scriptLogger = new LoggerConfiguration().MinimumLevel.Verbose().WriteTo.Sink(_logSink).CreateLogger();
}
public async Task<int> RunAsync()
{
try
{
var hello = await ScriptWorkerFrameCodec.ReadAsync(_pipe, ScriptWorkerFrameCodec.DefaultMaxFrameBytes);
if (hello is null) return ScriptWorkerEnvironment.ExitPipeClosed;
if (hello.BodyCase != Envelope.BodyOneofCase.Hello)
throw new ScriptWorkerProtocolException($"expected Hello, got {hello.BodyCase}");
Configure(hello.Hello);
// The nonce is echoed, not trusted: the host compares it with the one it put in OUR environment, which
// proves the process on the other end of its pipe is the one it launched.
await ScriptWorkerFrameCodec.WriteAsync(_pipe, Wrap(new Envelope
{
Ready = new Ready
{
Nonce = _nonce,
Pid = Environment.ProcessId,
RuntimeVersion = Environment.Version.ToString(),
},
}), _maxFrameBytes);
_logSink.Capacity = (int)Math.Max(1, hello.Hello.MaxLogLines == 0 ? 1000 : hello.Hello.MaxLogLines);
StartBackground(Math.Max(1, (int)hello.Hello.EvalThreads));
var writer = Task.Run(WriteLoopAsync);
var exit = await ReadLoopAsync();
_outbound.Writer.TryComplete();
try { await writer.WaitAsync(TimeSpan.FromSeconds(1)); } catch { /* exiting regardless */ }
return exit;
}
catch (ScriptWorkerProtocolException ex)
{
await Console.Error.WriteLineAsync($"ScriptWorker: protocol fault: {ex.Message}");
return ScriptWorkerEnvironment.ExitProtocolFault;
}
catch (IOException)
{
return ScriptWorkerEnvironment.ExitPipeClosed;
}
}
private void Configure(Hello hello)
{
if (hello.MaxFrameBytes > 0) _maxFrameBytes = (int)Math.Min(hello.MaxFrameBytes, 64u * 1024 * 1024);
if (hello.QueueCapacity > 0) _queueCapacity = (int)hello.QueueCapacity;
if (hello.HardGraceMs > 0) _hardGrace = TimeSpan.FromMilliseconds(hello.HardGraceMs);
if (hello.HeartbeatIntervalMs > 0) _heartbeatInterval = TimeSpan.FromMilliseconds(hello.HeartbeatIntervalMs);
}
private void StartBackground(int evalThreads)
{
for (var i = 0; i < evalThreads; i++)
{
new Thread(EvalThreadLoop) { IsBackground = true, Name = $"ScriptWorker.Eval{i}" }.Start();
}
new Thread(WatchdogLoop) { IsBackground = true, Name = "ScriptWorker.Watchdog" }.Start();
_ = Task.Run(HeartbeatLoopAsync);
_ = Task.Run(LogFlushLoopAsync);
if (_parentPid > 0) _ = Task.Run(ParentWatchLoopAsync);
}
private async Task<int> ReadLoopAsync()
{
while (true)
{
Envelope? frame;
try
{
frame = await ScriptWorkerFrameCodec.ReadAsync(_pipe, _maxFrameBytes);
}
catch (ScriptWorkerProtocolException ex)
{
await Console.Error.WriteLineAsync($"ScriptWorker: protocol fault: {ex.Message}");
return ScriptWorkerEnvironment.ExitProtocolFault;
}
catch (IOException)
{
return ScriptWorkerEnvironment.ExitPipeClosed;
}
if (frame is null) return ScriptWorkerEnvironment.ExitPipeClosed; // host gone: fatal by design
switch (frame.BodyCase)
{
case Envelope.BodyOneofCase.Load:
{
// Compile off the reader, so evaluations for already-loaded keys keep flowing during a load.
var load = frame.Load;
_ = Task.Run(() => Load(load));
break;
}
case Envelope.BodyOneofCase.Unload:
foreach (var key in frame.Unload.Keys)
if (_scripts.TryRemove(key, out var gone)) gone.Dispose();
break;
case Envelope.BodyOneofCase.Eval:
foreach (var item in frame.Eval.Items) Admit(item);
break;
case Envelope.BodyOneofCase.Cancel:
if (_running.TryGetValue((long)frame.Cancel.EvalId, out var toCancel))
{
try { toCancel.Cancellation.Cancel(); } catch (ObjectDisposedException) { /* just finished */ }
}
break;
case Envelope.BodyOneofCase.DiagRequest:
Send(new Envelope { DiagReply = Diagnostics() });
break;
case Envelope.BodyOneofCase.Shutdown:
return 0;
case Envelope.BodyOneofCase.Fault:
await Console.Error.WriteLineAsync($"ScriptWorker: host fault: {frame.Fault.Message}");
return ScriptWorkerEnvironment.ExitProtocolFault;
default:
await Console.Error.WriteLineAsync($"ScriptWorker: unexpected frame {frame.BodyCase}");
return ScriptWorkerEnvironment.ExitProtocolFault;
}
}
}
private void Load(LoadScripts load)
{
var result = new LoadResult { LoadId = load.LoadId };
foreach (var script in load.Scripts)
{
var outcome = new LoadOutcome { Key = script.Key, Ok = true };
try
{
var loaded = LoadedScript.Compile(script);
if (_scripts.TryRemove(script.Key, out var previous)) previous.Dispose();
_scripts[script.Key] = loaded;
}
catch (CompilationErrorException ex)
{
outcome.Ok = false;
outcome.Error = new ScriptError { Kind = ScriptError.Types.Kind.Compile, Message = ex.Message };
}
catch (ScriptSandboxViolationException ex)
{
outcome.Ok = false;
outcome.Error = new ScriptError { Kind = ScriptError.Types.Kind.Sandbox, Message = ex.Message };
}
catch (Exception ex)
{
outcome.Ok = false;
outcome.Error = new ScriptError { Kind = ScriptError.Types.Kind.Threw, Message = ex.Message };
}
result.Outcomes.Add(outcome);
}
if (ForbiddenAssemblyLoaded() is { } forbidden) DieForbidden(forbidden);
Send(new Envelope { LoadResult = result });
}
private void Admit(EvalRequest request)
{
if (_queue.Count >= _queueCapacity)
{
SendResult(new EvalResult
{
EvalId = request.EvalId,
Error = new ScriptError { Kind = ScriptError.Types.Kind.Busy, Message = "script worker busy" },
});
return;
}
_queue.Add(request);
}
private void EvalThreadLoop()
{
foreach (var request in _queue.GetConsumingEnumerable())
{
EvalResult result;
try
{
result = Evaluate(request);
}
catch (Exception ex)
{
result = new EvalResult
{
EvalId = request.EvalId,
Error = new ScriptError { Kind = ScriptError.Types.Kind.Threw, Message = ex.Message },
};
}
SendResult(result);
}
}
private EvalResult Evaluate(EvalRequest request)
{
var result = new EvalResult { EvalId = request.EvalId };
if (!_scripts.TryGetValue(request.Key, out var script))
{
result.Error = new ScriptError { Kind = ScriptError.Types.Kind.NotLoaded, Message = $"script '{request.Key}' is not loaded" };
return result;
}
var inputs = new Dictionary<string, DataValueSnapshot>(request.Inputs.Count, StringComparer.Ordinal);
foreach (var input in request.Inputs)
{
string path;
if (!string.IsNullOrEmpty(input.Path)) path = input.Path;
else if (input.Slot < script.InputPaths.Length) path = script.InputPaths[input.Slot];
else throw new ScriptWorkerProtocolException($"input slot {input.Slot} is not registered for '{request.Key}'");
if (input.Unsupported)
{
result.Error = new ScriptError
{
Kind = ScriptError.Types.Kind.UnsupportedType,
Message = $"input '{path}' has a value of unsupported type {input.UnsupportedType}",
};
return result;
}
inputs[path] = new DataValueSnapshot(
VariantCodec.Decode(input.Value),
input.StatusCode,
input.SourceTsTicks == 0 ? null : new DateTime(input.SourceTsTicks, DateTimeKind.Utc),
new DateTime(input.ServerTsTicks, DateTimeKind.Utc));
}
// ctx.Now follows the HOST clock (design §4.3) but keeps ticking: a fixed instant would never let a script
// that waits on ctx.Now finish.
var offset = request.NowUtcTicks == 0
? TimeSpan.Zero
: new DateTime(request.NowUtcTicks, DateTimeKind.Utc) - DateTime.UtcNow;
Func<DateTime> clock = () => DateTime.UtcNow + offset;
var logger = _scriptLogger.ForContext(ForwardingLogSink.KeyProperty, request.Key);
var soft = TimeSpan.FromMilliseconds(Math.Max(1, request.SoftTimeoutMs));
// Cancelled by the host's Cancel frame at the soft deadline, never by our own clock (see the proto).
using var cts = new CancellationTokenSource();
var grace = request.HardGraceMs > 0 ? TimeSpan.FromMilliseconds(request.HardGraceMs) : _hardGrace;
var running = new RunningEval(request.Key, Stopwatch.GetTimestamp(), soft + grace + WatchdogMargin, cts);
_running[(long)request.EvalId] = running;
var before = GC.GetAllocatedBytesForCurrentThread();
var sw = Stopwatch.StartNew();
try
{
object? value = script.Run(inputs, logger, clock, cts.Token, out var isPredicate);
if (isPredicate)
{
result.Predicate = (bool)value!;
}
else if (VariantCodec.TryEncode(value, out var variant, out var unsupported))
{
result.Value = variant;
}
else
{
result.Error = new ScriptError
{
Kind = ScriptError.Types.Kind.UnsupportedType,
Message = $"script returned a value of unsupported type {unsupported}",
};
}
}
catch (Exception ex)
{
result.Error = new ScriptError { Kind = ScriptError.Types.Kind.Threw, Message = ex.Message };
}
finally
{
_running.TryRemove((long)request.EvalId, out _);
}
result.AllocatedBytes = (ulong)Math.Max(0, GC.GetAllocatedBytesForCurrentThread() - before);
result.RunMicros = (uint)Math.Min(uint.MaxValue, sw.Elapsed.TotalMicroseconds);
return result;
}
private void WatchdogLoop()
{
while (true)
{
Thread.Sleep(50);
foreach (var (_, run) in _running)
{
if (Stopwatch.GetElapsedTime(run.StartedTimestamp) > run.FailFastAfter)
{
// The host should have killed us already; this names the culprit in the exit/dump.
Environment.FailFast($"ScriptWorker: script '{run.Key}' exceeded its hard deadline ({run.FailFastAfter.TotalMilliseconds:F0} ms)");
}
}
}
}
private async Task HeartbeatLoopAsync()
{
using var timer = new PeriodicTimer(_heartbeatInterval);
while (await timer.WaitForNextTickAsync())
{
Send(new Envelope
{
Heartbeat = new Heartbeat
{
Inflight = (uint)_running.Count,
Queued = (uint)_queue.Count,
HeapBytes = GC.GetTotalMemory(false),
Gen2Count = (uint)GC.CollectionCount(2),
},
});
}
}
private async Task LogFlushLoopAsync()
{
using var timer = new PeriodicTimer(TimeSpan.FromMilliseconds(100));
while (await timer.WaitForNextTickAsync())
{
if (_logSink.Drain() is { } batch) Send(new Envelope { Logs = batch });
}
}
private async Task ParentWatchLoopAsync()
{
using var timer = new PeriodicTimer(TimeSpan.FromMilliseconds(500));
while (await timer.WaitForNextTickAsync())
{
if (!ParentAlive())
{
await Console.Error.WriteLineAsync("ScriptWorker: parent process is gone; exiting.");
Environment.Exit(ScriptWorkerEnvironment.ExitParentGone);
}
}
}
private bool ParentAlive()
{
try
{
using var parent = Process.GetProcessById(_parentPid);
return !parent.HasExited;
}
catch (ArgumentException)
{
return false;
}
catch (InvalidOperationException)
{
return false;
}
}
private async Task WriteLoopAsync()
{
var reader = _outbound.Reader;
var pendingResults = new EvalResults();
try
{
while (await reader.WaitToReadAsync())
{
// Opportunistic coalescing (design §4.5): drain whatever is queued right now, merging every completed
// evaluation into one EvalResults frame; no timer, so an idle pipe adds no latency.
while (reader.TryRead(out var envelope))
{
if (envelope.BodyCase == Envelope.BodyOneofCase.Results)
{
pendingResults.Items.AddRange(envelope.Results.Items);
continue;
}
await ScriptWorkerFrameCodec.WriteAsync(_pipe, envelope, _maxFrameBytes);
}
if (pendingResults.Items.Count > 0)
{
await ScriptWorkerFrameCodec.WriteAsync(_pipe, Wrap(new Envelope { Results = pendingResults }), _maxFrameBytes);
pendingResults = new EvalResults();
}
await _pipe.FlushAsync();
}
}
catch (IOException)
{
Environment.Exit(ScriptWorkerEnvironment.ExitPipeClosed);
}
catch (ObjectDisposedException)
{
Environment.Exit(ScriptWorkerEnvironment.ExitPipeClosed);
}
}
private void SendResult(EvalResult result)
{
var results = new EvalResults();
results.Items.Add(result);
Send(new Envelope { Results = results });
}
private void Send(Envelope envelope) => _outbound.Writer.TryWrite(Wrap(envelope));
private Envelope Wrap(Envelope envelope)
{
envelope.ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion;
envelope.Seq = (ulong)Interlocked.Increment(ref _seq);
return envelope;
}
private static DiagnosticsReply Diagnostics()
{
var reply = new DiagnosticsReply
{
Pid = Environment.ProcessId,
WorkingDirectory = Environment.CurrentDirectory,
};
reply.LoadedAssemblies.AddRange(AppDomain.CurrentDomain.GetAssemblies()
.Select(a => a.GetName().Name ?? string.Empty)
.Where(n => n.Length > 0));
foreach (DictionaryEntry entry in Environment.GetEnvironmentVariables())
reply.EnvironmentNames.Add((string)entry.Key);
return reply;
}
private static string? ForbiddenAssemblyLoaded()
{
foreach (var assembly in AppDomain.CurrentDomain.GetAssemblies())
{
var name = assembly.GetName().Name ?? string.Empty;
foreach (var prefix in ScriptWorkerEnvironment.ForbiddenAssemblyPrefixes)
if (name.StartsWith(prefix, StringComparison.Ordinal)) return name;
}
return null;
}
private static void DieForbidden(string assembly)
{
Console.Error.WriteLine($"ScriptWorker: forbidden assembly '{assembly}' is loaded; exiting.");
Environment.Exit(ScriptWorkerEnvironment.ExitForbiddenAssembly);
}
private sealed record RunningEval(string Key, long StartedTimestamp, TimeSpan FailFastAfter, CancellationTokenSource Cancellation);
/// <summary>One loaded script: its compiled evaluator and its input slot order.</summary>
private sealed class LoadedScript : IDisposable
{
private readonly ScriptEvaluator<VirtualTagContext, object?>? _value;
private readonly ScriptEvaluator<AlarmPredicateContext, bool>? _predicate;
private LoadedScript(string[] inputPaths, ScriptEvaluator<VirtualTagContext, object?>? value,
ScriptEvaluator<AlarmPredicateContext, bool>? predicate)
{
InputPaths = inputPaths;
_value = value;
_predicate = predicate;
}
public string[] InputPaths { get; }
public static LoadedScript Compile(ScriptSource source) => source.Kind switch
{
ContextKind.VirtualTag or ContextKind.Calculation => new LoadedScript(
[.. source.InputPaths], ScriptEvaluator<VirtualTagContext, object?>.Compile(source.Source), null),
ContextKind.AlarmPredicate => new LoadedScript(
[.. source.InputPaths], null, ScriptEvaluator<AlarmPredicateContext, bool>.Compile(source.Source)),
_ => throw new ScriptWorkerProtocolException($"unknown script kind {source.Kind}"),
};
public object? Run(IReadOnlyDictionary<string, DataValueSnapshot> inputs, ILogger logger, Func<DateTime> clock,
CancellationToken ct, out bool isPredicate)
{
if (_predicate is not null)
{
isPredicate = true;
return _predicate.RunAsync(new AlarmPredicateContext(inputs, logger, clock), ct).GetAwaiter().GetResult();
}
isPredicate = false;
// Single-tag mode, as in the host: a cross-tag write is dropped.
return _value!.RunAsync(new VirtualTagContext(inputs, static (_, _) => { }, logger, clock), ct)
.GetAwaiter().GetResult();
}
public void Dispose()
{
_value?.Dispose();
_predicate?.Dispose();
}
}
/// <summary>Buffers <c>ctx.Logger</c> events for the host, bounded, counting what it had to drop.</summary>
private sealed class ForwardingLogSink : ILogEventSink
{
public const string KeyProperty = "ScriptWorkerKey";
private readonly ConcurrentQueue<LogLine> _lines = new();
private int _count;
private int _dropped;
public int Capacity { get; set; } = 1000;
public void Emit(LogEvent logEvent)
{
if (Interlocked.Increment(ref _count) > Capacity)
{
Interlocked.Decrement(ref _count);
Interlocked.Increment(ref _dropped);
return;
}
var key = logEvent.Properties.TryGetValue(KeyProperty, out var k) && k is ScalarValue { Value: string s } ? s : string.Empty;
_lines.Enqueue(new LogLine
{
Key = key,
Level = (uint)logEvent.Level,
Rendered = Truncate(logEvent.RenderMessage(), 8 * 1024),
Exception = Truncate(logEvent.Exception?.ToString() ?? string.Empty, 16 * 1024),
TsTicks = logEvent.Timestamp.UtcTicks,
});
}
public LogBatch? Drain()
{
var dropped = Interlocked.Exchange(ref _dropped, 0);
if (_lines.IsEmpty && dropped == 0) return null;
var batch = new LogBatch { Dropped = (uint)dropped };
while (_lines.TryDequeue(out var line))
{
Interlocked.Decrement(ref _count);
batch.Lines.Add(line);
}
return batch;
}
private static string Truncate(string value, int max) => value.Length <= max ? value : value[..max];
}
}
@@ -0,0 +1,32 @@
<Project Sdk="Microsoft.NET.Sdk">
<!-- lmxopcua#780 — the out-of-process, pre-emptible script runner's worker (design §3, §6). A plain console app the
host launches per shard over a named pipe. It references only the compile/run core and the frame protocol, so
the host's cluster, database, OPC UA and secrets stacks are not in its assembly set; it checks itself against
ScriptWorkerEnvironment.ForbiddenAssemblyPrefixes at runtime as well. -->
<PropertyGroup>
<OutputType>Exe</OutputType>
<RootNamespace>ZB.MOM.WW.OtOpcUa.ScriptWorker</RootNamespace>
<AssemblyName>ZB.MOM.WW.OtOpcUa.ScriptWorker</AssemblyName>
<InvariantGlobalization>true</InvariantGlobalization>
<!-- Workstation, non-concurrent GC: the heap hard limit (DOTNET_GCHeapHardLimit) is the worker's memory bound. -->
<ServerGarbageCollection>false</ServerGarbageCollection>
<ConcurrentGarbageCollection>false</ConcurrentGarbageCollection>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.Scripting\ZB.MOM.WW.OtOpcUa.Core.Scripting.csproj"/>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol\ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.csproj"/>
</ItemGroup>
<!-- Core.Scripting reaches Akka and Grpc.Core.Api transitively through Commons (telemetry meter, log messages),
none of which the worker executes. Excluding their assets keeps them out of the worker's deps.json and output
folder, so they are not loadable at all, not merely unloaded (design §6 "assembly allow-list by
construction"). A code path that did touch them would fail with FileNotFoundException, which the
containment tests would catch. -->
<ItemGroup>
<PackageReference Include="Akka" ExcludeAssets="all" PrivateAssets="all"/>
<PackageReference Include="Grpc.Core.Api" ExcludeAssets="all" PrivateAssets="all"/>
</ItemGroup>
</Project>
@@ -0,0 +1,43 @@
<Project>
<!--
lmxopcua#780 — places the ScriptWorker next to a host that launches it: the worker's build output is copied into
$(OutDir)script-worker/ after Build, and into $(PublishDir)script-worker/ after Publish. Imported by the Host and by
the worker test project. The worker is its own app (own deps.json / runtimeconfig.json), so it lives in its own folder
rather than beside the host's assemblies.
Publish note: the copy is the worker's framework-dependent BUILD output, run through the dotnet muxer. That suits the
docker-dev image (aspnet:10.0 runtime). A self-contained host publish (the Windows deploys) needs a self-contained
worker publish too; that is part of the installer work tracked on lmxopcua#780.
-->
<PropertyGroup>
<_OtScriptWorkerProject>$(MSBuildThisFileDirectory)../ZB.MOM.WW.OtOpcUa.ScriptWorker.csproj</_OtScriptWorkerProject>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="$(_OtScriptWorkerProject)" ReferenceOutputAssembly="false" Private="false" />
</ItemGroup>
<Target Name="_OtResolveScriptWorkerOutput">
<MSBuild Projects="$(_OtScriptWorkerProject)" Targets="GetTargetPath" Properties="Configuration=$(Configuration)">
<Output TaskParameter="TargetOutputs" ItemName="_OtScriptWorkerTargetPath" />
</MSBuild>
<PropertyGroup>
<_OtScriptWorkerOutDir>@(_OtScriptWorkerTargetPath->'%(RootDir)%(Directory)')</_OtScriptWorkerOutDir>
</PropertyGroup>
<ItemGroup>
<_OtScriptWorkerFiles Include="$(_OtScriptWorkerOutDir)**/*" />
</ItemGroup>
</Target>
<Target Name="_OtCopyScriptWorkerAfterBuild" AfterTargets="Build" DependsOnTargets="_OtResolveScriptWorkerOutput">
<Copy SourceFiles="@(_OtScriptWorkerFiles)"
DestinationFiles="@(_OtScriptWorkerFiles->'$(OutDir)script-worker/%(RecursiveDir)%(Filename)%(Extension)')"
SkipUnchangedFiles="true" />
</Target>
<Target Name="_OtCopyScriptWorkerAfterPublish" AfterTargets="Publish" DependsOnTargets="_OtResolveScriptWorkerOutput">
<Copy SourceFiles="@(_OtScriptWorkerFiles)"
DestinationFiles="@(_OtScriptWorkerFiles->'$(PublishDir)script-worker/%(RecursiveDir)%(Filename)%(Extension)')"
SkipUnchangedFiles="true" />
</Target>
</Project>