feat(scripting): out-of-process pre-emptible script runner behind Scripting:Runner (lmxopcua#780 P1)
v2-ci / unit-tests (push) Waiting to run

Opt-in (Scripting:Runner=OutOfProcess; InProcess stays the default and its code path is unchanged). Implements
the P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md with the owner's §14 decisions:

- Options: Scripting:Runner (InProcess | Shadow | OutOfProcess), Scripting:Runner:Seams, Scripting:Worker:*
  (Shards 2, HardGrace max(1 s,T), HeapLimitBytes 512 MiB, QuarantineAfterKills 2, UnavailableGrace 5 s,
  IsolationProbationRuns 20, PeCacheDir validated absolute + outside the content root (fail closed),
  ExecutablePath). Shadow is not implemented yet: runs InProcess with a startup Warning.
- Core.Scripting: IScriptRunner seam + InProcessScriptRunner (today's guard, the parity baseline).
- Core.Scripting.Protocol: protobuf envelopes, 4-byte LE framing (zero/oversize/truncated/garbage refused),
  Variant carrying the exact CLR type; unsupported types are explicit failures.
- ScriptWorker app: compiles in the worker, P-thread scheduler, host-driven soft cancel, fail-fast
  watchdog, heartbeats, ctx.Logger forwarding, exits on pipe EOF / parent gone, forbidden-assembly
  self-check; Akka and Grpc.Core.Api excluded from its folder. Shipped in script-worker/ (Build + Publish).
- Core.Scripting.OutOfProcess: K shards + isolation worker, soft T / hard T+G kill, attribution
  (deadline kill = one key; crash = suspects -> isolation; isolation crash = strike), kill-strike
  quarantine, backoff 250 ms -> 30 s, fail closed, cleared allow-listed env, nonce + pid handshake on a
  current-user pipe, Windows Job Object.
- VirtualTag + Calculation seams use the runner when selected; a runner-side failure (worker restarted /
  runner unavailable) publishes BadResourceUnavailable (0x80040000) — ScadaBridge#112. Script faults keep
  BadInternalError. The scripted-alarm seam stays in process (alarms last).

Tests: new ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests (Category=ScriptWorker, real processes; green on macOS and
on windev/Windows), seam tests in Host/Runtime/Calculation suites; mutation-checked.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 23:31:51 -05:00
parent f5c30ed0ef
commit 69a6e0b4c8
49 changed files with 5107 additions and 25 deletions
+7
View File
@@ -388,3 +388,10 @@ endpoints under `/api/script-analysis/*`, gated by the `ConfigEditor` policy
`ScriptAnalysisEndpoints.MapScriptAnalysisEndpoints`).
`Core.VirtualTags` is now just `DependencyGraph`, `ITagUpstreamSource` and `IHistoryWriter` — the dormant `VirtualTagEngine`/Source/Definition and `TimerTriggerScheduler` were deleted.
See `docs/ScriptEditor.md` for the full guide. Scripts may use the `{{equip}}` token for equipment-relative tag paths that resolve per-equipment at deploy time — see the "Equipment-relative tag paths" section in `docs/ScriptEditor.md`.
**Out-of-process script runner (lmxopcua#780, opt-in).** `Scripting:Runner` = `InProcess` (default, unchanged) | `OutOfProcess` | `Shadow` (not yet).
`OutOfProcess` runs the VirtualTag + Calculation seams' compiled scripts in `ZB.MOM.WW.OtOpcUa.ScriptWorker` processes
(`src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess`, protocol in `Core.Scripting.Protocol`), shipped in `script-worker/`
beside the host; a runner-side failure publishes `BadResourceUnavailable` (ScadaBridge-visible, ScadaBridge#112). The worker tests
are `tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests` (`Category=ScriptWorker`, real processes). See `docs/VirtualTags.md`
§"Out-of-process runner".
+4
View File
@@ -8,6 +8,8 @@
<Project Path="src/Core/ZB.MOM.WW.OtOpcUa.Core/ZB.MOM.WW.OtOpcUa.Core.csproj" />
<Project Path="src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Abstractions/ZB.MOM.WW.OtOpcUa.Core.Scripting.Abstractions.csproj" />
<Project Path="src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ZB.MOM.WW.OtOpcUa.Core.Scripting.csproj" />
<Project Path="src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.csproj" />
<Project Path="src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess.csproj" />
<Project Path="src/Core/ZB.MOM.WW.OtOpcUa.Core.VirtualTags/ZB.MOM.WW.OtOpcUa.Core.VirtualTags.csproj" />
<Project Path="src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms.csproj" />
<Project Path="src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.csproj" />
@@ -15,6 +17,7 @@
<Folder Name="/src/Server/">
<Project Path="src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ZB.MOM.WW.OtOpcUa.AdminUI.csproj" />
<Project Path="src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ZB.MOM.WW.OtOpcUa.ControlPlane.csproj" />
<Project Path="src/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker/ZB.MOM.WW.OtOpcUa.ScriptWorker.csproj" />
<Project Path="src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj" />
<Project Path="src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/ZB.MOM.WW.OtOpcUa.OpcUaServer.csproj" />
<Project Path="src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ZB.MOM.WW.OtOpcUa.Runtime.csproj" />
@@ -91,6 +94,7 @@
<Folder Name="/tests/Server/">
<Project Path="tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/ZB.MOM.WW.OtOpcUa.AdminUI.Tests.csproj" />
<Project Path="tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.csproj" />
<Project Path="tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests.csproj" />
<Project Path="tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests.csproj" />
<Project Path="tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/ZB.MOM.WW.OtOpcUa.Host.Tests.csproj" />
<Project Path="tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.csproj" />
+5
View File
@@ -211,6 +211,11 @@ scripted-alarm predicates) runs scripts through the shared `ScriptRunGuard`:
`Scripting:MaxAllocBytesPerRun` (64 MB) is quarantined. This is sampled after the
run returns, so a run that OOMs the process first is not stopped — that needs an
out-of-process runner (→ #750).
- **Out of process (opt-in, lmxopcua#780).** `Scripting:Runner=OutOfProcess` moves the
virtual-tag and Calculation seams' compiled runs into `ScriptWorker` processes, where a
run past `T + Scripting:Worker:HardGrace` is stopped by killing its worker and an OOM
kills a worker instead of the server. A run whose worker died under it reads
`BadResourceUnavailable`. See `docs/VirtualTags.md` §"Out-of-process runner".
- **Recursion.** Unbounded recursion fails the script with
`ScriptStackExhaustedException` (Bad, script log) instead of overflowing the
stack, which would terminate the whole server. The runtime compile inserts a
+10
View File
@@ -88,6 +88,16 @@ and supply `ServerHistorian__ApiKey` via the environment on the OtOpcUa host sid
`historian:write` + `historian:tags:write`. See [Historian.md](Historian.md) for the full config-key and
deployment-prerequisite reference. (The retired Wonderware TCP sidecar: [Historian.Wonderware.md](drivers/Historian.Wonderware.md).)
## Script worker processes (`Scripting:Runner=OutOfProcess`, lmxopcua#780)
Off by default. When enabled, a driver node launches up to `Scripting:Worker:Shards` + 1 child processes
(`ZB.MOM.WW.OtOpcUa.ScriptWorker`, from `script-worker/` beside the host) on its first compiled script evaluation and
kills them with it: a Job Object with kill-on-close on Windows, pipe end of stream plus a parent-pid poll elsewhere,
and a Shutdown frame then a kill on host stop. Their stderr lands in the host log prefixed `ScriptWorker[<shard>]`.
The worker currently runs under the host's account through the dotnet muxer; the dedicated low-privilege account,
pipe ACL and firewall outbound rule (design §14 #5/#6) are installer work still open on lmxopcua#780. See
`docs/VirtualTags.md` §"Out-of-process runner".
## Install / Uninstall
- `scripts/install/Install-Services.ps1 -Roles admin,driver` — installs `OtOpcUaHost`.
+44 -1
View File
@@ -37,11 +37,54 @@ User scripts are compiled via `Microsoft.CodeAnalysis.CSharp` (regular compiler,
#### Known resource limits (accepted trade-offs)
The sandbox cannot pre-empt a script. **Memory:** each run's own-thread allocation is sampled (`GC.GetAllocatedBytesForCurrentThread`) and a run over `Scripting:MaxAllocBytesPerRun` (default 64 MB) quarantines the script — but the sample is taken after the run returns, so a script that allocates fast enough (`new byte[int.MaxValue]` in a loop) can still OOM the process first. **CPU:** a tight loop keeps its pool thread after the timeout fires; `ScriptRunGuard` bounds that to one orphan per script by refusing further runs while it spins (Bad `"previous evaluation still running"`) and quarantines after three consecutive timeouts (a refusal is not a strike) — or immediately on a refusal behind an orphan that has run longer than `Scripting:StuckOrphanQuarantineFactor` (default 10) × the timeout, which is what quarantines a never-returning `while(true)`. Both are mitigations; pre-emptive CPU/memory limits need an out-of-process runner (→ #750). **Stack:** unbounded recursion used to terminate the whole server — a `StackOverflowException` cannot be caught, and lmxopcua#779 reproduced it out of process (local function, mutual recursion, self-referencing `Func<>`, anonymous self-application, recursion through LINQ: the child process died with "Stack overflow." / SIGABRT). `ScriptEvaluator.Compile` now rewrites the script so the body and every lambda, anonymous method and local function starts with `ScriptStackGuard.Check()` (`RuntimeHelpers.TryEnsureSufficientExecutionStack`), so the recursion fails the script with `ScriptStackExhaustedException` — Bad quality and a script-log entry like any throwing script — while there is still stack left. A runtime check rather than an analyzer rule because recursion is not decidable from syntax (anonymous self-application never names itself); bounded recursion keeps working. Limits: the depth reached depends on the evaluating thread's stack size; a recursion that never re-enters script code is not seen (the sandbox exposes no such API); a script can catch the exception, which only unwinds it. Pinned by `ScriptRecursionContainmentTests` (one child process per shape) and `ScriptStackGuardInjectionTests`. Script authoring is gated by the AdminUI `ConfigEditor` policy — **`Administrator` or `Designer`**, not Administrator alone — and that same policy gates deploy, so a Designer can author a script and put it into production; that gate is the primary control (AdminUI-001 deliberately admitted Designers to the script pages and `/api/script-analysis/*`; lmxopcua#779 corrected this sentence, which used to say Admin-only). See `docs/ScriptEditor.md` §"Runtime safety".
The sandbox cannot pre-empt a script. **Memory:** each run's own-thread allocation is sampled (`GC.GetAllocatedBytesForCurrentThread`) and a run over `Scripting:MaxAllocBytesPerRun` (default 64 MB) quarantines the script — but the sample is taken after the run returns, so a script that allocates fast enough (`new byte[int.MaxValue]` in a loop) can still OOM the process first. **CPU:** a tight loop keeps its pool thread after the timeout fires; `ScriptRunGuard` bounds that to one orphan per script by refusing further runs while it spins (Bad `"previous evaluation still running"`) and quarantines after three consecutive timeouts (a refusal is not a strike) — or immediately on a refusal behind an orphan that has run longer than `Scripting:StuckOrphanQuarantineFactor` (default 10) × the timeout, which is what quarantines a never-returning `while(true)`. Both are mitigations; pre-emptive CPU/memory limits need the out-of-process runner (→ #750 design, #780; opt-in via `Scripting:Runner=OutOfProcess`, below). **Stack:** unbounded recursion used to terminate the whole server — a `StackOverflowException` cannot be caught, and lmxopcua#779 reproduced it out of process (local function, mutual recursion, self-referencing `Func<>`, anonymous self-application, recursion through LINQ: the child process died with "Stack overflow." / SIGABRT). `ScriptEvaluator.Compile` now rewrites the script so the body and every lambda, anonymous method and local function starts with `ScriptStackGuard.Check()` (`RuntimeHelpers.TryEnsureSufficientExecutionStack`), so the recursion fails the script with `ScriptStackExhaustedException` — Bad quality and a script-log entry like any throwing script — while there is still stack left. A runtime check rather than an analyzer rule because recursion is not decidable from syntax (anonymous self-application never names itself); bounded recursion keeps working. Limits: the depth reached depends on the evaluating thread's stack size; a recursion that never re-enters script code is not seen (the sandbox exposes no such API); a script can catch the exception, which only unwinds it. Pinned by `ScriptRecursionContainmentTests` (one child process per shape) and `ScriptStackGuardInjectionTests`. Script authoring is gated by the AdminUI `ConfigEditor` policy — **`Administrator` or `Designer`**, not Administrator alone — and that same policy gates deploy, so a Designer can author a script and put it into production; that gate is the primary control (AdminUI-001 deliberately admitted Designers to the script pages and `/api/script-analysis/*`; lmxopcua#779 corrected this sentence, which used to say Admin-only). See `docs/ScriptEditor.md` §"Runtime safety".
`System.Threading.Tasks` is denied (Core.Scripting-003), which prevents `Task.Run` / `Parallel` fan-out that would spawn background work outliving the timeout.
#### Out-of-process runner (`Scripting:Runner`, lmxopcua#780)
`Scripting:Runner` selects where compiled scripts run: **`InProcess`** (the default and the rollback — everything
above, unchanged), **`OutOfProcess`**, or `Shadow` (design P2; not implemented yet, so it runs `InProcess` with a
startup Warning). `Scripting:Runner:Seams` narrows `OutOfProcess` to a subset (`VirtualTag`, `Calculation`; the
`ScriptedAlarm` seam stays in process for now — alarms go last, design §14 #10). Under `OutOfProcess` the virtual-tag
and Calculation evaluators hand each compiled run to `OutOfProcessScriptRunner`
(`Core.Scripting.OutOfProcess`), which runs it in a pool of `ZB.MOM.WW.OtOpcUa.ScriptWorker` processes:
- **K shards** (`Scripting:Worker:Shards`, default 2) placed by a stable FNV-1a hash of the key, plus an on-demand
**isolation worker** that runs suspects one at a time. No worker starts until the first compiled evaluation, so a
node without scripts never launches one. The passthrough fast path never leaves the host.
- **Two deadlines.** At the soft deadline `T` (the seam's budget, unchanged) the caller gets today's
`script timed out after …` failure and the run token is cancelled inside the worker; at `T + G`
(`Scripting:Worker:HardGrace`, default `max(1 s, T)`) the worker is **killed** (`Process.Kill(entireProcessTree)`,
plus a worker-side fail-fast watchdog) — the thread, its memory and its ALC go with it.
- **Attribution and quarantine** live in the host and survive restarts: a deadline kill strikes exactly its key
(other evaluations in flight are victims, not strikes); a crash with no deadline crossed makes every key in flight
a suspect that must complete `Scripting:Worker:IsolationProbationRuns` (20) clean runs in the isolation worker,
where a crash strikes exactly one key. `Scripting:Worker:QuarantineAfterKills` (2) strikes within an hour, or the
existing consecutive-timeout and allocation rules, quarantine with the usual `"script quarantined: "` prefix; a
source change or a redeploy lifts it.
- **Status codes.** Script faults keep `BadInternalError`. When the runner, not the script, failed — the worker
restarted under the run (`"script worker restarted"`) or no worker became ready within
`Scripting:Worker:UnavailableGrace` (5 s, `"script runner unavailable"`) — the tag publishes
**`BadResourceUnavailable` (0x80040000)** with its last value. This is ScadaBridge-visible (ScadaBridge#112).
The runner **fails closed**: no worker means Bad, never an in-process run.
- **Containment.** The worker gets a cleared, allow-listed environment (no `ServerHistorian__ApiKey`, no
connection strings), an empty temp working directory, `DOTNET_GCHeapHardLimit = Scripting:Worker:HeapLimitBytes`
(512 MiB), a Windows Job Object (kill-on-close, one process, a commit backstop), a random current-user-only pipe
with a launch nonce in its environment, and no Akka / gRPC assemblies in its folder. It exits on pipe end of stream
or when its parent pid disappears. Results cross as protobuf `Variant`s carrying the exact CLR type, so
`(short)ctx.GetTag(x).Value` behaves as in process; an input or result type with no tag (a `List<T>`, a `Guid`)
is an explicit `unsupported type: …` failure rather than a conversion — a behaviour change from in process.
- **Deployment.** The Host build copies the worker into `script-worker/` beside `OtOpcUa.Host.dll`
(`ScriptWorker.Deploy.targets`, Build and Publish); `Scripting:Worker:ExecutablePath` overrides it. It is run through
the dotnet muxer, so it needs a framework-dependent host (the docker-dev image). Not yet done (lmxopcua#780 P1
remainder): the PE cache (`Scripting:Worker:PeCacheDir` is validated — absolute, outside the content root — but not
used, so a worker restart recompiles), a self-contained worker for the Windows installs, the low-privilege worker
account and firewall rule, the runner-state gauge / readiness component, and the scripted-alarm seam.
Pinned by `tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests` (`Category=ScriptWorker`: real worker processes).
#### Quarantine: what a client sees (status code)
A quarantined script's output reads **`BadInternalError`** on OPC UA, the same code as a script that
@@ -665,6 +665,18 @@ per-shard state, restarts in the last hour and the quarantine list. Quarantines
- Default stays `InProcess`. Enable on the docker-dev rig only.
- Gate: §13 runaway / containment tests green on Linux (docker-dev) **and** Windows (windev box, which has
the .NET 10 toolchain).
- **Partly landed in lmxopcua#780 (P1 slice):** `Scripting:Runner` / `Scripting:Runner:Seams` / `Scripting:Worker:*`
(bound, clamped, `PeCacheDir` path rule fail-closed), `IScriptRunner` + `InProcessScriptRunner`, the protobuf frame
protocol (`Core.Scripting.Protocol`), the `ScriptWorker` app (compile in the worker, P-thread scheduler, watchdog,
heartbeats, log forwarding, pipe-EOF / parent-pid exit, forbidden-assembly self-check), `OutOfProcessScriptRunner`
(K shards + isolation worker, soft/hard deadlines, attribution, kill-strike quarantine, backoff, fail closed,
Job Object on Windows), the VirtualTag and Calculation seams over it, and `BadResourceUnavailable` for
runner-unavailable / worker-restarted (ScadaBridge#112). The §13 containment cases run on macOS and Windows
(windev). Deviations: supervision is a plain object owned by DI, not `ScriptRunnerActor`/`ScriptWorkerActor`;
the per-seam batch submit (P0) is still one-at-a-time. Remaining: the HMAC'd PE cache, a self-contained worker
publish + the low-privilege account / pipe ACL / firewall rule (installer), the runner-state gauge and readiness
component, victim re-evaluation on shard return, isolation-worker idle stop, the orphan sweep by nonce file, the
scripted-alarm seam, the docker-dev enablement and the §8 perf harness.
3. **P2, shadow mode on docker-dev and QA.**
- `Shadow` runs the **in-process** evaluation as the published result, and *also* submits to the worker.
- It compares value, status class and failure kind per evaluation, and counts
@@ -87,6 +87,19 @@ public sealed record VirtualTagEvalResult(bool Success, object? Value, string? R
/// <param name="reason">Description of the failure cause.</param>
/// <returns>A failed evaluation result.</returns>
public static VirtualTagEvalResult Failure(string reason) => new(false, null, reason);
/// <summary>
/// The OPC UA status a <see cref="Failure"/> publishes. Null (every failure the in-process path produces) means
/// the seam's long-standing <c>BadInternalError</c>; the out-of-process runner sets
/// <see cref="OpcUa.OpcUaStatus.BadResourceUnavailable"/> when the runner, not the script, failed (lmxopcua#780).
/// </summary>
public uint? StatusCode { get; init; }
/// <summary>A failure that is the runner's, not the script's: published <c>BadResourceUnavailable</c>.</summary>
/// <param name="reason">Description of the failure cause.</param>
/// <returns>A failed evaluation result carrying <see cref="OpcUa.OpcUaStatus.BadResourceUnavailable"/>.</returns>
public static VirtualTagEvalResult Unavailable(string reason) =>
new(false, null, reason) { StatusCode = OpcUa.OpcUaStatus.BadResourceUnavailable };
}
/// <summary>
@@ -47,6 +47,12 @@ public static class OpcUaStatus
/// see docs/VirtualTags.md §"Quarantine: what a client sees").</summary>
public const uint BadInternalError = 0x80020000;
/// <summary>The operating system / runtime could not provide a resource. The out-of-process script runner's code
/// for "the script did not run and it is not the script's fault" — its worker restarted under it, or no worker
/// was ready in time (lmxopcua#780, design §9). Distinct from <see cref="BadInternalError"/> so an operator, and
/// ScadaBridge, can tell a sick node from a broken script. Only produced under <c>Scripting:Runner=OutOfProcess</c>.</summary>
public const uint BadResourceUnavailable = 0x80040000;
/// <summary>Part 4 severity bucket of <paramref name="statusCode"/>: the top two bits.</summary>
/// <param name="statusCode">The status code to classify.</param>
/// <returns>0 for Good, 1 for Uncertain, 2 or 3 for Bad.</returns>
@@ -0,0 +1,505 @@
using System.Collections.Concurrent;
using System.Security.Cryptography;
using System.Text;
using Serilog;
using Serilog.Core;
using Serilog.Events;
using ZB.MOM.WW.OtOpcUa.Commons.Engines;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
/// <summary>
/// The pre-emptible <see cref="IScriptRunner"/> (lmxopcua#780, design §3–§9): compiled scripts run in a pool of
/// <c>ScriptWorker</c> processes — K shards placed by a stable hash of the script key, plus an isolation worker
/// for suspects — so a script that overruns its hard deadline is stopped by killing its worker, and an OOM, a
/// native crash or a fail-fast takes down a worker instead of the server.
/// </summary>
/// <remarks>
/// <list type="bullet">
/// <item><b>Two deadlines.</b> At the soft deadline <c>T</c> the caller gets today's timeout failure (and the
/// run token is cancelled inside the worker). At <c>T + G</c> (<see cref="ScriptWorkerOptions.EffectiveHardGrace"/>)
/// the worker is killed and the key takes a kill strike.</item>
/// <item><b>Attribution.</b> A deadline kill blames exactly one key; the other evaluations in flight on that
/// worker are victims and report <see cref="ScriptRunOutcome.WorkerRestartedReason"/> (no strike). A crash with
/// no deadline crossed makes every key in flight a suspect, and suspects run in the isolation worker one at a
/// time, where a crash names exactly one key. A suspect returns to its shard after
/// <see cref="ScriptWorkerOptions.IsolationProbationRuns"/> clean runs.</item>
/// <item><b>Quarantine</b> lives here, in the host, so it survives worker restarts: kill strikes (within an
/// hour), consecutive soft timeouts, or one allocation-budget breach. Reasons keep the
/// <see cref="ScriptRunGuard.QuarantinedReasonPrefix"/>, and a source change or <see cref="Reset"/> lifts them.</item>
/// <item><b>Fail closed.</b> When no worker is ready within <see cref="ScriptWorkerOptions.UnavailableGrace"/>
/// the result is <see cref="ScriptRunStatus.Unavailable"/> — never an in-process run.</item>
/// </list>
/// </remarks>
public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable, IDisposable
{
/// <summary>The worker entry assembly's file name.</summary>
public const string WorkerAssemblyFileName = "ZB.MOM.WW.OtOpcUa.ScriptWorker.dll";
/// <summary>The folder under the host's base directory the Host build places the worker in.</summary>
public const string WorkerFolderName = "script-worker";
/// <summary>Failure reason when the worker could not take more work.</summary>
public const string WorkerBusyReason = "script worker busy";
private static readonly TimeSpan KillStrikeWindow = TimeSpan.FromHours(1);
private readonly ScriptingOptions _options;
private readonly ScriptWorkerOptions _worker;
private readonly ILogger _log;
private readonly ILogger _scriptRoot;
private readonly WorkerShard[] _shards;
private readonly WorkerShard _isolation;
private readonly ConcurrentDictionary<string, KeyState> _keys = new(StringComparer.Ordinal);
private readonly ConcurrentDictionary<string, string> _compileFailures = new(StringComparer.Ordinal);
private int _disposed;
/// <summary>Initializes the runner. No process starts until the first evaluation.</summary>
/// <param name="options">The node's <c>Scripting</c> options (sanitized again here).</param>
/// <param name="log">Host diagnostics: starts, kills, quarantines; null ⇒ Serilog's global logger.</param>
/// <param name="scriptRoot">Fallback logger for forwarded <c>ctx.Logger</c> lines; null ⇒ <paramref name="log"/>.</param>
public OutOfProcessScriptRunner(ScriptingOptions options, ILogger? log = null, ILogger? scriptRoot = null)
: this(options, log, scriptRoot, heartbeatInterval: TimeSpan.FromSeconds(1), loadTimeout: TimeSpan.FromSeconds(60))
{
}
internal OutOfProcessScriptRunner(ScriptingOptions options, ILogger? log, ILogger? scriptRoot,
TimeSpan heartbeatInterval, TimeSpan loadTimeout)
{
ArgumentNullException.ThrowIfNull(options);
_options = options.Sanitized(out _);
_worker = _options.Worker;
_log = log ?? Log.Logger;
_scriptRoot = scriptRoot ?? _log;
LoadTimeout = loadTimeout;
WorkerPath = ResolveWorkerPath(_worker);
var evalThreads = Math.Max(2, Environment.ProcessorCount / _worker.Shards);
_shards = new WorkerShard[_worker.Shards];
for (var i = 0; i < _shards.Length; i++)
{
var label = i.ToString(System.Globalization.CultureInfo.InvariantCulture);
_shards[i] = NewShard(label, isolation: false, evalThreads, heartbeatInterval);
}
_isolation = NewShard("iso", isolation: true, evalThreads: 1, heartbeatInterval);
}
/// <summary>The worker entry assembly this runner launches.</summary>
public string WorkerPath { get; }
/// <summary>How long a compile in the worker may take before the worker is killed.</summary>
internal TimeSpan LoadTimeout { get; }
/// <summary>The shard workers (tests).</summary>
internal IReadOnlyList<WorkerShard> Shards => _shards;
/// <summary>The isolation worker (tests).</summary>
internal WorkerShard Isolation => _isolation;
/// <summary>Scoped keys currently treated as suspects (tests / diagnostics).</summary>
internal IReadOnlyCollection<string> SuspectKeys =>
_keys.Where(kv => kv.Value.Suspect).Select(kv => kv.Key).ToArray();
/// <summary>The shard a key is placed on when it is not a suspect: a stable FNV-1a hash, so placement survives restarts.</summary>
/// <param name="scopedKey">The scoped key.</param>
/// <param name="shardCount">K.</param>
/// <returns>The shard index.</returns>
internal static int ShardIndex(string scopedKey, int shardCount)
{
var hash = 2166136261u;
foreach (var b in Encoding.UTF8.GetBytes(scopedKey))
{
hash ^= b;
hash *= 16777619u;
}
return (int)(hash % (uint)shardCount);
}
/// <summary>The default worker path, or the configured one.</summary>
/// <param name="worker">The worker options.</param>
/// <returns>The absolute worker path.</returns>
public static string ResolveWorkerPath(ScriptWorkerOptions worker)
=> string.IsNullOrWhiteSpace(worker.ExecutablePath)
? Path.Combine(AppContext.BaseDirectory, WorkerFolderName, WorkerAssemblyFileName)
: Path.GetFullPath(worker.ExecutablePath);
/// <inheritdoc />
public async Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default)
{
ArgumentNullException.ThrowIfNull(invocation);
if (Volatile.Read(ref _disposed) != 0) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.RunnerUnavailableReason);
var key = InProcessScriptRunner.ScopedKey(invocation.Kind, invocation.Key);
var state = Admit(key, invocation.Source);
if (state.QuarantineReason is { } quarantined) return ScriptRunOutcome.Failed(quarantined);
var sha = SHA256.HashData(Encoding.UTF8.GetBytes(invocation.Source));
var failureKey = InProcessScriptRunner.ScopedKey(invocation.Kind, Convert.ToHexString(sha));
if (_compileFailures.TryGetValue(failureKey, out var knownFailure)) return ScriptRunOutcome.Failed(knownFailure);
if (Interlocked.CompareExchange(ref state.InFlight, 1, 0) != 0)
{
ScriptRunGuard.RefusedCounter.Add(1, new KeyValuePair<string, object?>("script", key));
return ScriptRunOutcome.Failed(ScriptRunGuard.PreviousRunStillRunningReason);
}
var handedOff = false;
SemaphoreSlim? gate = null;
try
{
state.ScriptLogger = invocation.ScriptLogger;
var shard = state.Suspect ? _isolation : _shards[ShardIndex(key, _shards.Length)];
if (shard.IsolationGate is { } isolationGate)
{
if (!await isolationGate.WaitAsync(_worker.UnavailableGrace, ct).ConfigureAwait(false))
return ScriptRunOutcome.Unavailable(ScriptRunOutcome.RunnerUnavailableReason);
gate = isolationGate;
}
var connection = await shard.GetReadyAsync(_worker.UnavailableGrace, ct).ConfigureAwait(false);
if (connection is null) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.RunnerUnavailableReason);
var paths = invocation.Inputs.Keys.Order(StringComparer.Ordinal).ToArray();
WorkerConnection.LoadEntry? entry;
try
{
entry = await connection
.EnsureLoadedAsync(key, WorkerShard.ToWire(invocation.Kind), invocation.Source, sha, paths)
.WaitAsync(LoadTimeout, ct).ConfigureAwait(false);
}
catch (TimeoutException)
{
// A compile that never finishes is the script's fault (a pathological source): the worker dies, the
// key takes a kill strike through the exit handler.
_log.Warning("Script {ScriptKey}: compile in ScriptWorker[{Shard}] exceeded {Timeout} s; killing the worker",
key, shard.Label, LoadTimeout.TotalSeconds);
connection.Kill(WorkerExitReason.KilledLoadTimeout, key);
return ScriptRunOutcome.Failed($"compile failure: compile did not finish within {LoadTimeout.TotalSeconds:F0} s");
}
if (entry?.Outcome is not { } outcome) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason);
if (!outcome.Ok)
{
var reason = outcome.Error?.Kind switch
{
ScriptError.Types.Kind.Compile => $"compile error: {outcome.Error.Message}",
ScriptError.Types.Kind.Sandbox => $"sandbox violation: {outcome.Error.Message}",
_ => $"compile failure: {outcome.Error?.Message}",
};
// Remembered until the next deploy boundary, so a broken script never re-enters Roslyn (design §7).
_compileFailures.TryAdd(failureKey, reason);
return ScriptRunOutcome.Failed(reason);
}
var pending = connection.Submit(key, BuildRequest(key, entry, invocation));
ArmHardDeadline(connection, pending, invocation.Timeout + _worker.EffectiveHardGrace(invocation.Timeout));
// From here the in-flight slot (and the isolation gate) belong to the evaluation itself: they are released
// when the worker answers or dies — not when this caller stops waiting at the soft deadline. That keeps a
// key single-flight between T and T + G, and the isolation worker strictly one-at-a-time.
handedOff = true;
var heldGate = gate;
_ = pending.Completion.ContinueWith(_ =>
{
Interlocked.Exchange(ref state.InFlight, 0);
heldGate?.Release();
}, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
EvalResult? result;
try
{
result = await pending.Completion.WaitAsync(invocation.Timeout, ct).ConfigureAwait(false);
}
catch (TimeoutException)
{
// As in process: the run token is cancelled only once the caller has its timeout, so a polling script
// ends itself (and frees its slot) without the worker being killed.
connection.Cancel(pending);
if (ct.IsCancellationRequested) return ScriptRunOutcome.Failed("evaluation cancelled");
if (NoteTimeout(key, state, invocation.Timeout)) return ScriptRunOutcome.Failed(state.QuarantineReason!);
return ScriptRunOutcome.Failed($"{ScriptEvalFailureReasons.TimedOutPrefix}{invocation.Timeout.TotalSeconds:F1}s");
}
if (result is null) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason);
return Complete(key, state, result);
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
return ScriptRunOutcome.Failed("evaluation cancelled");
}
finally
{
if (!handedOff)
{
Interlocked.Exchange(ref state.InFlight, 0);
gate?.Release();
}
}
}
/// <inheritdoc />
public void Reset(ScriptContextKind kind)
{
var prefix = InProcessScriptRunner.ScopedKey(kind, string.Empty);
var dropped = new List<string>();
foreach (var key in _keys.Keys)
{
if (key.StartsWith(prefix, StringComparison.Ordinal) && _keys.TryRemove(key, out _)) dropped.Add(key);
}
foreach (var key in _compileFailures.Keys)
{
if (key.StartsWith(prefix, StringComparison.Ordinal)) _compileFailures.TryRemove(key, out _);
}
foreach (var shard in _shards.Append(_isolation))
shard.Current?.Unload(dropped);
}
/// <inheritdoc />
public IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind)
{
var prefix = InProcessScriptRunner.ScopedKey(kind, string.Empty);
return _keys
.Where(kv => kv.Value.QuarantineReason is not null && kv.Key.StartsWith(prefix, StringComparison.Ordinal))
.Select(kv => kv.Key[prefix.Length..])
.ToArray();
}
/// <summary>Stops every worker.</summary>
/// <returns>A task that completes when they have ended.</returns>
public async ValueTask DisposeAsync()
{
if (Interlocked.Exchange(ref _disposed, 1) != 0) return;
await Task.WhenAll(_shards.Append(_isolation).Select(s => s.DisposeAsync().AsTask())).ConfigureAwait(false);
}
/// <summary>Stops every worker (blocking).</summary>
public void Dispose() => DisposeAsync().AsTask().GetAwaiter().GetResult();
private WorkerShard NewShard(string label, bool isolation, int evalThreads, TimeSpan heartbeatInterval)
{
var spec = new WorkerLaunchSpec(
Shard: label,
WorkerPath: WorkerPath,
HeapLimitBytes: _worker.HeapLimitBytes,
EvalThreads: evalThreads,
HardGrace: _worker.HardGrace ?? TimeSpan.FromSeconds(Math.Max(1, _options.RunTimeout.TotalSeconds)),
HeartbeatInterval: heartbeatInterval,
HandshakeTimeout: TimeSpan.FromSeconds(15));
return new WorkerShard(label, isolation,
ct => WorkerConnection.StartAsync(spec, _log, OnLogs, ct),
OnWorkerExit, _log);
}
private KeyState Admit(string key, string source)
{
while (true)
{
var state = _keys.GetOrAdd(key, static (_, src) => new KeyState(src), source);
if (string.Equals(state.Source, source, StringComparison.Ordinal)) return state;
// The script changed under this key: a clean slate, exactly as the in-process guard does. An evaluation
// of the old source still in flight releases the OLD state's slot when it ends.
if (_keys.TryUpdate(key, new KeyState(source), state)) continue;
}
}
private EvalRequest BuildRequest(string key, WorkerConnection.LoadEntry entry, ScriptInvocation invocation)
{
var request = new EvalRequest
{
Key = key,
NowUtcTicks = DateTime.UtcNow.Ticks,
SoftTimeoutMs = (uint)Math.Clamp(invocation.Timeout.TotalMilliseconds, 1, uint.MaxValue),
HardGraceMs = (uint)Math.Clamp(_worker.EffectiveHardGrace(invocation.Timeout).TotalMilliseconds, 1, uint.MaxValue),
};
foreach (var (path, snapshot) in invocation.Inputs)
{
if (snapshot is null) continue;
var input = new Input
{
StatusCode = snapshot.StatusCode,
SourceTsTicks = snapshot.SourceTimestampUtc is { } source ? VariantCodec.ToUtc(source).Ticks : 0,
ServerTsTicks = VariantCodec.ToUtc(snapshot.ServerTimestampUtc).Ticks,
};
if (entry.Slots.TryGetValue(path, out var slot)) input.Slot = slot;
else input.Path = path;
if (VariantCodec.TryEncode(snapshot.Value, out var variant, out var unsupported))
{
input.Value = variant;
}
else
{
// Never silently converted (design §14 #11): the worker answers UNSUPPORTED_TYPE for this input.
input.Unsupported = true;
input.UnsupportedType = unsupported ?? "?";
}
request.Inputs.Add(input);
}
return request;
}
private void ArmHardDeadline(WorkerConnection connection, PendingEval pending, TimeSpan after)
{
var cts = new CancellationTokenSource(after);
var registration = cts.Token.Register(() =>
{
if (pending.Completion.IsCompleted) return;
_log.Warning("Script {ScriptKey} exceeded its hard deadline ({Deadline} ms); killing ScriptWorker[{Shard}] pid {Pid}",
pending.Key, (long)after.TotalMilliseconds, connection.Shard, connection.Pid);
connection.Kill(WorkerExitReason.KilledDeadline, pending.Key);
});
_ = pending.Completion.ContinueWith(_ =>
{
registration.Dispose();
cts.Dispose();
}, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
}
private ScriptRunOutcome Complete(string key, KeyState state, EvalResult result)
{
// The script returned control (a value or a throw): the soft-timeout streak resets, as in process.
Interlocked.Exchange(ref state.ConsecutiveTimeouts, 0);
if (_options.MaxAllocBytesPerRun > 0 && (long)result.AllocatedBytes > _options.MaxAllocBytesPerRun)
{
Quarantine(key, state,
$"one evaluation allocated {result.AllocatedBytes / (1024.0 * 1024.0):F1} MB, over the " +
$"{_options.MaxAllocBytesPerRun / (1024.0 * 1024.0):F0} MB budget (Scripting:MaxAllocBytesPerRun); " +
"the script is not evaluated again until its source changes or it is redeployed",
"allocation");
return ScriptRunOutcome.Failed(state.QuarantineReason!);
}
if (state.Suspect && Interlocked.Increment(ref state.ProbationRuns) >= _worker.IsolationProbationRuns)
{
state.Suspect = false;
_log.Information("Script {ScriptKey} completed {Runs} clean run(s) in the isolation worker; returning it to its shard",
key, _worker.IsolationProbationRuns);
}
switch (result.OutcomeCase)
{
case EvalResult.OutcomeOneofCase.Value:
try
{
return ScriptRunOutcome.Ok(VariantCodec.Decode(result.Value));
}
catch (ScriptWorkerProtocolException ex)
{
return ScriptRunOutcome.Failed($"script worker returned an undecodable value: {ex.Message}");
}
case EvalResult.OutcomeOneofCase.Predicate:
return ScriptRunOutcome.Ok(result.Predicate);
case EvalResult.OutcomeOneofCase.Error:
return result.Error.Kind switch
{
ScriptError.Types.Kind.Threw => ScriptRunOutcome.Failed($"script threw: {result.Error.Message}"),
ScriptError.Types.Kind.UnsupportedType => ScriptRunOutcome.Failed($"unsupported type: {result.Error.Message}"),
ScriptError.Types.Kind.Busy => ScriptRunOutcome.Failed(WorkerBusyReason),
ScriptError.Types.Kind.NotLoaded => ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason),
_ => ScriptRunOutcome.Failed($"script failed: {result.Error.Message}"),
};
default:
return ScriptRunOutcome.Ok(null);
}
}
private bool NoteTimeout(string key, KeyState state, TimeSpan timeout)
{
var count = Interlocked.Increment(ref state.ConsecutiveTimeouts);
if (count < _options.QuarantineAfterConsecutiveTimeouts) return false;
Quarantine(key, state,
$"{count} consecutive evaluation(s) timed out (budget {timeout.TotalMilliseconds:F0} ms); " +
"the script is not evaluated again until its source changes or it is redeployed",
"timeouts");
return true;
}
private void OnWorkerExit(WorkerShard shard, WorkerExit exit)
{
switch (exit.Reason)
{
case WorkerExitReason.KilledDeadline when exit.CulpritKey is { } culprit:
KillStrike(culprit, "it exceeded its hard deadline and its worker was killed", "killed");
break;
case WorkerExitReason.KilledLoadTimeout when exit.CulpritKey is { } culprit:
KillStrike(culprit, "its compile did not finish and its worker was killed", "killed");
break;
case WorkerExitReason.Crash or WorkerExitReason.KilledHeartbeat or WorkerExitReason.ProtocolFault:
if (shard.IsIsolation)
{
// One evaluation at a time here: the crash is this key's (design §5.3.3).
foreach (var key in exit.InFlightKeys)
KillStrike(key, "it crashed the isolation worker", "isolated_crash");
}
else if (exit.InFlightKeys.Count > 0)
{
foreach (var key in exit.InFlightKeys)
{
if (!_keys.TryGetValue(key, out var state)) continue;
state.Suspect = true;
Interlocked.Exchange(ref state.ProbationRuns, 0);
}
_log.Warning("ScriptWorker[{Shard}] ended ({Reason}) with no deadline crossed; {Count} script(s) in flight move to the isolation worker: {Suspects}",
shard.Label, exit.Reason, exit.InFlightKeys.Count, string.Join(", ", exit.InFlightKeys));
}
break;
}
}
private void KillStrike(string key, string why, string metricReason)
{
if (!_keys.TryGetValue(key, out var state)) return;
int strikes;
lock (state.KillStrikes)
{
var now = DateTime.UtcNow;
state.KillStrikes.Add(now);
state.KillStrikes.RemoveAll(t => now - t > KillStrikeWindow);
strikes = state.KillStrikes.Count;
}
if (strikes >= _worker.QuarantineAfterKills)
{
Quarantine(key, state,
$"{why} ({strikes} time(s) in the last hour); the script is not evaluated again until its source changes or it is redeployed",
metricReason);
}
}
private void Quarantine(string key, KeyState state, string why, string metricReason)
{
var reason = ScriptRunGuard.QuarantinedReasonPrefix + why;
if (Interlocked.CompareExchange(ref state.QuarantineReason, reason, null) is not null) return;
ScriptRunGuard.QuarantinedCounter.Add(1,
new KeyValuePair<string, object?>("script", key),
new KeyValuePair<string, object?>("reason", metricReason));
_scriptRoot.ForContext(ScriptLoggerFactory.ScriptNameProperty, key).Error("Script {ScriptKey} quarantined: {Reason}", key, why);
}
private void OnLogs(LogBatch batch)
{
foreach (var line in batch.Lines)
{
var logger = _keys.TryGetValue(line.Key, out var state) && state.ScriptLogger is { } scoped ? scoped : _scriptRoot;
var level = line.Level <= (uint)LogEventLevel.Fatal ? (LogEventLevel)line.Level : LogEventLevel.Information;
if (string.IsNullOrEmpty(line.Exception))
logger.Write(level, "{ScriptMessage:l}", line.Rendered);
else
logger.Write(level, "{ScriptMessage:l}{NewLine}{ScriptException:l}", line.Rendered, Environment.NewLine, line.Exception);
}
if (batch.Dropped > 0)
_log.Warning("ScriptWorker dropped {Dropped} script log line(s): a script is logging faster than the pipe drains", batch.Dropped);
}
private sealed class KeyState(string source)
{
public readonly string Source = source;
public readonly List<DateTime> KillStrikes = [];
public int InFlight;
public int ConsecutiveTimeouts;
public int ProbationRuns;
public volatile bool Suspect;
public string? QuarantineReason;
public ILogger? ScriptLogger;
}
}
@@ -0,0 +1,114 @@
using System.ComponentModel;
using System.Diagnostics;
using System.Runtime.InteropServices;
using System.Runtime.Versioning;
using Microsoft.Win32.SafeHandles;
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
/// <summary>
/// A Windows Job Object holding one ScriptWorker (design §3.3, §5.2): <c>KILL_ON_JOB_CLOSE</c> so the kernel kills
/// the worker when the host dies (its handle closes with it), <c>ACTIVE_PROCESS = 1</c> so the worker can start no
/// child process, and a process-memory backstop slightly above the managed heap limit for native and JIT memory.
/// </summary>
[SupportedOSPlatform("windows")]
internal sealed class WindowsJobObject : IDisposable
{
private const int JobObjectExtendedLimitInformation = 9;
private const uint JOB_OBJECT_LIMIT_ACTIVE_PROCESS = 0x00000008;
private const uint JOB_OBJECT_LIMIT_PROCESS_MEMORY = 0x00000100;
private const uint JOB_OBJECT_LIMIT_DIE_ON_UNHANDLED_EXCEPTION = 0x00000400;
private const uint JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE = 0x00002000;
private readonly SafeFileHandle _handle;
private WindowsJobObject(SafeFileHandle handle) => _handle = handle;
/// <summary>Creates a job with the worker limits and assigns <paramref name="process"/> to it.</summary>
/// <param name="process">The freshly started worker.</param>
/// <param name="processMemoryLimitBytes">The per-process commit limit.</param>
/// <returns>The job; dispose it to kill the worker.</returns>
public static WindowsJobObject Assign(Process process, long processMemoryLimitBytes)
{
var handle = CreateJobObjectW(IntPtr.Zero, null);
if (handle.IsInvalid) throw new Win32Exception(Marshal.GetLastWin32Error(), "CreateJobObject failed");
var job = new WindowsJobObject(handle);
try
{
var info = new JOBOBJECT_EXTENDED_LIMIT_INFORMATION
{
BasicLimitInformation = new JOBOBJECT_BASIC_LIMIT_INFORMATION
{
LimitFlags = JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE | JOB_OBJECT_LIMIT_ACTIVE_PROCESS
| JOB_OBJECT_LIMIT_PROCESS_MEMORY | JOB_OBJECT_LIMIT_DIE_ON_UNHANDLED_EXCEPTION,
ActiveProcessLimit = 1,
},
ProcessMemoryLimit = (UIntPtr)(ulong)processMemoryLimitBytes,
};
if (!SetInformationJobObject(handle, JobObjectExtendedLimitInformation, ref info,
(uint)Marshal.SizeOf<JOBOBJECT_EXTENDED_LIMIT_INFORMATION>()))
throw new Win32Exception(Marshal.GetLastWin32Error(), "SetInformationJobObject failed");
if (!AssignProcessToJobObject(handle, process.Handle))
throw new Win32Exception(Marshal.GetLastWin32Error(), "AssignProcessToJobObject failed");
return job;
}
catch
{
job.Dispose();
throw;
}
}
/// <summary>Closes the job handle, which kills every process still in it.</summary>
public void Dispose() => _handle.Dispose();
[StructLayout(LayoutKind.Sequential)]
private struct JOBOBJECT_BASIC_LIMIT_INFORMATION
{
public long PerProcessUserTimeLimit;
public long PerJobUserTimeLimit;
public uint LimitFlags;
public UIntPtr MinimumWorkingSetSize;
public UIntPtr MaximumWorkingSetSize;
public uint ActiveProcessLimit;
public UIntPtr Affinity;
public uint PriorityClass;
public uint SchedulingClass;
}
[StructLayout(LayoutKind.Sequential)]
private struct IO_COUNTERS
{
public ulong ReadOperationCount;
public ulong WriteOperationCount;
public ulong OtherOperationCount;
public ulong ReadTransferCount;
public ulong WriteTransferCount;
public ulong OtherTransferCount;
}
[StructLayout(LayoutKind.Sequential)]
private struct JOBOBJECT_EXTENDED_LIMIT_INFORMATION
{
public JOBOBJECT_BASIC_LIMIT_INFORMATION BasicLimitInformation;
public IO_COUNTERS IoInfo;
public UIntPtr ProcessMemoryLimit;
public UIntPtr JobMemoryLimit;
public UIntPtr PeakProcessMemoryUsed;
public UIntPtr PeakJobMemoryUsed;
}
#pragma warning disable SYSLIB1054 // DllImport kept: LibraryImport would need AllowUnsafeBlocks for three calls.
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
private static extern SafeFileHandle CreateJobObjectW(IntPtr securityAttributes, string? name);
[DllImport("kernel32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool SetInformationJobObject(SafeFileHandle job, int infoClass,
ref JOBOBJECT_EXTENDED_LIMIT_INFORMATION info, uint length);
[DllImport("kernel32.dll", SetLastError = true)]
[return: MarshalAs(UnmanagedType.Bool)]
private static extern bool AssignProcessToJobObject(SafeFileHandle job, IntPtr process);
#pragma warning restore SYSLIB1054
}
@@ -0,0 +1,699 @@
using System.Collections.Concurrent;
using System.Diagnostics;
using System.IO.Pipes;
using System.Security.Cryptography;
using System.Text;
using System.Threading.Channels;
using Serilog;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
/// <summary>How a worker process ended.</summary>
internal enum WorkerExitReason
{
/// <summary>It exited by itself with code 0 (a Shutdown).</summary>
Clean,
/// <summary>The host killed it: a script crossed its hard deadline (<see cref="WorkerExit.CulpritKey"/>).</summary>
KilledDeadline,
/// <summary>The host killed it: no frame for three heartbeat intervals.</summary>
KilledHeartbeat,
/// <summary>The host killed it: a compile did not finish in time (<see cref="WorkerExit.CulpritKey"/>).</summary>
KilledLoadTimeout,
/// <summary>It died by itself (OOM, heap limit, fail-fast, external kill).</summary>
Crash,
/// <summary>A framing or handshake violation.</summary>
ProtocolFault,
/// <summary>The runner is shutting down.</summary>
Disposed,
}
/// <summary>The end of one worker process.</summary>
/// <param name="Reason">Why it ended.</param>
/// <param name="CulpritKey">The scoped key the host blamed, for a deadline or load kill.</param>
/// <param name="ExitCode">The OS exit code, when known.</param>
/// <param name="InFlightKeys">The scoped keys that had an evaluation outstanding when it ended.</param>
internal sealed record WorkerExit(WorkerExitReason Reason, string? CulpritKey, int? ExitCode, IReadOnlyList<string> InFlightKeys);
/// <summary>What the host needs to launch one worker.</summary>
/// <param name="Shard">The shard label (<c>0</c>, <c>1</c>, … or <c>iso</c>).</param>
/// <param name="WorkerPath">The worker entry assembly (<c>.dll</c>, run through the dotnet muxer) or apphost.</param>
/// <param name="HeapLimitBytes">The worker's <c>DOTNET_GCHeapHardLimit</c>.</param>
/// <param name="EvalThreads">The worker scheduler's width (1 for the isolation worker).</param>
/// <param name="HardGrace">The worker watchdog's grace past the soft deadline.</param>
/// <param name="HeartbeatInterval">How often the worker sends a heartbeat.</param>
/// <param name="HandshakeTimeout">How long a launch may take to reach Ready.</param>
internal sealed record WorkerLaunchSpec(
string Shard,
string WorkerPath,
long HeapLimitBytes,
int EvalThreads,
TimeSpan HardGrace,
TimeSpan HeartbeatInterval,
TimeSpan HandshakeTimeout);
/// <summary>
/// One live ScriptWorker process and its pipe (lmxopcua#780, design §3–§6): launch with a cleared, allow-listed
/// environment, a random pipe that accepts exactly one client, a nonce handshake, a frame reader and a coalescing
/// writer, heartbeat supervision, and the kill. It never restarts itself; <see cref="WorkerShard"/> does that.
/// </summary>
internal sealed class WorkerConnection : IAsyncDisposable
{
/// <summary>
/// Environment variables a worker inherits from the host (design §6): the runtime's own knobs, the temp
/// directories and the minimum a Windows process needs. Everything else — notably <c>ServerHistorian__ApiKey</c>,
/// ConfigDb connection strings and every other <c>Section__Key</c> override — is dropped.
/// </summary>
internal static readonly IReadOnlyList<string> InheritedEnvironment =
[
"PATH", "TEMP", "TMP", "TMPDIR", "HOME", "SystemRoot", "SYSTEMROOT", "windir", "SystemDrive", "ComSpec",
"PATHEXT", "NUMBER_OF_PROCESSORS", "PROCESSOR_ARCHITECTURE", "LOCALAPPDATA",
];
private readonly Process _process;
private readonly NamedPipeServerStream _pipe;
private readonly IDisposable? _job;
private readonly string _workDir;
private readonly ILogger _log;
private readonly Action<LogBatch> _onLogs;
private readonly Channel<Envelope> _outbound = Channel.CreateUnbounded<Envelope>(new UnboundedChannelOptions { SingleReader = true });
private readonly ConcurrentDictionary<ulong, PendingEval> _pending = new();
private readonly ConcurrentDictionary<ulong, TaskCompletionSource<LoadResult>> _loads = new();
private readonly ConcurrentDictionary<string, LoadEntry> _loaded = new(StringComparer.Ordinal);
private readonly TaskCompletionSource<WorkerExit> _exited = new(TaskCreationOptions.RunContinuationsAsynchronously);
private readonly CancellationTokenSource _stopping = new();
private readonly TimeSpan _heartbeatInterval;
private TaskCompletionSource<DiagnosticsReply>? _diag;
private (WorkerExitReason Reason, string? Culprit)? _killedFor;
private long _lastFrameTimestamp;
private long _seq;
private long _nextEvalId;
private long _nextLoadId;
private int _maxFrameBytes = ScriptWorkerFrameCodec.DefaultMaxFrameBytes;
private int _disposed;
private WorkerConnection(string shard, Process process, NamedPipeServerStream pipe, IDisposable? job, string workDir,
ILogger log, Action<LogBatch> onLogs, TimeSpan heartbeatInterval)
{
Shard = shard;
_process = process;
_pipe = pipe;
_job = job;
_workDir = workDir;
_log = log;
_onLogs = onLogs;
_heartbeatInterval = heartbeatInterval;
Pid = process.Id;
}
/// <summary>The shard label.</summary>
public string Shard { get; }
/// <summary>The worker's process id.</summary>
public int Pid { get; }
/// <summary>Completes when the worker has ended, with why.</summary>
public Task<WorkerExit> Exited => _exited.Task;
/// <summary>Whether the worker is still serving.</summary>
public bool IsAlive => !_exited.Task.IsCompleted && !_stopping.IsCancellationRequested;
/// <summary><see cref="Stopwatch.GetTimestamp"/> when the handshake completed.</summary>
public long ReadyTimestamp { get; private set; }
/// <summary>Evaluations currently outstanding on this worker.</summary>
public int InFlightCount => _pending.Count;
/// <summary>Launches a worker and completes the handshake.</summary>
/// <param name="spec">What to launch.</param>
/// <param name="log">Host diagnostics (stderr lines, exits).</param>
/// <param name="onLogs">Receives the worker's forwarded <c>ctx.Logger</c> batches.</param>
/// <param name="ct">Cancellation.</param>
/// <returns>A ready connection.</returns>
public static async Task<WorkerConnection> StartAsync(WorkerLaunchSpec spec, ILogger log, Action<LogBatch> onLogs, CancellationToken ct)
{
var pipeName = "otopcua-sw-" + Convert.ToHexString(RandomNumberGenerator.GetBytes(12)).ToLowerInvariant();
var nonce = Convert.ToHexString(RandomNumberGenerator.GetBytes(32));
var workDir = Path.Combine(Path.GetTempPath(), "otopcua-scriptworker", pipeName);
Directory.CreateDirectory(workDir);
// One instance, current user only: the pipe accepts exactly one connection, from this account.
var pipe = new NamedPipeServerStream(pipeName, PipeDirection.InOut, maxNumberOfServerInstances: 1,
PipeTransmissionMode.Byte, PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly);
Process? process = null;
IDisposable? job = null;
try
{
var psi = BuildStartInfo(spec, pipeName, nonce, workDir);
process = new Process { StartInfo = psi, EnableRaisingEvents = true };
if (!process.Start()) throw new InvalidOperationException("the worker process did not start");
if (OperatingSystem.IsWindows())
{
// Native and JIT memory backstop above the managed heap limit.
job = WindowsJobObject.Assign(process, spec.HeapLimitBytes + 256L * 1024 * 1024);
}
var shardLabel = spec.Shard;
process.ErrorDataReceived += (_, e) =>
{
if (e.Data is { Length: > 0 } line) log.Warning("ScriptWorker[{Shard}] {WorkerStderr}", shardLabel, line);
};
process.OutputDataReceived += (_, e) =>
{
if (e.Data is { Length: > 0 } line) log.Debug("ScriptWorker[{Shard}] stdout {WorkerStdout}", shardLabel, line);
};
process.BeginErrorReadLine();
process.BeginOutputReadLine();
using var handshakeCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
handshakeCts.CancelAfter(spec.HandshakeTimeout);
var exitedEarly = process.WaitForExitAsync(handshakeCts.Token);
var accept = pipe.WaitForConnectionAsync(handshakeCts.Token);
if (await Task.WhenAny(accept, exitedEarly).ConfigureAwait(false) != accept)
throw new InvalidOperationException(
$"the worker exited (code {SafeExitCode(process)}) before connecting to its pipe");
await accept.ConfigureAwait(false);
var hello = new Hello
{
Nonce = nonce,
MaxFrameBytes = ScriptWorkerFrameCodec.DefaultMaxFrameBytes,
EvalThreads = (uint)spec.EvalThreads,
QueueCapacity = 4096,
HardGraceMs = (uint)Math.Clamp(spec.HardGrace.TotalMilliseconds, 1, uint.MaxValue),
HeartbeatIntervalMs = (uint)spec.HeartbeatInterval.TotalMilliseconds,
MaxLogLines = 1000,
};
await HandshakeAsync(pipe, hello, nonce, process.Id, handshakeCts.Token).ConfigureAwait(false);
var connection = new WorkerConnection(spec.Shard, process, pipe, job, workDir, log, onLogs, spec.HeartbeatInterval);
connection.Begin();
return connection;
}
catch
{
try { if (process is { HasExited: false }) process.Kill(entireProcessTree: true); } catch { /* best effort */ }
job?.Dispose();
process?.Dispose();
await pipe.DisposeAsync().ConfigureAwait(false);
TryDeleteDirectory(workDir);
throw;
}
}
/// <summary>
/// The host half of the handshake: send <paramref name="hello"/>, require a <c>Ready</c> as the first frame,
/// echoing <paramref name="expectedNonce"/> (constant-time compare) from <paramref name="expectedPid"/>.
/// </summary>
/// <param name="stream">The accepted pipe.</param>
/// <param name="hello">The Hello to send.</param>
/// <param name="expectedNonce">The nonce put in the worker's environment.</param>
/// <param name="expectedPid">The launched process id; null skips the check.</param>
/// <param name="ct">Cancellation (the handshake timeout).</param>
/// <returns>The worker's Ready.</returns>
/// <exception cref="ScriptWorkerProtocolException">Anything but a matching Ready.</exception>
internal static async Task<Ready> HandshakeAsync(Stream stream, Hello hello, string expectedNonce, int? expectedPid, CancellationToken ct)
{
await ScriptWorkerFrameCodec.WriteAsync(stream, new Envelope
{
ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion,
Seq = 1,
Hello = hello,
}, ScriptWorkerFrameCodec.DefaultMaxFrameBytes, ct).ConfigureAwait(false);
await stream.FlushAsync(ct).ConfigureAwait(false);
var reply = await ScriptWorkerFrameCodec.ReadAsync(stream, ScriptWorkerFrameCodec.DefaultMaxFrameBytes, ct).ConfigureAwait(false)
?? throw new ScriptWorkerProtocolException("the worker closed the pipe during the handshake");
if (reply.BodyCase != Envelope.BodyOneofCase.Ready)
throw new ScriptWorkerProtocolException($"expected Ready, got {reply.BodyCase}");
var expected = Encoding.UTF8.GetBytes(expectedNonce);
var actual = Encoding.UTF8.GetBytes(reply.Ready.Nonce ?? string.Empty);
if (!CryptographicOperations.FixedTimeEquals(expected, actual))
throw new ScriptWorkerProtocolException("the worker's nonce does not match the launch nonce");
if (expectedPid is { } pid && reply.Ready.Pid != pid)
throw new ScriptWorkerProtocolException($"the pipe client reports pid {reply.Ready.Pid}, not the launched {pid}");
return reply.Ready;
}
private static ProcessStartInfo BuildStartInfo(WorkerLaunchSpec spec, string pipeName, string nonce, string workDir)
{
var psi = new ProcessStartInfo
{
UseShellExecute = false,
CreateNoWindow = true,
RedirectStandardError = true,
RedirectStandardOutput = true,
RedirectStandardInput = false,
WorkingDirectory = workDir,
};
if (spec.WorkerPath.EndsWith(".dll", StringComparison.OrdinalIgnoreCase))
{
psi.FileName = ResolveMuxer();
psi.ArgumentList.Add("exec");
psi.ArgumentList.Add(spec.WorkerPath);
}
else
{
psi.FileName = spec.WorkerPath;
}
// Cleared, then repopulated from an allow-list (design §6). ProcessStartInfo.Environment starts as a copy of
// the host's environment; ScadaBridge's sandbox inherits it whole, which this design rejects.
var inherited = psi.Environment.ToDictionary(kv => kv.Key, kv => kv.Value, StringComparer.Ordinal);
psi.Environment.Clear();
foreach (var (name, value) in inherited)
{
if (value is null) continue;
if (InheritedEnvironment.Contains(name, StringComparer.OrdinalIgnoreCase)
|| (name.StartsWith("DOTNET_", StringComparison.OrdinalIgnoreCase)
&& !name.StartsWith("DOTNET_GCHeap", StringComparison.OrdinalIgnoreCase)))
psi.Environment[name] = value;
}
psi.Environment["DOTNET_GCHeapHardLimit"] = "0x" + spec.HeapLimitBytes.ToString("X", System.Globalization.CultureInfo.InvariantCulture);
psi.Environment["DOTNET_gcServer"] = "0";
psi.Environment["DOTNET_CLI_TELEMETRY_OPTOUT"] = "1";
psi.Environment[ScriptWorkerEnvironment.PipeName] = pipeName;
psi.Environment[ScriptWorkerEnvironment.Nonce] = nonce;
psi.Environment[ScriptWorkerEnvironment.ParentPid] = Environment.ProcessId.ToString(System.Globalization.CultureInfo.InvariantCulture);
psi.Environment[ScriptWorkerEnvironment.Shard] = spec.Shard;
return psi;
}
/// <summary>The dotnet muxer that runs a framework-dependent worker <c>.dll</c>.</summary>
/// <returns>The muxer path, or plain <c>dotnet</c> (resolved on PATH) as a last resort.</returns>
internal static string ResolveMuxer()
{
var hostPath = Environment.GetEnvironmentVariable("DOTNET_HOST_PATH");
if (!string.IsNullOrEmpty(hostPath) && File.Exists(hostPath)) return hostPath;
var processPath = Environment.ProcessPath;
if (processPath is not null
&& string.Equals(Path.GetFileNameWithoutExtension(processPath), "dotnet", StringComparison.OrdinalIgnoreCase))
return processPath;
var root = Environment.GetEnvironmentVariable("DOTNET_ROOT");
if (!string.IsNullOrEmpty(root))
{
var candidate = Path.Combine(root, OperatingSystem.IsWindows() ? "dotnet.exe" : "dotnet");
if (File.Exists(candidate)) return candidate;
}
return "dotnet";
}
private void Begin()
{
ReadyTimestamp = Stopwatch.GetTimestamp();
Volatile.Write(ref _lastFrameTimestamp, ReadyTimestamp);
_ = Task.Run(ReadLoopAsync);
_ = Task.Run(WriteLoopAsync);
_ = Task.Run(HeartbeatMonitorAsync);
_ = Task.Run(WatchProcessAsync);
}
/// <summary>Ensures <paramref name="key"/> is compiled in this worker at <paramref name="sourceHash"/>.</summary>
/// <param name="key">The scoped key.</param>
/// <param name="kind">The context kind.</param>
/// <param name="source">The source.</param>
/// <param name="sourceHash">SHA-256 of the source.</param>
/// <param name="inputPaths">The slot order.</param>
/// <returns>The load outcome and the slot map; null when the worker ended first.</returns>
public Task<LoadEntry?> EnsureLoadedAsync(string key, ContextKind kind, string source, byte[] sourceHash, IReadOnlyList<string> inputPaths)
{
var hashHex = Convert.ToHexString(sourceHash);
if (_loaded.TryGetValue(key, out var existing) && existing.SourceHash == hashHex && existing.SameSlots(inputPaths))
return existing.Ready;
var entry = new LoadEntry(hashHex, inputPaths);
_loaded[key] = entry;
var loadId = (ulong)Interlocked.Increment(ref _nextLoadId);
var tcs = new TaskCompletionSource<LoadResult>(TaskCreationOptions.RunContinuationsAsynchronously);
_loads[loadId] = tcs;
var load = new LoadScripts { LoadId = loadId };
var script = new ScriptSource { Key = key, Kind = kind, Source = source, SourceSha256 = Google.Protobuf.ByteString.CopyFrom(sourceHash) };
script.InputPaths.AddRange(inputPaths);
load.Scripts.Add(script);
if (!Send(new Envelope { Load = load })) { _loads.TryRemove(loadId, out _); entry.Complete(null); return entry.Ready; }
_ = tcs.Task.ContinueWith(t =>
{
if (t.IsCompletedSuccessfully)
{
var outcome = t.Result.Outcomes.FirstOrDefault(o => o.Key == key);
entry.Complete(outcome ?? new LoadOutcome { Key = key, Ok = false, Error = new ScriptError { Kind = ScriptError.Types.Kind.Threw, Message = "no load outcome" } });
if (outcome is { Ok: false }) _loaded.TryRemove(new KeyValuePair<string, LoadEntry>(key, entry));
}
else
{
entry.Complete(null);
_loaded.TryRemove(new KeyValuePair<string, LoadEntry>(key, entry));
}
}, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
return entry.Ready;
}
/// <summary>Forgets a key's compiled form here and in the worker.</summary>
/// <param name="keys">The scoped keys.</param>
public void Unload(IReadOnlyCollection<string> keys)
{
var unload = new Unload();
foreach (var key in keys)
if (_loaded.TryRemove(key, out _)) unload.Keys.Add(key);
if (unload.Keys.Count > 0) Send(new Envelope { Unload = unload });
}
/// <summary>Submits one evaluation. Its <see cref="PendingEval.Completion"/> yields null when the worker ends first.</summary>
/// <param name="key">The scoped key.</param>
/// <param name="request">The request (its id is assigned here).</param>
/// <returns>The pending evaluation.</returns>
public PendingEval Submit(string key, EvalRequest request)
{
request.EvalId = (ulong)Interlocked.Increment(ref _nextEvalId);
var pending = new PendingEval(key, request.EvalId);
_pending[request.EvalId] = pending;
var batch = new EvalBatch();
batch.Items.Add(request);
if (!Send(new Envelope { Eval = batch }) || _exited.Task.IsCompleted)
{
if (_pending.TryRemove(request.EvalId, out _)) pending.Complete(null);
}
return pending;
}
/// <summary>Cancels a run's <c>ctx.CancellationToken</c> in the worker (sent at the soft deadline).</summary>
/// <param name="pending">The evaluation.</param>
public void Cancel(PendingEval pending)
{
if (!pending.Completion.IsCompleted) Send(new Envelope { Cancel = new Protocol.V1.Cancel { EvalId = pending.EvalId } });
}
/// <summary>Asks the worker for its process diagnostics (loaded assemblies, environment names).</summary>
/// <param name="timeout">How long to wait.</param>
/// <returns>The reply.</returns>
public async Task<DiagnosticsReply> DiagnosticsAsync(TimeSpan timeout)
{
var tcs = new TaskCompletionSource<DiagnosticsReply>(TaskCreationOptions.RunContinuationsAsynchronously);
Interlocked.Exchange(ref _diag, tcs);
Send(new Envelope { DiagRequest = new DiagnosticsRequest() });
return await tcs.Task.WaitAsync(timeout).ConfigureAwait(false);
}
/// <summary>Kills the worker (process tree). The first reason given is the one recorded.</summary>
/// <param name="reason">Why.</param>
/// <param name="culpritKey">The scoped key blamed, if any.</param>
public void Kill(WorkerExitReason reason, string? culpritKey)
{
lock (_pending)
{
_killedFor ??= (reason, culpritKey);
}
try
{
if (!_process.HasExited) _process.Kill(entireProcessTree: true);
}
catch (InvalidOperationException) { /* already gone */ }
catch (System.ComponentModel.Win32Exception ex)
{
_log.Warning(ex, "ScriptWorker[{Shard}] kill of pid {Pid} failed", Shard, Pid);
}
}
/// <summary>Asks the worker to stop, waits briefly, then kills it.</summary>
/// <returns>A task that completes when the worker has ended.</returns>
public async ValueTask DisposeAsync()
{
// Both the shard (on exit) and the runner (on shutdown) dispose a connection; only the first does the work.
if (Interlocked.Exchange(ref _disposed, 1) != 0) return;
if (!_exited.Task.IsCompleted)
{
lock (_pending) { _killedFor ??= (WorkerExitReason.Disposed, null); }
Send(new Envelope { Shutdown = new Shutdown() });
try
{
await _process.WaitForExitAsync().WaitAsync(TimeSpan.FromSeconds(1)).ConfigureAwait(false);
}
catch (TimeoutException)
{
Kill(WorkerExitReason.Disposed, null);
}
try { await _exited.Task.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false); } catch (TimeoutException) { }
}
_stopping.Cancel();
_outbound.Writer.TryComplete();
await _pipe.DisposeAsync().ConfigureAwait(false);
_job?.Dispose();
_process.Dispose();
TryDeleteDirectory(_workDir);
}
/// <summary>Test hook: closes the host end of the pipe without touching the process, as a dying host would.</summary>
internal void SeverPipeForTest() => _pipe.Dispose();
private bool Send(Envelope envelope)
{
envelope.ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion;
return _outbound.Writer.TryWrite(envelope);
}
private async Task WriteLoopAsync()
{
var reader = _outbound.Reader;
try
{
while (await reader.WaitToReadAsync(_stopping.Token).ConfigureAwait(false))
{
// Opportunistic batching (design §4.5): every evaluation queued right now goes out as ONE EvalBatch.
EvalBatch? batch = null;
while (reader.TryRead(out var envelope))
{
if (envelope.BodyCase == Envelope.BodyOneofCase.Eval)
{
(batch ??= new EvalBatch()).Items.AddRange(envelope.Eval.Items);
continue;
}
envelope.Seq = (ulong)Interlocked.Increment(ref _seq);
await ScriptWorkerFrameCodec.WriteAsync(_pipe, envelope, _maxFrameBytes, _stopping.Token).ConfigureAwait(false);
}
if (batch is not null)
{
await ScriptWorkerFrameCodec.WriteAsync(_pipe, new Envelope
{
ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion,
Seq = (ulong)Interlocked.Increment(ref _seq),
Eval = batch,
}, _maxFrameBytes, _stopping.Token).ConfigureAwait(false);
}
await _pipe.FlushAsync(_stopping.Token).ConfigureAwait(false);
}
}
catch (OperationCanceledException) { }
catch (Exception ex) when (ex is IOException or ObjectDisposedException or ScriptWorkerProtocolException)
{
if (!_stopping.IsCancellationRequested)
_log.Debug(ex, "ScriptWorker[{Shard}] pipe write ended", Shard);
Kill(WorkerExitReason.Crash, null);
}
}
private async Task ReadLoopAsync()
{
WorkerExitReason? fault = null;
try
{
while (true)
{
var frame = await ScriptWorkerFrameCodec.ReadAsync(_pipe, _maxFrameBytes, _stopping.Token).ConfigureAwait(false);
if (frame is null) break;
Volatile.Write(ref _lastFrameTimestamp, Stopwatch.GetTimestamp());
switch (frame.BodyCase)
{
case Envelope.BodyOneofCase.Results:
foreach (var result in frame.Results.Items)
{
// An eval id with no outstanding request is a protocol violation (design §6 "what the host
// trusts"), except for one that was already completed by a kill race — ignored then.
if (_pending.TryRemove(result.EvalId, out var pending)) pending.Complete(result);
else if (result.EvalId == 0 || result.EvalId > (ulong)Interlocked.Read(ref _nextEvalId))
throw new ScriptWorkerProtocolException($"result for unknown eval id {result.EvalId}");
}
break;
case Envelope.BodyOneofCase.LoadResult:
if (_loads.TryRemove(frame.LoadResult.LoadId, out var load)) load.TrySetResult(frame.LoadResult);
else throw new ScriptWorkerProtocolException($"load result for unknown load id {frame.LoadResult.LoadId}");
break;
case Envelope.BodyOneofCase.Logs:
try { _onLogs(frame.Logs); }
catch (Exception ex) { _log.Warning(ex, "ScriptWorker[{Shard}] log forwarding threw", Shard); }
break;
case Envelope.BodyOneofCase.Heartbeat:
break;
case Envelope.BodyOneofCase.DiagReply:
Volatile.Read(ref _diag)?.TrySetResult(frame.DiagReply);
break;
case Envelope.BodyOneofCase.Fault:
_log.Warning("ScriptWorker[{Shard}] reported a fault: {Fault}", Shard, frame.Fault.Message);
break;
default:
throw new ScriptWorkerProtocolException($"unexpected frame {frame.BodyCase} from the worker");
}
}
}
catch (OperationCanceledException) { }
catch (ScriptWorkerProtocolException ex)
{
_log.Warning("ScriptWorker[{Shard}] protocol violation: {Message}; killing pid {Pid}", Shard, ex.Message, Pid);
fault = WorkerExitReason.ProtocolFault;
}
catch (Exception ex) when (ex is IOException or ObjectDisposedException)
{
}
// The pipe is gone: the worker is dead or dying. Make sure, then report.
try
{
if (fault is { } f) Kill(f, null);
try
{
await _process.WaitForExitAsync().WaitAsync(TimeSpan.FromSeconds(2)).ConfigureAwait(false);
}
catch (TimeoutException)
{
Kill(WorkerExitReason.Crash, null);
try { await _process.WaitForExitAsync().WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false); } catch (TimeoutException) { }
}
}
catch (InvalidOperationException)
{
// The process object was disposed by a concurrent shutdown: the worker is already gone.
}
Finish();
}
private async Task WatchProcessAsync()
{
try { await _process.WaitForExitAsync(_stopping.Token).ConfigureAwait(false); }
catch (OperationCanceledException) { return; }
catch (InvalidOperationException) { return; }
// The reader normally finishes first on EOF; this covers a worker that died while the pipe stayed open.
await Task.Delay(TimeSpan.FromSeconds(2)).ConfigureAwait(false);
Finish();
}
private async Task HeartbeatMonitorAsync()
{
using var timer = new PeriodicTimer(_heartbeatInterval);
try
{
while (await timer.WaitForNextTickAsync(_stopping.Token).ConfigureAwait(false))
{
if (_exited.Task.IsCompleted) return;
var silent = Stopwatch.GetElapsedTime(Volatile.Read(ref _lastFrameTimestamp));
if (silent > _heartbeatInterval * 3)
{
_log.Warning("ScriptWorker[{Shard}] pid {Pid} sent nothing for {Silent} ms; killing it", Shard, Pid, (long)silent.TotalMilliseconds);
Kill(WorkerExitReason.KilledHeartbeat, null);
return;
}
}
}
catch (OperationCanceledException) { }
}
private void Finish()
{
int? code = null;
try { if (_process.HasExited) code = _process.ExitCode; } catch (InvalidOperationException) { }
(WorkerExitReason Reason, string? Culprit)? killed;
lock (_pending) { killed = _killedFor; }
var reason = killed?.Reason ?? (code == 0 ? WorkerExitReason.Clean : WorkerExitReason.Crash);
var inFlight = _pending.Values.Select(p => p.Key).Distinct(StringComparer.Ordinal).ToArray();
if (!_exited.TrySetResult(new WorkerExit(reason, killed?.Culprit, code, inFlight))) return;
// Every outstanding evaluation and load ends now; null tells the waiter "the worker went away".
foreach (var id in _pending.Keys.ToArray())
if (_pending.TryRemove(id, out var pending)) pending.Complete(null);
foreach (var id in _loads.Keys.ToArray())
if (_loads.TryRemove(id, out var load)) load.TrySetCanceled();
Volatile.Read(ref _diag)?.TrySetCanceled();
_outbound.Writer.TryComplete();
}
private static string SafeExitCode(Process process)
{
try { return process.HasExited ? process.ExitCode.ToString(System.Globalization.CultureInfo.InvariantCulture) : "?"; }
catch (InvalidOperationException) { return "?"; }
}
private static void TryDeleteDirectory(string dir)
{
try { if (Directory.Exists(dir)) Directory.Delete(dir, recursive: true); } catch { /* temp dir; best effort */ }
}
/// <summary>A key's compiled form in this worker.</summary>
internal sealed class LoadEntry
{
private readonly TaskCompletionSource<LoadEntry?> _ready = new(TaskCreationOptions.RunContinuationsAsynchronously);
public LoadEntry(string sourceHash, IReadOnlyList<string> inputPaths)
{
SourceHash = sourceHash;
InputPaths = inputPaths;
var slots = new Dictionary<string, uint>(inputPaths.Count, StringComparer.Ordinal);
for (var i = 0; i < inputPaths.Count; i++) slots[inputPaths[i]] = (uint)i;
Slots = slots;
}
public string SourceHash { get; }
public IReadOnlyList<string> InputPaths { get; }
/// <summary>Path → slot in the registered order.</summary>
public IReadOnlyDictionary<string, uint> Slots { get; }
/// <summary>The worker's verdict (null until it arrives).</summary>
public LoadOutcome? Outcome { get; private set; }
public Task<LoadEntry?> Ready => _ready.Task;
public bool SameSlots(IReadOnlyList<string> inputPaths)
{
if (inputPaths.Count != InputPaths.Count) return false;
for (var i = 0; i < inputPaths.Count; i++)
if (!string.Equals(inputPaths[i], InputPaths[i], StringComparison.Ordinal)) return false;
return true;
}
public void Complete(LoadOutcome? outcome)
{
Outcome = outcome;
_ready.TrySetResult(outcome is null ? null : this);
}
}
}
/// <summary>One evaluation outstanding on a worker.</summary>
internal sealed class PendingEval
{
private readonly TaskCompletionSource<EvalResult?> _completion = new(TaskCreationOptions.RunContinuationsAsynchronously);
public PendingEval(string key, ulong evalId)
{
Key = key;
EvalId = evalId;
}
/// <summary>The scoped key.</summary>
public string Key { get; }
/// <summary>The eval id on the wire.</summary>
public ulong EvalId { get; }
/// <summary>The worker's result, or null when the worker ended first.</summary>
public Task<EvalResult?> Completion => _completion.Task;
public void Complete(EvalResult? result) => _completion.TrySetResult(result);
}
@@ -0,0 +1,249 @@
using System.Diagnostics;
using System.Diagnostics.Metrics;
using Serilog;
using ZB.MOM.WW.OtOpcUa.Commons.Observability;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
/// <summary>
/// Supervises one shard's worker process (design §3.3, §5.3.5): starts it lazily on first use, reports its end to
/// the runner (for attribution), and restarts it after an exponential backoff — 250 ms doubling to a 30 s cap with
/// ±20 % jitter, reset once a worker has stayed up for 60 s. It never runs a script in the host when the worker
/// cannot start (fail closed, §14 #4): <see cref="GetReadyAsync"/> returns null and the caller reports
/// <c>BadResourceUnavailable</c>.
/// </summary>
internal sealed class WorkerShard : IAsyncDisposable
{
internal static readonly TimeSpan InitialBackoff = TimeSpan.FromMilliseconds(250);
internal static readonly TimeSpan MaxBackoff = TimeSpan.FromSeconds(30);
internal static readonly TimeSpan HealthyResetAfter = TimeSpan.FromSeconds(60);
private static readonly Counter<long> StartsCounter = OtOpcUaTelemetry.Meter.CreateCounter<long>(
"otopcua.script.worker.starts", unit: "{process}", description: "ScriptWorker processes started (tag: shard).");
private static readonly Counter<long> ExitsCounter = OtOpcUaTelemetry.Meter.CreateCounter<long>(
"otopcua.script.worker.exits", unit: "{process}",
description: "ScriptWorker processes ended (tags: shard, reason = clean | killed_deadline | killed_heartbeat | killed_load | crash | protocol | disposed).");
private readonly Func<CancellationToken, Task<WorkerConnection>> _launch;
private readonly Action<WorkerShard, WorkerExit> _onExit;
private readonly ILogger _log;
private readonly object _gate = new();
private readonly Random _jitter = new();
private WorkerConnection? _current;
private Task<WorkerConnection?>? _starting;
private long _backoffUntil;
private int _consecutiveFailures;
private bool _disposed;
public WorkerShard(string label, bool isolation, Func<CancellationToken, Task<WorkerConnection>> launch,
Action<WorkerShard, WorkerExit> onExit, ILogger log)
{
Label = label;
IsIsolation = isolation;
_launch = launch;
_onExit = onExit;
_log = log;
// The isolation worker runs one evaluation at a time, so a crash there names exactly one key (design §5.3.3).
IsolationGate = isolation ? new SemaphoreSlim(1, 1) : null;
}
/// <summary>The shard label (<c>0</c>…<c>K-1</c>, or <c>iso</c>).</summary>
public string Label { get; }
/// <summary>Whether this is the isolation ("penalty box") worker.</summary>
public bool IsIsolation { get; }
/// <summary>Serialises evaluations on the isolation worker; null on a normal shard.</summary>
public SemaphoreSlim? IsolationGate { get; }
/// <summary>The live worker, if any (diagnostics / tests).</summary>
public WorkerConnection? Current
{
get { lock (_gate) return _current is { IsAlive: true } c ? c : null; }
}
/// <summary>Consecutive start failures and crashes since the last healthy period (diagnostics / tests).</summary>
public int ConsecutiveFailures
{
get { lock (_gate) return _consecutiveFailures; }
}
/// <summary>
/// The ready worker, starting one if needed. Waits at most <paramref name="grace"/> — through a start in progress
/// or a backoff — and returns null if no worker is ready by then.
/// </summary>
/// <param name="grace">How long to wait.</param>
/// <param name="ct">Cancellation.</param>
/// <returns>The worker, or null.</returns>
public async Task<WorkerConnection?> GetReadyAsync(TimeSpan grace, CancellationToken ct)
{
var deadline = Stopwatch.GetTimestamp() + (long)(grace.TotalSeconds * Stopwatch.Frequency);
while (true)
{
Task<WorkerConnection?> starting;
TimeSpan wait;
lock (_gate)
{
if (_disposed) return null;
if (_current is { IsAlive: true } live) return live;
var now = Stopwatch.GetTimestamp();
if (_starting is null && now >= _backoffUntil)
{
_starting = StartCoreAsync();
}
if (_starting is not null)
{
starting = _starting;
wait = TimeSpan.Zero;
}
else
{
starting = null!;
wait = Stopwatch.GetElapsedTime(now, _backoffUntil);
}
}
var remaining = Stopwatch.GetElapsedTime(Stopwatch.GetTimestamp(), deadline);
if (remaining <= TimeSpan.Zero) return null;
if (starting is not null)
{
try
{
var started = await starting.WaitAsync(remaining, ct).ConfigureAwait(false);
if (started is { IsAlive: true }) return started;
}
catch (TimeoutException)
{
return null;
}
continue;
}
if (wait > remaining) return null;
await Task.Delay(wait, ct).ConfigureAwait(false);
}
}
private async Task<WorkerConnection?> StartCoreAsync()
{
await Task.Yield();
WorkerConnection? connection = null;
try
{
connection = await _launch(CancellationToken.None).ConfigureAwait(false);
StartsCounter.Add(1, new KeyValuePair<string, object?>("shard", Label));
_log.Information("ScriptWorker[{Shard}] started, pid {Pid}", Label, connection.Pid);
var started = connection;
_ = started.Exited.ContinueWith(t => OnExited(started, t.Result), CancellationToken.None,
TaskContinuationOptions.ExecuteSynchronously | TaskContinuationOptions.OnlyOnRanToCompletion, TaskScheduler.Default);
lock (_gate)
{
_starting = null;
if (_disposed)
{
_ = started.DisposeAsync().AsTask();
return null;
}
_current = started;
}
return started;
}
catch (Exception ex)
{
_log.Error(ex, "ScriptWorker[{Shard}] failed to start; scripts on it report runner unavailable until it does", Label);
lock (_gate)
{
_starting = null;
ScheduleBackoffLocked();
}
if (connection is not null) await connection.DisposeAsync().ConfigureAwait(false);
return null;
}
}
private void OnExited(WorkerConnection connection, WorkerExit exit)
{
ExitsCounter.Add(1, new KeyValuePair<string, object?>("shard", Label),
new KeyValuePair<string, object?>("reason", MetricReason(exit.Reason)));
lock (_gate)
{
if (ReferenceEquals(_current, connection)) _current = null;
if (!_disposed)
{
if (Stopwatch.GetElapsedTime(connection.ReadyTimestamp) >= HealthyResetAfter) _consecutiveFailures = 0;
ScheduleBackoffLocked();
}
}
if (exit.Reason != WorkerExitReason.Disposed)
{
_log.Warning("ScriptWorker[{Shard}] pid {Pid} ended: {Reason} (exit code {ExitCode}, culprit {Culprit}, {InFlight} evaluation(s) in flight)",
Label, connection.Pid, exit.Reason, exit.ExitCode, exit.CulpritKey ?? "none", exit.InFlightKeys.Count);
}
try { _onExit(this, exit); }
catch (Exception ex) { _log.Error(ex, "ScriptWorker[{Shard}] exit handler threw", Label); }
_ = connection.DisposeAsync().AsTask();
}
private void ScheduleBackoffLocked()
{
_consecutiveFailures++;
var delay = NextBackoff(_consecutiveFailures, _jitter.NextDouble());
_backoffUntil = Stopwatch.GetTimestamp() + (long)(delay.TotalSeconds * Stopwatch.Frequency);
}
/// <summary>The restart delay after <paramref name="failures"/> consecutive failures (design §5.3.5).</summary>
/// <param name="failures">1 for the first failure.</param>
/// <param name="unitJitter">A sample in [0, 1) mapped to ±20 %.</param>
/// <returns>The delay.</returns>
internal static TimeSpan NextBackoff(int failures, double unitJitter)
{
var exponent = Math.Clamp(failures - 1, 0, 30);
var baseMs = Math.Min(MaxBackoff.TotalMilliseconds, InitialBackoff.TotalMilliseconds * Math.Pow(2, exponent));
var factor = 0.8 + (0.4 * Math.Clamp(unitJitter, 0, 1));
return TimeSpan.FromMilliseconds(Math.Min(MaxBackoff.TotalMilliseconds * 1.2, baseMs * factor));
}
private static string MetricReason(WorkerExitReason reason) => reason switch
{
WorkerExitReason.Clean => "clean",
WorkerExitReason.KilledDeadline => "killed_deadline",
WorkerExitReason.KilledHeartbeat => "killed_heartbeat",
WorkerExitReason.KilledLoadTimeout => "killed_load",
WorkerExitReason.ProtocolFault => "protocol",
WorkerExitReason.Disposed => "disposed",
_ => "crash",
};
/// <summary>Stops the shard's worker.</summary>
/// <returns>A task that completes when the worker has ended.</returns>
public async ValueTask DisposeAsync()
{
WorkerConnection? current;
Task<WorkerConnection?>? starting;
lock (_gate)
{
_disposed = true;
current = _current;
starting = _starting;
_current = null;
}
if (starting is not null)
{
try { current ??= await starting.WaitAsync(TimeSpan.FromSeconds(15)).ConfigureAwait(false); } catch { /* shutting down */ }
}
if (current is not null) await current.DisposeAsync().ConfigureAwait(false);
IsolationGate?.Dispose();
}
/// <summary>Maps a context kind to the wire enum.</summary>
/// <param name="kind">The kind.</param>
/// <returns>The wire kind.</returns>
internal static ContextKind ToWire(ScriptContextKind kind) => kind switch
{
ScriptContextKind.VirtualTag => ContextKind.VirtualTag,
ScriptContextKind.Calculation => ContextKind.Calculation,
ScriptContextKind.AlarmPredicate => ContextKind.AlarmPredicate,
_ => ContextKind.Unspecified,
};
}
@@ -0,0 +1,20 @@
<Project Sdk="Microsoft.NET.Sdk">
<!-- lmxopcua#780 — the host side of the out-of-process, pre-emptible script runner (design §3, §5, §9): worker
launch + pipe handshake, the per-shard supervisor, attribution and quarantine. Selected by
Scripting:Runner=OutOfProcess; nothing here runs under the InProcess default. -->
<PropertyGroup>
<RootNamespace>ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess</RootNamespace>
<GenerateDocumentationFile>true</GenerateDocumentationFile>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.Core.Scripting\ZB.MOM.WW.OtOpcUa.Core.Scripting.csproj"/>
<ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol\ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.csproj"/>
</ItemGroup>
<ItemGroup>
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests"/>
</ItemGroup>
</Project>
@@ -0,0 +1,214 @@
// lmxopcua#780 — host <-> ScriptWorker IPC (design docs/plans/2026-11-09-out-of-process-script-runner-design.md §4).
// Framing: 4-byte little-endian length prefix + one serialized Envelope. Zero-length and oversize frames are protocol
// faults. One request frame and one reply frame per evaluation; the only other worker->host traffic is logs and
// heartbeats. Bump protocol_version on any incompatible change.
syntax = "proto3";
package otopcua.scriptworker.v1;
option csharp_namespace = "ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1";
message Envelope {
uint32 protocol_version = 1;
uint64 seq = 2;
oneof body {
Hello hello = 10; // host -> worker, first frame
Ready ready = 11; // worker -> host, reply to Hello
LoadScripts load = 12; // host -> worker
LoadResult load_result = 13; // worker -> host
Unload unload = 14; // host -> worker
EvalBatch eval = 15; // host -> worker
EvalResults results = 16; // worker -> host, streamed as runs complete
LogBatch logs = 17; // worker -> host, fire-and-forget, bounded
Heartbeat heartbeat = 18; // worker -> host, every heartbeat_interval_ms
Shutdown shutdown = 19; // host -> worker
Fault fault = 20; // either way; the sender closes after it
DiagnosticsRequest diag_request = 21; // host -> worker (tests / ops)
DiagnosticsReply diag_reply = 22; // worker -> host
Cancel cancel = 23; // host -> worker, at the soft deadline
}
}
message Hello {
string nonce = 1; // must be echoed in Ready
uint32 max_frame_bytes = 2;
uint32 eval_threads = 3; // the worker's scheduler width; 1 for the isolation worker
uint32 queue_capacity = 4; // queued evaluations above this answer BUSY
uint32 hard_grace_ms = 5; // worker watchdog: fail fast past soft + grace (+ margin)
uint32 heartbeat_interval_ms = 6;
uint32 max_log_lines = 7; // outbound log buffer cap
}
message Ready {
string nonce = 1;
int32 pid = 2;
string runtime_version = 3;
}
enum ContextKind {
CONTEXT_KIND_UNSPECIFIED = 0;
VIRTUAL_TAG = 1;
CALCULATION = 2;
ALARM_PREDICATE = 3;
}
message ScriptSource {
string key = 1;
bytes source_sha256 = 2;
ContextKind kind = 3;
string source = 4;
repeated string input_paths = 5; // slot order for EvalRequest.inputs
}
message LoadScripts {
uint64 load_id = 1;
repeated ScriptSource scripts = 2;
}
message LoadOutcome {
string key = 1;
bool ok = 2;
ScriptError error = 3; // COMPILE or SANDBOX when !ok
}
message LoadResult {
uint64 load_id = 1;
repeated LoadOutcome outcomes = 2;
}
message Unload {
repeated string keys = 1;
}
message EvalBatch {
repeated EvalRequest items = 1;
}
message EvalRequest {
uint64 eval_id = 1;
string key = 2;
repeated Input inputs = 3;
int64 now_utc_ticks = 4; // ctx.Now (host clock)
uint32 soft_timeout_ms = 5;
uint32 hard_grace_ms = 6; // this run's G; 0 = Hello.hard_grace_ms
}
message Input {
uint32 slot = 1; // index into ScriptSource.input_paths; ignored when path is set
Variant value = 2;
uint32 status_code = 3;
int64 source_ts_ticks = 4; // 0 = null
int64 server_ts_ticks = 5;
string path = 6; // inline path for an input with no registered slot
bool unsupported = 7; // the host could not encode the value's CLR type (see unsupported_type)
string unsupported_type = 8;
}
message EvalResult {
uint64 eval_id = 1;
oneof outcome {
Variant value = 2;
bool predicate = 3;
ScriptError error = 4;
}
uint64 allocated_bytes = 5; // GC.GetAllocatedBytesForCurrentThread around the run
uint32 run_micros = 6;
}
message EvalResults {
repeated EvalResult items = 1;
}
message ScriptError {
enum Kind {
THREW = 0;
COMPILE = 1;
SANDBOX = 2;
SOFT_TIMEOUT = 3;
UNSUPPORTED_TYPE = 4;
BUSY = 5;
NOT_LOADED = 6;
}
Kind kind = 1;
string message = 2;
}
message Variant {
enum TypeTag {
NULL = 0;
BOOLEAN = 1;
SBYTE = 2;
BYTE = 3;
INT16 = 4;
UINT16 = 5;
INT32 = 6;
UINT32 = 7;
INT64 = 8;
UINT64 = 9;
FLOAT = 10;
DOUBLE = 11;
STRING = 12;
DATETIME_UTC = 13;
ARRAY = 14;
}
TypeTag type = 1; // the exact CLR type: (short)ctx.GetTag(x).Value must keep working
oneof v {
bool b = 2;
sint64 i = 3;
uint64 u = 4;
double d = 5;
float f = 6;
string s = 7;
int64 dt_ticks = 8;
ArrayValue array = 9;
}
}
message ArrayValue {
Variant.TypeTag element_type = 1;
repeated Variant items = 2;
}
message LogBatch {
repeated LogLine lines = 1;
uint32 dropped = 2;
}
message LogLine {
string key = 1;
uint32 level = 2; // Serilog LogEventLevel
string rendered = 3;
string exception = 4;
int64 ts_ticks = 5;
}
message Heartbeat {
uint32 inflight = 1;
uint32 queued = 2;
int64 heap_bytes = 3;
uint32 gen2_count = 4;
}
message Shutdown {
}
// Sent by the host when it has answered the caller with a soft timeout: cancels the run's ctx.CancellationToken, so a
// polling script ends itself. The worker never cancels on its own clock, which keeps the in-process ordering — the
// caller always has its timeout before the script can observe cancellation.
message Cancel {
uint64 eval_id = 1;
}
message Fault {
string message = 1;
}
message DiagnosticsRequest {
}
message DiagnosticsReply {
int32 pid = 1;
repeated string loaded_assemblies = 2;
repeated string environment_names = 3;
string working_directory = 4;
}
@@ -0,0 +1,54 @@
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
/// <summary>
/// The environment variables the host sets on a ScriptWorker (design §6). The nonce travels in the environment,
/// never on the command line (mxaccessgw <c>WorkerBootstrap.md</c>): a command line is world-readable on most
/// systems.
/// </summary>
public static class ScriptWorkerEnvironment
{
/// <summary>The pipe name the worker connects to.</summary>
public const string PipeName = "OTOPCUA_SCRIPTWORKER_PIPE";
/// <summary>The one-time launch nonce the worker must echo in <c>Ready</c>.</summary>
public const string Nonce = "OTOPCUA_SCRIPTWORKER_NONCE";
/// <summary>The host's process id; the worker exits when it disappears.</summary>
public const string ParentPid = "OTOPCUA_SCRIPTWORKER_PARENT_PID";
/// <summary>The worker's shard label, for its own diagnostics.</summary>
public const string Shard = "OTOPCUA_SCRIPTWORKER_SHARD";
/// <summary>
/// Exit code of a worker whose pipe reached end of stream (the host went away). Distinct from a crash so the
/// host's exit reason metric can tell them apart.
/// </summary>
public const int ExitPipeClosed = 3;
/// <summary>Exit code of a worker whose handshake or framing failed.</summary>
public const int ExitProtocolFault = 4;
/// <summary>Exit code of a worker whose parent process disappeared.</summary>
public const int ExitParentGone = 5;
/// <summary>Exit code of a worker that found a forbidden assembly loaded in itself.</summary>
public const int ExitForbiddenAssembly = 6;
/// <summary>
/// Assembly-name prefixes that must never be loaded in a worker (design §6): the host's cluster, database,
/// OPC UA and secrets stacks. The worker checks itself against this list and exits if it finds one.
/// </summary>
public static readonly IReadOnlyList<string> ForbiddenAssemblyPrefixes =
[
"Akka",
"Microsoft.Data.SqlClient",
"System.Data.SqlClient",
"Microsoft.EntityFrameworkCore",
"Opc.Ua",
"ZB.MOM.WW.Secrets",
"ZB.MOM.WW.OtOpcUa.Runtime",
"ZB.MOM.WW.OtOpcUa.Host",
"ZB.MOM.WW.OtOpcUa.Driver.",
"Grpc.",
];
}
@@ -0,0 +1,116 @@
using System.Buffers;
using System.Buffers.Binary;
using Google.Protobuf;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
/// <summary>
/// Length-prefixed protobuf framing for the ScriptWorker pipe (lmxopcua#780, design §4.2): a 4-byte little-endian
/// length, then one serialized <see cref="Envelope"/>. Copies mxaccessgw's <c>WorkerFrameProtocol</c> discipline:
/// zero-length, oversize, truncated and unparseable frames are all a <see cref="ScriptWorkerProtocolException"/>,
/// after which the stream is unusable and the peer is dropped.
/// </summary>
public static class ScriptWorkerFrameCodec
{
/// <summary>The protocol version both ends put on every envelope.</summary>
public const uint ProtocolVersion = 1;
/// <summary>Default frame cap: 4 MiB.</summary>
public const int DefaultMaxFrameBytes = 4 * 1024 * 1024;
/// <summary>Writes one frame. Not thread-safe: callers serialise writes.</summary>
/// <param name="stream">The pipe.</param>
/// <param name="envelope">The envelope.</param>
/// <param name="maxFrameBytes">The negotiated cap; a larger envelope is refused before anything is written.</param>
/// <param name="ct">Cancellation.</param>
/// <returns>A task that completes when the frame is written (not flushed).</returns>
public static async ValueTask WriteAsync(Stream stream, Envelope envelope, int maxFrameBytes, CancellationToken ct = default)
{
ArgumentNullException.ThrowIfNull(stream);
ArgumentNullException.ThrowIfNull(envelope);
var size = envelope.CalculateSize();
if (size == 0) throw new ScriptWorkerProtocolException("refusing to write an empty frame");
if (size > maxFrameBytes)
throw new ScriptWorkerProtocolException($"frame of {size} bytes exceeds the {maxFrameBytes}-byte cap");
var buffer = ArrayPool<byte>.Shared.Rent(size + 4);
try
{
BinaryPrimitives.WriteInt32LittleEndian(buffer, size);
envelope.WriteTo(new Span<byte>(buffer, 4, size));
await stream.WriteAsync(buffer.AsMemory(0, size + 4), ct).ConfigureAwait(false);
}
finally
{
ArrayPool<byte>.Shared.Return(buffer);
}
}
/// <summary>Reads one frame.</summary>
/// <param name="stream">The pipe.</param>
/// <param name="maxFrameBytes">The negotiated cap.</param>
/// <param name="ct">Cancellation.</param>
/// <returns>The envelope, or null on a clean end of stream at a frame boundary.</returns>
/// <exception cref="ScriptWorkerProtocolException">A zero-length, oversize, truncated or unparseable frame.</exception>
public static async ValueTask<Envelope?> ReadAsync(Stream stream, int maxFrameBytes, CancellationToken ct = default)
{
ArgumentNullException.ThrowIfNull(stream);
var header = new byte[4];
var got = await ReadFullyAsync(stream, header, ct).ConfigureAwait(false);
if (got == 0) return null;
if (got < 4) throw new ScriptWorkerProtocolException("truncated frame header");
var size = BinaryPrimitives.ReadInt32LittleEndian(header);
if (size == 0) throw new ScriptWorkerProtocolException("zero-length frame");
if (size < 0 || size > maxFrameBytes)
throw new ScriptWorkerProtocolException($"frame length {size} is outside 1..{maxFrameBytes}");
var body = ArrayPool<byte>.Shared.Rent(size);
try
{
var read = await ReadFullyAsync(stream, body.AsMemory(0, size), ct).ConfigureAwait(false);
if (read < size) throw new ScriptWorkerProtocolException($"truncated frame: {read} of {size} bytes");
Envelope envelope;
try
{
envelope = Envelope.Parser.ParseFrom(body, 0, size);
}
catch (InvalidProtocolBufferException ex)
{
throw new ScriptWorkerProtocolException($"unparseable frame: {ex.Message}", ex);
}
if (envelope.ProtocolVersion != ProtocolVersion)
throw new ScriptWorkerProtocolException(
$"protocol version {envelope.ProtocolVersion} is not the supported {ProtocolVersion}");
if (envelope.BodyCase == Envelope.BodyOneofCase.None)
throw new ScriptWorkerProtocolException("frame carries no body");
return envelope;
}
finally
{
ArrayPool<byte>.Shared.Return(body);
}
}
private static async ValueTask<int> ReadFullyAsync(Stream stream, Memory<byte> buffer, CancellationToken ct)
{
var total = 0;
while (total < buffer.Length)
{
var n = await stream.ReadAsync(buffer[total..], ct).ConfigureAwait(false);
if (n == 0) break;
total += n;
}
return total;
}
}
/// <summary>A framing or schema violation on the ScriptWorker pipe; the connection is dropped.</summary>
public sealed class ScriptWorkerProtocolException : Exception
{
/// <summary>Initializes the exception.</summary>
/// <param name="message">What was wrong.</param>
/// <param name="inner">The underlying parser error, if any.</param>
public ScriptWorkerProtocolException(string message, Exception? inner = null) : base(message, inner) { }
}
@@ -0,0 +1,159 @@
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
/// <summary>
/// CLR value ↔ <see cref="Variant"/> (design §4.4). The exact boxed CLR type survives the round trip — a script
/// doing <c>(short)ctx.GetTag(x).Value</c> throws <see cref="InvalidCastException"/> if a <c>short</c> arrives as an
/// <c>int</c> — so the type tag names the CLR type, not just an OPC UA one. A type with no tag is never converted:
/// <see cref="TryEncode"/> refuses it, and the caller reports <c>UNSUPPORTED_TYPE</c> (design §14 #11).
/// </summary>
public static class VariantCodec
{
/// <summary>Encodes <paramref name="value"/>.</summary>
/// <param name="value">The boxed CLR value (null allowed).</param>
/// <param name="variant">The encoded variant when supported.</param>
/// <param name="unsupportedType">The CLR type name when not supported.</param>
/// <returns><c>true</c> when the type has a tag.</returns>
public static bool TryEncode(object? value, out Variant variant, out string? unsupportedType)
{
unsupportedType = null;
variant = new Variant();
switch (value)
{
case null: variant.Type = Variant.Types.TypeTag.Null; return true;
case bool b: variant.Type = Variant.Types.TypeTag.Boolean; variant.B = b; return true;
case sbyte v: variant.Type = Variant.Types.TypeTag.Sbyte; variant.I = v; return true;
case byte v: variant.Type = Variant.Types.TypeTag.Byte; variant.U = v; return true;
case short v: variant.Type = Variant.Types.TypeTag.Int16; variant.I = v; return true;
case ushort v: variant.Type = Variant.Types.TypeTag.Uint16; variant.U = v; return true;
case int v: variant.Type = Variant.Types.TypeTag.Int32; variant.I = v; return true;
case uint v: variant.Type = Variant.Types.TypeTag.Uint32; variant.U = v; return true;
case long v: variant.Type = Variant.Types.TypeTag.Int64; variant.I = v; return true;
case ulong v: variant.Type = Variant.Types.TypeTag.Uint64; variant.U = v; return true;
case float v: variant.Type = Variant.Types.TypeTag.Float; variant.F = v; return true;
case double v: variant.Type = Variant.Types.TypeTag.Double; variant.D = v; return true;
case string v: variant.Type = Variant.Types.TypeTag.String; variant.S = v; return true;
case DateTime v:
variant.Type = Variant.Types.TypeTag.DatetimeUtc;
variant.DtTicks = ToUtc(v).Ticks;
return true;
case Array array when array.Rank == 1 && TryElementTag(array.GetType().GetElementType()!, out var elementTag):
{
var av = new ArrayValue { ElementType = elementTag };
foreach (var item in array)
{
if (!TryEncode(item, out var element, out unsupportedType)) return false;
av.Items.Add(element);
}
variant.Type = Variant.Types.TypeTag.Array;
variant.Array = av;
return true;
}
default:
unsupportedType = value.GetType().FullName ?? value.GetType().Name;
return false;
}
}
/// <summary>Decodes <paramref name="variant"/> back to the boxed CLR value it was encoded from.</summary>
/// <param name="variant">The variant.</param>
/// <returns>The boxed value.</returns>
/// <exception cref="ScriptWorkerProtocolException">An unknown tag, or a payload that does not match the tag.</exception>
public static object? Decode(Variant? variant)
{
if (variant is null) return null;
return variant.Type switch
{
Variant.Types.TypeTag.Null => null,
Variant.Types.TypeTag.Boolean => Expect(variant, Variant.VOneofCase.B).B,
Variant.Types.TypeTag.Sbyte => checked((sbyte)Expect(variant, Variant.VOneofCase.I).I),
Variant.Types.TypeTag.Byte => checked((byte)Expect(variant, Variant.VOneofCase.U).U),
Variant.Types.TypeTag.Int16 => checked((short)Expect(variant, Variant.VOneofCase.I).I),
Variant.Types.TypeTag.Uint16 => checked((ushort)Expect(variant, Variant.VOneofCase.U).U),
Variant.Types.TypeTag.Int32 => checked((int)Expect(variant, Variant.VOneofCase.I).I),
Variant.Types.TypeTag.Uint32 => checked((uint)Expect(variant, Variant.VOneofCase.U).U),
Variant.Types.TypeTag.Int64 => Expect(variant, Variant.VOneofCase.I).I,
Variant.Types.TypeTag.Uint64 => Expect(variant, Variant.VOneofCase.U).U,
Variant.Types.TypeTag.Float => Expect(variant, Variant.VOneofCase.F).F,
Variant.Types.TypeTag.Double => Expect(variant, Variant.VOneofCase.D).D,
Variant.Types.TypeTag.String => Expect(variant, Variant.VOneofCase.S).S,
Variant.Types.TypeTag.DatetimeUtc => new DateTime(Expect(variant, Variant.VOneofCase.DtTicks).DtTicks, DateTimeKind.Utc),
Variant.Types.TypeTag.Array => DecodeArray(Expect(variant, Variant.VOneofCase.Array).Array),
_ => throw new ScriptWorkerProtocolException($"unknown variant type tag {(int)variant.Type}"),
};
}
/// <summary>A <see cref="DateTime"/> as UTC: a <c>Local</c> value is converted; an <c>Unspecified</c> one is taken as UTC.</summary>
/// <param name="value">The value.</param>
/// <returns>The UTC value.</returns>
public static DateTime ToUtc(DateTime value) => value.Kind switch
{
DateTimeKind.Utc => value,
DateTimeKind.Local => value.ToUniversalTime(),
_ => DateTime.SpecifyKind(value, DateTimeKind.Utc),
};
private static Variant Expect(Variant variant, Variant.VOneofCase expected)
{
// proto3 drops a default scalar (0, false, "") from the oneof, so an absent payload is the default value.
if (variant.VCase != expected && variant.VCase != Variant.VOneofCase.None)
throw new ScriptWorkerProtocolException($"variant tagged {variant.Type} carries a {variant.VCase} payload");
return variant;
}
private static Array DecodeArray(ArrayValue? array)
{
if (array is null) throw new ScriptWorkerProtocolException("array variant without an array payload");
var elementType = ClrType(array.ElementType);
var result = Array.CreateInstance(elementType, array.Items.Count);
for (var i = 0; i < array.Items.Count; i++)
{
var item = array.Items[i];
if (item.Type != array.ElementType && item.Type != Variant.Types.TypeTag.Null)
throw new ScriptWorkerProtocolException($"array of {array.ElementType} holds a {item.Type} element");
result.SetValue(Decode(item), i);
}
return result;
}
private static bool TryElementTag(Type elementType, out Variant.Types.TypeTag tag)
{
tag = elementType switch
{
_ when elementType == typeof(bool) => Variant.Types.TypeTag.Boolean,
_ when elementType == typeof(sbyte) => Variant.Types.TypeTag.Sbyte,
_ when elementType == typeof(byte) => Variant.Types.TypeTag.Byte,
_ when elementType == typeof(short) => Variant.Types.TypeTag.Int16,
_ when elementType == typeof(ushort) => Variant.Types.TypeTag.Uint16,
_ when elementType == typeof(int) => Variant.Types.TypeTag.Int32,
_ when elementType == typeof(uint) => Variant.Types.TypeTag.Uint32,
_ when elementType == typeof(long) => Variant.Types.TypeTag.Int64,
_ when elementType == typeof(ulong) => Variant.Types.TypeTag.Uint64,
_ when elementType == typeof(float) => Variant.Types.TypeTag.Float,
_ when elementType == typeof(double) => Variant.Types.TypeTag.Double,
_ when elementType == typeof(string) => Variant.Types.TypeTag.String,
_ when elementType == typeof(DateTime) => Variant.Types.TypeTag.DatetimeUtc,
_ => Variant.Types.TypeTag.Null,
};
return tag != Variant.Types.TypeTag.Null;
}
private static Type ClrType(Variant.Types.TypeTag tag) => tag switch
{
Variant.Types.TypeTag.Boolean => typeof(bool),
Variant.Types.TypeTag.Sbyte => typeof(sbyte),
Variant.Types.TypeTag.Byte => typeof(byte),
Variant.Types.TypeTag.Int16 => typeof(short),
Variant.Types.TypeTag.Uint16 => typeof(ushort),
Variant.Types.TypeTag.Int32 => typeof(int),
Variant.Types.TypeTag.Uint32 => typeof(uint),
Variant.Types.TypeTag.Int64 => typeof(long),
Variant.Types.TypeTag.Uint64 => typeof(ulong),
Variant.Types.TypeTag.Float => typeof(float),
Variant.Types.TypeTag.Double => typeof(double),
Variant.Types.TypeTag.String => typeof(string),
Variant.Types.TypeTag.DatetimeUtc => typeof(DateTime),
_ => throw new ScriptWorkerProtocolException($"array element type {tag} is not supported"),
};
}
@@ -0,0 +1,28 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<RootNamespace>ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol</RootNamespace>
<GenerateDocumentationFile>true</GenerateDocumentationFile>
<!-- protoc-generated members carry no XML docs. -->
<NoWarn>$(NoWarn);CS1591</NoWarn>
</PropertyGroup>
<!-- lmxopcua#780: the host <-> ScriptWorker frame protocol (design §4). Shared by the host-side runner and the
worker, so it references nothing but Google.Protobuf: the worker's assembly set stays small (design §6). -->
<ItemGroup>
<PackageReference Include="Google.Protobuf"/>
<PackageReference Include="Grpc.Tools">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
</ItemGroup>
<ItemGroup>
<Protobuf Include="Protos\script_worker.proto" GrpcServices="None"/>
</ItemGroup>
<ItemGroup>
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests"/>
</ItemGroup>
</Project>
@@ -0,0 +1,137 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting;
/// <summary>
/// The execution seam every host script owner can run a compiled script through (lmxopcua#780, design §11): a
/// script key, its source, its input snapshots and its budget go in; one <see cref="ScriptRunOutcome"/> comes
/// out. Implementations never throw for script faults.
/// </summary>
/// <remarks>
/// <para>
/// <see cref="InProcessScriptRunner"/> is today's <see cref="ScriptRunGuard"/> + <see cref="CompiledScriptCache{TContext, TResult}"/>
/// behind this interface (the parity baseline). The out-of-process runner
/// (<c>ZB.MOM.WW.OtOpcUa.Core.Scripting.Runner</c>) runs the same compile in a pre-emptible worker process.
/// The passthrough fast path (<c>PassthroughScript</c>) never reaches a runner: it runs no user code.
/// </para>
/// <para>
/// Keys are namespaced by <see cref="ScriptInvocation.Kind"/>, so a virtual-tag id and a calc RawPath that
/// happen to be equal never share quarantine state, and <see cref="Reset"/> for one seam leaves the other
/// seams' quarantines alone.
/// </para>
/// </remarks>
public interface IScriptRunner
{
/// <summary>Runs one compiled evaluation.</summary>
/// <param name="invocation">What to run.</param>
/// <param name="ct">Caller cancellation (shutdown); wins over the budget.</param>
/// <returns>The outcome; never faults for a script failure.</returns>
Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default);
/// <summary>
/// The deploy boundary for one seam: lifts every quarantine and forgets every remembered compile failure of
/// <paramref name="kind"/>, and drops its compiled scripts.
/// </summary>
/// <param name="kind">The seam's context kind.</param>
void Reset(ScriptContextKind kind);
/// <summary>The keys of <paramref name="kind"/> currently quarantined.</summary>
/// <param name="kind">The seam's context kind.</param>
/// <returns>The quarantined keys (un-namespaced).</returns>
IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind);
}
/// <summary>Which script context (and result type) a script compiles against.</summary>
public enum ScriptContextKind
{
/// <summary>A virtual tag: <c>VirtualTagContext</c> → <c>object?</c>.</summary>
VirtualTag = 1,
/// <summary>A Calculation tag: <c>VirtualTagContext</c> → <c>object?</c>.</summary>
Calculation = 2,
/// <summary>A scripted-alarm predicate: <c>AlarmPredicateContext</c> → <c>bool</c> (not run out of process yet).</summary>
AlarmPredicate = 3,
}
/// <summary>One evaluation handed to an <see cref="IScriptRunner"/>.</summary>
/// <param name="Key">The script instance key (virtual-tag id, calc RawPath, alarm id).</param>
/// <param name="Kind">The context kind.</param>
/// <param name="Source">The script source (also its identity: a change lifts the key's quarantine).</param>
/// <param name="Inputs">The read cache: dependency snapshots by path. Not mutated until the run completes.</param>
/// <param name="Timeout">The soft budget <c>T</c>.</param>
/// <param name="ScriptLogger">The per-evaluation script logger <c>ctx.Logger</c> writes through; null ⇒ a silent logger.</param>
public sealed record ScriptInvocation(
string Key,
ScriptContextKind Kind,
string Source,
IReadOnlyDictionary<string, DataValueSnapshot> Inputs,
TimeSpan Timeout,
Serilog.ILogger? ScriptLogger = null);
/// <summary>How an evaluation ended.</summary>
public enum ScriptRunStatus
{
/// <summary>The script returned <see cref="ScriptRunOutcome.Value"/>.</summary>
Ok = 0,
/// <summary>A script-attributable failure (threw, timed out, refused, quarantined, did not compile) — Bad as today.</summary>
Failed = 1,
/// <summary>
/// The runner could not run the script, and it is not the script's fault: its worker restarted under it, or no
/// worker became ready in time. Published as <c>BadResourceUnavailable</c> (design §9 / §14 #3).
/// </summary>
Unavailable = 2,
}
/// <summary>The result of one <see cref="IScriptRunner.RunAsync"/>.</summary>
/// <param name="Status">How it ended.</param>
/// <param name="Value">The script's result when <see cref="ScriptRunStatus.Ok"/>.</param>
/// <param name="Reason">The failure reason otherwise; the same strings the evaluators have always published.</param>
public sealed record ScriptRunOutcome(ScriptRunStatus Status, object? Value, string? Reason)
{
/// <summary>Reason when the worker running the script died under it (another script's kill, a crash).</summary>
public const string WorkerRestartedReason = "script worker restarted";
/// <summary>Reason when no worker became ready within <see cref="ScriptWorkerOptions.UnavailableGrace"/>.</summary>
public const string RunnerUnavailableReason = "script runner unavailable";
/// <summary>A successful run.</summary>
/// <param name="value">The script's result.</param>
/// <returns>The outcome.</returns>
public static ScriptRunOutcome Ok(object? value) => new(ScriptRunStatus.Ok, value, null);
/// <summary>A script-attributable failure.</summary>
/// <param name="reason">The reason.</param>
/// <returns>The outcome.</returns>
public static ScriptRunOutcome Failed(string reason) => new(ScriptRunStatus.Failed, null, reason);
/// <summary>A runner-attributable failure.</summary>
/// <param name="reason">The reason.</param>
/// <returns>The outcome.</returns>
public static ScriptRunOutcome Unavailable(string reason) => new(ScriptRunStatus.Unavailable, null, reason);
}
/// <summary>Maps an <see cref="IScriptRunner"/> outcome onto the value seams' result type.</summary>
public static class ScriptRunOutcomeExtensions
{
/// <summary>
/// <see cref="ScriptRunStatus.Ok"/> → <c>Ok(value)</c>; <see cref="ScriptRunStatus.Failed"/> → <c>Failure(reason)</c>
/// (published <c>BadInternalError</c>, as today); <see cref="ScriptRunStatus.Unavailable"/> →
/// <c>Unavailable(reason)</c> (published <c>BadResourceUnavailable</c>).
/// </summary>
/// <param name="outcome">The runner's outcome.</param>
/// <returns>The seam result.</returns>
public static Commons.Engines.VirtualTagEvalResult ToVirtualTagEvalResult(this ScriptRunOutcome outcome)
{
ArgumentNullException.ThrowIfNull(outcome);
return outcome.Status switch
{
ScriptRunStatus.Ok => Commons.Engines.VirtualTagEvalResult.Ok(outcome.Value),
ScriptRunStatus.Unavailable => Commons.Engines.VirtualTagEvalResult.Unavailable(
outcome.Reason ?? ScriptRunOutcome.RunnerUnavailableReason),
_ => Commons.Engines.VirtualTagEvalResult.Failure(outcome.Reason ?? "evaluator failure"),
};
}
}
@@ -0,0 +1,141 @@
using System.Collections.Concurrent;
using Serilog;
using Serilog.Core;
using ZB.MOM.WW.OtOpcUa.Commons.Engines;
using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
using ZB.MOM.WW.OtOpcUa.Core.VirtualTags;
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting;
/// <summary>
/// <see cref="IScriptRunner"/> over today's in-process machinery: a <see cref="CompiledScriptCache{TContext, TResult}"/>
/// per context kind and one <see cref="ScriptRunGuard"/> (lmxopcua#780, design §11). It produces the outcomes and
/// reason strings the seams' own in-process paths produce, which makes it the parity baseline the out-of-process
/// runner is tested against, and the in-process half of the future shadow mode (P2).
/// </summary>
/// <remarks>
/// The seams do not route their <see cref="ScriptRunnerMode.InProcess"/> path through this class: they keep their
/// existing guard code, so the default mode is unchanged byte for byte. Unlike those paths, this runner remembers
/// compile failures by source for every kind (Calculation already did, R07:P4).
/// </remarks>
public sealed class InProcessScriptRunner : IScriptRunner, IDisposable
{
private readonly CompiledScriptCache<VirtualTagContext, object?> _valueCache = new();
private readonly CompiledScriptCache<AlarmPredicateContext, bool> _predicateCache = new();
private readonly ConcurrentDictionary<string, string> _compileFailures = new(StringComparer.Ordinal);
private readonly ScriptRunGuard _guard;
/// <summary>Initializes the runner.</summary>
/// <param name="scripting">Run-guard knobs; null ⇒ defaults.</param>
/// <param name="logger">Where quarantine lines go; null ⇒ Serilog's global logger.</param>
public InProcessScriptRunner(ScriptingOptions? scripting = null, ILogger? logger = null)
{
_guard = new ScriptRunGuard(scripting, logger);
}
/// <summary>The run guard (diagnostics / tests).</summary>
public ScriptRunGuard RunGuard => _guard;
/// <inheritdoc />
public async Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default)
{
ArgumentNullException.ThrowIfNull(invocation);
var key = ScopedKey(invocation.Kind, invocation.Key);
var failureKey = ScopedKey(invocation.Kind, invocation.Source);
if (_compileFailures.TryGetValue(failureKey, out var known)) return ScriptRunOutcome.Failed(known);
var logger = invocation.ScriptLogger ?? Logger.None;
try
{
switch (invocation.Kind)
{
case ScriptContextKind.VirtualTag:
case ScriptContextKind.Calculation:
{
var evaluator = await Task.Run(() => _valueCache.GetOrCompile(invocation.Source), ct).ConfigureAwait(false);
var context = new VirtualTagContext(invocation.Inputs, static (_, _) => { }, logger);
var value = await _guard.RunAsync(key, invocation.Source, evaluator, context, invocation.Timeout, ct)
.ConfigureAwait(false);
return ScriptRunOutcome.Ok(value);
}
case ScriptContextKind.AlarmPredicate:
{
var evaluator = await Task.Run(() => _predicateCache.GetOrCompile(invocation.Source), ct).ConfigureAwait(false);
var context = new AlarmPredicateContext(invocation.Inputs, logger);
var value = await _guard.RunAsync(key, invocation.Source, evaluator, context, invocation.Timeout, ct)
.ConfigureAwait(false);
return ScriptRunOutcome.Ok(value);
}
default:
return ScriptRunOutcome.Failed($"unsupported script kind {invocation.Kind}");
}
}
catch (CompilationErrorException ex)
{
return RememberCompileFailure(failureKey, $"compile error: {ex.Message}");
}
catch (ScriptSandboxViolationException ex)
{
return RememberCompileFailure(failureKey, $"sandbox violation: {ex.Message}");
}
catch (ScriptTimeoutException)
{
return ScriptRunOutcome.Failed($"{ScriptEvalFailureReasons.TimedOutPrefix}{invocation.Timeout.TotalSeconds:F1}s");
}
catch (ScriptRunRefusedException ex)
{
return ScriptRunOutcome.Failed(ex.Message);
}
catch (ScriptQuarantinedException ex)
{
return ScriptRunOutcome.Failed(ex.Message);
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
return ScriptRunOutcome.Failed("evaluation cancelled");
}
catch (Exception ex)
{
return ScriptRunOutcome.Failed($"script threw: {ex.Message}");
}
}
/// <inheritdoc />
public void Reset(ScriptContextKind kind)
{
var prefix = ScopedKey(kind, string.Empty);
foreach (var k in _compileFailures.Keys)
if (k.StartsWith(prefix, StringComparison.Ordinal)) _compileFailures.TryRemove(k, out _);
foreach (var k in _guard.QuarantinedKeys)
if (k.StartsWith(prefix, StringComparison.Ordinal)) _guard.Reset(k);
}
/// <inheritdoc />
public IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind)
{
var prefix = ScopedKey(kind, string.Empty);
return _guard.QuarantinedKeys
.Where(k => k.StartsWith(prefix, StringComparison.Ordinal))
.Select(k => k[prefix.Length..])
.ToArray();
}
/// <inheritdoc />
public void Dispose()
{
_valueCache.Dispose();
_predicateCache.Dispose();
}
/// <summary>The runner-internal key: <c>{kind}:{key}</c>.</summary>
/// <param name="kind">The context kind.</param>
/// <param name="key">The seam's key.</param>
/// <returns>The namespaced key.</returns>
public static string ScopedKey(ScriptContextKind kind, string key) => $"{(int)kind}:{key}";
private ScriptRunOutcome RememberCompileFailure(string failureKey, string reason)
{
_compileFailures.TryAdd(failureKey, reason);
return ScriptRunOutcome.Failed(reason);
}
}
@@ -0,0 +1,213 @@
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting;
/// <summary>
/// Where compiled scripts run (<c>Scripting:Runner</c>, lmxopcua#780). <see cref="InProcess"/> is the default
/// and the documented rollback: it is today's <see cref="ScriptRunGuard"/> path, unchanged.
/// </summary>
public enum ScriptRunnerMode
{
/// <summary>Compiled scripts run in the host process under <see cref="ScriptRunGuard"/> (the default).</summary>
InProcess = 0,
/// <summary>
/// Run in process AND submit to the worker, comparing the two (design P2). Not implemented yet: a node
/// configured for it runs <see cref="InProcess"/> and logs a startup Warning.
/// </summary>
Shadow = 1,
/// <summary>
/// Compiled scripts of the selected seams run in the pre-emptible <c>ScriptWorker</c> process pool. The
/// runner fails closed: a worker that cannot start makes the script Bad
/// (<c>BadResourceUnavailable</c>), never an in-process run.
/// </summary>
OutOfProcess = 2,
}
/// <summary>The three host seams that run operator-authored C# (<c>Scripting:Runner:Seams</c>).</summary>
public enum ScriptSeam
{
/// <summary>Virtual tags (<c>RoslynVirtualTagEvaluator</c>).</summary>
VirtualTag = 0,
/// <summary>Calculation tags (<c>CalculationEvaluator</c>).</summary>
Calculation = 1,
/// <summary>Scripted-alarm predicates (<c>ScriptedAlarmEngine</c>). Last in the rollout (design §14 #10).</summary>
ScriptedAlarm = 2,
}
/// <summary>
/// The out-of-process worker pool's knobs (the <c>Scripting:Worker</c> section). Every value is clamped by
/// <see cref="Sanitized"/>, which reports each out-of-range key as a startup Warning, like the rest of the
/// <c>Scripting</c> section. Only consulted when <see cref="ScriptingOptions.Runner"/> is not
/// <see cref="ScriptRunnerMode.InProcess"/>.
/// </summary>
public sealed class ScriptWorkerOptions
{
/// <summary>Default <see cref="Shards"/>: 2 (design §14 #1).</summary>
public const int DefaultShards = 2;
/// <summary>Upper bound of <see cref="Shards"/>.</summary>
public const int MaxShards = 8;
/// <summary>Default <see cref="HeapLimitBytes"/>: 512 MiB.</summary>
public const long DefaultHeapLimitBytes = 512L * 1024 * 1024;
/// <summary>Lower bound of <see cref="HeapLimitBytes"/>: 64 MiB (a worker below this cannot host Roslyn).</summary>
public const long MinHeapLimitBytes = 64L * 1024 * 1024;
/// <summary>Default <see cref="QuarantineAfterKills"/>: 2.</summary>
public const int DefaultQuarantineAfterKills = 2;
/// <summary>Default <see cref="UnavailableGrace"/>: 5 s.</summary>
public static readonly TimeSpan DefaultUnavailableGrace = TimeSpan.FromSeconds(5);
/// <summary>Lower bound of an explicit <see cref="HardGrace"/>: 250 ms.</summary>
public static readonly TimeSpan MinHardGrace = TimeSpan.FromMilliseconds(250);
/// <summary>Upper bound of an explicit <see cref="HardGrace"/>: 30 s.</summary>
public static readonly TimeSpan MaxHardGrace = TimeSpan.FromSeconds(30);
/// <summary>Upper bound of <see cref="UnavailableGrace"/>: 60 s.</summary>
public static readonly TimeSpan MaxUnavailableGrace = TimeSpan.FromSeconds(60);
/// <summary>Default <see cref="IsolationProbationRuns"/>: 20.</summary>
public const int DefaultIsolationProbationRuns = 20;
/// <summary>Shard workers per node; a script is placed by a stable hash of its key. Clamped to 1–8.</summary>
public int Shards { get; set; } = DefaultShards;
/// <summary>
/// Grace after the soft deadline <c>T</c> before the worker running the script is killed. Null (the default)
/// means <c>max(1 s, T)</c> per run (design §14 #2); an explicit value is clamped to 250 ms – 30 s.
/// </summary>
public TimeSpan? HardGrace { get; set; }
/// <summary>The worker's <c>DOTNET_GCHeapHardLimit</c>. Default 512 MiB, minimum 64 MiB.</summary>
public long HeapLimitBytes { get; set; } = DefaultHeapLimitBytes;
/// <summary>Hard-deadline kills (within one hour) after which a script is quarantined. Default 2; 1–100.</summary>
public int QuarantineAfterKills { get; set; } = DefaultQuarantineAfterKills;
/// <summary>
/// How long an evaluation waits for its shard to become ready (a restart, a backoff) before it reports
/// <c>script runner unavailable</c>. Default 5 s; 0–60 s.
/// </summary>
public TimeSpan UnavailableGrace { get; set; } = DefaultUnavailableGrace;
/// <summary>
/// Successful runs a suspect script must complete in the isolation worker before it returns to its shard.
/// Default 20; 1–10 000.
/// </summary>
public int IsolationProbationRuns { get; set; } = DefaultIsolationProbationRuns;
/// <summary>
/// Directory of the node-local compiled-PE cache (design §7). Must be absolute and outside the content root
/// when set; startup fails otherwise (the family's fail-closed path rule). Reserved: the cache itself has
/// not landed yet (lmxopcua#780 P1 remainder), so a worker restart recompiles.
/// </summary>
public string? PeCacheDir { get; set; }
/// <summary>
/// Path of the worker entry assembly (<c>ZB.MOM.WW.OtOpcUa.ScriptWorker.dll</c>). Null (the default) means
/// <c>script-worker/</c> under the host's base directory, where the Host build places it.
/// </summary>
public string? ExecutablePath { get; set; }
/// <summary>The hard grace that applies to a run with soft budget <paramref name="softTimeout"/>.</summary>
/// <param name="softTimeout">The run's soft deadline <c>T</c>.</param>
/// <returns><see cref="HardGrace"/> when set, otherwise <c>max(1 s, T)</c>.</returns>
public TimeSpan EffectiveHardGrace(TimeSpan softTimeout)
=> HardGrace ?? (softTimeout > TimeSpan.FromSeconds(1) ? softTimeout : TimeSpan.FromSeconds(1));
/// <summary>Returns a clamped copy; one line per out-of-range key in <paramref name="adjustments"/>.</summary>
/// <param name="adjustments">The clamps applied, naming each key.</param>
/// <returns>A new, clamped instance.</returns>
public ScriptWorkerOptions Sanitized(out IReadOnlyList<string> adjustments)
{
const string prefix = ScriptingOptions.SectionName + ":Worker:";
var notes = new List<string>();
var shards = Shards;
if (shards < 1 || shards > MaxShards)
{
var clamped = Math.Clamp(shards, 1, MaxShards);
notes.Add($"{prefix}{nameof(Shards)} {shards} is outside 1–{MaxShards}; clamped to {clamped}.");
shards = clamped;
}
var grace = HardGrace;
if (grace is { } g && (g < MinHardGrace || g > MaxHardGrace))
{
var clamped = g < MinHardGrace ? MinHardGrace : MaxHardGrace;
notes.Add($"{prefix}{nameof(HardGrace)} {g} is outside {MinHardGrace}–{MaxHardGrace}; clamped to {clamped}.");
grace = clamped;
}
var heap = HeapLimitBytes;
if (heap < MinHeapLimitBytes)
{
notes.Add($"{prefix}{nameof(HeapLimitBytes)} {heap} is below the minimum; clamped to {MinHeapLimitBytes}.");
heap = MinHeapLimitBytes;
}
var kills = QuarantineAfterKills;
if (kills < 1 || kills > 100)
{
var clamped = Math.Clamp(kills, 1, 100);
notes.Add($"{prefix}{nameof(QuarantineAfterKills)} {kills} is outside 1–100; clamped to {clamped}.");
kills = clamped;
}
var unavailable = UnavailableGrace;
if (unavailable < TimeSpan.Zero || unavailable > MaxUnavailableGrace)
{
var clamped = unavailable < TimeSpan.Zero ? TimeSpan.Zero : MaxUnavailableGrace;
notes.Add($"{prefix}{nameof(UnavailableGrace)} {unavailable} is outside 0–{MaxUnavailableGrace}; clamped to {clamped}.");
unavailable = clamped;
}
var probation = IsolationProbationRuns;
if (probation < 1 || probation > 10_000)
{
var clamped = Math.Clamp(probation, 1, 10_000);
notes.Add($"{prefix}{nameof(IsolationProbationRuns)} {probation} is outside 1–10000; clamped to {clamped}.");
probation = clamped;
}
adjustments = notes;
return new ScriptWorkerOptions
{
Shards = shards,
HardGrace = grace,
HeapLimitBytes = heap,
QuarantineAfterKills = kills,
UnavailableGrace = unavailable,
IsolationProbationRuns = probation,
PeCacheDir = PeCacheDir,
ExecutablePath = ExecutablePath,
};
}
/// <summary>
/// Fail-closed path validation (design §7): <see cref="PeCacheDir"/>, when set, must be absolute and must not
/// sit inside <paramref name="contentRootPath"/>. Same rule as the family's <c>Secrets:SqlitePath</c>.
/// </summary>
/// <param name="contentRootPath">The host's content root.</param>
/// <returns>The violation, or null when the options are acceptable.</returns>
public string? ValidatePaths(string contentRootPath)
{
if (string.IsNullOrWhiteSpace(PeCacheDir)) return null;
const string key = ScriptingOptions.SectionName + ":Worker:" + nameof(PeCacheDir);
if (!Path.IsPathFullyQualified(PeCacheDir))
return $"{key} '{PeCacheDir}' must be an absolute path.";
var dir = Path.TrimEndingDirectorySeparator(Path.GetFullPath(PeCacheDir));
var root = Path.TrimEndingDirectorySeparator(Path.GetFullPath(contentRootPath));
var comparison = OperatingSystem.IsWindows() ? StringComparison.OrdinalIgnoreCase : StringComparison.Ordinal;
if (string.Equals(dir, root, comparison)
|| dir.StartsWith(root + Path.DirectorySeparatorChar, comparison)
|| dir.StartsWith(root + Path.AltDirectorySeparatorChar, comparison))
return $"{key} '{PeCacheDir}' is inside the content root '{contentRootPath}'; it must live outside it.";
return null;
}
}
@@ -72,6 +72,30 @@ public sealed class ScriptingOptions
/// </summary>
public double StuckOrphanQuarantineFactor { get; set; } = DefaultStuckOrphanQuarantineFactor;
/// <summary>
/// Where compiled scripts run (<c>Scripting:Runner</c>, lmxopcua#780). Default
/// <see cref="ScriptRunnerMode.InProcess"/> — today's path, unchanged, and the rollback for the other modes.
/// </summary>
public ScriptRunnerMode Runner { get; set; } = ScriptRunnerMode.InProcess;
/// <summary>
/// The seams <see cref="Runner"/> applies to when it is not <see cref="ScriptRunnerMode.InProcess"/>
/// (<c>Scripting:Runner:Seams</c>, e.g. <c>["Calculation"]</c>). Null or empty means all three. Bound by
/// <c>ScriptingOptionsBinding</c> from the documented key, because <c>Scripting:Runner</c> is itself a value.
/// </summary>
public IReadOnlyList<ScriptSeam>? RunnerSeams { get; set; }
/// <summary>The out-of-process worker pool's knobs (<c>Scripting:Worker</c>).</summary>
public ScriptWorkerOptions Worker { get; set; } = new();
/// <summary>Whether <paramref name="seam"/> is selected for out-of-process execution by these options.</summary>
/// <param name="seam">The seam.</param>
/// <returns><c>true</c> when <see cref="Runner"/> is <see cref="ScriptRunnerMode.OutOfProcess"/> and the seam is in
/// <see cref="RunnerSeams"/> (or that list is empty).</returns>
public bool IsOutOfProcess(ScriptSeam seam)
=> Runner == ScriptRunnerMode.OutOfProcess
&& (RunnerSeams is not { Count: > 0 } seams || seams.Contains(seam));
/// <summary>Upper bound <see cref="Sanitized"/> applies to <see cref="StuckOrphanQuarantineFactor"/>.</summary>
public const double MaxStuckOrphanQuarantineFactor = 1000;
@@ -158,6 +182,9 @@ public sealed class ScriptingOptions
runTimeout = MaxRunTimeout;
}
var worker = (Worker ?? new ScriptWorkerOptions()).Sanitized(out var workerNotes);
notes.AddRange(workerNotes);
adjustments = notes;
return new ScriptingOptions
{
@@ -165,6 +192,9 @@ public sealed class ScriptingOptions
MaxAllocBytesPerRun = alloc,
QuarantineAfterConsecutiveTimeouts = threshold,
StuckOrphanQuarantineFactor = factor,
Runner = Runner,
RunnerSeams = RunnerSeams is null ? null : [.. RunnerSeams],
Worker = worker,
};
}
}
@@ -137,7 +137,7 @@ public sealed class CalculationDriver : IDriver, IDependencyConsumer, ISubscriba
// _logger (ILogger<CalculationDriver>) implements the non-generic ILogger, so it is passed straight
// through — the evaluator's host-side diagnostics (compile errors, sandbox violations, execution
// throws) land on the SAME logger the driver itself was constructed with, not a separate one.
_evaluator = new CalculationEvaluator(_logger, _scriptRoot, _options.RunTimeout, _options.Scripting);
_evaluator = new CalculationEvaluator(_logger, _scriptRoot, _options.RunTimeout, _options.Scripting, _options.ScriptRunner);
RebuildTagTable(_options);
}
@@ -639,7 +639,7 @@ public sealed class CalculationDriver : IDriver, IDependencyConsumer, ISubscriba
if (!result.Success)
{
var reason = result.Reason ?? "evaluator failure";
toPublish = MarkEvaluationFailed(st, pending.TimestampUtc);
toPublish = MarkEvaluationFailed(st, pending.TimestampUtc, result.StatusCode ?? StatusBadInternalError);
// The script-log entry fires on the Good→Bad TRANSITION, or when the failure changes
// (R07:P4) — not on every trigger of a tag that is already Bad for the same reason, which
// flooded the script log at the input's cadence. The Bad publish was already once per
@@ -725,16 +725,19 @@ public sealed class CalculationDriver : IDriver, IDependencyConsumer, ISubscriba
/// returns before the evaluator ever runs), so there is no separate "inputs ready" condition here.</summary>
/// <param name="st">The tag's published state.</param>
/// <param name="timestampUtc">The trigger's timestamp.</param>
/// <param name="badStatus">The Bad code to publish (see the body).</param>
/// <returns>The Bad snapshot to publish on a fresh Good→Bad transition, else null.</returns>
private static (object? Value, uint StatusCode)? MarkEvaluationFailed(TagState st, DateTime timestampUtc)
private static (object? Value, uint StatusCode)? MarkEvaluationFailed(TagState st, DateTime timestampUtc, uint badStatus)
{
if (st.HasPublished && st.LastPublishedStatus == StatusBadInternalError) return null;
// badStatus is BadInternalError for every script failure; BadResourceUnavailable when the out-of-process
// runner could not run the script (lmxopcua#780). Once per transition, keyed on that code.
if (st.HasPublished && st.LastPublishedStatus == badStatus) return null;
st.HasPublished = true;
st.LastPublishedStatus = StatusBadInternalError;
st.LastPublishedStatus = badStatus;
st.LastTimestampUtc = timestampUtc;
// Carry the last-known value (null if none) with Bad quality.
return (st.HasValue ? st.LastValue : null, StatusBadInternalError);
return (st.HasValue ? st.LastValue : null, badStatus);
}
private void Publish(string rawPath, object? value, uint statusCode, DateTime timestampUtc)
@@ -40,14 +40,17 @@ public static class CalculationDriverFactoryExtensions
/// <param name="scriptRoot">Optional root script logger (fans failure entries onto the <c>script-logs</c> topic).</param>
/// <param name="scripting">The node-wide <c>Scripting</c> options every Calculation driver inherits; a key
/// authored in a driver's own config overrides that one knob. Null ⇒ the compiled defaults.</param>
/// <param name="scriptRunner">lmxopcua#780 — the node's out-of-process runner when the Calculation seam runs out
/// of process; null (the default) keeps every driver on the in-process guard.</param>
public static void Register(
DriverFactoryRegistry registry,
ILoggerFactory? loggerFactory = null,
ScriptRootLogger? scriptRoot = null,
ScriptingOptions? scripting = null)
ScriptingOptions? scripting = null,
IScriptRunner? scriptRunner = null)
{
ArgumentNullException.ThrowIfNull(registry);
registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory, scriptRoot, scripting));
registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory, scriptRoot, scripting, scriptRunner));
}
/// <summary>Public overload for the Server-side bootstrapper + test consumers.</summary>
@@ -74,10 +77,12 @@ public static class CalculationDriverFactoryExtensions
/// <param name="loggerFactory">Optional logger factory for per-instance diagnostics loggers.</param>
/// <param name="scriptRoot">Optional root script logger for failure fan-out.</param>
/// <param name="nodeScripting">The node-wide run-guard options; null ⇒ the compiled defaults.</param>
/// <param name="scriptRunner">The node's out-of-process runner, or null for in process.</param>
/// <returns>The constructed <see cref="CalculationDriver"/>.</returns>
public static CalculationDriver CreateInstance(
string driverInstanceId, string driverConfigJson,
ILoggerFactory? loggerFactory, ScriptRootLogger? scriptRoot, ScriptingOptions? nodeScripting)
ILoggerFactory? loggerFactory, ScriptRootLogger? scriptRoot, ScriptingOptions? nodeScripting,
IScriptRunner? scriptRunner = null)
{
ArgumentException.ThrowIfNullOrWhiteSpace(driverInstanceId);
ArgumentException.ThrowIfNullOrWhiteSpace(driverConfigJson);
@@ -101,6 +106,7 @@ public static class CalculationDriverFactoryExtensions
RunTimeout = dto.RunTimeoutMs is { } ms && ms > 0
? TimeSpan.FromMilliseconds(ms) : TimeSpan.FromSeconds(2),
Scripting = scripting,
ScriptRunner = scriptRunner,
};
return new CalculationDriver(options, driverInstanceId, scriptRoot, logger: logger);
@@ -18,4 +18,11 @@ public sealed class CalculationDriverOptions
/// <see cref="ScriptingOptions"/> defaults (64 MB, 3).
/// </summary>
public ScriptingOptions? Scripting { get; init; }
/// <summary>
/// lmxopcua#780 — the node's out-of-process runner when <c>Scripting:Runner=OutOfProcess</c> selects the
/// Calculation seam; null (the default) keeps the in-process run guard. Shared by every Calculation driver on the
/// node; keys are the calc tags' RawPaths.
/// </summary>
public IScriptRunner? ScriptRunner { get; init; }
}
@@ -59,6 +59,7 @@ public sealed class CalculationEvaluator : IScriptCacheOwner, IScriptQuarantineO
private readonly SerilogLogger _scriptRoot;
private readonly TimeSpan _runTimeout;
private readonly ScriptRunGuard _guard;
private readonly IScriptRunner? _runner;
// Negative compile cache (R07:P4), keyed by source hash. A compile error is a property of the SOURCE,
// which never changes between triggers (it is fixed in the deploy artifact) — yet every trigger of a
@@ -76,12 +77,16 @@ public sealed class CalculationEvaluator : IScriptCacheOwner, IScriptQuarantineO
/// <param name="scriptRoot">Root script logger; user <c>ctx.Logger.*</c> output flows through this to the Script-log page.</param>
/// <param name="runTimeout">Wall-clock budget per script run; defaults to 2 seconds (parity with the VT evaluator).</param>
/// <param name="scripting">Run-guard budget + quarantine threshold; null ⇒ <see cref="ScriptingOptions"/> defaults.</param>
/// <param name="scriptRunner">lmxopcua#780 — the out-of-process runner when the Calculation seam runs out of
/// process; null (the default) keeps the in-process guard path, unchanged.</param>
public CalculationEvaluator(
ILogger logger,
ScriptRootLogger scriptRoot,
TimeSpan? runTimeout = null,
ScriptingOptions? scripting = null)
ScriptingOptions? scripting = null,
IScriptRunner? scriptRunner = null)
{
_runner = scriptRunner;
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
_scriptRoot = (scriptRoot ?? throw new ArgumentNullException(nameof(scriptRoot))).Logger;
_runTimeout = runTimeout ?? TimeSpan.FromSeconds(2);
@@ -188,6 +193,15 @@ public sealed class CalculationEvaluator : IScriptCacheOwner, IScriptQuarantineO
calcTagId, path),
logger: scriptLog);
// lmxopcua#780 — OutOfProcess: the pre-emptible worker pool compiles and runs it, and owns timeouts, kills,
// quarantine and the negative compile cache for this seam. Unavailable publishes BadResourceUnavailable.
if (_runner is not null)
{
var outcome = await _runner.RunAsync(new ScriptInvocation(
calcTagId, ScriptContextKind.Calculation, scriptSource, readCache, _runTimeout, scriptLog), ct).ConfigureAwait(false);
return outcome.ToVirtualTagEvalResult();
}
// A source that already failed to compile fails the same way without touching Roslyn (R07:P4).
var sourceKey = HashSource(scriptSource);
if (_compileFailures.TryGetValue(sourceKey, out var knownFailure))
@@ -295,13 +309,20 @@ public sealed class CalculationEvaluator : IScriptCacheOwner, IScriptQuarantineO
// The compile-cache clear is this evaluator's deploy boundary — a redeploy lifts every quarantine
// (an edited source already lifts its own through the guard's source identity).
_guard.ResetAll();
_runner?.Reset(ScriptContextKind.Calculation);
}
/// <inheritdoc />
public void ResetQuarantines() => _guard.ResetAll();
public void ResetQuarantines()
{
_guard.ResetAll();
_runner?.Reset(ScriptContextKind.Calculation);
}
/// <inheritdoc />
public IReadOnlyCollection<string> QuarantinedScripts => _guard.QuarantinedKeys;
public IReadOnlyCollection<string> QuarantinedScripts => _runner is null
? _guard.QuarantinedKeys
: [.. _guard.QuarantinedKeys, .. _runner.QuarantinedKeys(ScriptContextKind.Calculation)];
/// <summary>Disposes the evaluator, draining + disposing every cached compile's collectible ALC.</summary>
public void Dispose()
@@ -30,6 +30,8 @@
<ItemGroup>
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.Driver.Calculation.Tests"/>
<!-- lmxopcua#780: the worker suite drives a real driver over the real ScriptWorker (IsEvaluationIdle). -->
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests"/>
</ItemGroup>
</Project>
@@ -0,0 +1,94 @@
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
using SerilogLogger = Serilog.ILogger;
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// <summary>
/// Which <see cref="IScriptRunner"/> each script seam uses on this node (lmxopcua#780, <c>Scripting:Runner</c>). Under
/// the <see cref="ScriptRunnerMode.InProcess"/> default every seam gets null and keeps its in-process guard path,
/// unchanged. Under <see cref="ScriptRunnerMode.OutOfProcess"/> the selected seams share one
/// <see cref="OutOfProcessScriptRunner"/>, which starts no worker until the first compiled evaluation, so a node with
/// no scripts (or an admin-only node) never launches one. Disposed with the host's service provider, which stops the
/// workers.
/// </summary>
public sealed class ScriptRunnerSelection : IAsyncDisposable, IDisposable
{
/// <summary>Seams the out-of-process runner can serve in this build. Scripted alarms go last (design §14 #10).</summary>
public static readonly IReadOnlyList<ScriptSeam> SupportedOutOfProcessSeams = [ScriptSeam.VirtualTag, ScriptSeam.Calculation];
private readonly OutOfProcessScriptRunner? _runner;
private readonly HashSet<ScriptSeam> _seams;
private ScriptRunnerSelection(OutOfProcessScriptRunner? runner, IEnumerable<ScriptSeam> seams)
{
_runner = runner;
_seams = [.. seams];
}
/// <summary>The out-of-process runner, when one is configured.</summary>
public IScriptRunner? OutOfProcessRunner => _runner;
/// <summary>The runner <paramref name="seam"/> must use, or null for its in-process path.</summary>
/// <param name="seam">The seam.</param>
/// <returns>The runner, or null.</returns>
public IScriptRunner? ForSeam(ScriptSeam seam) => _runner is not null && _seams.Contains(seam) ? _runner : null;
/// <summary>
/// Startup checks for the runner options: returns the Warnings to log, and throws when the configuration must
/// not start (a <c>PeCacheDir</c> that is relative or inside the content root — fail closed).
/// </summary>
/// <param name="options">The bound, sanitized <c>Scripting</c> options.</param>
/// <param name="contentRootPath">The host's content root.</param>
/// <returns>Warnings for the caller to log.</returns>
/// <exception cref="InvalidOperationException">The worker path options are unacceptable.</exception>
public static IReadOnlyList<string> Validate(ScriptingOptions options, string contentRootPath)
{
ArgumentNullException.ThrowIfNull(options);
var warnings = new List<string>();
if (options.Runner == ScriptRunnerMode.InProcess) return warnings;
if (options.Worker.ValidatePaths(contentRootPath) is { } violation)
throw new InvalidOperationException($"Scripting misconfiguration: {violation}");
if (options.Runner == ScriptRunnerMode.Shadow)
{
warnings.Add("Scripting:Runner=Shadow is not implemented yet (lmxopcua#780 P2); every seam runs InProcess.");
return warnings;
}
foreach (var seam in RequestedSeams(options))
{
if (!SupportedOutOfProcessSeams.Contains(seam))
warnings.Add($"Scripting:Runner=OutOfProcess: the {seam} seam cannot run out of process yet (lmxopcua#780); it stays InProcess.");
}
var workerPath = OutOfProcessScriptRunner.ResolveWorkerPath(options.Worker);
if (!File.Exists(workerPath))
warnings.Add($"Scripting:Runner=OutOfProcess: the script worker '{workerPath}' does not exist; every out-of-process " +
"script reports BadResourceUnavailable (the runner fails closed, it never falls back to in process).");
return warnings;
}
/// <summary>Builds the selection for <paramref name="options"/>.</summary>
/// <param name="options">The bound, sanitized <c>Scripting</c> options.</param>
/// <param name="log">Host diagnostics for the runner.</param>
/// <param name="scriptRoot">The root script logger forwarded <c>ctx.Logger</c> lines fall back to.</param>
/// <returns>The selection.</returns>
public static ScriptRunnerSelection Create(ScriptingOptions options, SerilogLogger log, SerilogLogger? scriptRoot)
{
ArgumentNullException.ThrowIfNull(options);
if (options.Runner != ScriptRunnerMode.OutOfProcess) return new ScriptRunnerSelection(null, []);
var seams = RequestedSeams(options).Where(SupportedOutOfProcessSeams.Contains).ToArray();
if (seams.Length == 0) return new ScriptRunnerSelection(null, []);
return new ScriptRunnerSelection(new OutOfProcessScriptRunner(options, log, scriptRoot), seams);
}
/// <inheritdoc />
public ValueTask DisposeAsync() => _runner?.DisposeAsync() ?? ValueTask.CompletedTask;
/// <inheritdoc />
public void Dispose() => _runner?.Dispose();
private static IEnumerable<ScriptSeam> RequestedSeams(ScriptingOptions options)
=> options.RunnerSeams is { Count: > 0 } seams ? seams.Distinct() : Enum.GetValues<ScriptSeam>();
}
@@ -18,6 +18,11 @@ public static class ScriptingOptionsBinding
{
ArgumentNullException.ThrowIfNull(configuration);
var bound = configuration.GetSection(ScriptingOptions.SectionName).Get<ScriptingOptions>() ?? new ScriptingOptions();
// lmxopcua#780: the seam subset lives under the runner key (Scripting:Runner:Seams), which the binder cannot
// reach because Scripting:Runner is itself a value. Bound explicitly; an unknown seam name is a startup error,
// like an unknown Scripting:Runner value.
var seams = configuration.GetSection($"{ScriptingOptions.SectionName}:Runner:Seams").Get<List<ScriptSeam>>();
if (seams is { Count: > 0 }) bound.RunnerSeams = seams;
return bound.Sanitized(out adjustments);
}
}
@@ -102,7 +102,10 @@ public static class DriverFactoryBootstrap
// failures; without this it writes to NullLogger in production.
ConfigureAbTransportLogging(sp);
Register(registry, loggerFactory, secretResolver, primaryRoleSource, ResolveAbCipDefaultTransport(sp),
sp.GetRequiredService<Driver.Galaxy.Runtime.IGalaxyAckIdentityDirectory>(), scriptRoot, scripting);
sp.GetRequiredService<Driver.Galaxy.Runtime.IGalaxyAckIdentityDirectory>(), scriptRoot, scripting,
// lmxopcua#780: null under the InProcess default (and in test graphs that register no selection).
sp.GetService<ZB.MOM.WW.OtOpcUa.Host.Configuration.ScriptRunnerSelection>()
?.ForSeam(ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptSeam.Calculation));
return registry;
});
services.AddSingleton<IDriverFactory>(sp =>
@@ -233,11 +236,12 @@ public static class DriverFactoryBootstrap
Driver.AbCip.AbCipTransport abCipDefaultTransport,
Driver.Galaxy.Runtime.IGalaxyAckIdentityDirectory galaxyAckIdentities,
ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptRootLogger? scriptRoot = null,
ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptingOptions? scripting = null)
ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptingOptions? scripting = null,
ZB.MOM.WW.OtOpcUa.Core.Scripting.IScriptRunner? scriptRunner = null)
{
Driver.AbCip.AbCipDriverFactoryExtensions.Register(registry, loggerFactory, abCipDefaultTransport);
Driver.AbLegacy.AbLegacyDriverFactoryExtensions.Register(registry, loggerFactory);
Driver.Calculation.CalculationDriverFactoryExtensions.Register(registry, loggerFactory, scriptRoot, scripting);
Driver.Calculation.CalculationDriverFactoryExtensions.Register(registry, loggerFactory, scriptRoot, scripting, scriptRunner);
Driver.FOCAS.FocasDriverFactoryExtensions.Register(registry, loggerFactory);
Driver.Galaxy.GalaxyDriverFactoryExtensions.Register(registry, secretResolver, loggerFactory, galaxyAckIdentities);
Driver.Modbus.ModbusDriverFactoryExtensions.Register(registry, loggerFactory);
@@ -53,6 +53,7 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac
private readonly SerilogLogger _scriptRoot;
private readonly TimeSpan _runTimeout;
private readonly ScriptRunGuard _guard;
private readonly IScriptRunner? _runner;
private bool _disposed;
/// <summary>Initializes a new RoslynVirtualTagEvaluator with the given loggers and optional timeout.</summary>
@@ -61,18 +62,28 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac
/// <param name="runTimeout">Maximum execution time for each script; null ⇒ <see cref="ScriptingOptions.RunTimeout"/>
/// of <paramref name="scripting"/> (2 s by default).</param>
/// <param name="scripting">Run-guard budget + quarantine threshold (the <c>Scripting</c> section); null ⇒ defaults.</param>
/// <param name="scriptRunner">
/// lmxopcua#780 — the out-of-process runner when <c>Scripting:Runner=OutOfProcess</c> selects the VirtualTag seam;
/// null (the default, <c>InProcess</c>) keeps the in-process guard path below, unchanged. The passthrough fast path
/// never reaches it.
/// </param>
public RoslynVirtualTagEvaluator(
ILogger<RoslynVirtualTagEvaluator> logger,
ScriptRootLogger scriptRoot,
TimeSpan? runTimeout = null,
ScriptingOptions? scripting = null)
ScriptingOptions? scripting = null,
IScriptRunner? scriptRunner = null)
{
_logger = logger;
_scriptRoot = (scriptRoot ?? throw new ArgumentNullException(nameof(scriptRoot))).Logger;
_runTimeout = runTimeout ?? (scripting ?? new ScriptingOptions()).Sanitized(out _).RunTimeout;
_guard = new ScriptRunGuard(scripting, _scriptRoot);
_runner = scriptRunner;
}
/// <summary>The out-of-process runner this evaluator delegates compiled runs to, or null (in process).</summary>
public IScriptRunner? ScriptRunner => _runner;
/// <summary>The run guard every compiled evaluation goes through (diagnostics / tests).</summary>
public ScriptRunGuard RunGuard => _guard;
@@ -147,6 +158,16 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac
virtualTagId, path),
logger: scriptLog);
// lmxopcua#780 — OutOfProcess: the compiled run goes to the pre-emptible worker pool, which compiles in the
// worker and owns timeouts, kills and quarantine for this seam. An Unavailable outcome (worker restarted, no
// worker ready) publishes BadResourceUnavailable; everything else keeps today's Bad contract.
if (_runner is not null)
{
var outcome = await _runner.RunAsync(new ScriptInvocation(
virtualTagId, ScriptContextKind.VirtualTag, expression, readCache, _runTimeout, scriptLog)).ConfigureAwait(false);
return outcome.ToVirtualTagEvalResult();
}
// 02/S12: fetch + run inside a bounded loop so an ObjectDisposedException caused by a concurrent
// apply-boundary ClearCompiledScripts (which disposed the evaluator between GetOrCompile and the
// run's disposed guard) re-fetches and retries exactly once. The loop runs at most twice — the
@@ -235,13 +256,20 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac
// The compile-cache clear IS the deploy boundary for this evaluator: a redeploy lifts every quarantine
// (an edited expression already lifts its own through the guard's source identity).
_guard.ResetAll();
_runner?.Reset(ScriptContextKind.VirtualTag);
}
/// <inheritdoc />
public void ResetQuarantines() => _guard.ResetAll();
public void ResetQuarantines()
{
_guard.ResetAll();
_runner?.Reset(ScriptContextKind.VirtualTag);
}
/// <inheritdoc />
public IReadOnlyCollection<string> QuarantinedScripts => _guard.QuarantinedKeys;
public IReadOnlyCollection<string> QuarantinedScripts => _runner is null
? _guard.QuarantinedKeys
: [.. _guard.QuarantinedKeys, .. _runner.QuarantinedKeys(ScriptContextKind.VirtualTag)];
/// <summary>Disposes the evaluator and cleans up all cached script evaluators.</summary>
public void Dispose()
+13 -1
View File
@@ -376,6 +376,16 @@ if (hasDriver)
Log.Warning("Scripting misconfiguration detected at startup: {Adjustment}", adjustment);
builder.Services.AddSingleton(scriptingOptions);
// lmxopcua#780: Scripting:Runner (InProcess default / Shadow / OutOfProcess). Under the default every seam keeps its
// in-process guard path unchanged; OutOfProcess hands the selected seams' compiled runs to the pre-emptible
// ScriptWorker pool. A PeCacheDir that is relative or inside the content root stops startup (fail closed).
foreach (var warning in ZB.MOM.WW.OtOpcUa.Host.Configuration.ScriptRunnerSelection.Validate(
scriptingOptions, builder.Environment.ContentRootPath))
Log.Warning("Scripting runner: {RunnerWarning}", warning);
Log.Information("Scripting runner: {RunnerMode}", scriptingOptions.Runner);
builder.Services.AddSingleton(sp => ZB.MOM.WW.OtOpcUa.Host.Configuration.ScriptRunnerSelection.Create(
sp.GetRequiredService<ScriptingOptions>(), Log.Logger, sp.GetService<ScriptRootLogger>()?.Logger));
// lmxopcua#777: the ONE operator-write timeout (DriverWrites:Timeout) every write bound is derived from — the
// driver child's cancel + abandon, its queue budget, DriverHostActor's Ask and the inbound-write gateway's Ask.
// Checked against the resilience tiers' Write timeouts here; an inconsistency is a startup Warning (a tier
@@ -392,7 +402,9 @@ if (hasDriver)
new RoslynVirtualTagEvaluator(
sp.GetRequiredService<ILoggerFactory>().CreateLogger<RoslynVirtualTagEvaluator>(),
sp.GetRequiredService<ScriptRootLogger>(),
scripting: sp.GetRequiredService<ScriptingOptions>()));
scripting: sp.GetRequiredService<ScriptingOptions>(),
scriptRunner: sp.GetRequiredService<ZB.MOM.WW.OtOpcUa.Host.Configuration.ScriptRunnerSelection>()
.ForSeam(ScriptSeam.VirtualTag)));
builder.Services.AddSingleton<IVirtualTagEvaluator>(sp => sp.GetRequiredService<RoslynVirtualTagEvaluator>());
// Script-log fan-out (Layer 0). The DPS publisher resolves the ActorSystem lazily so it never
@@ -57,6 +57,7 @@
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Configuration\ZB.MOM.WW.OtOpcUa.Configuration.csproj"/>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core\ZB.MOM.WW.OtOpcUa.Core.csproj"/>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.Scripting\ZB.MOM.WW.OtOpcUa.Core.Scripting.csproj"/>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess\ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess.csproj"/>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.VirtualTags\ZB.MOM.WW.OtOpcUa.Core.VirtualTags.csproj"/>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms\ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms.csproj"/>
<ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.Security\ZB.MOM.WW.OtOpcUa.Security.csproj"/>
@@ -90,4 +91,7 @@
<ProjectReference Include="..\..\Drivers\ZB.MOM.WW.OtOpcUa.Driver.TwinCAT\ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.csproj"/>
</ItemGroup>
<!-- lmxopcua#780: the out-of-process script runner's worker ships in script-worker/ beside the host. -->
<Import Project="..\ZB.MOM.WW.OtOpcUa.ScriptWorker\build\ScriptWorker.Deploy.targets"/>
</Project>
@@ -391,7 +391,8 @@ public sealed class VirtualTagActor : ReceiveActor, IWithUnboundedStash
var result = evaluation.Result;
if (!result.Success)
{
OnEvaluationFailed(result.Reason ?? "evaluator failure", "Warning", timestampUtc);
OnEvaluationFailed(result.Reason ?? "evaluator failure", "Warning", timestampUtc,
result.StatusCode ?? OpcUaStatus.BadInternalError);
return;
}
@@ -472,7 +473,10 @@ public sealed class VirtualTagActor : ReceiveActor, IWithUnboundedStash
/// <param name="reason">The failure reason for the script-log entry.</param>
/// <param name="level">The script-log level ("Warning" for a Failure result, "Error" for a throw).</param>
/// <param name="timestampUtc">The triggering dependency change's timestamp, stamped on the Bad result.</param>
private void OnEvaluationFailed(string reason, string level, DateTime timestampUtc)
/// <param name="badStatus">The Bad code to publish: <see cref="OpcUaStatus.BadInternalError"/> for every script
/// failure, <see cref="OpcUaStatus.BadResourceUnavailable"/> when the out-of-process runner could not run the
/// script (lmxopcua#780). The once-per-transition dedup is on this code, so a switch between the two publishes.</param>
private void OnEvaluationFailed(string reason, string level, DateTime timestampUtc, uint badStatus = OpcUaStatus.BadInternalError)
{
OtOpcUaTelemetry.VirtualTagEval.Add(1, new KeyValuePair<string, object?>("outcome", "fail"));
@@ -488,15 +492,15 @@ public sealed class VirtualTagActor : ReceiveActor, IWithUnboundedStash
}
if (admission.Admitted) PublishLog(level, reason);
if ((_hasPublished && _lastPublishedStatus == OpcUaStatus.BadInternalError) || !AllDependenciesArrived())
if ((_hasPublished && _lastPublishedStatus == badStatus) || !AllDependenciesArrived())
{
return;
}
_hasPublished = true;
_lastPublishedStatus = OpcUaStatus.BadInternalError;
_lastPublishedStatus = badStatus;
Context.Parent.Tell(new EvaluationResult(
_virtualTagId, _hasLastValue ? _lastValue : null, timestampUtc, CorrelationId.NewId(), OpcUaStatus.BadInternalError));
_virtualTagId, _hasLastValue ? _lastValue : null, timestampUtc, CorrelationId.NewId(), badStatus));
}
/// <summary>Inputs-ready gate (the retired cascade engine's rule): true once
@@ -0,0 +1,36 @@
using System.IO.Pipes;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker;
/// <summary>Entry point: connect to the host's pipe named in the environment and serve until told to stop.</summary>
internal static class Program
{
private static async Task<int> Main()
{
var pipeName = Environment.GetEnvironmentVariable(ScriptWorkerEnvironment.PipeName);
var nonce = Environment.GetEnvironmentVariable(ScriptWorkerEnvironment.Nonce);
if (string.IsNullOrEmpty(pipeName) || string.IsNullOrEmpty(nonce))
{
await Console.Error.WriteLineAsync("ScriptWorker: launched without a pipe name or nonce; exiting.");
return ScriptWorkerEnvironment.ExitProtocolFault;
}
_ = int.TryParse(Environment.GetEnvironmentVariable(ScriptWorkerEnvironment.ParentPid), out var parentPid);
// CurrentUserOnly: the client verifies the server end is owned by this account as well.
var pipe = new NamedPipeClientStream(".", pipeName, PipeDirection.InOut,
PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly);
try
{
await pipe.ConnectAsync(TimeSpan.FromSeconds(10), CancellationToken.None);
}
catch (Exception ex)
{
await Console.Error.WriteLineAsync($"ScriptWorker: could not connect to the host pipe: {ex.Message}");
return ScriptWorkerEnvironment.ExitProtocolFault;
}
var host = new WorkerHost(pipe, nonce, parentPid);
return await host.RunAsync();
}
}
@@ -0,0 +1,576 @@
using System.Collections;
using System.Collections.Concurrent;
using System.Diagnostics;
using System.IO.Pipes;
using System.Threading.Channels;
using Serilog;
using Serilog.Core;
using Serilog.Events;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
using ZB.MOM.WW.OtOpcUa.Core.VirtualTags;
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker;
/// <summary>
/// One worker's whole life (lmxopcua#780, design §3.2): handshake, compile-on-load, a bounded evaluation scheduler
/// on dedicated threads, a hard-deadline watchdog, heartbeats, and log forwarding. The host is authoritative for
/// deadlines (it kills this process); the watchdog is the belt to that brace and fails fast slightly later, so the
/// exit names the key that overran.
/// </summary>
/// <remarks>
/// Anything that means "the host is gone" — pipe end of stream, a protocol fault, the parent pid disappearing —
/// ends the process. The worker never outlives its host (design §3.3).
/// </remarks>
internal sealed class WorkerHost
{
/// <summary>The watchdog fails fast this long after the host's own hard deadline, so the host normally wins.</summary>
private static readonly TimeSpan WatchdogMargin = TimeSpan.FromMilliseconds(500);
private readonly NamedPipeClientStream _pipe;
private readonly string _nonce;
private readonly int _parentPid;
private readonly Channel<Envelope> _outbound = Channel.CreateUnbounded<Envelope>(
new UnboundedChannelOptions { SingleReader = true });
private readonly ConcurrentDictionary<string, LoadedScript> _scripts = new(StringComparer.Ordinal);
private readonly ConcurrentDictionary<long, RunningEval> _running = new();
private readonly BlockingCollection<EvalRequest> _queue = new(new ConcurrentQueue<EvalRequest>());
private readonly ForwardingLogSink _logSink;
private readonly ILogger _scriptLogger;
private int _maxFrameBytes = ScriptWorkerFrameCodec.DefaultMaxFrameBytes;
private int _queueCapacity = 4096;
private TimeSpan _hardGrace = TimeSpan.FromSeconds(1);
private TimeSpan _heartbeatInterval = TimeSpan.FromSeconds(1);
private long _seq;
public WorkerHost(NamedPipeClientStream pipe, string nonce, int parentPid)
{
_pipe = pipe;
_nonce = nonce;
_parentPid = parentPid;
_logSink = new ForwardingLogSink();
_scriptLogger = new LoggerConfiguration().MinimumLevel.Verbose().WriteTo.Sink(_logSink).CreateLogger();
}
public async Task<int> RunAsync()
{
try
{
var hello = await ScriptWorkerFrameCodec.ReadAsync(_pipe, ScriptWorkerFrameCodec.DefaultMaxFrameBytes);
if (hello is null) return ScriptWorkerEnvironment.ExitPipeClosed;
if (hello.BodyCase != Envelope.BodyOneofCase.Hello)
throw new ScriptWorkerProtocolException($"expected Hello, got {hello.BodyCase}");
Configure(hello.Hello);
// The nonce is echoed, not trusted: the host compares it with the one it put in OUR environment, which
// proves the process on the other end of its pipe is the one it launched.
await ScriptWorkerFrameCodec.WriteAsync(_pipe, Wrap(new Envelope
{
Ready = new Ready
{
Nonce = _nonce,
Pid = Environment.ProcessId,
RuntimeVersion = Environment.Version.ToString(),
},
}), _maxFrameBytes);
_logSink.Capacity = (int)Math.Max(1, hello.Hello.MaxLogLines == 0 ? 1000 : hello.Hello.MaxLogLines);
StartBackground(Math.Max(1, (int)hello.Hello.EvalThreads));
var writer = Task.Run(WriteLoopAsync);
var exit = await ReadLoopAsync();
_outbound.Writer.TryComplete();
try { await writer.WaitAsync(TimeSpan.FromSeconds(1)); } catch { /* exiting regardless */ }
return exit;
}
catch (ScriptWorkerProtocolException ex)
{
await Console.Error.WriteLineAsync($"ScriptWorker: protocol fault: {ex.Message}");
return ScriptWorkerEnvironment.ExitProtocolFault;
}
catch (IOException)
{
return ScriptWorkerEnvironment.ExitPipeClosed;
}
}
private void Configure(Hello hello)
{
if (hello.MaxFrameBytes > 0) _maxFrameBytes = (int)Math.Min(hello.MaxFrameBytes, 64u * 1024 * 1024);
if (hello.QueueCapacity > 0) _queueCapacity = (int)hello.QueueCapacity;
if (hello.HardGraceMs > 0) _hardGrace = TimeSpan.FromMilliseconds(hello.HardGraceMs);
if (hello.HeartbeatIntervalMs > 0) _heartbeatInterval = TimeSpan.FromMilliseconds(hello.HeartbeatIntervalMs);
}
private void StartBackground(int evalThreads)
{
for (var i = 0; i < evalThreads; i++)
{
new Thread(EvalThreadLoop) { IsBackground = true, Name = $"ScriptWorker.Eval{i}" }.Start();
}
new Thread(WatchdogLoop) { IsBackground = true, Name = "ScriptWorker.Watchdog" }.Start();
_ = Task.Run(HeartbeatLoopAsync);
_ = Task.Run(LogFlushLoopAsync);
if (_parentPid > 0) _ = Task.Run(ParentWatchLoopAsync);
}
private async Task<int> ReadLoopAsync()
{
while (true)
{
Envelope? frame;
try
{
frame = await ScriptWorkerFrameCodec.ReadAsync(_pipe, _maxFrameBytes);
}
catch (ScriptWorkerProtocolException ex)
{
await Console.Error.WriteLineAsync($"ScriptWorker: protocol fault: {ex.Message}");
return ScriptWorkerEnvironment.ExitProtocolFault;
}
catch (IOException)
{
return ScriptWorkerEnvironment.ExitPipeClosed;
}
if (frame is null) return ScriptWorkerEnvironment.ExitPipeClosed; // host gone: fatal by design
switch (frame.BodyCase)
{
case Envelope.BodyOneofCase.Load:
{
// Compile off the reader, so evaluations for already-loaded keys keep flowing during a load.
var load = frame.Load;
_ = Task.Run(() => Load(load));
break;
}
case Envelope.BodyOneofCase.Unload:
foreach (var key in frame.Unload.Keys)
if (_scripts.TryRemove(key, out var gone)) gone.Dispose();
break;
case Envelope.BodyOneofCase.Eval:
foreach (var item in frame.Eval.Items) Admit(item);
break;
case Envelope.BodyOneofCase.Cancel:
if (_running.TryGetValue((long)frame.Cancel.EvalId, out var toCancel))
{
try { toCancel.Cancellation.Cancel(); } catch (ObjectDisposedException) { /* just finished */ }
}
break;
case Envelope.BodyOneofCase.DiagRequest:
Send(new Envelope { DiagReply = Diagnostics() });
break;
case Envelope.BodyOneofCase.Shutdown:
return 0;
case Envelope.BodyOneofCase.Fault:
await Console.Error.WriteLineAsync($"ScriptWorker: host fault: {frame.Fault.Message}");
return ScriptWorkerEnvironment.ExitProtocolFault;
default:
await Console.Error.WriteLineAsync($"ScriptWorker: unexpected frame {frame.BodyCase}");
return ScriptWorkerEnvironment.ExitProtocolFault;
}
}
}
private void Load(LoadScripts load)
{
var result = new LoadResult { LoadId = load.LoadId };
foreach (var script in load.Scripts)
{
var outcome = new LoadOutcome { Key = script.Key, Ok = true };
try
{
var loaded = LoadedScript.Compile(script);
if (_scripts.TryRemove(script.Key, out var previous)) previous.Dispose();
_scripts[script.Key] = loaded;
}
catch (CompilationErrorException ex)
{
outcome.Ok = false;
outcome.Error = new ScriptError { Kind = ScriptError.Types.Kind.Compile, Message = ex.Message };
}
catch (ScriptSandboxViolationException ex)
{
outcome.Ok = false;
outcome.Error = new ScriptError { Kind = ScriptError.Types.Kind.Sandbox, Message = ex.Message };
}
catch (Exception ex)
{
outcome.Ok = false;
outcome.Error = new ScriptError { Kind = ScriptError.Types.Kind.Threw, Message = ex.Message };
}
result.Outcomes.Add(outcome);
}
if (ForbiddenAssemblyLoaded() is { } forbidden) DieForbidden(forbidden);
Send(new Envelope { LoadResult = result });
}
private void Admit(EvalRequest request)
{
if (_queue.Count >= _queueCapacity)
{
SendResult(new EvalResult
{
EvalId = request.EvalId,
Error = new ScriptError { Kind = ScriptError.Types.Kind.Busy, Message = "script worker busy" },
});
return;
}
_queue.Add(request);
}
private void EvalThreadLoop()
{
foreach (var request in _queue.GetConsumingEnumerable())
{
EvalResult result;
try
{
result = Evaluate(request);
}
catch (Exception ex)
{
result = new EvalResult
{
EvalId = request.EvalId,
Error = new ScriptError { Kind = ScriptError.Types.Kind.Threw, Message = ex.Message },
};
}
SendResult(result);
}
}
private EvalResult Evaluate(EvalRequest request)
{
var result = new EvalResult { EvalId = request.EvalId };
if (!_scripts.TryGetValue(request.Key, out var script))
{
result.Error = new ScriptError { Kind = ScriptError.Types.Kind.NotLoaded, Message = $"script '{request.Key}' is not loaded" };
return result;
}
var inputs = new Dictionary<string, DataValueSnapshot>(request.Inputs.Count, StringComparer.Ordinal);
foreach (var input in request.Inputs)
{
string path;
if (!string.IsNullOrEmpty(input.Path)) path = input.Path;
else if (input.Slot < script.InputPaths.Length) path = script.InputPaths[input.Slot];
else throw new ScriptWorkerProtocolException($"input slot {input.Slot} is not registered for '{request.Key}'");
if (input.Unsupported)
{
result.Error = new ScriptError
{
Kind = ScriptError.Types.Kind.UnsupportedType,
Message = $"input '{path}' has a value of unsupported type {input.UnsupportedType}",
};
return result;
}
inputs[path] = new DataValueSnapshot(
VariantCodec.Decode(input.Value),
input.StatusCode,
input.SourceTsTicks == 0 ? null : new DateTime(input.SourceTsTicks, DateTimeKind.Utc),
new DateTime(input.ServerTsTicks, DateTimeKind.Utc));
}
// ctx.Now follows the HOST clock (design §4.3) but keeps ticking: a fixed instant would never let a script
// that waits on ctx.Now finish.
var offset = request.NowUtcTicks == 0
? TimeSpan.Zero
: new DateTime(request.NowUtcTicks, DateTimeKind.Utc) - DateTime.UtcNow;
Func<DateTime> clock = () => DateTime.UtcNow + offset;
var logger = _scriptLogger.ForContext(ForwardingLogSink.KeyProperty, request.Key);
var soft = TimeSpan.FromMilliseconds(Math.Max(1, request.SoftTimeoutMs));
// Cancelled by the host's Cancel frame at the soft deadline, never by our own clock (see the proto).
using var cts = new CancellationTokenSource();
var grace = request.HardGraceMs > 0 ? TimeSpan.FromMilliseconds(request.HardGraceMs) : _hardGrace;
var running = new RunningEval(request.Key, Stopwatch.GetTimestamp(), soft + grace + WatchdogMargin, cts);
_running[(long)request.EvalId] = running;
var before = GC.GetAllocatedBytesForCurrentThread();
var sw = Stopwatch.StartNew();
try
{
object? value = script.Run(inputs, logger, clock, cts.Token, out var isPredicate);
if (isPredicate)
{
result.Predicate = (bool)value!;
}
else if (VariantCodec.TryEncode(value, out var variant, out var unsupported))
{
result.Value = variant;
}
else
{
result.Error = new ScriptError
{
Kind = ScriptError.Types.Kind.UnsupportedType,
Message = $"script returned a value of unsupported type {unsupported}",
};
}
}
catch (Exception ex)
{
result.Error = new ScriptError { Kind = ScriptError.Types.Kind.Threw, Message = ex.Message };
}
finally
{
_running.TryRemove((long)request.EvalId, out _);
}
result.AllocatedBytes = (ulong)Math.Max(0, GC.GetAllocatedBytesForCurrentThread() - before);
result.RunMicros = (uint)Math.Min(uint.MaxValue, sw.Elapsed.TotalMicroseconds);
return result;
}
private void WatchdogLoop()
{
while (true)
{
Thread.Sleep(50);
foreach (var (_, run) in _running)
{
if (Stopwatch.GetElapsedTime(run.StartedTimestamp) > run.FailFastAfter)
{
// The host should have killed us already; this names the culprit in the exit/dump.
Environment.FailFast($"ScriptWorker: script '{run.Key}' exceeded its hard deadline ({run.FailFastAfter.TotalMilliseconds:F0} ms)");
}
}
}
}
private async Task HeartbeatLoopAsync()
{
using var timer = new PeriodicTimer(_heartbeatInterval);
while (await timer.WaitForNextTickAsync())
{
Send(new Envelope
{
Heartbeat = new Heartbeat
{
Inflight = (uint)_running.Count,
Queued = (uint)_queue.Count,
HeapBytes = GC.GetTotalMemory(false),
Gen2Count = (uint)GC.CollectionCount(2),
},
});
}
}
private async Task LogFlushLoopAsync()
{
using var timer = new PeriodicTimer(TimeSpan.FromMilliseconds(100));
while (await timer.WaitForNextTickAsync())
{
if (_logSink.Drain() is { } batch) Send(new Envelope { Logs = batch });
}
}
private async Task ParentWatchLoopAsync()
{
using var timer = new PeriodicTimer(TimeSpan.FromMilliseconds(500));
while (await timer.WaitForNextTickAsync())
{
if (!ParentAlive())
{
await Console.Error.WriteLineAsync("ScriptWorker: parent process is gone; exiting.");
Environment.Exit(ScriptWorkerEnvironment.ExitParentGone);
}
}
}
private bool ParentAlive()
{
try
{
using var parent = Process.GetProcessById(_parentPid);
return !parent.HasExited;
}
catch (ArgumentException)
{
return false;
}
catch (InvalidOperationException)
{
return false;
}
}
private async Task WriteLoopAsync()
{
var reader = _outbound.Reader;
var pendingResults = new EvalResults();
try
{
while (await reader.WaitToReadAsync())
{
// Opportunistic coalescing (design §4.5): drain whatever is queued right now, merging every completed
// evaluation into one EvalResults frame; no timer, so an idle pipe adds no latency.
while (reader.TryRead(out var envelope))
{
if (envelope.BodyCase == Envelope.BodyOneofCase.Results)
{
pendingResults.Items.AddRange(envelope.Results.Items);
continue;
}
await ScriptWorkerFrameCodec.WriteAsync(_pipe, envelope, _maxFrameBytes);
}
if (pendingResults.Items.Count > 0)
{
await ScriptWorkerFrameCodec.WriteAsync(_pipe, Wrap(new Envelope { Results = pendingResults }), _maxFrameBytes);
pendingResults = new EvalResults();
}
await _pipe.FlushAsync();
}
}
catch (IOException)
{
Environment.Exit(ScriptWorkerEnvironment.ExitPipeClosed);
}
catch (ObjectDisposedException)
{
Environment.Exit(ScriptWorkerEnvironment.ExitPipeClosed);
}
}
private void SendResult(EvalResult result)
{
var results = new EvalResults();
results.Items.Add(result);
Send(new Envelope { Results = results });
}
private void Send(Envelope envelope) => _outbound.Writer.TryWrite(Wrap(envelope));
private Envelope Wrap(Envelope envelope)
{
envelope.ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion;
envelope.Seq = (ulong)Interlocked.Increment(ref _seq);
return envelope;
}
private static DiagnosticsReply Diagnostics()
{
var reply = new DiagnosticsReply
{
Pid = Environment.ProcessId,
WorkingDirectory = Environment.CurrentDirectory,
};
reply.LoadedAssemblies.AddRange(AppDomain.CurrentDomain.GetAssemblies()
.Select(a => a.GetName().Name ?? string.Empty)
.Where(n => n.Length > 0));
foreach (DictionaryEntry entry in Environment.GetEnvironmentVariables())
reply.EnvironmentNames.Add((string)entry.Key);
return reply;
}
private static string? ForbiddenAssemblyLoaded()
{
foreach (var assembly in AppDomain.CurrentDomain.GetAssemblies())
{
var name = assembly.GetName().Name ?? string.Empty;
foreach (var prefix in ScriptWorkerEnvironment.ForbiddenAssemblyPrefixes)
if (name.StartsWith(prefix, StringComparison.Ordinal)) return name;
}
return null;
}
private static void DieForbidden(string assembly)
{
Console.Error.WriteLine($"ScriptWorker: forbidden assembly '{assembly}' is loaded; exiting.");
Environment.Exit(ScriptWorkerEnvironment.ExitForbiddenAssembly);
}
private sealed record RunningEval(string Key, long StartedTimestamp, TimeSpan FailFastAfter, CancellationTokenSource Cancellation);
/// <summary>One loaded script: its compiled evaluator and its input slot order.</summary>
private sealed class LoadedScript : IDisposable
{
private readonly ScriptEvaluator<VirtualTagContext, object?>? _value;
private readonly ScriptEvaluator<AlarmPredicateContext, bool>? _predicate;
private LoadedScript(string[] inputPaths, ScriptEvaluator<VirtualTagContext, object?>? value,
ScriptEvaluator<AlarmPredicateContext, bool>? predicate)
{
InputPaths = inputPaths;
_value = value;
_predicate = predicate;
}
public string[] InputPaths { get; }
public static LoadedScript Compile(ScriptSource source) => source.Kind switch
{
ContextKind.VirtualTag or ContextKind.Calculation => new LoadedScript(
[.. source.InputPaths], ScriptEvaluator<VirtualTagContext, object?>.Compile(source.Source), null),
ContextKind.AlarmPredicate => new LoadedScript(
[.. source.InputPaths], null, ScriptEvaluator<AlarmPredicateContext, bool>.Compile(source.Source)),
_ => throw new ScriptWorkerProtocolException($"unknown script kind {source.Kind}"),
};
public object? Run(IReadOnlyDictionary<string, DataValueSnapshot> inputs, ILogger logger, Func<DateTime> clock,
CancellationToken ct, out bool isPredicate)
{
if (_predicate is not null)
{
isPredicate = true;
return _predicate.RunAsync(new AlarmPredicateContext(inputs, logger, clock), ct).GetAwaiter().GetResult();
}
isPredicate = false;
// Single-tag mode, as in the host: a cross-tag write is dropped.
return _value!.RunAsync(new VirtualTagContext(inputs, static (_, _) => { }, logger, clock), ct)
.GetAwaiter().GetResult();
}
public void Dispose()
{
_value?.Dispose();
_predicate?.Dispose();
}
}
/// <summary>Buffers <c>ctx.Logger</c> events for the host, bounded, counting what it had to drop.</summary>
private sealed class ForwardingLogSink : ILogEventSink
{
public const string KeyProperty = "ScriptWorkerKey";
private readonly ConcurrentQueue<LogLine> _lines = new();
private int _count;
private int _dropped;
public int Capacity { get; set; } = 1000;
public void Emit(LogEvent logEvent)
{
if (Interlocked.Increment(ref _count) > Capacity)
{
Interlocked.Decrement(ref _count);
Interlocked.Increment(ref _dropped);
return;
}
var key = logEvent.Properties.TryGetValue(KeyProperty, out var k) && k is ScalarValue { Value: string s } ? s : string.Empty;
_lines.Enqueue(new LogLine
{
Key = key,
Level = (uint)logEvent.Level,
Rendered = Truncate(logEvent.RenderMessage(), 8 * 1024),
Exception = Truncate(logEvent.Exception?.ToString() ?? string.Empty, 16 * 1024),
TsTicks = logEvent.Timestamp.UtcTicks,
});
}
public LogBatch? Drain()
{
var dropped = Interlocked.Exchange(ref _dropped, 0);
if (_lines.IsEmpty && dropped == 0) return null;
var batch = new LogBatch { Dropped = (uint)dropped };
while (_lines.TryDequeue(out var line))
{
Interlocked.Decrement(ref _count);
batch.Lines.Add(line);
}
return batch;
}
private static string Truncate(string value, int max) => value.Length <= max ? value : value[..max];
}
}
@@ -0,0 +1,32 @@
<Project Sdk="Microsoft.NET.Sdk">
<!-- lmxopcua#780 — the out-of-process, pre-emptible script runner's worker (design §3, §6). A plain console app the
host launches per shard over a named pipe. It references only the compile/run core and the frame protocol, so
the host's cluster, database, OPC UA and secrets stacks are not in its assembly set; it checks itself against
ScriptWorkerEnvironment.ForbiddenAssemblyPrefixes at runtime as well. -->
<PropertyGroup>
<OutputType>Exe</OutputType>
<RootNamespace>ZB.MOM.WW.OtOpcUa.ScriptWorker</RootNamespace>
<AssemblyName>ZB.MOM.WW.OtOpcUa.ScriptWorker</AssemblyName>
<InvariantGlobalization>true</InvariantGlobalization>
<!-- Workstation, non-concurrent GC: the heap hard limit (DOTNET_GCHeapHardLimit) is the worker's memory bound. -->
<ServerGarbageCollection>false</ServerGarbageCollection>
<ConcurrentGarbageCollection>false</ConcurrentGarbageCollection>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.Scripting\ZB.MOM.WW.OtOpcUa.Core.Scripting.csproj"/>
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol\ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.csproj"/>
</ItemGroup>
<!-- Core.Scripting reaches Akka and Grpc.Core.Api transitively through Commons (telemetry meter, log messages),
none of which the worker executes. Excluding their assets keeps them out of the worker's deps.json and output
folder, so they are not loadable at all, not merely unloaded (design §6 "assembly allow-list by
construction"). A code path that did touch them would fail with FileNotFoundException, which the
containment tests would catch. -->
<ItemGroup>
<PackageReference Include="Akka" ExcludeAssets="all" PrivateAssets="all"/>
<PackageReference Include="Grpc.Core.Api" ExcludeAssets="all" PrivateAssets="all"/>
</ItemGroup>
</Project>
@@ -0,0 +1,43 @@
<Project>
<!--
lmxopcua#780 — places the ScriptWorker next to a host that launches it: the worker's build output is copied into
$(OutDir)script-worker/ after Build, and into $(PublishDir)script-worker/ after Publish. Imported by the Host and by
the worker test project. The worker is its own app (own deps.json / runtimeconfig.json), so it lives in its own folder
rather than beside the host's assemblies.
Publish note: the copy is the worker's framework-dependent BUILD output, run through the dotnet muxer. That suits the
docker-dev image (aspnet:10.0 runtime). A self-contained host publish (the Windows deploys) needs a self-contained
worker publish too; that is part of the installer work tracked on lmxopcua#780.
-->
<PropertyGroup>
<_OtScriptWorkerProject>$(MSBuildThisFileDirectory)../ZB.MOM.WW.OtOpcUa.ScriptWorker.csproj</_OtScriptWorkerProject>
</PropertyGroup>
<ItemGroup>
<ProjectReference Include="$(_OtScriptWorkerProject)" ReferenceOutputAssembly="false" Private="false" />
</ItemGroup>
<Target Name="_OtResolveScriptWorkerOutput">
<MSBuild Projects="$(_OtScriptWorkerProject)" Targets="GetTargetPath" Properties="Configuration=$(Configuration)">
<Output TaskParameter="TargetOutputs" ItemName="_OtScriptWorkerTargetPath" />
</MSBuild>
<PropertyGroup>
<_OtScriptWorkerOutDir>@(_OtScriptWorkerTargetPath->'%(RootDir)%(Directory)')</_OtScriptWorkerOutDir>
</PropertyGroup>
<ItemGroup>
<_OtScriptWorkerFiles Include="$(_OtScriptWorkerOutDir)**/*" />
</ItemGroup>
</Target>
<Target Name="_OtCopyScriptWorkerAfterBuild" AfterTargets="Build" DependsOnTargets="_OtResolveScriptWorkerOutput">
<Copy SourceFiles="@(_OtScriptWorkerFiles)"
DestinationFiles="@(_OtScriptWorkerFiles->'$(OutDir)script-worker/%(RecursiveDir)%(Filename)%(Extension)')"
SkipUnchangedFiles="true" />
</Target>
<Target Name="_OtCopyScriptWorkerAfterPublish" AfterTargets="Publish" DependsOnTargets="_OtResolveScriptWorkerOutput">
<Copy SourceFiles="@(_OtScriptWorkerFiles)"
DestinationFiles="@(_OtScriptWorkerFiles->'$(PublishDir)script-worker/%(RecursiveDir)%(Filename)%(Extension)')"
SkipUnchangedFiles="true" />
</Target>
</Project>
@@ -0,0 +1,122 @@
using System.Collections.Concurrent;
using Serilog;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
namespace ZB.MOM.WW.OtOpcUa.Driver.Calculation.Tests;
/// <summary>
/// lmxopcua#780 — with <c>Scripting:Runner=OutOfProcess</c> selecting the Calculation seam, the driver hands each
/// compiled run to the node's <see cref="IScriptRunner"/>: keyed by the calc tag's RawPath, with the driver's own
/// budget. A runner-side failure publishes <c>BadResourceUnavailable</c> (0x80040000); a script failure keeps
/// <c>BadInternalError</c>. The passthrough fast path never reaches the runner.
/// </summary>
public sealed class CalculationDriverScriptRunnerTests
{
private const uint StatusGood = 0u;
private const uint StatusBadInternalError = 0x80020000u;
private const uint StatusBadResourceUnavailable = 0x80040000u;
private const string Tag = "calc/e/x";
private sealed class FakeRunner : IScriptRunner
{
public ConcurrentQueue<ScriptRunOutcome> Outcomes { get; } = new();
public ConcurrentQueue<ScriptInvocation> Invocations { get; } = new();
public List<ScriptContextKind> Resets { get; } = [];
public Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default)
{
Invocations.Enqueue(invocation);
return Task.FromResult(Outcomes.TryDequeue(out var next) ? next : ScriptRunOutcome.Ok(null));
}
public void Reset(ScriptContextKind kind) => Resets.Add(kind);
public IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind) => ["q-" + kind];
}
private static RawTagEntry CalcTag(string source)
{
var json = $"{{\"scriptId\":\"s\",\"changeTriggered\":true,\"scriptSource\":{System.Text.Json.JsonSerializer.Serialize(source)}}}";
return new RawTagEntry(Tag, json, WriteIdempotent: false);
}
private static async Task Settle(CalculationDriver driver)
{
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
while (DateTime.UtcNow < deadline && !driver.IsEvaluationIdle) await Task.Delay(10);
driver.IsEvaluationIdle.ShouldBeTrue();
}
private static (CalculationDriver Driver, ConcurrentQueue<DataChangeEventArgs> Published) Build(FakeRunner runner, string source)
{
var driver = new CalculationDriver(
new CalculationDriverOptions { RawTags = [CalcTag(source)], RunTimeout = TimeSpan.FromMilliseconds(1234), ScriptRunner = runner },
"calc-drv", new ScriptRootLogger(new LoggerConfiguration().CreateLogger()));
var published = new ConcurrentQueue<DataChangeEventArgs>();
driver.OnDataChange += (_, e) => published.Enqueue(e);
return (driver, published);
}
[Fact]
public async Task A_runner_failure_publishes_BadResourceUnavailable_and_a_script_failure_BadInternalError()
{
var runner = new FakeRunner();
runner.Outcomes.Enqueue(ScriptRunOutcome.Ok(30.0));
runner.Outcomes.Enqueue(ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason));
runner.Outcomes.Enqueue(ScriptRunOutcome.Failed("script threw: boom"));
var (driver, published) = Build(runner, "return (double)ctx.GetTag(\"src/a\").Value * 3;");
await driver.InitializeAsync("{}", TestContext.Current.CancellationToken);
driver.OnDependencyValue("src/a", 10.0, StatusGood, DateTime.UtcNow);
await Settle(driver);
published.TryDequeue(out var good).ShouldBeTrue();
good!.Snapshot.StatusCode.ShouldBe(StatusGood);
good.Snapshot.Value.ShouldBe(30.0);
var invocation = runner.Invocations.Single();
invocation.Key.ShouldBe(Tag);
invocation.Kind.ShouldBe(ScriptContextKind.Calculation);
invocation.Timeout.ShouldBe(TimeSpan.FromMilliseconds(1234));
invocation.Inputs["src/a"].Value.ShouldBe(10.0);
driver.OnDependencyValue("src/a", 11.0, StatusGood, DateTime.UtcNow);
await Settle(driver);
published.TryDequeue(out var unavailable).ShouldBeTrue();
unavailable!.Snapshot.StatusCode.ShouldBe(StatusBadResourceUnavailable);
unavailable.Snapshot.Value.ShouldBe(30.0, "the last known value rides out with the Bad status");
driver.OnDependencyValue("src/a", 12.0, StatusGood, DateTime.UtcNow);
await Settle(driver);
published.TryDequeue(out var scriptFault).ShouldBeTrue();
scriptFault!.Snapshot.StatusCode.ShouldBe(StatusBadInternalError);
}
[Fact]
public async Task The_passthrough_fast_path_never_reaches_the_runner()
{
var runner = new FakeRunner();
var (driver, published) = Build(runner, "return ctx.GetTag(\"src/a\").Value;");
await driver.InitializeAsync("{}", TestContext.Current.CancellationToken);
driver.OnDependencyValue("src/a", 4.0, StatusGood, DateTime.UtcNow);
await Settle(driver);
published.TryDequeue(out var mirrored).ShouldBeTrue();
mirrored!.Snapshot.Value.ShouldBe(4.0);
runner.Invocations.ShouldBeEmpty();
}
[Fact]
public void The_deploy_boundary_resets_the_runners_calculation_seam_and_quarantines_are_reported()
{
var runner = new FakeRunner();
using var evaluator = new CalculationEvaluator(Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance,
new ScriptRootLogger(new LoggerConfiguration().CreateLogger()), scriptRunner: runner);
evaluator.ClearCompiledScripts();
evaluator.ResetQuarantines();
runner.Resets.ShouldBe([ScriptContextKind.Calculation, ScriptContextKind.Calculation]);
evaluator.QuarantinedScripts.ShouldBe(["q-Calculation"]);
}
}
@@ -0,0 +1,170 @@
using Microsoft.Extensions.Configuration;
using Serilog;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Configuration;
/// <summary>
/// lmxopcua#780 — <c>Scripting:Runner</c> binds by its documented keys (design §12), defaults to InProcess (every seam
/// gets no runner, so the in-process path is untouched), and the worker options clamp and fail closed.
/// </summary>
public sealed class ScriptRunnerSelectionTests
{
private static IConfiguration Config(Dictionary<string, string?> values) =>
new ConfigurationBuilder().AddInMemoryCollection(values).Build();
private static readonly ILogger Silent = new LoggerConfiguration().CreateLogger();
[Fact]
public void The_default_is_InProcess_and_selects_no_runner_for_any_seam()
{
var bound = ScriptingOptionsBinding.Bind(Config(new()), out var adjustments);
adjustments.ShouldBeEmpty();
bound.Runner.ShouldBe(ScriptRunnerMode.InProcess);
ScriptRunnerSelection.Validate(bound, "/srv/app").ShouldBeEmpty();
using var selection = ScriptRunnerSelection.Create(bound, Silent, null);
foreach (var seam in Enum.GetValues<ScriptSeam>()) selection.ForSeam(seam).ShouldBeNull(seam.ToString());
selection.OutOfProcessRunner.ShouldBeNull();
}
[Fact]
public void Every_documented_runner_key_binds()
{
var bound = ScriptingOptionsBinding.Bind(Config(new()
{
["Scripting:Runner"] = "OutOfProcess",
["Scripting:Runner:Seams:0"] = "Calculation",
["Scripting:Worker:Shards"] = "3",
["Scripting:Worker:HardGrace"] = "00:00:02",
["Scripting:Worker:HeapLimitBytes"] = "268435456",
["Scripting:Worker:QuarantineAfterKills"] = "4",
["Scripting:Worker:UnavailableGrace"] = "00:00:07",
["Scripting:Worker:IsolationProbationRuns"] = "9",
["Scripting:Worker:PeCacheDir"] = "/var/cache/otopcua-pe",
["Scripting:Worker:ExecutablePath"] = "/opt/w/ZB.MOM.WW.OtOpcUa.ScriptWorker.dll",
}), out var adjustments);
adjustments.ShouldBeEmpty();
bound.Runner.ShouldBe(ScriptRunnerMode.OutOfProcess);
bound.RunnerSeams.ShouldBe([ScriptSeam.Calculation]);
bound.Worker.Shards.ShouldBe(3);
bound.Worker.HardGrace.ShouldBe(TimeSpan.FromSeconds(2));
bound.Worker.HeapLimitBytes.ShouldBe(256L * 1024 * 1024);
bound.Worker.QuarantineAfterKills.ShouldBe(4);
bound.Worker.UnavailableGrace.ShouldBe(TimeSpan.FromSeconds(7));
bound.Worker.IsolationProbationRuns.ShouldBe(9);
bound.Worker.PeCacheDir.ShouldBe("/var/cache/otopcua-pe");
bound.Worker.ExecutablePath.ShouldBe("/opt/w/ZB.MOM.WW.OtOpcUa.ScriptWorker.dll");
bound.IsOutOfProcess(ScriptSeam.Calculation).ShouldBeTrue();
bound.IsOutOfProcess(ScriptSeam.VirtualTag).ShouldBeFalse();
}
[Fact]
public void A_misspelled_runner_mode_fails_startup()
=> Should.Throw<InvalidOperationException>(() =>
ScriptingOptionsBinding.Bind(Config(new() { ["Scripting:Runner"] = "OutOfProc" }), out _));
[Fact]
public void Worker_values_out_of_range_are_clamped_and_reported()
{
var bound = ScriptingOptionsBinding.Bind(Config(new()
{
["Scripting:Worker:Shards"] = "64",
["Scripting:Worker:HardGrace"] = "00:00:00.010",
["Scripting:Worker:HeapLimitBytes"] = "1024",
["Scripting:Worker:QuarantineAfterKills"] = "0",
["Scripting:Worker:UnavailableGrace"] = "00:05:00",
["Scripting:Worker:IsolationProbationRuns"] = "0",
}), out var adjustments);
bound.Worker.Shards.ShouldBe(ScriptWorkerOptions.MaxShards);
bound.Worker.HardGrace.ShouldBe(ScriptWorkerOptions.MinHardGrace);
bound.Worker.HeapLimitBytes.ShouldBe(ScriptWorkerOptions.MinHeapLimitBytes);
bound.Worker.QuarantineAfterKills.ShouldBe(1);
bound.Worker.UnavailableGrace.ShouldBe(ScriptWorkerOptions.MaxUnavailableGrace);
bound.Worker.IsolationProbationRuns.ShouldBe(1);
adjustments.Count.ShouldBe(6);
adjustments.ShouldAllBe(a => a.StartsWith("Scripting:Worker:"));
}
[Fact]
public void The_hard_grace_defaults_to_max_of_one_second_and_the_budget()
{
var worker = new ScriptWorkerOptions();
worker.EffectiveHardGrace(TimeSpan.FromMilliseconds(250)).ShouldBe(TimeSpan.FromSeconds(1));
worker.EffectiveHardGrace(TimeSpan.FromSeconds(2)).ShouldBe(TimeSpan.FromSeconds(2));
new ScriptWorkerOptions { HardGrace = TimeSpan.FromMilliseconds(400) }
.EffectiveHardGrace(TimeSpan.FromSeconds(2)).ShouldBe(TimeSpan.FromMilliseconds(400));
}
[Theory]
[InlineData("relative/cache", "must be an absolute path")]
[InlineData("{root}/pe-cache", "inside the content root")]
[InlineData("{root}", "inside the content root")]
public void A_pe_cache_dir_that_is_relative_or_inside_the_content_root_fails_startup(string dir, string why)
{
var root = Path.Combine(Path.GetTempPath(), "otopcua-root-" + Guid.NewGuid().ToString("N"));
var options = new ScriptingOptions
{
Runner = ScriptRunnerMode.OutOfProcess,
Worker = new ScriptWorkerOptions { PeCacheDir = dir.Replace("{root}", root) },
};
Should.Throw<InvalidOperationException>(() => ScriptRunnerSelection.Validate(options, root)).Message.ShouldContain(why);
}
[Fact]
public void A_pe_cache_dir_outside_the_content_root_is_accepted()
{
var root = Path.Combine(Path.GetTempPath(), "otopcua-root");
var options = new ScriptingOptions
{
Runner = ScriptRunnerMode.OutOfProcess,
Worker = new ScriptWorkerOptions { PeCacheDir = Path.Combine(Path.GetTempPath(), "otopcua-root-sibling") },
};
Should.NotThrow(() => ScriptRunnerSelection.Validate(options, root));
}
[Fact]
public void Shadow_is_not_implemented_yet_and_runs_in_process_with_a_warning()
{
var options = new ScriptingOptions { Runner = ScriptRunnerMode.Shadow };
ScriptRunnerSelection.Validate(options, "/srv").ShouldHaveSingleItem().ShouldContain("Shadow");
using var selection = ScriptRunnerSelection.Create(options, Silent, null);
selection.OutOfProcessRunner.ShouldBeNull();
}
[Fact]
public void OutOfProcess_selects_only_the_supported_requested_seams()
{
var all = new ScriptingOptions { Runner = ScriptRunnerMode.OutOfProcess };
ScriptRunnerSelection.Validate(all, "/srv").ShouldContain(w => w.Contains("ScriptedAlarm seam cannot run out of process yet"));
using (var selection = ScriptRunnerSelection.Create(all, Silent, null))
{
selection.ForSeam(ScriptSeam.VirtualTag).ShouldNotBeNull();
selection.ForSeam(ScriptSeam.Calculation).ShouldBeSameAs(selection.ForSeam(ScriptSeam.VirtualTag), "one node-wide runner");
selection.ForSeam(ScriptSeam.ScriptedAlarm).ShouldBeNull();
}
var calcOnly = new ScriptingOptions { Runner = ScriptRunnerMode.OutOfProcess, RunnerSeams = [ScriptSeam.Calculation] };
using (var selection = ScriptRunnerSelection.Create(calcOnly, Silent, null))
{
selection.ForSeam(ScriptSeam.Calculation).ShouldNotBeNull();
selection.ForSeam(ScriptSeam.VirtualTag).ShouldBeNull();
}
}
[Fact]
public void A_missing_worker_is_a_startup_warning_not_a_silent_in_process_fallback()
{
var options = new ScriptingOptions
{
Runner = ScriptRunnerMode.OutOfProcess,
RunnerSeams = [ScriptSeam.VirtualTag],
Worker = new ScriptWorkerOptions { ExecutablePath = Path.Combine(Path.GetTempPath(), "missing-" + Guid.NewGuid().ToString("N"), "w.dll") },
};
ScriptRunnerSelection.Validate(options, "/srv").ShouldHaveSingleItem().ShouldContain("BadResourceUnavailable");
}
}
@@ -0,0 +1,109 @@
using System.Collections.Concurrent;
using Microsoft.Extensions.Logging.Abstractions;
using Serilog;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
using ZB.MOM.WW.OtOpcUa.Host.Engines;
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Engines;
/// <summary>
/// lmxopcua#780 — the virtual-tag seam over <see cref="IScriptRunner"/>: with a runner (OutOfProcess) the compiled run
/// is the runner's, keyed by virtual-tag id with the evaluator's budget, and its Unavailable outcome becomes a failure
/// carrying <see cref="OpcUaStatus.BadResourceUnavailable"/>. Without one (the InProcess default) nothing changes.
/// </summary>
public sealed class RoslynVirtualTagEvaluatorScriptRunnerTests
{
private sealed class FakeRunner : IScriptRunner
{
public ConcurrentQueue<ScriptRunOutcome> Outcomes { get; } = new();
public ConcurrentQueue<ScriptInvocation> Invocations { get; } = new();
public List<ScriptContextKind> Resets { get; } = [];
public Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default)
{
Invocations.Enqueue(invocation);
return Task.FromResult(Outcomes.TryDequeue(out var next) ? next : ScriptRunOutcome.Ok(null));
}
public void Reset(ScriptContextKind kind) => Resets.Add(kind);
public IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind) => ["vt-q"];
}
private static RoslynVirtualTagEvaluator Build(IScriptRunner? runner) =>
new(NullLogger<RoslynVirtualTagEvaluator>.Instance, new ScriptRootLogger(new LoggerConfiguration().CreateLogger()),
runTimeout: TimeSpan.FromMilliseconds(1500), scriptRunner: runner);
private static Dictionary<string, DataValueSnapshot> Deps(int a) => new(StringComparer.Ordinal)
{
["a"] = new DataValueSnapshot(a, 0u, DateTime.UtcNow, DateTime.UtcNow),
};
[Fact]
public async Task Compiled_runs_go_to_the_runner_and_its_outcomes_map_onto_the_seam_contract()
{
var runner = new FakeRunner();
runner.Outcomes.Enqueue(ScriptRunOutcome.Ok(42));
runner.Outcomes.Enqueue(ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason));
runner.Outcomes.Enqueue(ScriptRunOutcome.Failed("script threw: boom"));
using var sut = Build(runner);
const string script = "return (int)ctx.GetTag(\"a\").Value + 1;";
var ok = await sut.EvaluateAsync("vt-1", script, Deps(41));
ok.Success.ShouldBeTrue();
ok.Value.ShouldBe(42);
ok.StatusCode.ShouldBeNull();
var unavailable = await sut.EvaluateAsync("vt-1", script, Deps(41));
unavailable.Success.ShouldBeFalse();
unavailable.Reason.ShouldBe(ScriptRunOutcome.WorkerRestartedReason);
unavailable.StatusCode.ShouldBe(OpcUaStatus.BadResourceUnavailable);
var failed = await sut.EvaluateAsync("vt-1", script, Deps(41));
failed.Reason.ShouldBe("script threw: boom");
failed.StatusCode.ShouldBeNull("a script failure keeps BadInternalError");
var invocation = runner.Invocations.First();
invocation.Key.ShouldBe("vt-1");
invocation.Kind.ShouldBe(ScriptContextKind.VirtualTag);
invocation.Source.ShouldBe(script);
invocation.Timeout.ShouldBe(TimeSpan.FromMilliseconds(1500));
invocation.Inputs["a"].Value.ShouldBe(41);
invocation.ScriptLogger.ShouldNotBeNull();
}
[Fact]
public async Task The_passthrough_fast_path_and_an_empty_expression_never_reach_the_runner()
{
var runner = new FakeRunner();
using var sut = Build(runner);
(await sut.EvaluateAsync("vt-p", "return ctx.GetTag(\"a\").Value;", Deps(7))).Value.ShouldBe(7);
(await sut.EvaluateAsync("vt-e", " ", Deps(7))).Success.ShouldBeFalse();
runner.Invocations.ShouldBeEmpty();
}
[Fact]
public void The_deploy_boundary_resets_the_runners_virtual_tag_seam_and_its_quarantines_are_reported()
{
var runner = new FakeRunner();
using var sut = Build(runner);
sut.ClearCompiledScripts();
sut.ResetQuarantines();
runner.Resets.ShouldBe([ScriptContextKind.VirtualTag, ScriptContextKind.VirtualTag]);
sut.QuarantinedScripts.ShouldBe(["vt-q"]);
}
/// <summary>Positive control for the default: no runner means the in-process Roslyn path actually runs the script.</summary>
[Fact]
public async Task Without_a_runner_the_script_runs_in_process()
{
using var sut = Build(runner: null);
sut.ScriptRunner.ShouldBeNull();
(await sut.EvaluateAsync("vt-1", "return (int)ctx.GetTag(\"a\").Value + 1;", Deps(41))).Value.ShouldBe(42);
sut.RunGuard.StartedRunCount.ShouldBe(1);
}
}
@@ -0,0 +1,64 @@
using System.Collections.Concurrent;
using Akka.Actor;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.Engines;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
using ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags;
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.VirtualTags;
/// <summary>
/// lmxopcua#780 — when the out-of-process runner, not the script, fails (its worker restarted, no worker ready), the
/// virtual tag publishes <see cref="OpcUaStatus.BadResourceUnavailable"/> with its last value, so a client can tell a
/// sick node from a broken script. Every other failure keeps the long-standing <see cref="OpcUaStatus.BadInternalError"/>.
/// </summary>
public sealed class VirtualTagActorRunnerUnavailableTests : RuntimeActorTestBase
{
private sealed class QueuedEvaluator : IVirtualTagEvaluator
{
public ConcurrentQueue<VirtualTagEvalResult> Results { get; } = new();
public VirtualTagEvalResult Evaluate(string virtualTagId, string expression, IReadOnlyDictionary<string, object?> dependencies)
=> Results.TryDequeue(out var next) ? next : VirtualTagEvalResult.NoChange;
}
[Fact]
public void A_runner_unavailable_failure_publishes_BadResourceUnavailable_and_a_script_failure_BadInternalError()
{
var parent = CreateTestProbe();
var evaluator = new QueuedEvaluator();
evaluator.Results.Enqueue(VirtualTagEvalResult.Ok(5));
evaluator.Results.Enqueue(VirtualTagEvalResult.Unavailable("script worker restarted"));
evaluator.Results.Enqueue(VirtualTagEvalResult.Unavailable("script worker restarted"));
evaluator.Results.Enqueue(VirtualTagEvalResult.Failure("script threw: boom"));
var actor = parent.ChildActorOf(VirtualTagActor.Props("vt-1", "x", evaluator: evaluator, dependencyRefs: ["a"]));
actor.Tell(new VirtualTagActor.DependencyValueChanged("a", 1, DateTime.UtcNow));
parent.ExpectMsg<VirtualTagActor.EvaluationResult>().StatusCode.ShouldBe(OpcUaStatus.Good);
actor.Tell(new VirtualTagActor.DependencyValueChanged("a", 2, DateTime.UtcNow));
var unavailable = parent.ExpectMsg<VirtualTagActor.EvaluationResult>(PresenceBudget);
unavailable.StatusCode.ShouldBe(OpcUaStatus.BadResourceUnavailable);
unavailable.Value.ShouldBe(5, "the last known value rides out with the Bad status");
// Once per transition: the same code again publishes nothing.
actor.Tell(new VirtualTagActor.DependencyValueChanged("a", 3, DateTime.UtcNow));
parent.ExpectNoMsg(TimeSpan.FromMilliseconds(300));
// A script failure is a different Bad and is published.
actor.Tell(new VirtualTagActor.DependencyValueChanged("a", 4, DateTime.UtcNow));
var scriptFault = parent.ExpectMsg<VirtualTagActor.EvaluationResult>(PresenceBudget);
scriptFault.StatusCode.ShouldBe(OpcUaStatus.BadInternalError);
scriptFault.Value.ShouldBe(5);
}
[Fact]
public void A_failure_without_a_status_keeps_BadInternalError()
{
VirtualTagEvalResult.Failure("x").StatusCode.ShouldBeNull();
VirtualTagEvalResult.Unavailable("x").StatusCode.ShouldBe(OpcUaStatus.BadResourceUnavailable);
OpcUaStatus.BadResourceUnavailable.ShouldBe(0x80040000u, "the OPC UA Part 6 code ScadaBridge maps");
}
}
@@ -0,0 +1,382 @@
using System.Diagnostics;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests;
/// <summary>Worker-backed tests spawn real processes and measure wall-clock bounds, so they never run in parallel.</summary>
[CollectionDefinition(Name, DisableParallelization = true)]
public sealed class ScriptWorkerCollection
{
public const string Name = "ScriptWorker";
}
/// <summary>
/// lmxopcua#780 design §13 "Containment": each case runs against the real <c>ScriptWorker</c> process, and each has a
/// positive control — a healthy sibling on the SAME shard (every runner here has one shard) that keeps producing
/// correct values before and after the event.
/// </summary>
[Collection(ScriptWorkerCollection.Name)]
[Trait("Category", "ScriptWorker")]
public sealed class ContainmentTests
{
private const string Healthy = "return 7;";
private const string Runaway = "while (true) { }";
private const string Cooperative = "while (!ctx.CancellationToken.IsCancellationRequested) { } return 1;";
/// <summary>Spins for the number of milliseconds in input <c>ms</c>, so one source serves slow and fast runs.</summary>
private const string Spin = "var ms = (int)ctx.GetTag(\"ms\").Value; var end = ctx.Now.AddMilliseconds(ms); while (ctx.Now < end) { } return ms;";
private static IReadOnlyDictionary<string, DataValueSnapshot> Ms(int ms) => Scripts.Input("ms", ms);
private static CancellationToken Ct => TestContext.Current.CancellationToken;
[Fact]
public async Task A_healthy_script_runs_out_of_process_and_reads_its_inputs()
{
await using var runner = Scripts.Runner(Scripts.Options());
var outcome = await runner.RunAsync(Scripts.Vt("vt", "return (int)ctx.GetTag(\"x\").Value + 1;",
inputs: Scripts.Input("x", 41)), Ct);
outcome.Status.ShouldBe(ScriptRunStatus.Ok, outcome.Reason);
outcome.Value.ShouldBe(42);
runner.Shards[0].Current.ShouldNotBeNull().Pid.ShouldNotBe(Environment.ProcessId);
}
/// <summary>
/// The headline case: <c>while(true){}</c> ignores the token. In process it would spin a host core forever; here
/// the caller gets the usual timeout at T, the worker is killed at T + G, the host burns no CPU on it, and the
/// sibling on the same shard keeps working. Mutation check: with the hard-deadline kill disabled the worker is
/// never killed and the Exited wait times out.
/// </summary>
[Fact]
public async Task A_runaway_script_is_killed_at_its_hard_deadline_and_its_sibling_keeps_working()
{
var softT = TimeSpan.FromMilliseconds(400);
var grace = TimeSpan.FromMilliseconds(500);
await using var runner = Scripts.Runner(Scripts.Options(hardGrace: grace, quarantineAfterKills: 2));
(await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7);
var worker = runner.Shards[0].Current.ShouldNotBeNull();
var sw = Stopwatch.StartNew();
var outcome = await runner.RunAsync(Scripts.Vt("runaway", Runaway, softT), Ct);
var softElapsed = sw.Elapsed;
outcome.Status.ShouldBe(ScriptRunStatus.Failed);
outcome.Reason!.ShouldStartWith("script timed out after ");
softElapsed.ShouldBeLessThan(softT + TimeSpan.FromMilliseconds(400), "the caller is answered at the soft deadline");
var exit = await worker.Exited.WaitAsync(TimeSpan.FromSeconds(10), Ct);
var killElapsed = sw.Elapsed;
exit.Reason.ShouldBe(WorkerExitReason.KilledDeadline);
exit.CulpritKey.ShouldBe(InProcessScriptRunner.ScopedKey(ScriptContextKind.VirtualTag, "runaway"));
killElapsed.ShouldBeGreaterThanOrEqualTo(softT + grace - TimeSpan.FromMilliseconds(50));
// Design budget is T + G + 100 ms; the slack here is for a loaded CI box (process reap + exit notification).
killElapsed.ShouldBeLessThan(softT + grace + TimeSpan.FromMilliseconds(1500));
// No orphan in the host: an in-process while(true) would burn one host core for the whole window.
var cpuBefore = Process.GetCurrentProcess().TotalProcessorTime;
await Task.Delay(TimeSpan.FromMilliseconds(1500), Ct);
var cpuUsed = Process.GetCurrentProcess().TotalProcessorTime - cpuBefore;
cpuUsed.ShouldBeLessThan(TimeSpan.FromMilliseconds(1000), "the runaway's thread died with its worker");
// The sibling recovers on a fresh worker.
(await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7);
runner.Shards[0].Current.ShouldNotBeNull().Pid.ShouldNotBe(worker.Pid);
// A second kill quarantines (QuarantineAfterKills = 2); the third trigger is refused without running.
var second = runner.Shards[0].Current!;
(await runner.RunAsync(Scripts.Vt("runaway", Runaway, softT), Ct)).Status.ShouldBe(ScriptRunStatus.Failed);
(await second.Exited.WaitAsync(TimeSpan.FromSeconds(10), Ct)).Reason.ShouldBe(WorkerExitReason.KilledDeadline);
// The strike is recorded by the shard's exit handler, which runs asynchronously to the Exited completion.
await Scripts.Eventually(() => runner.QuarantinedKeys(ScriptContextKind.VirtualTag).Count == 1,
TimeSpan.FromSeconds(5), "the second kill strike");
var refused = await runner.RunAsync(Scripts.Vt("runaway", Runaway, softT), Ct);
refused.Reason!.ShouldStartWith(ScriptRunGuard.QuarantinedReasonPrefix);
refused.Reason.ShouldContain("hard deadline");
runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBe(["runaway"]);
(await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7);
// Lifted by a source change (the operator fixed it)…
(await runner.RunAsync(Scripts.Vt("runaway", "return 1;"), Ct)).Value.ShouldBe(1);
runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBeEmpty();
}
[Fact]
public async Task A_redeploy_lifts_a_kill_quarantine_for_its_seam_only()
{
var softT = TimeSpan.FromMilliseconds(300);
await using var runner = Scripts.Runner(Scripts.Options(hardGrace: TimeSpan.FromMilliseconds(300), quarantineAfterKills: 1));
(await runner.RunAsync(Scripts.Vt("runaway", Runaway, softT), Ct)).Status.ShouldBe(ScriptRunStatus.Failed);
await Scripts.Eventually(() => runner.QuarantinedKeys(ScriptContextKind.VirtualTag).Count == 1, TimeSpan.FromSeconds(10), "the kill quarantine");
runner.Reset(ScriptContextKind.Calculation);
runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBe(["runaway"], "another seam's redeploy leaves it alone");
runner.Reset(ScriptContextKind.VirtualTag);
runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBeEmpty();
}
/// <summary>A script that polls its token ends itself at T: no kill, no strike beyond the timeout (design §13).</summary>
[Fact]
public async Task A_cooperative_script_ends_at_its_soft_deadline_without_a_kill()
{
var softT = TimeSpan.FromMilliseconds(400);
var grace = TimeSpan.FromMilliseconds(1500);
await using var runner = Scripts.Runner(Scripts.Options(hardGrace: grace, quarantineAfterKills: 1));
(await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7);
var worker = runner.Shards[0].Current.ShouldNotBeNull();
var outcome = await runner.RunAsync(Scripts.Vt("polite", Cooperative, softT), Ct);
outcome.Reason!.ShouldStartWith("script timed out after ");
await Task.Delay(softT + grace + TimeSpan.FromMilliseconds(500), Ct);
worker.IsAlive.ShouldBeTrue("a script that ended itself must not cost the shard a restart");
runner.Shards[0].Current!.Pid.ShouldBe(worker.Pid);
runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBeEmpty("no kill strike (QuarantineAfterKills = 1)");
(await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7);
}
[Fact]
public async Task A_second_trigger_while_the_first_still_runs_is_refused_not_queued()
{
await using var runner = Scripts.Runner(Scripts.Options());
(await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0);
var first = runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(1500)), Ct);
await Task.Delay(200, Ct);
var second = await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct);
second.Reason.ShouldBe(ScriptRunGuard.PreviousRunStillRunningReason);
(await first).Value.ShouldBe(1500);
}
/// <summary>
/// <c>kill -9</c> of the worker: the evaluation in flight is a victim — Unavailable "script worker restarted"
/// (BadResourceUnavailable), not a script fault — and, with no deadline crossed, a suspect that proves itself in
/// the isolation worker and then returns to its shard.
/// </summary>
[Fact]
public async Task An_externally_killed_worker_makes_its_in_flight_script_a_suspect_that_recovers_in_isolation()
{
await using var runner = Scripts.Runner(Scripts.Options(probation: 2));
(await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0);
var worker = runner.Shards[0].Current.ShouldNotBeNull();
var inFlight = runner.RunAsync(Scripts.Vt("spin", Spin, TimeSpan.FromSeconds(20), Ms(5000)), Ct);
await Task.Delay(300, Ct);
Process.GetProcessById(worker.Pid).Kill();
var victim = await inFlight.WaitAsync(TimeSpan.FromSeconds(10), Ct);
victim.Status.ShouldBe(ScriptRunStatus.Unavailable);
victim.Reason.ShouldBe(ScriptRunOutcome.WorkerRestartedReason);
(await worker.Exited).Reason.ShouldBe(WorkerExitReason.Crash);
runner.SuspectKeys.ShouldBe([InProcessScriptRunner.ScopedKey(ScriptContextKind.VirtualTag, "spin")]);
(await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0);
runner.Isolation.Current.ShouldNotBeNull("a suspect runs in the isolation worker");
(await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0);
runner.SuspectKeys.ShouldBeEmpty("two clean runs (probation = 2) return it to its shard");
runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBeEmpty();
// The sibling recovers on a fresh shard worker.
(await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7);
runner.Shards[0].Current!.Pid.ShouldNotBe(worker.Pid);
}
/// <summary>The isolation worker runs one evaluation at a time, so a crash there names exactly one key.</summary>
[Fact]
public async Task A_crash_in_the_isolation_worker_strikes_its_one_key_and_quarantines_it()
{
await using var runner = Scripts.Runner(Scripts.Options(quarantineAfterKills: 2));
(await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0);
async Task<ScriptRunOutcome> RunAndKill(Func<WorkerConnection?> worker)
{
var run = runner.RunAsync(Scripts.Vt("spin", Spin, TimeSpan.FromSeconds(20), Ms(5000)), Ct);
var live = await Scripts.Eventually(worker, TimeSpan.FromSeconds(15), "a live worker");
await Scripts.Eventually(() => live.InFlightCount == 1, TimeSpan.FromSeconds(15), "the evaluation in flight");
await Task.Delay(200, Ct);
Process.GetProcessById(live.Pid).Kill();
return await run.WaitAsync(TimeSpan.FromSeconds(10), Ct);
}
(await RunAndKill(() => runner.Shards[0].Current)).Reason.ShouldBe(ScriptRunOutcome.WorkerRestartedReason);
runner.SuspectKeys.Count.ShouldBe(1);
runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBeEmpty("an ambiguous shard crash is not a strike");
(await RunAndKill(() => runner.Isolation.Current)).Reason.ShouldBe(ScriptRunOutcome.WorkerRestartedReason);
runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBeEmpty("one isolation crash is one strike");
(await RunAndKill(() => runner.Isolation.Current)).Reason.ShouldBe(ScriptRunOutcome.WorkerRestartedReason);
await Scripts.Eventually(() => runner.QuarantinedKeys(ScriptContextKind.VirtualTag).Count == 1,
TimeSpan.FromSeconds(5), "the isolated-crash quarantine");
var refused = await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct);
refused.Reason!.ShouldStartWith(ScriptRunGuard.QuarantinedReasonPrefix);
refused.Reason.ShouldContain("isolation worker");
(await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7);
}
/// <summary>A deadline kill blames exactly one key: the sibling in flight on the same worker is a victim, not a suspect.</summary>
[Fact]
public async Task A_deadline_kill_blames_only_its_culprit()
{
await using var runner = Scripts.Runner(Scripts.Options(hardGrace: TimeSpan.FromMilliseconds(300), quarantineAfterKills: 1));
(await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0);
(await runner.RunAsync(Scripts.Vt("warm", Healthy), Ct)).Value.ShouldBe(7);
// Compile the runaway first, so both evaluations are on the worker at once.
_ = runner.Shards[0].Current!;
var victim = runner.RunAsync(Scripts.Vt("spin", Spin, TimeSpan.FromSeconds(20), Ms(5000)), Ct);
await Task.Delay(200, Ct);
var culprit = await runner.RunAsync(Scripts.Vt("runaway", Runaway, TimeSpan.FromMilliseconds(300)), Ct);
culprit.Reason!.ShouldStartWith("script timed out after ");
var victimOutcome = await victim.WaitAsync(TimeSpan.FromSeconds(15), Ct);
victimOutcome.Status.ShouldBe(ScriptRunStatus.Unavailable);
victimOutcome.Reason.ShouldBe(ScriptRunOutcome.WorkerRestartedReason);
await Scripts.Eventually(() => runner.QuarantinedKeys(ScriptContextKind.VirtualTag).Count == 1, TimeSpan.FromSeconds(5), "the culprit's quarantine");
runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBe(["runaway"]);
runner.SuspectKeys.ShouldBeEmpty("a deadline kill is unambiguous: nobody else is suspected");
(await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0);
}
/// <summary>An allocation bomb dies in its worker (heap hard limit); the host's own memory does not move.</summary>
[Fact]
public async Task An_allocation_bomb_is_contained_in_its_worker()
{
await using var runner = Scripts.Runner(Scripts.Options(heapLimit: 128L * 1024 * 1024));
(await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7);
GC.Collect();
var hostBefore = GC.GetTotalMemory(forceFullCollection: true);
var outcome = await runner.RunAsync(Scripts.Vt("bomb",
"var hoard = new System.Collections.Generic.List<byte[]>(); while (true) { hoard.Add(new byte[1 << 20]); }",
TimeSpan.FromSeconds(10)), Ct);
var hostAfter = GC.GetTotalMemory(forceFullCollection: true);
outcome.Status.ShouldNotBe(ScriptRunStatus.Ok);
(hostAfter - hostBefore).ShouldBeLessThan(32L * 1024 * 1024, "the hoard lived in the worker's heap, not the host's");
await Scripts.Eventually(() => runner.Shards[0].Current, TimeSpan.FromSeconds(10), "a worker");
(await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7);
}
/// <summary>Fail closed (design §14 #4): no worker means Bad, never an in-process run.</summary>
[Fact]
public async Task A_worker_that_cannot_start_reports_unavailable_and_never_runs_in_process()
{
var missing = Path.Combine(Path.GetTempPath(), "no-such-dir-" + Guid.NewGuid().ToString("N"), OutOfProcessScriptRunner.WorkerAssemblyFileName);
await using var runner = Scripts.Runner(Scripts.Options(workerPath: missing, unavailableGrace: TimeSpan.FromSeconds(1)));
var outcome = await runner.RunAsync(Scripts.Vt("vt", "return 1;"), Ct);
outcome.Status.ShouldBe(ScriptRunStatus.Unavailable);
outcome.Reason.ShouldBe(ScriptRunOutcome.RunnerUnavailableReason);
outcome.ToVirtualTagEvalResult().StatusCode.ShouldBe(ZB.MOM.WW.OtOpcUa.Commons.OpcUa.OpcUaStatus.BadResourceUnavailable);
runner.Shards[0].ConsecutiveFailures.ShouldBeGreaterThan(0, "the failed start backs off");
}
/// <summary>A worker whose pipe reaches end of stream exits by itself (the Linux "host died" path, design §3.3).</summary>
[Fact]
public async Task A_worker_exits_by_itself_when_its_pipe_closes()
{
await using var runner = Scripts.Runner(Scripts.Options());
(await runner.RunAsync(Scripts.Vt("vt", Healthy), Ct)).Value.ShouldBe(7);
var worker = runner.Shards[0].Current.ShouldNotBeNull();
using var process = Process.GetProcessById(worker.Pid);
var sw = Stopwatch.StartNew();
worker.SeverPipeForTest();
await process.WaitForExitAsync(Ct).WaitAsync(TimeSpan.FromSeconds(5), Ct);
sw.Elapsed.ShouldBeLessThan(TimeSpan.FromSeconds(2));
var exit = await worker.Exited.WaitAsync(TimeSpan.FromSeconds(5), Ct);
exit.ExitCode.ShouldBe(ScriptWorkerEnvironment.ExitPipeClosed, "it noticed the EOF; the host did not have to kill it");
}
/// <summary>Design §6: the environment is cleared and repopulated from an allow-list. Positive control first.</summary>
[Fact]
public async Task The_worker_environment_carries_no_host_secrets()
{
const string apiKey = "ServerHistorian__ApiKey";
const string configDb = "ConnectionStrings__ConfigDb";
Environment.SetEnvironmentVariable(apiKey, "sekrit");
Environment.SetEnvironmentVariable(configDb, "Server=x;Password=y");
try
{
Environment.GetEnvironmentVariable(apiKey).ShouldBe("sekrit", "positive control: the host has it");
await using var runner = Scripts.Runner(Scripts.Options());
(await runner.RunAsync(Scripts.Vt("vt", Healthy), Ct)).Value.ShouldBe(7);
var diag = await runner.Shards[0].Current!.DiagnosticsAsync(TimeSpan.FromSeconds(10));
diag.EnvironmentNames.ShouldNotContain(apiKey);
diag.EnvironmentNames.ShouldNotContain(configDb);
diag.EnvironmentNames.ShouldContain(ScriptWorkerEnvironment.Nonce, "the allow-listed launch variables do arrive");
diag.EnvironmentNames.ShouldContain("DOTNET_GCHeapHardLimit");
diag.WorkingDirectory.ShouldNotBe(Environment.CurrentDirectory);
// An empty per-worker directory under the temp root (macOS reports /var as /private/var, so match the leaf).
diag.WorkingDirectory.ShouldContain(Path.Combine("otopcua-scriptworker", "otopcua-sw-"));
Directory.EnumerateFileSystemEntries(diag.WorkingDirectory).ShouldBeEmpty();
}
finally
{
Environment.SetEnvironmentVariable(apiKey, null);
Environment.SetEnvironmentVariable(configDb, null);
}
}
/// <summary>Design §6: the host's cluster, database, OPC UA and secrets stacks are not in the worker.</summary>
[Fact]
public async Task The_worker_loads_none_of_the_host_stacks()
{
await using var runner = Scripts.Runner(Scripts.Options());
(await runner.RunAsync(Scripts.Vt("vt", "return System.Math.Abs(-3);"), Ct)).Value.ShouldBe(3);
var diag = await runner.Shards[0].Current!.DiagnosticsAsync(TimeSpan.FromSeconds(10));
diag.LoadedAssemblies.ShouldContain("ZB.MOM.WW.OtOpcUa.Core.Scripting", "positive control: the compile ran here");
diag.LoadedAssemblies.ShouldContain("Microsoft.CodeAnalysis.CSharp");
foreach (var prefix in ScriptWorkerEnvironment.ForbiddenAssemblyPrefixes)
diag.LoadedAssemblies.ShouldNotContain(a => a.StartsWith(prefix, StringComparison.Ordinal), prefix);
// Not merely unloaded: Akka is not even deployable beside the worker.
var folder = Path.GetDirectoryName(runner.WorkerPath)!;
File.Exists(Path.Combine(folder, "ZB.MOM.WW.OtOpcUa.ScriptWorker.dll")).ShouldBeTrue("positive control");
File.Exists(Path.Combine(folder, "Akka.dll")).ShouldBeFalse();
File.Exists(Path.Combine(folder, "Grpc.Core.Api.dll")).ShouldBeFalse();
}
[Fact]
public async Task Script_log_lines_come_back_through_the_invocations_logger()
{
var sink = new CapturingSink();
await using var runner = Scripts.Runner(Scripts.Options());
(await runner.RunAsync(Scripts.Vt("vt", "ctx.Logger.Information(\"hello {N}\", 42); return 1;", logger: sink.CreateLogger()), Ct))
.Value.ShouldBe(1);
await Scripts.Eventually(() => sink.Rendered.Contains("hello 42"), TimeSpan.FromSeconds(5), "the forwarded log line");
}
[Fact]
public async Task An_alarm_predicate_runs_out_of_process()
{
await using var runner = Scripts.Runner(Scripts.Options());
var outcome = await runner.RunAsync(new ScriptInvocation("alarm-1", ScriptContextKind.AlarmPredicate,
"return (double)ctx.GetTag(\"t\").Value > 50;", Scripts.Input("t", 75.0), TimeSpan.FromSeconds(5)), Ct);
outcome.Status.ShouldBe(ScriptRunStatus.Ok, outcome.Reason);
outcome.Value.ShouldBe(true);
}
/// <summary>Design §14 #11: an unsupported type is an explicit failure, never a silent conversion.</summary>
[Fact]
public async Task Unsupported_input_and_result_types_are_explicit_failures()
{
await using var runner = Scripts.Runner(Scripts.Options());
var result = await runner.RunAsync(Scripts.Vt("list", "return new System.Collections.Generic.List<int> { 1 };"), Ct);
result.Status.ShouldBe(ScriptRunStatus.Failed);
result.Reason!.ShouldStartWith("unsupported type: script returned a value of unsupported type System.Collections.Generic.List");
var input = await runner.RunAsync(Scripts.Vt("guid", "return 1;", inputs: Scripts.Input("g", Guid.NewGuid())), Ct);
input.Status.ShouldBe(ScriptRunStatus.Failed);
input.Reason!.ShouldStartWith("unsupported type: input 'g' has a value of unsupported type System.Guid");
}
}
@@ -0,0 +1,86 @@
using System.IO.Pipes;
using System.Security.Cryptography;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests;
/// <summary>lmxopcua#780 design §6 "Pipe hardening": only the launched process, echoing its nonce, is accepted.</summary>
public sealed class HandshakeTests
{
private static CancellationToken Ct => TestContext.Current.CancellationToken;
private static async Task<Exception?> HandshakeWith(Func<Envelope, Envelope> reply, string expectedNonce = "right", int? expectedPid = 1234)
{
var name = "otopcua-sw-test-" + Convert.ToHexString(RandomNumberGenerator.GetBytes(6));
await using var server = new NamedPipeServerStream(name, PipeDirection.InOut, 1, PipeTransmissionMode.Byte,
PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly);
await using var client = new NamedPipeClientStream(".", name, PipeDirection.InOut, PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly);
var accept = server.WaitForConnectionAsync(Ct);
await client.ConnectAsync(TimeSpan.FromSeconds(5), Ct);
await accept;
var rogue = Task.Run(async () =>
{
var hello = await ScriptWorkerFrameCodec.ReadAsync(client, ScriptWorkerFrameCodec.DefaultMaxFrameBytes, Ct);
await ScriptWorkerFrameCodec.WriteAsync(client, reply(hello!), ScriptWorkerFrameCodec.DefaultMaxFrameBytes, Ct);
await client.FlushAsync(Ct);
}, Ct);
try
{
using var timeout = CancellationTokenSource.CreateLinkedTokenSource(Ct);
timeout.CancelAfter(TimeSpan.FromSeconds(5));
await WorkerConnection.HandshakeAsync(server, new Hello { Nonce = expectedNonce }, expectedNonce, expectedPid, timeout.Token);
await rogue;
return null;
}
catch (Exception ex)
{
return ex;
}
}
private static Envelope Ready(string nonce, int pid) => new()
{
ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion,
Ready = new Ready { Nonce = nonce, Pid = pid },
};
[Fact]
public async Task The_launched_process_echoing_its_nonce_is_accepted()
=> (await HandshakeWith(_ => Ready("right", 1234))).ShouldBeNull("positive control");
[Fact]
public async Task A_wrong_nonce_is_refused()
=> (await HandshakeWith(_ => Ready("wrong", 1234))).ShouldBeOfType<ScriptWorkerProtocolException>()
.Message.ShouldContain("nonce");
[Fact]
public async Task A_client_that_is_not_the_launched_process_is_refused()
=> (await HandshakeWith(_ => Ready("right", 999))).ShouldBeOfType<ScriptWorkerProtocolException>()
.Message.ShouldContain("pid");
[Fact]
public async Task Anything_but_ready_first_is_refused()
=> (await HandshakeWith(_ => new Envelope
{
ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion,
Heartbeat = new Heartbeat(),
})).ShouldBeOfType<ScriptWorkerProtocolException>();
[Fact]
public async Task The_hello_carries_the_nonce_to_echo()
{
string? seen = null;
(await HandshakeWith(hello =>
{
seen = hello.Hello.Nonce;
return Ready(hello.Hello.Nonce, 1234);
})).ShouldBeNull();
seen.ShouldBe("right");
}
}
@@ -0,0 +1,95 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests;
/// <summary>
/// lmxopcua#780 design §13 "Parity": the same script, through <see cref="InProcessScriptRunner"/> (today's guard)
/// and through the worker, gives the same status, reason, value and exact CLR value type.
/// </summary>
[Collection(ScriptWorkerCollection.Name)]
[Trait("Category", "ScriptWorker")]
public sealed class ParityTests
{
private static CancellationToken Ct => TestContext.Current.CancellationToken;
public static TheoryData<string, string> Cases() => new()
{
{ "arithmetic", "return 1 + 2;" },
{ "string", "return \"a\" + ctx.GetTag(\"x\").Value;" },
{ "throw", "throw new System.InvalidOperationException(\"boom\");" },
{ "compile error", "return undefinedThing;" },
{ "sandbox violation", "return System.IO.File.Exists(\"x\");" },
{ "missing input status", "return ctx.GetTag(\"missing\").StatusCode;" },
{ "input status", "return ctx.GetTag(\"x\").StatusCode;" },
{ "source timestamp", "return ctx.GetTag(\"x\").SourceTimestampUtc;" },
{ "clock", "return ctx.Now > new System.DateTime(2000, 1, 1);" },
{ "null", "return null;" },
{ "array", "return new double[] { 1.5, 2.5 };" },
{ "cast of wrong type", "return (short)ctx.GetTag(\"x\").Value;" },
};
private static readonly IReadOnlyDictionary<string, DataValueSnapshot> Inputs =
new Dictionary<string, DataValueSnapshot>(StringComparer.Ordinal)
{
["x"] = new DataValueSnapshot(5, 0x40000000u, new DateTime(2026, 1, 2, 3, 4, 5, DateTimeKind.Utc), DateTime.UtcNow),
};
[Theory]
[MemberData(nameof(Cases))]
public async Task In_and_out_of_process_agree(string name, string source)
{
using var inProcess = new InProcessScriptRunner();
await using var outOfProcess = Scripts.Runner(Scripts.Options());
var invocation = Scripts.Vt(name, source, inputs: Inputs);
var expected = await inProcess.RunAsync(invocation, Ct);
var actual = await outOfProcess.RunAsync(invocation, Ct);
actual.Status.ShouldBe(expected.Status, $"{name}: {actual.Reason} vs {expected.Reason}");
actual.Reason.ShouldBe(expected.Reason);
Same(actual.Value, expected.Value, name);
}
public static TheoryData<string, object> Typed() => new()
{
{ "bool", true }, { "sbyte", (sbyte)-2 }, { "byte", (byte)2 }, { "short", (short)-3 }, { "ushort", (ushort)3 },
{ "int", -4 }, { "uint", 4u }, { "long", -5L }, { "ulong", 5UL }, { "float", 1.25f }, { "double", 2.5 },
{ "string", "s" }, { "System.DateTime", new DateTime(2026, 11, 9, 0, 0, 0, DateTimeKind.Utc) },
};
/// <summary>
/// Unboxing parity (pins design assumption A-7): <c>(T)ctx.GetTag("x").Value</c> works out of process for every
/// tag type, which needs the exact boxed CLR type to survive the wire.
/// </summary>
[Theory]
[MemberData(nameof(Typed))]
public async Task Unboxing_to_the_exact_type_matches(string typeName, object value)
{
using var inProcess = new InProcessScriptRunner();
await using var outOfProcess = Scripts.Runner(Scripts.Options());
var invocation = Scripts.Vt("unbox-" + typeName, $"var v = ({typeName})ctx.GetTag(\"x\").Value; return v;",
inputs: Scripts.Input("x", value));
var expected = await inProcess.RunAsync(invocation, Ct);
var actual = await outOfProcess.RunAsync(invocation, Ct);
expected.Status.ShouldBe(ScriptRunStatus.Ok, expected.Reason);
actual.Status.ShouldBe(ScriptRunStatus.Ok, actual.Reason);
Same(actual.Value, expected.Value, typeName);
}
private static void Same(object? actual, object? expected, string because)
{
if (expected is null)
{
actual.ShouldBeNull(because);
return;
}
actual.ShouldNotBeNull(because);
actual.GetType().ShouldBe(expected.GetType(), because);
if (expected is Array array) ((Array)actual).Cast<object?>().ShouldBe(array.Cast<object?>(), because);
else actual.ShouldBe(expected, because);
}
}
@@ -0,0 +1,198 @@
using System.Buffers.Binary;
using Google.Protobuf;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests;
/// <summary>lmxopcua#780 design §13 — frame codec and the CLR ↔ Variant mapping.</summary>
public sealed class ProtocolTests
{
public static IEnumerable<object?[]> Values()
{
object?[] values =
[
null, true, false, (sbyte)-5, (byte)250, (short)-3, (ushort)7, -42, 42u, long.MinValue, ulong.MaxValue,
1.5f, double.NaN, 3.25, "text", string.Empty, new DateTime(2026, 11, 9, 1, 2, 3, DateTimeKind.Utc),
new[] { 1, 2, 3 }, new short[] { -1, 1 }, new[] { "a", "b" }, new[] { true, false }, Array.Empty<double>(),
new[] { new DateTime(2020, 1, 1, 0, 0, 0, DateTimeKind.Utc) }, (short)0, 0, 0L, 0.0,
];
return values.Select(v => new[] { v });
}
/// <summary>The exact boxed CLR type survives (design §4.4): <c>(short)ctx.GetTag(x).Value</c> must keep working.</summary>
[Theory]
[MemberData(nameof(Values))]
public void Every_type_tag_round_trips_with_its_exact_clr_type(object? value)
{
VariantCodec.TryEncode(value, out var variant, out var unsupported).ShouldBeTrue();
unsupported.ShouldBeNull();
// Through the wire, not just the object: a payload the serializer drops must still decode.
var wire = Variant.Parser.ParseFrom(variant.ToByteArray());
var decoded = VariantCodec.Decode(wire);
if (value is null)
{
decoded.ShouldBeNull();
return;
}
decoded.ShouldNotBeNull();
decoded.GetType().ShouldBe(value.GetType());
if (value is Array expected)
((Array)decoded).Cast<object?>().ShouldBe(expected.Cast<object?>());
else if (value is double d && double.IsNaN(d))
double.IsNaN((double)decoded).ShouldBeTrue();
else
decoded.ShouldBe(value);
}
[Fact]
public void Unsupported_types_are_refused_not_converted()
{
foreach (var value in new object[] { new List<int> { 1 }, 1.5m, Guid.NewGuid(), new { A = 1 }, new int[1, 1], new object[] { 1 } })
{
VariantCodec.TryEncode(value, out _, out var type).ShouldBeFalse(value.GetType().Name);
type.ShouldNotBeNullOrEmpty();
}
}
[Fact]
public void Local_datetime_crosses_as_utc()
{
var local = new DateTime(2026, 6, 1, 12, 0, 0, DateTimeKind.Local);
VariantCodec.TryEncode(local, out var variant, out _).ShouldBeTrue();
var decoded = (DateTime)VariantCodec.Decode(variant)!;
decoded.Kind.ShouldBe(DateTimeKind.Utc);
decoded.ShouldBe(local.ToUniversalTime());
}
[Fact]
public void A_payload_that_contradicts_its_tag_is_a_protocol_fault()
{
Should.Throw<ScriptWorkerProtocolException>(() => VariantCodec.Decode(new Variant { Type = Variant.Types.TypeTag.Int16, S = "x" }));
Should.Throw<ScriptWorkerProtocolException>(() => VariantCodec.Decode(new Variant { Type = (Variant.Types.TypeTag)99 }));
Should.Throw<OverflowException>(() => VariantCodec.Decode(new Variant { Type = Variant.Types.TypeTag.Int16, I = 1 << 20 }));
}
private static Envelope Sample() => new()
{
ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion,
Seq = 7,
Heartbeat = new Heartbeat { Inflight = 3, HeapBytes = 1234 },
};
[Fact]
public async Task A_frame_round_trips()
{
using var stream = new MemoryStream();
await ScriptWorkerFrameCodec.WriteAsync(stream, Sample(), 1024, TestContext.Current.CancellationToken);
stream.Position = 0;
var read = await ScriptWorkerFrameCodec.ReadAsync(stream, 1024, TestContext.Current.CancellationToken);
read.ShouldBe(Sample());
(await ScriptWorkerFrameCodec.ReadAsync(stream, 1024, TestContext.Current.CancellationToken)).ShouldBeNull("a clean end of stream at a frame boundary");
}
[Fact]
public async Task Oversize_frames_are_refused_on_write_and_on_read()
{
using var stream = new MemoryStream();
await Should.ThrowAsync<ScriptWorkerProtocolException>(async () =>
await ScriptWorkerFrameCodec.WriteAsync(stream, Sample(), 4, TestContext.Current.CancellationToken));
stream.Length.ShouldBe(0, "nothing is written for a refused frame");
await ScriptWorkerFrameCodec.WriteAsync(stream, Sample(), 1024, TestContext.Current.CancellationToken);
stream.Position = 0;
await Should.ThrowAsync<ScriptWorkerProtocolException>(async () =>
await ScriptWorkerFrameCodec.ReadAsync(stream, 4, TestContext.Current.CancellationToken));
}
public static TheoryData<string, byte[]> BadFrames()
{
var good = Sample().ToByteArray();
var header = new byte[4];
BinaryPrimitives.WriteInt32LittleEndian(header, good.Length);
var wrongVersion = new Envelope { ProtocolVersion = 99, Heartbeat = new Heartbeat() }.ToByteArray();
var noBody = new Envelope { ProtocolVersion = 1, Seq = 1 }.ToByteArray();
return new TheoryData<string, byte[]>
{
{ "zero length", new byte[] { 0, 0, 0, 0 } },
{ "negative length", new byte[] { 0xFF, 0xFF, 0xFF, 0xFF } },
{ "truncated header", new byte[] { 5, 0 } },
{ "truncated body", [.. header, .. good[..^1]] },
{ "garbage body", [5, 0, 0, 0, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF] },
{ "wrong protocol version", Frame(wrongVersion) },
{ "no body", Frame(noBody) },
};
}
private static byte[] Frame(byte[] body)
{
var header = new byte[4];
BinaryPrimitives.WriteInt32LittleEndian(header, body.Length);
return [.. header, .. body];
}
[Theory]
[MemberData(nameof(BadFrames))]
public async Task Malformed_frames_are_protocol_faults(string why, byte[] bytes)
{
using var stream = new MemoryStream(bytes);
await Should.ThrowAsync<ScriptWorkerProtocolException>(async () =>
await ScriptWorkerFrameCodec.ReadAsync(stream, 1024, TestContext.Current.CancellationToken), why);
}
/// <summary>Random bytes never escape as anything but a protocol fault (or a clean end of stream).</summary>
[Fact]
public async Task Fuzzed_input_only_ever_yields_a_protocol_fault()
{
var random = new Random(780);
for (var i = 0; i < 3000; i++)
{
var bytes = new byte[random.Next(0, 64)];
random.NextBytes(bytes);
if (bytes.Length >= 4 && random.Next(2) == 0)
BinaryPrimitives.WriteInt32LittleEndian(bytes, random.Next(1, bytes.Length));
using var stream = new MemoryStream(bytes);
try
{
while (await ScriptWorkerFrameCodec.ReadAsync(stream, 1024, TestContext.Current.CancellationToken) is { } envelope)
{
// A frame that parses must still decode its variants without an unexpected exception type.
if (envelope.BodyCase == Envelope.BodyOneofCase.Results)
foreach (var r in envelope.Results.Items)
try { VariantCodec.Decode(r.Value); } catch (Exception ex) when (ex is ScriptWorkerProtocolException or OverflowException) { }
}
}
catch (ScriptWorkerProtocolException)
{
}
}
}
[Fact]
public void Backoff_doubles_from_250ms_to_a_30s_cap_with_20_percent_jitter()
{
WorkerShard.NextBackoff(1, 0.5).ShouldBe(TimeSpan.FromMilliseconds(250));
WorkerShard.NextBackoff(2, 0.5).ShouldBe(TimeSpan.FromMilliseconds(500));
WorkerShard.NextBackoff(3, 0.5).ShouldBe(TimeSpan.FromSeconds(1));
WorkerShard.NextBackoff(20, 0.5).ShouldBe(TimeSpan.FromSeconds(30));
WorkerShard.NextBackoff(1, 0).ShouldBe(TimeSpan.FromMilliseconds(200));
WorkerShard.NextBackoff(1, 0.9999999).TotalMilliseconds.ShouldBe(300, tolerance: 0.01);
WorkerShard.NextBackoff(40, 1).ShouldBe(TimeSpan.FromSeconds(36));
}
[Fact]
public void Shard_placement_is_stable_and_in_range()
{
// FNV-1a, not string.GetHashCode (randomised per process): a script stays on its shard across restarts.
OutOfProcessScriptRunner.ShardIndex("1:vt-1", 2).ShouldBe(OutOfProcessScriptRunner.ShardIndex("1:vt-1", 2));
var spread = Enumerable.Range(0, 400).Select(i => OutOfProcessScriptRunner.ShardIndex($"1:vt-{i}", 4)).ToArray();
spread.ShouldAllBe(i => i >= 0 && i < 4);
spread.Distinct().Count().ShouldBe(4);
OutOfProcessScriptRunner.ShardIndex("abc", 1000).ShouldBe((int)(0x1A47E90Bu % 1000));
}
}
@@ -0,0 +1,88 @@
using System.Collections.Concurrent;
using System.Diagnostics;
using Serilog;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
using ZB.MOM.WW.OtOpcUa.Driver.Calculation;
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests;
/// <summary>
/// lmxopcua#780 — the whole path a client sees: a real Calculation driver whose seam runs out of process on the real
/// worker. A script failure publishes <c>BadInternalError</c> as today; the worker dying under a script publishes
/// <c>BadResourceUnavailable</c> (0x80040000), the ScadaBridge-visible code; and the tag recovers Good afterwards.
/// </summary>
[Collection(ScriptWorkerCollection.Name)]
[Trait("Category", "ScriptWorker")]
public sealed class SeamEndToEndTests
{
private const uint StatusGood = 0u;
private const uint StatusBadInternalError = 0x80020000u;
private const uint StatusBadResourceUnavailable = 0x80040000u;
private const string Tag = "calc/e/spin";
/// <summary>Spins for <c>src/ms</c> milliseconds, throws when it is negative, and returns it otherwise.</summary>
private const string Script =
"var ms = (double)ctx.GetTag(\"src/ms\").Value; if (ms < 0) throw new System.InvalidOperationException(\"negative\"); " +
"var end = ctx.Now.AddMilliseconds(ms); while (ctx.Now < end) { } return ms;";
private static CancellationToken Ct => TestContext.Current.CancellationToken;
private static async Task Settle(CalculationDriver driver)
{
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(60);
while (DateTime.UtcNow < deadline && !driver.IsEvaluationIdle) await Task.Delay(10);
driver.IsEvaluationIdle.ShouldBeTrue();
}
[Fact]
public async Task A_calculation_tag_publishes_BadResourceUnavailable_only_when_the_worker_died_under_it()
{
await using var runner = Scripts.Runner(Scripts.Options());
var json = $"{{\"scriptId\":\"s\",\"changeTriggered\":true,\"scriptSource\":{System.Text.Json.JsonSerializer.Serialize(Script)}}}";
var driver = new CalculationDriver(
new CalculationDriverOptions { RawTags = [new RawTagEntry(Tag, json, WriteIdempotent: false)], RunTimeout = TimeSpan.FromSeconds(20), ScriptRunner = runner },
"calc-drv", new ScriptRootLogger(new LoggerConfiguration().CreateLogger()));
var published = new ConcurrentQueue<DataChangeEventArgs>();
driver.OnDataChange += (_, e) => published.Enqueue(e);
await driver.InitializeAsync("{}", Ct);
driver.OnDependencyValue("src/ms", 0.0, StatusGood, DateTime.UtcNow);
await Settle(driver);
published.TryDequeue(out var good).ShouldBeTrue();
good!.Snapshot.StatusCode.ShouldBe(StatusGood);
good.Snapshot.Value.ShouldBe(0.0);
// A script fault: BadInternalError, exactly as in process.
driver.OnDependencyValue("src/ms", -1.0, StatusGood, DateTime.UtcNow);
await Settle(driver);
published.TryDequeue(out var threw).ShouldBeTrue();
threw!.Snapshot.StatusCode.ShouldBe(StatusBadInternalError);
driver.OnDependencyValue("src/ms", 1.0, StatusGood, DateTime.UtcNow);
await Settle(driver);
published.TryDequeue(out var back).ShouldBeTrue();
back!.Snapshot.StatusCode.ShouldBe(StatusGood);
// The worker dies under a run: the runner, not the script, failed.
driver.OnDependencyValue("src/ms", 5000.0, StatusGood, DateTime.UtcNow);
var worker = await Scripts.Eventually(() => runner.Shards[0].Current is { InFlightCount: 1 } w ? w : null,
TimeSpan.FromSeconds(10), "the run in flight");
await Task.Delay(200, Ct);
Process.GetProcessById(worker.Pid).Kill();
await Settle(driver);
published.TryDequeue(out var unavailable).ShouldBeTrue();
unavailable!.Snapshot.StatusCode.ShouldBe(StatusBadResourceUnavailable);
unavailable.Snapshot.Value.ShouldBe(1.0, "the last good value rides out with the Bad status");
// Recovery: the next trigger runs (in the isolation worker, as a suspect) and publishes Good.
driver.OnDependencyValue("src/ms", 2.0, StatusGood, DateTime.UtcNow);
await Settle(driver);
published.TryDequeue(out var recovered).ShouldBeTrue();
recovered!.Snapshot.StatusCode.ShouldBe(StatusGood);
recovered.Snapshot.Value.ShouldBe(2.0);
await driver.ShutdownAsync(Ct);
}
}
@@ -0,0 +1,82 @@
using System.Collections.Concurrent;
using Serilog;
using Serilog.Core;
using Serilog.Events;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests;
/// <summary>Collects Serilog events in memory.</summary>
internal sealed class CapturingSink : ILogEventSink
{
public ConcurrentQueue<LogEvent> Events { get; } = new();
public void Emit(LogEvent logEvent) => Events.Enqueue(logEvent);
public ILogger CreateLogger() => new LoggerConfiguration().MinimumLevel.Verbose().WriteTo.Sink(this).CreateLogger();
public IEnumerable<string> Rendered => Events.Select(e => e.RenderMessage());
}
internal static class Scripts
{
public static readonly IReadOnlyDictionary<string, DataValueSnapshot> NoInputs =
new Dictionary<string, DataValueSnapshot>(StringComparer.Ordinal);
public static IReadOnlyDictionary<string, DataValueSnapshot> Input(string path, object? value, uint status = 0)
=> new Dictionary<string, DataValueSnapshot>(StringComparer.Ordinal)
{
[path] = new DataValueSnapshot(value, status, DateTime.UtcNow, DateTime.UtcNow),
};
public static ScriptInvocation Vt(string key, string source, TimeSpan? timeout = null,
IReadOnlyDictionary<string, DataValueSnapshot>? inputs = null, ILogger? logger = null)
=> new(key, ScriptContextKind.VirtualTag, source, inputs ?? NoInputs, timeout ?? TimeSpan.FromSeconds(5), logger);
/// <summary>Options for a single-shard runner with short budgets, so the containment tests run in seconds.</summary>
public static ScriptingOptions Options(int shards = 1, TimeSpan? hardGrace = null, int quarantineAfterKills = 2,
int probation = 20, TimeSpan? unavailableGrace = null, string? workerPath = null, long heapLimit = 256L * 1024 * 1024)
=> new()
{
Runner = ScriptRunnerMode.OutOfProcess,
QuarantineAfterConsecutiveTimeouts = 100,
Worker = new ScriptWorkerOptions
{
Shards = shards,
HardGrace = hardGrace ?? TimeSpan.FromMilliseconds(500),
QuarantineAfterKills = quarantineAfterKills,
IsolationProbationRuns = probation,
UnavailableGrace = unavailableGrace ?? TimeSpan.FromSeconds(30),
ExecutablePath = workerPath,
HeapLimitBytes = heapLimit,
},
};
public static OutOfProcessScriptRunner Runner(ScriptingOptions options, ILogger? log = null)
=> new(options, log ?? new LoggerConfiguration().CreateLogger(), null,
heartbeatInterval: TimeSpan.FromSeconds(1), loadTimeout: TimeSpan.FromSeconds(60));
public static async Task<T> Eventually<T>(Func<T?> probe, TimeSpan timeout, string what) where T : class
{
var deadline = DateTime.UtcNow + timeout;
while (DateTime.UtcNow < deadline)
{
if (probe() is { } value) return value;
await Task.Delay(20);
}
throw new TimeoutException($"timed out waiting for {what}");
}
public static async Task Eventually(Func<bool> probe, TimeSpan timeout, string what)
{
var deadline = DateTime.UtcNow + timeout;
while (DateTime.UtcNow < deadline)
{
if (probe()) return;
await Task.Delay(20);
}
throw new TimeoutException($"timed out waiting for {what}");
}
}
@@ -0,0 +1,30 @@
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<IsPackable>false</IsPackable>
<IsTestProject>true</IsTestProject>
<RootNamespace>ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests</RootNamespace>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="xunit.v3"/>
<PackageReference Include="Shouldly"/>
<PackageReference Include="Microsoft.NET.Test.Sdk"/>
<PackageReference Include="xunit.runner.visualstudio">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\..\src\Core\ZB.MOM.WW.OtOpcUa.Core.Scripting\ZB.MOM.WW.OtOpcUa.Core.Scripting.csproj"/>
<ProjectReference Include="..\..\..\src\Core\ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol\ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.csproj"/>
<ProjectReference Include="..\..\..\src\Core\ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess\ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess.csproj"/>
<!-- End-to-end seam test: a real Calculation driver over the real worker. -->
<ProjectReference Include="..\..\..\src\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Calculation\ZB.MOM.WW.OtOpcUa.Driver.Calculation.csproj"/>
</ItemGroup>
<!-- The real worker, built and placed in script-worker/ beside the tests, exactly as beside the Host. -->
<Import Project="..\..\..\src\Server\ZB.MOM.WW.OtOpcUa.ScriptWorker\build\ScriptWorker.Deploy.targets"/>
</Project>