diff --git a/CLAUDE.md b/CLAUDE.md index 48330704..f4636df1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -388,3 +388,10 @@ endpoints under `/api/script-analysis/*`, gated by the `ConfigEditor` policy `ScriptAnalysisEndpoints.MapScriptAnalysisEndpoints`). `Core.VirtualTags` is now just `DependencyGraph`, `ITagUpstreamSource` and `IHistoryWriter` — the dormant `VirtualTagEngine`/Source/Definition and `TimerTriggerScheduler` were deleted. See `docs/ScriptEditor.md` for the full guide. Scripts may use the `{{equip}}` token for equipment-relative tag paths that resolve per-equipment at deploy time — see the "Equipment-relative tag paths" section in `docs/ScriptEditor.md`. + +**Out-of-process script runner (lmxopcua#780, opt-in).** `Scripting:Runner` = `InProcess` (default, unchanged) | `OutOfProcess` | `Shadow` (not yet). +`OutOfProcess` runs the VirtualTag + Calculation seams' compiled scripts in `ZB.MOM.WW.OtOpcUa.ScriptWorker` processes +(`src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess`, protocol in `Core.Scripting.Protocol`), shipped in `script-worker/` +beside the host; a runner-side failure publishes `BadResourceUnavailable` (ScadaBridge-visible, ScadaBridge#112). The worker tests +are `tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests` (`Category=ScriptWorker`, real processes). See `docs/VirtualTags.md` +§"Out-of-process runner". diff --git a/ZB.MOM.WW.OtOpcUa.slnx b/ZB.MOM.WW.OtOpcUa.slnx index 4621530b..fd177ab1 100644 --- a/ZB.MOM.WW.OtOpcUa.slnx +++ b/ZB.MOM.WW.OtOpcUa.slnx @@ -8,6 +8,8 @@ + + @@ -15,6 +17,7 @@ + @@ -91,6 +94,7 @@ + diff --git a/docs/ScriptEditor.md b/docs/ScriptEditor.md index 30f1a443..5c46e437 100644 --- a/docs/ScriptEditor.md +++ b/docs/ScriptEditor.md @@ -211,6 +211,11 @@ scripted-alarm predicates) runs scripts through the shared `ScriptRunGuard`: `Scripting:MaxAllocBytesPerRun` (64 MB) is quarantined. This is sampled after the run returns, so a run that OOMs the process first is not stopped — that needs an out-of-process runner (→ #750). +- **Out of process (opt-in, lmxopcua#780).** `Scripting:Runner=OutOfProcess` moves the + virtual-tag and Calculation seams' compiled runs into `ScriptWorker` processes, where a + run past `T + Scripting:Worker:HardGrace` is stopped by killing its worker and an OOM + kills a worker instead of the server. A run whose worker died under it reads + `BadResourceUnavailable`. See `docs/VirtualTags.md` §"Out-of-process runner". - **Recursion.** Unbounded recursion fails the script with `ScriptStackExhaustedException` (Bad, script log) instead of overflowing the stack, which would terminate the whole server. The runtime compile inserts a diff --git a/docs/ServiceHosting.md b/docs/ServiceHosting.md index 48b13b9b..3ad4427d 100644 --- a/docs/ServiceHosting.md +++ b/docs/ServiceHosting.md @@ -88,6 +88,16 @@ and supply `ServerHistorian__ApiKey` via the environment on the OtOpcUa host sid `historian:write` + `historian:tags:write`. See [Historian.md](Historian.md) for the full config-key and deployment-prerequisite reference. (The retired Wonderware TCP sidecar: [Historian.Wonderware.md](drivers/Historian.Wonderware.md).) +## Script worker processes (`Scripting:Runner=OutOfProcess`, lmxopcua#780) + +Off by default. When enabled, a driver node launches up to `Scripting:Worker:Shards` + 1 child processes +(`ZB.MOM.WW.OtOpcUa.ScriptWorker`, from `script-worker/` beside the host) on its first compiled script evaluation and +kills them with it: a Job Object with kill-on-close on Windows, pipe end of stream plus a parent-pid poll elsewhere, +and a Shutdown frame then a kill on host stop. Their stderr lands in the host log prefixed `ScriptWorker[]`. +The worker currently runs under the host's account through the dotnet muxer; the dedicated low-privilege account, +pipe ACL and firewall outbound rule (design §14 #5/#6) are installer work still open on lmxopcua#780. See +`docs/VirtualTags.md` §"Out-of-process runner". + ## Install / Uninstall - `scripts/install/Install-Services.ps1 -Roles admin,driver` — installs `OtOpcUaHost`. diff --git a/docs/VirtualTags.md b/docs/VirtualTags.md index 851977d1..c8f91f40 100644 --- a/docs/VirtualTags.md +++ b/docs/VirtualTags.md @@ -37,11 +37,54 @@ User scripts are compiled via `Microsoft.CodeAnalysis.CSharp` (regular compiler, #### Known resource limits (accepted trade-offs) -The sandbox cannot pre-empt a script. **Memory:** each run's own-thread allocation is sampled (`GC.GetAllocatedBytesForCurrentThread`) and a run over `Scripting:MaxAllocBytesPerRun` (default 64 MB) quarantines the script — but the sample is taken after the run returns, so a script that allocates fast enough (`new byte[int.MaxValue]` in a loop) can still OOM the process first. **CPU:** a tight loop keeps its pool thread after the timeout fires; `ScriptRunGuard` bounds that to one orphan per script by refusing further runs while it spins (Bad `"previous evaluation still running"`) and quarantines after three consecutive timeouts (a refusal is not a strike) — or immediately on a refusal behind an orphan that has run longer than `Scripting:StuckOrphanQuarantineFactor` (default 10) × the timeout, which is what quarantines a never-returning `while(true)`. Both are mitigations; pre-emptive CPU/memory limits need an out-of-process runner (→ #750). **Stack:** unbounded recursion used to terminate the whole server — a `StackOverflowException` cannot be caught, and lmxopcua#779 reproduced it out of process (local function, mutual recursion, self-referencing `Func<>`, anonymous self-application, recursion through LINQ: the child process died with "Stack overflow." / SIGABRT). `ScriptEvaluator.Compile` now rewrites the script so the body and every lambda, anonymous method and local function starts with `ScriptStackGuard.Check()` (`RuntimeHelpers.TryEnsureSufficientExecutionStack`), so the recursion fails the script with `ScriptStackExhaustedException` — Bad quality and a script-log entry like any throwing script — while there is still stack left. A runtime check rather than an analyzer rule because recursion is not decidable from syntax (anonymous self-application never names itself); bounded recursion keeps working. Limits: the depth reached depends on the evaluating thread's stack size; a recursion that never re-enters script code is not seen (the sandbox exposes no such API); a script can catch the exception, which only unwinds it. Pinned by `ScriptRecursionContainmentTests` (one child process per shape) and `ScriptStackGuardInjectionTests`. Script authoring is gated by the AdminUI `ConfigEditor` policy — **`Administrator` or `Designer`**, not Administrator alone — and that same policy gates deploy, so a Designer can author a script and put it into production; that gate is the primary control (AdminUI-001 deliberately admitted Designers to the script pages and `/api/script-analysis/*`; lmxopcua#779 corrected this sentence, which used to say Admin-only). See `docs/ScriptEditor.md` §"Runtime safety". +The sandbox cannot pre-empt a script. **Memory:** each run's own-thread allocation is sampled (`GC.GetAllocatedBytesForCurrentThread`) and a run over `Scripting:MaxAllocBytesPerRun` (default 64 MB) quarantines the script — but the sample is taken after the run returns, so a script that allocates fast enough (`new byte[int.MaxValue]` in a loop) can still OOM the process first. **CPU:** a tight loop keeps its pool thread after the timeout fires; `ScriptRunGuard` bounds that to one orphan per script by refusing further runs while it spins (Bad `"previous evaluation still running"`) and quarantines after three consecutive timeouts (a refusal is not a strike) — or immediately on a refusal behind an orphan that has run longer than `Scripting:StuckOrphanQuarantineFactor` (default 10) × the timeout, which is what quarantines a never-returning `while(true)`. Both are mitigations; pre-emptive CPU/memory limits need the out-of-process runner (→ #750 design, #780; opt-in via `Scripting:Runner=OutOfProcess`, below). **Stack:** unbounded recursion used to terminate the whole server — a `StackOverflowException` cannot be caught, and lmxopcua#779 reproduced it out of process (local function, mutual recursion, self-referencing `Func<>`, anonymous self-application, recursion through LINQ: the child process died with "Stack overflow." / SIGABRT). `ScriptEvaluator.Compile` now rewrites the script so the body and every lambda, anonymous method and local function starts with `ScriptStackGuard.Check()` (`RuntimeHelpers.TryEnsureSufficientExecutionStack`), so the recursion fails the script with `ScriptStackExhaustedException` — Bad quality and a script-log entry like any throwing script — while there is still stack left. A runtime check rather than an analyzer rule because recursion is not decidable from syntax (anonymous self-application never names itself); bounded recursion keeps working. Limits: the depth reached depends on the evaluating thread's stack size; a recursion that never re-enters script code is not seen (the sandbox exposes no such API); a script can catch the exception, which only unwinds it. Pinned by `ScriptRecursionContainmentTests` (one child process per shape) and `ScriptStackGuardInjectionTests`. Script authoring is gated by the AdminUI `ConfigEditor` policy — **`Administrator` or `Designer`**, not Administrator alone — and that same policy gates deploy, so a Designer can author a script and put it into production; that gate is the primary control (AdminUI-001 deliberately admitted Designers to the script pages and `/api/script-analysis/*`; lmxopcua#779 corrected this sentence, which used to say Admin-only). See `docs/ScriptEditor.md` §"Runtime safety". `System.Threading.Tasks` is denied (Core.Scripting-003), which prevents `Task.Run` / `Parallel` fan-out that would spawn background work outliving the timeout. +#### Out-of-process runner (`Scripting:Runner`, lmxopcua#780) + +`Scripting:Runner` selects where compiled scripts run: **`InProcess`** (the default and the rollback — everything +above, unchanged), **`OutOfProcess`**, or `Shadow` (design P2; not implemented yet, so it runs `InProcess` with a +startup Warning). `Scripting:Runner:Seams` narrows `OutOfProcess` to a subset (`VirtualTag`, `Calculation`; the +`ScriptedAlarm` seam stays in process for now — alarms go last, design §14 #10). Under `OutOfProcess` the virtual-tag +and Calculation evaluators hand each compiled run to `OutOfProcessScriptRunner` +(`Core.Scripting.OutOfProcess`), which runs it in a pool of `ZB.MOM.WW.OtOpcUa.ScriptWorker` processes: + +- **K shards** (`Scripting:Worker:Shards`, default 2) placed by a stable FNV-1a hash of the key, plus an on-demand + **isolation worker** that runs suspects one at a time. No worker starts until the first compiled evaluation, so a + node without scripts never launches one. The passthrough fast path never leaves the host. +- **Two deadlines.** At the soft deadline `T` (the seam's budget, unchanged) the caller gets today's + `script timed out after …` failure and the run token is cancelled inside the worker; at `T + G` + (`Scripting:Worker:HardGrace`, default `max(1 s, T)`) the worker is **killed** (`Process.Kill(entireProcessTree)`, + plus a worker-side fail-fast watchdog) — the thread, its memory and its ALC go with it. +- **Attribution and quarantine** live in the host and survive restarts: a deadline kill strikes exactly its key + (other evaluations in flight are victims, not strikes); a crash with no deadline crossed makes every key in flight + a suspect that must complete `Scripting:Worker:IsolationProbationRuns` (20) clean runs in the isolation worker, + where a crash strikes exactly one key. `Scripting:Worker:QuarantineAfterKills` (2) strikes within an hour, or the + existing consecutive-timeout and allocation rules, quarantine with the usual `"script quarantined: "` prefix; a + source change or a redeploy lifts it. +- **Status codes.** Script faults keep `BadInternalError`. When the runner, not the script, failed — the worker + restarted under the run (`"script worker restarted"`) or no worker became ready within + `Scripting:Worker:UnavailableGrace` (5 s, `"script runner unavailable"`) — the tag publishes + **`BadResourceUnavailable` (0x80040000)** with its last value. This is ScadaBridge-visible (ScadaBridge#112). + The runner **fails closed**: no worker means Bad, never an in-process run. +- **Containment.** The worker gets a cleared, allow-listed environment (no `ServerHistorian__ApiKey`, no + connection strings), an empty temp working directory, `DOTNET_GCHeapHardLimit = Scripting:Worker:HeapLimitBytes` + (512 MiB), a Windows Job Object (kill-on-close, one process, a commit backstop), a random current-user-only pipe + with a launch nonce in its environment, and no Akka / gRPC assemblies in its folder. It exits on pipe end of stream + or when its parent pid disappears. Results cross as protobuf `Variant`s carrying the exact CLR type, so + `(short)ctx.GetTag(x).Value` behaves as in process; an input or result type with no tag (a `List`, a `Guid`) + is an explicit `unsupported type: …` failure rather than a conversion — a behaviour change from in process. +- **Deployment.** The Host build copies the worker into `script-worker/` beside `OtOpcUa.Host.dll` + (`ScriptWorker.Deploy.targets`, Build and Publish); `Scripting:Worker:ExecutablePath` overrides it. It is run through + the dotnet muxer, so it needs a framework-dependent host (the docker-dev image). Not yet done (lmxopcua#780 P1 + remainder): the PE cache (`Scripting:Worker:PeCacheDir` is validated — absolute, outside the content root — but not + used, so a worker restart recompiles), a self-contained worker for the Windows installs, the low-privilege worker + account and firewall rule, the runner-state gauge / readiness component, and the scripted-alarm seam. + +Pinned by `tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests` (`Category=ScriptWorker`: real worker processes). + #### Quarantine: what a client sees (status code) A quarantined script's output reads **`BadInternalError`** on OPC UA, the same code as a script that diff --git a/docs/plans/2026-11-09-out-of-process-script-runner-design.md b/docs/plans/2026-11-09-out-of-process-script-runner-design.md index 9bccf3c4..6e851c80 100644 --- a/docs/plans/2026-11-09-out-of-process-script-runner-design.md +++ b/docs/plans/2026-11-09-out-of-process-script-runner-design.md @@ -665,6 +665,18 @@ per-shard state, restarts in the last hour and the quarantine list. Quarantines - Default stays `InProcess`. Enable on the docker-dev rig only. - Gate: §13 runaway / containment tests green on Linux (docker-dev) **and** Windows (windev box, which has the .NET 10 toolchain). + - **Partly landed in lmxopcua#780 (P1 slice):** `Scripting:Runner` / `Scripting:Runner:Seams` / `Scripting:Worker:*` + (bound, clamped, `PeCacheDir` path rule fail-closed), `IScriptRunner` + `InProcessScriptRunner`, the protobuf frame + protocol (`Core.Scripting.Protocol`), the `ScriptWorker` app (compile in the worker, P-thread scheduler, watchdog, + heartbeats, log forwarding, pipe-EOF / parent-pid exit, forbidden-assembly self-check), `OutOfProcessScriptRunner` + (K shards + isolation worker, soft/hard deadlines, attribution, kill-strike quarantine, backoff, fail closed, + Job Object on Windows), the VirtualTag and Calculation seams over it, and `BadResourceUnavailable` for + runner-unavailable / worker-restarted (ScadaBridge#112). The §13 containment cases run on macOS and Windows + (windev). Deviations: supervision is a plain object owned by DI, not `ScriptRunnerActor`/`ScriptWorkerActor`; + the per-seam batch submit (P0) is still one-at-a-time. Remaining: the HMAC'd PE cache, a self-contained worker + publish + the low-privilege account / pipe ACL / firewall rule (installer), the runner-state gauge and readiness + component, victim re-evaluation on shard return, isolation-worker idle stop, the orphan sweep by nonce file, the + scripted-alarm seam, the docker-dev enablement and the §8 perf harness. 3. **P2, shadow mode on docker-dev and QA.** - `Shadow` runs the **in-process** evaluation as the published result, and *also* submits to the worker. - It compares value, status class and failure kind per evaluation, and counts diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Commons/Engines/IVirtualTagEvaluator.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Commons/Engines/IVirtualTagEvaluator.cs index 28cea959..67f43a1b 100644 --- a/src/Core/ZB.MOM.WW.OtOpcUa.Commons/Engines/IVirtualTagEvaluator.cs +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Commons/Engines/IVirtualTagEvaluator.cs @@ -87,6 +87,19 @@ public sealed record VirtualTagEvalResult(bool Success, object? Value, string? R /// Description of the failure cause. /// A failed evaluation result. public static VirtualTagEvalResult Failure(string reason) => new(false, null, reason); + + /// + /// The OPC UA status a publishes. Null (every failure the in-process path produces) means + /// the seam's long-standing BadInternalError; the out-of-process runner sets + /// when the runner, not the script, failed (lmxopcua#780). + /// + public uint? StatusCode { get; init; } + + /// A failure that is the runner's, not the script's: published BadResourceUnavailable. + /// Description of the failure cause. + /// A failed evaluation result carrying . + public static VirtualTagEvalResult Unavailable(string reason) => + new(false, null, reason) { StatusCode = OpcUa.OpcUaStatus.BadResourceUnavailable }; } /// diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/OpcUaStatus.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/OpcUaStatus.cs index ed65b70c..16fa6d7a 100644 --- a/src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/OpcUaStatus.cs +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Commons/OpcUa/OpcUaStatus.cs @@ -47,6 +47,12 @@ public static class OpcUaStatus /// see docs/VirtualTags.md §"Quarantine: what a client sees"). public const uint BadInternalError = 0x80020000; + /// The operating system / runtime could not provide a resource. The out-of-process script runner's code + /// for "the script did not run and it is not the script's fault" — its worker restarted under it, or no worker + /// was ready in time (lmxopcua#780, design §9). Distinct from so an operator, and + /// ScadaBridge, can tell a sick node from a broken script. Only produced under Scripting:Runner=OutOfProcess. + public const uint BadResourceUnavailable = 0x80040000; + /// Part 4 severity bucket of : the top two bits. /// The status code to classify. /// 0 for Good, 1 for Uncertain, 2 or 3 for Bad. diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/OutOfProcessScriptRunner.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/OutOfProcessScriptRunner.cs new file mode 100644 index 00000000..e3e99528 --- /dev/null +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/OutOfProcessScriptRunner.cs @@ -0,0 +1,505 @@ +using System.Collections.Concurrent; +using System.Security.Cryptography; +using System.Text; +using Serilog; +using Serilog.Core; +using Serilog.Events; +using ZB.MOM.WW.OtOpcUa.Commons.Engines; +using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol; +using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1; + +namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess; + +/// +/// The pre-emptible (lmxopcua#780, design §3–§9): compiled scripts run in a pool of +/// ScriptWorker processes — K shards placed by a stable hash of the script key, plus an isolation worker +/// for suspects — so a script that overruns its hard deadline is stopped by killing its worker, and an OOM, a +/// native crash or a fail-fast takes down a worker instead of the server. +/// +/// +/// +/// Two deadlines. At the soft deadline T the caller gets today's timeout failure (and the +/// run token is cancelled inside the worker). At T + G () +/// the worker is killed and the key takes a kill strike. +/// Attribution. A deadline kill blames exactly one key; the other evaluations in flight on that +/// worker are victims and report (no strike). A crash with +/// no deadline crossed makes every key in flight a suspect, and suspects run in the isolation worker one at a +/// time, where a crash names exactly one key. A suspect returns to its shard after +/// clean runs. +/// Quarantine lives here, in the host, so it survives worker restarts: kill strikes (within an +/// hour), consecutive soft timeouts, or one allocation-budget breach. Reasons keep the +/// , and a source change or lifts them. +/// Fail closed. When no worker is ready within +/// the result is — never an in-process run. +/// +/// +public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable, IDisposable +{ + /// The worker entry assembly's file name. + public const string WorkerAssemblyFileName = "ZB.MOM.WW.OtOpcUa.ScriptWorker.dll"; + + /// The folder under the host's base directory the Host build places the worker in. + public const string WorkerFolderName = "script-worker"; + + /// Failure reason when the worker could not take more work. + public const string WorkerBusyReason = "script worker busy"; + + private static readonly TimeSpan KillStrikeWindow = TimeSpan.FromHours(1); + + private readonly ScriptingOptions _options; + private readonly ScriptWorkerOptions _worker; + private readonly ILogger _log; + private readonly ILogger _scriptRoot; + private readonly WorkerShard[] _shards; + private readonly WorkerShard _isolation; + private readonly ConcurrentDictionary _keys = new(StringComparer.Ordinal); + private readonly ConcurrentDictionary _compileFailures = new(StringComparer.Ordinal); + private int _disposed; + + /// Initializes the runner. No process starts until the first evaluation. + /// The node's Scripting options (sanitized again here). + /// Host diagnostics: starts, kills, quarantines; null ⇒ Serilog's global logger. + /// Fallback logger for forwarded ctx.Logger lines; null ⇒ . + public OutOfProcessScriptRunner(ScriptingOptions options, ILogger? log = null, ILogger? scriptRoot = null) + : this(options, log, scriptRoot, heartbeatInterval: TimeSpan.FromSeconds(1), loadTimeout: TimeSpan.FromSeconds(60)) + { + } + + internal OutOfProcessScriptRunner(ScriptingOptions options, ILogger? log, ILogger? scriptRoot, + TimeSpan heartbeatInterval, TimeSpan loadTimeout) + { + ArgumentNullException.ThrowIfNull(options); + _options = options.Sanitized(out _); + _worker = _options.Worker; + _log = log ?? Log.Logger; + _scriptRoot = scriptRoot ?? _log; + LoadTimeout = loadTimeout; + WorkerPath = ResolveWorkerPath(_worker); + + var evalThreads = Math.Max(2, Environment.ProcessorCount / _worker.Shards); + _shards = new WorkerShard[_worker.Shards]; + for (var i = 0; i < _shards.Length; i++) + { + var label = i.ToString(System.Globalization.CultureInfo.InvariantCulture); + _shards[i] = NewShard(label, isolation: false, evalThreads, heartbeatInterval); + } + _isolation = NewShard("iso", isolation: true, evalThreads: 1, heartbeatInterval); + } + + /// The worker entry assembly this runner launches. + public string WorkerPath { get; } + + /// How long a compile in the worker may take before the worker is killed. + internal TimeSpan LoadTimeout { get; } + + /// The shard workers (tests). + internal IReadOnlyList Shards => _shards; + + /// The isolation worker (tests). + internal WorkerShard Isolation => _isolation; + + /// Scoped keys currently treated as suspects (tests / diagnostics). + internal IReadOnlyCollection SuspectKeys => + _keys.Where(kv => kv.Value.Suspect).Select(kv => kv.Key).ToArray(); + + /// The shard a key is placed on when it is not a suspect: a stable FNV-1a hash, so placement survives restarts. + /// The scoped key. + /// K. + /// The shard index. + internal static int ShardIndex(string scopedKey, int shardCount) + { + var hash = 2166136261u; + foreach (var b in Encoding.UTF8.GetBytes(scopedKey)) + { + hash ^= b; + hash *= 16777619u; + } + return (int)(hash % (uint)shardCount); + } + + /// The default worker path, or the configured one. + /// The worker options. + /// The absolute worker path. + public static string ResolveWorkerPath(ScriptWorkerOptions worker) + => string.IsNullOrWhiteSpace(worker.ExecutablePath) + ? Path.Combine(AppContext.BaseDirectory, WorkerFolderName, WorkerAssemblyFileName) + : Path.GetFullPath(worker.ExecutablePath); + + /// + public async Task RunAsync(ScriptInvocation invocation, CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(invocation); + if (Volatile.Read(ref _disposed) != 0) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.RunnerUnavailableReason); + + var key = InProcessScriptRunner.ScopedKey(invocation.Kind, invocation.Key); + var state = Admit(key, invocation.Source); + if (state.QuarantineReason is { } quarantined) return ScriptRunOutcome.Failed(quarantined); + + var sha = SHA256.HashData(Encoding.UTF8.GetBytes(invocation.Source)); + var failureKey = InProcessScriptRunner.ScopedKey(invocation.Kind, Convert.ToHexString(sha)); + if (_compileFailures.TryGetValue(failureKey, out var knownFailure)) return ScriptRunOutcome.Failed(knownFailure); + + if (Interlocked.CompareExchange(ref state.InFlight, 1, 0) != 0) + { + ScriptRunGuard.RefusedCounter.Add(1, new KeyValuePair("script", key)); + return ScriptRunOutcome.Failed(ScriptRunGuard.PreviousRunStillRunningReason); + } + + var handedOff = false; + SemaphoreSlim? gate = null; + try + { + state.ScriptLogger = invocation.ScriptLogger; + var shard = state.Suspect ? _isolation : _shards[ShardIndex(key, _shards.Length)]; + if (shard.IsolationGate is { } isolationGate) + { + if (!await isolationGate.WaitAsync(_worker.UnavailableGrace, ct).ConfigureAwait(false)) + return ScriptRunOutcome.Unavailable(ScriptRunOutcome.RunnerUnavailableReason); + gate = isolationGate; + } + + var connection = await shard.GetReadyAsync(_worker.UnavailableGrace, ct).ConfigureAwait(false); + if (connection is null) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.RunnerUnavailableReason); + + var paths = invocation.Inputs.Keys.Order(StringComparer.Ordinal).ToArray(); + WorkerConnection.LoadEntry? entry; + try + { + entry = await connection + .EnsureLoadedAsync(key, WorkerShard.ToWire(invocation.Kind), invocation.Source, sha, paths) + .WaitAsync(LoadTimeout, ct).ConfigureAwait(false); + } + catch (TimeoutException) + { + // A compile that never finishes is the script's fault (a pathological source): the worker dies, the + // key takes a kill strike through the exit handler. + _log.Warning("Script {ScriptKey}: compile in ScriptWorker[{Shard}] exceeded {Timeout} s; killing the worker", + key, shard.Label, LoadTimeout.TotalSeconds); + connection.Kill(WorkerExitReason.KilledLoadTimeout, key); + return ScriptRunOutcome.Failed($"compile failure: compile did not finish within {LoadTimeout.TotalSeconds:F0} s"); + } + if (entry?.Outcome is not { } outcome) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason); + if (!outcome.Ok) + { + var reason = outcome.Error?.Kind switch + { + ScriptError.Types.Kind.Compile => $"compile error: {outcome.Error.Message}", + ScriptError.Types.Kind.Sandbox => $"sandbox violation: {outcome.Error.Message}", + _ => $"compile failure: {outcome.Error?.Message}", + }; + // Remembered until the next deploy boundary, so a broken script never re-enters Roslyn (design §7). + _compileFailures.TryAdd(failureKey, reason); + return ScriptRunOutcome.Failed(reason); + } + + var pending = connection.Submit(key, BuildRequest(key, entry, invocation)); + ArmHardDeadline(connection, pending, invocation.Timeout + _worker.EffectiveHardGrace(invocation.Timeout)); + + // From here the in-flight slot (and the isolation gate) belong to the evaluation itself: they are released + // when the worker answers or dies — not when this caller stops waiting at the soft deadline. That keeps a + // key single-flight between T and T + G, and the isolation worker strictly one-at-a-time. + handedOff = true; + var heldGate = gate; + _ = pending.Completion.ContinueWith(_ => + { + Interlocked.Exchange(ref state.InFlight, 0); + heldGate?.Release(); + }, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default); + + EvalResult? result; + try + { + result = await pending.Completion.WaitAsync(invocation.Timeout, ct).ConfigureAwait(false); + } + catch (TimeoutException) + { + // As in process: the run token is cancelled only once the caller has its timeout, so a polling script + // ends itself (and frees its slot) without the worker being killed. + connection.Cancel(pending); + if (ct.IsCancellationRequested) return ScriptRunOutcome.Failed("evaluation cancelled"); + if (NoteTimeout(key, state, invocation.Timeout)) return ScriptRunOutcome.Failed(state.QuarantineReason!); + return ScriptRunOutcome.Failed($"{ScriptEvalFailureReasons.TimedOutPrefix}{invocation.Timeout.TotalSeconds:F1}s"); + } + + if (result is null) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason); + return Complete(key, state, result); + } + catch (OperationCanceledException) when (ct.IsCancellationRequested) + { + return ScriptRunOutcome.Failed("evaluation cancelled"); + } + finally + { + if (!handedOff) + { + Interlocked.Exchange(ref state.InFlight, 0); + gate?.Release(); + } + } + } + + /// + public void Reset(ScriptContextKind kind) + { + var prefix = InProcessScriptRunner.ScopedKey(kind, string.Empty); + var dropped = new List(); + foreach (var key in _keys.Keys) + { + if (key.StartsWith(prefix, StringComparison.Ordinal) && _keys.TryRemove(key, out _)) dropped.Add(key); + } + foreach (var key in _compileFailures.Keys) + { + if (key.StartsWith(prefix, StringComparison.Ordinal)) _compileFailures.TryRemove(key, out _); + } + foreach (var shard in _shards.Append(_isolation)) + shard.Current?.Unload(dropped); + } + + /// + public IReadOnlyCollection QuarantinedKeys(ScriptContextKind kind) + { + var prefix = InProcessScriptRunner.ScopedKey(kind, string.Empty); + return _keys + .Where(kv => kv.Value.QuarantineReason is not null && kv.Key.StartsWith(prefix, StringComparison.Ordinal)) + .Select(kv => kv.Key[prefix.Length..]) + .ToArray(); + } + + /// Stops every worker. + /// A task that completes when they have ended. + public async ValueTask DisposeAsync() + { + if (Interlocked.Exchange(ref _disposed, 1) != 0) return; + await Task.WhenAll(_shards.Append(_isolation).Select(s => s.DisposeAsync().AsTask())).ConfigureAwait(false); + } + + /// Stops every worker (blocking). + public void Dispose() => DisposeAsync().AsTask().GetAwaiter().GetResult(); + + private WorkerShard NewShard(string label, bool isolation, int evalThreads, TimeSpan heartbeatInterval) + { + var spec = new WorkerLaunchSpec( + Shard: label, + WorkerPath: WorkerPath, + HeapLimitBytes: _worker.HeapLimitBytes, + EvalThreads: evalThreads, + HardGrace: _worker.HardGrace ?? TimeSpan.FromSeconds(Math.Max(1, _options.RunTimeout.TotalSeconds)), + HeartbeatInterval: heartbeatInterval, + HandshakeTimeout: TimeSpan.FromSeconds(15)); + return new WorkerShard(label, isolation, + ct => WorkerConnection.StartAsync(spec, _log, OnLogs, ct), + OnWorkerExit, _log); + } + + private KeyState Admit(string key, string source) + { + while (true) + { + var state = _keys.GetOrAdd(key, static (_, src) => new KeyState(src), source); + if (string.Equals(state.Source, source, StringComparison.Ordinal)) return state; + // The script changed under this key: a clean slate, exactly as the in-process guard does. An evaluation + // of the old source still in flight releases the OLD state's slot when it ends. + if (_keys.TryUpdate(key, new KeyState(source), state)) continue; + } + } + + private EvalRequest BuildRequest(string key, WorkerConnection.LoadEntry entry, ScriptInvocation invocation) + { + var request = new EvalRequest + { + Key = key, + NowUtcTicks = DateTime.UtcNow.Ticks, + SoftTimeoutMs = (uint)Math.Clamp(invocation.Timeout.TotalMilliseconds, 1, uint.MaxValue), + HardGraceMs = (uint)Math.Clamp(_worker.EffectiveHardGrace(invocation.Timeout).TotalMilliseconds, 1, uint.MaxValue), + }; + foreach (var (path, snapshot) in invocation.Inputs) + { + if (snapshot is null) continue; + var input = new Input + { + StatusCode = snapshot.StatusCode, + SourceTsTicks = snapshot.SourceTimestampUtc is { } source ? VariantCodec.ToUtc(source).Ticks : 0, + ServerTsTicks = VariantCodec.ToUtc(snapshot.ServerTimestampUtc).Ticks, + }; + if (entry.Slots.TryGetValue(path, out var slot)) input.Slot = slot; + else input.Path = path; + if (VariantCodec.TryEncode(snapshot.Value, out var variant, out var unsupported)) + { + input.Value = variant; + } + else + { + // Never silently converted (design §14 #11): the worker answers UNSUPPORTED_TYPE for this input. + input.Unsupported = true; + input.UnsupportedType = unsupported ?? "?"; + } + request.Inputs.Add(input); + } + return request; + } + + private void ArmHardDeadline(WorkerConnection connection, PendingEval pending, TimeSpan after) + { + var cts = new CancellationTokenSource(after); + var registration = cts.Token.Register(() => + { + if (pending.Completion.IsCompleted) return; + _log.Warning("Script {ScriptKey} exceeded its hard deadline ({Deadline} ms); killing ScriptWorker[{Shard}] pid {Pid}", + pending.Key, (long)after.TotalMilliseconds, connection.Shard, connection.Pid); + connection.Kill(WorkerExitReason.KilledDeadline, pending.Key); + }); + _ = pending.Completion.ContinueWith(_ => + { + registration.Dispose(); + cts.Dispose(); + }, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default); + } + + private ScriptRunOutcome Complete(string key, KeyState state, EvalResult result) + { + // The script returned control (a value or a throw): the soft-timeout streak resets, as in process. + Interlocked.Exchange(ref state.ConsecutiveTimeouts, 0); + + if (_options.MaxAllocBytesPerRun > 0 && (long)result.AllocatedBytes > _options.MaxAllocBytesPerRun) + { + Quarantine(key, state, + $"one evaluation allocated {result.AllocatedBytes / (1024.0 * 1024.0):F1} MB, over the " + + $"{_options.MaxAllocBytesPerRun / (1024.0 * 1024.0):F0} MB budget (Scripting:MaxAllocBytesPerRun); " + + "the script is not evaluated again until its source changes or it is redeployed", + "allocation"); + return ScriptRunOutcome.Failed(state.QuarantineReason!); + } + + if (state.Suspect && Interlocked.Increment(ref state.ProbationRuns) >= _worker.IsolationProbationRuns) + { + state.Suspect = false; + _log.Information("Script {ScriptKey} completed {Runs} clean run(s) in the isolation worker; returning it to its shard", + key, _worker.IsolationProbationRuns); + } + + switch (result.OutcomeCase) + { + case EvalResult.OutcomeOneofCase.Value: + try + { + return ScriptRunOutcome.Ok(VariantCodec.Decode(result.Value)); + } + catch (ScriptWorkerProtocolException ex) + { + return ScriptRunOutcome.Failed($"script worker returned an undecodable value: {ex.Message}"); + } + case EvalResult.OutcomeOneofCase.Predicate: + return ScriptRunOutcome.Ok(result.Predicate); + case EvalResult.OutcomeOneofCase.Error: + return result.Error.Kind switch + { + ScriptError.Types.Kind.Threw => ScriptRunOutcome.Failed($"script threw: {result.Error.Message}"), + ScriptError.Types.Kind.UnsupportedType => ScriptRunOutcome.Failed($"unsupported type: {result.Error.Message}"), + ScriptError.Types.Kind.Busy => ScriptRunOutcome.Failed(WorkerBusyReason), + ScriptError.Types.Kind.NotLoaded => ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason), + _ => ScriptRunOutcome.Failed($"script failed: {result.Error.Message}"), + }; + default: + return ScriptRunOutcome.Ok(null); + } + } + + private bool NoteTimeout(string key, KeyState state, TimeSpan timeout) + { + var count = Interlocked.Increment(ref state.ConsecutiveTimeouts); + if (count < _options.QuarantineAfterConsecutiveTimeouts) return false; + Quarantine(key, state, + $"{count} consecutive evaluation(s) timed out (budget {timeout.TotalMilliseconds:F0} ms); " + + "the script is not evaluated again until its source changes or it is redeployed", + "timeouts"); + return true; + } + + private void OnWorkerExit(WorkerShard shard, WorkerExit exit) + { + switch (exit.Reason) + { + case WorkerExitReason.KilledDeadline when exit.CulpritKey is { } culprit: + KillStrike(culprit, "it exceeded its hard deadline and its worker was killed", "killed"); + break; + case WorkerExitReason.KilledLoadTimeout when exit.CulpritKey is { } culprit: + KillStrike(culprit, "its compile did not finish and its worker was killed", "killed"); + break; + case WorkerExitReason.Crash or WorkerExitReason.KilledHeartbeat or WorkerExitReason.ProtocolFault: + if (shard.IsIsolation) + { + // One evaluation at a time here: the crash is this key's (design §5.3.3). + foreach (var key in exit.InFlightKeys) + KillStrike(key, "it crashed the isolation worker", "isolated_crash"); + } + else if (exit.InFlightKeys.Count > 0) + { + foreach (var key in exit.InFlightKeys) + { + if (!_keys.TryGetValue(key, out var state)) continue; + state.Suspect = true; + Interlocked.Exchange(ref state.ProbationRuns, 0); + } + _log.Warning("ScriptWorker[{Shard}] ended ({Reason}) with no deadline crossed; {Count} script(s) in flight move to the isolation worker: {Suspects}", + shard.Label, exit.Reason, exit.InFlightKeys.Count, string.Join(", ", exit.InFlightKeys)); + } + break; + } + } + + private void KillStrike(string key, string why, string metricReason) + { + if (!_keys.TryGetValue(key, out var state)) return; + int strikes; + lock (state.KillStrikes) + { + var now = DateTime.UtcNow; + state.KillStrikes.Add(now); + state.KillStrikes.RemoveAll(t => now - t > KillStrikeWindow); + strikes = state.KillStrikes.Count; + } + if (strikes >= _worker.QuarantineAfterKills) + { + Quarantine(key, state, + $"{why} ({strikes} time(s) in the last hour); the script is not evaluated again until its source changes or it is redeployed", + metricReason); + } + } + + private void Quarantine(string key, KeyState state, string why, string metricReason) + { + var reason = ScriptRunGuard.QuarantinedReasonPrefix + why; + if (Interlocked.CompareExchange(ref state.QuarantineReason, reason, null) is not null) return; + ScriptRunGuard.QuarantinedCounter.Add(1, + new KeyValuePair("script", key), + new KeyValuePair("reason", metricReason)); + _scriptRoot.ForContext(ScriptLoggerFactory.ScriptNameProperty, key).Error("Script {ScriptKey} quarantined: {Reason}", key, why); + } + + private void OnLogs(LogBatch batch) + { + foreach (var line in batch.Lines) + { + var logger = _keys.TryGetValue(line.Key, out var state) && state.ScriptLogger is { } scoped ? scoped : _scriptRoot; + var level = line.Level <= (uint)LogEventLevel.Fatal ? (LogEventLevel)line.Level : LogEventLevel.Information; + if (string.IsNullOrEmpty(line.Exception)) + logger.Write(level, "{ScriptMessage:l}", line.Rendered); + else + logger.Write(level, "{ScriptMessage:l}{NewLine}{ScriptException:l}", line.Rendered, Environment.NewLine, line.Exception); + } + if (batch.Dropped > 0) + _log.Warning("ScriptWorker dropped {Dropped} script log line(s): a script is logging faster than the pipe drains", batch.Dropped); + } + + private sealed class KeyState(string source) + { + public readonly string Source = source; + public readonly List KillStrikes = []; + public int InFlight; + public int ConsecutiveTimeouts; + public int ProbationRuns; + public volatile bool Suspect; + public string? QuarantineReason; + public ILogger? ScriptLogger; + } +} diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/WindowsJobObject.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/WindowsJobObject.cs new file mode 100644 index 00000000..c44d6829 --- /dev/null +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/WindowsJobObject.cs @@ -0,0 +1,114 @@ +using System.ComponentModel; +using System.Diagnostics; +using System.Runtime.InteropServices; +using System.Runtime.Versioning; +using Microsoft.Win32.SafeHandles; + +namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess; + +/// +/// A Windows Job Object holding one ScriptWorker (design §3.3, §5.2): KILL_ON_JOB_CLOSE so the kernel kills +/// the worker when the host dies (its handle closes with it), ACTIVE_PROCESS = 1 so the worker can start no +/// child process, and a process-memory backstop slightly above the managed heap limit for native and JIT memory. +/// +[SupportedOSPlatform("windows")] +internal sealed class WindowsJobObject : IDisposable +{ + private const int JobObjectExtendedLimitInformation = 9; + private const uint JOB_OBJECT_LIMIT_ACTIVE_PROCESS = 0x00000008; + private const uint JOB_OBJECT_LIMIT_PROCESS_MEMORY = 0x00000100; + private const uint JOB_OBJECT_LIMIT_DIE_ON_UNHANDLED_EXCEPTION = 0x00000400; + private const uint JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE = 0x00002000; + + private readonly SafeFileHandle _handle; + + private WindowsJobObject(SafeFileHandle handle) => _handle = handle; + + /// Creates a job with the worker limits and assigns to it. + /// The freshly started worker. + /// The per-process commit limit. + /// The job; dispose it to kill the worker. + public static WindowsJobObject Assign(Process process, long processMemoryLimitBytes) + { + var handle = CreateJobObjectW(IntPtr.Zero, null); + if (handle.IsInvalid) throw new Win32Exception(Marshal.GetLastWin32Error(), "CreateJobObject failed"); + var job = new WindowsJobObject(handle); + try + { + var info = new JOBOBJECT_EXTENDED_LIMIT_INFORMATION + { + BasicLimitInformation = new JOBOBJECT_BASIC_LIMIT_INFORMATION + { + LimitFlags = JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE | JOB_OBJECT_LIMIT_ACTIVE_PROCESS + | JOB_OBJECT_LIMIT_PROCESS_MEMORY | JOB_OBJECT_LIMIT_DIE_ON_UNHANDLED_EXCEPTION, + ActiveProcessLimit = 1, + }, + ProcessMemoryLimit = (UIntPtr)(ulong)processMemoryLimitBytes, + }; + if (!SetInformationJobObject(handle, JobObjectExtendedLimitInformation, ref info, + (uint)Marshal.SizeOf())) + throw new Win32Exception(Marshal.GetLastWin32Error(), "SetInformationJobObject failed"); + if (!AssignProcessToJobObject(handle, process.Handle)) + throw new Win32Exception(Marshal.GetLastWin32Error(), "AssignProcessToJobObject failed"); + return job; + } + catch + { + job.Dispose(); + throw; + } + } + + /// Closes the job handle, which kills every process still in it. + public void Dispose() => _handle.Dispose(); + + [StructLayout(LayoutKind.Sequential)] + private struct JOBOBJECT_BASIC_LIMIT_INFORMATION + { + public long PerProcessUserTimeLimit; + public long PerJobUserTimeLimit; + public uint LimitFlags; + public UIntPtr MinimumWorkingSetSize; + public UIntPtr MaximumWorkingSetSize; + public uint ActiveProcessLimit; + public UIntPtr Affinity; + public uint PriorityClass; + public uint SchedulingClass; + } + + [StructLayout(LayoutKind.Sequential)] + private struct IO_COUNTERS + { + public ulong ReadOperationCount; + public ulong WriteOperationCount; + public ulong OtherOperationCount; + public ulong ReadTransferCount; + public ulong WriteTransferCount; + public ulong OtherTransferCount; + } + + [StructLayout(LayoutKind.Sequential)] + private struct JOBOBJECT_EXTENDED_LIMIT_INFORMATION + { + public JOBOBJECT_BASIC_LIMIT_INFORMATION BasicLimitInformation; + public IO_COUNTERS IoInfo; + public UIntPtr ProcessMemoryLimit; + public UIntPtr JobMemoryLimit; + public UIntPtr PeakProcessMemoryUsed; + public UIntPtr PeakJobMemoryUsed; + } + +#pragma warning disable SYSLIB1054 // DllImport kept: LibraryImport would need AllowUnsafeBlocks for three calls. + [DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)] + private static extern SafeFileHandle CreateJobObjectW(IntPtr securityAttributes, string? name); + + [DllImport("kernel32.dll", SetLastError = true)] + [return: MarshalAs(UnmanagedType.Bool)] + private static extern bool SetInformationJobObject(SafeFileHandle job, int infoClass, + ref JOBOBJECT_EXTENDED_LIMIT_INFORMATION info, uint length); + + [DllImport("kernel32.dll", SetLastError = true)] + [return: MarshalAs(UnmanagedType.Bool)] + private static extern bool AssignProcessToJobObject(SafeFileHandle job, IntPtr process); +#pragma warning restore SYSLIB1054 +} diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/WorkerConnection.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/WorkerConnection.cs new file mode 100644 index 00000000..4b456f00 --- /dev/null +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/WorkerConnection.cs @@ -0,0 +1,699 @@ +using System.Collections.Concurrent; +using System.Diagnostics; +using System.IO.Pipes; +using System.Security.Cryptography; +using System.Text; +using System.Threading.Channels; +using Serilog; +using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol; +using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1; + +namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess; + +/// How a worker process ended. +internal enum WorkerExitReason +{ + /// It exited by itself with code 0 (a Shutdown). + Clean, + + /// The host killed it: a script crossed its hard deadline (). + KilledDeadline, + + /// The host killed it: no frame for three heartbeat intervals. + KilledHeartbeat, + + /// The host killed it: a compile did not finish in time (). + KilledLoadTimeout, + + /// It died by itself (OOM, heap limit, fail-fast, external kill). + Crash, + + /// A framing or handshake violation. + ProtocolFault, + + /// The runner is shutting down. + Disposed, +} + +/// The end of one worker process. +/// Why it ended. +/// The scoped key the host blamed, for a deadline or load kill. +/// The OS exit code, when known. +/// The scoped keys that had an evaluation outstanding when it ended. +internal sealed record WorkerExit(WorkerExitReason Reason, string? CulpritKey, int? ExitCode, IReadOnlyList InFlightKeys); + +/// What the host needs to launch one worker. +/// The shard label (0, 1, … or iso). +/// The worker entry assembly (.dll, run through the dotnet muxer) or apphost. +/// The worker's DOTNET_GCHeapHardLimit. +/// The worker scheduler's width (1 for the isolation worker). +/// The worker watchdog's grace past the soft deadline. +/// How often the worker sends a heartbeat. +/// How long a launch may take to reach Ready. +internal sealed record WorkerLaunchSpec( + string Shard, + string WorkerPath, + long HeapLimitBytes, + int EvalThreads, + TimeSpan HardGrace, + TimeSpan HeartbeatInterval, + TimeSpan HandshakeTimeout); + +/// +/// One live ScriptWorker process and its pipe (lmxopcua#780, design §3–§6): launch with a cleared, allow-listed +/// environment, a random pipe that accepts exactly one client, a nonce handshake, a frame reader and a coalescing +/// writer, heartbeat supervision, and the kill. It never restarts itself; does that. +/// +internal sealed class WorkerConnection : IAsyncDisposable +{ + /// + /// Environment variables a worker inherits from the host (design §6): the runtime's own knobs, the temp + /// directories and the minimum a Windows process needs. Everything else — notably ServerHistorian__ApiKey, + /// ConfigDb connection strings and every other Section__Key override — is dropped. + /// + internal static readonly IReadOnlyList InheritedEnvironment = + [ + "PATH", "TEMP", "TMP", "TMPDIR", "HOME", "SystemRoot", "SYSTEMROOT", "windir", "SystemDrive", "ComSpec", + "PATHEXT", "NUMBER_OF_PROCESSORS", "PROCESSOR_ARCHITECTURE", "LOCALAPPDATA", + ]; + + private readonly Process _process; + private readonly NamedPipeServerStream _pipe; + private readonly IDisposable? _job; + private readonly string _workDir; + private readonly ILogger _log; + private readonly Action _onLogs; + private readonly Channel _outbound = Channel.CreateUnbounded(new UnboundedChannelOptions { SingleReader = true }); + private readonly ConcurrentDictionary _pending = new(); + private readonly ConcurrentDictionary> _loads = new(); + private readonly ConcurrentDictionary _loaded = new(StringComparer.Ordinal); + private readonly TaskCompletionSource _exited = new(TaskCreationOptions.RunContinuationsAsynchronously); + private readonly CancellationTokenSource _stopping = new(); + private readonly TimeSpan _heartbeatInterval; + private TaskCompletionSource? _diag; + private (WorkerExitReason Reason, string? Culprit)? _killedFor; + private long _lastFrameTimestamp; + private long _seq; + private long _nextEvalId; + private long _nextLoadId; + private int _maxFrameBytes = ScriptWorkerFrameCodec.DefaultMaxFrameBytes; + private int _disposed; + + private WorkerConnection(string shard, Process process, NamedPipeServerStream pipe, IDisposable? job, string workDir, + ILogger log, Action onLogs, TimeSpan heartbeatInterval) + { + Shard = shard; + _process = process; + _pipe = pipe; + _job = job; + _workDir = workDir; + _log = log; + _onLogs = onLogs; + _heartbeatInterval = heartbeatInterval; + Pid = process.Id; + } + + /// The shard label. + public string Shard { get; } + + /// The worker's process id. + public int Pid { get; } + + /// Completes when the worker has ended, with why. + public Task Exited => _exited.Task; + + /// Whether the worker is still serving. + public bool IsAlive => !_exited.Task.IsCompleted && !_stopping.IsCancellationRequested; + + /// when the handshake completed. + public long ReadyTimestamp { get; private set; } + + /// Evaluations currently outstanding on this worker. + public int InFlightCount => _pending.Count; + + /// Launches a worker and completes the handshake. + /// What to launch. + /// Host diagnostics (stderr lines, exits). + /// Receives the worker's forwarded ctx.Logger batches. + /// Cancellation. + /// A ready connection. + public static async Task StartAsync(WorkerLaunchSpec spec, ILogger log, Action onLogs, CancellationToken ct) + { + var pipeName = "otopcua-sw-" + Convert.ToHexString(RandomNumberGenerator.GetBytes(12)).ToLowerInvariant(); + var nonce = Convert.ToHexString(RandomNumberGenerator.GetBytes(32)); + var workDir = Path.Combine(Path.GetTempPath(), "otopcua-scriptworker", pipeName); + Directory.CreateDirectory(workDir); + + // One instance, current user only: the pipe accepts exactly one connection, from this account. + var pipe = new NamedPipeServerStream(pipeName, PipeDirection.InOut, maxNumberOfServerInstances: 1, + PipeTransmissionMode.Byte, PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly); + Process? process = null; + IDisposable? job = null; + try + { + var psi = BuildStartInfo(spec, pipeName, nonce, workDir); + process = new Process { StartInfo = psi, EnableRaisingEvents = true }; + if (!process.Start()) throw new InvalidOperationException("the worker process did not start"); + if (OperatingSystem.IsWindows()) + { + // Native and JIT memory backstop above the managed heap limit. + job = WindowsJobObject.Assign(process, spec.HeapLimitBytes + 256L * 1024 * 1024); + } + var shardLabel = spec.Shard; + process.ErrorDataReceived += (_, e) => + { + if (e.Data is { Length: > 0 } line) log.Warning("ScriptWorker[{Shard}] {WorkerStderr}", shardLabel, line); + }; + process.OutputDataReceived += (_, e) => + { + if (e.Data is { Length: > 0 } line) log.Debug("ScriptWorker[{Shard}] stdout {WorkerStdout}", shardLabel, line); + }; + process.BeginErrorReadLine(); + process.BeginOutputReadLine(); + + using var handshakeCts = CancellationTokenSource.CreateLinkedTokenSource(ct); + handshakeCts.CancelAfter(spec.HandshakeTimeout); + var exitedEarly = process.WaitForExitAsync(handshakeCts.Token); + var accept = pipe.WaitForConnectionAsync(handshakeCts.Token); + if (await Task.WhenAny(accept, exitedEarly).ConfigureAwait(false) != accept) + throw new InvalidOperationException( + $"the worker exited (code {SafeExitCode(process)}) before connecting to its pipe"); + await accept.ConfigureAwait(false); + + var hello = new Hello + { + Nonce = nonce, + MaxFrameBytes = ScriptWorkerFrameCodec.DefaultMaxFrameBytes, + EvalThreads = (uint)spec.EvalThreads, + QueueCapacity = 4096, + HardGraceMs = (uint)Math.Clamp(spec.HardGrace.TotalMilliseconds, 1, uint.MaxValue), + HeartbeatIntervalMs = (uint)spec.HeartbeatInterval.TotalMilliseconds, + MaxLogLines = 1000, + }; + await HandshakeAsync(pipe, hello, nonce, process.Id, handshakeCts.Token).ConfigureAwait(false); + + var connection = new WorkerConnection(spec.Shard, process, pipe, job, workDir, log, onLogs, spec.HeartbeatInterval); + connection.Begin(); + return connection; + } + catch + { + try { if (process is { HasExited: false }) process.Kill(entireProcessTree: true); } catch { /* best effort */ } + job?.Dispose(); + process?.Dispose(); + await pipe.DisposeAsync().ConfigureAwait(false); + TryDeleteDirectory(workDir); + throw; + } + } + + /// + /// The host half of the handshake: send , require a Ready as the first frame, + /// echoing (constant-time compare) from . + /// + /// The accepted pipe. + /// The Hello to send. + /// The nonce put in the worker's environment. + /// The launched process id; null skips the check. + /// Cancellation (the handshake timeout). + /// The worker's Ready. + /// Anything but a matching Ready. + internal static async Task HandshakeAsync(Stream stream, Hello hello, string expectedNonce, int? expectedPid, CancellationToken ct) + { + await ScriptWorkerFrameCodec.WriteAsync(stream, new Envelope + { + ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion, + Seq = 1, + Hello = hello, + }, ScriptWorkerFrameCodec.DefaultMaxFrameBytes, ct).ConfigureAwait(false); + await stream.FlushAsync(ct).ConfigureAwait(false); + + var reply = await ScriptWorkerFrameCodec.ReadAsync(stream, ScriptWorkerFrameCodec.DefaultMaxFrameBytes, ct).ConfigureAwait(false) + ?? throw new ScriptWorkerProtocolException("the worker closed the pipe during the handshake"); + if (reply.BodyCase != Envelope.BodyOneofCase.Ready) + throw new ScriptWorkerProtocolException($"expected Ready, got {reply.BodyCase}"); + var expected = Encoding.UTF8.GetBytes(expectedNonce); + var actual = Encoding.UTF8.GetBytes(reply.Ready.Nonce ?? string.Empty); + if (!CryptographicOperations.FixedTimeEquals(expected, actual)) + throw new ScriptWorkerProtocolException("the worker's nonce does not match the launch nonce"); + if (expectedPid is { } pid && reply.Ready.Pid != pid) + throw new ScriptWorkerProtocolException($"the pipe client reports pid {reply.Ready.Pid}, not the launched {pid}"); + return reply.Ready; + } + + private static ProcessStartInfo BuildStartInfo(WorkerLaunchSpec spec, string pipeName, string nonce, string workDir) + { + var psi = new ProcessStartInfo + { + UseShellExecute = false, + CreateNoWindow = true, + RedirectStandardError = true, + RedirectStandardOutput = true, + RedirectStandardInput = false, + WorkingDirectory = workDir, + }; + if (spec.WorkerPath.EndsWith(".dll", StringComparison.OrdinalIgnoreCase)) + { + psi.FileName = ResolveMuxer(); + psi.ArgumentList.Add("exec"); + psi.ArgumentList.Add(spec.WorkerPath); + } + else + { + psi.FileName = spec.WorkerPath; + } + + // Cleared, then repopulated from an allow-list (design §6). ProcessStartInfo.Environment starts as a copy of + // the host's environment; ScadaBridge's sandbox inherits it whole, which this design rejects. + var inherited = psi.Environment.ToDictionary(kv => kv.Key, kv => kv.Value, StringComparer.Ordinal); + psi.Environment.Clear(); + foreach (var (name, value) in inherited) + { + if (value is null) continue; + if (InheritedEnvironment.Contains(name, StringComparer.OrdinalIgnoreCase) + || (name.StartsWith("DOTNET_", StringComparison.OrdinalIgnoreCase) + && !name.StartsWith("DOTNET_GCHeap", StringComparison.OrdinalIgnoreCase))) + psi.Environment[name] = value; + } + psi.Environment["DOTNET_GCHeapHardLimit"] = "0x" + spec.HeapLimitBytes.ToString("X", System.Globalization.CultureInfo.InvariantCulture); + psi.Environment["DOTNET_gcServer"] = "0"; + psi.Environment["DOTNET_CLI_TELEMETRY_OPTOUT"] = "1"; + psi.Environment[ScriptWorkerEnvironment.PipeName] = pipeName; + psi.Environment[ScriptWorkerEnvironment.Nonce] = nonce; + psi.Environment[ScriptWorkerEnvironment.ParentPid] = Environment.ProcessId.ToString(System.Globalization.CultureInfo.InvariantCulture); + psi.Environment[ScriptWorkerEnvironment.Shard] = spec.Shard; + return psi; + } + + /// The dotnet muxer that runs a framework-dependent worker .dll. + /// The muxer path, or plain dotnet (resolved on PATH) as a last resort. + internal static string ResolveMuxer() + { + var hostPath = Environment.GetEnvironmentVariable("DOTNET_HOST_PATH"); + if (!string.IsNullOrEmpty(hostPath) && File.Exists(hostPath)) return hostPath; + var processPath = Environment.ProcessPath; + if (processPath is not null + && string.Equals(Path.GetFileNameWithoutExtension(processPath), "dotnet", StringComparison.OrdinalIgnoreCase)) + return processPath; + var root = Environment.GetEnvironmentVariable("DOTNET_ROOT"); + if (!string.IsNullOrEmpty(root)) + { + var candidate = Path.Combine(root, OperatingSystem.IsWindows() ? "dotnet.exe" : "dotnet"); + if (File.Exists(candidate)) return candidate; + } + return "dotnet"; + } + + private void Begin() + { + ReadyTimestamp = Stopwatch.GetTimestamp(); + Volatile.Write(ref _lastFrameTimestamp, ReadyTimestamp); + _ = Task.Run(ReadLoopAsync); + _ = Task.Run(WriteLoopAsync); + _ = Task.Run(HeartbeatMonitorAsync); + _ = Task.Run(WatchProcessAsync); + } + + /// Ensures is compiled in this worker at . + /// The scoped key. + /// The context kind. + /// The source. + /// SHA-256 of the source. + /// The slot order. + /// The load outcome and the slot map; null when the worker ended first. + public Task EnsureLoadedAsync(string key, ContextKind kind, string source, byte[] sourceHash, IReadOnlyList inputPaths) + { + var hashHex = Convert.ToHexString(sourceHash); + if (_loaded.TryGetValue(key, out var existing) && existing.SourceHash == hashHex && existing.SameSlots(inputPaths)) + return existing.Ready; + + var entry = new LoadEntry(hashHex, inputPaths); + _loaded[key] = entry; + var loadId = (ulong)Interlocked.Increment(ref _nextLoadId); + var tcs = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + _loads[loadId] = tcs; + var load = new LoadScripts { LoadId = loadId }; + var script = new ScriptSource { Key = key, Kind = kind, Source = source, SourceSha256 = Google.Protobuf.ByteString.CopyFrom(sourceHash) }; + script.InputPaths.AddRange(inputPaths); + load.Scripts.Add(script); + if (!Send(new Envelope { Load = load })) { _loads.TryRemove(loadId, out _); entry.Complete(null); return entry.Ready; } + + _ = tcs.Task.ContinueWith(t => + { + if (t.IsCompletedSuccessfully) + { + var outcome = t.Result.Outcomes.FirstOrDefault(o => o.Key == key); + entry.Complete(outcome ?? new LoadOutcome { Key = key, Ok = false, Error = new ScriptError { Kind = ScriptError.Types.Kind.Threw, Message = "no load outcome" } }); + if (outcome is { Ok: false }) _loaded.TryRemove(new KeyValuePair(key, entry)); + } + else + { + entry.Complete(null); + _loaded.TryRemove(new KeyValuePair(key, entry)); + } + }, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default); + return entry.Ready; + } + + /// Forgets a key's compiled form here and in the worker. + /// The scoped keys. + public void Unload(IReadOnlyCollection keys) + { + var unload = new Unload(); + foreach (var key in keys) + if (_loaded.TryRemove(key, out _)) unload.Keys.Add(key); + if (unload.Keys.Count > 0) Send(new Envelope { Unload = unload }); + } + + /// Submits one evaluation. Its yields null when the worker ends first. + /// The scoped key. + /// The request (its id is assigned here). + /// The pending evaluation. + public PendingEval Submit(string key, EvalRequest request) + { + request.EvalId = (ulong)Interlocked.Increment(ref _nextEvalId); + var pending = new PendingEval(key, request.EvalId); + _pending[request.EvalId] = pending; + var batch = new EvalBatch(); + batch.Items.Add(request); + if (!Send(new Envelope { Eval = batch }) || _exited.Task.IsCompleted) + { + if (_pending.TryRemove(request.EvalId, out _)) pending.Complete(null); + } + return pending; + } + + /// Cancels a run's ctx.CancellationToken in the worker (sent at the soft deadline). + /// The evaluation. + public void Cancel(PendingEval pending) + { + if (!pending.Completion.IsCompleted) Send(new Envelope { Cancel = new Protocol.V1.Cancel { EvalId = pending.EvalId } }); + } + + /// Asks the worker for its process diagnostics (loaded assemblies, environment names). + /// How long to wait. + /// The reply. + public async Task DiagnosticsAsync(TimeSpan timeout) + { + var tcs = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + Interlocked.Exchange(ref _diag, tcs); + Send(new Envelope { DiagRequest = new DiagnosticsRequest() }); + return await tcs.Task.WaitAsync(timeout).ConfigureAwait(false); + } + + /// Kills the worker (process tree). The first reason given is the one recorded. + /// Why. + /// The scoped key blamed, if any. + public void Kill(WorkerExitReason reason, string? culpritKey) + { + lock (_pending) + { + _killedFor ??= (reason, culpritKey); + } + try + { + if (!_process.HasExited) _process.Kill(entireProcessTree: true); + } + catch (InvalidOperationException) { /* already gone */ } + catch (System.ComponentModel.Win32Exception ex) + { + _log.Warning(ex, "ScriptWorker[{Shard}] kill of pid {Pid} failed", Shard, Pid); + } + } + + /// Asks the worker to stop, waits briefly, then kills it. + /// A task that completes when the worker has ended. + public async ValueTask DisposeAsync() + { + // Both the shard (on exit) and the runner (on shutdown) dispose a connection; only the first does the work. + if (Interlocked.Exchange(ref _disposed, 1) != 0) return; + if (!_exited.Task.IsCompleted) + { + lock (_pending) { _killedFor ??= (WorkerExitReason.Disposed, null); } + Send(new Envelope { Shutdown = new Shutdown() }); + try + { + await _process.WaitForExitAsync().WaitAsync(TimeSpan.FromSeconds(1)).ConfigureAwait(false); + } + catch (TimeoutException) + { + Kill(WorkerExitReason.Disposed, null); + } + try { await _exited.Task.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false); } catch (TimeoutException) { } + } + _stopping.Cancel(); + _outbound.Writer.TryComplete(); + await _pipe.DisposeAsync().ConfigureAwait(false); + _job?.Dispose(); + _process.Dispose(); + TryDeleteDirectory(_workDir); + } + + /// Test hook: closes the host end of the pipe without touching the process, as a dying host would. + internal void SeverPipeForTest() => _pipe.Dispose(); + + private bool Send(Envelope envelope) + { + envelope.ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion; + return _outbound.Writer.TryWrite(envelope); + } + + private async Task WriteLoopAsync() + { + var reader = _outbound.Reader; + try + { + while (await reader.WaitToReadAsync(_stopping.Token).ConfigureAwait(false)) + { + // Opportunistic batching (design §4.5): every evaluation queued right now goes out as ONE EvalBatch. + EvalBatch? batch = null; + while (reader.TryRead(out var envelope)) + { + if (envelope.BodyCase == Envelope.BodyOneofCase.Eval) + { + (batch ??= new EvalBatch()).Items.AddRange(envelope.Eval.Items); + continue; + } + envelope.Seq = (ulong)Interlocked.Increment(ref _seq); + await ScriptWorkerFrameCodec.WriteAsync(_pipe, envelope, _maxFrameBytes, _stopping.Token).ConfigureAwait(false); + } + if (batch is not null) + { + await ScriptWorkerFrameCodec.WriteAsync(_pipe, new Envelope + { + ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion, + Seq = (ulong)Interlocked.Increment(ref _seq), + Eval = batch, + }, _maxFrameBytes, _stopping.Token).ConfigureAwait(false); + } + await _pipe.FlushAsync(_stopping.Token).ConfigureAwait(false); + } + } + catch (OperationCanceledException) { } + catch (Exception ex) when (ex is IOException or ObjectDisposedException or ScriptWorkerProtocolException) + { + if (!_stopping.IsCancellationRequested) + _log.Debug(ex, "ScriptWorker[{Shard}] pipe write ended", Shard); + Kill(WorkerExitReason.Crash, null); + } + } + + private async Task ReadLoopAsync() + { + WorkerExitReason? fault = null; + try + { + while (true) + { + var frame = await ScriptWorkerFrameCodec.ReadAsync(_pipe, _maxFrameBytes, _stopping.Token).ConfigureAwait(false); + if (frame is null) break; + Volatile.Write(ref _lastFrameTimestamp, Stopwatch.GetTimestamp()); + switch (frame.BodyCase) + { + case Envelope.BodyOneofCase.Results: + foreach (var result in frame.Results.Items) + { + // An eval id with no outstanding request is a protocol violation (design §6 "what the host + // trusts"), except for one that was already completed by a kill race — ignored then. + if (_pending.TryRemove(result.EvalId, out var pending)) pending.Complete(result); + else if (result.EvalId == 0 || result.EvalId > (ulong)Interlocked.Read(ref _nextEvalId)) + throw new ScriptWorkerProtocolException($"result for unknown eval id {result.EvalId}"); + } + break; + case Envelope.BodyOneofCase.LoadResult: + if (_loads.TryRemove(frame.LoadResult.LoadId, out var load)) load.TrySetResult(frame.LoadResult); + else throw new ScriptWorkerProtocolException($"load result for unknown load id {frame.LoadResult.LoadId}"); + break; + case Envelope.BodyOneofCase.Logs: + try { _onLogs(frame.Logs); } + catch (Exception ex) { _log.Warning(ex, "ScriptWorker[{Shard}] log forwarding threw", Shard); } + break; + case Envelope.BodyOneofCase.Heartbeat: + break; + case Envelope.BodyOneofCase.DiagReply: + Volatile.Read(ref _diag)?.TrySetResult(frame.DiagReply); + break; + case Envelope.BodyOneofCase.Fault: + _log.Warning("ScriptWorker[{Shard}] reported a fault: {Fault}", Shard, frame.Fault.Message); + break; + default: + throw new ScriptWorkerProtocolException($"unexpected frame {frame.BodyCase} from the worker"); + } + } + } + catch (OperationCanceledException) { } + catch (ScriptWorkerProtocolException ex) + { + _log.Warning("ScriptWorker[{Shard}] protocol violation: {Message}; killing pid {Pid}", Shard, ex.Message, Pid); + fault = WorkerExitReason.ProtocolFault; + } + catch (Exception ex) when (ex is IOException or ObjectDisposedException) + { + } + + // The pipe is gone: the worker is dead or dying. Make sure, then report. + try + { + if (fault is { } f) Kill(f, null); + try + { + await _process.WaitForExitAsync().WaitAsync(TimeSpan.FromSeconds(2)).ConfigureAwait(false); + } + catch (TimeoutException) + { + Kill(WorkerExitReason.Crash, null); + try { await _process.WaitForExitAsync().WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false); } catch (TimeoutException) { } + } + } + catch (InvalidOperationException) + { + // The process object was disposed by a concurrent shutdown: the worker is already gone. + } + Finish(); + } + + private async Task WatchProcessAsync() + { + try { await _process.WaitForExitAsync(_stopping.Token).ConfigureAwait(false); } + catch (OperationCanceledException) { return; } + catch (InvalidOperationException) { return; } + // The reader normally finishes first on EOF; this covers a worker that died while the pipe stayed open. + await Task.Delay(TimeSpan.FromSeconds(2)).ConfigureAwait(false); + Finish(); + } + + private async Task HeartbeatMonitorAsync() + { + using var timer = new PeriodicTimer(_heartbeatInterval); + try + { + while (await timer.WaitForNextTickAsync(_stopping.Token).ConfigureAwait(false)) + { + if (_exited.Task.IsCompleted) return; + var silent = Stopwatch.GetElapsedTime(Volatile.Read(ref _lastFrameTimestamp)); + if (silent > _heartbeatInterval * 3) + { + _log.Warning("ScriptWorker[{Shard}] pid {Pid} sent nothing for {Silent} ms; killing it", Shard, Pid, (long)silent.TotalMilliseconds); + Kill(WorkerExitReason.KilledHeartbeat, null); + return; + } + } + } + catch (OperationCanceledException) { } + } + + private void Finish() + { + int? code = null; + try { if (_process.HasExited) code = _process.ExitCode; } catch (InvalidOperationException) { } + + (WorkerExitReason Reason, string? Culprit)? killed; + lock (_pending) { killed = _killedFor; } + var reason = killed?.Reason ?? (code == 0 ? WorkerExitReason.Clean : WorkerExitReason.Crash); + var inFlight = _pending.Values.Select(p => p.Key).Distinct(StringComparer.Ordinal).ToArray(); + if (!_exited.TrySetResult(new WorkerExit(reason, killed?.Culprit, code, inFlight))) return; + + // Every outstanding evaluation and load ends now; null tells the waiter "the worker went away". + foreach (var id in _pending.Keys.ToArray()) + if (_pending.TryRemove(id, out var pending)) pending.Complete(null); + foreach (var id in _loads.Keys.ToArray()) + if (_loads.TryRemove(id, out var load)) load.TrySetCanceled(); + Volatile.Read(ref _diag)?.TrySetCanceled(); + _outbound.Writer.TryComplete(); + } + + private static string SafeExitCode(Process process) + { + try { return process.HasExited ? process.ExitCode.ToString(System.Globalization.CultureInfo.InvariantCulture) : "?"; } + catch (InvalidOperationException) { return "?"; } + } + + private static void TryDeleteDirectory(string dir) + { + try { if (Directory.Exists(dir)) Directory.Delete(dir, recursive: true); } catch { /* temp dir; best effort */ } + } + + /// A key's compiled form in this worker. + internal sealed class LoadEntry + { + private readonly TaskCompletionSource _ready = new(TaskCreationOptions.RunContinuationsAsynchronously); + + public LoadEntry(string sourceHash, IReadOnlyList inputPaths) + { + SourceHash = sourceHash; + InputPaths = inputPaths; + var slots = new Dictionary(inputPaths.Count, StringComparer.Ordinal); + for (var i = 0; i < inputPaths.Count; i++) slots[inputPaths[i]] = (uint)i; + Slots = slots; + } + + public string SourceHash { get; } + + public IReadOnlyList InputPaths { get; } + + /// Path → slot in the registered order. + public IReadOnlyDictionary Slots { get; } + + /// The worker's verdict (null until it arrives). + public LoadOutcome? Outcome { get; private set; } + + public Task Ready => _ready.Task; + + public bool SameSlots(IReadOnlyList inputPaths) + { + if (inputPaths.Count != InputPaths.Count) return false; + for (var i = 0; i < inputPaths.Count; i++) + if (!string.Equals(inputPaths[i], InputPaths[i], StringComparison.Ordinal)) return false; + return true; + } + + public void Complete(LoadOutcome? outcome) + { + Outcome = outcome; + _ready.TrySetResult(outcome is null ? null : this); + } + } +} + +/// One evaluation outstanding on a worker. +internal sealed class PendingEval +{ + private readonly TaskCompletionSource _completion = new(TaskCreationOptions.RunContinuationsAsynchronously); + + public PendingEval(string key, ulong evalId) + { + Key = key; + EvalId = evalId; + } + + /// The scoped key. + public string Key { get; } + + /// The eval id on the wire. + public ulong EvalId { get; } + + /// The worker's result, or null when the worker ended first. + public Task Completion => _completion.Task; + + public void Complete(EvalResult? result) => _completion.TrySetResult(result); +} diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/WorkerShard.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/WorkerShard.cs new file mode 100644 index 00000000..32febbab --- /dev/null +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/WorkerShard.cs @@ -0,0 +1,249 @@ +using System.Diagnostics; +using System.Diagnostics.Metrics; +using Serilog; +using ZB.MOM.WW.OtOpcUa.Commons.Observability; +using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1; + +namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess; + +/// +/// Supervises one shard's worker process (design §3.3, §5.3.5): starts it lazily on first use, reports its end to +/// the runner (for attribution), and restarts it after an exponential backoff — 250 ms doubling to a 30 s cap with +/// ±20 % jitter, reset once a worker has stayed up for 60 s. It never runs a script in the host when the worker +/// cannot start (fail closed, §14 #4): returns null and the caller reports +/// BadResourceUnavailable. +/// +internal sealed class WorkerShard : IAsyncDisposable +{ + internal static readonly TimeSpan InitialBackoff = TimeSpan.FromMilliseconds(250); + internal static readonly TimeSpan MaxBackoff = TimeSpan.FromSeconds(30); + internal static readonly TimeSpan HealthyResetAfter = TimeSpan.FromSeconds(60); + + private static readonly Counter StartsCounter = OtOpcUaTelemetry.Meter.CreateCounter( + "otopcua.script.worker.starts", unit: "{process}", description: "ScriptWorker processes started (tag: shard)."); + + private static readonly Counter ExitsCounter = OtOpcUaTelemetry.Meter.CreateCounter( + "otopcua.script.worker.exits", unit: "{process}", + description: "ScriptWorker processes ended (tags: shard, reason = clean | killed_deadline | killed_heartbeat | killed_load | crash | protocol | disposed)."); + + private readonly Func> _launch; + private readonly Action _onExit; + private readonly ILogger _log; + private readonly object _gate = new(); + private readonly Random _jitter = new(); + private WorkerConnection? _current; + private Task? _starting; + private long _backoffUntil; + private int _consecutiveFailures; + private bool _disposed; + + public WorkerShard(string label, bool isolation, Func> launch, + Action onExit, ILogger log) + { + Label = label; + IsIsolation = isolation; + _launch = launch; + _onExit = onExit; + _log = log; + // The isolation worker runs one evaluation at a time, so a crash there names exactly one key (design §5.3.3). + IsolationGate = isolation ? new SemaphoreSlim(1, 1) : null; + } + + /// The shard label (0…K-1, or iso). + public string Label { get; } + + /// Whether this is the isolation ("penalty box") worker. + public bool IsIsolation { get; } + + /// Serialises evaluations on the isolation worker; null on a normal shard. + public SemaphoreSlim? IsolationGate { get; } + + /// The live worker, if any (diagnostics / tests). + public WorkerConnection? Current + { + get { lock (_gate) return _current is { IsAlive: true } c ? c : null; } + } + + /// Consecutive start failures and crashes since the last healthy period (diagnostics / tests). + public int ConsecutiveFailures + { + get { lock (_gate) return _consecutiveFailures; } + } + + /// + /// The ready worker, starting one if needed. Waits at most — through a start in progress + /// or a backoff — and returns null if no worker is ready by then. + /// + /// How long to wait. + /// Cancellation. + /// The worker, or null. + public async Task GetReadyAsync(TimeSpan grace, CancellationToken ct) + { + var deadline = Stopwatch.GetTimestamp() + (long)(grace.TotalSeconds * Stopwatch.Frequency); + while (true) + { + Task starting; + TimeSpan wait; + lock (_gate) + { + if (_disposed) return null; + if (_current is { IsAlive: true } live) return live; + var now = Stopwatch.GetTimestamp(); + if (_starting is null && now >= _backoffUntil) + { + _starting = StartCoreAsync(); + } + if (_starting is not null) + { + starting = _starting; + wait = TimeSpan.Zero; + } + else + { + starting = null!; + wait = Stopwatch.GetElapsedTime(now, _backoffUntil); + } + } + + var remaining = Stopwatch.GetElapsedTime(Stopwatch.GetTimestamp(), deadline); + if (remaining <= TimeSpan.Zero) return null; + if (starting is not null) + { + try + { + var started = await starting.WaitAsync(remaining, ct).ConfigureAwait(false); + if (started is { IsAlive: true }) return started; + } + catch (TimeoutException) + { + return null; + } + continue; + } + if (wait > remaining) return null; + await Task.Delay(wait, ct).ConfigureAwait(false); + } + } + + private async Task StartCoreAsync() + { + await Task.Yield(); + WorkerConnection? connection = null; + try + { + connection = await _launch(CancellationToken.None).ConfigureAwait(false); + StartsCounter.Add(1, new KeyValuePair("shard", Label)); + _log.Information("ScriptWorker[{Shard}] started, pid {Pid}", Label, connection.Pid); + var started = connection; + _ = started.Exited.ContinueWith(t => OnExited(started, t.Result), CancellationToken.None, + TaskContinuationOptions.ExecuteSynchronously | TaskContinuationOptions.OnlyOnRanToCompletion, TaskScheduler.Default); + lock (_gate) + { + _starting = null; + if (_disposed) + { + _ = started.DisposeAsync().AsTask(); + return null; + } + _current = started; + } + return started; + } + catch (Exception ex) + { + _log.Error(ex, "ScriptWorker[{Shard}] failed to start; scripts on it report runner unavailable until it does", Label); + lock (_gate) + { + _starting = null; + ScheduleBackoffLocked(); + } + if (connection is not null) await connection.DisposeAsync().ConfigureAwait(false); + return null; + } + } + + private void OnExited(WorkerConnection connection, WorkerExit exit) + { + ExitsCounter.Add(1, new KeyValuePair("shard", Label), + new KeyValuePair("reason", MetricReason(exit.Reason))); + lock (_gate) + { + if (ReferenceEquals(_current, connection)) _current = null; + if (!_disposed) + { + if (Stopwatch.GetElapsedTime(connection.ReadyTimestamp) >= HealthyResetAfter) _consecutiveFailures = 0; + ScheduleBackoffLocked(); + } + } + if (exit.Reason != WorkerExitReason.Disposed) + { + _log.Warning("ScriptWorker[{Shard}] pid {Pid} ended: {Reason} (exit code {ExitCode}, culprit {Culprit}, {InFlight} evaluation(s) in flight)", + Label, connection.Pid, exit.Reason, exit.ExitCode, exit.CulpritKey ?? "none", exit.InFlightKeys.Count); + } + try { _onExit(this, exit); } + catch (Exception ex) { _log.Error(ex, "ScriptWorker[{Shard}] exit handler threw", Label); } + _ = connection.DisposeAsync().AsTask(); + } + + private void ScheduleBackoffLocked() + { + _consecutiveFailures++; + var delay = NextBackoff(_consecutiveFailures, _jitter.NextDouble()); + _backoffUntil = Stopwatch.GetTimestamp() + (long)(delay.TotalSeconds * Stopwatch.Frequency); + } + + /// The restart delay after consecutive failures (design §5.3.5). + /// 1 for the first failure. + /// A sample in [0, 1) mapped to ±20 %. + /// The delay. + internal static TimeSpan NextBackoff(int failures, double unitJitter) + { + var exponent = Math.Clamp(failures - 1, 0, 30); + var baseMs = Math.Min(MaxBackoff.TotalMilliseconds, InitialBackoff.TotalMilliseconds * Math.Pow(2, exponent)); + var factor = 0.8 + (0.4 * Math.Clamp(unitJitter, 0, 1)); + return TimeSpan.FromMilliseconds(Math.Min(MaxBackoff.TotalMilliseconds * 1.2, baseMs * factor)); + } + + private static string MetricReason(WorkerExitReason reason) => reason switch + { + WorkerExitReason.Clean => "clean", + WorkerExitReason.KilledDeadline => "killed_deadline", + WorkerExitReason.KilledHeartbeat => "killed_heartbeat", + WorkerExitReason.KilledLoadTimeout => "killed_load", + WorkerExitReason.ProtocolFault => "protocol", + WorkerExitReason.Disposed => "disposed", + _ => "crash", + }; + + /// Stops the shard's worker. + /// A task that completes when the worker has ended. + public async ValueTask DisposeAsync() + { + WorkerConnection? current; + Task? starting; + lock (_gate) + { + _disposed = true; + current = _current; + starting = _starting; + _current = null; + } + if (starting is not null) + { + try { current ??= await starting.WaitAsync(TimeSpan.FromSeconds(15)).ConfigureAwait(false); } catch { /* shutting down */ } + } + if (current is not null) await current.DisposeAsync().ConfigureAwait(false); + IsolationGate?.Dispose(); + } + + /// Maps a context kind to the wire enum. + /// The kind. + /// The wire kind. + internal static ContextKind ToWire(ScriptContextKind kind) => kind switch + { + ScriptContextKind.VirtualTag => ContextKind.VirtualTag, + ScriptContextKind.Calculation => ContextKind.Calculation, + ScriptContextKind.AlarmPredicate => ContextKind.AlarmPredicate, + _ => ContextKind.Unspecified, + }; +} diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess.csproj b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess.csproj new file mode 100644 index 00000000..057b919c --- /dev/null +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess.csproj @@ -0,0 +1,20 @@ + + + + + ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess + true + + + + + + + + + + + + diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol/Protos/script_worker.proto b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol/Protos/script_worker.proto new file mode 100644 index 00000000..f27c40b5 --- /dev/null +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol/Protos/script_worker.proto @@ -0,0 +1,214 @@ +// lmxopcua#780 — host <-> ScriptWorker IPC (design docs/plans/2026-11-09-out-of-process-script-runner-design.md §4). +// Framing: 4-byte little-endian length prefix + one serialized Envelope. Zero-length and oversize frames are protocol +// faults. One request frame and one reply frame per evaluation; the only other worker->host traffic is logs and +// heartbeats. Bump protocol_version on any incompatible change. +syntax = "proto3"; + +package otopcua.scriptworker.v1; + +option csharp_namespace = "ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1"; + +message Envelope { + uint32 protocol_version = 1; + uint64 seq = 2; + oneof body { + Hello hello = 10; // host -> worker, first frame + Ready ready = 11; // worker -> host, reply to Hello + LoadScripts load = 12; // host -> worker + LoadResult load_result = 13; // worker -> host + Unload unload = 14; // host -> worker + EvalBatch eval = 15; // host -> worker + EvalResults results = 16; // worker -> host, streamed as runs complete + LogBatch logs = 17; // worker -> host, fire-and-forget, bounded + Heartbeat heartbeat = 18; // worker -> host, every heartbeat_interval_ms + Shutdown shutdown = 19; // host -> worker + Fault fault = 20; // either way; the sender closes after it + DiagnosticsRequest diag_request = 21; // host -> worker (tests / ops) + DiagnosticsReply diag_reply = 22; // worker -> host + Cancel cancel = 23; // host -> worker, at the soft deadline + } +} + +message Hello { + string nonce = 1; // must be echoed in Ready + uint32 max_frame_bytes = 2; + uint32 eval_threads = 3; // the worker's scheduler width; 1 for the isolation worker + uint32 queue_capacity = 4; // queued evaluations above this answer BUSY + uint32 hard_grace_ms = 5; // worker watchdog: fail fast past soft + grace (+ margin) + uint32 heartbeat_interval_ms = 6; + uint32 max_log_lines = 7; // outbound log buffer cap +} + +message Ready { + string nonce = 1; + int32 pid = 2; + string runtime_version = 3; +} + +enum ContextKind { + CONTEXT_KIND_UNSPECIFIED = 0; + VIRTUAL_TAG = 1; + CALCULATION = 2; + ALARM_PREDICATE = 3; +} + +message ScriptSource { + string key = 1; + bytes source_sha256 = 2; + ContextKind kind = 3; + string source = 4; + repeated string input_paths = 5; // slot order for EvalRequest.inputs +} + +message LoadScripts { + uint64 load_id = 1; + repeated ScriptSource scripts = 2; +} + +message LoadOutcome { + string key = 1; + bool ok = 2; + ScriptError error = 3; // COMPILE or SANDBOX when !ok +} + +message LoadResult { + uint64 load_id = 1; + repeated LoadOutcome outcomes = 2; +} + +message Unload { + repeated string keys = 1; +} + +message EvalBatch { + repeated EvalRequest items = 1; +} + +message EvalRequest { + uint64 eval_id = 1; + string key = 2; + repeated Input inputs = 3; + int64 now_utc_ticks = 4; // ctx.Now (host clock) + uint32 soft_timeout_ms = 5; + uint32 hard_grace_ms = 6; // this run's G; 0 = Hello.hard_grace_ms +} + +message Input { + uint32 slot = 1; // index into ScriptSource.input_paths; ignored when path is set + Variant value = 2; + uint32 status_code = 3; + int64 source_ts_ticks = 4; // 0 = null + int64 server_ts_ticks = 5; + string path = 6; // inline path for an input with no registered slot + bool unsupported = 7; // the host could not encode the value's CLR type (see unsupported_type) + string unsupported_type = 8; +} + +message EvalResult { + uint64 eval_id = 1; + oneof outcome { + Variant value = 2; + bool predicate = 3; + ScriptError error = 4; + } + uint64 allocated_bytes = 5; // GC.GetAllocatedBytesForCurrentThread around the run + uint32 run_micros = 6; +} + +message EvalResults { + repeated EvalResult items = 1; +} + +message ScriptError { + enum Kind { + THREW = 0; + COMPILE = 1; + SANDBOX = 2; + SOFT_TIMEOUT = 3; + UNSUPPORTED_TYPE = 4; + BUSY = 5; + NOT_LOADED = 6; + } + Kind kind = 1; + string message = 2; +} + +message Variant { + enum TypeTag { + NULL = 0; + BOOLEAN = 1; + SBYTE = 2; + BYTE = 3; + INT16 = 4; + UINT16 = 5; + INT32 = 6; + UINT32 = 7; + INT64 = 8; + UINT64 = 9; + FLOAT = 10; + DOUBLE = 11; + STRING = 12; + DATETIME_UTC = 13; + ARRAY = 14; + } + TypeTag type = 1; // the exact CLR type: (short)ctx.GetTag(x).Value must keep working + oneof v { + bool b = 2; + sint64 i = 3; + uint64 u = 4; + double d = 5; + float f = 6; + string s = 7; + int64 dt_ticks = 8; + ArrayValue array = 9; + } +} + +message ArrayValue { + Variant.TypeTag element_type = 1; + repeated Variant items = 2; +} + +message LogBatch { + repeated LogLine lines = 1; + uint32 dropped = 2; +} + +message LogLine { + string key = 1; + uint32 level = 2; // Serilog LogEventLevel + string rendered = 3; + string exception = 4; + int64 ts_ticks = 5; +} + +message Heartbeat { + uint32 inflight = 1; + uint32 queued = 2; + int64 heap_bytes = 3; + uint32 gen2_count = 4; +} + +message Shutdown { +} + +// Sent by the host when it has answered the caller with a soft timeout: cancels the run's ctx.CancellationToken, so a +// polling script ends itself. The worker never cancels on its own clock, which keeps the in-process ordering — the +// caller always has its timeout before the script can observe cancellation. +message Cancel { + uint64 eval_id = 1; +} + +message Fault { + string message = 1; +} + +message DiagnosticsRequest { +} + +message DiagnosticsReply { + int32 pid = 1; + repeated string loaded_assemblies = 2; + repeated string environment_names = 3; + string working_directory = 4; +} diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol/ScriptWorkerEnvironment.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol/ScriptWorkerEnvironment.cs new file mode 100644 index 00000000..1f1a9478 --- /dev/null +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol/ScriptWorkerEnvironment.cs @@ -0,0 +1,54 @@ +namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol; + +/// +/// The environment variables the host sets on a ScriptWorker (design §6). The nonce travels in the environment, +/// never on the command line (mxaccessgw WorkerBootstrap.md): a command line is world-readable on most +/// systems. +/// +public static class ScriptWorkerEnvironment +{ + /// The pipe name the worker connects to. + public const string PipeName = "OTOPCUA_SCRIPTWORKER_PIPE"; + + /// The one-time launch nonce the worker must echo in Ready. + public const string Nonce = "OTOPCUA_SCRIPTWORKER_NONCE"; + + /// The host's process id; the worker exits when it disappears. + public const string ParentPid = "OTOPCUA_SCRIPTWORKER_PARENT_PID"; + + /// The worker's shard label, for its own diagnostics. + public const string Shard = "OTOPCUA_SCRIPTWORKER_SHARD"; + + /// + /// Exit code of a worker whose pipe reached end of stream (the host went away). Distinct from a crash so the + /// host's exit reason metric can tell them apart. + /// + public const int ExitPipeClosed = 3; + + /// Exit code of a worker whose handshake or framing failed. + public const int ExitProtocolFault = 4; + + /// Exit code of a worker whose parent process disappeared. + public const int ExitParentGone = 5; + + /// Exit code of a worker that found a forbidden assembly loaded in itself. + public const int ExitForbiddenAssembly = 6; + + /// + /// Assembly-name prefixes that must never be loaded in a worker (design §6): the host's cluster, database, + /// OPC UA and secrets stacks. The worker checks itself against this list and exits if it finds one. + /// + public static readonly IReadOnlyList ForbiddenAssemblyPrefixes = + [ + "Akka", + "Microsoft.Data.SqlClient", + "System.Data.SqlClient", + "Microsoft.EntityFrameworkCore", + "Opc.Ua", + "ZB.MOM.WW.Secrets", + "ZB.MOM.WW.OtOpcUa.Runtime", + "ZB.MOM.WW.OtOpcUa.Host", + "ZB.MOM.WW.OtOpcUa.Driver.", + "Grpc.", + ]; +} diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol/ScriptWorkerFrameCodec.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol/ScriptWorkerFrameCodec.cs new file mode 100644 index 00000000..ea54851f --- /dev/null +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol/ScriptWorkerFrameCodec.cs @@ -0,0 +1,116 @@ +using System.Buffers; +using System.Buffers.Binary; +using Google.Protobuf; +using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1; + +namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol; + +/// +/// Length-prefixed protobuf framing for the ScriptWorker pipe (lmxopcua#780, design §4.2): a 4-byte little-endian +/// length, then one serialized . Copies mxaccessgw's WorkerFrameProtocol discipline: +/// zero-length, oversize, truncated and unparseable frames are all a , +/// after which the stream is unusable and the peer is dropped. +/// +public static class ScriptWorkerFrameCodec +{ + /// The protocol version both ends put on every envelope. + public const uint ProtocolVersion = 1; + + /// Default frame cap: 4 MiB. + public const int DefaultMaxFrameBytes = 4 * 1024 * 1024; + + /// Writes one frame. Not thread-safe: callers serialise writes. + /// The pipe. + /// The envelope. + /// The negotiated cap; a larger envelope is refused before anything is written. + /// Cancellation. + /// A task that completes when the frame is written (not flushed). + public static async ValueTask WriteAsync(Stream stream, Envelope envelope, int maxFrameBytes, CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(stream); + ArgumentNullException.ThrowIfNull(envelope); + var size = envelope.CalculateSize(); + if (size == 0) throw new ScriptWorkerProtocolException("refusing to write an empty frame"); + if (size > maxFrameBytes) + throw new ScriptWorkerProtocolException($"frame of {size} bytes exceeds the {maxFrameBytes}-byte cap"); + + var buffer = ArrayPool.Shared.Rent(size + 4); + try + { + BinaryPrimitives.WriteInt32LittleEndian(buffer, size); + envelope.WriteTo(new Span(buffer, 4, size)); + await stream.WriteAsync(buffer.AsMemory(0, size + 4), ct).ConfigureAwait(false); + } + finally + { + ArrayPool.Shared.Return(buffer); + } + } + + /// Reads one frame. + /// The pipe. + /// The negotiated cap. + /// Cancellation. + /// The envelope, or null on a clean end of stream at a frame boundary. + /// A zero-length, oversize, truncated or unparseable frame. + public static async ValueTask ReadAsync(Stream stream, int maxFrameBytes, CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(stream); + var header = new byte[4]; + var got = await ReadFullyAsync(stream, header, ct).ConfigureAwait(false); + if (got == 0) return null; + if (got < 4) throw new ScriptWorkerProtocolException("truncated frame header"); + + var size = BinaryPrimitives.ReadInt32LittleEndian(header); + if (size == 0) throw new ScriptWorkerProtocolException("zero-length frame"); + if (size < 0 || size > maxFrameBytes) + throw new ScriptWorkerProtocolException($"frame length {size} is outside 1..{maxFrameBytes}"); + + var body = ArrayPool.Shared.Rent(size); + try + { + var read = await ReadFullyAsync(stream, body.AsMemory(0, size), ct).ConfigureAwait(false); + if (read < size) throw new ScriptWorkerProtocolException($"truncated frame: {read} of {size} bytes"); + Envelope envelope; + try + { + envelope = Envelope.Parser.ParseFrom(body, 0, size); + } + catch (InvalidProtocolBufferException ex) + { + throw new ScriptWorkerProtocolException($"unparseable frame: {ex.Message}", ex); + } + if (envelope.ProtocolVersion != ProtocolVersion) + throw new ScriptWorkerProtocolException( + $"protocol version {envelope.ProtocolVersion} is not the supported {ProtocolVersion}"); + if (envelope.BodyCase == Envelope.BodyOneofCase.None) + throw new ScriptWorkerProtocolException("frame carries no body"); + return envelope; + } + finally + { + ArrayPool.Shared.Return(body); + } + } + + private static async ValueTask ReadFullyAsync(Stream stream, Memory buffer, CancellationToken ct) + { + var total = 0; + while (total < buffer.Length) + { + var n = await stream.ReadAsync(buffer[total..], ct).ConfigureAwait(false); + if (n == 0) break; + total += n; + } + return total; + } +} + +/// A framing or schema violation on the ScriptWorker pipe; the connection is dropped. +public sealed class ScriptWorkerProtocolException : Exception +{ + /// Initializes the exception. + /// What was wrong. + /// The underlying parser error, if any. + public ScriptWorkerProtocolException(string message, Exception? inner = null) : base(message, inner) { } +} diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol/VariantCodec.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol/VariantCodec.cs new file mode 100644 index 00000000..d37ffde4 --- /dev/null +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol/VariantCodec.cs @@ -0,0 +1,159 @@ +using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1; + +namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol; + +/// +/// CLR value ↔ (design §4.4). The exact boxed CLR type survives the round trip — a script +/// doing (short)ctx.GetTag(x).Value throws if a short arrives as an +/// int — so the type tag names the CLR type, not just an OPC UA one. A type with no tag is never converted: +/// refuses it, and the caller reports UNSUPPORTED_TYPE (design §14 #11). +/// +public static class VariantCodec +{ + /// Encodes . + /// The boxed CLR value (null allowed). + /// The encoded variant when supported. + /// The CLR type name when not supported. + /// true when the type has a tag. + public static bool TryEncode(object? value, out Variant variant, out string? unsupportedType) + { + unsupportedType = null; + variant = new Variant(); + switch (value) + { + case null: variant.Type = Variant.Types.TypeTag.Null; return true; + case bool b: variant.Type = Variant.Types.TypeTag.Boolean; variant.B = b; return true; + case sbyte v: variant.Type = Variant.Types.TypeTag.Sbyte; variant.I = v; return true; + case byte v: variant.Type = Variant.Types.TypeTag.Byte; variant.U = v; return true; + case short v: variant.Type = Variant.Types.TypeTag.Int16; variant.I = v; return true; + case ushort v: variant.Type = Variant.Types.TypeTag.Uint16; variant.U = v; return true; + case int v: variant.Type = Variant.Types.TypeTag.Int32; variant.I = v; return true; + case uint v: variant.Type = Variant.Types.TypeTag.Uint32; variant.U = v; return true; + case long v: variant.Type = Variant.Types.TypeTag.Int64; variant.I = v; return true; + case ulong v: variant.Type = Variant.Types.TypeTag.Uint64; variant.U = v; return true; + case float v: variant.Type = Variant.Types.TypeTag.Float; variant.F = v; return true; + case double v: variant.Type = Variant.Types.TypeTag.Double; variant.D = v; return true; + case string v: variant.Type = Variant.Types.TypeTag.String; variant.S = v; return true; + case DateTime v: + variant.Type = Variant.Types.TypeTag.DatetimeUtc; + variant.DtTicks = ToUtc(v).Ticks; + return true; + case Array array when array.Rank == 1 && TryElementTag(array.GetType().GetElementType()!, out var elementTag): + { + var av = new ArrayValue { ElementType = elementTag }; + foreach (var item in array) + { + if (!TryEncode(item, out var element, out unsupportedType)) return false; + av.Items.Add(element); + } + variant.Type = Variant.Types.TypeTag.Array; + variant.Array = av; + return true; + } + default: + unsupportedType = value.GetType().FullName ?? value.GetType().Name; + return false; + } + } + + /// Decodes back to the boxed CLR value it was encoded from. + /// The variant. + /// The boxed value. + /// An unknown tag, or a payload that does not match the tag. + public static object? Decode(Variant? variant) + { + if (variant is null) return null; + return variant.Type switch + { + Variant.Types.TypeTag.Null => null, + Variant.Types.TypeTag.Boolean => Expect(variant, Variant.VOneofCase.B).B, + Variant.Types.TypeTag.Sbyte => checked((sbyte)Expect(variant, Variant.VOneofCase.I).I), + Variant.Types.TypeTag.Byte => checked((byte)Expect(variant, Variant.VOneofCase.U).U), + Variant.Types.TypeTag.Int16 => checked((short)Expect(variant, Variant.VOneofCase.I).I), + Variant.Types.TypeTag.Uint16 => checked((ushort)Expect(variant, Variant.VOneofCase.U).U), + Variant.Types.TypeTag.Int32 => checked((int)Expect(variant, Variant.VOneofCase.I).I), + Variant.Types.TypeTag.Uint32 => checked((uint)Expect(variant, Variant.VOneofCase.U).U), + Variant.Types.TypeTag.Int64 => Expect(variant, Variant.VOneofCase.I).I, + Variant.Types.TypeTag.Uint64 => Expect(variant, Variant.VOneofCase.U).U, + Variant.Types.TypeTag.Float => Expect(variant, Variant.VOneofCase.F).F, + Variant.Types.TypeTag.Double => Expect(variant, Variant.VOneofCase.D).D, + Variant.Types.TypeTag.String => Expect(variant, Variant.VOneofCase.S).S, + Variant.Types.TypeTag.DatetimeUtc => new DateTime(Expect(variant, Variant.VOneofCase.DtTicks).DtTicks, DateTimeKind.Utc), + Variant.Types.TypeTag.Array => DecodeArray(Expect(variant, Variant.VOneofCase.Array).Array), + _ => throw new ScriptWorkerProtocolException($"unknown variant type tag {(int)variant.Type}"), + }; + } + + /// A as UTC: a Local value is converted; an Unspecified one is taken as UTC. + /// The value. + /// The UTC value. + public static DateTime ToUtc(DateTime value) => value.Kind switch + { + DateTimeKind.Utc => value, + DateTimeKind.Local => value.ToUniversalTime(), + _ => DateTime.SpecifyKind(value, DateTimeKind.Utc), + }; + + private static Variant Expect(Variant variant, Variant.VOneofCase expected) + { + // proto3 drops a default scalar (0, false, "") from the oneof, so an absent payload is the default value. + if (variant.VCase != expected && variant.VCase != Variant.VOneofCase.None) + throw new ScriptWorkerProtocolException($"variant tagged {variant.Type} carries a {variant.VCase} payload"); + return variant; + } + + private static Array DecodeArray(ArrayValue? array) + { + if (array is null) throw new ScriptWorkerProtocolException("array variant without an array payload"); + var elementType = ClrType(array.ElementType); + var result = Array.CreateInstance(elementType, array.Items.Count); + for (var i = 0; i < array.Items.Count; i++) + { + var item = array.Items[i]; + if (item.Type != array.ElementType && item.Type != Variant.Types.TypeTag.Null) + throw new ScriptWorkerProtocolException($"array of {array.ElementType} holds a {item.Type} element"); + result.SetValue(Decode(item), i); + } + return result; + } + + private static bool TryElementTag(Type elementType, out Variant.Types.TypeTag tag) + { + tag = elementType switch + { + _ when elementType == typeof(bool) => Variant.Types.TypeTag.Boolean, + _ when elementType == typeof(sbyte) => Variant.Types.TypeTag.Sbyte, + _ when elementType == typeof(byte) => Variant.Types.TypeTag.Byte, + _ when elementType == typeof(short) => Variant.Types.TypeTag.Int16, + _ when elementType == typeof(ushort) => Variant.Types.TypeTag.Uint16, + _ when elementType == typeof(int) => Variant.Types.TypeTag.Int32, + _ when elementType == typeof(uint) => Variant.Types.TypeTag.Uint32, + _ when elementType == typeof(long) => Variant.Types.TypeTag.Int64, + _ when elementType == typeof(ulong) => Variant.Types.TypeTag.Uint64, + _ when elementType == typeof(float) => Variant.Types.TypeTag.Float, + _ when elementType == typeof(double) => Variant.Types.TypeTag.Double, + _ when elementType == typeof(string) => Variant.Types.TypeTag.String, + _ when elementType == typeof(DateTime) => Variant.Types.TypeTag.DatetimeUtc, + _ => Variant.Types.TypeTag.Null, + }; + return tag != Variant.Types.TypeTag.Null; + } + + private static Type ClrType(Variant.Types.TypeTag tag) => tag switch + { + Variant.Types.TypeTag.Boolean => typeof(bool), + Variant.Types.TypeTag.Sbyte => typeof(sbyte), + Variant.Types.TypeTag.Byte => typeof(byte), + Variant.Types.TypeTag.Int16 => typeof(short), + Variant.Types.TypeTag.Uint16 => typeof(ushort), + Variant.Types.TypeTag.Int32 => typeof(int), + Variant.Types.TypeTag.Uint32 => typeof(uint), + Variant.Types.TypeTag.Int64 => typeof(long), + Variant.Types.TypeTag.Uint64 => typeof(ulong), + Variant.Types.TypeTag.Float => typeof(float), + Variant.Types.TypeTag.Double => typeof(double), + Variant.Types.TypeTag.String => typeof(string), + Variant.Types.TypeTag.DatetimeUtc => typeof(DateTime), + _ => throw new ScriptWorkerProtocolException($"array element type {tag} is not supported"), + }; +} diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.csproj b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.csproj new file mode 100644 index 00000000..ddfa49b6 --- /dev/null +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.csproj @@ -0,0 +1,28 @@ + + + + ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol + true + + $(NoWarn);CS1591 + + + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + + + + + + + + + + diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/IScriptRunner.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/IScriptRunner.cs new file mode 100644 index 00000000..0678896d --- /dev/null +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/IScriptRunner.cs @@ -0,0 +1,137 @@ +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +namespace ZB.MOM.WW.OtOpcUa.Core.Scripting; + +/// +/// The execution seam every host script owner can run a compiled script through (lmxopcua#780, design §11): a +/// script key, its source, its input snapshots and its budget go in; one comes +/// out. Implementations never throw for script faults. +/// +/// +/// +/// is today's + +/// behind this interface (the parity baseline). The out-of-process runner +/// (ZB.MOM.WW.OtOpcUa.Core.Scripting.Runner) runs the same compile in a pre-emptible worker process. +/// The passthrough fast path (PassthroughScript) never reaches a runner: it runs no user code. +/// +/// +/// Keys are namespaced by , so a virtual-tag id and a calc RawPath that +/// happen to be equal never share quarantine state, and for one seam leaves the other +/// seams' quarantines alone. +/// +/// +public interface IScriptRunner +{ + /// Runs one compiled evaluation. + /// What to run. + /// Caller cancellation (shutdown); wins over the budget. + /// The outcome; never faults for a script failure. + Task RunAsync(ScriptInvocation invocation, CancellationToken ct = default); + + /// + /// The deploy boundary for one seam: lifts every quarantine and forgets every remembered compile failure of + /// , and drops its compiled scripts. + /// + /// The seam's context kind. + void Reset(ScriptContextKind kind); + + /// The keys of currently quarantined. + /// The seam's context kind. + /// The quarantined keys (un-namespaced). + IReadOnlyCollection QuarantinedKeys(ScriptContextKind kind); +} + +/// Which script context (and result type) a script compiles against. +public enum ScriptContextKind +{ + /// A virtual tag: VirtualTagContext → object?. + VirtualTag = 1, + + /// A Calculation tag: VirtualTagContext → object?. + Calculation = 2, + + /// A scripted-alarm predicate: AlarmPredicateContext → bool (not run out of process yet). + AlarmPredicate = 3, +} + +/// One evaluation handed to an . +/// The script instance key (virtual-tag id, calc RawPath, alarm id). +/// The context kind. +/// The script source (also its identity: a change lifts the key's quarantine). +/// The read cache: dependency snapshots by path. Not mutated until the run completes. +/// The soft budget T. +/// The per-evaluation script logger ctx.Logger writes through; null ⇒ a silent logger. +public sealed record ScriptInvocation( + string Key, + ScriptContextKind Kind, + string Source, + IReadOnlyDictionary Inputs, + TimeSpan Timeout, + Serilog.ILogger? ScriptLogger = null); + +/// How an evaluation ended. +public enum ScriptRunStatus +{ + /// The script returned . + Ok = 0, + + /// A script-attributable failure (threw, timed out, refused, quarantined, did not compile) — Bad as today. + Failed = 1, + + /// + /// The runner could not run the script, and it is not the script's fault: its worker restarted under it, or no + /// worker became ready in time. Published as BadResourceUnavailable (design §9 / §14 #3). + /// + Unavailable = 2, +} + +/// The result of one . +/// How it ended. +/// The script's result when . +/// The failure reason otherwise; the same strings the evaluators have always published. +public sealed record ScriptRunOutcome(ScriptRunStatus Status, object? Value, string? Reason) +{ + /// Reason when the worker running the script died under it (another script's kill, a crash). + public const string WorkerRestartedReason = "script worker restarted"; + + /// Reason when no worker became ready within . + public const string RunnerUnavailableReason = "script runner unavailable"; + + /// A successful run. + /// The script's result. + /// The outcome. + public static ScriptRunOutcome Ok(object? value) => new(ScriptRunStatus.Ok, value, null); + + /// A script-attributable failure. + /// The reason. + /// The outcome. + public static ScriptRunOutcome Failed(string reason) => new(ScriptRunStatus.Failed, null, reason); + + /// A runner-attributable failure. + /// The reason. + /// The outcome. + public static ScriptRunOutcome Unavailable(string reason) => new(ScriptRunStatus.Unavailable, null, reason); +} + +/// Maps an outcome onto the value seams' result type. +public static class ScriptRunOutcomeExtensions +{ + /// + /// → Ok(value); → Failure(reason) + /// (published BadInternalError, as today); → + /// Unavailable(reason) (published BadResourceUnavailable). + /// + /// The runner's outcome. + /// The seam result. + public static Commons.Engines.VirtualTagEvalResult ToVirtualTagEvalResult(this ScriptRunOutcome outcome) + { + ArgumentNullException.ThrowIfNull(outcome); + return outcome.Status switch + { + ScriptRunStatus.Ok => Commons.Engines.VirtualTagEvalResult.Ok(outcome.Value), + ScriptRunStatus.Unavailable => Commons.Engines.VirtualTagEvalResult.Unavailable( + outcome.Reason ?? ScriptRunOutcome.RunnerUnavailableReason), + _ => Commons.Engines.VirtualTagEvalResult.Failure(outcome.Reason ?? "evaluator failure"), + }; + } +} diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/InProcessScriptRunner.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/InProcessScriptRunner.cs new file mode 100644 index 00000000..7ed57ee7 --- /dev/null +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/InProcessScriptRunner.cs @@ -0,0 +1,141 @@ +using System.Collections.Concurrent; +using Serilog; +using Serilog.Core; +using ZB.MOM.WW.OtOpcUa.Commons.Engines; +using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms; +using ZB.MOM.WW.OtOpcUa.Core.VirtualTags; + +namespace ZB.MOM.WW.OtOpcUa.Core.Scripting; + +/// +/// over today's in-process machinery: a +/// per context kind and one (lmxopcua#780, design §11). It produces the outcomes and +/// reason strings the seams' own in-process paths produce, which makes it the parity baseline the out-of-process +/// runner is tested against, and the in-process half of the future shadow mode (P2). +/// +/// +/// The seams do not route their path through this class: they keep their +/// existing guard code, so the default mode is unchanged byte for byte. Unlike those paths, this runner remembers +/// compile failures by source for every kind (Calculation already did, R07:P4). +/// +public sealed class InProcessScriptRunner : IScriptRunner, IDisposable +{ + private readonly CompiledScriptCache _valueCache = new(); + private readonly CompiledScriptCache _predicateCache = new(); + private readonly ConcurrentDictionary _compileFailures = new(StringComparer.Ordinal); + private readonly ScriptRunGuard _guard; + + /// Initializes the runner. + /// Run-guard knobs; null ⇒ defaults. + /// Where quarantine lines go; null ⇒ Serilog's global logger. + public InProcessScriptRunner(ScriptingOptions? scripting = null, ILogger? logger = null) + { + _guard = new ScriptRunGuard(scripting, logger); + } + + /// The run guard (diagnostics / tests). + public ScriptRunGuard RunGuard => _guard; + + /// + public async Task RunAsync(ScriptInvocation invocation, CancellationToken ct = default) + { + ArgumentNullException.ThrowIfNull(invocation); + var key = ScopedKey(invocation.Kind, invocation.Key); + var failureKey = ScopedKey(invocation.Kind, invocation.Source); + if (_compileFailures.TryGetValue(failureKey, out var known)) return ScriptRunOutcome.Failed(known); + var logger = invocation.ScriptLogger ?? Logger.None; + + try + { + switch (invocation.Kind) + { + case ScriptContextKind.VirtualTag: + case ScriptContextKind.Calculation: + { + var evaluator = await Task.Run(() => _valueCache.GetOrCompile(invocation.Source), ct).ConfigureAwait(false); + var context = new VirtualTagContext(invocation.Inputs, static (_, _) => { }, logger); + var value = await _guard.RunAsync(key, invocation.Source, evaluator, context, invocation.Timeout, ct) + .ConfigureAwait(false); + return ScriptRunOutcome.Ok(value); + } + case ScriptContextKind.AlarmPredicate: + { + var evaluator = await Task.Run(() => _predicateCache.GetOrCompile(invocation.Source), ct).ConfigureAwait(false); + var context = new AlarmPredicateContext(invocation.Inputs, logger); + var value = await _guard.RunAsync(key, invocation.Source, evaluator, context, invocation.Timeout, ct) + .ConfigureAwait(false); + return ScriptRunOutcome.Ok(value); + } + default: + return ScriptRunOutcome.Failed($"unsupported script kind {invocation.Kind}"); + } + } + catch (CompilationErrorException ex) + { + return RememberCompileFailure(failureKey, $"compile error: {ex.Message}"); + } + catch (ScriptSandboxViolationException ex) + { + return RememberCompileFailure(failureKey, $"sandbox violation: {ex.Message}"); + } + catch (ScriptTimeoutException) + { + return ScriptRunOutcome.Failed($"{ScriptEvalFailureReasons.TimedOutPrefix}{invocation.Timeout.TotalSeconds:F1}s"); + } + catch (ScriptRunRefusedException ex) + { + return ScriptRunOutcome.Failed(ex.Message); + } + catch (ScriptQuarantinedException ex) + { + return ScriptRunOutcome.Failed(ex.Message); + } + catch (OperationCanceledException) when (ct.IsCancellationRequested) + { + return ScriptRunOutcome.Failed("evaluation cancelled"); + } + catch (Exception ex) + { + return ScriptRunOutcome.Failed($"script threw: {ex.Message}"); + } + } + + /// + public void Reset(ScriptContextKind kind) + { + var prefix = ScopedKey(kind, string.Empty); + foreach (var k in _compileFailures.Keys) + if (k.StartsWith(prefix, StringComparison.Ordinal)) _compileFailures.TryRemove(k, out _); + foreach (var k in _guard.QuarantinedKeys) + if (k.StartsWith(prefix, StringComparison.Ordinal)) _guard.Reset(k); + } + + /// + public IReadOnlyCollection QuarantinedKeys(ScriptContextKind kind) + { + var prefix = ScopedKey(kind, string.Empty); + return _guard.QuarantinedKeys + .Where(k => k.StartsWith(prefix, StringComparison.Ordinal)) + .Select(k => k[prefix.Length..]) + .ToArray(); + } + + /// + public void Dispose() + { + _valueCache.Dispose(); + _predicateCache.Dispose(); + } + + /// The runner-internal key: {kind}:{key}. + /// The context kind. + /// The seam's key. + /// The namespaced key. + public static string ScopedKey(ScriptContextKind kind, string key) => $"{(int)kind}:{key}"; + + private ScriptRunOutcome RememberCompileFailure(string failureKey, string reason) + { + _compileFailures.TryAdd(failureKey, reason); + return ScriptRunOutcome.Failed(reason); + } +} diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ScriptRunnerOptions.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ScriptRunnerOptions.cs new file mode 100644 index 00000000..54e03457 --- /dev/null +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ScriptRunnerOptions.cs @@ -0,0 +1,213 @@ +namespace ZB.MOM.WW.OtOpcUa.Core.Scripting; + +/// +/// Where compiled scripts run (Scripting:Runner, lmxopcua#780). is the default +/// and the documented rollback: it is today's path, unchanged. +/// +public enum ScriptRunnerMode +{ + /// Compiled scripts run in the host process under (the default). + InProcess = 0, + + /// + /// Run in process AND submit to the worker, comparing the two (design P2). Not implemented yet: a node + /// configured for it runs and logs a startup Warning. + /// + Shadow = 1, + + /// + /// Compiled scripts of the selected seams run in the pre-emptible ScriptWorker process pool. The + /// runner fails closed: a worker that cannot start makes the script Bad + /// (BadResourceUnavailable), never an in-process run. + /// + OutOfProcess = 2, +} + +/// The three host seams that run operator-authored C# (Scripting:Runner:Seams). +public enum ScriptSeam +{ + /// Virtual tags (RoslynVirtualTagEvaluator). + VirtualTag = 0, + + /// Calculation tags (CalculationEvaluator). + Calculation = 1, + + /// Scripted-alarm predicates (ScriptedAlarmEngine). Last in the rollout (design §14 #10). + ScriptedAlarm = 2, +} + +/// +/// The out-of-process worker pool's knobs (the Scripting:Worker section). Every value is clamped by +/// , which reports each out-of-range key as a startup Warning, like the rest of the +/// Scripting section. Only consulted when is not +/// . +/// +public sealed class ScriptWorkerOptions +{ + /// Default : 2 (design §14 #1). + public const int DefaultShards = 2; + + /// Upper bound of . + public const int MaxShards = 8; + + /// Default : 512 MiB. + public const long DefaultHeapLimitBytes = 512L * 1024 * 1024; + + /// Lower bound of : 64 MiB (a worker below this cannot host Roslyn). + public const long MinHeapLimitBytes = 64L * 1024 * 1024; + + /// Default : 2. + public const int DefaultQuarantineAfterKills = 2; + + /// Default : 5 s. + public static readonly TimeSpan DefaultUnavailableGrace = TimeSpan.FromSeconds(5); + + /// Lower bound of an explicit : 250 ms. + public static readonly TimeSpan MinHardGrace = TimeSpan.FromMilliseconds(250); + + /// Upper bound of an explicit : 30 s. + public static readonly TimeSpan MaxHardGrace = TimeSpan.FromSeconds(30); + + /// Upper bound of : 60 s. + public static readonly TimeSpan MaxUnavailableGrace = TimeSpan.FromSeconds(60); + + /// Default : 20. + public const int DefaultIsolationProbationRuns = 20; + + /// Shard workers per node; a script is placed by a stable hash of its key. Clamped to 1–8. + public int Shards { get; set; } = DefaultShards; + + /// + /// Grace after the soft deadline T before the worker running the script is killed. Null (the default) + /// means max(1 s, T) per run (design §14 #2); an explicit value is clamped to 250 ms – 30 s. + /// + public TimeSpan? HardGrace { get; set; } + + /// The worker's DOTNET_GCHeapHardLimit. Default 512 MiB, minimum 64 MiB. + public long HeapLimitBytes { get; set; } = DefaultHeapLimitBytes; + + /// Hard-deadline kills (within one hour) after which a script is quarantined. Default 2; 1–100. + public int QuarantineAfterKills { get; set; } = DefaultQuarantineAfterKills; + + /// + /// How long an evaluation waits for its shard to become ready (a restart, a backoff) before it reports + /// script runner unavailable. Default 5 s; 0–60 s. + /// + public TimeSpan UnavailableGrace { get; set; } = DefaultUnavailableGrace; + + /// + /// Successful runs a suspect script must complete in the isolation worker before it returns to its shard. + /// Default 20; 1–10 000. + /// + public int IsolationProbationRuns { get; set; } = DefaultIsolationProbationRuns; + + /// + /// Directory of the node-local compiled-PE cache (design §7). Must be absolute and outside the content root + /// when set; startup fails otherwise (the family's fail-closed path rule). Reserved: the cache itself has + /// not landed yet (lmxopcua#780 P1 remainder), so a worker restart recompiles. + /// + public string? PeCacheDir { get; set; } + + /// + /// Path of the worker entry assembly (ZB.MOM.WW.OtOpcUa.ScriptWorker.dll). Null (the default) means + /// script-worker/ under the host's base directory, where the Host build places it. + /// + public string? ExecutablePath { get; set; } + + /// The hard grace that applies to a run with soft budget . + /// The run's soft deadline T. + /// when set, otherwise max(1 s, T). + public TimeSpan EffectiveHardGrace(TimeSpan softTimeout) + => HardGrace ?? (softTimeout > TimeSpan.FromSeconds(1) ? softTimeout : TimeSpan.FromSeconds(1)); + + /// Returns a clamped copy; one line per out-of-range key in . + /// The clamps applied, naming each key. + /// A new, clamped instance. + public ScriptWorkerOptions Sanitized(out IReadOnlyList adjustments) + { + const string prefix = ScriptingOptions.SectionName + ":Worker:"; + var notes = new List(); + + var shards = Shards; + if (shards < 1 || shards > MaxShards) + { + var clamped = Math.Clamp(shards, 1, MaxShards); + notes.Add($"{prefix}{nameof(Shards)} {shards} is outside 1–{MaxShards}; clamped to {clamped}."); + shards = clamped; + } + + var grace = HardGrace; + if (grace is { } g && (g < MinHardGrace || g > MaxHardGrace)) + { + var clamped = g < MinHardGrace ? MinHardGrace : MaxHardGrace; + notes.Add($"{prefix}{nameof(HardGrace)} {g} is outside {MinHardGrace}–{MaxHardGrace}; clamped to {clamped}."); + grace = clamped; + } + + var heap = HeapLimitBytes; + if (heap < MinHeapLimitBytes) + { + notes.Add($"{prefix}{nameof(HeapLimitBytes)} {heap} is below the minimum; clamped to {MinHeapLimitBytes}."); + heap = MinHeapLimitBytes; + } + + var kills = QuarantineAfterKills; + if (kills < 1 || kills > 100) + { + var clamped = Math.Clamp(kills, 1, 100); + notes.Add($"{prefix}{nameof(QuarantineAfterKills)} {kills} is outside 1–100; clamped to {clamped}."); + kills = clamped; + } + + var unavailable = UnavailableGrace; + if (unavailable < TimeSpan.Zero || unavailable > MaxUnavailableGrace) + { + var clamped = unavailable < TimeSpan.Zero ? TimeSpan.Zero : MaxUnavailableGrace; + notes.Add($"{prefix}{nameof(UnavailableGrace)} {unavailable} is outside 0–{MaxUnavailableGrace}; clamped to {clamped}."); + unavailable = clamped; + } + + var probation = IsolationProbationRuns; + if (probation < 1 || probation > 10_000) + { + var clamped = Math.Clamp(probation, 1, 10_000); + notes.Add($"{prefix}{nameof(IsolationProbationRuns)} {probation} is outside 1–10000; clamped to {clamped}."); + probation = clamped; + } + + adjustments = notes; + return new ScriptWorkerOptions + { + Shards = shards, + HardGrace = grace, + HeapLimitBytes = heap, + QuarantineAfterKills = kills, + UnavailableGrace = unavailable, + IsolationProbationRuns = probation, + PeCacheDir = PeCacheDir, + ExecutablePath = ExecutablePath, + }; + } + + /// + /// Fail-closed path validation (design §7): , when set, must be absolute and must not + /// sit inside . Same rule as the family's Secrets:SqlitePath. + /// + /// The host's content root. + /// The violation, or null when the options are acceptable. + public string? ValidatePaths(string contentRootPath) + { + if (string.IsNullOrWhiteSpace(PeCacheDir)) return null; + const string key = ScriptingOptions.SectionName + ":Worker:" + nameof(PeCacheDir); + if (!Path.IsPathFullyQualified(PeCacheDir)) + return $"{key} '{PeCacheDir}' must be an absolute path."; + var dir = Path.TrimEndingDirectorySeparator(Path.GetFullPath(PeCacheDir)); + var root = Path.TrimEndingDirectorySeparator(Path.GetFullPath(contentRootPath)); + var comparison = OperatingSystem.IsWindows() ? StringComparison.OrdinalIgnoreCase : StringComparison.Ordinal; + if (string.Equals(dir, root, comparison) + || dir.StartsWith(root + Path.DirectorySeparatorChar, comparison) + || dir.StartsWith(root + Path.AltDirectorySeparatorChar, comparison)) + return $"{key} '{PeCacheDir}' is inside the content root '{contentRootPath}'; it must live outside it."; + return null; + } +} diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ScriptingOptions.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ScriptingOptions.cs index b6a4e48f..bae47f1f 100644 --- a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ScriptingOptions.cs +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ScriptingOptions.cs @@ -72,6 +72,30 @@ public sealed class ScriptingOptions /// public double StuckOrphanQuarantineFactor { get; set; } = DefaultStuckOrphanQuarantineFactor; + /// + /// Where compiled scripts run (Scripting:Runner, lmxopcua#780). Default + /// — today's path, unchanged, and the rollback for the other modes. + /// + public ScriptRunnerMode Runner { get; set; } = ScriptRunnerMode.InProcess; + + /// + /// The seams applies to when it is not + /// (Scripting:Runner:Seams, e.g. ["Calculation"]). Null or empty means all three. Bound by + /// ScriptingOptionsBinding from the documented key, because Scripting:Runner is itself a value. + /// + public IReadOnlyList? RunnerSeams { get; set; } + + /// The out-of-process worker pool's knobs (Scripting:Worker). + public ScriptWorkerOptions Worker { get; set; } = new(); + + /// Whether is selected for out-of-process execution by these options. + /// The seam. + /// true when is and the seam is in + /// (or that list is empty). + public bool IsOutOfProcess(ScriptSeam seam) + => Runner == ScriptRunnerMode.OutOfProcess + && (RunnerSeams is not { Count: > 0 } seams || seams.Contains(seam)); + /// Upper bound applies to . public const double MaxStuckOrphanQuarantineFactor = 1000; @@ -158,6 +182,9 @@ public sealed class ScriptingOptions runTimeout = MaxRunTimeout; } + var worker = (Worker ?? new ScriptWorkerOptions()).Sanitized(out var workerNotes); + notes.AddRange(workerNotes); + adjustments = notes; return new ScriptingOptions { @@ -165,6 +192,9 @@ public sealed class ScriptingOptions MaxAllocBytesPerRun = alloc, QuarantineAfterConsecutiveTimeouts = threshold, StuckOrphanQuarantineFactor = factor, + Runner = Runner, + RunnerSeams = RunnerSeams is null ? null : [.. RunnerSeams], + Worker = worker, }; } } diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationDriver.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationDriver.cs index 9f8ef41b..1a12ae7c 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationDriver.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationDriver.cs @@ -137,7 +137,7 @@ public sealed class CalculationDriver : IDriver, IDependencyConsumer, ISubscriba // _logger (ILogger) implements the non-generic ILogger, so it is passed straight // through — the evaluator's host-side diagnostics (compile errors, sandbox violations, execution // throws) land on the SAME logger the driver itself was constructed with, not a separate one. - _evaluator = new CalculationEvaluator(_logger, _scriptRoot, _options.RunTimeout, _options.Scripting); + _evaluator = new CalculationEvaluator(_logger, _scriptRoot, _options.RunTimeout, _options.Scripting, _options.ScriptRunner); RebuildTagTable(_options); } @@ -639,7 +639,7 @@ public sealed class CalculationDriver : IDriver, IDependencyConsumer, ISubscriba if (!result.Success) { var reason = result.Reason ?? "evaluator failure"; - toPublish = MarkEvaluationFailed(st, pending.TimestampUtc); + toPublish = MarkEvaluationFailed(st, pending.TimestampUtc, result.StatusCode ?? StatusBadInternalError); // The script-log entry fires on the Good→Bad TRANSITION, or when the failure changes // (R07:P4) — not on every trigger of a tag that is already Bad for the same reason, which // flooded the script log at the input's cadence. The Bad publish was already once per @@ -725,16 +725,19 @@ public sealed class CalculationDriver : IDriver, IDependencyConsumer, ISubscriba /// returns before the evaluator ever runs), so there is no separate "inputs ready" condition here. /// The tag's published state. /// The trigger's timestamp. + /// The Bad code to publish (see the body). /// The Bad snapshot to publish on a fresh Good→Bad transition, else null. - private static (object? Value, uint StatusCode)? MarkEvaluationFailed(TagState st, DateTime timestampUtc) + private static (object? Value, uint StatusCode)? MarkEvaluationFailed(TagState st, DateTime timestampUtc, uint badStatus) { - if (st.HasPublished && st.LastPublishedStatus == StatusBadInternalError) return null; + // badStatus is BadInternalError for every script failure; BadResourceUnavailable when the out-of-process + // runner could not run the script (lmxopcua#780). Once per transition, keyed on that code. + if (st.HasPublished && st.LastPublishedStatus == badStatus) return null; st.HasPublished = true; - st.LastPublishedStatus = StatusBadInternalError; + st.LastPublishedStatus = badStatus; st.LastTimestampUtc = timestampUtc; // Carry the last-known value (null if none) with Bad quality. - return (st.HasValue ? st.LastValue : null, StatusBadInternalError); + return (st.HasValue ? st.LastValue : null, badStatus); } private void Publish(string rawPath, object? value, uint statusCode, DateTime timestampUtc) diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationDriverFactoryExtensions.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationDriverFactoryExtensions.cs index 2c417de1..3bc8dac6 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationDriverFactoryExtensions.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationDriverFactoryExtensions.cs @@ -40,14 +40,17 @@ public static class CalculationDriverFactoryExtensions /// Optional root script logger (fans failure entries onto the script-logs topic). /// The node-wide Scripting options every Calculation driver inherits; a key /// authored in a driver's own config overrides that one knob. Null ⇒ the compiled defaults. + /// lmxopcua#780 — the node's out-of-process runner when the Calculation seam runs out + /// of process; null (the default) keeps every driver on the in-process guard. public static void Register( DriverFactoryRegistry registry, ILoggerFactory? loggerFactory = null, ScriptRootLogger? scriptRoot = null, - ScriptingOptions? scripting = null) + ScriptingOptions? scripting = null, + IScriptRunner? scriptRunner = null) { ArgumentNullException.ThrowIfNull(registry); - registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory, scriptRoot, scripting)); + registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory, scriptRoot, scripting, scriptRunner)); } /// Public overload for the Server-side bootstrapper + test consumers. @@ -74,10 +77,12 @@ public static class CalculationDriverFactoryExtensions /// Optional logger factory for per-instance diagnostics loggers. /// Optional root script logger for failure fan-out. /// The node-wide run-guard options; null ⇒ the compiled defaults. + /// The node's out-of-process runner, or null for in process. /// The constructed . public static CalculationDriver CreateInstance( string driverInstanceId, string driverConfigJson, - ILoggerFactory? loggerFactory, ScriptRootLogger? scriptRoot, ScriptingOptions? nodeScripting) + ILoggerFactory? loggerFactory, ScriptRootLogger? scriptRoot, ScriptingOptions? nodeScripting, + IScriptRunner? scriptRunner = null) { ArgumentException.ThrowIfNullOrWhiteSpace(driverInstanceId); ArgumentException.ThrowIfNullOrWhiteSpace(driverConfigJson); @@ -101,6 +106,7 @@ public static class CalculationDriverFactoryExtensions RunTimeout = dto.RunTimeoutMs is { } ms && ms > 0 ? TimeSpan.FromMilliseconds(ms) : TimeSpan.FromSeconds(2), Scripting = scripting, + ScriptRunner = scriptRunner, }; return new CalculationDriver(options, driverInstanceId, scriptRoot, logger: logger); diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationDriverOptions.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationDriverOptions.cs index f00f50e4..28e12055 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationDriverOptions.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationDriverOptions.cs @@ -18,4 +18,11 @@ public sealed class CalculationDriverOptions /// defaults (64 MB, 3). /// public ScriptingOptions? Scripting { get; init; } + + /// + /// lmxopcua#780 — the node's out-of-process runner when Scripting:Runner=OutOfProcess selects the + /// Calculation seam; null (the default) keeps the in-process run guard. Shared by every Calculation driver on the + /// node; keys are the calc tags' RawPaths. + /// + public IScriptRunner? ScriptRunner { get; init; } } diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationEvaluator.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationEvaluator.cs index dd7085e3..6af66db6 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationEvaluator.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/CalculationEvaluator.cs @@ -59,6 +59,7 @@ public sealed class CalculationEvaluator : IScriptCacheOwner, IScriptQuarantineO private readonly SerilogLogger _scriptRoot; private readonly TimeSpan _runTimeout; private readonly ScriptRunGuard _guard; + private readonly IScriptRunner? _runner; // Negative compile cache (R07:P4), keyed by source hash. A compile error is a property of the SOURCE, // which never changes between triggers (it is fixed in the deploy artifact) — yet every trigger of a @@ -76,12 +77,16 @@ public sealed class CalculationEvaluator : IScriptCacheOwner, IScriptQuarantineO /// Root script logger; user ctx.Logger.* output flows through this to the Script-log page. /// Wall-clock budget per script run; defaults to 2 seconds (parity with the VT evaluator). /// Run-guard budget + quarantine threshold; null ⇒ defaults. + /// lmxopcua#780 — the out-of-process runner when the Calculation seam runs out of + /// process; null (the default) keeps the in-process guard path, unchanged. public CalculationEvaluator( ILogger logger, ScriptRootLogger scriptRoot, TimeSpan? runTimeout = null, - ScriptingOptions? scripting = null) + ScriptingOptions? scripting = null, + IScriptRunner? scriptRunner = null) { + _runner = scriptRunner; _logger = logger ?? throw new ArgumentNullException(nameof(logger)); _scriptRoot = (scriptRoot ?? throw new ArgumentNullException(nameof(scriptRoot))).Logger; _runTimeout = runTimeout ?? TimeSpan.FromSeconds(2); @@ -188,6 +193,15 @@ public sealed class CalculationEvaluator : IScriptCacheOwner, IScriptQuarantineO calcTagId, path), logger: scriptLog); + // lmxopcua#780 — OutOfProcess: the pre-emptible worker pool compiles and runs it, and owns timeouts, kills, + // quarantine and the negative compile cache for this seam. Unavailable publishes BadResourceUnavailable. + if (_runner is not null) + { + var outcome = await _runner.RunAsync(new ScriptInvocation( + calcTagId, ScriptContextKind.Calculation, scriptSource, readCache, _runTimeout, scriptLog), ct).ConfigureAwait(false); + return outcome.ToVirtualTagEvalResult(); + } + // A source that already failed to compile fails the same way without touching Roslyn (R07:P4). var sourceKey = HashSource(scriptSource); if (_compileFailures.TryGetValue(sourceKey, out var knownFailure)) @@ -295,13 +309,20 @@ public sealed class CalculationEvaluator : IScriptCacheOwner, IScriptQuarantineO // The compile-cache clear is this evaluator's deploy boundary — a redeploy lifts every quarantine // (an edited source already lifts its own through the guard's source identity). _guard.ResetAll(); + _runner?.Reset(ScriptContextKind.Calculation); } /// - public void ResetQuarantines() => _guard.ResetAll(); + public void ResetQuarantines() + { + _guard.ResetAll(); + _runner?.Reset(ScriptContextKind.Calculation); + } /// - public IReadOnlyCollection QuarantinedScripts => _guard.QuarantinedKeys; + public IReadOnlyCollection QuarantinedScripts => _runner is null + ? _guard.QuarantinedKeys + : [.. _guard.QuarantinedKeys, .. _runner.QuarantinedKeys(ScriptContextKind.Calculation)]; /// Disposes the evaluator, draining + disposing every cached compile's collectible ALC. public void Dispose() diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/ZB.MOM.WW.OtOpcUa.Driver.Calculation.csproj b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/ZB.MOM.WW.OtOpcUa.Driver.Calculation.csproj index ea2926ef..631789f7 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/ZB.MOM.WW.OtOpcUa.Driver.Calculation.csproj +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation/ZB.MOM.WW.OtOpcUa.Driver.Calculation.csproj @@ -30,6 +30,8 @@ + + diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ScriptRunnerSelection.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ScriptRunnerSelection.cs new file mode 100644 index 00000000..19c7b46c --- /dev/null +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ScriptRunnerSelection.cs @@ -0,0 +1,94 @@ +using ZB.MOM.WW.OtOpcUa.Core.Scripting; +using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess; +using SerilogLogger = Serilog.ILogger; + +namespace ZB.MOM.WW.OtOpcUa.Host.Configuration; + +/// +/// Which each script seam uses on this node (lmxopcua#780, Scripting:Runner). Under +/// the default every seam gets null and keeps its in-process guard path, +/// unchanged. Under the selected seams share one +/// , which starts no worker until the first compiled evaluation, so a node with +/// no scripts (or an admin-only node) never launches one. Disposed with the host's service provider, which stops the +/// workers. +/// +public sealed class ScriptRunnerSelection : IAsyncDisposable, IDisposable +{ + /// Seams the out-of-process runner can serve in this build. Scripted alarms go last (design §14 #10). + public static readonly IReadOnlyList SupportedOutOfProcessSeams = [ScriptSeam.VirtualTag, ScriptSeam.Calculation]; + + private readonly OutOfProcessScriptRunner? _runner; + private readonly HashSet _seams; + + private ScriptRunnerSelection(OutOfProcessScriptRunner? runner, IEnumerable seams) + { + _runner = runner; + _seams = [.. seams]; + } + + /// The out-of-process runner, when one is configured. + public IScriptRunner? OutOfProcessRunner => _runner; + + /// The runner must use, or null for its in-process path. + /// The seam. + /// The runner, or null. + public IScriptRunner? ForSeam(ScriptSeam seam) => _runner is not null && _seams.Contains(seam) ? _runner : null; + + /// + /// Startup checks for the runner options: returns the Warnings to log, and throws when the configuration must + /// not start (a PeCacheDir that is relative or inside the content root — fail closed). + /// + /// The bound, sanitized Scripting options. + /// The host's content root. + /// Warnings for the caller to log. + /// The worker path options are unacceptable. + public static IReadOnlyList Validate(ScriptingOptions options, string contentRootPath) + { + ArgumentNullException.ThrowIfNull(options); + var warnings = new List(); + if (options.Runner == ScriptRunnerMode.InProcess) return warnings; + + if (options.Worker.ValidatePaths(contentRootPath) is { } violation) + throw new InvalidOperationException($"Scripting misconfiguration: {violation}"); + + if (options.Runner == ScriptRunnerMode.Shadow) + { + warnings.Add("Scripting:Runner=Shadow is not implemented yet (lmxopcua#780 P2); every seam runs InProcess."); + return warnings; + } + + foreach (var seam in RequestedSeams(options)) + { + if (!SupportedOutOfProcessSeams.Contains(seam)) + warnings.Add($"Scripting:Runner=OutOfProcess: the {seam} seam cannot run out of process yet (lmxopcua#780); it stays InProcess."); + } + var workerPath = OutOfProcessScriptRunner.ResolveWorkerPath(options.Worker); + if (!File.Exists(workerPath)) + warnings.Add($"Scripting:Runner=OutOfProcess: the script worker '{workerPath}' does not exist; every out-of-process " + + "script reports BadResourceUnavailable (the runner fails closed, it never falls back to in process)."); + return warnings; + } + + /// Builds the selection for . + /// The bound, sanitized Scripting options. + /// Host diagnostics for the runner. + /// The root script logger forwarded ctx.Logger lines fall back to. + /// The selection. + public static ScriptRunnerSelection Create(ScriptingOptions options, SerilogLogger log, SerilogLogger? scriptRoot) + { + ArgumentNullException.ThrowIfNull(options); + if (options.Runner != ScriptRunnerMode.OutOfProcess) return new ScriptRunnerSelection(null, []); + var seams = RequestedSeams(options).Where(SupportedOutOfProcessSeams.Contains).ToArray(); + if (seams.Length == 0) return new ScriptRunnerSelection(null, []); + return new ScriptRunnerSelection(new OutOfProcessScriptRunner(options, log, scriptRoot), seams); + } + + /// + public ValueTask DisposeAsync() => _runner?.DisposeAsync() ?? ValueTask.CompletedTask; + + /// + public void Dispose() => _runner?.Dispose(); + + private static IEnumerable RequestedSeams(ScriptingOptions options) + => options.RunnerSeams is { Count: > 0 } seams ? seams.Distinct() : Enum.GetValues(); +} diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ScriptingOptionsBinding.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ScriptingOptionsBinding.cs index 1155748b..b2e0517e 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ScriptingOptionsBinding.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Configuration/ScriptingOptionsBinding.cs @@ -18,6 +18,11 @@ public static class ScriptingOptionsBinding { ArgumentNullException.ThrowIfNull(configuration); var bound = configuration.GetSection(ScriptingOptions.SectionName).Get() ?? new ScriptingOptions(); + // lmxopcua#780: the seam subset lives under the runner key (Scripting:Runner:Seams), which the binder cannot + // reach because Scripting:Runner is itself a value. Bound explicitly; an unknown seam name is a startup error, + // like an unknown Scripting:Runner value. + var seams = configuration.GetSection($"{ScriptingOptions.SectionName}:Runner:Seams").Get>(); + if (seams is { Count: > 0 }) bound.RunnerSeams = seams; return bound.Sanitized(out adjustments); } } diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs index 792f0253..429548b4 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Drivers/DriverFactoryBootstrap.cs @@ -102,7 +102,10 @@ public static class DriverFactoryBootstrap // failures; without this it writes to NullLogger in production. ConfigureAbTransportLogging(sp); Register(registry, loggerFactory, secretResolver, primaryRoleSource, ResolveAbCipDefaultTransport(sp), - sp.GetRequiredService(), scriptRoot, scripting); + sp.GetRequiredService(), scriptRoot, scripting, + // lmxopcua#780: null under the InProcess default (and in test graphs that register no selection). + sp.GetService() + ?.ForSeam(ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptSeam.Calculation)); return registry; }); services.AddSingleton(sp => @@ -233,11 +236,12 @@ public static class DriverFactoryBootstrap Driver.AbCip.AbCipTransport abCipDefaultTransport, Driver.Galaxy.Runtime.IGalaxyAckIdentityDirectory galaxyAckIdentities, ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptRootLogger? scriptRoot = null, - ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptingOptions? scripting = null) + ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptingOptions? scripting = null, + ZB.MOM.WW.OtOpcUa.Core.Scripting.IScriptRunner? scriptRunner = null) { Driver.AbCip.AbCipDriverFactoryExtensions.Register(registry, loggerFactory, abCipDefaultTransport); Driver.AbLegacy.AbLegacyDriverFactoryExtensions.Register(registry, loggerFactory); - Driver.Calculation.CalculationDriverFactoryExtensions.Register(registry, loggerFactory, scriptRoot, scripting); + Driver.Calculation.CalculationDriverFactoryExtensions.Register(registry, loggerFactory, scriptRoot, scripting, scriptRunner); Driver.FOCAS.FocasDriverFactoryExtensions.Register(registry, loggerFactory); Driver.Galaxy.GalaxyDriverFactoryExtensions.Register(registry, secretResolver, loggerFactory, galaxyAckIdentities); Driver.Modbus.ModbusDriverFactoryExtensions.Register(registry, loggerFactory); diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Engines/RoslynVirtualTagEvaluator.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Engines/RoslynVirtualTagEvaluator.cs index 73752265..cc2c5095 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Engines/RoslynVirtualTagEvaluator.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Engines/RoslynVirtualTagEvaluator.cs @@ -53,6 +53,7 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac private readonly SerilogLogger _scriptRoot; private readonly TimeSpan _runTimeout; private readonly ScriptRunGuard _guard; + private readonly IScriptRunner? _runner; private bool _disposed; /// Initializes a new RoslynVirtualTagEvaluator with the given loggers and optional timeout. @@ -61,18 +62,28 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac /// Maximum execution time for each script; null ⇒ /// of (2 s by default). /// Run-guard budget + quarantine threshold (the Scripting section); null ⇒ defaults. + /// + /// lmxopcua#780 — the out-of-process runner when Scripting:Runner=OutOfProcess selects the VirtualTag seam; + /// null (the default, InProcess) keeps the in-process guard path below, unchanged. The passthrough fast path + /// never reaches it. + /// public RoslynVirtualTagEvaluator( ILogger logger, ScriptRootLogger scriptRoot, TimeSpan? runTimeout = null, - ScriptingOptions? scripting = null) + ScriptingOptions? scripting = null, + IScriptRunner? scriptRunner = null) { _logger = logger; _scriptRoot = (scriptRoot ?? throw new ArgumentNullException(nameof(scriptRoot))).Logger; _runTimeout = runTimeout ?? (scripting ?? new ScriptingOptions()).Sanitized(out _).RunTimeout; _guard = new ScriptRunGuard(scripting, _scriptRoot); + _runner = scriptRunner; } + /// The out-of-process runner this evaluator delegates compiled runs to, or null (in process). + public IScriptRunner? ScriptRunner => _runner; + /// The run guard every compiled evaluation goes through (diagnostics / tests). public ScriptRunGuard RunGuard => _guard; @@ -147,6 +158,16 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac virtualTagId, path), logger: scriptLog); + // lmxopcua#780 — OutOfProcess: the compiled run goes to the pre-emptible worker pool, which compiles in the + // worker and owns timeouts, kills and quarantine for this seam. An Unavailable outcome (worker restarted, no + // worker ready) publishes BadResourceUnavailable; everything else keeps today's Bad contract. + if (_runner is not null) + { + var outcome = await _runner.RunAsync(new ScriptInvocation( + virtualTagId, ScriptContextKind.VirtualTag, expression, readCache, _runTimeout, scriptLog)).ConfigureAwait(false); + return outcome.ToVirtualTagEvalResult(); + } + // 02/S12: fetch + run inside a bounded loop so an ObjectDisposedException caused by a concurrent // apply-boundary ClearCompiledScripts (which disposed the evaluator between GetOrCompile and the // run's disposed guard) re-fetches and retries exactly once. The loop runs at most twice — the @@ -235,13 +256,20 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac // The compile-cache clear IS the deploy boundary for this evaluator: a redeploy lifts every quarantine // (an edited expression already lifts its own through the guard's source identity). _guard.ResetAll(); + _runner?.Reset(ScriptContextKind.VirtualTag); } /// - public void ResetQuarantines() => _guard.ResetAll(); + public void ResetQuarantines() + { + _guard.ResetAll(); + _runner?.Reset(ScriptContextKind.VirtualTag); + } /// - public IReadOnlyCollection QuarantinedScripts => _guard.QuarantinedKeys; + public IReadOnlyCollection QuarantinedScripts => _runner is null + ? _guard.QuarantinedKeys + : [.. _guard.QuarantinedKeys, .. _runner.QuarantinedKeys(ScriptContextKind.VirtualTag)]; /// Disposes the evaluator and cleans up all cached script evaluators. public void Dispose() diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs index fc3cb39a..4277552e 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/Program.cs @@ -376,6 +376,16 @@ if (hasDriver) Log.Warning("Scripting misconfiguration detected at startup: {Adjustment}", adjustment); builder.Services.AddSingleton(scriptingOptions); + // lmxopcua#780: Scripting:Runner (InProcess default / Shadow / OutOfProcess). Under the default every seam keeps its + // in-process guard path unchanged; OutOfProcess hands the selected seams' compiled runs to the pre-emptible + // ScriptWorker pool. A PeCacheDir that is relative or inside the content root stops startup (fail closed). + foreach (var warning in ZB.MOM.WW.OtOpcUa.Host.Configuration.ScriptRunnerSelection.Validate( + scriptingOptions, builder.Environment.ContentRootPath)) + Log.Warning("Scripting runner: {RunnerWarning}", warning); + Log.Information("Scripting runner: {RunnerMode}", scriptingOptions.Runner); + builder.Services.AddSingleton(sp => ZB.MOM.WW.OtOpcUa.Host.Configuration.ScriptRunnerSelection.Create( + sp.GetRequiredService(), Log.Logger, sp.GetService()?.Logger)); + // lmxopcua#777: the ONE operator-write timeout (DriverWrites:Timeout) every write bound is derived from — the // driver child's cancel + abandon, its queue budget, DriverHostActor's Ask and the inbound-write gateway's Ask. // Checked against the resilience tiers' Write timeouts here; an inconsistency is a startup Warning (a tier @@ -392,7 +402,9 @@ if (hasDriver) new RoslynVirtualTagEvaluator( sp.GetRequiredService().CreateLogger(), sp.GetRequiredService(), - scripting: sp.GetRequiredService())); + scripting: sp.GetRequiredService(), + scriptRunner: sp.GetRequiredService() + .ForSeam(ScriptSeam.VirtualTag))); builder.Services.AddSingleton(sp => sp.GetRequiredService()); // Script-log fan-out (Layer 0). The DPS publisher resolves the ActorSystem lazily so it never diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj b/src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj index 34a25433..159ff0bb 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj @@ -57,6 +57,7 @@ + @@ -90,4 +91,7 @@ + + + diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagActor.cs b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagActor.cs index 0d9e0cf8..f55bc965 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagActor.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagActor.cs @@ -391,7 +391,8 @@ public sealed class VirtualTagActor : ReceiveActor, IWithUnboundedStash var result = evaluation.Result; if (!result.Success) { - OnEvaluationFailed(result.Reason ?? "evaluator failure", "Warning", timestampUtc); + OnEvaluationFailed(result.Reason ?? "evaluator failure", "Warning", timestampUtc, + result.StatusCode ?? OpcUaStatus.BadInternalError); return; } @@ -472,7 +473,10 @@ public sealed class VirtualTagActor : ReceiveActor, IWithUnboundedStash /// The failure reason for the script-log entry. /// The script-log level ("Warning" for a Failure result, "Error" for a throw). /// The triggering dependency change's timestamp, stamped on the Bad result. - private void OnEvaluationFailed(string reason, string level, DateTime timestampUtc) + /// The Bad code to publish: for every script + /// failure, when the out-of-process runner could not run the + /// script (lmxopcua#780). The once-per-transition dedup is on this code, so a switch between the two publishes. + private void OnEvaluationFailed(string reason, string level, DateTime timestampUtc, uint badStatus = OpcUaStatus.BadInternalError) { OtOpcUaTelemetry.VirtualTagEval.Add(1, new KeyValuePair("outcome", "fail")); @@ -488,15 +492,15 @@ public sealed class VirtualTagActor : ReceiveActor, IWithUnboundedStash } if (admission.Admitted) PublishLog(level, reason); - if ((_hasPublished && _lastPublishedStatus == OpcUaStatus.BadInternalError) || !AllDependenciesArrived()) + if ((_hasPublished && _lastPublishedStatus == badStatus) || !AllDependenciesArrived()) { return; } _hasPublished = true; - _lastPublishedStatus = OpcUaStatus.BadInternalError; + _lastPublishedStatus = badStatus; Context.Parent.Tell(new EvaluationResult( - _virtualTagId, _hasLastValue ? _lastValue : null, timestampUtc, CorrelationId.NewId(), OpcUaStatus.BadInternalError)); + _virtualTagId, _hasLastValue ? _lastValue : null, timestampUtc, CorrelationId.NewId(), badStatus)); } /// Inputs-ready gate (the retired cascade engine's rule): true once diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker/Program.cs b/src/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker/Program.cs new file mode 100644 index 00000000..6ffa161e --- /dev/null +++ b/src/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker/Program.cs @@ -0,0 +1,36 @@ +using System.IO.Pipes; +using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol; + +namespace ZB.MOM.WW.OtOpcUa.ScriptWorker; + +/// Entry point: connect to the host's pipe named in the environment and serve until told to stop. +internal static class Program +{ + private static async Task Main() + { + var pipeName = Environment.GetEnvironmentVariable(ScriptWorkerEnvironment.PipeName); + var nonce = Environment.GetEnvironmentVariable(ScriptWorkerEnvironment.Nonce); + if (string.IsNullOrEmpty(pipeName) || string.IsNullOrEmpty(nonce)) + { + await Console.Error.WriteLineAsync("ScriptWorker: launched without a pipe name or nonce; exiting."); + return ScriptWorkerEnvironment.ExitProtocolFault; + } + _ = int.TryParse(Environment.GetEnvironmentVariable(ScriptWorkerEnvironment.ParentPid), out var parentPid); + + // CurrentUserOnly: the client verifies the server end is owned by this account as well. + var pipe = new NamedPipeClientStream(".", pipeName, PipeDirection.InOut, + PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly); + try + { + await pipe.ConnectAsync(TimeSpan.FromSeconds(10), CancellationToken.None); + } + catch (Exception ex) + { + await Console.Error.WriteLineAsync($"ScriptWorker: could not connect to the host pipe: {ex.Message}"); + return ScriptWorkerEnvironment.ExitProtocolFault; + } + + var host = new WorkerHost(pipe, nonce, parentPid); + return await host.RunAsync(); + } +} diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker/WorkerHost.cs b/src/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker/WorkerHost.cs new file mode 100644 index 00000000..858f0328 --- /dev/null +++ b/src/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker/WorkerHost.cs @@ -0,0 +1,576 @@ +using System.Collections; +using System.Collections.Concurrent; +using System.Diagnostics; +using System.IO.Pipes; +using System.Threading.Channels; +using Serilog; +using Serilog.Core; +using Serilog.Events; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Core.Scripting; +using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol; +using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1; +using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms; +using ZB.MOM.WW.OtOpcUa.Core.VirtualTags; + +namespace ZB.MOM.WW.OtOpcUa.ScriptWorker; + +/// +/// One worker's whole life (lmxopcua#780, design §3.2): handshake, compile-on-load, a bounded evaluation scheduler +/// on dedicated threads, a hard-deadline watchdog, heartbeats, and log forwarding. The host is authoritative for +/// deadlines (it kills this process); the watchdog is the belt to that brace and fails fast slightly later, so the +/// exit names the key that overran. +/// +/// +/// Anything that means "the host is gone" — pipe end of stream, a protocol fault, the parent pid disappearing — +/// ends the process. The worker never outlives its host (design §3.3). +/// +internal sealed class WorkerHost +{ + /// The watchdog fails fast this long after the host's own hard deadline, so the host normally wins. + private static readonly TimeSpan WatchdogMargin = TimeSpan.FromMilliseconds(500); + + private readonly NamedPipeClientStream _pipe; + private readonly string _nonce; + private readonly int _parentPid; + private readonly Channel _outbound = Channel.CreateUnbounded( + new UnboundedChannelOptions { SingleReader = true }); + private readonly ConcurrentDictionary _scripts = new(StringComparer.Ordinal); + private readonly ConcurrentDictionary _running = new(); + private readonly BlockingCollection _queue = new(new ConcurrentQueue()); + private readonly ForwardingLogSink _logSink; + private readonly ILogger _scriptLogger; + private int _maxFrameBytes = ScriptWorkerFrameCodec.DefaultMaxFrameBytes; + private int _queueCapacity = 4096; + private TimeSpan _hardGrace = TimeSpan.FromSeconds(1); + private TimeSpan _heartbeatInterval = TimeSpan.FromSeconds(1); + private long _seq; + + public WorkerHost(NamedPipeClientStream pipe, string nonce, int parentPid) + { + _pipe = pipe; + _nonce = nonce; + _parentPid = parentPid; + _logSink = new ForwardingLogSink(); + _scriptLogger = new LoggerConfiguration().MinimumLevel.Verbose().WriteTo.Sink(_logSink).CreateLogger(); + } + + public async Task RunAsync() + { + try + { + var hello = await ScriptWorkerFrameCodec.ReadAsync(_pipe, ScriptWorkerFrameCodec.DefaultMaxFrameBytes); + if (hello is null) return ScriptWorkerEnvironment.ExitPipeClosed; + if (hello.BodyCase != Envelope.BodyOneofCase.Hello) + throw new ScriptWorkerProtocolException($"expected Hello, got {hello.BodyCase}"); + Configure(hello.Hello); + + // The nonce is echoed, not trusted: the host compares it with the one it put in OUR environment, which + // proves the process on the other end of its pipe is the one it launched. + await ScriptWorkerFrameCodec.WriteAsync(_pipe, Wrap(new Envelope + { + Ready = new Ready + { + Nonce = _nonce, + Pid = Environment.ProcessId, + RuntimeVersion = Environment.Version.ToString(), + }, + }), _maxFrameBytes); + + _logSink.Capacity = (int)Math.Max(1, hello.Hello.MaxLogLines == 0 ? 1000 : hello.Hello.MaxLogLines); + StartBackground(Math.Max(1, (int)hello.Hello.EvalThreads)); + var writer = Task.Run(WriteLoopAsync); + var exit = await ReadLoopAsync(); + _outbound.Writer.TryComplete(); + try { await writer.WaitAsync(TimeSpan.FromSeconds(1)); } catch { /* exiting regardless */ } + return exit; + } + catch (ScriptWorkerProtocolException ex) + { + await Console.Error.WriteLineAsync($"ScriptWorker: protocol fault: {ex.Message}"); + return ScriptWorkerEnvironment.ExitProtocolFault; + } + catch (IOException) + { + return ScriptWorkerEnvironment.ExitPipeClosed; + } + } + + private void Configure(Hello hello) + { + if (hello.MaxFrameBytes > 0) _maxFrameBytes = (int)Math.Min(hello.MaxFrameBytes, 64u * 1024 * 1024); + if (hello.QueueCapacity > 0) _queueCapacity = (int)hello.QueueCapacity; + if (hello.HardGraceMs > 0) _hardGrace = TimeSpan.FromMilliseconds(hello.HardGraceMs); + if (hello.HeartbeatIntervalMs > 0) _heartbeatInterval = TimeSpan.FromMilliseconds(hello.HeartbeatIntervalMs); + } + + private void StartBackground(int evalThreads) + { + for (var i = 0; i < evalThreads; i++) + { + new Thread(EvalThreadLoop) { IsBackground = true, Name = $"ScriptWorker.Eval{i}" }.Start(); + } + new Thread(WatchdogLoop) { IsBackground = true, Name = "ScriptWorker.Watchdog" }.Start(); + _ = Task.Run(HeartbeatLoopAsync); + _ = Task.Run(LogFlushLoopAsync); + if (_parentPid > 0) _ = Task.Run(ParentWatchLoopAsync); + } + + private async Task ReadLoopAsync() + { + while (true) + { + Envelope? frame; + try + { + frame = await ScriptWorkerFrameCodec.ReadAsync(_pipe, _maxFrameBytes); + } + catch (ScriptWorkerProtocolException ex) + { + await Console.Error.WriteLineAsync($"ScriptWorker: protocol fault: {ex.Message}"); + return ScriptWorkerEnvironment.ExitProtocolFault; + } + catch (IOException) + { + return ScriptWorkerEnvironment.ExitPipeClosed; + } + if (frame is null) return ScriptWorkerEnvironment.ExitPipeClosed; // host gone: fatal by design + + switch (frame.BodyCase) + { + case Envelope.BodyOneofCase.Load: + { + // Compile off the reader, so evaluations for already-loaded keys keep flowing during a load. + var load = frame.Load; + _ = Task.Run(() => Load(load)); + break; + } + case Envelope.BodyOneofCase.Unload: + foreach (var key in frame.Unload.Keys) + if (_scripts.TryRemove(key, out var gone)) gone.Dispose(); + break; + case Envelope.BodyOneofCase.Eval: + foreach (var item in frame.Eval.Items) Admit(item); + break; + case Envelope.BodyOneofCase.Cancel: + if (_running.TryGetValue((long)frame.Cancel.EvalId, out var toCancel)) + { + try { toCancel.Cancellation.Cancel(); } catch (ObjectDisposedException) { /* just finished */ } + } + break; + case Envelope.BodyOneofCase.DiagRequest: + Send(new Envelope { DiagReply = Diagnostics() }); + break; + case Envelope.BodyOneofCase.Shutdown: + return 0; + case Envelope.BodyOneofCase.Fault: + await Console.Error.WriteLineAsync($"ScriptWorker: host fault: {frame.Fault.Message}"); + return ScriptWorkerEnvironment.ExitProtocolFault; + default: + await Console.Error.WriteLineAsync($"ScriptWorker: unexpected frame {frame.BodyCase}"); + return ScriptWorkerEnvironment.ExitProtocolFault; + } + } + } + + private void Load(LoadScripts load) + { + var result = new LoadResult { LoadId = load.LoadId }; + foreach (var script in load.Scripts) + { + var outcome = new LoadOutcome { Key = script.Key, Ok = true }; + try + { + var loaded = LoadedScript.Compile(script); + if (_scripts.TryRemove(script.Key, out var previous)) previous.Dispose(); + _scripts[script.Key] = loaded; + } + catch (CompilationErrorException ex) + { + outcome.Ok = false; + outcome.Error = new ScriptError { Kind = ScriptError.Types.Kind.Compile, Message = ex.Message }; + } + catch (ScriptSandboxViolationException ex) + { + outcome.Ok = false; + outcome.Error = new ScriptError { Kind = ScriptError.Types.Kind.Sandbox, Message = ex.Message }; + } + catch (Exception ex) + { + outcome.Ok = false; + outcome.Error = new ScriptError { Kind = ScriptError.Types.Kind.Threw, Message = ex.Message }; + } + result.Outcomes.Add(outcome); + } + if (ForbiddenAssemblyLoaded() is { } forbidden) DieForbidden(forbidden); + Send(new Envelope { LoadResult = result }); + } + + private void Admit(EvalRequest request) + { + if (_queue.Count >= _queueCapacity) + { + SendResult(new EvalResult + { + EvalId = request.EvalId, + Error = new ScriptError { Kind = ScriptError.Types.Kind.Busy, Message = "script worker busy" }, + }); + return; + } + _queue.Add(request); + } + + private void EvalThreadLoop() + { + foreach (var request in _queue.GetConsumingEnumerable()) + { + EvalResult result; + try + { + result = Evaluate(request); + } + catch (Exception ex) + { + result = new EvalResult + { + EvalId = request.EvalId, + Error = new ScriptError { Kind = ScriptError.Types.Kind.Threw, Message = ex.Message }, + }; + } + SendResult(result); + } + } + + private EvalResult Evaluate(EvalRequest request) + { + var result = new EvalResult { EvalId = request.EvalId }; + if (!_scripts.TryGetValue(request.Key, out var script)) + { + result.Error = new ScriptError { Kind = ScriptError.Types.Kind.NotLoaded, Message = $"script '{request.Key}' is not loaded" }; + return result; + } + + var inputs = new Dictionary(request.Inputs.Count, StringComparer.Ordinal); + foreach (var input in request.Inputs) + { + string path; + if (!string.IsNullOrEmpty(input.Path)) path = input.Path; + else if (input.Slot < script.InputPaths.Length) path = script.InputPaths[input.Slot]; + else throw new ScriptWorkerProtocolException($"input slot {input.Slot} is not registered for '{request.Key}'"); + + if (input.Unsupported) + { + result.Error = new ScriptError + { + Kind = ScriptError.Types.Kind.UnsupportedType, + Message = $"input '{path}' has a value of unsupported type {input.UnsupportedType}", + }; + return result; + } + inputs[path] = new DataValueSnapshot( + VariantCodec.Decode(input.Value), + input.StatusCode, + input.SourceTsTicks == 0 ? null : new DateTime(input.SourceTsTicks, DateTimeKind.Utc), + new DateTime(input.ServerTsTicks, DateTimeKind.Utc)); + } + + // ctx.Now follows the HOST clock (design §4.3) but keeps ticking: a fixed instant would never let a script + // that waits on ctx.Now finish. + var offset = request.NowUtcTicks == 0 + ? TimeSpan.Zero + : new DateTime(request.NowUtcTicks, DateTimeKind.Utc) - DateTime.UtcNow; + Func clock = () => DateTime.UtcNow + offset; + var logger = _scriptLogger.ForContext(ForwardingLogSink.KeyProperty, request.Key); + var soft = TimeSpan.FromMilliseconds(Math.Max(1, request.SoftTimeoutMs)); + + // Cancelled by the host's Cancel frame at the soft deadline, never by our own clock (see the proto). + using var cts = new CancellationTokenSource(); + var grace = request.HardGraceMs > 0 ? TimeSpan.FromMilliseconds(request.HardGraceMs) : _hardGrace; + var running = new RunningEval(request.Key, Stopwatch.GetTimestamp(), soft + grace + WatchdogMargin, cts); + _running[(long)request.EvalId] = running; + var before = GC.GetAllocatedBytesForCurrentThread(); + var sw = Stopwatch.StartNew(); + try + { + object? value = script.Run(inputs, logger, clock, cts.Token, out var isPredicate); + if (isPredicate) + { + result.Predicate = (bool)value!; + } + else if (VariantCodec.TryEncode(value, out var variant, out var unsupported)) + { + result.Value = variant; + } + else + { + result.Error = new ScriptError + { + Kind = ScriptError.Types.Kind.UnsupportedType, + Message = $"script returned a value of unsupported type {unsupported}", + }; + } + } + catch (Exception ex) + { + result.Error = new ScriptError { Kind = ScriptError.Types.Kind.Threw, Message = ex.Message }; + } + finally + { + _running.TryRemove((long)request.EvalId, out _); + } + result.AllocatedBytes = (ulong)Math.Max(0, GC.GetAllocatedBytesForCurrentThread() - before); + result.RunMicros = (uint)Math.Min(uint.MaxValue, sw.Elapsed.TotalMicroseconds); + return result; + } + + private void WatchdogLoop() + { + while (true) + { + Thread.Sleep(50); + foreach (var (_, run) in _running) + { + if (Stopwatch.GetElapsedTime(run.StartedTimestamp) > run.FailFastAfter) + { + // The host should have killed us already; this names the culprit in the exit/dump. + Environment.FailFast($"ScriptWorker: script '{run.Key}' exceeded its hard deadline ({run.FailFastAfter.TotalMilliseconds:F0} ms)"); + } + } + } + } + + private async Task HeartbeatLoopAsync() + { + using var timer = new PeriodicTimer(_heartbeatInterval); + while (await timer.WaitForNextTickAsync()) + { + Send(new Envelope + { + Heartbeat = new Heartbeat + { + Inflight = (uint)_running.Count, + Queued = (uint)_queue.Count, + HeapBytes = GC.GetTotalMemory(false), + Gen2Count = (uint)GC.CollectionCount(2), + }, + }); + } + } + + private async Task LogFlushLoopAsync() + { + using var timer = new PeriodicTimer(TimeSpan.FromMilliseconds(100)); + while (await timer.WaitForNextTickAsync()) + { + if (_logSink.Drain() is { } batch) Send(new Envelope { Logs = batch }); + } + } + + private async Task ParentWatchLoopAsync() + { + using var timer = new PeriodicTimer(TimeSpan.FromMilliseconds(500)); + while (await timer.WaitForNextTickAsync()) + { + if (!ParentAlive()) + { + await Console.Error.WriteLineAsync("ScriptWorker: parent process is gone; exiting."); + Environment.Exit(ScriptWorkerEnvironment.ExitParentGone); + } + } + } + + private bool ParentAlive() + { + try + { + using var parent = Process.GetProcessById(_parentPid); + return !parent.HasExited; + } + catch (ArgumentException) + { + return false; + } + catch (InvalidOperationException) + { + return false; + } + } + + private async Task WriteLoopAsync() + { + var reader = _outbound.Reader; + var pendingResults = new EvalResults(); + try + { + while (await reader.WaitToReadAsync()) + { + // Opportunistic coalescing (design §4.5): drain whatever is queued right now, merging every completed + // evaluation into one EvalResults frame; no timer, so an idle pipe adds no latency. + while (reader.TryRead(out var envelope)) + { + if (envelope.BodyCase == Envelope.BodyOneofCase.Results) + { + pendingResults.Items.AddRange(envelope.Results.Items); + continue; + } + await ScriptWorkerFrameCodec.WriteAsync(_pipe, envelope, _maxFrameBytes); + } + if (pendingResults.Items.Count > 0) + { + await ScriptWorkerFrameCodec.WriteAsync(_pipe, Wrap(new Envelope { Results = pendingResults }), _maxFrameBytes); + pendingResults = new EvalResults(); + } + await _pipe.FlushAsync(); + } + } + catch (IOException) + { + Environment.Exit(ScriptWorkerEnvironment.ExitPipeClosed); + } + catch (ObjectDisposedException) + { + Environment.Exit(ScriptWorkerEnvironment.ExitPipeClosed); + } + } + + private void SendResult(EvalResult result) + { + var results = new EvalResults(); + results.Items.Add(result); + Send(new Envelope { Results = results }); + } + + private void Send(Envelope envelope) => _outbound.Writer.TryWrite(Wrap(envelope)); + + private Envelope Wrap(Envelope envelope) + { + envelope.ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion; + envelope.Seq = (ulong)Interlocked.Increment(ref _seq); + return envelope; + } + + private static DiagnosticsReply Diagnostics() + { + var reply = new DiagnosticsReply + { + Pid = Environment.ProcessId, + WorkingDirectory = Environment.CurrentDirectory, + }; + reply.LoadedAssemblies.AddRange(AppDomain.CurrentDomain.GetAssemblies() + .Select(a => a.GetName().Name ?? string.Empty) + .Where(n => n.Length > 0)); + foreach (DictionaryEntry entry in Environment.GetEnvironmentVariables()) + reply.EnvironmentNames.Add((string)entry.Key); + return reply; + } + + private static string? ForbiddenAssemblyLoaded() + { + foreach (var assembly in AppDomain.CurrentDomain.GetAssemblies()) + { + var name = assembly.GetName().Name ?? string.Empty; + foreach (var prefix in ScriptWorkerEnvironment.ForbiddenAssemblyPrefixes) + if (name.StartsWith(prefix, StringComparison.Ordinal)) return name; + } + return null; + } + + private static void DieForbidden(string assembly) + { + Console.Error.WriteLine($"ScriptWorker: forbidden assembly '{assembly}' is loaded; exiting."); + Environment.Exit(ScriptWorkerEnvironment.ExitForbiddenAssembly); + } + + private sealed record RunningEval(string Key, long StartedTimestamp, TimeSpan FailFastAfter, CancellationTokenSource Cancellation); + + /// One loaded script: its compiled evaluator and its input slot order. + private sealed class LoadedScript : IDisposable + { + private readonly ScriptEvaluator? _value; + private readonly ScriptEvaluator? _predicate; + + private LoadedScript(string[] inputPaths, ScriptEvaluator? value, + ScriptEvaluator? predicate) + { + InputPaths = inputPaths; + _value = value; + _predicate = predicate; + } + + public string[] InputPaths { get; } + + public static LoadedScript Compile(ScriptSource source) => source.Kind switch + { + ContextKind.VirtualTag or ContextKind.Calculation => new LoadedScript( + [.. source.InputPaths], ScriptEvaluator.Compile(source.Source), null), + ContextKind.AlarmPredicate => new LoadedScript( + [.. source.InputPaths], null, ScriptEvaluator.Compile(source.Source)), + _ => throw new ScriptWorkerProtocolException($"unknown script kind {source.Kind}"), + }; + + public object? Run(IReadOnlyDictionary inputs, ILogger logger, Func clock, + CancellationToken ct, out bool isPredicate) + { + if (_predicate is not null) + { + isPredicate = true; + return _predicate.RunAsync(new AlarmPredicateContext(inputs, logger, clock), ct).GetAwaiter().GetResult(); + } + isPredicate = false; + // Single-tag mode, as in the host: a cross-tag write is dropped. + return _value!.RunAsync(new VirtualTagContext(inputs, static (_, _) => { }, logger, clock), ct) + .GetAwaiter().GetResult(); + } + + public void Dispose() + { + _value?.Dispose(); + _predicate?.Dispose(); + } + } + + /// Buffers ctx.Logger events for the host, bounded, counting what it had to drop. + private sealed class ForwardingLogSink : ILogEventSink + { + public const string KeyProperty = "ScriptWorkerKey"; + private readonly ConcurrentQueue _lines = new(); + private int _count; + private int _dropped; + + public int Capacity { get; set; } = 1000; + + public void Emit(LogEvent logEvent) + { + if (Interlocked.Increment(ref _count) > Capacity) + { + Interlocked.Decrement(ref _count); + Interlocked.Increment(ref _dropped); + return; + } + var key = logEvent.Properties.TryGetValue(KeyProperty, out var k) && k is ScalarValue { Value: string s } ? s : string.Empty; + _lines.Enqueue(new LogLine + { + Key = key, + Level = (uint)logEvent.Level, + Rendered = Truncate(logEvent.RenderMessage(), 8 * 1024), + Exception = Truncate(logEvent.Exception?.ToString() ?? string.Empty, 16 * 1024), + TsTicks = logEvent.Timestamp.UtcTicks, + }); + } + + public LogBatch? Drain() + { + var dropped = Interlocked.Exchange(ref _dropped, 0); + if (_lines.IsEmpty && dropped == 0) return null; + var batch = new LogBatch { Dropped = (uint)dropped }; + while (_lines.TryDequeue(out var line)) + { + Interlocked.Decrement(ref _count); + batch.Lines.Add(line); + } + return batch; + } + + private static string Truncate(string value, int max) => value.Length <= max ? value : value[..max]; + } +} diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker/ZB.MOM.WW.OtOpcUa.ScriptWorker.csproj b/src/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker/ZB.MOM.WW.OtOpcUa.ScriptWorker.csproj new file mode 100644 index 00000000..90eda2c5 --- /dev/null +++ b/src/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker/ZB.MOM.WW.OtOpcUa.ScriptWorker.csproj @@ -0,0 +1,32 @@ + + + + + Exe + ZB.MOM.WW.OtOpcUa.ScriptWorker + ZB.MOM.WW.OtOpcUa.ScriptWorker + true + + false + false + + + + + + + + + + + + + + diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker/build/ScriptWorker.Deploy.targets b/src/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker/build/ScriptWorker.Deploy.targets new file mode 100644 index 00000000..853fc5e9 --- /dev/null +++ b/src/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker/build/ScriptWorker.Deploy.targets @@ -0,0 +1,43 @@ + + + + <_OtScriptWorkerProject>$(MSBuildThisFileDirectory)../ZB.MOM.WW.OtOpcUa.ScriptWorker.csproj + + + + + + + + + + + + <_OtScriptWorkerOutDir>@(_OtScriptWorkerTargetPath->'%(RootDir)%(Directory)') + + + <_OtScriptWorkerFiles Include="$(_OtScriptWorkerOutDir)**/*" /> + + + + + + + + + + + diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation.Tests/CalculationDriverScriptRunnerTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation.Tests/CalculationDriverScriptRunnerTests.cs new file mode 100644 index 00000000..d336f9ec --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Calculation.Tests/CalculationDriverScriptRunnerTests.cs @@ -0,0 +1,122 @@ +using System.Collections.Concurrent; +using Serilog; +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Core.Scripting; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Calculation.Tests; + +/// +/// lmxopcua#780 — with Scripting:Runner=OutOfProcess selecting the Calculation seam, the driver hands each +/// compiled run to the node's : keyed by the calc tag's RawPath, with the driver's own +/// budget. A runner-side failure publishes BadResourceUnavailable (0x80040000); a script failure keeps +/// BadInternalError. The passthrough fast path never reaches the runner. +/// +public sealed class CalculationDriverScriptRunnerTests +{ + private const uint StatusGood = 0u; + private const uint StatusBadInternalError = 0x80020000u; + private const uint StatusBadResourceUnavailable = 0x80040000u; + private const string Tag = "calc/e/x"; + + private sealed class FakeRunner : IScriptRunner + { + public ConcurrentQueue Outcomes { get; } = new(); + public ConcurrentQueue Invocations { get; } = new(); + public List Resets { get; } = []; + + public Task RunAsync(ScriptInvocation invocation, CancellationToken ct = default) + { + Invocations.Enqueue(invocation); + return Task.FromResult(Outcomes.TryDequeue(out var next) ? next : ScriptRunOutcome.Ok(null)); + } + + public void Reset(ScriptContextKind kind) => Resets.Add(kind); + + public IReadOnlyCollection QuarantinedKeys(ScriptContextKind kind) => ["q-" + kind]; + } + + private static RawTagEntry CalcTag(string source) + { + var json = $"{{\"scriptId\":\"s\",\"changeTriggered\":true,\"scriptSource\":{System.Text.Json.JsonSerializer.Serialize(source)}}}"; + return new RawTagEntry(Tag, json, WriteIdempotent: false); + } + + private static async Task Settle(CalculationDriver driver) + { + var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30); + while (DateTime.UtcNow < deadline && !driver.IsEvaluationIdle) await Task.Delay(10); + driver.IsEvaluationIdle.ShouldBeTrue(); + } + + private static (CalculationDriver Driver, ConcurrentQueue Published) Build(FakeRunner runner, string source) + { + var driver = new CalculationDriver( + new CalculationDriverOptions { RawTags = [CalcTag(source)], RunTimeout = TimeSpan.FromMilliseconds(1234), ScriptRunner = runner }, + "calc-drv", new ScriptRootLogger(new LoggerConfiguration().CreateLogger())); + var published = new ConcurrentQueue(); + driver.OnDataChange += (_, e) => published.Enqueue(e); + return (driver, published); + } + + [Fact] + public async Task A_runner_failure_publishes_BadResourceUnavailable_and_a_script_failure_BadInternalError() + { + var runner = new FakeRunner(); + runner.Outcomes.Enqueue(ScriptRunOutcome.Ok(30.0)); + runner.Outcomes.Enqueue(ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason)); + runner.Outcomes.Enqueue(ScriptRunOutcome.Failed("script threw: boom")); + var (driver, published) = Build(runner, "return (double)ctx.GetTag(\"src/a\").Value * 3;"); + await driver.InitializeAsync("{}", TestContext.Current.CancellationToken); + + driver.OnDependencyValue("src/a", 10.0, StatusGood, DateTime.UtcNow); + await Settle(driver); + published.TryDequeue(out var good).ShouldBeTrue(); + good!.Snapshot.StatusCode.ShouldBe(StatusGood); + good.Snapshot.Value.ShouldBe(30.0); + + var invocation = runner.Invocations.Single(); + invocation.Key.ShouldBe(Tag); + invocation.Kind.ShouldBe(ScriptContextKind.Calculation); + invocation.Timeout.ShouldBe(TimeSpan.FromMilliseconds(1234)); + invocation.Inputs["src/a"].Value.ShouldBe(10.0); + + driver.OnDependencyValue("src/a", 11.0, StatusGood, DateTime.UtcNow); + await Settle(driver); + published.TryDequeue(out var unavailable).ShouldBeTrue(); + unavailable!.Snapshot.StatusCode.ShouldBe(StatusBadResourceUnavailable); + unavailable.Snapshot.Value.ShouldBe(30.0, "the last known value rides out with the Bad status"); + + driver.OnDependencyValue("src/a", 12.0, StatusGood, DateTime.UtcNow); + await Settle(driver); + published.TryDequeue(out var scriptFault).ShouldBeTrue(); + scriptFault!.Snapshot.StatusCode.ShouldBe(StatusBadInternalError); + } + + [Fact] + public async Task The_passthrough_fast_path_never_reaches_the_runner() + { + var runner = new FakeRunner(); + var (driver, published) = Build(runner, "return ctx.GetTag(\"src/a\").Value;"); + await driver.InitializeAsync("{}", TestContext.Current.CancellationToken); + + driver.OnDependencyValue("src/a", 4.0, StatusGood, DateTime.UtcNow); + await Settle(driver); + published.TryDequeue(out var mirrored).ShouldBeTrue(); + mirrored!.Snapshot.Value.ShouldBe(4.0); + runner.Invocations.ShouldBeEmpty(); + } + + [Fact] + public void The_deploy_boundary_resets_the_runners_calculation_seam_and_quarantines_are_reported() + { + var runner = new FakeRunner(); + using var evaluator = new CalculationEvaluator(Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance, + new ScriptRootLogger(new LoggerConfiguration().CreateLogger()), scriptRunner: runner); + evaluator.ClearCompiledScripts(); + evaluator.ResetQuarantines(); + runner.Resets.ShouldBe([ScriptContextKind.Calculation, ScriptContextKind.Calculation]); + evaluator.QuarantinedScripts.ShouldBe(["q-Calculation"]); + } +} diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/ScriptRunnerSelectionTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/ScriptRunnerSelectionTests.cs new file mode 100644 index 00000000..0cf7688c --- /dev/null +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Configuration/ScriptRunnerSelectionTests.cs @@ -0,0 +1,170 @@ +using Microsoft.Extensions.Configuration; +using Serilog; +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Scripting; +using ZB.MOM.WW.OtOpcUa.Host.Configuration; + +namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Configuration; + +/// +/// lmxopcua#780 — Scripting:Runner binds by its documented keys (design §12), defaults to InProcess (every seam +/// gets no runner, so the in-process path is untouched), and the worker options clamp and fail closed. +/// +public sealed class ScriptRunnerSelectionTests +{ + private static IConfiguration Config(Dictionary values) => + new ConfigurationBuilder().AddInMemoryCollection(values).Build(); + + private static readonly ILogger Silent = new LoggerConfiguration().CreateLogger(); + + [Fact] + public void The_default_is_InProcess_and_selects_no_runner_for_any_seam() + { + var bound = ScriptingOptionsBinding.Bind(Config(new()), out var adjustments); + adjustments.ShouldBeEmpty(); + bound.Runner.ShouldBe(ScriptRunnerMode.InProcess); + ScriptRunnerSelection.Validate(bound, "/srv/app").ShouldBeEmpty(); + using var selection = ScriptRunnerSelection.Create(bound, Silent, null); + foreach (var seam in Enum.GetValues()) selection.ForSeam(seam).ShouldBeNull(seam.ToString()); + selection.OutOfProcessRunner.ShouldBeNull(); + } + + [Fact] + public void Every_documented_runner_key_binds() + { + var bound = ScriptingOptionsBinding.Bind(Config(new() + { + ["Scripting:Runner"] = "OutOfProcess", + ["Scripting:Runner:Seams:0"] = "Calculation", + ["Scripting:Worker:Shards"] = "3", + ["Scripting:Worker:HardGrace"] = "00:00:02", + ["Scripting:Worker:HeapLimitBytes"] = "268435456", + ["Scripting:Worker:QuarantineAfterKills"] = "4", + ["Scripting:Worker:UnavailableGrace"] = "00:00:07", + ["Scripting:Worker:IsolationProbationRuns"] = "9", + ["Scripting:Worker:PeCacheDir"] = "/var/cache/otopcua-pe", + ["Scripting:Worker:ExecutablePath"] = "/opt/w/ZB.MOM.WW.OtOpcUa.ScriptWorker.dll", + }), out var adjustments); + + adjustments.ShouldBeEmpty(); + bound.Runner.ShouldBe(ScriptRunnerMode.OutOfProcess); + bound.RunnerSeams.ShouldBe([ScriptSeam.Calculation]); + bound.Worker.Shards.ShouldBe(3); + bound.Worker.HardGrace.ShouldBe(TimeSpan.FromSeconds(2)); + bound.Worker.HeapLimitBytes.ShouldBe(256L * 1024 * 1024); + bound.Worker.QuarantineAfterKills.ShouldBe(4); + bound.Worker.UnavailableGrace.ShouldBe(TimeSpan.FromSeconds(7)); + bound.Worker.IsolationProbationRuns.ShouldBe(9); + bound.Worker.PeCacheDir.ShouldBe("/var/cache/otopcua-pe"); + bound.Worker.ExecutablePath.ShouldBe("/opt/w/ZB.MOM.WW.OtOpcUa.ScriptWorker.dll"); + bound.IsOutOfProcess(ScriptSeam.Calculation).ShouldBeTrue(); + bound.IsOutOfProcess(ScriptSeam.VirtualTag).ShouldBeFalse(); + } + + [Fact] + public void A_misspelled_runner_mode_fails_startup() + => Should.Throw(() => + ScriptingOptionsBinding.Bind(Config(new() { ["Scripting:Runner"] = "OutOfProc" }), out _)); + + [Fact] + public void Worker_values_out_of_range_are_clamped_and_reported() + { + var bound = ScriptingOptionsBinding.Bind(Config(new() + { + ["Scripting:Worker:Shards"] = "64", + ["Scripting:Worker:HardGrace"] = "00:00:00.010", + ["Scripting:Worker:HeapLimitBytes"] = "1024", + ["Scripting:Worker:QuarantineAfterKills"] = "0", + ["Scripting:Worker:UnavailableGrace"] = "00:05:00", + ["Scripting:Worker:IsolationProbationRuns"] = "0", + }), out var adjustments); + + bound.Worker.Shards.ShouldBe(ScriptWorkerOptions.MaxShards); + bound.Worker.HardGrace.ShouldBe(ScriptWorkerOptions.MinHardGrace); + bound.Worker.HeapLimitBytes.ShouldBe(ScriptWorkerOptions.MinHeapLimitBytes); + bound.Worker.QuarantineAfterKills.ShouldBe(1); + bound.Worker.UnavailableGrace.ShouldBe(ScriptWorkerOptions.MaxUnavailableGrace); + bound.Worker.IsolationProbationRuns.ShouldBe(1); + adjustments.Count.ShouldBe(6); + adjustments.ShouldAllBe(a => a.StartsWith("Scripting:Worker:")); + } + + [Fact] + public void The_hard_grace_defaults_to_max_of_one_second_and_the_budget() + { + var worker = new ScriptWorkerOptions(); + worker.EffectiveHardGrace(TimeSpan.FromMilliseconds(250)).ShouldBe(TimeSpan.FromSeconds(1)); + worker.EffectiveHardGrace(TimeSpan.FromSeconds(2)).ShouldBe(TimeSpan.FromSeconds(2)); + new ScriptWorkerOptions { HardGrace = TimeSpan.FromMilliseconds(400) } + .EffectiveHardGrace(TimeSpan.FromSeconds(2)).ShouldBe(TimeSpan.FromMilliseconds(400)); + } + + [Theory] + [InlineData("relative/cache", "must be an absolute path")] + [InlineData("{root}/pe-cache", "inside the content root")] + [InlineData("{root}", "inside the content root")] + public void A_pe_cache_dir_that_is_relative_or_inside_the_content_root_fails_startup(string dir, string why) + { + var root = Path.Combine(Path.GetTempPath(), "otopcua-root-" + Guid.NewGuid().ToString("N")); + var options = new ScriptingOptions + { + Runner = ScriptRunnerMode.OutOfProcess, + Worker = new ScriptWorkerOptions { PeCacheDir = dir.Replace("{root}", root) }, + }; + Should.Throw(() => ScriptRunnerSelection.Validate(options, root)).Message.ShouldContain(why); + } + + [Fact] + public void A_pe_cache_dir_outside_the_content_root_is_accepted() + { + var root = Path.Combine(Path.GetTempPath(), "otopcua-root"); + var options = new ScriptingOptions + { + Runner = ScriptRunnerMode.OutOfProcess, + Worker = new ScriptWorkerOptions { PeCacheDir = Path.Combine(Path.GetTempPath(), "otopcua-root-sibling") }, + }; + Should.NotThrow(() => ScriptRunnerSelection.Validate(options, root)); + } + + [Fact] + public void Shadow_is_not_implemented_yet_and_runs_in_process_with_a_warning() + { + var options = new ScriptingOptions { Runner = ScriptRunnerMode.Shadow }; + ScriptRunnerSelection.Validate(options, "/srv").ShouldHaveSingleItem().ShouldContain("Shadow"); + using var selection = ScriptRunnerSelection.Create(options, Silent, null); + selection.OutOfProcessRunner.ShouldBeNull(); + } + + [Fact] + public void OutOfProcess_selects_only_the_supported_requested_seams() + { + var all = new ScriptingOptions { Runner = ScriptRunnerMode.OutOfProcess }; + ScriptRunnerSelection.Validate(all, "/srv").ShouldContain(w => w.Contains("ScriptedAlarm seam cannot run out of process yet")); + using (var selection = ScriptRunnerSelection.Create(all, Silent, null)) + { + selection.ForSeam(ScriptSeam.VirtualTag).ShouldNotBeNull(); + selection.ForSeam(ScriptSeam.Calculation).ShouldBeSameAs(selection.ForSeam(ScriptSeam.VirtualTag), "one node-wide runner"); + selection.ForSeam(ScriptSeam.ScriptedAlarm).ShouldBeNull(); + } + + var calcOnly = new ScriptingOptions { Runner = ScriptRunnerMode.OutOfProcess, RunnerSeams = [ScriptSeam.Calculation] }; + using (var selection = ScriptRunnerSelection.Create(calcOnly, Silent, null)) + { + selection.ForSeam(ScriptSeam.Calculation).ShouldNotBeNull(); + selection.ForSeam(ScriptSeam.VirtualTag).ShouldBeNull(); + } + } + + [Fact] + public void A_missing_worker_is_a_startup_warning_not_a_silent_in_process_fallback() + { + var options = new ScriptingOptions + { + Runner = ScriptRunnerMode.OutOfProcess, + RunnerSeams = [ScriptSeam.VirtualTag], + Worker = new ScriptWorkerOptions { ExecutablePath = Path.Combine(Path.GetTempPath(), "missing-" + Guid.NewGuid().ToString("N"), "w.dll") }, + }; + ScriptRunnerSelection.Validate(options, "/srv").ShouldHaveSingleItem().ShouldContain("BadResourceUnavailable"); + } +} diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Engines/RoslynVirtualTagEvaluatorScriptRunnerTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Engines/RoslynVirtualTagEvaluatorScriptRunnerTests.cs new file mode 100644 index 00000000..2c5e87ed --- /dev/null +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/Engines/RoslynVirtualTagEvaluatorScriptRunnerTests.cs @@ -0,0 +1,109 @@ +using System.Collections.Concurrent; +using Microsoft.Extensions.Logging.Abstractions; +using Serilog; +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Commons.OpcUa; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Core.Scripting; +using ZB.MOM.WW.OtOpcUa.Host.Engines; + +namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Engines; + +/// +/// lmxopcua#780 — the virtual-tag seam over : with a runner (OutOfProcess) the compiled run +/// is the runner's, keyed by virtual-tag id with the evaluator's budget, and its Unavailable outcome becomes a failure +/// carrying . Without one (the InProcess default) nothing changes. +/// +public sealed class RoslynVirtualTagEvaluatorScriptRunnerTests +{ + private sealed class FakeRunner : IScriptRunner + { + public ConcurrentQueue Outcomes { get; } = new(); + public ConcurrentQueue Invocations { get; } = new(); + public List Resets { get; } = []; + + public Task RunAsync(ScriptInvocation invocation, CancellationToken ct = default) + { + Invocations.Enqueue(invocation); + return Task.FromResult(Outcomes.TryDequeue(out var next) ? next : ScriptRunOutcome.Ok(null)); + } + + public void Reset(ScriptContextKind kind) => Resets.Add(kind); + + public IReadOnlyCollection QuarantinedKeys(ScriptContextKind kind) => ["vt-q"]; + } + + private static RoslynVirtualTagEvaluator Build(IScriptRunner? runner) => + new(NullLogger.Instance, new ScriptRootLogger(new LoggerConfiguration().CreateLogger()), + runTimeout: TimeSpan.FromMilliseconds(1500), scriptRunner: runner); + + private static Dictionary Deps(int a) => new(StringComparer.Ordinal) + { + ["a"] = new DataValueSnapshot(a, 0u, DateTime.UtcNow, DateTime.UtcNow), + }; + + [Fact] + public async Task Compiled_runs_go_to_the_runner_and_its_outcomes_map_onto_the_seam_contract() + { + var runner = new FakeRunner(); + runner.Outcomes.Enqueue(ScriptRunOutcome.Ok(42)); + runner.Outcomes.Enqueue(ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason)); + runner.Outcomes.Enqueue(ScriptRunOutcome.Failed("script threw: boom")); + using var sut = Build(runner); + const string script = "return (int)ctx.GetTag(\"a\").Value + 1;"; + + var ok = await sut.EvaluateAsync("vt-1", script, Deps(41)); + ok.Success.ShouldBeTrue(); + ok.Value.ShouldBe(42); + ok.StatusCode.ShouldBeNull(); + + var unavailable = await sut.EvaluateAsync("vt-1", script, Deps(41)); + unavailable.Success.ShouldBeFalse(); + unavailable.Reason.ShouldBe(ScriptRunOutcome.WorkerRestartedReason); + unavailable.StatusCode.ShouldBe(OpcUaStatus.BadResourceUnavailable); + + var failed = await sut.EvaluateAsync("vt-1", script, Deps(41)); + failed.Reason.ShouldBe("script threw: boom"); + failed.StatusCode.ShouldBeNull("a script failure keeps BadInternalError"); + + var invocation = runner.Invocations.First(); + invocation.Key.ShouldBe("vt-1"); + invocation.Kind.ShouldBe(ScriptContextKind.VirtualTag); + invocation.Source.ShouldBe(script); + invocation.Timeout.ShouldBe(TimeSpan.FromMilliseconds(1500)); + invocation.Inputs["a"].Value.ShouldBe(41); + invocation.ScriptLogger.ShouldNotBeNull(); + } + + [Fact] + public async Task The_passthrough_fast_path_and_an_empty_expression_never_reach_the_runner() + { + var runner = new FakeRunner(); + using var sut = Build(runner); + (await sut.EvaluateAsync("vt-p", "return ctx.GetTag(\"a\").Value;", Deps(7))).Value.ShouldBe(7); + (await sut.EvaluateAsync("vt-e", " ", Deps(7))).Success.ShouldBeFalse(); + runner.Invocations.ShouldBeEmpty(); + } + + [Fact] + public void The_deploy_boundary_resets_the_runners_virtual_tag_seam_and_its_quarantines_are_reported() + { + var runner = new FakeRunner(); + using var sut = Build(runner); + sut.ClearCompiledScripts(); + sut.ResetQuarantines(); + runner.Resets.ShouldBe([ScriptContextKind.VirtualTag, ScriptContextKind.VirtualTag]); + sut.QuarantinedScripts.ShouldBe(["vt-q"]); + } + + /// Positive control for the default: no runner means the in-process Roslyn path actually runs the script. + [Fact] + public async Task Without_a_runner_the_script_runs_in_process() + { + using var sut = Build(runner: null); + sut.ScriptRunner.ShouldBeNull(); + (await sut.EvaluateAsync("vt-1", "return (int)ctx.GetTag(\"a\").Value + 1;", Deps(41))).Value.ShouldBe(42); + sut.RunGuard.StartedRunCount.ShouldBe(1); + } +} diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagActorRunnerUnavailableTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagActorRunnerUnavailableTests.cs new file mode 100644 index 00000000..252bfeb5 --- /dev/null +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/VirtualTags/VirtualTagActorRunnerUnavailableTests.cs @@ -0,0 +1,64 @@ +using System.Collections.Concurrent; +using Akka.Actor; +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Commons.Engines; +using ZB.MOM.WW.OtOpcUa.Commons.OpcUa; +using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness; +using ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags; + +namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.VirtualTags; + +/// +/// lmxopcua#780 — when the out-of-process runner, not the script, fails (its worker restarted, no worker ready), the +/// virtual tag publishes with its last value, so a client can tell a +/// sick node from a broken script. Every other failure keeps the long-standing . +/// +public sealed class VirtualTagActorRunnerUnavailableTests : RuntimeActorTestBase +{ + private sealed class QueuedEvaluator : IVirtualTagEvaluator + { + public ConcurrentQueue Results { get; } = new(); + + public VirtualTagEvalResult Evaluate(string virtualTagId, string expression, IReadOnlyDictionary dependencies) + => Results.TryDequeue(out var next) ? next : VirtualTagEvalResult.NoChange; + } + + [Fact] + public void A_runner_unavailable_failure_publishes_BadResourceUnavailable_and_a_script_failure_BadInternalError() + { + var parent = CreateTestProbe(); + var evaluator = new QueuedEvaluator(); + evaluator.Results.Enqueue(VirtualTagEvalResult.Ok(5)); + evaluator.Results.Enqueue(VirtualTagEvalResult.Unavailable("script worker restarted")); + evaluator.Results.Enqueue(VirtualTagEvalResult.Unavailable("script worker restarted")); + evaluator.Results.Enqueue(VirtualTagEvalResult.Failure("script threw: boom")); + var actor = parent.ChildActorOf(VirtualTagActor.Props("vt-1", "x", evaluator: evaluator, dependencyRefs: ["a"])); + + actor.Tell(new VirtualTagActor.DependencyValueChanged("a", 1, DateTime.UtcNow)); + parent.ExpectMsg().StatusCode.ShouldBe(OpcUaStatus.Good); + + actor.Tell(new VirtualTagActor.DependencyValueChanged("a", 2, DateTime.UtcNow)); + var unavailable = parent.ExpectMsg(PresenceBudget); + unavailable.StatusCode.ShouldBe(OpcUaStatus.BadResourceUnavailable); + unavailable.Value.ShouldBe(5, "the last known value rides out with the Bad status"); + + // Once per transition: the same code again publishes nothing. + actor.Tell(new VirtualTagActor.DependencyValueChanged("a", 3, DateTime.UtcNow)); + parent.ExpectNoMsg(TimeSpan.FromMilliseconds(300)); + + // A script failure is a different Bad and is published. + actor.Tell(new VirtualTagActor.DependencyValueChanged("a", 4, DateTime.UtcNow)); + var scriptFault = parent.ExpectMsg(PresenceBudget); + scriptFault.StatusCode.ShouldBe(OpcUaStatus.BadInternalError); + scriptFault.Value.ShouldBe(5); + } + + [Fact] + public void A_failure_without_a_status_keeps_BadInternalError() + { + VirtualTagEvalResult.Failure("x").StatusCode.ShouldBeNull(); + VirtualTagEvalResult.Unavailable("x").StatusCode.ShouldBe(OpcUaStatus.BadResourceUnavailable); + OpcUaStatus.BadResourceUnavailable.ShouldBe(0x80040000u, "the OPC UA Part 6 code ScadaBridge maps"); + } +} diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/ContainmentTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/ContainmentTests.cs new file mode 100644 index 00000000..482cd8e3 --- /dev/null +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/ContainmentTests.cs @@ -0,0 +1,382 @@ +using System.Diagnostics; +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Core.Scripting; +using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess; +using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol; + +namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests; + +/// Worker-backed tests spawn real processes and measure wall-clock bounds, so they never run in parallel. +[CollectionDefinition(Name, DisableParallelization = true)] +public sealed class ScriptWorkerCollection +{ + public const string Name = "ScriptWorker"; +} + +/// +/// lmxopcua#780 design §13 "Containment": each case runs against the real ScriptWorker process, and each has a +/// positive control — a healthy sibling on the SAME shard (every runner here has one shard) that keeps producing +/// correct values before and after the event. +/// +[Collection(ScriptWorkerCollection.Name)] +[Trait("Category", "ScriptWorker")] +public sealed class ContainmentTests +{ + private const string Healthy = "return 7;"; + private const string Runaway = "while (true) { }"; + private const string Cooperative = "while (!ctx.CancellationToken.IsCancellationRequested) { } return 1;"; + + /// Spins for the number of milliseconds in input ms, so one source serves slow and fast runs. + private const string Spin = "var ms = (int)ctx.GetTag(\"ms\").Value; var end = ctx.Now.AddMilliseconds(ms); while (ctx.Now < end) { } return ms;"; + + private static IReadOnlyDictionary Ms(int ms) => Scripts.Input("ms", ms); + + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + [Fact] + public async Task A_healthy_script_runs_out_of_process_and_reads_its_inputs() + { + await using var runner = Scripts.Runner(Scripts.Options()); + var outcome = await runner.RunAsync(Scripts.Vt("vt", "return (int)ctx.GetTag(\"x\").Value + 1;", + inputs: Scripts.Input("x", 41)), Ct); + + outcome.Status.ShouldBe(ScriptRunStatus.Ok, outcome.Reason); + outcome.Value.ShouldBe(42); + runner.Shards[0].Current.ShouldNotBeNull().Pid.ShouldNotBe(Environment.ProcessId); + } + + /// + /// The headline case: while(true){} ignores the token. In process it would spin a host core forever; here + /// the caller gets the usual timeout at T, the worker is killed at T + G, the host burns no CPU on it, and the + /// sibling on the same shard keeps working. Mutation check: with the hard-deadline kill disabled the worker is + /// never killed and the Exited wait times out. + /// + [Fact] + public async Task A_runaway_script_is_killed_at_its_hard_deadline_and_its_sibling_keeps_working() + { + var softT = TimeSpan.FromMilliseconds(400); + var grace = TimeSpan.FromMilliseconds(500); + await using var runner = Scripts.Runner(Scripts.Options(hardGrace: grace, quarantineAfterKills: 2)); + + (await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7); + var worker = runner.Shards[0].Current.ShouldNotBeNull(); + + var sw = Stopwatch.StartNew(); + var outcome = await runner.RunAsync(Scripts.Vt("runaway", Runaway, softT), Ct); + var softElapsed = sw.Elapsed; + outcome.Status.ShouldBe(ScriptRunStatus.Failed); + outcome.Reason!.ShouldStartWith("script timed out after "); + softElapsed.ShouldBeLessThan(softT + TimeSpan.FromMilliseconds(400), "the caller is answered at the soft deadline"); + + var exit = await worker.Exited.WaitAsync(TimeSpan.FromSeconds(10), Ct); + var killElapsed = sw.Elapsed; + exit.Reason.ShouldBe(WorkerExitReason.KilledDeadline); + exit.CulpritKey.ShouldBe(InProcessScriptRunner.ScopedKey(ScriptContextKind.VirtualTag, "runaway")); + killElapsed.ShouldBeGreaterThanOrEqualTo(softT + grace - TimeSpan.FromMilliseconds(50)); + // Design budget is T + G + 100 ms; the slack here is for a loaded CI box (process reap + exit notification). + killElapsed.ShouldBeLessThan(softT + grace + TimeSpan.FromMilliseconds(1500)); + + // No orphan in the host: an in-process while(true) would burn one host core for the whole window. + var cpuBefore = Process.GetCurrentProcess().TotalProcessorTime; + await Task.Delay(TimeSpan.FromMilliseconds(1500), Ct); + var cpuUsed = Process.GetCurrentProcess().TotalProcessorTime - cpuBefore; + cpuUsed.ShouldBeLessThan(TimeSpan.FromMilliseconds(1000), "the runaway's thread died with its worker"); + + // The sibling recovers on a fresh worker. + (await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7); + runner.Shards[0].Current.ShouldNotBeNull().Pid.ShouldNotBe(worker.Pid); + + // A second kill quarantines (QuarantineAfterKills = 2); the third trigger is refused without running. + var second = runner.Shards[0].Current!; + (await runner.RunAsync(Scripts.Vt("runaway", Runaway, softT), Ct)).Status.ShouldBe(ScriptRunStatus.Failed); + (await second.Exited.WaitAsync(TimeSpan.FromSeconds(10), Ct)).Reason.ShouldBe(WorkerExitReason.KilledDeadline); + // The strike is recorded by the shard's exit handler, which runs asynchronously to the Exited completion. + await Scripts.Eventually(() => runner.QuarantinedKeys(ScriptContextKind.VirtualTag).Count == 1, + TimeSpan.FromSeconds(5), "the second kill strike"); + var refused = await runner.RunAsync(Scripts.Vt("runaway", Runaway, softT), Ct); + refused.Reason!.ShouldStartWith(ScriptRunGuard.QuarantinedReasonPrefix); + refused.Reason.ShouldContain("hard deadline"); + runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBe(["runaway"]); + (await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7); + + // Lifted by a source change (the operator fixed it)… + (await runner.RunAsync(Scripts.Vt("runaway", "return 1;"), Ct)).Value.ShouldBe(1); + runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBeEmpty(); + } + + [Fact] + public async Task A_redeploy_lifts_a_kill_quarantine_for_its_seam_only() + { + var softT = TimeSpan.FromMilliseconds(300); + await using var runner = Scripts.Runner(Scripts.Options(hardGrace: TimeSpan.FromMilliseconds(300), quarantineAfterKills: 1)); + + (await runner.RunAsync(Scripts.Vt("runaway", Runaway, softT), Ct)).Status.ShouldBe(ScriptRunStatus.Failed); + await Scripts.Eventually(() => runner.QuarantinedKeys(ScriptContextKind.VirtualTag).Count == 1, TimeSpan.FromSeconds(10), "the kill quarantine"); + + runner.Reset(ScriptContextKind.Calculation); + runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBe(["runaway"], "another seam's redeploy leaves it alone"); + runner.Reset(ScriptContextKind.VirtualTag); + runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBeEmpty(); + } + + /// A script that polls its token ends itself at T: no kill, no strike beyond the timeout (design §13). + [Fact] + public async Task A_cooperative_script_ends_at_its_soft_deadline_without_a_kill() + { + var softT = TimeSpan.FromMilliseconds(400); + var grace = TimeSpan.FromMilliseconds(1500); + await using var runner = Scripts.Runner(Scripts.Options(hardGrace: grace, quarantineAfterKills: 1)); + (await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7); + var worker = runner.Shards[0].Current.ShouldNotBeNull(); + + var outcome = await runner.RunAsync(Scripts.Vt("polite", Cooperative, softT), Ct); + outcome.Reason!.ShouldStartWith("script timed out after "); + + await Task.Delay(softT + grace + TimeSpan.FromMilliseconds(500), Ct); + worker.IsAlive.ShouldBeTrue("a script that ended itself must not cost the shard a restart"); + runner.Shards[0].Current!.Pid.ShouldBe(worker.Pid); + runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBeEmpty("no kill strike (QuarantineAfterKills = 1)"); + (await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7); + } + + [Fact] + public async Task A_second_trigger_while_the_first_still_runs_is_refused_not_queued() + { + await using var runner = Scripts.Runner(Scripts.Options()); + (await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0); + + var first = runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(1500)), Ct); + await Task.Delay(200, Ct); + var second = await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct); + second.Reason.ShouldBe(ScriptRunGuard.PreviousRunStillRunningReason); + (await first).Value.ShouldBe(1500); + } + + /// + /// kill -9 of the worker: the evaluation in flight is a victim — Unavailable "script worker restarted" + /// (BadResourceUnavailable), not a script fault — and, with no deadline crossed, a suspect that proves itself in + /// the isolation worker and then returns to its shard. + /// + [Fact] + public async Task An_externally_killed_worker_makes_its_in_flight_script_a_suspect_that_recovers_in_isolation() + { + await using var runner = Scripts.Runner(Scripts.Options(probation: 2)); + (await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0); + var worker = runner.Shards[0].Current.ShouldNotBeNull(); + + var inFlight = runner.RunAsync(Scripts.Vt("spin", Spin, TimeSpan.FromSeconds(20), Ms(5000)), Ct); + await Task.Delay(300, Ct); + Process.GetProcessById(worker.Pid).Kill(); + + var victim = await inFlight.WaitAsync(TimeSpan.FromSeconds(10), Ct); + victim.Status.ShouldBe(ScriptRunStatus.Unavailable); + victim.Reason.ShouldBe(ScriptRunOutcome.WorkerRestartedReason); + (await worker.Exited).Reason.ShouldBe(WorkerExitReason.Crash); + runner.SuspectKeys.ShouldBe([InProcessScriptRunner.ScopedKey(ScriptContextKind.VirtualTag, "spin")]); + + (await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0); + runner.Isolation.Current.ShouldNotBeNull("a suspect runs in the isolation worker"); + (await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0); + runner.SuspectKeys.ShouldBeEmpty("two clean runs (probation = 2) return it to its shard"); + runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBeEmpty(); + + // The sibling recovers on a fresh shard worker. + (await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7); + runner.Shards[0].Current!.Pid.ShouldNotBe(worker.Pid); + } + + /// The isolation worker runs one evaluation at a time, so a crash there names exactly one key. + [Fact] + public async Task A_crash_in_the_isolation_worker_strikes_its_one_key_and_quarantines_it() + { + await using var runner = Scripts.Runner(Scripts.Options(quarantineAfterKills: 2)); + (await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0); + + async Task RunAndKill(Func worker) + { + var run = runner.RunAsync(Scripts.Vt("spin", Spin, TimeSpan.FromSeconds(20), Ms(5000)), Ct); + var live = await Scripts.Eventually(worker, TimeSpan.FromSeconds(15), "a live worker"); + await Scripts.Eventually(() => live.InFlightCount == 1, TimeSpan.FromSeconds(15), "the evaluation in flight"); + await Task.Delay(200, Ct); + Process.GetProcessById(live.Pid).Kill(); + return await run.WaitAsync(TimeSpan.FromSeconds(10), Ct); + } + + (await RunAndKill(() => runner.Shards[0].Current)).Reason.ShouldBe(ScriptRunOutcome.WorkerRestartedReason); + runner.SuspectKeys.Count.ShouldBe(1); + runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBeEmpty("an ambiguous shard crash is not a strike"); + + (await RunAndKill(() => runner.Isolation.Current)).Reason.ShouldBe(ScriptRunOutcome.WorkerRestartedReason); + runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBeEmpty("one isolation crash is one strike"); + (await RunAndKill(() => runner.Isolation.Current)).Reason.ShouldBe(ScriptRunOutcome.WorkerRestartedReason); + await Scripts.Eventually(() => runner.QuarantinedKeys(ScriptContextKind.VirtualTag).Count == 1, + TimeSpan.FromSeconds(5), "the isolated-crash quarantine"); + + var refused = await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct); + refused.Reason!.ShouldStartWith(ScriptRunGuard.QuarantinedReasonPrefix); + refused.Reason.ShouldContain("isolation worker"); + (await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7); + } + + /// A deadline kill blames exactly one key: the sibling in flight on the same worker is a victim, not a suspect. + [Fact] + public async Task A_deadline_kill_blames_only_its_culprit() + { + await using var runner = Scripts.Runner(Scripts.Options(hardGrace: TimeSpan.FromMilliseconds(300), quarantineAfterKills: 1)); + (await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0); + (await runner.RunAsync(Scripts.Vt("warm", Healthy), Ct)).Value.ShouldBe(7); + // Compile the runaway first, so both evaluations are on the worker at once. + _ = runner.Shards[0].Current!; + + var victim = runner.RunAsync(Scripts.Vt("spin", Spin, TimeSpan.FromSeconds(20), Ms(5000)), Ct); + await Task.Delay(200, Ct); + var culprit = await runner.RunAsync(Scripts.Vt("runaway", Runaway, TimeSpan.FromMilliseconds(300)), Ct); + culprit.Reason!.ShouldStartWith("script timed out after "); + + var victimOutcome = await victim.WaitAsync(TimeSpan.FromSeconds(15), Ct); + victimOutcome.Status.ShouldBe(ScriptRunStatus.Unavailable); + victimOutcome.Reason.ShouldBe(ScriptRunOutcome.WorkerRestartedReason); + await Scripts.Eventually(() => runner.QuarantinedKeys(ScriptContextKind.VirtualTag).Count == 1, TimeSpan.FromSeconds(5), "the culprit's quarantine"); + runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBe(["runaway"]); + runner.SuspectKeys.ShouldBeEmpty("a deadline kill is unambiguous: nobody else is suspected"); + (await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0); + } + + /// An allocation bomb dies in its worker (heap hard limit); the host's own memory does not move. + [Fact] + public async Task An_allocation_bomb_is_contained_in_its_worker() + { + await using var runner = Scripts.Runner(Scripts.Options(heapLimit: 128L * 1024 * 1024)); + (await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7); + + GC.Collect(); + var hostBefore = GC.GetTotalMemory(forceFullCollection: true); + var outcome = await runner.RunAsync(Scripts.Vt("bomb", + "var hoard = new System.Collections.Generic.List(); while (true) { hoard.Add(new byte[1 << 20]); }", + TimeSpan.FromSeconds(10)), Ct); + var hostAfter = GC.GetTotalMemory(forceFullCollection: true); + + outcome.Status.ShouldNotBe(ScriptRunStatus.Ok); + (hostAfter - hostBefore).ShouldBeLessThan(32L * 1024 * 1024, "the hoard lived in the worker's heap, not the host's"); + await Scripts.Eventually(() => runner.Shards[0].Current, TimeSpan.FromSeconds(10), "a worker"); + (await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7); + } + + /// Fail closed (design §14 #4): no worker means Bad, never an in-process run. + [Fact] + public async Task A_worker_that_cannot_start_reports_unavailable_and_never_runs_in_process() + { + var missing = Path.Combine(Path.GetTempPath(), "no-such-dir-" + Guid.NewGuid().ToString("N"), OutOfProcessScriptRunner.WorkerAssemblyFileName); + await using var runner = Scripts.Runner(Scripts.Options(workerPath: missing, unavailableGrace: TimeSpan.FromSeconds(1))); + + var outcome = await runner.RunAsync(Scripts.Vt("vt", "return 1;"), Ct); + outcome.Status.ShouldBe(ScriptRunStatus.Unavailable); + outcome.Reason.ShouldBe(ScriptRunOutcome.RunnerUnavailableReason); + outcome.ToVirtualTagEvalResult().StatusCode.ShouldBe(ZB.MOM.WW.OtOpcUa.Commons.OpcUa.OpcUaStatus.BadResourceUnavailable); + runner.Shards[0].ConsecutiveFailures.ShouldBeGreaterThan(0, "the failed start backs off"); + } + + /// A worker whose pipe reaches end of stream exits by itself (the Linux "host died" path, design §3.3). + [Fact] + public async Task A_worker_exits_by_itself_when_its_pipe_closes() + { + await using var runner = Scripts.Runner(Scripts.Options()); + (await runner.RunAsync(Scripts.Vt("vt", Healthy), Ct)).Value.ShouldBe(7); + var worker = runner.Shards[0].Current.ShouldNotBeNull(); + using var process = Process.GetProcessById(worker.Pid); + + var sw = Stopwatch.StartNew(); + worker.SeverPipeForTest(); + await process.WaitForExitAsync(Ct).WaitAsync(TimeSpan.FromSeconds(5), Ct); + sw.Elapsed.ShouldBeLessThan(TimeSpan.FromSeconds(2)); + var exit = await worker.Exited.WaitAsync(TimeSpan.FromSeconds(5), Ct); + exit.ExitCode.ShouldBe(ScriptWorkerEnvironment.ExitPipeClosed, "it noticed the EOF; the host did not have to kill it"); + } + + /// Design §6: the environment is cleared and repopulated from an allow-list. Positive control first. + [Fact] + public async Task The_worker_environment_carries_no_host_secrets() + { + const string apiKey = "ServerHistorian__ApiKey"; + const string configDb = "ConnectionStrings__ConfigDb"; + Environment.SetEnvironmentVariable(apiKey, "sekrit"); + Environment.SetEnvironmentVariable(configDb, "Server=x;Password=y"); + try + { + Environment.GetEnvironmentVariable(apiKey).ShouldBe("sekrit", "positive control: the host has it"); + await using var runner = Scripts.Runner(Scripts.Options()); + (await runner.RunAsync(Scripts.Vt("vt", Healthy), Ct)).Value.ShouldBe(7); + + var diag = await runner.Shards[0].Current!.DiagnosticsAsync(TimeSpan.FromSeconds(10)); + diag.EnvironmentNames.ShouldNotContain(apiKey); + diag.EnvironmentNames.ShouldNotContain(configDb); + diag.EnvironmentNames.ShouldContain(ScriptWorkerEnvironment.Nonce, "the allow-listed launch variables do arrive"); + diag.EnvironmentNames.ShouldContain("DOTNET_GCHeapHardLimit"); + diag.WorkingDirectory.ShouldNotBe(Environment.CurrentDirectory); + // An empty per-worker directory under the temp root (macOS reports /var as /private/var, so match the leaf). + diag.WorkingDirectory.ShouldContain(Path.Combine("otopcua-scriptworker", "otopcua-sw-")); + Directory.EnumerateFileSystemEntries(diag.WorkingDirectory).ShouldBeEmpty(); + } + finally + { + Environment.SetEnvironmentVariable(apiKey, null); + Environment.SetEnvironmentVariable(configDb, null); + } + } + + /// Design §6: the host's cluster, database, OPC UA and secrets stacks are not in the worker. + [Fact] + public async Task The_worker_loads_none_of_the_host_stacks() + { + await using var runner = Scripts.Runner(Scripts.Options()); + (await runner.RunAsync(Scripts.Vt("vt", "return System.Math.Abs(-3);"), Ct)).Value.ShouldBe(3); + var diag = await runner.Shards[0].Current!.DiagnosticsAsync(TimeSpan.FromSeconds(10)); + + diag.LoadedAssemblies.ShouldContain("ZB.MOM.WW.OtOpcUa.Core.Scripting", "positive control: the compile ran here"); + diag.LoadedAssemblies.ShouldContain("Microsoft.CodeAnalysis.CSharp"); + foreach (var prefix in ScriptWorkerEnvironment.ForbiddenAssemblyPrefixes) + diag.LoadedAssemblies.ShouldNotContain(a => a.StartsWith(prefix, StringComparison.Ordinal), prefix); + + // Not merely unloaded: Akka is not even deployable beside the worker. + var folder = Path.GetDirectoryName(runner.WorkerPath)!; + File.Exists(Path.Combine(folder, "ZB.MOM.WW.OtOpcUa.ScriptWorker.dll")).ShouldBeTrue("positive control"); + File.Exists(Path.Combine(folder, "Akka.dll")).ShouldBeFalse(); + File.Exists(Path.Combine(folder, "Grpc.Core.Api.dll")).ShouldBeFalse(); + } + + [Fact] + public async Task Script_log_lines_come_back_through_the_invocations_logger() + { + var sink = new CapturingSink(); + await using var runner = Scripts.Runner(Scripts.Options()); + (await runner.RunAsync(Scripts.Vt("vt", "ctx.Logger.Information(\"hello {N}\", 42); return 1;", logger: sink.CreateLogger()), Ct)) + .Value.ShouldBe(1); + await Scripts.Eventually(() => sink.Rendered.Contains("hello 42"), TimeSpan.FromSeconds(5), "the forwarded log line"); + } + + [Fact] + public async Task An_alarm_predicate_runs_out_of_process() + { + await using var runner = Scripts.Runner(Scripts.Options()); + var outcome = await runner.RunAsync(new ScriptInvocation("alarm-1", ScriptContextKind.AlarmPredicate, + "return (double)ctx.GetTag(\"t\").Value > 50;", Scripts.Input("t", 75.0), TimeSpan.FromSeconds(5)), Ct); + outcome.Status.ShouldBe(ScriptRunStatus.Ok, outcome.Reason); + outcome.Value.ShouldBe(true); + } + + /// Design §14 #11: an unsupported type is an explicit failure, never a silent conversion. + [Fact] + public async Task Unsupported_input_and_result_types_are_explicit_failures() + { + await using var runner = Scripts.Runner(Scripts.Options()); + var result = await runner.RunAsync(Scripts.Vt("list", "return new System.Collections.Generic.List { 1 };"), Ct); + result.Status.ShouldBe(ScriptRunStatus.Failed); + result.Reason!.ShouldStartWith("unsupported type: script returned a value of unsupported type System.Collections.Generic.List"); + + var input = await runner.RunAsync(Scripts.Vt("guid", "return 1;", inputs: Scripts.Input("g", Guid.NewGuid())), Ct); + input.Status.ShouldBe(ScriptRunStatus.Failed); + input.Reason!.ShouldStartWith("unsupported type: input 'g' has a value of unsupported type System.Guid"); + } +} diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/HandshakeTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/HandshakeTests.cs new file mode 100644 index 00000000..60aa1a8d --- /dev/null +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/HandshakeTests.cs @@ -0,0 +1,86 @@ +using System.IO.Pipes; +using System.Security.Cryptography; +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess; +using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol; +using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1; + +namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests; + +/// lmxopcua#780 design §6 "Pipe hardening": only the launched process, echoing its nonce, is accepted. +public sealed class HandshakeTests +{ + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + private static async Task HandshakeWith(Func reply, string expectedNonce = "right", int? expectedPid = 1234) + { + var name = "otopcua-sw-test-" + Convert.ToHexString(RandomNumberGenerator.GetBytes(6)); + await using var server = new NamedPipeServerStream(name, PipeDirection.InOut, 1, PipeTransmissionMode.Byte, + PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly); + await using var client = new NamedPipeClientStream(".", name, PipeDirection.InOut, PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly); + var accept = server.WaitForConnectionAsync(Ct); + await client.ConnectAsync(TimeSpan.FromSeconds(5), Ct); + await accept; + + var rogue = Task.Run(async () => + { + var hello = await ScriptWorkerFrameCodec.ReadAsync(client, ScriptWorkerFrameCodec.DefaultMaxFrameBytes, Ct); + await ScriptWorkerFrameCodec.WriteAsync(client, reply(hello!), ScriptWorkerFrameCodec.DefaultMaxFrameBytes, Ct); + await client.FlushAsync(Ct); + }, Ct); + + try + { + using var timeout = CancellationTokenSource.CreateLinkedTokenSource(Ct); + timeout.CancelAfter(TimeSpan.FromSeconds(5)); + await WorkerConnection.HandshakeAsync(server, new Hello { Nonce = expectedNonce }, expectedNonce, expectedPid, timeout.Token); + await rogue; + return null; + } + catch (Exception ex) + { + return ex; + } + } + + private static Envelope Ready(string nonce, int pid) => new() + { + ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion, + Ready = new Ready { Nonce = nonce, Pid = pid }, + }; + + [Fact] + public async Task The_launched_process_echoing_its_nonce_is_accepted() + => (await HandshakeWith(_ => Ready("right", 1234))).ShouldBeNull("positive control"); + + [Fact] + public async Task A_wrong_nonce_is_refused() + => (await HandshakeWith(_ => Ready("wrong", 1234))).ShouldBeOfType() + .Message.ShouldContain("nonce"); + + [Fact] + public async Task A_client_that_is_not_the_launched_process_is_refused() + => (await HandshakeWith(_ => Ready("right", 999))).ShouldBeOfType() + .Message.ShouldContain("pid"); + + [Fact] + public async Task Anything_but_ready_first_is_refused() + => (await HandshakeWith(_ => new Envelope + { + ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion, + Heartbeat = new Heartbeat(), + })).ShouldBeOfType(); + + [Fact] + public async Task The_hello_carries_the_nonce_to_echo() + { + string? seen = null; + (await HandshakeWith(hello => + { + seen = hello.Hello.Nonce; + return Ready(hello.Hello.Nonce, 1234); + })).ShouldBeNull(); + seen.ShouldBe("right"); + } +} diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/ParityTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/ParityTests.cs new file mode 100644 index 00000000..7056e536 --- /dev/null +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/ParityTests.cs @@ -0,0 +1,95 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Core.Scripting; + +namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests; + +/// +/// lmxopcua#780 design §13 "Parity": the same script, through (today's guard) +/// and through the worker, gives the same status, reason, value and exact CLR value type. +/// +[Collection(ScriptWorkerCollection.Name)] +[Trait("Category", "ScriptWorker")] +public sealed class ParityTests +{ + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + public static TheoryData Cases() => new() + { + { "arithmetic", "return 1 + 2;" }, + { "string", "return \"a\" + ctx.GetTag(\"x\").Value;" }, + { "throw", "throw new System.InvalidOperationException(\"boom\");" }, + { "compile error", "return undefinedThing;" }, + { "sandbox violation", "return System.IO.File.Exists(\"x\");" }, + { "missing input status", "return ctx.GetTag(\"missing\").StatusCode;" }, + { "input status", "return ctx.GetTag(\"x\").StatusCode;" }, + { "source timestamp", "return ctx.GetTag(\"x\").SourceTimestampUtc;" }, + { "clock", "return ctx.Now > new System.DateTime(2000, 1, 1);" }, + { "null", "return null;" }, + { "array", "return new double[] { 1.5, 2.5 };" }, + { "cast of wrong type", "return (short)ctx.GetTag(\"x\").Value;" }, + }; + + private static readonly IReadOnlyDictionary Inputs = + new Dictionary(StringComparer.Ordinal) + { + ["x"] = new DataValueSnapshot(5, 0x40000000u, new DateTime(2026, 1, 2, 3, 4, 5, DateTimeKind.Utc), DateTime.UtcNow), + }; + + [Theory] + [MemberData(nameof(Cases))] + public async Task In_and_out_of_process_agree(string name, string source) + { + using var inProcess = new InProcessScriptRunner(); + await using var outOfProcess = Scripts.Runner(Scripts.Options()); + var invocation = Scripts.Vt(name, source, inputs: Inputs); + + var expected = await inProcess.RunAsync(invocation, Ct); + var actual = await outOfProcess.RunAsync(invocation, Ct); + + actual.Status.ShouldBe(expected.Status, $"{name}: {actual.Reason} vs {expected.Reason}"); + actual.Reason.ShouldBe(expected.Reason); + Same(actual.Value, expected.Value, name); + } + + public static TheoryData Typed() => new() + { + { "bool", true }, { "sbyte", (sbyte)-2 }, { "byte", (byte)2 }, { "short", (short)-3 }, { "ushort", (ushort)3 }, + { "int", -4 }, { "uint", 4u }, { "long", -5L }, { "ulong", 5UL }, { "float", 1.25f }, { "double", 2.5 }, + { "string", "s" }, { "System.DateTime", new DateTime(2026, 11, 9, 0, 0, 0, DateTimeKind.Utc) }, + }; + + /// + /// Unboxing parity (pins design assumption A-7): (T)ctx.GetTag("x").Value works out of process for every + /// tag type, which needs the exact boxed CLR type to survive the wire. + /// + [Theory] + [MemberData(nameof(Typed))] + public async Task Unboxing_to_the_exact_type_matches(string typeName, object value) + { + using var inProcess = new InProcessScriptRunner(); + await using var outOfProcess = Scripts.Runner(Scripts.Options()); + var invocation = Scripts.Vt("unbox-" + typeName, $"var v = ({typeName})ctx.GetTag(\"x\").Value; return v;", + inputs: Scripts.Input("x", value)); + + var expected = await inProcess.RunAsync(invocation, Ct); + var actual = await outOfProcess.RunAsync(invocation, Ct); + expected.Status.ShouldBe(ScriptRunStatus.Ok, expected.Reason); + actual.Status.ShouldBe(ScriptRunStatus.Ok, actual.Reason); + Same(actual.Value, expected.Value, typeName); + } + + private static void Same(object? actual, object? expected, string because) + { + if (expected is null) + { + actual.ShouldBeNull(because); + return; + } + actual.ShouldNotBeNull(because); + actual.GetType().ShouldBe(expected.GetType(), because); + if (expected is Array array) ((Array)actual).Cast().ShouldBe(array.Cast(), because); + else actual.ShouldBe(expected, because); + } +} diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/ProtocolTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/ProtocolTests.cs new file mode 100644 index 00000000..f20be772 --- /dev/null +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/ProtocolTests.cs @@ -0,0 +1,198 @@ +using System.Buffers.Binary; +using Google.Protobuf; +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess; +using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol; +using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1; + +namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests; + +/// lmxopcua#780 design §13 — frame codec and the CLR ↔ Variant mapping. +public sealed class ProtocolTests +{ + public static IEnumerable Values() + { + object?[] values = + [ + null, true, false, (sbyte)-5, (byte)250, (short)-3, (ushort)7, -42, 42u, long.MinValue, ulong.MaxValue, + 1.5f, double.NaN, 3.25, "text", string.Empty, new DateTime(2026, 11, 9, 1, 2, 3, DateTimeKind.Utc), + new[] { 1, 2, 3 }, new short[] { -1, 1 }, new[] { "a", "b" }, new[] { true, false }, Array.Empty(), + new[] { new DateTime(2020, 1, 1, 0, 0, 0, DateTimeKind.Utc) }, (short)0, 0, 0L, 0.0, + ]; + return values.Select(v => new[] { v }); + } + + /// The exact boxed CLR type survives (design §4.4): (short)ctx.GetTag(x).Value must keep working. + [Theory] + [MemberData(nameof(Values))] + public void Every_type_tag_round_trips_with_its_exact_clr_type(object? value) + { + VariantCodec.TryEncode(value, out var variant, out var unsupported).ShouldBeTrue(); + unsupported.ShouldBeNull(); + + // Through the wire, not just the object: a payload the serializer drops must still decode. + var wire = Variant.Parser.ParseFrom(variant.ToByteArray()); + var decoded = VariantCodec.Decode(wire); + + if (value is null) + { + decoded.ShouldBeNull(); + return; + } + decoded.ShouldNotBeNull(); + decoded.GetType().ShouldBe(value.GetType()); + if (value is Array expected) + ((Array)decoded).Cast().ShouldBe(expected.Cast()); + else if (value is double d && double.IsNaN(d)) + double.IsNaN((double)decoded).ShouldBeTrue(); + else + decoded.ShouldBe(value); + } + + [Fact] + public void Unsupported_types_are_refused_not_converted() + { + foreach (var value in new object[] { new List { 1 }, 1.5m, Guid.NewGuid(), new { A = 1 }, new int[1, 1], new object[] { 1 } }) + { + VariantCodec.TryEncode(value, out _, out var type).ShouldBeFalse(value.GetType().Name); + type.ShouldNotBeNullOrEmpty(); + } + } + + [Fact] + public void Local_datetime_crosses_as_utc() + { + var local = new DateTime(2026, 6, 1, 12, 0, 0, DateTimeKind.Local); + VariantCodec.TryEncode(local, out var variant, out _).ShouldBeTrue(); + var decoded = (DateTime)VariantCodec.Decode(variant)!; + decoded.Kind.ShouldBe(DateTimeKind.Utc); + decoded.ShouldBe(local.ToUniversalTime()); + } + + [Fact] + public void A_payload_that_contradicts_its_tag_is_a_protocol_fault() + { + Should.Throw(() => VariantCodec.Decode(new Variant { Type = Variant.Types.TypeTag.Int16, S = "x" })); + Should.Throw(() => VariantCodec.Decode(new Variant { Type = (Variant.Types.TypeTag)99 })); + Should.Throw(() => VariantCodec.Decode(new Variant { Type = Variant.Types.TypeTag.Int16, I = 1 << 20 })); + } + + private static Envelope Sample() => new() + { + ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion, + Seq = 7, + Heartbeat = new Heartbeat { Inflight = 3, HeapBytes = 1234 }, + }; + + [Fact] + public async Task A_frame_round_trips() + { + using var stream = new MemoryStream(); + await ScriptWorkerFrameCodec.WriteAsync(stream, Sample(), 1024, TestContext.Current.CancellationToken); + stream.Position = 0; + var read = await ScriptWorkerFrameCodec.ReadAsync(stream, 1024, TestContext.Current.CancellationToken); + read.ShouldBe(Sample()); + (await ScriptWorkerFrameCodec.ReadAsync(stream, 1024, TestContext.Current.CancellationToken)).ShouldBeNull("a clean end of stream at a frame boundary"); + } + + [Fact] + public async Task Oversize_frames_are_refused_on_write_and_on_read() + { + using var stream = new MemoryStream(); + await Should.ThrowAsync(async () => + await ScriptWorkerFrameCodec.WriteAsync(stream, Sample(), 4, TestContext.Current.CancellationToken)); + stream.Length.ShouldBe(0, "nothing is written for a refused frame"); + + await ScriptWorkerFrameCodec.WriteAsync(stream, Sample(), 1024, TestContext.Current.CancellationToken); + stream.Position = 0; + await Should.ThrowAsync(async () => + await ScriptWorkerFrameCodec.ReadAsync(stream, 4, TestContext.Current.CancellationToken)); + } + + public static TheoryData BadFrames() + { + var good = Sample().ToByteArray(); + var header = new byte[4]; + BinaryPrimitives.WriteInt32LittleEndian(header, good.Length); + var wrongVersion = new Envelope { ProtocolVersion = 99, Heartbeat = new Heartbeat() }.ToByteArray(); + var noBody = new Envelope { ProtocolVersion = 1, Seq = 1 }.ToByteArray(); + return new TheoryData + { + { "zero length", new byte[] { 0, 0, 0, 0 } }, + { "negative length", new byte[] { 0xFF, 0xFF, 0xFF, 0xFF } }, + { "truncated header", new byte[] { 5, 0 } }, + { "truncated body", [.. header, .. good[..^1]] }, + { "garbage body", [5, 0, 0, 0, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF] }, + { "wrong protocol version", Frame(wrongVersion) }, + { "no body", Frame(noBody) }, + }; + } + + private static byte[] Frame(byte[] body) + { + var header = new byte[4]; + BinaryPrimitives.WriteInt32LittleEndian(header, body.Length); + return [.. header, .. body]; + } + + [Theory] + [MemberData(nameof(BadFrames))] + public async Task Malformed_frames_are_protocol_faults(string why, byte[] bytes) + { + using var stream = new MemoryStream(bytes); + await Should.ThrowAsync(async () => + await ScriptWorkerFrameCodec.ReadAsync(stream, 1024, TestContext.Current.CancellationToken), why); + } + + /// Random bytes never escape as anything but a protocol fault (or a clean end of stream). + [Fact] + public async Task Fuzzed_input_only_ever_yields_a_protocol_fault() + { + var random = new Random(780); + for (var i = 0; i < 3000; i++) + { + var bytes = new byte[random.Next(0, 64)]; + random.NextBytes(bytes); + if (bytes.Length >= 4 && random.Next(2) == 0) + BinaryPrimitives.WriteInt32LittleEndian(bytes, random.Next(1, bytes.Length)); + using var stream = new MemoryStream(bytes); + try + { + while (await ScriptWorkerFrameCodec.ReadAsync(stream, 1024, TestContext.Current.CancellationToken) is { } envelope) + { + // A frame that parses must still decode its variants without an unexpected exception type. + if (envelope.BodyCase == Envelope.BodyOneofCase.Results) + foreach (var r in envelope.Results.Items) + try { VariantCodec.Decode(r.Value); } catch (Exception ex) when (ex is ScriptWorkerProtocolException or OverflowException) { } + } + } + catch (ScriptWorkerProtocolException) + { + } + } + } + + [Fact] + public void Backoff_doubles_from_250ms_to_a_30s_cap_with_20_percent_jitter() + { + WorkerShard.NextBackoff(1, 0.5).ShouldBe(TimeSpan.FromMilliseconds(250)); + WorkerShard.NextBackoff(2, 0.5).ShouldBe(TimeSpan.FromMilliseconds(500)); + WorkerShard.NextBackoff(3, 0.5).ShouldBe(TimeSpan.FromSeconds(1)); + WorkerShard.NextBackoff(20, 0.5).ShouldBe(TimeSpan.FromSeconds(30)); + WorkerShard.NextBackoff(1, 0).ShouldBe(TimeSpan.FromMilliseconds(200)); + WorkerShard.NextBackoff(1, 0.9999999).TotalMilliseconds.ShouldBe(300, tolerance: 0.01); + WorkerShard.NextBackoff(40, 1).ShouldBe(TimeSpan.FromSeconds(36)); + } + + [Fact] + public void Shard_placement_is_stable_and_in_range() + { + // FNV-1a, not string.GetHashCode (randomised per process): a script stays on its shard across restarts. + OutOfProcessScriptRunner.ShardIndex("1:vt-1", 2).ShouldBe(OutOfProcessScriptRunner.ShardIndex("1:vt-1", 2)); + var spread = Enumerable.Range(0, 400).Select(i => OutOfProcessScriptRunner.ShardIndex($"1:vt-{i}", 4)).ToArray(); + spread.ShouldAllBe(i => i >= 0 && i < 4); + spread.Distinct().Count().ShouldBe(4); + OutOfProcessScriptRunner.ShardIndex("abc", 1000).ShouldBe((int)(0x1A47E90Bu % 1000)); + } +} diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/SeamEndToEndTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/SeamEndToEndTests.cs new file mode 100644 index 00000000..6f5f22a9 --- /dev/null +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/SeamEndToEndTests.cs @@ -0,0 +1,88 @@ +using System.Collections.Concurrent; +using System.Diagnostics; +using Serilog; +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Core.Scripting; +using ZB.MOM.WW.OtOpcUa.Driver.Calculation; + +namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests; + +/// +/// lmxopcua#780 — the whole path a client sees: a real Calculation driver whose seam runs out of process on the real +/// worker. A script failure publishes BadInternalError as today; the worker dying under a script publishes +/// BadResourceUnavailable (0x80040000), the ScadaBridge-visible code; and the tag recovers Good afterwards. +/// +[Collection(ScriptWorkerCollection.Name)] +[Trait("Category", "ScriptWorker")] +public sealed class SeamEndToEndTests +{ + private const uint StatusGood = 0u; + private const uint StatusBadInternalError = 0x80020000u; + private const uint StatusBadResourceUnavailable = 0x80040000u; + private const string Tag = "calc/e/spin"; + + /// Spins for src/ms milliseconds, throws when it is negative, and returns it otherwise. + private const string Script = + "var ms = (double)ctx.GetTag(\"src/ms\").Value; if (ms < 0) throw new System.InvalidOperationException(\"negative\"); " + + "var end = ctx.Now.AddMilliseconds(ms); while (ctx.Now < end) { } return ms;"; + + private static CancellationToken Ct => TestContext.Current.CancellationToken; + + private static async Task Settle(CalculationDriver driver) + { + var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(60); + while (DateTime.UtcNow < deadline && !driver.IsEvaluationIdle) await Task.Delay(10); + driver.IsEvaluationIdle.ShouldBeTrue(); + } + + [Fact] + public async Task A_calculation_tag_publishes_BadResourceUnavailable_only_when_the_worker_died_under_it() + { + await using var runner = Scripts.Runner(Scripts.Options()); + var json = $"{{\"scriptId\":\"s\",\"changeTriggered\":true,\"scriptSource\":{System.Text.Json.JsonSerializer.Serialize(Script)}}}"; + var driver = new CalculationDriver( + new CalculationDriverOptions { RawTags = [new RawTagEntry(Tag, json, WriteIdempotent: false)], RunTimeout = TimeSpan.FromSeconds(20), ScriptRunner = runner }, + "calc-drv", new ScriptRootLogger(new LoggerConfiguration().CreateLogger())); + var published = new ConcurrentQueue(); + driver.OnDataChange += (_, e) => published.Enqueue(e); + await driver.InitializeAsync("{}", Ct); + + driver.OnDependencyValue("src/ms", 0.0, StatusGood, DateTime.UtcNow); + await Settle(driver); + published.TryDequeue(out var good).ShouldBeTrue(); + good!.Snapshot.StatusCode.ShouldBe(StatusGood); + good.Snapshot.Value.ShouldBe(0.0); + + // A script fault: BadInternalError, exactly as in process. + driver.OnDependencyValue("src/ms", -1.0, StatusGood, DateTime.UtcNow); + await Settle(driver); + published.TryDequeue(out var threw).ShouldBeTrue(); + threw!.Snapshot.StatusCode.ShouldBe(StatusBadInternalError); + + driver.OnDependencyValue("src/ms", 1.0, StatusGood, DateTime.UtcNow); + await Settle(driver); + published.TryDequeue(out var back).ShouldBeTrue(); + back!.Snapshot.StatusCode.ShouldBe(StatusGood); + + // The worker dies under a run: the runner, not the script, failed. + driver.OnDependencyValue("src/ms", 5000.0, StatusGood, DateTime.UtcNow); + var worker = await Scripts.Eventually(() => runner.Shards[0].Current is { InFlightCount: 1 } w ? w : null, + TimeSpan.FromSeconds(10), "the run in flight"); + await Task.Delay(200, Ct); + Process.GetProcessById(worker.Pid).Kill(); + await Settle(driver); + published.TryDequeue(out var unavailable).ShouldBeTrue(); + unavailable!.Snapshot.StatusCode.ShouldBe(StatusBadResourceUnavailable); + unavailable.Snapshot.Value.ShouldBe(1.0, "the last good value rides out with the Bad status"); + + // Recovery: the next trigger runs (in the isolation worker, as a suspect) and publishes Good. + driver.OnDependencyValue("src/ms", 2.0, StatusGood, DateTime.UtcNow); + await Settle(driver); + published.TryDequeue(out var recovered).ShouldBeTrue(); + recovered!.Snapshot.StatusCode.ShouldBe(StatusGood); + recovered.Snapshot.Value.ShouldBe(2.0); + await driver.ShutdownAsync(Ct); + } +} diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/TestSupport.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/TestSupport.cs new file mode 100644 index 00000000..d0612986 --- /dev/null +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/TestSupport.cs @@ -0,0 +1,82 @@ +using System.Collections.Concurrent; +using Serilog; +using Serilog.Core; +using Serilog.Events; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Core.Scripting; +using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess; + +namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests; + +/// Collects Serilog events in memory. +internal sealed class CapturingSink : ILogEventSink +{ + public ConcurrentQueue Events { get; } = new(); + + public void Emit(LogEvent logEvent) => Events.Enqueue(logEvent); + + public ILogger CreateLogger() => new LoggerConfiguration().MinimumLevel.Verbose().WriteTo.Sink(this).CreateLogger(); + + public IEnumerable Rendered => Events.Select(e => e.RenderMessage()); +} + +internal static class Scripts +{ + public static readonly IReadOnlyDictionary NoInputs = + new Dictionary(StringComparer.Ordinal); + + public static IReadOnlyDictionary Input(string path, object? value, uint status = 0) + => new Dictionary(StringComparer.Ordinal) + { + [path] = new DataValueSnapshot(value, status, DateTime.UtcNow, DateTime.UtcNow), + }; + + public static ScriptInvocation Vt(string key, string source, TimeSpan? timeout = null, + IReadOnlyDictionary? inputs = null, ILogger? logger = null) + => new(key, ScriptContextKind.VirtualTag, source, inputs ?? NoInputs, timeout ?? TimeSpan.FromSeconds(5), logger); + + /// Options for a single-shard runner with short budgets, so the containment tests run in seconds. + public static ScriptingOptions Options(int shards = 1, TimeSpan? hardGrace = null, int quarantineAfterKills = 2, + int probation = 20, TimeSpan? unavailableGrace = null, string? workerPath = null, long heapLimit = 256L * 1024 * 1024) + => new() + { + Runner = ScriptRunnerMode.OutOfProcess, + QuarantineAfterConsecutiveTimeouts = 100, + Worker = new ScriptWorkerOptions + { + Shards = shards, + HardGrace = hardGrace ?? TimeSpan.FromMilliseconds(500), + QuarantineAfterKills = quarantineAfterKills, + IsolationProbationRuns = probation, + UnavailableGrace = unavailableGrace ?? TimeSpan.FromSeconds(30), + ExecutablePath = workerPath, + HeapLimitBytes = heapLimit, + }, + }; + + public static OutOfProcessScriptRunner Runner(ScriptingOptions options, ILogger? log = null) + => new(options, log ?? new LoggerConfiguration().CreateLogger(), null, + heartbeatInterval: TimeSpan.FromSeconds(1), loadTimeout: TimeSpan.FromSeconds(60)); + + public static async Task Eventually(Func probe, TimeSpan timeout, string what) where T : class + { + var deadline = DateTime.UtcNow + timeout; + while (DateTime.UtcNow < deadline) + { + if (probe() is { } value) return value; + await Task.Delay(20); + } + throw new TimeoutException($"timed out waiting for {what}"); + } + + public static async Task Eventually(Func probe, TimeSpan timeout, string what) + { + var deadline = DateTime.UtcNow + timeout; + while (DateTime.UtcNow < deadline) + { + if (probe()) return; + await Task.Delay(20); + } + throw new TimeoutException($"timed out waiting for {what}"); + } +} diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests.csproj b/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests.csproj new file mode 100644 index 00000000..eaa5e773 --- /dev/null +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests.csproj @@ -0,0 +1,30 @@ + + + + false + true + ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests + + + + + + + + all + runtime; build; native; contentfiles; analyzers; buildtransitive + + + + + + + + + + + + + + +