feat(scripting): out-of-process pre-emptible script runner behind Scripting:Runner (lmxopcua#780 P1)
v2-ci / unit-tests (push) Waiting to run
v2-ci / unit-tests (push) Waiting to run
Opt-in (Scripting:Runner=OutOfProcess; InProcess stays the default and its code path is unchanged). Implements the P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md with the owner's §14 decisions: - Options: Scripting:Runner (InProcess | Shadow | OutOfProcess), Scripting:Runner:Seams, Scripting:Worker:* (Shards 2, HardGrace max(1 s,T), HeapLimitBytes 512 MiB, QuarantineAfterKills 2, UnavailableGrace 5 s, IsolationProbationRuns 20, PeCacheDir validated absolute + outside the content root (fail closed), ExecutablePath). Shadow is not implemented yet: runs InProcess with a startup Warning. - Core.Scripting: IScriptRunner seam + InProcessScriptRunner (today's guard, the parity baseline). - Core.Scripting.Protocol: protobuf envelopes, 4-byte LE framing (zero/oversize/truncated/garbage refused), Variant carrying the exact CLR type; unsupported types are explicit failures. - ScriptWorker app: compiles in the worker, P-thread scheduler, host-driven soft cancel, fail-fast watchdog, heartbeats, ctx.Logger forwarding, exits on pipe EOF / parent gone, forbidden-assembly self-check; Akka and Grpc.Core.Api excluded from its folder. Shipped in script-worker/ (Build + Publish). - Core.Scripting.OutOfProcess: K shards + isolation worker, soft T / hard T+G kill, attribution (deadline kill = one key; crash = suspects -> isolation; isolation crash = strike), kill-strike quarantine, backoff 250 ms -> 30 s, fail closed, cleared allow-listed env, nonce + pid handshake on a current-user pipe, Windows Job Object. - VirtualTag + Calculation seams use the runner when selected; a runner-side failure (worker restarted / runner unavailable) publishes BadResourceUnavailable (0x80040000) — ScadaBridge#112. Script faults keep BadInternalError. The scripted-alarm seam stays in process (alarms last). Tests: new ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests (Category=ScriptWorker, real processes; green on macOS and on windev/Windows), seam tests in Host/Runtime/Calculation suites; mutation-checked. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
@@ -388,3 +388,10 @@ endpoints under `/api/script-analysis/*`, gated by the `ConfigEditor` policy
|
||||
`ScriptAnalysisEndpoints.MapScriptAnalysisEndpoints`).
|
||||
`Core.VirtualTags` is now just `DependencyGraph`, `ITagUpstreamSource` and `IHistoryWriter` — the dormant `VirtualTagEngine`/Source/Definition and `TimerTriggerScheduler` were deleted.
|
||||
See `docs/ScriptEditor.md` for the full guide. Scripts may use the `{{equip}}` token for equipment-relative tag paths that resolve per-equipment at deploy time — see the "Equipment-relative tag paths" section in `docs/ScriptEditor.md`.
|
||||
|
||||
**Out-of-process script runner (lmxopcua#780, opt-in).** `Scripting:Runner` = `InProcess` (default, unchanged) | `OutOfProcess` | `Shadow` (not yet).
|
||||
`OutOfProcess` runs the VirtualTag + Calculation seams' compiled scripts in `ZB.MOM.WW.OtOpcUa.ScriptWorker` processes
|
||||
(`src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess`, protocol in `Core.Scripting.Protocol`), shipped in `script-worker/`
|
||||
beside the host; a runner-side failure publishes `BadResourceUnavailable` (ScadaBridge-visible, ScadaBridge#112). The worker tests
|
||||
are `tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests` (`Category=ScriptWorker`, real processes). See `docs/VirtualTags.md`
|
||||
§"Out-of-process runner".
|
||||
|
||||
@@ -8,6 +8,8 @@
|
||||
<Project Path="src/Core/ZB.MOM.WW.OtOpcUa.Core/ZB.MOM.WW.OtOpcUa.Core.csproj" />
|
||||
<Project Path="src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Abstractions/ZB.MOM.WW.OtOpcUa.Core.Scripting.Abstractions.csproj" />
|
||||
<Project Path="src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ZB.MOM.WW.OtOpcUa.Core.Scripting.csproj" />
|
||||
<Project Path="src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol/ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.csproj" />
|
||||
<Project Path="src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess.csproj" />
|
||||
<Project Path="src/Core/ZB.MOM.WW.OtOpcUa.Core.VirtualTags/ZB.MOM.WW.OtOpcUa.Core.VirtualTags.csproj" />
|
||||
<Project Path="src/Core/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms/ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms.csproj" />
|
||||
<Project Path="src/Core/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian/ZB.MOM.WW.OtOpcUa.Core.AlarmHistorian.csproj" />
|
||||
@@ -15,6 +17,7 @@
|
||||
<Folder Name="/src/Server/">
|
||||
<Project Path="src/Server/ZB.MOM.WW.OtOpcUa.AdminUI/ZB.MOM.WW.OtOpcUa.AdminUI.csproj" />
|
||||
<Project Path="src/Server/ZB.MOM.WW.OtOpcUa.ControlPlane/ZB.MOM.WW.OtOpcUa.ControlPlane.csproj" />
|
||||
<Project Path="src/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker/ZB.MOM.WW.OtOpcUa.ScriptWorker.csproj" />
|
||||
<Project Path="src/Server/ZB.MOM.WW.OtOpcUa.Host/ZB.MOM.WW.OtOpcUa.Host.csproj" />
|
||||
<Project Path="src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/ZB.MOM.WW.OtOpcUa.OpcUaServer.csproj" />
|
||||
<Project Path="src/Server/ZB.MOM.WW.OtOpcUa.Runtime/ZB.MOM.WW.OtOpcUa.Runtime.csproj" />
|
||||
@@ -91,6 +94,7 @@
|
||||
<Folder Name="/tests/Server/">
|
||||
<Project Path="tests/Server/ZB.MOM.WW.OtOpcUa.AdminUI.Tests/ZB.MOM.WW.OtOpcUa.AdminUI.Tests.csproj" />
|
||||
<Project Path="tests/Server/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests/ZB.MOM.WW.OtOpcUa.ControlPlane.Tests.csproj" />
|
||||
<Project Path="tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests.csproj" />
|
||||
<Project Path="tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests.csproj" />
|
||||
<Project Path="tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/ZB.MOM.WW.OtOpcUa.Host.Tests.csproj" />
|
||||
<Project Path="tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests.csproj" />
|
||||
|
||||
@@ -211,6 +211,11 @@ scripted-alarm predicates) runs scripts through the shared `ScriptRunGuard`:
|
||||
`Scripting:MaxAllocBytesPerRun` (64 MB) is quarantined. This is sampled after the
|
||||
run returns, so a run that OOMs the process first is not stopped — that needs an
|
||||
out-of-process runner (→ #750).
|
||||
- **Out of process (opt-in, lmxopcua#780).** `Scripting:Runner=OutOfProcess` moves the
|
||||
virtual-tag and Calculation seams' compiled runs into `ScriptWorker` processes, where a
|
||||
run past `T + Scripting:Worker:HardGrace` is stopped by killing its worker and an OOM
|
||||
kills a worker instead of the server. A run whose worker died under it reads
|
||||
`BadResourceUnavailable`. See `docs/VirtualTags.md` §"Out-of-process runner".
|
||||
- **Recursion.** Unbounded recursion fails the script with
|
||||
`ScriptStackExhaustedException` (Bad, script log) instead of overflowing the
|
||||
stack, which would terminate the whole server. The runtime compile inserts a
|
||||
|
||||
@@ -88,6 +88,16 @@ and supply `ServerHistorian__ApiKey` via the environment on the OtOpcUa host sid
|
||||
`historian:write` + `historian:tags:write`. See [Historian.md](Historian.md) for the full config-key and
|
||||
deployment-prerequisite reference. (The retired Wonderware TCP sidecar: [Historian.Wonderware.md](drivers/Historian.Wonderware.md).)
|
||||
|
||||
## Script worker processes (`Scripting:Runner=OutOfProcess`, lmxopcua#780)
|
||||
|
||||
Off by default. When enabled, a driver node launches up to `Scripting:Worker:Shards` + 1 child processes
|
||||
(`ZB.MOM.WW.OtOpcUa.ScriptWorker`, from `script-worker/` beside the host) on its first compiled script evaluation and
|
||||
kills them with it: a Job Object with kill-on-close on Windows, pipe end of stream plus a parent-pid poll elsewhere,
|
||||
and a Shutdown frame then a kill on host stop. Their stderr lands in the host log prefixed `ScriptWorker[<shard>]`.
|
||||
The worker currently runs under the host's account through the dotnet muxer; the dedicated low-privilege account,
|
||||
pipe ACL and firewall outbound rule (design §14 #5/#6) are installer work still open on lmxopcua#780. See
|
||||
`docs/VirtualTags.md` §"Out-of-process runner".
|
||||
|
||||
## Install / Uninstall
|
||||
|
||||
- `scripts/install/Install-Services.ps1 -Roles admin,driver` — installs `OtOpcUaHost`.
|
||||
|
||||
+44
-1
@@ -37,11 +37,54 @@ User scripts are compiled via `Microsoft.CodeAnalysis.CSharp` (regular compiler,
|
||||
|
||||
#### Known resource limits (accepted trade-offs)
|
||||
|
||||
The sandbox cannot pre-empt a script. **Memory:** each run's own-thread allocation is sampled (`GC.GetAllocatedBytesForCurrentThread`) and a run over `Scripting:MaxAllocBytesPerRun` (default 64 MB) quarantines the script — but the sample is taken after the run returns, so a script that allocates fast enough (`new byte[int.MaxValue]` in a loop) can still OOM the process first. **CPU:** a tight loop keeps its pool thread after the timeout fires; `ScriptRunGuard` bounds that to one orphan per script by refusing further runs while it spins (Bad `"previous evaluation still running"`) and quarantines after three consecutive timeouts (a refusal is not a strike) — or immediately on a refusal behind an orphan that has run longer than `Scripting:StuckOrphanQuarantineFactor` (default 10) × the timeout, which is what quarantines a never-returning `while(true)`. Both are mitigations; pre-emptive CPU/memory limits need an out-of-process runner (→ #750). **Stack:** unbounded recursion used to terminate the whole server — a `StackOverflowException` cannot be caught, and lmxopcua#779 reproduced it out of process (local function, mutual recursion, self-referencing `Func<>`, anonymous self-application, recursion through LINQ: the child process died with "Stack overflow." / SIGABRT). `ScriptEvaluator.Compile` now rewrites the script so the body and every lambda, anonymous method and local function starts with `ScriptStackGuard.Check()` (`RuntimeHelpers.TryEnsureSufficientExecutionStack`), so the recursion fails the script with `ScriptStackExhaustedException` — Bad quality and a script-log entry like any throwing script — while there is still stack left. A runtime check rather than an analyzer rule because recursion is not decidable from syntax (anonymous self-application never names itself); bounded recursion keeps working. Limits: the depth reached depends on the evaluating thread's stack size; a recursion that never re-enters script code is not seen (the sandbox exposes no such API); a script can catch the exception, which only unwinds it. Pinned by `ScriptRecursionContainmentTests` (one child process per shape) and `ScriptStackGuardInjectionTests`. Script authoring is gated by the AdminUI `ConfigEditor` policy — **`Administrator` or `Designer`**, not Administrator alone — and that same policy gates deploy, so a Designer can author a script and put it into production; that gate is the primary control (AdminUI-001 deliberately admitted Designers to the script pages and `/api/script-analysis/*`; lmxopcua#779 corrected this sentence, which used to say Admin-only). See `docs/ScriptEditor.md` §"Runtime safety".
|
||||
The sandbox cannot pre-empt a script. **Memory:** each run's own-thread allocation is sampled (`GC.GetAllocatedBytesForCurrentThread`) and a run over `Scripting:MaxAllocBytesPerRun` (default 64 MB) quarantines the script — but the sample is taken after the run returns, so a script that allocates fast enough (`new byte[int.MaxValue]` in a loop) can still OOM the process first. **CPU:** a tight loop keeps its pool thread after the timeout fires; `ScriptRunGuard` bounds that to one orphan per script by refusing further runs while it spins (Bad `"previous evaluation still running"`) and quarantines after three consecutive timeouts (a refusal is not a strike) — or immediately on a refusal behind an orphan that has run longer than `Scripting:StuckOrphanQuarantineFactor` (default 10) × the timeout, which is what quarantines a never-returning `while(true)`. Both are mitigations; pre-emptive CPU/memory limits need the out-of-process runner (→ #750 design, #780; opt-in via `Scripting:Runner=OutOfProcess`, below). **Stack:** unbounded recursion used to terminate the whole server — a `StackOverflowException` cannot be caught, and lmxopcua#779 reproduced it out of process (local function, mutual recursion, self-referencing `Func<>`, anonymous self-application, recursion through LINQ: the child process died with "Stack overflow." / SIGABRT). `ScriptEvaluator.Compile` now rewrites the script so the body and every lambda, anonymous method and local function starts with `ScriptStackGuard.Check()` (`RuntimeHelpers.TryEnsureSufficientExecutionStack`), so the recursion fails the script with `ScriptStackExhaustedException` — Bad quality and a script-log entry like any throwing script — while there is still stack left. A runtime check rather than an analyzer rule because recursion is not decidable from syntax (anonymous self-application never names itself); bounded recursion keeps working. Limits: the depth reached depends on the evaluating thread's stack size; a recursion that never re-enters script code is not seen (the sandbox exposes no such API); a script can catch the exception, which only unwinds it. Pinned by `ScriptRecursionContainmentTests` (one child process per shape) and `ScriptStackGuardInjectionTests`. Script authoring is gated by the AdminUI `ConfigEditor` policy — **`Administrator` or `Designer`**, not Administrator alone — and that same policy gates deploy, so a Designer can author a script and put it into production; that gate is the primary control (AdminUI-001 deliberately admitted Designers to the script pages and `/api/script-analysis/*`; lmxopcua#779 corrected this sentence, which used to say Admin-only). See `docs/ScriptEditor.md` §"Runtime safety".
|
||||
|
||||
`System.Threading.Tasks` is denied (Core.Scripting-003), which prevents `Task.Run` / `Parallel` fan-out that would spawn background work outliving the timeout.
|
||||
|
||||
|
||||
#### Out-of-process runner (`Scripting:Runner`, lmxopcua#780)
|
||||
|
||||
`Scripting:Runner` selects where compiled scripts run: **`InProcess`** (the default and the rollback — everything
|
||||
above, unchanged), **`OutOfProcess`**, or `Shadow` (design P2; not implemented yet, so it runs `InProcess` with a
|
||||
startup Warning). `Scripting:Runner:Seams` narrows `OutOfProcess` to a subset (`VirtualTag`, `Calculation`; the
|
||||
`ScriptedAlarm` seam stays in process for now — alarms go last, design §14 #10). Under `OutOfProcess` the virtual-tag
|
||||
and Calculation evaluators hand each compiled run to `OutOfProcessScriptRunner`
|
||||
(`Core.Scripting.OutOfProcess`), which runs it in a pool of `ZB.MOM.WW.OtOpcUa.ScriptWorker` processes:
|
||||
|
||||
- **K shards** (`Scripting:Worker:Shards`, default 2) placed by a stable FNV-1a hash of the key, plus an on-demand
|
||||
**isolation worker** that runs suspects one at a time. No worker starts until the first compiled evaluation, so a
|
||||
node without scripts never launches one. The passthrough fast path never leaves the host.
|
||||
- **Two deadlines.** At the soft deadline `T` (the seam's budget, unchanged) the caller gets today's
|
||||
`script timed out after …` failure and the run token is cancelled inside the worker; at `T + G`
|
||||
(`Scripting:Worker:HardGrace`, default `max(1 s, T)`) the worker is **killed** (`Process.Kill(entireProcessTree)`,
|
||||
plus a worker-side fail-fast watchdog) — the thread, its memory and its ALC go with it.
|
||||
- **Attribution and quarantine** live in the host and survive restarts: a deadline kill strikes exactly its key
|
||||
(other evaluations in flight are victims, not strikes); a crash with no deadline crossed makes every key in flight
|
||||
a suspect that must complete `Scripting:Worker:IsolationProbationRuns` (20) clean runs in the isolation worker,
|
||||
where a crash strikes exactly one key. `Scripting:Worker:QuarantineAfterKills` (2) strikes within an hour, or the
|
||||
existing consecutive-timeout and allocation rules, quarantine with the usual `"script quarantined: "` prefix; a
|
||||
source change or a redeploy lifts it.
|
||||
- **Status codes.** Script faults keep `BadInternalError`. When the runner, not the script, failed — the worker
|
||||
restarted under the run (`"script worker restarted"`) or no worker became ready within
|
||||
`Scripting:Worker:UnavailableGrace` (5 s, `"script runner unavailable"`) — the tag publishes
|
||||
**`BadResourceUnavailable` (0x80040000)** with its last value. This is ScadaBridge-visible (ScadaBridge#112).
|
||||
The runner **fails closed**: no worker means Bad, never an in-process run.
|
||||
- **Containment.** The worker gets a cleared, allow-listed environment (no `ServerHistorian__ApiKey`, no
|
||||
connection strings), an empty temp working directory, `DOTNET_GCHeapHardLimit = Scripting:Worker:HeapLimitBytes`
|
||||
(512 MiB), a Windows Job Object (kill-on-close, one process, a commit backstop), a random current-user-only pipe
|
||||
with a launch nonce in its environment, and no Akka / gRPC assemblies in its folder. It exits on pipe end of stream
|
||||
or when its parent pid disappears. Results cross as protobuf `Variant`s carrying the exact CLR type, so
|
||||
`(short)ctx.GetTag(x).Value` behaves as in process; an input or result type with no tag (a `List<T>`, a `Guid`)
|
||||
is an explicit `unsupported type: …` failure rather than a conversion — a behaviour change from in process.
|
||||
- **Deployment.** The Host build copies the worker into `script-worker/` beside `OtOpcUa.Host.dll`
|
||||
(`ScriptWorker.Deploy.targets`, Build and Publish); `Scripting:Worker:ExecutablePath` overrides it. It is run through
|
||||
the dotnet muxer, so it needs a framework-dependent host (the docker-dev image). Not yet done (lmxopcua#780 P1
|
||||
remainder): the PE cache (`Scripting:Worker:PeCacheDir` is validated — absolute, outside the content root — but not
|
||||
used, so a worker restart recompiles), a self-contained worker for the Windows installs, the low-privilege worker
|
||||
account and firewall rule, the runner-state gauge / readiness component, and the scripted-alarm seam.
|
||||
|
||||
Pinned by `tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests` (`Category=ScriptWorker`: real worker processes).
|
||||
|
||||
#### Quarantine: what a client sees (status code)
|
||||
|
||||
A quarantined script's output reads **`BadInternalError`** on OPC UA, the same code as a script that
|
||||
|
||||
@@ -665,6 +665,18 @@ per-shard state, restarts in the last hour and the quarantine list. Quarantines
|
||||
- Default stays `InProcess`. Enable on the docker-dev rig only.
|
||||
- Gate: §13 runaway / containment tests green on Linux (docker-dev) **and** Windows (windev box, which has
|
||||
the .NET 10 toolchain).
|
||||
- **Partly landed in lmxopcua#780 (P1 slice):** `Scripting:Runner` / `Scripting:Runner:Seams` / `Scripting:Worker:*`
|
||||
(bound, clamped, `PeCacheDir` path rule fail-closed), `IScriptRunner` + `InProcessScriptRunner`, the protobuf frame
|
||||
protocol (`Core.Scripting.Protocol`), the `ScriptWorker` app (compile in the worker, P-thread scheduler, watchdog,
|
||||
heartbeats, log forwarding, pipe-EOF / parent-pid exit, forbidden-assembly self-check), `OutOfProcessScriptRunner`
|
||||
(K shards + isolation worker, soft/hard deadlines, attribution, kill-strike quarantine, backoff, fail closed,
|
||||
Job Object on Windows), the VirtualTag and Calculation seams over it, and `BadResourceUnavailable` for
|
||||
runner-unavailable / worker-restarted (ScadaBridge#112). The §13 containment cases run on macOS and Windows
|
||||
(windev). Deviations: supervision is a plain object owned by DI, not `ScriptRunnerActor`/`ScriptWorkerActor`;
|
||||
the per-seam batch submit (P0) is still one-at-a-time. Remaining: the HMAC'd PE cache, a self-contained worker
|
||||
publish + the low-privilege account / pipe ACL / firewall rule (installer), the runner-state gauge and readiness
|
||||
component, victim re-evaluation on shard return, isolation-worker idle stop, the orphan sweep by nonce file, the
|
||||
scripted-alarm seam, the docker-dev enablement and the §8 perf harness.
|
||||
3. **P2, shadow mode on docker-dev and QA.**
|
||||
- `Shadow` runs the **in-process** evaluation as the published result, and *also* submits to the worker.
|
||||
- It compares value, status class and failure kind per evaluation, and counts
|
||||
|
||||
@@ -87,6 +87,19 @@ public sealed record VirtualTagEvalResult(bool Success, object? Value, string? R
|
||||
/// <param name="reason">Description of the failure cause.</param>
|
||||
/// <returns>A failed evaluation result.</returns>
|
||||
public static VirtualTagEvalResult Failure(string reason) => new(false, null, reason);
|
||||
|
||||
/// <summary>
|
||||
/// The OPC UA status a <see cref="Failure"/> publishes. Null (every failure the in-process path produces) means
|
||||
/// the seam's long-standing <c>BadInternalError</c>; the out-of-process runner sets
|
||||
/// <see cref="OpcUa.OpcUaStatus.BadResourceUnavailable"/> when the runner, not the script, failed (lmxopcua#780).
|
||||
/// </summary>
|
||||
public uint? StatusCode { get; init; }
|
||||
|
||||
/// <summary>A failure that is the runner's, not the script's: published <c>BadResourceUnavailable</c>.</summary>
|
||||
/// <param name="reason">Description of the failure cause.</param>
|
||||
/// <returns>A failed evaluation result carrying <see cref="OpcUa.OpcUaStatus.BadResourceUnavailable"/>.</returns>
|
||||
public static VirtualTagEvalResult Unavailable(string reason) =>
|
||||
new(false, null, reason) { StatusCode = OpcUa.OpcUaStatus.BadResourceUnavailable };
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
@@ -47,6 +47,12 @@ public static class OpcUaStatus
|
||||
/// see docs/VirtualTags.md §"Quarantine: what a client sees").</summary>
|
||||
public const uint BadInternalError = 0x80020000;
|
||||
|
||||
/// <summary>The operating system / runtime could not provide a resource. The out-of-process script runner's code
|
||||
/// for "the script did not run and it is not the script's fault" — its worker restarted under it, or no worker
|
||||
/// was ready in time (lmxopcua#780, design §9). Distinct from <see cref="BadInternalError"/> so an operator, and
|
||||
/// ScadaBridge, can tell a sick node from a broken script. Only produced under <c>Scripting:Runner=OutOfProcess</c>.</summary>
|
||||
public const uint BadResourceUnavailable = 0x80040000;
|
||||
|
||||
/// <summary>Part 4 severity bucket of <paramref name="statusCode"/>: the top two bits.</summary>
|
||||
/// <param name="statusCode">The status code to classify.</param>
|
||||
/// <returns>0 for Good, 1 for Uncertain, 2 or 3 for Bad.</returns>
|
||||
|
||||
@@ -0,0 +1,505 @@
|
||||
using System.Collections.Concurrent;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
using Serilog;
|
||||
using Serilog.Core;
|
||||
using Serilog.Events;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Engines;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
|
||||
|
||||
/// <summary>
|
||||
/// The pre-emptible <see cref="IScriptRunner"/> (lmxopcua#780, design §3–§9): compiled scripts run in a pool of
|
||||
/// <c>ScriptWorker</c> processes — K shards placed by a stable hash of the script key, plus an isolation worker
|
||||
/// for suspects — so a script that overruns its hard deadline is stopped by killing its worker, and an OOM, a
|
||||
/// native crash or a fail-fast takes down a worker instead of the server.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <list type="bullet">
|
||||
/// <item><b>Two deadlines.</b> At the soft deadline <c>T</c> the caller gets today's timeout failure (and the
|
||||
/// run token is cancelled inside the worker). At <c>T + G</c> (<see cref="ScriptWorkerOptions.EffectiveHardGrace"/>)
|
||||
/// the worker is killed and the key takes a kill strike.</item>
|
||||
/// <item><b>Attribution.</b> A deadline kill blames exactly one key; the other evaluations in flight on that
|
||||
/// worker are victims and report <see cref="ScriptRunOutcome.WorkerRestartedReason"/> (no strike). A crash with
|
||||
/// no deadline crossed makes every key in flight a suspect, and suspects run in the isolation worker one at a
|
||||
/// time, where a crash names exactly one key. A suspect returns to its shard after
|
||||
/// <see cref="ScriptWorkerOptions.IsolationProbationRuns"/> clean runs.</item>
|
||||
/// <item><b>Quarantine</b> lives here, in the host, so it survives worker restarts: kill strikes (within an
|
||||
/// hour), consecutive soft timeouts, or one allocation-budget breach. Reasons keep the
|
||||
/// <see cref="ScriptRunGuard.QuarantinedReasonPrefix"/>, and a source change or <see cref="Reset"/> lifts them.</item>
|
||||
/// <item><b>Fail closed.</b> When no worker is ready within <see cref="ScriptWorkerOptions.UnavailableGrace"/>
|
||||
/// the result is <see cref="ScriptRunStatus.Unavailable"/> — never an in-process run.</item>
|
||||
/// </list>
|
||||
/// </remarks>
|
||||
public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable, IDisposable
|
||||
{
|
||||
/// <summary>The worker entry assembly's file name.</summary>
|
||||
public const string WorkerAssemblyFileName = "ZB.MOM.WW.OtOpcUa.ScriptWorker.dll";
|
||||
|
||||
/// <summary>The folder under the host's base directory the Host build places the worker in.</summary>
|
||||
public const string WorkerFolderName = "script-worker";
|
||||
|
||||
/// <summary>Failure reason when the worker could not take more work.</summary>
|
||||
public const string WorkerBusyReason = "script worker busy";
|
||||
|
||||
private static readonly TimeSpan KillStrikeWindow = TimeSpan.FromHours(1);
|
||||
|
||||
private readonly ScriptingOptions _options;
|
||||
private readonly ScriptWorkerOptions _worker;
|
||||
private readonly ILogger _log;
|
||||
private readonly ILogger _scriptRoot;
|
||||
private readonly WorkerShard[] _shards;
|
||||
private readonly WorkerShard _isolation;
|
||||
private readonly ConcurrentDictionary<string, KeyState> _keys = new(StringComparer.Ordinal);
|
||||
private readonly ConcurrentDictionary<string, string> _compileFailures = new(StringComparer.Ordinal);
|
||||
private int _disposed;
|
||||
|
||||
/// <summary>Initializes the runner. No process starts until the first evaluation.</summary>
|
||||
/// <param name="options">The node's <c>Scripting</c> options (sanitized again here).</param>
|
||||
/// <param name="log">Host diagnostics: starts, kills, quarantines; null ⇒ Serilog's global logger.</param>
|
||||
/// <param name="scriptRoot">Fallback logger for forwarded <c>ctx.Logger</c> lines; null ⇒ <paramref name="log"/>.</param>
|
||||
public OutOfProcessScriptRunner(ScriptingOptions options, ILogger? log = null, ILogger? scriptRoot = null)
|
||||
: this(options, log, scriptRoot, heartbeatInterval: TimeSpan.FromSeconds(1), loadTimeout: TimeSpan.FromSeconds(60))
|
||||
{
|
||||
}
|
||||
|
||||
internal OutOfProcessScriptRunner(ScriptingOptions options, ILogger? log, ILogger? scriptRoot,
|
||||
TimeSpan heartbeatInterval, TimeSpan loadTimeout)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
_options = options.Sanitized(out _);
|
||||
_worker = _options.Worker;
|
||||
_log = log ?? Log.Logger;
|
||||
_scriptRoot = scriptRoot ?? _log;
|
||||
LoadTimeout = loadTimeout;
|
||||
WorkerPath = ResolveWorkerPath(_worker);
|
||||
|
||||
var evalThreads = Math.Max(2, Environment.ProcessorCount / _worker.Shards);
|
||||
_shards = new WorkerShard[_worker.Shards];
|
||||
for (var i = 0; i < _shards.Length; i++)
|
||||
{
|
||||
var label = i.ToString(System.Globalization.CultureInfo.InvariantCulture);
|
||||
_shards[i] = NewShard(label, isolation: false, evalThreads, heartbeatInterval);
|
||||
}
|
||||
_isolation = NewShard("iso", isolation: true, evalThreads: 1, heartbeatInterval);
|
||||
}
|
||||
|
||||
/// <summary>The worker entry assembly this runner launches.</summary>
|
||||
public string WorkerPath { get; }
|
||||
|
||||
/// <summary>How long a compile in the worker may take before the worker is killed.</summary>
|
||||
internal TimeSpan LoadTimeout { get; }
|
||||
|
||||
/// <summary>The shard workers (tests).</summary>
|
||||
internal IReadOnlyList<WorkerShard> Shards => _shards;
|
||||
|
||||
/// <summary>The isolation worker (tests).</summary>
|
||||
internal WorkerShard Isolation => _isolation;
|
||||
|
||||
/// <summary>Scoped keys currently treated as suspects (tests / diagnostics).</summary>
|
||||
internal IReadOnlyCollection<string> SuspectKeys =>
|
||||
_keys.Where(kv => kv.Value.Suspect).Select(kv => kv.Key).ToArray();
|
||||
|
||||
/// <summary>The shard a key is placed on when it is not a suspect: a stable FNV-1a hash, so placement survives restarts.</summary>
|
||||
/// <param name="scopedKey">The scoped key.</param>
|
||||
/// <param name="shardCount">K.</param>
|
||||
/// <returns>The shard index.</returns>
|
||||
internal static int ShardIndex(string scopedKey, int shardCount)
|
||||
{
|
||||
var hash = 2166136261u;
|
||||
foreach (var b in Encoding.UTF8.GetBytes(scopedKey))
|
||||
{
|
||||
hash ^= b;
|
||||
hash *= 16777619u;
|
||||
}
|
||||
return (int)(hash % (uint)shardCount);
|
||||
}
|
||||
|
||||
/// <summary>The default worker path, or the configured one.</summary>
|
||||
/// <param name="worker">The worker options.</param>
|
||||
/// <returns>The absolute worker path.</returns>
|
||||
public static string ResolveWorkerPath(ScriptWorkerOptions worker)
|
||||
=> string.IsNullOrWhiteSpace(worker.ExecutablePath)
|
||||
? Path.Combine(AppContext.BaseDirectory, WorkerFolderName, WorkerAssemblyFileName)
|
||||
: Path.GetFullPath(worker.ExecutablePath);
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(invocation);
|
||||
if (Volatile.Read(ref _disposed) != 0) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.RunnerUnavailableReason);
|
||||
|
||||
var key = InProcessScriptRunner.ScopedKey(invocation.Kind, invocation.Key);
|
||||
var state = Admit(key, invocation.Source);
|
||||
if (state.QuarantineReason is { } quarantined) return ScriptRunOutcome.Failed(quarantined);
|
||||
|
||||
var sha = SHA256.HashData(Encoding.UTF8.GetBytes(invocation.Source));
|
||||
var failureKey = InProcessScriptRunner.ScopedKey(invocation.Kind, Convert.ToHexString(sha));
|
||||
if (_compileFailures.TryGetValue(failureKey, out var knownFailure)) return ScriptRunOutcome.Failed(knownFailure);
|
||||
|
||||
if (Interlocked.CompareExchange(ref state.InFlight, 1, 0) != 0)
|
||||
{
|
||||
ScriptRunGuard.RefusedCounter.Add(1, new KeyValuePair<string, object?>("script", key));
|
||||
return ScriptRunOutcome.Failed(ScriptRunGuard.PreviousRunStillRunningReason);
|
||||
}
|
||||
|
||||
var handedOff = false;
|
||||
SemaphoreSlim? gate = null;
|
||||
try
|
||||
{
|
||||
state.ScriptLogger = invocation.ScriptLogger;
|
||||
var shard = state.Suspect ? _isolation : _shards[ShardIndex(key, _shards.Length)];
|
||||
if (shard.IsolationGate is { } isolationGate)
|
||||
{
|
||||
if (!await isolationGate.WaitAsync(_worker.UnavailableGrace, ct).ConfigureAwait(false))
|
||||
return ScriptRunOutcome.Unavailable(ScriptRunOutcome.RunnerUnavailableReason);
|
||||
gate = isolationGate;
|
||||
}
|
||||
|
||||
var connection = await shard.GetReadyAsync(_worker.UnavailableGrace, ct).ConfigureAwait(false);
|
||||
if (connection is null) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.RunnerUnavailableReason);
|
||||
|
||||
var paths = invocation.Inputs.Keys.Order(StringComparer.Ordinal).ToArray();
|
||||
WorkerConnection.LoadEntry? entry;
|
||||
try
|
||||
{
|
||||
entry = await connection
|
||||
.EnsureLoadedAsync(key, WorkerShard.ToWire(invocation.Kind), invocation.Source, sha, paths)
|
||||
.WaitAsync(LoadTimeout, ct).ConfigureAwait(false);
|
||||
}
|
||||
catch (TimeoutException)
|
||||
{
|
||||
// A compile that never finishes is the script's fault (a pathological source): the worker dies, the
|
||||
// key takes a kill strike through the exit handler.
|
||||
_log.Warning("Script {ScriptKey}: compile in ScriptWorker[{Shard}] exceeded {Timeout} s; killing the worker",
|
||||
key, shard.Label, LoadTimeout.TotalSeconds);
|
||||
connection.Kill(WorkerExitReason.KilledLoadTimeout, key);
|
||||
return ScriptRunOutcome.Failed($"compile failure: compile did not finish within {LoadTimeout.TotalSeconds:F0} s");
|
||||
}
|
||||
if (entry?.Outcome is not { } outcome) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason);
|
||||
if (!outcome.Ok)
|
||||
{
|
||||
var reason = outcome.Error?.Kind switch
|
||||
{
|
||||
ScriptError.Types.Kind.Compile => $"compile error: {outcome.Error.Message}",
|
||||
ScriptError.Types.Kind.Sandbox => $"sandbox violation: {outcome.Error.Message}",
|
||||
_ => $"compile failure: {outcome.Error?.Message}",
|
||||
};
|
||||
// Remembered until the next deploy boundary, so a broken script never re-enters Roslyn (design §7).
|
||||
_compileFailures.TryAdd(failureKey, reason);
|
||||
return ScriptRunOutcome.Failed(reason);
|
||||
}
|
||||
|
||||
var pending = connection.Submit(key, BuildRequest(key, entry, invocation));
|
||||
ArmHardDeadline(connection, pending, invocation.Timeout + _worker.EffectiveHardGrace(invocation.Timeout));
|
||||
|
||||
// From here the in-flight slot (and the isolation gate) belong to the evaluation itself: they are released
|
||||
// when the worker answers or dies — not when this caller stops waiting at the soft deadline. That keeps a
|
||||
// key single-flight between T and T + G, and the isolation worker strictly one-at-a-time.
|
||||
handedOff = true;
|
||||
var heldGate = gate;
|
||||
_ = pending.Completion.ContinueWith(_ =>
|
||||
{
|
||||
Interlocked.Exchange(ref state.InFlight, 0);
|
||||
heldGate?.Release();
|
||||
}, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
|
||||
|
||||
EvalResult? result;
|
||||
try
|
||||
{
|
||||
result = await pending.Completion.WaitAsync(invocation.Timeout, ct).ConfigureAwait(false);
|
||||
}
|
||||
catch (TimeoutException)
|
||||
{
|
||||
// As in process: the run token is cancelled only once the caller has its timeout, so a polling script
|
||||
// ends itself (and frees its slot) without the worker being killed.
|
||||
connection.Cancel(pending);
|
||||
if (ct.IsCancellationRequested) return ScriptRunOutcome.Failed("evaluation cancelled");
|
||||
if (NoteTimeout(key, state, invocation.Timeout)) return ScriptRunOutcome.Failed(state.QuarantineReason!);
|
||||
return ScriptRunOutcome.Failed($"{ScriptEvalFailureReasons.TimedOutPrefix}{invocation.Timeout.TotalSeconds:F1}s");
|
||||
}
|
||||
|
||||
if (result is null) return ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason);
|
||||
return Complete(key, state, result);
|
||||
}
|
||||
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
||||
{
|
||||
return ScriptRunOutcome.Failed("evaluation cancelled");
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (!handedOff)
|
||||
{
|
||||
Interlocked.Exchange(ref state.InFlight, 0);
|
||||
gate?.Release();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Reset(ScriptContextKind kind)
|
||||
{
|
||||
var prefix = InProcessScriptRunner.ScopedKey(kind, string.Empty);
|
||||
var dropped = new List<string>();
|
||||
foreach (var key in _keys.Keys)
|
||||
{
|
||||
if (key.StartsWith(prefix, StringComparison.Ordinal) && _keys.TryRemove(key, out _)) dropped.Add(key);
|
||||
}
|
||||
foreach (var key in _compileFailures.Keys)
|
||||
{
|
||||
if (key.StartsWith(prefix, StringComparison.Ordinal)) _compileFailures.TryRemove(key, out _);
|
||||
}
|
||||
foreach (var shard in _shards.Append(_isolation))
|
||||
shard.Current?.Unload(dropped);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind)
|
||||
{
|
||||
var prefix = InProcessScriptRunner.ScopedKey(kind, string.Empty);
|
||||
return _keys
|
||||
.Where(kv => kv.Value.QuarantineReason is not null && kv.Key.StartsWith(prefix, StringComparison.Ordinal))
|
||||
.Select(kv => kv.Key[prefix.Length..])
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
/// <summary>Stops every worker.</summary>
|
||||
/// <returns>A task that completes when they have ended.</returns>
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
if (Interlocked.Exchange(ref _disposed, 1) != 0) return;
|
||||
await Task.WhenAll(_shards.Append(_isolation).Select(s => s.DisposeAsync().AsTask())).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>Stops every worker (blocking).</summary>
|
||||
public void Dispose() => DisposeAsync().AsTask().GetAwaiter().GetResult();
|
||||
|
||||
private WorkerShard NewShard(string label, bool isolation, int evalThreads, TimeSpan heartbeatInterval)
|
||||
{
|
||||
var spec = new WorkerLaunchSpec(
|
||||
Shard: label,
|
||||
WorkerPath: WorkerPath,
|
||||
HeapLimitBytes: _worker.HeapLimitBytes,
|
||||
EvalThreads: evalThreads,
|
||||
HardGrace: _worker.HardGrace ?? TimeSpan.FromSeconds(Math.Max(1, _options.RunTimeout.TotalSeconds)),
|
||||
HeartbeatInterval: heartbeatInterval,
|
||||
HandshakeTimeout: TimeSpan.FromSeconds(15));
|
||||
return new WorkerShard(label, isolation,
|
||||
ct => WorkerConnection.StartAsync(spec, _log, OnLogs, ct),
|
||||
OnWorkerExit, _log);
|
||||
}
|
||||
|
||||
private KeyState Admit(string key, string source)
|
||||
{
|
||||
while (true)
|
||||
{
|
||||
var state = _keys.GetOrAdd(key, static (_, src) => new KeyState(src), source);
|
||||
if (string.Equals(state.Source, source, StringComparison.Ordinal)) return state;
|
||||
// The script changed under this key: a clean slate, exactly as the in-process guard does. An evaluation
|
||||
// of the old source still in flight releases the OLD state's slot when it ends.
|
||||
if (_keys.TryUpdate(key, new KeyState(source), state)) continue;
|
||||
}
|
||||
}
|
||||
|
||||
private EvalRequest BuildRequest(string key, WorkerConnection.LoadEntry entry, ScriptInvocation invocation)
|
||||
{
|
||||
var request = new EvalRequest
|
||||
{
|
||||
Key = key,
|
||||
NowUtcTicks = DateTime.UtcNow.Ticks,
|
||||
SoftTimeoutMs = (uint)Math.Clamp(invocation.Timeout.TotalMilliseconds, 1, uint.MaxValue),
|
||||
HardGraceMs = (uint)Math.Clamp(_worker.EffectiveHardGrace(invocation.Timeout).TotalMilliseconds, 1, uint.MaxValue),
|
||||
};
|
||||
foreach (var (path, snapshot) in invocation.Inputs)
|
||||
{
|
||||
if (snapshot is null) continue;
|
||||
var input = new Input
|
||||
{
|
||||
StatusCode = snapshot.StatusCode,
|
||||
SourceTsTicks = snapshot.SourceTimestampUtc is { } source ? VariantCodec.ToUtc(source).Ticks : 0,
|
||||
ServerTsTicks = VariantCodec.ToUtc(snapshot.ServerTimestampUtc).Ticks,
|
||||
};
|
||||
if (entry.Slots.TryGetValue(path, out var slot)) input.Slot = slot;
|
||||
else input.Path = path;
|
||||
if (VariantCodec.TryEncode(snapshot.Value, out var variant, out var unsupported))
|
||||
{
|
||||
input.Value = variant;
|
||||
}
|
||||
else
|
||||
{
|
||||
// Never silently converted (design §14 #11): the worker answers UNSUPPORTED_TYPE for this input.
|
||||
input.Unsupported = true;
|
||||
input.UnsupportedType = unsupported ?? "?";
|
||||
}
|
||||
request.Inputs.Add(input);
|
||||
}
|
||||
return request;
|
||||
}
|
||||
|
||||
private void ArmHardDeadline(WorkerConnection connection, PendingEval pending, TimeSpan after)
|
||||
{
|
||||
var cts = new CancellationTokenSource(after);
|
||||
var registration = cts.Token.Register(() =>
|
||||
{
|
||||
if (pending.Completion.IsCompleted) return;
|
||||
_log.Warning("Script {ScriptKey} exceeded its hard deadline ({Deadline} ms); killing ScriptWorker[{Shard}] pid {Pid}",
|
||||
pending.Key, (long)after.TotalMilliseconds, connection.Shard, connection.Pid);
|
||||
connection.Kill(WorkerExitReason.KilledDeadline, pending.Key);
|
||||
});
|
||||
_ = pending.Completion.ContinueWith(_ =>
|
||||
{
|
||||
registration.Dispose();
|
||||
cts.Dispose();
|
||||
}, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
|
||||
}
|
||||
|
||||
private ScriptRunOutcome Complete(string key, KeyState state, EvalResult result)
|
||||
{
|
||||
// The script returned control (a value or a throw): the soft-timeout streak resets, as in process.
|
||||
Interlocked.Exchange(ref state.ConsecutiveTimeouts, 0);
|
||||
|
||||
if (_options.MaxAllocBytesPerRun > 0 && (long)result.AllocatedBytes > _options.MaxAllocBytesPerRun)
|
||||
{
|
||||
Quarantine(key, state,
|
||||
$"one evaluation allocated {result.AllocatedBytes / (1024.0 * 1024.0):F1} MB, over the " +
|
||||
$"{_options.MaxAllocBytesPerRun / (1024.0 * 1024.0):F0} MB budget (Scripting:MaxAllocBytesPerRun); " +
|
||||
"the script is not evaluated again until its source changes or it is redeployed",
|
||||
"allocation");
|
||||
return ScriptRunOutcome.Failed(state.QuarantineReason!);
|
||||
}
|
||||
|
||||
if (state.Suspect && Interlocked.Increment(ref state.ProbationRuns) >= _worker.IsolationProbationRuns)
|
||||
{
|
||||
state.Suspect = false;
|
||||
_log.Information("Script {ScriptKey} completed {Runs} clean run(s) in the isolation worker; returning it to its shard",
|
||||
key, _worker.IsolationProbationRuns);
|
||||
}
|
||||
|
||||
switch (result.OutcomeCase)
|
||||
{
|
||||
case EvalResult.OutcomeOneofCase.Value:
|
||||
try
|
||||
{
|
||||
return ScriptRunOutcome.Ok(VariantCodec.Decode(result.Value));
|
||||
}
|
||||
catch (ScriptWorkerProtocolException ex)
|
||||
{
|
||||
return ScriptRunOutcome.Failed($"script worker returned an undecodable value: {ex.Message}");
|
||||
}
|
||||
case EvalResult.OutcomeOneofCase.Predicate:
|
||||
return ScriptRunOutcome.Ok(result.Predicate);
|
||||
case EvalResult.OutcomeOneofCase.Error:
|
||||
return result.Error.Kind switch
|
||||
{
|
||||
ScriptError.Types.Kind.Threw => ScriptRunOutcome.Failed($"script threw: {result.Error.Message}"),
|
||||
ScriptError.Types.Kind.UnsupportedType => ScriptRunOutcome.Failed($"unsupported type: {result.Error.Message}"),
|
||||
ScriptError.Types.Kind.Busy => ScriptRunOutcome.Failed(WorkerBusyReason),
|
||||
ScriptError.Types.Kind.NotLoaded => ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason),
|
||||
_ => ScriptRunOutcome.Failed($"script failed: {result.Error.Message}"),
|
||||
};
|
||||
default:
|
||||
return ScriptRunOutcome.Ok(null);
|
||||
}
|
||||
}
|
||||
|
||||
private bool NoteTimeout(string key, KeyState state, TimeSpan timeout)
|
||||
{
|
||||
var count = Interlocked.Increment(ref state.ConsecutiveTimeouts);
|
||||
if (count < _options.QuarantineAfterConsecutiveTimeouts) return false;
|
||||
Quarantine(key, state,
|
||||
$"{count} consecutive evaluation(s) timed out (budget {timeout.TotalMilliseconds:F0} ms); " +
|
||||
"the script is not evaluated again until its source changes or it is redeployed",
|
||||
"timeouts");
|
||||
return true;
|
||||
}
|
||||
|
||||
private void OnWorkerExit(WorkerShard shard, WorkerExit exit)
|
||||
{
|
||||
switch (exit.Reason)
|
||||
{
|
||||
case WorkerExitReason.KilledDeadline when exit.CulpritKey is { } culprit:
|
||||
KillStrike(culprit, "it exceeded its hard deadline and its worker was killed", "killed");
|
||||
break;
|
||||
case WorkerExitReason.KilledLoadTimeout when exit.CulpritKey is { } culprit:
|
||||
KillStrike(culprit, "its compile did not finish and its worker was killed", "killed");
|
||||
break;
|
||||
case WorkerExitReason.Crash or WorkerExitReason.KilledHeartbeat or WorkerExitReason.ProtocolFault:
|
||||
if (shard.IsIsolation)
|
||||
{
|
||||
// One evaluation at a time here: the crash is this key's (design §5.3.3).
|
||||
foreach (var key in exit.InFlightKeys)
|
||||
KillStrike(key, "it crashed the isolation worker", "isolated_crash");
|
||||
}
|
||||
else if (exit.InFlightKeys.Count > 0)
|
||||
{
|
||||
foreach (var key in exit.InFlightKeys)
|
||||
{
|
||||
if (!_keys.TryGetValue(key, out var state)) continue;
|
||||
state.Suspect = true;
|
||||
Interlocked.Exchange(ref state.ProbationRuns, 0);
|
||||
}
|
||||
_log.Warning("ScriptWorker[{Shard}] ended ({Reason}) with no deadline crossed; {Count} script(s) in flight move to the isolation worker: {Suspects}",
|
||||
shard.Label, exit.Reason, exit.InFlightKeys.Count, string.Join(", ", exit.InFlightKeys));
|
||||
}
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
private void KillStrike(string key, string why, string metricReason)
|
||||
{
|
||||
if (!_keys.TryGetValue(key, out var state)) return;
|
||||
int strikes;
|
||||
lock (state.KillStrikes)
|
||||
{
|
||||
var now = DateTime.UtcNow;
|
||||
state.KillStrikes.Add(now);
|
||||
state.KillStrikes.RemoveAll(t => now - t > KillStrikeWindow);
|
||||
strikes = state.KillStrikes.Count;
|
||||
}
|
||||
if (strikes >= _worker.QuarantineAfterKills)
|
||||
{
|
||||
Quarantine(key, state,
|
||||
$"{why} ({strikes} time(s) in the last hour); the script is not evaluated again until its source changes or it is redeployed",
|
||||
metricReason);
|
||||
}
|
||||
}
|
||||
|
||||
private void Quarantine(string key, KeyState state, string why, string metricReason)
|
||||
{
|
||||
var reason = ScriptRunGuard.QuarantinedReasonPrefix + why;
|
||||
if (Interlocked.CompareExchange(ref state.QuarantineReason, reason, null) is not null) return;
|
||||
ScriptRunGuard.QuarantinedCounter.Add(1,
|
||||
new KeyValuePair<string, object?>("script", key),
|
||||
new KeyValuePair<string, object?>("reason", metricReason));
|
||||
_scriptRoot.ForContext(ScriptLoggerFactory.ScriptNameProperty, key).Error("Script {ScriptKey} quarantined: {Reason}", key, why);
|
||||
}
|
||||
|
||||
private void OnLogs(LogBatch batch)
|
||||
{
|
||||
foreach (var line in batch.Lines)
|
||||
{
|
||||
var logger = _keys.TryGetValue(line.Key, out var state) && state.ScriptLogger is { } scoped ? scoped : _scriptRoot;
|
||||
var level = line.Level <= (uint)LogEventLevel.Fatal ? (LogEventLevel)line.Level : LogEventLevel.Information;
|
||||
if (string.IsNullOrEmpty(line.Exception))
|
||||
logger.Write(level, "{ScriptMessage:l}", line.Rendered);
|
||||
else
|
||||
logger.Write(level, "{ScriptMessage:l}{NewLine}{ScriptException:l}", line.Rendered, Environment.NewLine, line.Exception);
|
||||
}
|
||||
if (batch.Dropped > 0)
|
||||
_log.Warning("ScriptWorker dropped {Dropped} script log line(s): a script is logging faster than the pipe drains", batch.Dropped);
|
||||
}
|
||||
|
||||
private sealed class KeyState(string source)
|
||||
{
|
||||
public readonly string Source = source;
|
||||
public readonly List<DateTime> KillStrikes = [];
|
||||
public int InFlight;
|
||||
public int ConsecutiveTimeouts;
|
||||
public int ProbationRuns;
|
||||
public volatile bool Suspect;
|
||||
public string? QuarantineReason;
|
||||
public ILogger? ScriptLogger;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,114 @@
|
||||
using System.ComponentModel;
|
||||
using System.Diagnostics;
|
||||
using System.Runtime.InteropServices;
|
||||
using System.Runtime.Versioning;
|
||||
using Microsoft.Win32.SafeHandles;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
|
||||
|
||||
/// <summary>
|
||||
/// A Windows Job Object holding one ScriptWorker (design §3.3, §5.2): <c>KILL_ON_JOB_CLOSE</c> so the kernel kills
|
||||
/// the worker when the host dies (its handle closes with it), <c>ACTIVE_PROCESS = 1</c> so the worker can start no
|
||||
/// child process, and a process-memory backstop slightly above the managed heap limit for native and JIT memory.
|
||||
/// </summary>
|
||||
[SupportedOSPlatform("windows")]
|
||||
internal sealed class WindowsJobObject : IDisposable
|
||||
{
|
||||
private const int JobObjectExtendedLimitInformation = 9;
|
||||
private const uint JOB_OBJECT_LIMIT_ACTIVE_PROCESS = 0x00000008;
|
||||
private const uint JOB_OBJECT_LIMIT_PROCESS_MEMORY = 0x00000100;
|
||||
private const uint JOB_OBJECT_LIMIT_DIE_ON_UNHANDLED_EXCEPTION = 0x00000400;
|
||||
private const uint JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE = 0x00002000;
|
||||
|
||||
private readonly SafeFileHandle _handle;
|
||||
|
||||
private WindowsJobObject(SafeFileHandle handle) => _handle = handle;
|
||||
|
||||
/// <summary>Creates a job with the worker limits and assigns <paramref name="process"/> to it.</summary>
|
||||
/// <param name="process">The freshly started worker.</param>
|
||||
/// <param name="processMemoryLimitBytes">The per-process commit limit.</param>
|
||||
/// <returns>The job; dispose it to kill the worker.</returns>
|
||||
public static WindowsJobObject Assign(Process process, long processMemoryLimitBytes)
|
||||
{
|
||||
var handle = CreateJobObjectW(IntPtr.Zero, null);
|
||||
if (handle.IsInvalid) throw new Win32Exception(Marshal.GetLastWin32Error(), "CreateJobObject failed");
|
||||
var job = new WindowsJobObject(handle);
|
||||
try
|
||||
{
|
||||
var info = new JOBOBJECT_EXTENDED_LIMIT_INFORMATION
|
||||
{
|
||||
BasicLimitInformation = new JOBOBJECT_BASIC_LIMIT_INFORMATION
|
||||
{
|
||||
LimitFlags = JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE | JOB_OBJECT_LIMIT_ACTIVE_PROCESS
|
||||
| JOB_OBJECT_LIMIT_PROCESS_MEMORY | JOB_OBJECT_LIMIT_DIE_ON_UNHANDLED_EXCEPTION,
|
||||
ActiveProcessLimit = 1,
|
||||
},
|
||||
ProcessMemoryLimit = (UIntPtr)(ulong)processMemoryLimitBytes,
|
||||
};
|
||||
if (!SetInformationJobObject(handle, JobObjectExtendedLimitInformation, ref info,
|
||||
(uint)Marshal.SizeOf<JOBOBJECT_EXTENDED_LIMIT_INFORMATION>()))
|
||||
throw new Win32Exception(Marshal.GetLastWin32Error(), "SetInformationJobObject failed");
|
||||
if (!AssignProcessToJobObject(handle, process.Handle))
|
||||
throw new Win32Exception(Marshal.GetLastWin32Error(), "AssignProcessToJobObject failed");
|
||||
return job;
|
||||
}
|
||||
catch
|
||||
{
|
||||
job.Dispose();
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Closes the job handle, which kills every process still in it.</summary>
|
||||
public void Dispose() => _handle.Dispose();
|
||||
|
||||
[StructLayout(LayoutKind.Sequential)]
|
||||
private struct JOBOBJECT_BASIC_LIMIT_INFORMATION
|
||||
{
|
||||
public long PerProcessUserTimeLimit;
|
||||
public long PerJobUserTimeLimit;
|
||||
public uint LimitFlags;
|
||||
public UIntPtr MinimumWorkingSetSize;
|
||||
public UIntPtr MaximumWorkingSetSize;
|
||||
public uint ActiveProcessLimit;
|
||||
public UIntPtr Affinity;
|
||||
public uint PriorityClass;
|
||||
public uint SchedulingClass;
|
||||
}
|
||||
|
||||
[StructLayout(LayoutKind.Sequential)]
|
||||
private struct IO_COUNTERS
|
||||
{
|
||||
public ulong ReadOperationCount;
|
||||
public ulong WriteOperationCount;
|
||||
public ulong OtherOperationCount;
|
||||
public ulong ReadTransferCount;
|
||||
public ulong WriteTransferCount;
|
||||
public ulong OtherTransferCount;
|
||||
}
|
||||
|
||||
[StructLayout(LayoutKind.Sequential)]
|
||||
private struct JOBOBJECT_EXTENDED_LIMIT_INFORMATION
|
||||
{
|
||||
public JOBOBJECT_BASIC_LIMIT_INFORMATION BasicLimitInformation;
|
||||
public IO_COUNTERS IoInfo;
|
||||
public UIntPtr ProcessMemoryLimit;
|
||||
public UIntPtr JobMemoryLimit;
|
||||
public UIntPtr PeakProcessMemoryUsed;
|
||||
public UIntPtr PeakJobMemoryUsed;
|
||||
}
|
||||
|
||||
#pragma warning disable SYSLIB1054 // DllImport kept: LibraryImport would need AllowUnsafeBlocks for three calls.
|
||||
[DllImport("kernel32.dll", SetLastError = true, CharSet = CharSet.Unicode)]
|
||||
private static extern SafeFileHandle CreateJobObjectW(IntPtr securityAttributes, string? name);
|
||||
|
||||
[DllImport("kernel32.dll", SetLastError = true)]
|
||||
[return: MarshalAs(UnmanagedType.Bool)]
|
||||
private static extern bool SetInformationJobObject(SafeFileHandle job, int infoClass,
|
||||
ref JOBOBJECT_EXTENDED_LIMIT_INFORMATION info, uint length);
|
||||
|
||||
[DllImport("kernel32.dll", SetLastError = true)]
|
||||
[return: MarshalAs(UnmanagedType.Bool)]
|
||||
private static extern bool AssignProcessToJobObject(SafeFileHandle job, IntPtr process);
|
||||
#pragma warning restore SYSLIB1054
|
||||
}
|
||||
@@ -0,0 +1,699 @@
|
||||
using System.Collections.Concurrent;
|
||||
using System.Diagnostics;
|
||||
using System.IO.Pipes;
|
||||
using System.Security.Cryptography;
|
||||
using System.Text;
|
||||
using System.Threading.Channels;
|
||||
using Serilog;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
|
||||
|
||||
/// <summary>How a worker process ended.</summary>
|
||||
internal enum WorkerExitReason
|
||||
{
|
||||
/// <summary>It exited by itself with code 0 (a Shutdown).</summary>
|
||||
Clean,
|
||||
|
||||
/// <summary>The host killed it: a script crossed its hard deadline (<see cref="WorkerExit.CulpritKey"/>).</summary>
|
||||
KilledDeadline,
|
||||
|
||||
/// <summary>The host killed it: no frame for three heartbeat intervals.</summary>
|
||||
KilledHeartbeat,
|
||||
|
||||
/// <summary>The host killed it: a compile did not finish in time (<see cref="WorkerExit.CulpritKey"/>).</summary>
|
||||
KilledLoadTimeout,
|
||||
|
||||
/// <summary>It died by itself (OOM, heap limit, fail-fast, external kill).</summary>
|
||||
Crash,
|
||||
|
||||
/// <summary>A framing or handshake violation.</summary>
|
||||
ProtocolFault,
|
||||
|
||||
/// <summary>The runner is shutting down.</summary>
|
||||
Disposed,
|
||||
}
|
||||
|
||||
/// <summary>The end of one worker process.</summary>
|
||||
/// <param name="Reason">Why it ended.</param>
|
||||
/// <param name="CulpritKey">The scoped key the host blamed, for a deadline or load kill.</param>
|
||||
/// <param name="ExitCode">The OS exit code, when known.</param>
|
||||
/// <param name="InFlightKeys">The scoped keys that had an evaluation outstanding when it ended.</param>
|
||||
internal sealed record WorkerExit(WorkerExitReason Reason, string? CulpritKey, int? ExitCode, IReadOnlyList<string> InFlightKeys);
|
||||
|
||||
/// <summary>What the host needs to launch one worker.</summary>
|
||||
/// <param name="Shard">The shard label (<c>0</c>, <c>1</c>, … or <c>iso</c>).</param>
|
||||
/// <param name="WorkerPath">The worker entry assembly (<c>.dll</c>, run through the dotnet muxer) or apphost.</param>
|
||||
/// <param name="HeapLimitBytes">The worker's <c>DOTNET_GCHeapHardLimit</c>.</param>
|
||||
/// <param name="EvalThreads">The worker scheduler's width (1 for the isolation worker).</param>
|
||||
/// <param name="HardGrace">The worker watchdog's grace past the soft deadline.</param>
|
||||
/// <param name="HeartbeatInterval">How often the worker sends a heartbeat.</param>
|
||||
/// <param name="HandshakeTimeout">How long a launch may take to reach Ready.</param>
|
||||
internal sealed record WorkerLaunchSpec(
|
||||
string Shard,
|
||||
string WorkerPath,
|
||||
long HeapLimitBytes,
|
||||
int EvalThreads,
|
||||
TimeSpan HardGrace,
|
||||
TimeSpan HeartbeatInterval,
|
||||
TimeSpan HandshakeTimeout);
|
||||
|
||||
/// <summary>
|
||||
/// One live ScriptWorker process and its pipe (lmxopcua#780, design §3–§6): launch with a cleared, allow-listed
|
||||
/// environment, a random pipe that accepts exactly one client, a nonce handshake, a frame reader and a coalescing
|
||||
/// writer, heartbeat supervision, and the kill. It never restarts itself; <see cref="WorkerShard"/> does that.
|
||||
/// </summary>
|
||||
internal sealed class WorkerConnection : IAsyncDisposable
|
||||
{
|
||||
/// <summary>
|
||||
/// Environment variables a worker inherits from the host (design §6): the runtime's own knobs, the temp
|
||||
/// directories and the minimum a Windows process needs. Everything else — notably <c>ServerHistorian__ApiKey</c>,
|
||||
/// ConfigDb connection strings and every other <c>Section__Key</c> override — is dropped.
|
||||
/// </summary>
|
||||
internal static readonly IReadOnlyList<string> InheritedEnvironment =
|
||||
[
|
||||
"PATH", "TEMP", "TMP", "TMPDIR", "HOME", "SystemRoot", "SYSTEMROOT", "windir", "SystemDrive", "ComSpec",
|
||||
"PATHEXT", "NUMBER_OF_PROCESSORS", "PROCESSOR_ARCHITECTURE", "LOCALAPPDATA",
|
||||
];
|
||||
|
||||
private readonly Process _process;
|
||||
private readonly NamedPipeServerStream _pipe;
|
||||
private readonly IDisposable? _job;
|
||||
private readonly string _workDir;
|
||||
private readonly ILogger _log;
|
||||
private readonly Action<LogBatch> _onLogs;
|
||||
private readonly Channel<Envelope> _outbound = Channel.CreateUnbounded<Envelope>(new UnboundedChannelOptions { SingleReader = true });
|
||||
private readonly ConcurrentDictionary<ulong, PendingEval> _pending = new();
|
||||
private readonly ConcurrentDictionary<ulong, TaskCompletionSource<LoadResult>> _loads = new();
|
||||
private readonly ConcurrentDictionary<string, LoadEntry> _loaded = new(StringComparer.Ordinal);
|
||||
private readonly TaskCompletionSource<WorkerExit> _exited = new(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
private readonly CancellationTokenSource _stopping = new();
|
||||
private readonly TimeSpan _heartbeatInterval;
|
||||
private TaskCompletionSource<DiagnosticsReply>? _diag;
|
||||
private (WorkerExitReason Reason, string? Culprit)? _killedFor;
|
||||
private long _lastFrameTimestamp;
|
||||
private long _seq;
|
||||
private long _nextEvalId;
|
||||
private long _nextLoadId;
|
||||
private int _maxFrameBytes = ScriptWorkerFrameCodec.DefaultMaxFrameBytes;
|
||||
private int _disposed;
|
||||
|
||||
private WorkerConnection(string shard, Process process, NamedPipeServerStream pipe, IDisposable? job, string workDir,
|
||||
ILogger log, Action<LogBatch> onLogs, TimeSpan heartbeatInterval)
|
||||
{
|
||||
Shard = shard;
|
||||
_process = process;
|
||||
_pipe = pipe;
|
||||
_job = job;
|
||||
_workDir = workDir;
|
||||
_log = log;
|
||||
_onLogs = onLogs;
|
||||
_heartbeatInterval = heartbeatInterval;
|
||||
Pid = process.Id;
|
||||
}
|
||||
|
||||
/// <summary>The shard label.</summary>
|
||||
public string Shard { get; }
|
||||
|
||||
/// <summary>The worker's process id.</summary>
|
||||
public int Pid { get; }
|
||||
|
||||
/// <summary>Completes when the worker has ended, with why.</summary>
|
||||
public Task<WorkerExit> Exited => _exited.Task;
|
||||
|
||||
/// <summary>Whether the worker is still serving.</summary>
|
||||
public bool IsAlive => !_exited.Task.IsCompleted && !_stopping.IsCancellationRequested;
|
||||
|
||||
/// <summary><see cref="Stopwatch.GetTimestamp"/> when the handshake completed.</summary>
|
||||
public long ReadyTimestamp { get; private set; }
|
||||
|
||||
/// <summary>Evaluations currently outstanding on this worker.</summary>
|
||||
public int InFlightCount => _pending.Count;
|
||||
|
||||
/// <summary>Launches a worker and completes the handshake.</summary>
|
||||
/// <param name="spec">What to launch.</param>
|
||||
/// <param name="log">Host diagnostics (stderr lines, exits).</param>
|
||||
/// <param name="onLogs">Receives the worker's forwarded <c>ctx.Logger</c> batches.</param>
|
||||
/// <param name="ct">Cancellation.</param>
|
||||
/// <returns>A ready connection.</returns>
|
||||
public static async Task<WorkerConnection> StartAsync(WorkerLaunchSpec spec, ILogger log, Action<LogBatch> onLogs, CancellationToken ct)
|
||||
{
|
||||
var pipeName = "otopcua-sw-" + Convert.ToHexString(RandomNumberGenerator.GetBytes(12)).ToLowerInvariant();
|
||||
var nonce = Convert.ToHexString(RandomNumberGenerator.GetBytes(32));
|
||||
var workDir = Path.Combine(Path.GetTempPath(), "otopcua-scriptworker", pipeName);
|
||||
Directory.CreateDirectory(workDir);
|
||||
|
||||
// One instance, current user only: the pipe accepts exactly one connection, from this account.
|
||||
var pipe = new NamedPipeServerStream(pipeName, PipeDirection.InOut, maxNumberOfServerInstances: 1,
|
||||
PipeTransmissionMode.Byte, PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly);
|
||||
Process? process = null;
|
||||
IDisposable? job = null;
|
||||
try
|
||||
{
|
||||
var psi = BuildStartInfo(spec, pipeName, nonce, workDir);
|
||||
process = new Process { StartInfo = psi, EnableRaisingEvents = true };
|
||||
if (!process.Start()) throw new InvalidOperationException("the worker process did not start");
|
||||
if (OperatingSystem.IsWindows())
|
||||
{
|
||||
// Native and JIT memory backstop above the managed heap limit.
|
||||
job = WindowsJobObject.Assign(process, spec.HeapLimitBytes + 256L * 1024 * 1024);
|
||||
}
|
||||
var shardLabel = spec.Shard;
|
||||
process.ErrorDataReceived += (_, e) =>
|
||||
{
|
||||
if (e.Data is { Length: > 0 } line) log.Warning("ScriptWorker[{Shard}] {WorkerStderr}", shardLabel, line);
|
||||
};
|
||||
process.OutputDataReceived += (_, e) =>
|
||||
{
|
||||
if (e.Data is { Length: > 0 } line) log.Debug("ScriptWorker[{Shard}] stdout {WorkerStdout}", shardLabel, line);
|
||||
};
|
||||
process.BeginErrorReadLine();
|
||||
process.BeginOutputReadLine();
|
||||
|
||||
using var handshakeCts = CancellationTokenSource.CreateLinkedTokenSource(ct);
|
||||
handshakeCts.CancelAfter(spec.HandshakeTimeout);
|
||||
var exitedEarly = process.WaitForExitAsync(handshakeCts.Token);
|
||||
var accept = pipe.WaitForConnectionAsync(handshakeCts.Token);
|
||||
if (await Task.WhenAny(accept, exitedEarly).ConfigureAwait(false) != accept)
|
||||
throw new InvalidOperationException(
|
||||
$"the worker exited (code {SafeExitCode(process)}) before connecting to its pipe");
|
||||
await accept.ConfigureAwait(false);
|
||||
|
||||
var hello = new Hello
|
||||
{
|
||||
Nonce = nonce,
|
||||
MaxFrameBytes = ScriptWorkerFrameCodec.DefaultMaxFrameBytes,
|
||||
EvalThreads = (uint)spec.EvalThreads,
|
||||
QueueCapacity = 4096,
|
||||
HardGraceMs = (uint)Math.Clamp(spec.HardGrace.TotalMilliseconds, 1, uint.MaxValue),
|
||||
HeartbeatIntervalMs = (uint)spec.HeartbeatInterval.TotalMilliseconds,
|
||||
MaxLogLines = 1000,
|
||||
};
|
||||
await HandshakeAsync(pipe, hello, nonce, process.Id, handshakeCts.Token).ConfigureAwait(false);
|
||||
|
||||
var connection = new WorkerConnection(spec.Shard, process, pipe, job, workDir, log, onLogs, spec.HeartbeatInterval);
|
||||
connection.Begin();
|
||||
return connection;
|
||||
}
|
||||
catch
|
||||
{
|
||||
try { if (process is { HasExited: false }) process.Kill(entireProcessTree: true); } catch { /* best effort */ }
|
||||
job?.Dispose();
|
||||
process?.Dispose();
|
||||
await pipe.DisposeAsync().ConfigureAwait(false);
|
||||
TryDeleteDirectory(workDir);
|
||||
throw;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The host half of the handshake: send <paramref name="hello"/>, require a <c>Ready</c> as the first frame,
|
||||
/// echoing <paramref name="expectedNonce"/> (constant-time compare) from <paramref name="expectedPid"/>.
|
||||
/// </summary>
|
||||
/// <param name="stream">The accepted pipe.</param>
|
||||
/// <param name="hello">The Hello to send.</param>
|
||||
/// <param name="expectedNonce">The nonce put in the worker's environment.</param>
|
||||
/// <param name="expectedPid">The launched process id; null skips the check.</param>
|
||||
/// <param name="ct">Cancellation (the handshake timeout).</param>
|
||||
/// <returns>The worker's Ready.</returns>
|
||||
/// <exception cref="ScriptWorkerProtocolException">Anything but a matching Ready.</exception>
|
||||
internal static async Task<Ready> HandshakeAsync(Stream stream, Hello hello, string expectedNonce, int? expectedPid, CancellationToken ct)
|
||||
{
|
||||
await ScriptWorkerFrameCodec.WriteAsync(stream, new Envelope
|
||||
{
|
||||
ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion,
|
||||
Seq = 1,
|
||||
Hello = hello,
|
||||
}, ScriptWorkerFrameCodec.DefaultMaxFrameBytes, ct).ConfigureAwait(false);
|
||||
await stream.FlushAsync(ct).ConfigureAwait(false);
|
||||
|
||||
var reply = await ScriptWorkerFrameCodec.ReadAsync(stream, ScriptWorkerFrameCodec.DefaultMaxFrameBytes, ct).ConfigureAwait(false)
|
||||
?? throw new ScriptWorkerProtocolException("the worker closed the pipe during the handshake");
|
||||
if (reply.BodyCase != Envelope.BodyOneofCase.Ready)
|
||||
throw new ScriptWorkerProtocolException($"expected Ready, got {reply.BodyCase}");
|
||||
var expected = Encoding.UTF8.GetBytes(expectedNonce);
|
||||
var actual = Encoding.UTF8.GetBytes(reply.Ready.Nonce ?? string.Empty);
|
||||
if (!CryptographicOperations.FixedTimeEquals(expected, actual))
|
||||
throw new ScriptWorkerProtocolException("the worker's nonce does not match the launch nonce");
|
||||
if (expectedPid is { } pid && reply.Ready.Pid != pid)
|
||||
throw new ScriptWorkerProtocolException($"the pipe client reports pid {reply.Ready.Pid}, not the launched {pid}");
|
||||
return reply.Ready;
|
||||
}
|
||||
|
||||
private static ProcessStartInfo BuildStartInfo(WorkerLaunchSpec spec, string pipeName, string nonce, string workDir)
|
||||
{
|
||||
var psi = new ProcessStartInfo
|
||||
{
|
||||
UseShellExecute = false,
|
||||
CreateNoWindow = true,
|
||||
RedirectStandardError = true,
|
||||
RedirectStandardOutput = true,
|
||||
RedirectStandardInput = false,
|
||||
WorkingDirectory = workDir,
|
||||
};
|
||||
if (spec.WorkerPath.EndsWith(".dll", StringComparison.OrdinalIgnoreCase))
|
||||
{
|
||||
psi.FileName = ResolveMuxer();
|
||||
psi.ArgumentList.Add("exec");
|
||||
psi.ArgumentList.Add(spec.WorkerPath);
|
||||
}
|
||||
else
|
||||
{
|
||||
psi.FileName = spec.WorkerPath;
|
||||
}
|
||||
|
||||
// Cleared, then repopulated from an allow-list (design §6). ProcessStartInfo.Environment starts as a copy of
|
||||
// the host's environment; ScadaBridge's sandbox inherits it whole, which this design rejects.
|
||||
var inherited = psi.Environment.ToDictionary(kv => kv.Key, kv => kv.Value, StringComparer.Ordinal);
|
||||
psi.Environment.Clear();
|
||||
foreach (var (name, value) in inherited)
|
||||
{
|
||||
if (value is null) continue;
|
||||
if (InheritedEnvironment.Contains(name, StringComparer.OrdinalIgnoreCase)
|
||||
|| (name.StartsWith("DOTNET_", StringComparison.OrdinalIgnoreCase)
|
||||
&& !name.StartsWith("DOTNET_GCHeap", StringComparison.OrdinalIgnoreCase)))
|
||||
psi.Environment[name] = value;
|
||||
}
|
||||
psi.Environment["DOTNET_GCHeapHardLimit"] = "0x" + spec.HeapLimitBytes.ToString("X", System.Globalization.CultureInfo.InvariantCulture);
|
||||
psi.Environment["DOTNET_gcServer"] = "0";
|
||||
psi.Environment["DOTNET_CLI_TELEMETRY_OPTOUT"] = "1";
|
||||
psi.Environment[ScriptWorkerEnvironment.PipeName] = pipeName;
|
||||
psi.Environment[ScriptWorkerEnvironment.Nonce] = nonce;
|
||||
psi.Environment[ScriptWorkerEnvironment.ParentPid] = Environment.ProcessId.ToString(System.Globalization.CultureInfo.InvariantCulture);
|
||||
psi.Environment[ScriptWorkerEnvironment.Shard] = spec.Shard;
|
||||
return psi;
|
||||
}
|
||||
|
||||
/// <summary>The dotnet muxer that runs a framework-dependent worker <c>.dll</c>.</summary>
|
||||
/// <returns>The muxer path, or plain <c>dotnet</c> (resolved on PATH) as a last resort.</returns>
|
||||
internal static string ResolveMuxer()
|
||||
{
|
||||
var hostPath = Environment.GetEnvironmentVariable("DOTNET_HOST_PATH");
|
||||
if (!string.IsNullOrEmpty(hostPath) && File.Exists(hostPath)) return hostPath;
|
||||
var processPath = Environment.ProcessPath;
|
||||
if (processPath is not null
|
||||
&& string.Equals(Path.GetFileNameWithoutExtension(processPath), "dotnet", StringComparison.OrdinalIgnoreCase))
|
||||
return processPath;
|
||||
var root = Environment.GetEnvironmentVariable("DOTNET_ROOT");
|
||||
if (!string.IsNullOrEmpty(root))
|
||||
{
|
||||
var candidate = Path.Combine(root, OperatingSystem.IsWindows() ? "dotnet.exe" : "dotnet");
|
||||
if (File.Exists(candidate)) return candidate;
|
||||
}
|
||||
return "dotnet";
|
||||
}
|
||||
|
||||
private void Begin()
|
||||
{
|
||||
ReadyTimestamp = Stopwatch.GetTimestamp();
|
||||
Volatile.Write(ref _lastFrameTimestamp, ReadyTimestamp);
|
||||
_ = Task.Run(ReadLoopAsync);
|
||||
_ = Task.Run(WriteLoopAsync);
|
||||
_ = Task.Run(HeartbeatMonitorAsync);
|
||||
_ = Task.Run(WatchProcessAsync);
|
||||
}
|
||||
|
||||
/// <summary>Ensures <paramref name="key"/> is compiled in this worker at <paramref name="sourceHash"/>.</summary>
|
||||
/// <param name="key">The scoped key.</param>
|
||||
/// <param name="kind">The context kind.</param>
|
||||
/// <param name="source">The source.</param>
|
||||
/// <param name="sourceHash">SHA-256 of the source.</param>
|
||||
/// <param name="inputPaths">The slot order.</param>
|
||||
/// <returns>The load outcome and the slot map; null when the worker ended first.</returns>
|
||||
public Task<LoadEntry?> EnsureLoadedAsync(string key, ContextKind kind, string source, byte[] sourceHash, IReadOnlyList<string> inputPaths)
|
||||
{
|
||||
var hashHex = Convert.ToHexString(sourceHash);
|
||||
if (_loaded.TryGetValue(key, out var existing) && existing.SourceHash == hashHex && existing.SameSlots(inputPaths))
|
||||
return existing.Ready;
|
||||
|
||||
var entry = new LoadEntry(hashHex, inputPaths);
|
||||
_loaded[key] = entry;
|
||||
var loadId = (ulong)Interlocked.Increment(ref _nextLoadId);
|
||||
var tcs = new TaskCompletionSource<LoadResult>(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
_loads[loadId] = tcs;
|
||||
var load = new LoadScripts { LoadId = loadId };
|
||||
var script = new ScriptSource { Key = key, Kind = kind, Source = source, SourceSha256 = Google.Protobuf.ByteString.CopyFrom(sourceHash) };
|
||||
script.InputPaths.AddRange(inputPaths);
|
||||
load.Scripts.Add(script);
|
||||
if (!Send(new Envelope { Load = load })) { _loads.TryRemove(loadId, out _); entry.Complete(null); return entry.Ready; }
|
||||
|
||||
_ = tcs.Task.ContinueWith(t =>
|
||||
{
|
||||
if (t.IsCompletedSuccessfully)
|
||||
{
|
||||
var outcome = t.Result.Outcomes.FirstOrDefault(o => o.Key == key);
|
||||
entry.Complete(outcome ?? new LoadOutcome { Key = key, Ok = false, Error = new ScriptError { Kind = ScriptError.Types.Kind.Threw, Message = "no load outcome" } });
|
||||
if (outcome is { Ok: false }) _loaded.TryRemove(new KeyValuePair<string, LoadEntry>(key, entry));
|
||||
}
|
||||
else
|
||||
{
|
||||
entry.Complete(null);
|
||||
_loaded.TryRemove(new KeyValuePair<string, LoadEntry>(key, entry));
|
||||
}
|
||||
}, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
|
||||
return entry.Ready;
|
||||
}
|
||||
|
||||
/// <summary>Forgets a key's compiled form here and in the worker.</summary>
|
||||
/// <param name="keys">The scoped keys.</param>
|
||||
public void Unload(IReadOnlyCollection<string> keys)
|
||||
{
|
||||
var unload = new Unload();
|
||||
foreach (var key in keys)
|
||||
if (_loaded.TryRemove(key, out _)) unload.Keys.Add(key);
|
||||
if (unload.Keys.Count > 0) Send(new Envelope { Unload = unload });
|
||||
}
|
||||
|
||||
/// <summary>Submits one evaluation. Its <see cref="PendingEval.Completion"/> yields null when the worker ends first.</summary>
|
||||
/// <param name="key">The scoped key.</param>
|
||||
/// <param name="request">The request (its id is assigned here).</param>
|
||||
/// <returns>The pending evaluation.</returns>
|
||||
public PendingEval Submit(string key, EvalRequest request)
|
||||
{
|
||||
request.EvalId = (ulong)Interlocked.Increment(ref _nextEvalId);
|
||||
var pending = new PendingEval(key, request.EvalId);
|
||||
_pending[request.EvalId] = pending;
|
||||
var batch = new EvalBatch();
|
||||
batch.Items.Add(request);
|
||||
if (!Send(new Envelope { Eval = batch }) || _exited.Task.IsCompleted)
|
||||
{
|
||||
if (_pending.TryRemove(request.EvalId, out _)) pending.Complete(null);
|
||||
}
|
||||
return pending;
|
||||
}
|
||||
|
||||
/// <summary>Cancels a run's <c>ctx.CancellationToken</c> in the worker (sent at the soft deadline).</summary>
|
||||
/// <param name="pending">The evaluation.</param>
|
||||
public void Cancel(PendingEval pending)
|
||||
{
|
||||
if (!pending.Completion.IsCompleted) Send(new Envelope { Cancel = new Protocol.V1.Cancel { EvalId = pending.EvalId } });
|
||||
}
|
||||
|
||||
/// <summary>Asks the worker for its process diagnostics (loaded assemblies, environment names).</summary>
|
||||
/// <param name="timeout">How long to wait.</param>
|
||||
/// <returns>The reply.</returns>
|
||||
public async Task<DiagnosticsReply> DiagnosticsAsync(TimeSpan timeout)
|
||||
{
|
||||
var tcs = new TaskCompletionSource<DiagnosticsReply>(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
Interlocked.Exchange(ref _diag, tcs);
|
||||
Send(new Envelope { DiagRequest = new DiagnosticsRequest() });
|
||||
return await tcs.Task.WaitAsync(timeout).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>Kills the worker (process tree). The first reason given is the one recorded.</summary>
|
||||
/// <param name="reason">Why.</param>
|
||||
/// <param name="culpritKey">The scoped key blamed, if any.</param>
|
||||
public void Kill(WorkerExitReason reason, string? culpritKey)
|
||||
{
|
||||
lock (_pending)
|
||||
{
|
||||
_killedFor ??= (reason, culpritKey);
|
||||
}
|
||||
try
|
||||
{
|
||||
if (!_process.HasExited) _process.Kill(entireProcessTree: true);
|
||||
}
|
||||
catch (InvalidOperationException) { /* already gone */ }
|
||||
catch (System.ComponentModel.Win32Exception ex)
|
||||
{
|
||||
_log.Warning(ex, "ScriptWorker[{Shard}] kill of pid {Pid} failed", Shard, Pid);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Asks the worker to stop, waits briefly, then kills it.</summary>
|
||||
/// <returns>A task that completes when the worker has ended.</returns>
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
// Both the shard (on exit) and the runner (on shutdown) dispose a connection; only the first does the work.
|
||||
if (Interlocked.Exchange(ref _disposed, 1) != 0) return;
|
||||
if (!_exited.Task.IsCompleted)
|
||||
{
|
||||
lock (_pending) { _killedFor ??= (WorkerExitReason.Disposed, null); }
|
||||
Send(new Envelope { Shutdown = new Shutdown() });
|
||||
try
|
||||
{
|
||||
await _process.WaitForExitAsync().WaitAsync(TimeSpan.FromSeconds(1)).ConfigureAwait(false);
|
||||
}
|
||||
catch (TimeoutException)
|
||||
{
|
||||
Kill(WorkerExitReason.Disposed, null);
|
||||
}
|
||||
try { await _exited.Task.WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false); } catch (TimeoutException) { }
|
||||
}
|
||||
_stopping.Cancel();
|
||||
_outbound.Writer.TryComplete();
|
||||
await _pipe.DisposeAsync().ConfigureAwait(false);
|
||||
_job?.Dispose();
|
||||
_process.Dispose();
|
||||
TryDeleteDirectory(_workDir);
|
||||
}
|
||||
|
||||
/// <summary>Test hook: closes the host end of the pipe without touching the process, as a dying host would.</summary>
|
||||
internal void SeverPipeForTest() => _pipe.Dispose();
|
||||
|
||||
private bool Send(Envelope envelope)
|
||||
{
|
||||
envelope.ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion;
|
||||
return _outbound.Writer.TryWrite(envelope);
|
||||
}
|
||||
|
||||
private async Task WriteLoopAsync()
|
||||
{
|
||||
var reader = _outbound.Reader;
|
||||
try
|
||||
{
|
||||
while (await reader.WaitToReadAsync(_stopping.Token).ConfigureAwait(false))
|
||||
{
|
||||
// Opportunistic batching (design §4.5): every evaluation queued right now goes out as ONE EvalBatch.
|
||||
EvalBatch? batch = null;
|
||||
while (reader.TryRead(out var envelope))
|
||||
{
|
||||
if (envelope.BodyCase == Envelope.BodyOneofCase.Eval)
|
||||
{
|
||||
(batch ??= new EvalBatch()).Items.AddRange(envelope.Eval.Items);
|
||||
continue;
|
||||
}
|
||||
envelope.Seq = (ulong)Interlocked.Increment(ref _seq);
|
||||
await ScriptWorkerFrameCodec.WriteAsync(_pipe, envelope, _maxFrameBytes, _stopping.Token).ConfigureAwait(false);
|
||||
}
|
||||
if (batch is not null)
|
||||
{
|
||||
await ScriptWorkerFrameCodec.WriteAsync(_pipe, new Envelope
|
||||
{
|
||||
ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion,
|
||||
Seq = (ulong)Interlocked.Increment(ref _seq),
|
||||
Eval = batch,
|
||||
}, _maxFrameBytes, _stopping.Token).ConfigureAwait(false);
|
||||
}
|
||||
await _pipe.FlushAsync(_stopping.Token).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) { }
|
||||
catch (Exception ex) when (ex is IOException or ObjectDisposedException or ScriptWorkerProtocolException)
|
||||
{
|
||||
if (!_stopping.IsCancellationRequested)
|
||||
_log.Debug(ex, "ScriptWorker[{Shard}] pipe write ended", Shard);
|
||||
Kill(WorkerExitReason.Crash, null);
|
||||
}
|
||||
}
|
||||
|
||||
private async Task ReadLoopAsync()
|
||||
{
|
||||
WorkerExitReason? fault = null;
|
||||
try
|
||||
{
|
||||
while (true)
|
||||
{
|
||||
var frame = await ScriptWorkerFrameCodec.ReadAsync(_pipe, _maxFrameBytes, _stopping.Token).ConfigureAwait(false);
|
||||
if (frame is null) break;
|
||||
Volatile.Write(ref _lastFrameTimestamp, Stopwatch.GetTimestamp());
|
||||
switch (frame.BodyCase)
|
||||
{
|
||||
case Envelope.BodyOneofCase.Results:
|
||||
foreach (var result in frame.Results.Items)
|
||||
{
|
||||
// An eval id with no outstanding request is a protocol violation (design §6 "what the host
|
||||
// trusts"), except for one that was already completed by a kill race — ignored then.
|
||||
if (_pending.TryRemove(result.EvalId, out var pending)) pending.Complete(result);
|
||||
else if (result.EvalId == 0 || result.EvalId > (ulong)Interlocked.Read(ref _nextEvalId))
|
||||
throw new ScriptWorkerProtocolException($"result for unknown eval id {result.EvalId}");
|
||||
}
|
||||
break;
|
||||
case Envelope.BodyOneofCase.LoadResult:
|
||||
if (_loads.TryRemove(frame.LoadResult.LoadId, out var load)) load.TrySetResult(frame.LoadResult);
|
||||
else throw new ScriptWorkerProtocolException($"load result for unknown load id {frame.LoadResult.LoadId}");
|
||||
break;
|
||||
case Envelope.BodyOneofCase.Logs:
|
||||
try { _onLogs(frame.Logs); }
|
||||
catch (Exception ex) { _log.Warning(ex, "ScriptWorker[{Shard}] log forwarding threw", Shard); }
|
||||
break;
|
||||
case Envelope.BodyOneofCase.Heartbeat:
|
||||
break;
|
||||
case Envelope.BodyOneofCase.DiagReply:
|
||||
Volatile.Read(ref _diag)?.TrySetResult(frame.DiagReply);
|
||||
break;
|
||||
case Envelope.BodyOneofCase.Fault:
|
||||
_log.Warning("ScriptWorker[{Shard}] reported a fault: {Fault}", Shard, frame.Fault.Message);
|
||||
break;
|
||||
default:
|
||||
throw new ScriptWorkerProtocolException($"unexpected frame {frame.BodyCase} from the worker");
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) { }
|
||||
catch (ScriptWorkerProtocolException ex)
|
||||
{
|
||||
_log.Warning("ScriptWorker[{Shard}] protocol violation: {Message}; killing pid {Pid}", Shard, ex.Message, Pid);
|
||||
fault = WorkerExitReason.ProtocolFault;
|
||||
}
|
||||
catch (Exception ex) when (ex is IOException or ObjectDisposedException)
|
||||
{
|
||||
}
|
||||
|
||||
// The pipe is gone: the worker is dead or dying. Make sure, then report.
|
||||
try
|
||||
{
|
||||
if (fault is { } f) Kill(f, null);
|
||||
try
|
||||
{
|
||||
await _process.WaitForExitAsync().WaitAsync(TimeSpan.FromSeconds(2)).ConfigureAwait(false);
|
||||
}
|
||||
catch (TimeoutException)
|
||||
{
|
||||
Kill(WorkerExitReason.Crash, null);
|
||||
try { await _process.WaitForExitAsync().WaitAsync(TimeSpan.FromSeconds(5)).ConfigureAwait(false); } catch (TimeoutException) { }
|
||||
}
|
||||
}
|
||||
catch (InvalidOperationException)
|
||||
{
|
||||
// The process object was disposed by a concurrent shutdown: the worker is already gone.
|
||||
}
|
||||
Finish();
|
||||
}
|
||||
|
||||
private async Task WatchProcessAsync()
|
||||
{
|
||||
try { await _process.WaitForExitAsync(_stopping.Token).ConfigureAwait(false); }
|
||||
catch (OperationCanceledException) { return; }
|
||||
catch (InvalidOperationException) { return; }
|
||||
// The reader normally finishes first on EOF; this covers a worker that died while the pipe stayed open.
|
||||
await Task.Delay(TimeSpan.FromSeconds(2)).ConfigureAwait(false);
|
||||
Finish();
|
||||
}
|
||||
|
||||
private async Task HeartbeatMonitorAsync()
|
||||
{
|
||||
using var timer = new PeriodicTimer(_heartbeatInterval);
|
||||
try
|
||||
{
|
||||
while (await timer.WaitForNextTickAsync(_stopping.Token).ConfigureAwait(false))
|
||||
{
|
||||
if (_exited.Task.IsCompleted) return;
|
||||
var silent = Stopwatch.GetElapsedTime(Volatile.Read(ref _lastFrameTimestamp));
|
||||
if (silent > _heartbeatInterval * 3)
|
||||
{
|
||||
_log.Warning("ScriptWorker[{Shard}] pid {Pid} sent nothing for {Silent} ms; killing it", Shard, Pid, (long)silent.TotalMilliseconds);
|
||||
Kill(WorkerExitReason.KilledHeartbeat, null);
|
||||
return;
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (OperationCanceledException) { }
|
||||
}
|
||||
|
||||
private void Finish()
|
||||
{
|
||||
int? code = null;
|
||||
try { if (_process.HasExited) code = _process.ExitCode; } catch (InvalidOperationException) { }
|
||||
|
||||
(WorkerExitReason Reason, string? Culprit)? killed;
|
||||
lock (_pending) { killed = _killedFor; }
|
||||
var reason = killed?.Reason ?? (code == 0 ? WorkerExitReason.Clean : WorkerExitReason.Crash);
|
||||
var inFlight = _pending.Values.Select(p => p.Key).Distinct(StringComparer.Ordinal).ToArray();
|
||||
if (!_exited.TrySetResult(new WorkerExit(reason, killed?.Culprit, code, inFlight))) return;
|
||||
|
||||
// Every outstanding evaluation and load ends now; null tells the waiter "the worker went away".
|
||||
foreach (var id in _pending.Keys.ToArray())
|
||||
if (_pending.TryRemove(id, out var pending)) pending.Complete(null);
|
||||
foreach (var id in _loads.Keys.ToArray())
|
||||
if (_loads.TryRemove(id, out var load)) load.TrySetCanceled();
|
||||
Volatile.Read(ref _diag)?.TrySetCanceled();
|
||||
_outbound.Writer.TryComplete();
|
||||
}
|
||||
|
||||
private static string SafeExitCode(Process process)
|
||||
{
|
||||
try { return process.HasExited ? process.ExitCode.ToString(System.Globalization.CultureInfo.InvariantCulture) : "?"; }
|
||||
catch (InvalidOperationException) { return "?"; }
|
||||
}
|
||||
|
||||
private static void TryDeleteDirectory(string dir)
|
||||
{
|
||||
try { if (Directory.Exists(dir)) Directory.Delete(dir, recursive: true); } catch { /* temp dir; best effort */ }
|
||||
}
|
||||
|
||||
/// <summary>A key's compiled form in this worker.</summary>
|
||||
internal sealed class LoadEntry
|
||||
{
|
||||
private readonly TaskCompletionSource<LoadEntry?> _ready = new(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
|
||||
public LoadEntry(string sourceHash, IReadOnlyList<string> inputPaths)
|
||||
{
|
||||
SourceHash = sourceHash;
|
||||
InputPaths = inputPaths;
|
||||
var slots = new Dictionary<string, uint>(inputPaths.Count, StringComparer.Ordinal);
|
||||
for (var i = 0; i < inputPaths.Count; i++) slots[inputPaths[i]] = (uint)i;
|
||||
Slots = slots;
|
||||
}
|
||||
|
||||
public string SourceHash { get; }
|
||||
|
||||
public IReadOnlyList<string> InputPaths { get; }
|
||||
|
||||
/// <summary>Path → slot in the registered order.</summary>
|
||||
public IReadOnlyDictionary<string, uint> Slots { get; }
|
||||
|
||||
/// <summary>The worker's verdict (null until it arrives).</summary>
|
||||
public LoadOutcome? Outcome { get; private set; }
|
||||
|
||||
public Task<LoadEntry?> Ready => _ready.Task;
|
||||
|
||||
public bool SameSlots(IReadOnlyList<string> inputPaths)
|
||||
{
|
||||
if (inputPaths.Count != InputPaths.Count) return false;
|
||||
for (var i = 0; i < inputPaths.Count; i++)
|
||||
if (!string.Equals(inputPaths[i], InputPaths[i], StringComparison.Ordinal)) return false;
|
||||
return true;
|
||||
}
|
||||
|
||||
public void Complete(LoadOutcome? outcome)
|
||||
{
|
||||
Outcome = outcome;
|
||||
_ready.TrySetResult(outcome is null ? null : this);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>One evaluation outstanding on a worker.</summary>
|
||||
internal sealed class PendingEval
|
||||
{
|
||||
private readonly TaskCompletionSource<EvalResult?> _completion = new(TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
|
||||
public PendingEval(string key, ulong evalId)
|
||||
{
|
||||
Key = key;
|
||||
EvalId = evalId;
|
||||
}
|
||||
|
||||
/// <summary>The scoped key.</summary>
|
||||
public string Key { get; }
|
||||
|
||||
/// <summary>The eval id on the wire.</summary>
|
||||
public ulong EvalId { get; }
|
||||
|
||||
/// <summary>The worker's result, or null when the worker ended first.</summary>
|
||||
public Task<EvalResult?> Completion => _completion.Task;
|
||||
|
||||
public void Complete(EvalResult? result) => _completion.TrySetResult(result);
|
||||
}
|
||||
@@ -0,0 +1,249 @@
|
||||
using System.Diagnostics;
|
||||
using System.Diagnostics.Metrics;
|
||||
using Serilog;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Observability;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
|
||||
|
||||
/// <summary>
|
||||
/// Supervises one shard's worker process (design §3.3, §5.3.5): starts it lazily on first use, reports its end to
|
||||
/// the runner (for attribution), and restarts it after an exponential backoff — 250 ms doubling to a 30 s cap with
|
||||
/// ±20 % jitter, reset once a worker has stayed up for 60 s. It never runs a script in the host when the worker
|
||||
/// cannot start (fail closed, §14 #4): <see cref="GetReadyAsync"/> returns null and the caller reports
|
||||
/// <c>BadResourceUnavailable</c>.
|
||||
/// </summary>
|
||||
internal sealed class WorkerShard : IAsyncDisposable
|
||||
{
|
||||
internal static readonly TimeSpan InitialBackoff = TimeSpan.FromMilliseconds(250);
|
||||
internal static readonly TimeSpan MaxBackoff = TimeSpan.FromSeconds(30);
|
||||
internal static readonly TimeSpan HealthyResetAfter = TimeSpan.FromSeconds(60);
|
||||
|
||||
private static readonly Counter<long> StartsCounter = OtOpcUaTelemetry.Meter.CreateCounter<long>(
|
||||
"otopcua.script.worker.starts", unit: "{process}", description: "ScriptWorker processes started (tag: shard).");
|
||||
|
||||
private static readonly Counter<long> ExitsCounter = OtOpcUaTelemetry.Meter.CreateCounter<long>(
|
||||
"otopcua.script.worker.exits", unit: "{process}",
|
||||
description: "ScriptWorker processes ended (tags: shard, reason = clean | killed_deadline | killed_heartbeat | killed_load | crash | protocol | disposed).");
|
||||
|
||||
private readonly Func<CancellationToken, Task<WorkerConnection>> _launch;
|
||||
private readonly Action<WorkerShard, WorkerExit> _onExit;
|
||||
private readonly ILogger _log;
|
||||
private readonly object _gate = new();
|
||||
private readonly Random _jitter = new();
|
||||
private WorkerConnection? _current;
|
||||
private Task<WorkerConnection?>? _starting;
|
||||
private long _backoffUntil;
|
||||
private int _consecutiveFailures;
|
||||
private bool _disposed;
|
||||
|
||||
public WorkerShard(string label, bool isolation, Func<CancellationToken, Task<WorkerConnection>> launch,
|
||||
Action<WorkerShard, WorkerExit> onExit, ILogger log)
|
||||
{
|
||||
Label = label;
|
||||
IsIsolation = isolation;
|
||||
_launch = launch;
|
||||
_onExit = onExit;
|
||||
_log = log;
|
||||
// The isolation worker runs one evaluation at a time, so a crash there names exactly one key (design §5.3.3).
|
||||
IsolationGate = isolation ? new SemaphoreSlim(1, 1) : null;
|
||||
}
|
||||
|
||||
/// <summary>The shard label (<c>0</c>…<c>K-1</c>, or <c>iso</c>).</summary>
|
||||
public string Label { get; }
|
||||
|
||||
/// <summary>Whether this is the isolation ("penalty box") worker.</summary>
|
||||
public bool IsIsolation { get; }
|
||||
|
||||
/// <summary>Serialises evaluations on the isolation worker; null on a normal shard.</summary>
|
||||
public SemaphoreSlim? IsolationGate { get; }
|
||||
|
||||
/// <summary>The live worker, if any (diagnostics / tests).</summary>
|
||||
public WorkerConnection? Current
|
||||
{
|
||||
get { lock (_gate) return _current is { IsAlive: true } c ? c : null; }
|
||||
}
|
||||
|
||||
/// <summary>Consecutive start failures and crashes since the last healthy period (diagnostics / tests).</summary>
|
||||
public int ConsecutiveFailures
|
||||
{
|
||||
get { lock (_gate) return _consecutiveFailures; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The ready worker, starting one if needed. Waits at most <paramref name="grace"/> — through a start in progress
|
||||
/// or a backoff — and returns null if no worker is ready by then.
|
||||
/// </summary>
|
||||
/// <param name="grace">How long to wait.</param>
|
||||
/// <param name="ct">Cancellation.</param>
|
||||
/// <returns>The worker, or null.</returns>
|
||||
public async Task<WorkerConnection?> GetReadyAsync(TimeSpan grace, CancellationToken ct)
|
||||
{
|
||||
var deadline = Stopwatch.GetTimestamp() + (long)(grace.TotalSeconds * Stopwatch.Frequency);
|
||||
while (true)
|
||||
{
|
||||
Task<WorkerConnection?> starting;
|
||||
TimeSpan wait;
|
||||
lock (_gate)
|
||||
{
|
||||
if (_disposed) return null;
|
||||
if (_current is { IsAlive: true } live) return live;
|
||||
var now = Stopwatch.GetTimestamp();
|
||||
if (_starting is null && now >= _backoffUntil)
|
||||
{
|
||||
_starting = StartCoreAsync();
|
||||
}
|
||||
if (_starting is not null)
|
||||
{
|
||||
starting = _starting;
|
||||
wait = TimeSpan.Zero;
|
||||
}
|
||||
else
|
||||
{
|
||||
starting = null!;
|
||||
wait = Stopwatch.GetElapsedTime(now, _backoffUntil);
|
||||
}
|
||||
}
|
||||
|
||||
var remaining = Stopwatch.GetElapsedTime(Stopwatch.GetTimestamp(), deadline);
|
||||
if (remaining <= TimeSpan.Zero) return null;
|
||||
if (starting is not null)
|
||||
{
|
||||
try
|
||||
{
|
||||
var started = await starting.WaitAsync(remaining, ct).ConfigureAwait(false);
|
||||
if (started is { IsAlive: true }) return started;
|
||||
}
|
||||
catch (TimeoutException)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if (wait > remaining) return null;
|
||||
await Task.Delay(wait, ct).ConfigureAwait(false);
|
||||
}
|
||||
}
|
||||
|
||||
private async Task<WorkerConnection?> StartCoreAsync()
|
||||
{
|
||||
await Task.Yield();
|
||||
WorkerConnection? connection = null;
|
||||
try
|
||||
{
|
||||
connection = await _launch(CancellationToken.None).ConfigureAwait(false);
|
||||
StartsCounter.Add(1, new KeyValuePair<string, object?>("shard", Label));
|
||||
_log.Information("ScriptWorker[{Shard}] started, pid {Pid}", Label, connection.Pid);
|
||||
var started = connection;
|
||||
_ = started.Exited.ContinueWith(t => OnExited(started, t.Result), CancellationToken.None,
|
||||
TaskContinuationOptions.ExecuteSynchronously | TaskContinuationOptions.OnlyOnRanToCompletion, TaskScheduler.Default);
|
||||
lock (_gate)
|
||||
{
|
||||
_starting = null;
|
||||
if (_disposed)
|
||||
{
|
||||
_ = started.DisposeAsync().AsTask();
|
||||
return null;
|
||||
}
|
||||
_current = started;
|
||||
}
|
||||
return started;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.Error(ex, "ScriptWorker[{Shard}] failed to start; scripts on it report runner unavailable until it does", Label);
|
||||
lock (_gate)
|
||||
{
|
||||
_starting = null;
|
||||
ScheduleBackoffLocked();
|
||||
}
|
||||
if (connection is not null) await connection.DisposeAsync().ConfigureAwait(false);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
private void OnExited(WorkerConnection connection, WorkerExit exit)
|
||||
{
|
||||
ExitsCounter.Add(1, new KeyValuePair<string, object?>("shard", Label),
|
||||
new KeyValuePair<string, object?>("reason", MetricReason(exit.Reason)));
|
||||
lock (_gate)
|
||||
{
|
||||
if (ReferenceEquals(_current, connection)) _current = null;
|
||||
if (!_disposed)
|
||||
{
|
||||
if (Stopwatch.GetElapsedTime(connection.ReadyTimestamp) >= HealthyResetAfter) _consecutiveFailures = 0;
|
||||
ScheduleBackoffLocked();
|
||||
}
|
||||
}
|
||||
if (exit.Reason != WorkerExitReason.Disposed)
|
||||
{
|
||||
_log.Warning("ScriptWorker[{Shard}] pid {Pid} ended: {Reason} (exit code {ExitCode}, culprit {Culprit}, {InFlight} evaluation(s) in flight)",
|
||||
Label, connection.Pid, exit.Reason, exit.ExitCode, exit.CulpritKey ?? "none", exit.InFlightKeys.Count);
|
||||
}
|
||||
try { _onExit(this, exit); }
|
||||
catch (Exception ex) { _log.Error(ex, "ScriptWorker[{Shard}] exit handler threw", Label); }
|
||||
_ = connection.DisposeAsync().AsTask();
|
||||
}
|
||||
|
||||
private void ScheduleBackoffLocked()
|
||||
{
|
||||
_consecutiveFailures++;
|
||||
var delay = NextBackoff(_consecutiveFailures, _jitter.NextDouble());
|
||||
_backoffUntil = Stopwatch.GetTimestamp() + (long)(delay.TotalSeconds * Stopwatch.Frequency);
|
||||
}
|
||||
|
||||
/// <summary>The restart delay after <paramref name="failures"/> consecutive failures (design §5.3.5).</summary>
|
||||
/// <param name="failures">1 for the first failure.</param>
|
||||
/// <param name="unitJitter">A sample in [0, 1) mapped to ±20 %.</param>
|
||||
/// <returns>The delay.</returns>
|
||||
internal static TimeSpan NextBackoff(int failures, double unitJitter)
|
||||
{
|
||||
var exponent = Math.Clamp(failures - 1, 0, 30);
|
||||
var baseMs = Math.Min(MaxBackoff.TotalMilliseconds, InitialBackoff.TotalMilliseconds * Math.Pow(2, exponent));
|
||||
var factor = 0.8 + (0.4 * Math.Clamp(unitJitter, 0, 1));
|
||||
return TimeSpan.FromMilliseconds(Math.Min(MaxBackoff.TotalMilliseconds * 1.2, baseMs * factor));
|
||||
}
|
||||
|
||||
private static string MetricReason(WorkerExitReason reason) => reason switch
|
||||
{
|
||||
WorkerExitReason.Clean => "clean",
|
||||
WorkerExitReason.KilledDeadline => "killed_deadline",
|
||||
WorkerExitReason.KilledHeartbeat => "killed_heartbeat",
|
||||
WorkerExitReason.KilledLoadTimeout => "killed_load",
|
||||
WorkerExitReason.ProtocolFault => "protocol",
|
||||
WorkerExitReason.Disposed => "disposed",
|
||||
_ => "crash",
|
||||
};
|
||||
|
||||
/// <summary>Stops the shard's worker.</summary>
|
||||
/// <returns>A task that completes when the worker has ended.</returns>
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
WorkerConnection? current;
|
||||
Task<WorkerConnection?>? starting;
|
||||
lock (_gate)
|
||||
{
|
||||
_disposed = true;
|
||||
current = _current;
|
||||
starting = _starting;
|
||||
_current = null;
|
||||
}
|
||||
if (starting is not null)
|
||||
{
|
||||
try { current ??= await starting.WaitAsync(TimeSpan.FromSeconds(15)).ConfigureAwait(false); } catch { /* shutting down */ }
|
||||
}
|
||||
if (current is not null) await current.DisposeAsync().ConfigureAwait(false);
|
||||
IsolationGate?.Dispose();
|
||||
}
|
||||
|
||||
/// <summary>Maps a context kind to the wire enum.</summary>
|
||||
/// <param name="kind">The kind.</param>
|
||||
/// <returns>The wire kind.</returns>
|
||||
internal static ContextKind ToWire(ScriptContextKind kind) => kind switch
|
||||
{
|
||||
ScriptContextKind.VirtualTag => ContextKind.VirtualTag,
|
||||
ScriptContextKind.Calculation => ContextKind.Calculation,
|
||||
ScriptContextKind.AlarmPredicate => ContextKind.AlarmPredicate,
|
||||
_ => ContextKind.Unspecified,
|
||||
};
|
||||
}
|
||||
+20
@@ -0,0 +1,20 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<!-- lmxopcua#780 — the host side of the out-of-process, pre-emptible script runner (design §3, §5, §9): worker
|
||||
launch + pipe handshake, the per-shard supervisor, attribution and quarantine. Selected by
|
||||
Scripting:Runner=OutOfProcess; nothing here runs under the InProcess default. -->
|
||||
<PropertyGroup>
|
||||
<RootNamespace>ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess</RootNamespace>
|
||||
<GenerateDocumentationFile>true</GenerateDocumentationFile>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.Core.Scripting\ZB.MOM.WW.OtOpcUa.Core.Scripting.csproj"/>
|
||||
<ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol\ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.csproj"/>
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests"/>
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
@@ -0,0 +1,214 @@
|
||||
// lmxopcua#780 — host <-> ScriptWorker IPC (design docs/plans/2026-11-09-out-of-process-script-runner-design.md §4).
|
||||
// Framing: 4-byte little-endian length prefix + one serialized Envelope. Zero-length and oversize frames are protocol
|
||||
// faults. One request frame and one reply frame per evaluation; the only other worker->host traffic is logs and
|
||||
// heartbeats. Bump protocol_version on any incompatible change.
|
||||
syntax = "proto3";
|
||||
|
||||
package otopcua.scriptworker.v1;
|
||||
|
||||
option csharp_namespace = "ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1";
|
||||
|
||||
message Envelope {
|
||||
uint32 protocol_version = 1;
|
||||
uint64 seq = 2;
|
||||
oneof body {
|
||||
Hello hello = 10; // host -> worker, first frame
|
||||
Ready ready = 11; // worker -> host, reply to Hello
|
||||
LoadScripts load = 12; // host -> worker
|
||||
LoadResult load_result = 13; // worker -> host
|
||||
Unload unload = 14; // host -> worker
|
||||
EvalBatch eval = 15; // host -> worker
|
||||
EvalResults results = 16; // worker -> host, streamed as runs complete
|
||||
LogBatch logs = 17; // worker -> host, fire-and-forget, bounded
|
||||
Heartbeat heartbeat = 18; // worker -> host, every heartbeat_interval_ms
|
||||
Shutdown shutdown = 19; // host -> worker
|
||||
Fault fault = 20; // either way; the sender closes after it
|
||||
DiagnosticsRequest diag_request = 21; // host -> worker (tests / ops)
|
||||
DiagnosticsReply diag_reply = 22; // worker -> host
|
||||
Cancel cancel = 23; // host -> worker, at the soft deadline
|
||||
}
|
||||
}
|
||||
|
||||
message Hello {
|
||||
string nonce = 1; // must be echoed in Ready
|
||||
uint32 max_frame_bytes = 2;
|
||||
uint32 eval_threads = 3; // the worker's scheduler width; 1 for the isolation worker
|
||||
uint32 queue_capacity = 4; // queued evaluations above this answer BUSY
|
||||
uint32 hard_grace_ms = 5; // worker watchdog: fail fast past soft + grace (+ margin)
|
||||
uint32 heartbeat_interval_ms = 6;
|
||||
uint32 max_log_lines = 7; // outbound log buffer cap
|
||||
}
|
||||
|
||||
message Ready {
|
||||
string nonce = 1;
|
||||
int32 pid = 2;
|
||||
string runtime_version = 3;
|
||||
}
|
||||
|
||||
enum ContextKind {
|
||||
CONTEXT_KIND_UNSPECIFIED = 0;
|
||||
VIRTUAL_TAG = 1;
|
||||
CALCULATION = 2;
|
||||
ALARM_PREDICATE = 3;
|
||||
}
|
||||
|
||||
message ScriptSource {
|
||||
string key = 1;
|
||||
bytes source_sha256 = 2;
|
||||
ContextKind kind = 3;
|
||||
string source = 4;
|
||||
repeated string input_paths = 5; // slot order for EvalRequest.inputs
|
||||
}
|
||||
|
||||
message LoadScripts {
|
||||
uint64 load_id = 1;
|
||||
repeated ScriptSource scripts = 2;
|
||||
}
|
||||
|
||||
message LoadOutcome {
|
||||
string key = 1;
|
||||
bool ok = 2;
|
||||
ScriptError error = 3; // COMPILE or SANDBOX when !ok
|
||||
}
|
||||
|
||||
message LoadResult {
|
||||
uint64 load_id = 1;
|
||||
repeated LoadOutcome outcomes = 2;
|
||||
}
|
||||
|
||||
message Unload {
|
||||
repeated string keys = 1;
|
||||
}
|
||||
|
||||
message EvalBatch {
|
||||
repeated EvalRequest items = 1;
|
||||
}
|
||||
|
||||
message EvalRequest {
|
||||
uint64 eval_id = 1;
|
||||
string key = 2;
|
||||
repeated Input inputs = 3;
|
||||
int64 now_utc_ticks = 4; // ctx.Now (host clock)
|
||||
uint32 soft_timeout_ms = 5;
|
||||
uint32 hard_grace_ms = 6; // this run's G; 0 = Hello.hard_grace_ms
|
||||
}
|
||||
|
||||
message Input {
|
||||
uint32 slot = 1; // index into ScriptSource.input_paths; ignored when path is set
|
||||
Variant value = 2;
|
||||
uint32 status_code = 3;
|
||||
int64 source_ts_ticks = 4; // 0 = null
|
||||
int64 server_ts_ticks = 5;
|
||||
string path = 6; // inline path for an input with no registered slot
|
||||
bool unsupported = 7; // the host could not encode the value's CLR type (see unsupported_type)
|
||||
string unsupported_type = 8;
|
||||
}
|
||||
|
||||
message EvalResult {
|
||||
uint64 eval_id = 1;
|
||||
oneof outcome {
|
||||
Variant value = 2;
|
||||
bool predicate = 3;
|
||||
ScriptError error = 4;
|
||||
}
|
||||
uint64 allocated_bytes = 5; // GC.GetAllocatedBytesForCurrentThread around the run
|
||||
uint32 run_micros = 6;
|
||||
}
|
||||
|
||||
message EvalResults {
|
||||
repeated EvalResult items = 1;
|
||||
}
|
||||
|
||||
message ScriptError {
|
||||
enum Kind {
|
||||
THREW = 0;
|
||||
COMPILE = 1;
|
||||
SANDBOX = 2;
|
||||
SOFT_TIMEOUT = 3;
|
||||
UNSUPPORTED_TYPE = 4;
|
||||
BUSY = 5;
|
||||
NOT_LOADED = 6;
|
||||
}
|
||||
Kind kind = 1;
|
||||
string message = 2;
|
||||
}
|
||||
|
||||
message Variant {
|
||||
enum TypeTag {
|
||||
NULL = 0;
|
||||
BOOLEAN = 1;
|
||||
SBYTE = 2;
|
||||
BYTE = 3;
|
||||
INT16 = 4;
|
||||
UINT16 = 5;
|
||||
INT32 = 6;
|
||||
UINT32 = 7;
|
||||
INT64 = 8;
|
||||
UINT64 = 9;
|
||||
FLOAT = 10;
|
||||
DOUBLE = 11;
|
||||
STRING = 12;
|
||||
DATETIME_UTC = 13;
|
||||
ARRAY = 14;
|
||||
}
|
||||
TypeTag type = 1; // the exact CLR type: (short)ctx.GetTag(x).Value must keep working
|
||||
oneof v {
|
||||
bool b = 2;
|
||||
sint64 i = 3;
|
||||
uint64 u = 4;
|
||||
double d = 5;
|
||||
float f = 6;
|
||||
string s = 7;
|
||||
int64 dt_ticks = 8;
|
||||
ArrayValue array = 9;
|
||||
}
|
||||
}
|
||||
|
||||
message ArrayValue {
|
||||
Variant.TypeTag element_type = 1;
|
||||
repeated Variant items = 2;
|
||||
}
|
||||
|
||||
message LogBatch {
|
||||
repeated LogLine lines = 1;
|
||||
uint32 dropped = 2;
|
||||
}
|
||||
|
||||
message LogLine {
|
||||
string key = 1;
|
||||
uint32 level = 2; // Serilog LogEventLevel
|
||||
string rendered = 3;
|
||||
string exception = 4;
|
||||
int64 ts_ticks = 5;
|
||||
}
|
||||
|
||||
message Heartbeat {
|
||||
uint32 inflight = 1;
|
||||
uint32 queued = 2;
|
||||
int64 heap_bytes = 3;
|
||||
uint32 gen2_count = 4;
|
||||
}
|
||||
|
||||
message Shutdown {
|
||||
}
|
||||
|
||||
// Sent by the host when it has answered the caller with a soft timeout: cancels the run's ctx.CancellationToken, so a
|
||||
// polling script ends itself. The worker never cancels on its own clock, which keeps the in-process ordering — the
|
||||
// caller always has its timeout before the script can observe cancellation.
|
||||
message Cancel {
|
||||
uint64 eval_id = 1;
|
||||
}
|
||||
|
||||
message Fault {
|
||||
string message = 1;
|
||||
}
|
||||
|
||||
message DiagnosticsRequest {
|
||||
}
|
||||
|
||||
message DiagnosticsReply {
|
||||
int32 pid = 1;
|
||||
repeated string loaded_assemblies = 2;
|
||||
repeated string environment_names = 3;
|
||||
string working_directory = 4;
|
||||
}
|
||||
@@ -0,0 +1,54 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
|
||||
|
||||
/// <summary>
|
||||
/// The environment variables the host sets on a ScriptWorker (design §6). The nonce travels in the environment,
|
||||
/// never on the command line (mxaccessgw <c>WorkerBootstrap.md</c>): a command line is world-readable on most
|
||||
/// systems.
|
||||
/// </summary>
|
||||
public static class ScriptWorkerEnvironment
|
||||
{
|
||||
/// <summary>The pipe name the worker connects to.</summary>
|
||||
public const string PipeName = "OTOPCUA_SCRIPTWORKER_PIPE";
|
||||
|
||||
/// <summary>The one-time launch nonce the worker must echo in <c>Ready</c>.</summary>
|
||||
public const string Nonce = "OTOPCUA_SCRIPTWORKER_NONCE";
|
||||
|
||||
/// <summary>The host's process id; the worker exits when it disappears.</summary>
|
||||
public const string ParentPid = "OTOPCUA_SCRIPTWORKER_PARENT_PID";
|
||||
|
||||
/// <summary>The worker's shard label, for its own diagnostics.</summary>
|
||||
public const string Shard = "OTOPCUA_SCRIPTWORKER_SHARD";
|
||||
|
||||
/// <summary>
|
||||
/// Exit code of a worker whose pipe reached end of stream (the host went away). Distinct from a crash so the
|
||||
/// host's exit reason metric can tell them apart.
|
||||
/// </summary>
|
||||
public const int ExitPipeClosed = 3;
|
||||
|
||||
/// <summary>Exit code of a worker whose handshake or framing failed.</summary>
|
||||
public const int ExitProtocolFault = 4;
|
||||
|
||||
/// <summary>Exit code of a worker whose parent process disappeared.</summary>
|
||||
public const int ExitParentGone = 5;
|
||||
|
||||
/// <summary>Exit code of a worker that found a forbidden assembly loaded in itself.</summary>
|
||||
public const int ExitForbiddenAssembly = 6;
|
||||
|
||||
/// <summary>
|
||||
/// Assembly-name prefixes that must never be loaded in a worker (design §6): the host's cluster, database,
|
||||
/// OPC UA and secrets stacks. The worker checks itself against this list and exits if it finds one.
|
||||
/// </summary>
|
||||
public static readonly IReadOnlyList<string> ForbiddenAssemblyPrefixes =
|
||||
[
|
||||
"Akka",
|
||||
"Microsoft.Data.SqlClient",
|
||||
"System.Data.SqlClient",
|
||||
"Microsoft.EntityFrameworkCore",
|
||||
"Opc.Ua",
|
||||
"ZB.MOM.WW.Secrets",
|
||||
"ZB.MOM.WW.OtOpcUa.Runtime",
|
||||
"ZB.MOM.WW.OtOpcUa.Host",
|
||||
"ZB.MOM.WW.OtOpcUa.Driver.",
|
||||
"Grpc.",
|
||||
];
|
||||
}
|
||||
@@ -0,0 +1,116 @@
|
||||
using System.Buffers;
|
||||
using System.Buffers.Binary;
|
||||
using Google.Protobuf;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
|
||||
|
||||
/// <summary>
|
||||
/// Length-prefixed protobuf framing for the ScriptWorker pipe (lmxopcua#780, design §4.2): a 4-byte little-endian
|
||||
/// length, then one serialized <see cref="Envelope"/>. Copies mxaccessgw's <c>WorkerFrameProtocol</c> discipline:
|
||||
/// zero-length, oversize, truncated and unparseable frames are all a <see cref="ScriptWorkerProtocolException"/>,
|
||||
/// after which the stream is unusable and the peer is dropped.
|
||||
/// </summary>
|
||||
public static class ScriptWorkerFrameCodec
|
||||
{
|
||||
/// <summary>The protocol version both ends put on every envelope.</summary>
|
||||
public const uint ProtocolVersion = 1;
|
||||
|
||||
/// <summary>Default frame cap: 4 MiB.</summary>
|
||||
public const int DefaultMaxFrameBytes = 4 * 1024 * 1024;
|
||||
|
||||
/// <summary>Writes one frame. Not thread-safe: callers serialise writes.</summary>
|
||||
/// <param name="stream">The pipe.</param>
|
||||
/// <param name="envelope">The envelope.</param>
|
||||
/// <param name="maxFrameBytes">The negotiated cap; a larger envelope is refused before anything is written.</param>
|
||||
/// <param name="ct">Cancellation.</param>
|
||||
/// <returns>A task that completes when the frame is written (not flushed).</returns>
|
||||
public static async ValueTask WriteAsync(Stream stream, Envelope envelope, int maxFrameBytes, CancellationToken ct = default)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(stream);
|
||||
ArgumentNullException.ThrowIfNull(envelope);
|
||||
var size = envelope.CalculateSize();
|
||||
if (size == 0) throw new ScriptWorkerProtocolException("refusing to write an empty frame");
|
||||
if (size > maxFrameBytes)
|
||||
throw new ScriptWorkerProtocolException($"frame of {size} bytes exceeds the {maxFrameBytes}-byte cap");
|
||||
|
||||
var buffer = ArrayPool<byte>.Shared.Rent(size + 4);
|
||||
try
|
||||
{
|
||||
BinaryPrimitives.WriteInt32LittleEndian(buffer, size);
|
||||
envelope.WriteTo(new Span<byte>(buffer, 4, size));
|
||||
await stream.WriteAsync(buffer.AsMemory(0, size + 4), ct).ConfigureAwait(false);
|
||||
}
|
||||
finally
|
||||
{
|
||||
ArrayPool<byte>.Shared.Return(buffer);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Reads one frame.</summary>
|
||||
/// <param name="stream">The pipe.</param>
|
||||
/// <param name="maxFrameBytes">The negotiated cap.</param>
|
||||
/// <param name="ct">Cancellation.</param>
|
||||
/// <returns>The envelope, or null on a clean end of stream at a frame boundary.</returns>
|
||||
/// <exception cref="ScriptWorkerProtocolException">A zero-length, oversize, truncated or unparseable frame.</exception>
|
||||
public static async ValueTask<Envelope?> ReadAsync(Stream stream, int maxFrameBytes, CancellationToken ct = default)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(stream);
|
||||
var header = new byte[4];
|
||||
var got = await ReadFullyAsync(stream, header, ct).ConfigureAwait(false);
|
||||
if (got == 0) return null;
|
||||
if (got < 4) throw new ScriptWorkerProtocolException("truncated frame header");
|
||||
|
||||
var size = BinaryPrimitives.ReadInt32LittleEndian(header);
|
||||
if (size == 0) throw new ScriptWorkerProtocolException("zero-length frame");
|
||||
if (size < 0 || size > maxFrameBytes)
|
||||
throw new ScriptWorkerProtocolException($"frame length {size} is outside 1..{maxFrameBytes}");
|
||||
|
||||
var body = ArrayPool<byte>.Shared.Rent(size);
|
||||
try
|
||||
{
|
||||
var read = await ReadFullyAsync(stream, body.AsMemory(0, size), ct).ConfigureAwait(false);
|
||||
if (read < size) throw new ScriptWorkerProtocolException($"truncated frame: {read} of {size} bytes");
|
||||
Envelope envelope;
|
||||
try
|
||||
{
|
||||
envelope = Envelope.Parser.ParseFrom(body, 0, size);
|
||||
}
|
||||
catch (InvalidProtocolBufferException ex)
|
||||
{
|
||||
throw new ScriptWorkerProtocolException($"unparseable frame: {ex.Message}", ex);
|
||||
}
|
||||
if (envelope.ProtocolVersion != ProtocolVersion)
|
||||
throw new ScriptWorkerProtocolException(
|
||||
$"protocol version {envelope.ProtocolVersion} is not the supported {ProtocolVersion}");
|
||||
if (envelope.BodyCase == Envelope.BodyOneofCase.None)
|
||||
throw new ScriptWorkerProtocolException("frame carries no body");
|
||||
return envelope;
|
||||
}
|
||||
finally
|
||||
{
|
||||
ArrayPool<byte>.Shared.Return(body);
|
||||
}
|
||||
}
|
||||
|
||||
private static async ValueTask<int> ReadFullyAsync(Stream stream, Memory<byte> buffer, CancellationToken ct)
|
||||
{
|
||||
var total = 0;
|
||||
while (total < buffer.Length)
|
||||
{
|
||||
var n = await stream.ReadAsync(buffer[total..], ct).ConfigureAwait(false);
|
||||
if (n == 0) break;
|
||||
total += n;
|
||||
}
|
||||
return total;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>A framing or schema violation on the ScriptWorker pipe; the connection is dropped.</summary>
|
||||
public sealed class ScriptWorkerProtocolException : Exception
|
||||
{
|
||||
/// <summary>Initializes the exception.</summary>
|
||||
/// <param name="message">What was wrong.</param>
|
||||
/// <param name="inner">The underlying parser error, if any.</param>
|
||||
public ScriptWorkerProtocolException(string message, Exception? inner = null) : base(message, inner) { }
|
||||
}
|
||||
@@ -0,0 +1,159 @@
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
|
||||
|
||||
/// <summary>
|
||||
/// CLR value ↔ <see cref="Variant"/> (design §4.4). The exact boxed CLR type survives the round trip — a script
|
||||
/// doing <c>(short)ctx.GetTag(x).Value</c> throws <see cref="InvalidCastException"/> if a <c>short</c> arrives as an
|
||||
/// <c>int</c> — so the type tag names the CLR type, not just an OPC UA one. A type with no tag is never converted:
|
||||
/// <see cref="TryEncode"/> refuses it, and the caller reports <c>UNSUPPORTED_TYPE</c> (design §14 #11).
|
||||
/// </summary>
|
||||
public static class VariantCodec
|
||||
{
|
||||
/// <summary>Encodes <paramref name="value"/>.</summary>
|
||||
/// <param name="value">The boxed CLR value (null allowed).</param>
|
||||
/// <param name="variant">The encoded variant when supported.</param>
|
||||
/// <param name="unsupportedType">The CLR type name when not supported.</param>
|
||||
/// <returns><c>true</c> when the type has a tag.</returns>
|
||||
public static bool TryEncode(object? value, out Variant variant, out string? unsupportedType)
|
||||
{
|
||||
unsupportedType = null;
|
||||
variant = new Variant();
|
||||
switch (value)
|
||||
{
|
||||
case null: variant.Type = Variant.Types.TypeTag.Null; return true;
|
||||
case bool b: variant.Type = Variant.Types.TypeTag.Boolean; variant.B = b; return true;
|
||||
case sbyte v: variant.Type = Variant.Types.TypeTag.Sbyte; variant.I = v; return true;
|
||||
case byte v: variant.Type = Variant.Types.TypeTag.Byte; variant.U = v; return true;
|
||||
case short v: variant.Type = Variant.Types.TypeTag.Int16; variant.I = v; return true;
|
||||
case ushort v: variant.Type = Variant.Types.TypeTag.Uint16; variant.U = v; return true;
|
||||
case int v: variant.Type = Variant.Types.TypeTag.Int32; variant.I = v; return true;
|
||||
case uint v: variant.Type = Variant.Types.TypeTag.Uint32; variant.U = v; return true;
|
||||
case long v: variant.Type = Variant.Types.TypeTag.Int64; variant.I = v; return true;
|
||||
case ulong v: variant.Type = Variant.Types.TypeTag.Uint64; variant.U = v; return true;
|
||||
case float v: variant.Type = Variant.Types.TypeTag.Float; variant.F = v; return true;
|
||||
case double v: variant.Type = Variant.Types.TypeTag.Double; variant.D = v; return true;
|
||||
case string v: variant.Type = Variant.Types.TypeTag.String; variant.S = v; return true;
|
||||
case DateTime v:
|
||||
variant.Type = Variant.Types.TypeTag.DatetimeUtc;
|
||||
variant.DtTicks = ToUtc(v).Ticks;
|
||||
return true;
|
||||
case Array array when array.Rank == 1 && TryElementTag(array.GetType().GetElementType()!, out var elementTag):
|
||||
{
|
||||
var av = new ArrayValue { ElementType = elementTag };
|
||||
foreach (var item in array)
|
||||
{
|
||||
if (!TryEncode(item, out var element, out unsupportedType)) return false;
|
||||
av.Items.Add(element);
|
||||
}
|
||||
variant.Type = Variant.Types.TypeTag.Array;
|
||||
variant.Array = av;
|
||||
return true;
|
||||
}
|
||||
default:
|
||||
unsupportedType = value.GetType().FullName ?? value.GetType().Name;
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Decodes <paramref name="variant"/> back to the boxed CLR value it was encoded from.</summary>
|
||||
/// <param name="variant">The variant.</param>
|
||||
/// <returns>The boxed value.</returns>
|
||||
/// <exception cref="ScriptWorkerProtocolException">An unknown tag, or a payload that does not match the tag.</exception>
|
||||
public static object? Decode(Variant? variant)
|
||||
{
|
||||
if (variant is null) return null;
|
||||
return variant.Type switch
|
||||
{
|
||||
Variant.Types.TypeTag.Null => null,
|
||||
Variant.Types.TypeTag.Boolean => Expect(variant, Variant.VOneofCase.B).B,
|
||||
Variant.Types.TypeTag.Sbyte => checked((sbyte)Expect(variant, Variant.VOneofCase.I).I),
|
||||
Variant.Types.TypeTag.Byte => checked((byte)Expect(variant, Variant.VOneofCase.U).U),
|
||||
Variant.Types.TypeTag.Int16 => checked((short)Expect(variant, Variant.VOneofCase.I).I),
|
||||
Variant.Types.TypeTag.Uint16 => checked((ushort)Expect(variant, Variant.VOneofCase.U).U),
|
||||
Variant.Types.TypeTag.Int32 => checked((int)Expect(variant, Variant.VOneofCase.I).I),
|
||||
Variant.Types.TypeTag.Uint32 => checked((uint)Expect(variant, Variant.VOneofCase.U).U),
|
||||
Variant.Types.TypeTag.Int64 => Expect(variant, Variant.VOneofCase.I).I,
|
||||
Variant.Types.TypeTag.Uint64 => Expect(variant, Variant.VOneofCase.U).U,
|
||||
Variant.Types.TypeTag.Float => Expect(variant, Variant.VOneofCase.F).F,
|
||||
Variant.Types.TypeTag.Double => Expect(variant, Variant.VOneofCase.D).D,
|
||||
Variant.Types.TypeTag.String => Expect(variant, Variant.VOneofCase.S).S,
|
||||
Variant.Types.TypeTag.DatetimeUtc => new DateTime(Expect(variant, Variant.VOneofCase.DtTicks).DtTicks, DateTimeKind.Utc),
|
||||
Variant.Types.TypeTag.Array => DecodeArray(Expect(variant, Variant.VOneofCase.Array).Array),
|
||||
_ => throw new ScriptWorkerProtocolException($"unknown variant type tag {(int)variant.Type}"),
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>A <see cref="DateTime"/> as UTC: a <c>Local</c> value is converted; an <c>Unspecified</c> one is taken as UTC.</summary>
|
||||
/// <param name="value">The value.</param>
|
||||
/// <returns>The UTC value.</returns>
|
||||
public static DateTime ToUtc(DateTime value) => value.Kind switch
|
||||
{
|
||||
DateTimeKind.Utc => value,
|
||||
DateTimeKind.Local => value.ToUniversalTime(),
|
||||
_ => DateTime.SpecifyKind(value, DateTimeKind.Utc),
|
||||
};
|
||||
|
||||
private static Variant Expect(Variant variant, Variant.VOneofCase expected)
|
||||
{
|
||||
// proto3 drops a default scalar (0, false, "") from the oneof, so an absent payload is the default value.
|
||||
if (variant.VCase != expected && variant.VCase != Variant.VOneofCase.None)
|
||||
throw new ScriptWorkerProtocolException($"variant tagged {variant.Type} carries a {variant.VCase} payload");
|
||||
return variant;
|
||||
}
|
||||
|
||||
private static Array DecodeArray(ArrayValue? array)
|
||||
{
|
||||
if (array is null) throw new ScriptWorkerProtocolException("array variant without an array payload");
|
||||
var elementType = ClrType(array.ElementType);
|
||||
var result = Array.CreateInstance(elementType, array.Items.Count);
|
||||
for (var i = 0; i < array.Items.Count; i++)
|
||||
{
|
||||
var item = array.Items[i];
|
||||
if (item.Type != array.ElementType && item.Type != Variant.Types.TypeTag.Null)
|
||||
throw new ScriptWorkerProtocolException($"array of {array.ElementType} holds a {item.Type} element");
|
||||
result.SetValue(Decode(item), i);
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
private static bool TryElementTag(Type elementType, out Variant.Types.TypeTag tag)
|
||||
{
|
||||
tag = elementType switch
|
||||
{
|
||||
_ when elementType == typeof(bool) => Variant.Types.TypeTag.Boolean,
|
||||
_ when elementType == typeof(sbyte) => Variant.Types.TypeTag.Sbyte,
|
||||
_ when elementType == typeof(byte) => Variant.Types.TypeTag.Byte,
|
||||
_ when elementType == typeof(short) => Variant.Types.TypeTag.Int16,
|
||||
_ when elementType == typeof(ushort) => Variant.Types.TypeTag.Uint16,
|
||||
_ when elementType == typeof(int) => Variant.Types.TypeTag.Int32,
|
||||
_ when elementType == typeof(uint) => Variant.Types.TypeTag.Uint32,
|
||||
_ when elementType == typeof(long) => Variant.Types.TypeTag.Int64,
|
||||
_ when elementType == typeof(ulong) => Variant.Types.TypeTag.Uint64,
|
||||
_ when elementType == typeof(float) => Variant.Types.TypeTag.Float,
|
||||
_ when elementType == typeof(double) => Variant.Types.TypeTag.Double,
|
||||
_ when elementType == typeof(string) => Variant.Types.TypeTag.String,
|
||||
_ when elementType == typeof(DateTime) => Variant.Types.TypeTag.DatetimeUtc,
|
||||
_ => Variant.Types.TypeTag.Null,
|
||||
};
|
||||
return tag != Variant.Types.TypeTag.Null;
|
||||
}
|
||||
|
||||
private static Type ClrType(Variant.Types.TypeTag tag) => tag switch
|
||||
{
|
||||
Variant.Types.TypeTag.Boolean => typeof(bool),
|
||||
Variant.Types.TypeTag.Sbyte => typeof(sbyte),
|
||||
Variant.Types.TypeTag.Byte => typeof(byte),
|
||||
Variant.Types.TypeTag.Int16 => typeof(short),
|
||||
Variant.Types.TypeTag.Uint16 => typeof(ushort),
|
||||
Variant.Types.TypeTag.Int32 => typeof(int),
|
||||
Variant.Types.TypeTag.Uint32 => typeof(uint),
|
||||
Variant.Types.TypeTag.Int64 => typeof(long),
|
||||
Variant.Types.TypeTag.Uint64 => typeof(ulong),
|
||||
Variant.Types.TypeTag.Float => typeof(float),
|
||||
Variant.Types.TypeTag.Double => typeof(double),
|
||||
Variant.Types.TypeTag.String => typeof(string),
|
||||
Variant.Types.TypeTag.DatetimeUtc => typeof(DateTime),
|
||||
_ => throw new ScriptWorkerProtocolException($"array element type {tag} is not supported"),
|
||||
};
|
||||
}
|
||||
+28
@@ -0,0 +1,28 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<RootNamespace>ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol</RootNamespace>
|
||||
<GenerateDocumentationFile>true</GenerateDocumentationFile>
|
||||
<!-- protoc-generated members carry no XML docs. -->
|
||||
<NoWarn>$(NoWarn);CS1591</NoWarn>
|
||||
</PropertyGroup>
|
||||
|
||||
<!-- lmxopcua#780: the host <-> ScriptWorker frame protocol (design §4). Shared by the host-side runner and the
|
||||
worker, so it references nothing but Google.Protobuf: the worker's assembly set stays small (design §6). -->
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Google.Protobuf"/>
|
||||
<PackageReference Include="Grpc.Tools">
|
||||
<PrivateAssets>all</PrivateAssets>
|
||||
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
|
||||
</PackageReference>
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<Protobuf Include="Protos\script_worker.proto" GrpcServices="None"/>
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests"/>
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
@@ -0,0 +1,137 @@
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
||||
|
||||
/// <summary>
|
||||
/// The execution seam every host script owner can run a compiled script through (lmxopcua#780, design §11): a
|
||||
/// script key, its source, its input snapshots and its budget go in; one <see cref="ScriptRunOutcome"/> comes
|
||||
/// out. Implementations never throw for script faults.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <see cref="InProcessScriptRunner"/> is today's <see cref="ScriptRunGuard"/> + <see cref="CompiledScriptCache{TContext, TResult}"/>
|
||||
/// behind this interface (the parity baseline). The out-of-process runner
|
||||
/// (<c>ZB.MOM.WW.OtOpcUa.Core.Scripting.Runner</c>) runs the same compile in a pre-emptible worker process.
|
||||
/// The passthrough fast path (<c>PassthroughScript</c>) never reaches a runner: it runs no user code.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Keys are namespaced by <see cref="ScriptInvocation.Kind"/>, so a virtual-tag id and a calc RawPath that
|
||||
/// happen to be equal never share quarantine state, and <see cref="Reset"/> for one seam leaves the other
|
||||
/// seams' quarantines alone.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public interface IScriptRunner
|
||||
{
|
||||
/// <summary>Runs one compiled evaluation.</summary>
|
||||
/// <param name="invocation">What to run.</param>
|
||||
/// <param name="ct">Caller cancellation (shutdown); wins over the budget.</param>
|
||||
/// <returns>The outcome; never faults for a script failure.</returns>
|
||||
Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default);
|
||||
|
||||
/// <summary>
|
||||
/// The deploy boundary for one seam: lifts every quarantine and forgets every remembered compile failure of
|
||||
/// <paramref name="kind"/>, and drops its compiled scripts.
|
||||
/// </summary>
|
||||
/// <param name="kind">The seam's context kind.</param>
|
||||
void Reset(ScriptContextKind kind);
|
||||
|
||||
/// <summary>The keys of <paramref name="kind"/> currently quarantined.</summary>
|
||||
/// <param name="kind">The seam's context kind.</param>
|
||||
/// <returns>The quarantined keys (un-namespaced).</returns>
|
||||
IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind);
|
||||
}
|
||||
|
||||
/// <summary>Which script context (and result type) a script compiles against.</summary>
|
||||
public enum ScriptContextKind
|
||||
{
|
||||
/// <summary>A virtual tag: <c>VirtualTagContext</c> → <c>object?</c>.</summary>
|
||||
VirtualTag = 1,
|
||||
|
||||
/// <summary>A Calculation tag: <c>VirtualTagContext</c> → <c>object?</c>.</summary>
|
||||
Calculation = 2,
|
||||
|
||||
/// <summary>A scripted-alarm predicate: <c>AlarmPredicateContext</c> → <c>bool</c> (not run out of process yet).</summary>
|
||||
AlarmPredicate = 3,
|
||||
}
|
||||
|
||||
/// <summary>One evaluation handed to an <see cref="IScriptRunner"/>.</summary>
|
||||
/// <param name="Key">The script instance key (virtual-tag id, calc RawPath, alarm id).</param>
|
||||
/// <param name="Kind">The context kind.</param>
|
||||
/// <param name="Source">The script source (also its identity: a change lifts the key's quarantine).</param>
|
||||
/// <param name="Inputs">The read cache: dependency snapshots by path. Not mutated until the run completes.</param>
|
||||
/// <param name="Timeout">The soft budget <c>T</c>.</param>
|
||||
/// <param name="ScriptLogger">The per-evaluation script logger <c>ctx.Logger</c> writes through; null ⇒ a silent logger.</param>
|
||||
public sealed record ScriptInvocation(
|
||||
string Key,
|
||||
ScriptContextKind Kind,
|
||||
string Source,
|
||||
IReadOnlyDictionary<string, DataValueSnapshot> Inputs,
|
||||
TimeSpan Timeout,
|
||||
Serilog.ILogger? ScriptLogger = null);
|
||||
|
||||
/// <summary>How an evaluation ended.</summary>
|
||||
public enum ScriptRunStatus
|
||||
{
|
||||
/// <summary>The script returned <see cref="ScriptRunOutcome.Value"/>.</summary>
|
||||
Ok = 0,
|
||||
|
||||
/// <summary>A script-attributable failure (threw, timed out, refused, quarantined, did not compile) — Bad as today.</summary>
|
||||
Failed = 1,
|
||||
|
||||
/// <summary>
|
||||
/// The runner could not run the script, and it is not the script's fault: its worker restarted under it, or no
|
||||
/// worker became ready in time. Published as <c>BadResourceUnavailable</c> (design §9 / §14 #3).
|
||||
/// </summary>
|
||||
Unavailable = 2,
|
||||
}
|
||||
|
||||
/// <summary>The result of one <see cref="IScriptRunner.RunAsync"/>.</summary>
|
||||
/// <param name="Status">How it ended.</param>
|
||||
/// <param name="Value">The script's result when <see cref="ScriptRunStatus.Ok"/>.</param>
|
||||
/// <param name="Reason">The failure reason otherwise; the same strings the evaluators have always published.</param>
|
||||
public sealed record ScriptRunOutcome(ScriptRunStatus Status, object? Value, string? Reason)
|
||||
{
|
||||
/// <summary>Reason when the worker running the script died under it (another script's kill, a crash).</summary>
|
||||
public const string WorkerRestartedReason = "script worker restarted";
|
||||
|
||||
/// <summary>Reason when no worker became ready within <see cref="ScriptWorkerOptions.UnavailableGrace"/>.</summary>
|
||||
public const string RunnerUnavailableReason = "script runner unavailable";
|
||||
|
||||
/// <summary>A successful run.</summary>
|
||||
/// <param name="value">The script's result.</param>
|
||||
/// <returns>The outcome.</returns>
|
||||
public static ScriptRunOutcome Ok(object? value) => new(ScriptRunStatus.Ok, value, null);
|
||||
|
||||
/// <summary>A script-attributable failure.</summary>
|
||||
/// <param name="reason">The reason.</param>
|
||||
/// <returns>The outcome.</returns>
|
||||
public static ScriptRunOutcome Failed(string reason) => new(ScriptRunStatus.Failed, null, reason);
|
||||
|
||||
/// <summary>A runner-attributable failure.</summary>
|
||||
/// <param name="reason">The reason.</param>
|
||||
/// <returns>The outcome.</returns>
|
||||
public static ScriptRunOutcome Unavailable(string reason) => new(ScriptRunStatus.Unavailable, null, reason);
|
||||
}
|
||||
|
||||
/// <summary>Maps an <see cref="IScriptRunner"/> outcome onto the value seams' result type.</summary>
|
||||
public static class ScriptRunOutcomeExtensions
|
||||
{
|
||||
/// <summary>
|
||||
/// <see cref="ScriptRunStatus.Ok"/> → <c>Ok(value)</c>; <see cref="ScriptRunStatus.Failed"/> → <c>Failure(reason)</c>
|
||||
/// (published <c>BadInternalError</c>, as today); <see cref="ScriptRunStatus.Unavailable"/> →
|
||||
/// <c>Unavailable(reason)</c> (published <c>BadResourceUnavailable</c>).
|
||||
/// </summary>
|
||||
/// <param name="outcome">The runner's outcome.</param>
|
||||
/// <returns>The seam result.</returns>
|
||||
public static Commons.Engines.VirtualTagEvalResult ToVirtualTagEvalResult(this ScriptRunOutcome outcome)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(outcome);
|
||||
return outcome.Status switch
|
||||
{
|
||||
ScriptRunStatus.Ok => Commons.Engines.VirtualTagEvalResult.Ok(outcome.Value),
|
||||
ScriptRunStatus.Unavailable => Commons.Engines.VirtualTagEvalResult.Unavailable(
|
||||
outcome.Reason ?? ScriptRunOutcome.RunnerUnavailableReason),
|
||||
_ => Commons.Engines.VirtualTagEvalResult.Failure(outcome.Reason ?? "evaluator failure"),
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,141 @@
|
||||
using System.Collections.Concurrent;
|
||||
using Serilog;
|
||||
using Serilog.Core;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Engines;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.VirtualTags;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
||||
|
||||
/// <summary>
|
||||
/// <see cref="IScriptRunner"/> over today's in-process machinery: a <see cref="CompiledScriptCache{TContext, TResult}"/>
|
||||
/// per context kind and one <see cref="ScriptRunGuard"/> (lmxopcua#780, design §11). It produces the outcomes and
|
||||
/// reason strings the seams' own in-process paths produce, which makes it the parity baseline the out-of-process
|
||||
/// runner is tested against, and the in-process half of the future shadow mode (P2).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The seams do not route their <see cref="ScriptRunnerMode.InProcess"/> path through this class: they keep their
|
||||
/// existing guard code, so the default mode is unchanged byte for byte. Unlike those paths, this runner remembers
|
||||
/// compile failures by source for every kind (Calculation already did, R07:P4).
|
||||
/// </remarks>
|
||||
public sealed class InProcessScriptRunner : IScriptRunner, IDisposable
|
||||
{
|
||||
private readonly CompiledScriptCache<VirtualTagContext, object?> _valueCache = new();
|
||||
private readonly CompiledScriptCache<AlarmPredicateContext, bool> _predicateCache = new();
|
||||
private readonly ConcurrentDictionary<string, string> _compileFailures = new(StringComparer.Ordinal);
|
||||
private readonly ScriptRunGuard _guard;
|
||||
|
||||
/// <summary>Initializes the runner.</summary>
|
||||
/// <param name="scripting">Run-guard knobs; null ⇒ defaults.</param>
|
||||
/// <param name="logger">Where quarantine lines go; null ⇒ Serilog's global logger.</param>
|
||||
public InProcessScriptRunner(ScriptingOptions? scripting = null, ILogger? logger = null)
|
||||
{
|
||||
_guard = new ScriptRunGuard(scripting, logger);
|
||||
}
|
||||
|
||||
/// <summary>The run guard (diagnostics / tests).</summary>
|
||||
public ScriptRunGuard RunGuard => _guard;
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(invocation);
|
||||
var key = ScopedKey(invocation.Kind, invocation.Key);
|
||||
var failureKey = ScopedKey(invocation.Kind, invocation.Source);
|
||||
if (_compileFailures.TryGetValue(failureKey, out var known)) return ScriptRunOutcome.Failed(known);
|
||||
var logger = invocation.ScriptLogger ?? Logger.None;
|
||||
|
||||
try
|
||||
{
|
||||
switch (invocation.Kind)
|
||||
{
|
||||
case ScriptContextKind.VirtualTag:
|
||||
case ScriptContextKind.Calculation:
|
||||
{
|
||||
var evaluator = await Task.Run(() => _valueCache.GetOrCompile(invocation.Source), ct).ConfigureAwait(false);
|
||||
var context = new VirtualTagContext(invocation.Inputs, static (_, _) => { }, logger);
|
||||
var value = await _guard.RunAsync(key, invocation.Source, evaluator, context, invocation.Timeout, ct)
|
||||
.ConfigureAwait(false);
|
||||
return ScriptRunOutcome.Ok(value);
|
||||
}
|
||||
case ScriptContextKind.AlarmPredicate:
|
||||
{
|
||||
var evaluator = await Task.Run(() => _predicateCache.GetOrCompile(invocation.Source), ct).ConfigureAwait(false);
|
||||
var context = new AlarmPredicateContext(invocation.Inputs, logger);
|
||||
var value = await _guard.RunAsync(key, invocation.Source, evaluator, context, invocation.Timeout, ct)
|
||||
.ConfigureAwait(false);
|
||||
return ScriptRunOutcome.Ok(value);
|
||||
}
|
||||
default:
|
||||
return ScriptRunOutcome.Failed($"unsupported script kind {invocation.Kind}");
|
||||
}
|
||||
}
|
||||
catch (CompilationErrorException ex)
|
||||
{
|
||||
return RememberCompileFailure(failureKey, $"compile error: {ex.Message}");
|
||||
}
|
||||
catch (ScriptSandboxViolationException ex)
|
||||
{
|
||||
return RememberCompileFailure(failureKey, $"sandbox violation: {ex.Message}");
|
||||
}
|
||||
catch (ScriptTimeoutException)
|
||||
{
|
||||
return ScriptRunOutcome.Failed($"{ScriptEvalFailureReasons.TimedOutPrefix}{invocation.Timeout.TotalSeconds:F1}s");
|
||||
}
|
||||
catch (ScriptRunRefusedException ex)
|
||||
{
|
||||
return ScriptRunOutcome.Failed(ex.Message);
|
||||
}
|
||||
catch (ScriptQuarantinedException ex)
|
||||
{
|
||||
return ScriptRunOutcome.Failed(ex.Message);
|
||||
}
|
||||
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
||||
{
|
||||
return ScriptRunOutcome.Failed("evaluation cancelled");
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
return ScriptRunOutcome.Failed($"script threw: {ex.Message}");
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Reset(ScriptContextKind kind)
|
||||
{
|
||||
var prefix = ScopedKey(kind, string.Empty);
|
||||
foreach (var k in _compileFailures.Keys)
|
||||
if (k.StartsWith(prefix, StringComparison.Ordinal)) _compileFailures.TryRemove(k, out _);
|
||||
foreach (var k in _guard.QuarantinedKeys)
|
||||
if (k.StartsWith(prefix, StringComparison.Ordinal)) _guard.Reset(k);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind)
|
||||
{
|
||||
var prefix = ScopedKey(kind, string.Empty);
|
||||
return _guard.QuarantinedKeys
|
||||
.Where(k => k.StartsWith(prefix, StringComparison.Ordinal))
|
||||
.Select(k => k[prefix.Length..])
|
||||
.ToArray();
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Dispose()
|
||||
{
|
||||
_valueCache.Dispose();
|
||||
_predicateCache.Dispose();
|
||||
}
|
||||
|
||||
/// <summary>The runner-internal key: <c>{kind}:{key}</c>.</summary>
|
||||
/// <param name="kind">The context kind.</param>
|
||||
/// <param name="key">The seam's key.</param>
|
||||
/// <returns>The namespaced key.</returns>
|
||||
public static string ScopedKey(ScriptContextKind kind, string key) => $"{(int)kind}:{key}";
|
||||
|
||||
private ScriptRunOutcome RememberCompileFailure(string failureKey, string reason)
|
||||
{
|
||||
_compileFailures.TryAdd(failureKey, reason);
|
||||
return ScriptRunOutcome.Failed(reason);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,213 @@
|
||||
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
||||
|
||||
/// <summary>
|
||||
/// Where compiled scripts run (<c>Scripting:Runner</c>, lmxopcua#780). <see cref="InProcess"/> is the default
|
||||
/// and the documented rollback: it is today's <see cref="ScriptRunGuard"/> path, unchanged.
|
||||
/// </summary>
|
||||
public enum ScriptRunnerMode
|
||||
{
|
||||
/// <summary>Compiled scripts run in the host process under <see cref="ScriptRunGuard"/> (the default).</summary>
|
||||
InProcess = 0,
|
||||
|
||||
/// <summary>
|
||||
/// Run in process AND submit to the worker, comparing the two (design P2). Not implemented yet: a node
|
||||
/// configured for it runs <see cref="InProcess"/> and logs a startup Warning.
|
||||
/// </summary>
|
||||
Shadow = 1,
|
||||
|
||||
/// <summary>
|
||||
/// Compiled scripts of the selected seams run in the pre-emptible <c>ScriptWorker</c> process pool. The
|
||||
/// runner fails closed: a worker that cannot start makes the script Bad
|
||||
/// (<c>BadResourceUnavailable</c>), never an in-process run.
|
||||
/// </summary>
|
||||
OutOfProcess = 2,
|
||||
}
|
||||
|
||||
/// <summary>The three host seams that run operator-authored C# (<c>Scripting:Runner:Seams</c>).</summary>
|
||||
public enum ScriptSeam
|
||||
{
|
||||
/// <summary>Virtual tags (<c>RoslynVirtualTagEvaluator</c>).</summary>
|
||||
VirtualTag = 0,
|
||||
|
||||
/// <summary>Calculation tags (<c>CalculationEvaluator</c>).</summary>
|
||||
Calculation = 1,
|
||||
|
||||
/// <summary>Scripted-alarm predicates (<c>ScriptedAlarmEngine</c>). Last in the rollout (design §14 #10).</summary>
|
||||
ScriptedAlarm = 2,
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The out-of-process worker pool's knobs (the <c>Scripting:Worker</c> section). Every value is clamped by
|
||||
/// <see cref="Sanitized"/>, which reports each out-of-range key as a startup Warning, like the rest of the
|
||||
/// <c>Scripting</c> section. Only consulted when <see cref="ScriptingOptions.Runner"/> is not
|
||||
/// <see cref="ScriptRunnerMode.InProcess"/>.
|
||||
/// </summary>
|
||||
public sealed class ScriptWorkerOptions
|
||||
{
|
||||
/// <summary>Default <see cref="Shards"/>: 2 (design §14 #1).</summary>
|
||||
public const int DefaultShards = 2;
|
||||
|
||||
/// <summary>Upper bound of <see cref="Shards"/>.</summary>
|
||||
public const int MaxShards = 8;
|
||||
|
||||
/// <summary>Default <see cref="HeapLimitBytes"/>: 512 MiB.</summary>
|
||||
public const long DefaultHeapLimitBytes = 512L * 1024 * 1024;
|
||||
|
||||
/// <summary>Lower bound of <see cref="HeapLimitBytes"/>: 64 MiB (a worker below this cannot host Roslyn).</summary>
|
||||
public const long MinHeapLimitBytes = 64L * 1024 * 1024;
|
||||
|
||||
/// <summary>Default <see cref="QuarantineAfterKills"/>: 2.</summary>
|
||||
public const int DefaultQuarantineAfterKills = 2;
|
||||
|
||||
/// <summary>Default <see cref="UnavailableGrace"/>: 5 s.</summary>
|
||||
public static readonly TimeSpan DefaultUnavailableGrace = TimeSpan.FromSeconds(5);
|
||||
|
||||
/// <summary>Lower bound of an explicit <see cref="HardGrace"/>: 250 ms.</summary>
|
||||
public static readonly TimeSpan MinHardGrace = TimeSpan.FromMilliseconds(250);
|
||||
|
||||
/// <summary>Upper bound of an explicit <see cref="HardGrace"/>: 30 s.</summary>
|
||||
public static readonly TimeSpan MaxHardGrace = TimeSpan.FromSeconds(30);
|
||||
|
||||
/// <summary>Upper bound of <see cref="UnavailableGrace"/>: 60 s.</summary>
|
||||
public static readonly TimeSpan MaxUnavailableGrace = TimeSpan.FromSeconds(60);
|
||||
|
||||
/// <summary>Default <see cref="IsolationProbationRuns"/>: 20.</summary>
|
||||
public const int DefaultIsolationProbationRuns = 20;
|
||||
|
||||
/// <summary>Shard workers per node; a script is placed by a stable hash of its key. Clamped to 1–8.</summary>
|
||||
public int Shards { get; set; } = DefaultShards;
|
||||
|
||||
/// <summary>
|
||||
/// Grace after the soft deadline <c>T</c> before the worker running the script is killed. Null (the default)
|
||||
/// means <c>max(1 s, T)</c> per run (design §14 #2); an explicit value is clamped to 250 ms – 30 s.
|
||||
/// </summary>
|
||||
public TimeSpan? HardGrace { get; set; }
|
||||
|
||||
/// <summary>The worker's <c>DOTNET_GCHeapHardLimit</c>. Default 512 MiB, minimum 64 MiB.</summary>
|
||||
public long HeapLimitBytes { get; set; } = DefaultHeapLimitBytes;
|
||||
|
||||
/// <summary>Hard-deadline kills (within one hour) after which a script is quarantined. Default 2; 1–100.</summary>
|
||||
public int QuarantineAfterKills { get; set; } = DefaultQuarantineAfterKills;
|
||||
|
||||
/// <summary>
|
||||
/// How long an evaluation waits for its shard to become ready (a restart, a backoff) before it reports
|
||||
/// <c>script runner unavailable</c>. Default 5 s; 0–60 s.
|
||||
/// </summary>
|
||||
public TimeSpan UnavailableGrace { get; set; } = DefaultUnavailableGrace;
|
||||
|
||||
/// <summary>
|
||||
/// Successful runs a suspect script must complete in the isolation worker before it returns to its shard.
|
||||
/// Default 20; 1–10 000.
|
||||
/// </summary>
|
||||
public int IsolationProbationRuns { get; set; } = DefaultIsolationProbationRuns;
|
||||
|
||||
/// <summary>
|
||||
/// Directory of the node-local compiled-PE cache (design §7). Must be absolute and outside the content root
|
||||
/// when set; startup fails otherwise (the family's fail-closed path rule). Reserved: the cache itself has
|
||||
/// not landed yet (lmxopcua#780 P1 remainder), so a worker restart recompiles.
|
||||
/// </summary>
|
||||
public string? PeCacheDir { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Path of the worker entry assembly (<c>ZB.MOM.WW.OtOpcUa.ScriptWorker.dll</c>). Null (the default) means
|
||||
/// <c>script-worker/</c> under the host's base directory, where the Host build places it.
|
||||
/// </summary>
|
||||
public string? ExecutablePath { get; set; }
|
||||
|
||||
/// <summary>The hard grace that applies to a run with soft budget <paramref name="softTimeout"/>.</summary>
|
||||
/// <param name="softTimeout">The run's soft deadline <c>T</c>.</param>
|
||||
/// <returns><see cref="HardGrace"/> when set, otherwise <c>max(1 s, T)</c>.</returns>
|
||||
public TimeSpan EffectiveHardGrace(TimeSpan softTimeout)
|
||||
=> HardGrace ?? (softTimeout > TimeSpan.FromSeconds(1) ? softTimeout : TimeSpan.FromSeconds(1));
|
||||
|
||||
/// <summary>Returns a clamped copy; one line per out-of-range key in <paramref name="adjustments"/>.</summary>
|
||||
/// <param name="adjustments">The clamps applied, naming each key.</param>
|
||||
/// <returns>A new, clamped instance.</returns>
|
||||
public ScriptWorkerOptions Sanitized(out IReadOnlyList<string> adjustments)
|
||||
{
|
||||
const string prefix = ScriptingOptions.SectionName + ":Worker:";
|
||||
var notes = new List<string>();
|
||||
|
||||
var shards = Shards;
|
||||
if (shards < 1 || shards > MaxShards)
|
||||
{
|
||||
var clamped = Math.Clamp(shards, 1, MaxShards);
|
||||
notes.Add($"{prefix}{nameof(Shards)} {shards} is outside 1–{MaxShards}; clamped to {clamped}.");
|
||||
shards = clamped;
|
||||
}
|
||||
|
||||
var grace = HardGrace;
|
||||
if (grace is { } g && (g < MinHardGrace || g > MaxHardGrace))
|
||||
{
|
||||
var clamped = g < MinHardGrace ? MinHardGrace : MaxHardGrace;
|
||||
notes.Add($"{prefix}{nameof(HardGrace)} {g} is outside {MinHardGrace}–{MaxHardGrace}; clamped to {clamped}.");
|
||||
grace = clamped;
|
||||
}
|
||||
|
||||
var heap = HeapLimitBytes;
|
||||
if (heap < MinHeapLimitBytes)
|
||||
{
|
||||
notes.Add($"{prefix}{nameof(HeapLimitBytes)} {heap} is below the minimum; clamped to {MinHeapLimitBytes}.");
|
||||
heap = MinHeapLimitBytes;
|
||||
}
|
||||
|
||||
var kills = QuarantineAfterKills;
|
||||
if (kills < 1 || kills > 100)
|
||||
{
|
||||
var clamped = Math.Clamp(kills, 1, 100);
|
||||
notes.Add($"{prefix}{nameof(QuarantineAfterKills)} {kills} is outside 1–100; clamped to {clamped}.");
|
||||
kills = clamped;
|
||||
}
|
||||
|
||||
var unavailable = UnavailableGrace;
|
||||
if (unavailable < TimeSpan.Zero || unavailable > MaxUnavailableGrace)
|
||||
{
|
||||
var clamped = unavailable < TimeSpan.Zero ? TimeSpan.Zero : MaxUnavailableGrace;
|
||||
notes.Add($"{prefix}{nameof(UnavailableGrace)} {unavailable} is outside 0–{MaxUnavailableGrace}; clamped to {clamped}.");
|
||||
unavailable = clamped;
|
||||
}
|
||||
|
||||
var probation = IsolationProbationRuns;
|
||||
if (probation < 1 || probation > 10_000)
|
||||
{
|
||||
var clamped = Math.Clamp(probation, 1, 10_000);
|
||||
notes.Add($"{prefix}{nameof(IsolationProbationRuns)} {probation} is outside 1–10000; clamped to {clamped}.");
|
||||
probation = clamped;
|
||||
}
|
||||
|
||||
adjustments = notes;
|
||||
return new ScriptWorkerOptions
|
||||
{
|
||||
Shards = shards,
|
||||
HardGrace = grace,
|
||||
HeapLimitBytes = heap,
|
||||
QuarantineAfterKills = kills,
|
||||
UnavailableGrace = unavailable,
|
||||
IsolationProbationRuns = probation,
|
||||
PeCacheDir = PeCacheDir,
|
||||
ExecutablePath = ExecutablePath,
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Fail-closed path validation (design §7): <see cref="PeCacheDir"/>, when set, must be absolute and must not
|
||||
/// sit inside <paramref name="contentRootPath"/>. Same rule as the family's <c>Secrets:SqlitePath</c>.
|
||||
/// </summary>
|
||||
/// <param name="contentRootPath">The host's content root.</param>
|
||||
/// <returns>The violation, or null when the options are acceptable.</returns>
|
||||
public string? ValidatePaths(string contentRootPath)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(PeCacheDir)) return null;
|
||||
const string key = ScriptingOptions.SectionName + ":Worker:" + nameof(PeCacheDir);
|
||||
if (!Path.IsPathFullyQualified(PeCacheDir))
|
||||
return $"{key} '{PeCacheDir}' must be an absolute path.";
|
||||
var dir = Path.TrimEndingDirectorySeparator(Path.GetFullPath(PeCacheDir));
|
||||
var root = Path.TrimEndingDirectorySeparator(Path.GetFullPath(contentRootPath));
|
||||
var comparison = OperatingSystem.IsWindows() ? StringComparison.OrdinalIgnoreCase : StringComparison.Ordinal;
|
||||
if (string.Equals(dir, root, comparison)
|
||||
|| dir.StartsWith(root + Path.DirectorySeparatorChar, comparison)
|
||||
|| dir.StartsWith(root + Path.AltDirectorySeparatorChar, comparison))
|
||||
return $"{key} '{PeCacheDir}' is inside the content root '{contentRootPath}'; it must live outside it.";
|
||||
return null;
|
||||
}
|
||||
}
|
||||
@@ -72,6 +72,30 @@ public sealed class ScriptingOptions
|
||||
/// </summary>
|
||||
public double StuckOrphanQuarantineFactor { get; set; } = DefaultStuckOrphanQuarantineFactor;
|
||||
|
||||
/// <summary>
|
||||
/// Where compiled scripts run (<c>Scripting:Runner</c>, lmxopcua#780). Default
|
||||
/// <see cref="ScriptRunnerMode.InProcess"/> — today's path, unchanged, and the rollback for the other modes.
|
||||
/// </summary>
|
||||
public ScriptRunnerMode Runner { get; set; } = ScriptRunnerMode.InProcess;
|
||||
|
||||
/// <summary>
|
||||
/// The seams <see cref="Runner"/> applies to when it is not <see cref="ScriptRunnerMode.InProcess"/>
|
||||
/// (<c>Scripting:Runner:Seams</c>, e.g. <c>["Calculation"]</c>). Null or empty means all three. Bound by
|
||||
/// <c>ScriptingOptionsBinding</c> from the documented key, because <c>Scripting:Runner</c> is itself a value.
|
||||
/// </summary>
|
||||
public IReadOnlyList<ScriptSeam>? RunnerSeams { get; set; }
|
||||
|
||||
/// <summary>The out-of-process worker pool's knobs (<c>Scripting:Worker</c>).</summary>
|
||||
public ScriptWorkerOptions Worker { get; set; } = new();
|
||||
|
||||
/// <summary>Whether <paramref name="seam"/> is selected for out-of-process execution by these options.</summary>
|
||||
/// <param name="seam">The seam.</param>
|
||||
/// <returns><c>true</c> when <see cref="Runner"/> is <see cref="ScriptRunnerMode.OutOfProcess"/> and the seam is in
|
||||
/// <see cref="RunnerSeams"/> (or that list is empty).</returns>
|
||||
public bool IsOutOfProcess(ScriptSeam seam)
|
||||
=> Runner == ScriptRunnerMode.OutOfProcess
|
||||
&& (RunnerSeams is not { Count: > 0 } seams || seams.Contains(seam));
|
||||
|
||||
/// <summary>Upper bound <see cref="Sanitized"/> applies to <see cref="StuckOrphanQuarantineFactor"/>.</summary>
|
||||
public const double MaxStuckOrphanQuarantineFactor = 1000;
|
||||
|
||||
@@ -158,6 +182,9 @@ public sealed class ScriptingOptions
|
||||
runTimeout = MaxRunTimeout;
|
||||
}
|
||||
|
||||
var worker = (Worker ?? new ScriptWorkerOptions()).Sanitized(out var workerNotes);
|
||||
notes.AddRange(workerNotes);
|
||||
|
||||
adjustments = notes;
|
||||
return new ScriptingOptions
|
||||
{
|
||||
@@ -165,6 +192,9 @@ public sealed class ScriptingOptions
|
||||
MaxAllocBytesPerRun = alloc,
|
||||
QuarantineAfterConsecutiveTimeouts = threshold,
|
||||
StuckOrphanQuarantineFactor = factor,
|
||||
Runner = Runner,
|
||||
RunnerSeams = RunnerSeams is null ? null : [.. RunnerSeams],
|
||||
Worker = worker,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
@@ -137,7 +137,7 @@ public sealed class CalculationDriver : IDriver, IDependencyConsumer, ISubscriba
|
||||
// _logger (ILogger<CalculationDriver>) implements the non-generic ILogger, so it is passed straight
|
||||
// through — the evaluator's host-side diagnostics (compile errors, sandbox violations, execution
|
||||
// throws) land on the SAME logger the driver itself was constructed with, not a separate one.
|
||||
_evaluator = new CalculationEvaluator(_logger, _scriptRoot, _options.RunTimeout, _options.Scripting);
|
||||
_evaluator = new CalculationEvaluator(_logger, _scriptRoot, _options.RunTimeout, _options.Scripting, _options.ScriptRunner);
|
||||
|
||||
RebuildTagTable(_options);
|
||||
}
|
||||
@@ -639,7 +639,7 @@ public sealed class CalculationDriver : IDriver, IDependencyConsumer, ISubscriba
|
||||
if (!result.Success)
|
||||
{
|
||||
var reason = result.Reason ?? "evaluator failure";
|
||||
toPublish = MarkEvaluationFailed(st, pending.TimestampUtc);
|
||||
toPublish = MarkEvaluationFailed(st, pending.TimestampUtc, result.StatusCode ?? StatusBadInternalError);
|
||||
// The script-log entry fires on the Good→Bad TRANSITION, or when the failure changes
|
||||
// (R07:P4) — not on every trigger of a tag that is already Bad for the same reason, which
|
||||
// flooded the script log at the input's cadence. The Bad publish was already once per
|
||||
@@ -725,16 +725,19 @@ public sealed class CalculationDriver : IDriver, IDependencyConsumer, ISubscriba
|
||||
/// returns before the evaluator ever runs), so there is no separate "inputs ready" condition here.</summary>
|
||||
/// <param name="st">The tag's published state.</param>
|
||||
/// <param name="timestampUtc">The trigger's timestamp.</param>
|
||||
/// <param name="badStatus">The Bad code to publish (see the body).</param>
|
||||
/// <returns>The Bad snapshot to publish on a fresh Good→Bad transition, else null.</returns>
|
||||
private static (object? Value, uint StatusCode)? MarkEvaluationFailed(TagState st, DateTime timestampUtc)
|
||||
private static (object? Value, uint StatusCode)? MarkEvaluationFailed(TagState st, DateTime timestampUtc, uint badStatus)
|
||||
{
|
||||
if (st.HasPublished && st.LastPublishedStatus == StatusBadInternalError) return null;
|
||||
// badStatus is BadInternalError for every script failure; BadResourceUnavailable when the out-of-process
|
||||
// runner could not run the script (lmxopcua#780). Once per transition, keyed on that code.
|
||||
if (st.HasPublished && st.LastPublishedStatus == badStatus) return null;
|
||||
|
||||
st.HasPublished = true;
|
||||
st.LastPublishedStatus = StatusBadInternalError;
|
||||
st.LastPublishedStatus = badStatus;
|
||||
st.LastTimestampUtc = timestampUtc;
|
||||
// Carry the last-known value (null if none) with Bad quality.
|
||||
return (st.HasValue ? st.LastValue : null, StatusBadInternalError);
|
||||
return (st.HasValue ? st.LastValue : null, badStatus);
|
||||
}
|
||||
|
||||
private void Publish(string rawPath, object? value, uint statusCode, DateTime timestampUtc)
|
||||
|
||||
+9
-3
@@ -40,14 +40,17 @@ public static class CalculationDriverFactoryExtensions
|
||||
/// <param name="scriptRoot">Optional root script logger (fans failure entries onto the <c>script-logs</c> topic).</param>
|
||||
/// <param name="scripting">The node-wide <c>Scripting</c> options every Calculation driver inherits; a key
|
||||
/// authored in a driver's own config overrides that one knob. Null ⇒ the compiled defaults.</param>
|
||||
/// <param name="scriptRunner">lmxopcua#780 — the node's out-of-process runner when the Calculation seam runs out
|
||||
/// of process; null (the default) keeps every driver on the in-process guard.</param>
|
||||
public static void Register(
|
||||
DriverFactoryRegistry registry,
|
||||
ILoggerFactory? loggerFactory = null,
|
||||
ScriptRootLogger? scriptRoot = null,
|
||||
ScriptingOptions? scripting = null)
|
||||
ScriptingOptions? scripting = null,
|
||||
IScriptRunner? scriptRunner = null)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(registry);
|
||||
registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory, scriptRoot, scripting));
|
||||
registry.Register(DriverTypeName, (id, json) => CreateInstance(id, json, loggerFactory, scriptRoot, scripting, scriptRunner));
|
||||
}
|
||||
|
||||
/// <summary>Public overload for the Server-side bootstrapper + test consumers.</summary>
|
||||
@@ -74,10 +77,12 @@ public static class CalculationDriverFactoryExtensions
|
||||
/// <param name="loggerFactory">Optional logger factory for per-instance diagnostics loggers.</param>
|
||||
/// <param name="scriptRoot">Optional root script logger for failure fan-out.</param>
|
||||
/// <param name="nodeScripting">The node-wide run-guard options; null ⇒ the compiled defaults.</param>
|
||||
/// <param name="scriptRunner">The node's out-of-process runner, or null for in process.</param>
|
||||
/// <returns>The constructed <see cref="CalculationDriver"/>.</returns>
|
||||
public static CalculationDriver CreateInstance(
|
||||
string driverInstanceId, string driverConfigJson,
|
||||
ILoggerFactory? loggerFactory, ScriptRootLogger? scriptRoot, ScriptingOptions? nodeScripting)
|
||||
ILoggerFactory? loggerFactory, ScriptRootLogger? scriptRoot, ScriptingOptions? nodeScripting,
|
||||
IScriptRunner? scriptRunner = null)
|
||||
{
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(driverInstanceId);
|
||||
ArgumentException.ThrowIfNullOrWhiteSpace(driverConfigJson);
|
||||
@@ -101,6 +106,7 @@ public static class CalculationDriverFactoryExtensions
|
||||
RunTimeout = dto.RunTimeoutMs is { } ms && ms > 0
|
||||
? TimeSpan.FromMilliseconds(ms) : TimeSpan.FromSeconds(2),
|
||||
Scripting = scripting,
|
||||
ScriptRunner = scriptRunner,
|
||||
};
|
||||
|
||||
return new CalculationDriver(options, driverInstanceId, scriptRoot, logger: logger);
|
||||
|
||||
@@ -18,4 +18,11 @@ public sealed class CalculationDriverOptions
|
||||
/// <see cref="ScriptingOptions"/> defaults (64 MB, 3).
|
||||
/// </summary>
|
||||
public ScriptingOptions? Scripting { get; init; }
|
||||
|
||||
/// <summary>
|
||||
/// lmxopcua#780 — the node's out-of-process runner when <c>Scripting:Runner=OutOfProcess</c> selects the
|
||||
/// Calculation seam; null (the default) keeps the in-process run guard. Shared by every Calculation driver on the
|
||||
/// node; keys are the calc tags' RawPaths.
|
||||
/// </summary>
|
||||
public IScriptRunner? ScriptRunner { get; init; }
|
||||
}
|
||||
|
||||
@@ -59,6 +59,7 @@ public sealed class CalculationEvaluator : IScriptCacheOwner, IScriptQuarantineO
|
||||
private readonly SerilogLogger _scriptRoot;
|
||||
private readonly TimeSpan _runTimeout;
|
||||
private readonly ScriptRunGuard _guard;
|
||||
private readonly IScriptRunner? _runner;
|
||||
|
||||
// Negative compile cache (R07:P4), keyed by source hash. A compile error is a property of the SOURCE,
|
||||
// which never changes between triggers (it is fixed in the deploy artifact) — yet every trigger of a
|
||||
@@ -76,12 +77,16 @@ public sealed class CalculationEvaluator : IScriptCacheOwner, IScriptQuarantineO
|
||||
/// <param name="scriptRoot">Root script logger; user <c>ctx.Logger.*</c> output flows through this to the Script-log page.</param>
|
||||
/// <param name="runTimeout">Wall-clock budget per script run; defaults to 2 seconds (parity with the VT evaluator).</param>
|
||||
/// <param name="scripting">Run-guard budget + quarantine threshold; null ⇒ <see cref="ScriptingOptions"/> defaults.</param>
|
||||
/// <param name="scriptRunner">lmxopcua#780 — the out-of-process runner when the Calculation seam runs out of
|
||||
/// process; null (the default) keeps the in-process guard path, unchanged.</param>
|
||||
public CalculationEvaluator(
|
||||
ILogger logger,
|
||||
ScriptRootLogger scriptRoot,
|
||||
TimeSpan? runTimeout = null,
|
||||
ScriptingOptions? scripting = null)
|
||||
ScriptingOptions? scripting = null,
|
||||
IScriptRunner? scriptRunner = null)
|
||||
{
|
||||
_runner = scriptRunner;
|
||||
_logger = logger ?? throw new ArgumentNullException(nameof(logger));
|
||||
_scriptRoot = (scriptRoot ?? throw new ArgumentNullException(nameof(scriptRoot))).Logger;
|
||||
_runTimeout = runTimeout ?? TimeSpan.FromSeconds(2);
|
||||
@@ -188,6 +193,15 @@ public sealed class CalculationEvaluator : IScriptCacheOwner, IScriptQuarantineO
|
||||
calcTagId, path),
|
||||
logger: scriptLog);
|
||||
|
||||
// lmxopcua#780 — OutOfProcess: the pre-emptible worker pool compiles and runs it, and owns timeouts, kills,
|
||||
// quarantine and the negative compile cache for this seam. Unavailable publishes BadResourceUnavailable.
|
||||
if (_runner is not null)
|
||||
{
|
||||
var outcome = await _runner.RunAsync(new ScriptInvocation(
|
||||
calcTagId, ScriptContextKind.Calculation, scriptSource, readCache, _runTimeout, scriptLog), ct).ConfigureAwait(false);
|
||||
return outcome.ToVirtualTagEvalResult();
|
||||
}
|
||||
|
||||
// A source that already failed to compile fails the same way without touching Roslyn (R07:P4).
|
||||
var sourceKey = HashSource(scriptSource);
|
||||
if (_compileFailures.TryGetValue(sourceKey, out var knownFailure))
|
||||
@@ -295,13 +309,20 @@ public sealed class CalculationEvaluator : IScriptCacheOwner, IScriptQuarantineO
|
||||
// The compile-cache clear is this evaluator's deploy boundary — a redeploy lifts every quarantine
|
||||
// (an edited source already lifts its own through the guard's source identity).
|
||||
_guard.ResetAll();
|
||||
_runner?.Reset(ScriptContextKind.Calculation);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void ResetQuarantines() => _guard.ResetAll();
|
||||
public void ResetQuarantines()
|
||||
{
|
||||
_guard.ResetAll();
|
||||
_runner?.Reset(ScriptContextKind.Calculation);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public IReadOnlyCollection<string> QuarantinedScripts => _guard.QuarantinedKeys;
|
||||
public IReadOnlyCollection<string> QuarantinedScripts => _runner is null
|
||||
? _guard.QuarantinedKeys
|
||||
: [.. _guard.QuarantinedKeys, .. _runner.QuarantinedKeys(ScriptContextKind.Calculation)];
|
||||
|
||||
/// <summary>Disposes the evaluator, draining + disposing every cached compile's collectible ALC.</summary>
|
||||
public void Dispose()
|
||||
|
||||
+2
@@ -30,6 +30,8 @@
|
||||
|
||||
<ItemGroup>
|
||||
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.Driver.Calculation.Tests"/>
|
||||
<!-- lmxopcua#780: the worker suite drives a real driver over the real ScriptWorker (IsEvaluationIdle). -->
|
||||
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests"/>
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
|
||||
using SerilogLogger = Serilog.ILogger;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
||||
|
||||
/// <summary>
|
||||
/// Which <see cref="IScriptRunner"/> each script seam uses on this node (lmxopcua#780, <c>Scripting:Runner</c>). Under
|
||||
/// the <see cref="ScriptRunnerMode.InProcess"/> default every seam gets null and keeps its in-process guard path,
|
||||
/// unchanged. Under <see cref="ScriptRunnerMode.OutOfProcess"/> the selected seams share one
|
||||
/// <see cref="OutOfProcessScriptRunner"/>, which starts no worker until the first compiled evaluation, so a node with
|
||||
/// no scripts (or an admin-only node) never launches one. Disposed with the host's service provider, which stops the
|
||||
/// workers.
|
||||
/// </summary>
|
||||
public sealed class ScriptRunnerSelection : IAsyncDisposable, IDisposable
|
||||
{
|
||||
/// <summary>Seams the out-of-process runner can serve in this build. Scripted alarms go last (design §14 #10).</summary>
|
||||
public static readonly IReadOnlyList<ScriptSeam> SupportedOutOfProcessSeams = [ScriptSeam.VirtualTag, ScriptSeam.Calculation];
|
||||
|
||||
private readonly OutOfProcessScriptRunner? _runner;
|
||||
private readonly HashSet<ScriptSeam> _seams;
|
||||
|
||||
private ScriptRunnerSelection(OutOfProcessScriptRunner? runner, IEnumerable<ScriptSeam> seams)
|
||||
{
|
||||
_runner = runner;
|
||||
_seams = [.. seams];
|
||||
}
|
||||
|
||||
/// <summary>The out-of-process runner, when one is configured.</summary>
|
||||
public IScriptRunner? OutOfProcessRunner => _runner;
|
||||
|
||||
/// <summary>The runner <paramref name="seam"/> must use, or null for its in-process path.</summary>
|
||||
/// <param name="seam">The seam.</param>
|
||||
/// <returns>The runner, or null.</returns>
|
||||
public IScriptRunner? ForSeam(ScriptSeam seam) => _runner is not null && _seams.Contains(seam) ? _runner : null;
|
||||
|
||||
/// <summary>
|
||||
/// Startup checks for the runner options: returns the Warnings to log, and throws when the configuration must
|
||||
/// not start (a <c>PeCacheDir</c> that is relative or inside the content root — fail closed).
|
||||
/// </summary>
|
||||
/// <param name="options">The bound, sanitized <c>Scripting</c> options.</param>
|
||||
/// <param name="contentRootPath">The host's content root.</param>
|
||||
/// <returns>Warnings for the caller to log.</returns>
|
||||
/// <exception cref="InvalidOperationException">The worker path options are unacceptable.</exception>
|
||||
public static IReadOnlyList<string> Validate(ScriptingOptions options, string contentRootPath)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
var warnings = new List<string>();
|
||||
if (options.Runner == ScriptRunnerMode.InProcess) return warnings;
|
||||
|
||||
if (options.Worker.ValidatePaths(contentRootPath) is { } violation)
|
||||
throw new InvalidOperationException($"Scripting misconfiguration: {violation}");
|
||||
|
||||
if (options.Runner == ScriptRunnerMode.Shadow)
|
||||
{
|
||||
warnings.Add("Scripting:Runner=Shadow is not implemented yet (lmxopcua#780 P2); every seam runs InProcess.");
|
||||
return warnings;
|
||||
}
|
||||
|
||||
foreach (var seam in RequestedSeams(options))
|
||||
{
|
||||
if (!SupportedOutOfProcessSeams.Contains(seam))
|
||||
warnings.Add($"Scripting:Runner=OutOfProcess: the {seam} seam cannot run out of process yet (lmxopcua#780); it stays InProcess.");
|
||||
}
|
||||
var workerPath = OutOfProcessScriptRunner.ResolveWorkerPath(options.Worker);
|
||||
if (!File.Exists(workerPath))
|
||||
warnings.Add($"Scripting:Runner=OutOfProcess: the script worker '{workerPath}' does not exist; every out-of-process " +
|
||||
"script reports BadResourceUnavailable (the runner fails closed, it never falls back to in process).");
|
||||
return warnings;
|
||||
}
|
||||
|
||||
/// <summary>Builds the selection for <paramref name="options"/>.</summary>
|
||||
/// <param name="options">The bound, sanitized <c>Scripting</c> options.</param>
|
||||
/// <param name="log">Host diagnostics for the runner.</param>
|
||||
/// <param name="scriptRoot">The root script logger forwarded <c>ctx.Logger</c> lines fall back to.</param>
|
||||
/// <returns>The selection.</returns>
|
||||
public static ScriptRunnerSelection Create(ScriptingOptions options, SerilogLogger log, SerilogLogger? scriptRoot)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
if (options.Runner != ScriptRunnerMode.OutOfProcess) return new ScriptRunnerSelection(null, []);
|
||||
var seams = RequestedSeams(options).Where(SupportedOutOfProcessSeams.Contains).ToArray();
|
||||
if (seams.Length == 0) return new ScriptRunnerSelection(null, []);
|
||||
return new ScriptRunnerSelection(new OutOfProcessScriptRunner(options, log, scriptRoot), seams);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public ValueTask DisposeAsync() => _runner?.DisposeAsync() ?? ValueTask.CompletedTask;
|
||||
|
||||
/// <inheritdoc />
|
||||
public void Dispose() => _runner?.Dispose();
|
||||
|
||||
private static IEnumerable<ScriptSeam> RequestedSeams(ScriptingOptions options)
|
||||
=> options.RunnerSeams is { Count: > 0 } seams ? seams.Distinct() : Enum.GetValues<ScriptSeam>();
|
||||
}
|
||||
@@ -18,6 +18,11 @@ public static class ScriptingOptionsBinding
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(configuration);
|
||||
var bound = configuration.GetSection(ScriptingOptions.SectionName).Get<ScriptingOptions>() ?? new ScriptingOptions();
|
||||
// lmxopcua#780: the seam subset lives under the runner key (Scripting:Runner:Seams), which the binder cannot
|
||||
// reach because Scripting:Runner is itself a value. Bound explicitly; an unknown seam name is a startup error,
|
||||
// like an unknown Scripting:Runner value.
|
||||
var seams = configuration.GetSection($"{ScriptingOptions.SectionName}:Runner:Seams").Get<List<ScriptSeam>>();
|
||||
if (seams is { Count: > 0 }) bound.RunnerSeams = seams;
|
||||
return bound.Sanitized(out adjustments);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -102,7 +102,10 @@ public static class DriverFactoryBootstrap
|
||||
// failures; without this it writes to NullLogger in production.
|
||||
ConfigureAbTransportLogging(sp);
|
||||
Register(registry, loggerFactory, secretResolver, primaryRoleSource, ResolveAbCipDefaultTransport(sp),
|
||||
sp.GetRequiredService<Driver.Galaxy.Runtime.IGalaxyAckIdentityDirectory>(), scriptRoot, scripting);
|
||||
sp.GetRequiredService<Driver.Galaxy.Runtime.IGalaxyAckIdentityDirectory>(), scriptRoot, scripting,
|
||||
// lmxopcua#780: null under the InProcess default (and in test graphs that register no selection).
|
||||
sp.GetService<ZB.MOM.WW.OtOpcUa.Host.Configuration.ScriptRunnerSelection>()
|
||||
?.ForSeam(ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptSeam.Calculation));
|
||||
return registry;
|
||||
});
|
||||
services.AddSingleton<IDriverFactory>(sp =>
|
||||
@@ -233,11 +236,12 @@ public static class DriverFactoryBootstrap
|
||||
Driver.AbCip.AbCipTransport abCipDefaultTransport,
|
||||
Driver.Galaxy.Runtime.IGalaxyAckIdentityDirectory galaxyAckIdentities,
|
||||
ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptRootLogger? scriptRoot = null,
|
||||
ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptingOptions? scripting = null)
|
||||
ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptingOptions? scripting = null,
|
||||
ZB.MOM.WW.OtOpcUa.Core.Scripting.IScriptRunner? scriptRunner = null)
|
||||
{
|
||||
Driver.AbCip.AbCipDriverFactoryExtensions.Register(registry, loggerFactory, abCipDefaultTransport);
|
||||
Driver.AbLegacy.AbLegacyDriverFactoryExtensions.Register(registry, loggerFactory);
|
||||
Driver.Calculation.CalculationDriverFactoryExtensions.Register(registry, loggerFactory, scriptRoot, scripting);
|
||||
Driver.Calculation.CalculationDriverFactoryExtensions.Register(registry, loggerFactory, scriptRoot, scripting, scriptRunner);
|
||||
Driver.FOCAS.FocasDriverFactoryExtensions.Register(registry, loggerFactory);
|
||||
Driver.Galaxy.GalaxyDriverFactoryExtensions.Register(registry, secretResolver, loggerFactory, galaxyAckIdentities);
|
||||
Driver.Modbus.ModbusDriverFactoryExtensions.Register(registry, loggerFactory);
|
||||
|
||||
@@ -53,6 +53,7 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac
|
||||
private readonly SerilogLogger _scriptRoot;
|
||||
private readonly TimeSpan _runTimeout;
|
||||
private readonly ScriptRunGuard _guard;
|
||||
private readonly IScriptRunner? _runner;
|
||||
private bool _disposed;
|
||||
|
||||
/// <summary>Initializes a new RoslynVirtualTagEvaluator with the given loggers and optional timeout.</summary>
|
||||
@@ -61,18 +62,28 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac
|
||||
/// <param name="runTimeout">Maximum execution time for each script; null ⇒ <see cref="ScriptingOptions.RunTimeout"/>
|
||||
/// of <paramref name="scripting"/> (2 s by default).</param>
|
||||
/// <param name="scripting">Run-guard budget + quarantine threshold (the <c>Scripting</c> section); null ⇒ defaults.</param>
|
||||
/// <param name="scriptRunner">
|
||||
/// lmxopcua#780 — the out-of-process runner when <c>Scripting:Runner=OutOfProcess</c> selects the VirtualTag seam;
|
||||
/// null (the default, <c>InProcess</c>) keeps the in-process guard path below, unchanged. The passthrough fast path
|
||||
/// never reaches it.
|
||||
/// </param>
|
||||
public RoslynVirtualTagEvaluator(
|
||||
ILogger<RoslynVirtualTagEvaluator> logger,
|
||||
ScriptRootLogger scriptRoot,
|
||||
TimeSpan? runTimeout = null,
|
||||
ScriptingOptions? scripting = null)
|
||||
ScriptingOptions? scripting = null,
|
||||
IScriptRunner? scriptRunner = null)
|
||||
{
|
||||
_logger = logger;
|
||||
_scriptRoot = (scriptRoot ?? throw new ArgumentNullException(nameof(scriptRoot))).Logger;
|
||||
_runTimeout = runTimeout ?? (scripting ?? new ScriptingOptions()).Sanitized(out _).RunTimeout;
|
||||
_guard = new ScriptRunGuard(scripting, _scriptRoot);
|
||||
_runner = scriptRunner;
|
||||
}
|
||||
|
||||
/// <summary>The out-of-process runner this evaluator delegates compiled runs to, or null (in process).</summary>
|
||||
public IScriptRunner? ScriptRunner => _runner;
|
||||
|
||||
/// <summary>The run guard every compiled evaluation goes through (diagnostics / tests).</summary>
|
||||
public ScriptRunGuard RunGuard => _guard;
|
||||
|
||||
@@ -147,6 +158,16 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac
|
||||
virtualTagId, path),
|
||||
logger: scriptLog);
|
||||
|
||||
// lmxopcua#780 — OutOfProcess: the compiled run goes to the pre-emptible worker pool, which compiles in the
|
||||
// worker and owns timeouts, kills and quarantine for this seam. An Unavailable outcome (worker restarted, no
|
||||
// worker ready) publishes BadResourceUnavailable; everything else keeps today's Bad contract.
|
||||
if (_runner is not null)
|
||||
{
|
||||
var outcome = await _runner.RunAsync(new ScriptInvocation(
|
||||
virtualTagId, ScriptContextKind.VirtualTag, expression, readCache, _runTimeout, scriptLog)).ConfigureAwait(false);
|
||||
return outcome.ToVirtualTagEvalResult();
|
||||
}
|
||||
|
||||
// 02/S12: fetch + run inside a bounded loop so an ObjectDisposedException caused by a concurrent
|
||||
// apply-boundary ClearCompiledScripts (which disposed the evaluator between GetOrCompile and the
|
||||
// run's disposed guard) re-fetches and retries exactly once. The loop runs at most twice — the
|
||||
@@ -235,13 +256,20 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac
|
||||
// The compile-cache clear IS the deploy boundary for this evaluator: a redeploy lifts every quarantine
|
||||
// (an edited expression already lifts its own through the guard's source identity).
|
||||
_guard.ResetAll();
|
||||
_runner?.Reset(ScriptContextKind.VirtualTag);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public void ResetQuarantines() => _guard.ResetAll();
|
||||
public void ResetQuarantines()
|
||||
{
|
||||
_guard.ResetAll();
|
||||
_runner?.Reset(ScriptContextKind.VirtualTag);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public IReadOnlyCollection<string> QuarantinedScripts => _guard.QuarantinedKeys;
|
||||
public IReadOnlyCollection<string> QuarantinedScripts => _runner is null
|
||||
? _guard.QuarantinedKeys
|
||||
: [.. _guard.QuarantinedKeys, .. _runner.QuarantinedKeys(ScriptContextKind.VirtualTag)];
|
||||
|
||||
/// <summary>Disposes the evaluator and cleans up all cached script evaluators.</summary>
|
||||
public void Dispose()
|
||||
|
||||
@@ -376,6 +376,16 @@ if (hasDriver)
|
||||
Log.Warning("Scripting misconfiguration detected at startup: {Adjustment}", adjustment);
|
||||
builder.Services.AddSingleton(scriptingOptions);
|
||||
|
||||
// lmxopcua#780: Scripting:Runner (InProcess default / Shadow / OutOfProcess). Under the default every seam keeps its
|
||||
// in-process guard path unchanged; OutOfProcess hands the selected seams' compiled runs to the pre-emptible
|
||||
// ScriptWorker pool. A PeCacheDir that is relative or inside the content root stops startup (fail closed).
|
||||
foreach (var warning in ZB.MOM.WW.OtOpcUa.Host.Configuration.ScriptRunnerSelection.Validate(
|
||||
scriptingOptions, builder.Environment.ContentRootPath))
|
||||
Log.Warning("Scripting runner: {RunnerWarning}", warning);
|
||||
Log.Information("Scripting runner: {RunnerMode}", scriptingOptions.Runner);
|
||||
builder.Services.AddSingleton(sp => ZB.MOM.WW.OtOpcUa.Host.Configuration.ScriptRunnerSelection.Create(
|
||||
sp.GetRequiredService<ScriptingOptions>(), Log.Logger, sp.GetService<ScriptRootLogger>()?.Logger));
|
||||
|
||||
// lmxopcua#777: the ONE operator-write timeout (DriverWrites:Timeout) every write bound is derived from — the
|
||||
// driver child's cancel + abandon, its queue budget, DriverHostActor's Ask and the inbound-write gateway's Ask.
|
||||
// Checked against the resilience tiers' Write timeouts here; an inconsistency is a startup Warning (a tier
|
||||
@@ -392,7 +402,9 @@ if (hasDriver)
|
||||
new RoslynVirtualTagEvaluator(
|
||||
sp.GetRequiredService<ILoggerFactory>().CreateLogger<RoslynVirtualTagEvaluator>(),
|
||||
sp.GetRequiredService<ScriptRootLogger>(),
|
||||
scripting: sp.GetRequiredService<ScriptingOptions>()));
|
||||
scripting: sp.GetRequiredService<ScriptingOptions>(),
|
||||
scriptRunner: sp.GetRequiredService<ZB.MOM.WW.OtOpcUa.Host.Configuration.ScriptRunnerSelection>()
|
||||
.ForSeam(ScriptSeam.VirtualTag)));
|
||||
builder.Services.AddSingleton<IVirtualTagEvaluator>(sp => sp.GetRequiredService<RoslynVirtualTagEvaluator>());
|
||||
|
||||
// Script-log fan-out (Layer 0). The DPS publisher resolves the ActorSystem lazily so it never
|
||||
|
||||
@@ -57,6 +57,7 @@
|
||||
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Configuration\ZB.MOM.WW.OtOpcUa.Configuration.csproj"/>
|
||||
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core\ZB.MOM.WW.OtOpcUa.Core.csproj"/>
|
||||
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.Scripting\ZB.MOM.WW.OtOpcUa.Core.Scripting.csproj"/>
|
||||
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess\ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess.csproj"/>
|
||||
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.VirtualTags\ZB.MOM.WW.OtOpcUa.Core.VirtualTags.csproj"/>
|
||||
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms\ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms.csproj"/>
|
||||
<ProjectReference Include="..\ZB.MOM.WW.OtOpcUa.Security\ZB.MOM.WW.OtOpcUa.Security.csproj"/>
|
||||
@@ -90,4 +91,7 @@
|
||||
<ProjectReference Include="..\..\Drivers\ZB.MOM.WW.OtOpcUa.Driver.TwinCAT\ZB.MOM.WW.OtOpcUa.Driver.TwinCAT.csproj"/>
|
||||
</ItemGroup>
|
||||
|
||||
<!-- lmxopcua#780: the out-of-process script runner's worker ships in script-worker/ beside the host. -->
|
||||
<Import Project="..\ZB.MOM.WW.OtOpcUa.ScriptWorker\build\ScriptWorker.Deploy.targets"/>
|
||||
|
||||
</Project>
|
||||
|
||||
@@ -391,7 +391,8 @@ public sealed class VirtualTagActor : ReceiveActor, IWithUnboundedStash
|
||||
var result = evaluation.Result;
|
||||
if (!result.Success)
|
||||
{
|
||||
OnEvaluationFailed(result.Reason ?? "evaluator failure", "Warning", timestampUtc);
|
||||
OnEvaluationFailed(result.Reason ?? "evaluator failure", "Warning", timestampUtc,
|
||||
result.StatusCode ?? OpcUaStatus.BadInternalError);
|
||||
return;
|
||||
}
|
||||
|
||||
@@ -472,7 +473,10 @@ public sealed class VirtualTagActor : ReceiveActor, IWithUnboundedStash
|
||||
/// <param name="reason">The failure reason for the script-log entry.</param>
|
||||
/// <param name="level">The script-log level ("Warning" for a Failure result, "Error" for a throw).</param>
|
||||
/// <param name="timestampUtc">The triggering dependency change's timestamp, stamped on the Bad result.</param>
|
||||
private void OnEvaluationFailed(string reason, string level, DateTime timestampUtc)
|
||||
/// <param name="badStatus">The Bad code to publish: <see cref="OpcUaStatus.BadInternalError"/> for every script
|
||||
/// failure, <see cref="OpcUaStatus.BadResourceUnavailable"/> when the out-of-process runner could not run the
|
||||
/// script (lmxopcua#780). The once-per-transition dedup is on this code, so a switch between the two publishes.</param>
|
||||
private void OnEvaluationFailed(string reason, string level, DateTime timestampUtc, uint badStatus = OpcUaStatus.BadInternalError)
|
||||
{
|
||||
OtOpcUaTelemetry.VirtualTagEval.Add(1, new KeyValuePair<string, object?>("outcome", "fail"));
|
||||
|
||||
@@ -488,15 +492,15 @@ public sealed class VirtualTagActor : ReceiveActor, IWithUnboundedStash
|
||||
}
|
||||
if (admission.Admitted) PublishLog(level, reason);
|
||||
|
||||
if ((_hasPublished && _lastPublishedStatus == OpcUaStatus.BadInternalError) || !AllDependenciesArrived())
|
||||
if ((_hasPublished && _lastPublishedStatus == badStatus) || !AllDependenciesArrived())
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
_hasPublished = true;
|
||||
_lastPublishedStatus = OpcUaStatus.BadInternalError;
|
||||
_lastPublishedStatus = badStatus;
|
||||
Context.Parent.Tell(new EvaluationResult(
|
||||
_virtualTagId, _hasLastValue ? _lastValue : null, timestampUtc, CorrelationId.NewId(), OpcUaStatus.BadInternalError));
|
||||
_virtualTagId, _hasLastValue ? _lastValue : null, timestampUtc, CorrelationId.NewId(), badStatus));
|
||||
}
|
||||
|
||||
/// <summary>Inputs-ready gate (the retired cascade engine's rule): true once
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
using System.IO.Pipes;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker;
|
||||
|
||||
/// <summary>Entry point: connect to the host's pipe named in the environment and serve until told to stop.</summary>
|
||||
internal static class Program
|
||||
{
|
||||
private static async Task<int> Main()
|
||||
{
|
||||
var pipeName = Environment.GetEnvironmentVariable(ScriptWorkerEnvironment.PipeName);
|
||||
var nonce = Environment.GetEnvironmentVariable(ScriptWorkerEnvironment.Nonce);
|
||||
if (string.IsNullOrEmpty(pipeName) || string.IsNullOrEmpty(nonce))
|
||||
{
|
||||
await Console.Error.WriteLineAsync("ScriptWorker: launched without a pipe name or nonce; exiting.");
|
||||
return ScriptWorkerEnvironment.ExitProtocolFault;
|
||||
}
|
||||
_ = int.TryParse(Environment.GetEnvironmentVariable(ScriptWorkerEnvironment.ParentPid), out var parentPid);
|
||||
|
||||
// CurrentUserOnly: the client verifies the server end is owned by this account as well.
|
||||
var pipe = new NamedPipeClientStream(".", pipeName, PipeDirection.InOut,
|
||||
PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly);
|
||||
try
|
||||
{
|
||||
await pipe.ConnectAsync(TimeSpan.FromSeconds(10), CancellationToken.None);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
await Console.Error.WriteLineAsync($"ScriptWorker: could not connect to the host pipe: {ex.Message}");
|
||||
return ScriptWorkerEnvironment.ExitProtocolFault;
|
||||
}
|
||||
|
||||
var host = new WorkerHost(pipe, nonce, parentPid);
|
||||
return await host.RunAsync();
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,576 @@
|
||||
using System.Collections;
|
||||
using System.Collections.Concurrent;
|
||||
using System.Diagnostics;
|
||||
using System.IO.Pipes;
|
||||
using System.Threading.Channels;
|
||||
using Serilog;
|
||||
using Serilog.Core;
|
||||
using Serilog.Events;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.VirtualTags;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker;
|
||||
|
||||
/// <summary>
|
||||
/// One worker's whole life (lmxopcua#780, design §3.2): handshake, compile-on-load, a bounded evaluation scheduler
|
||||
/// on dedicated threads, a hard-deadline watchdog, heartbeats, and log forwarding. The host is authoritative for
|
||||
/// deadlines (it kills this process); the watchdog is the belt to that brace and fails fast slightly later, so the
|
||||
/// exit names the key that overran.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// Anything that means "the host is gone" — pipe end of stream, a protocol fault, the parent pid disappearing —
|
||||
/// ends the process. The worker never outlives its host (design §3.3).
|
||||
/// </remarks>
|
||||
internal sealed class WorkerHost
|
||||
{
|
||||
/// <summary>The watchdog fails fast this long after the host's own hard deadline, so the host normally wins.</summary>
|
||||
private static readonly TimeSpan WatchdogMargin = TimeSpan.FromMilliseconds(500);
|
||||
|
||||
private readonly NamedPipeClientStream _pipe;
|
||||
private readonly string _nonce;
|
||||
private readonly int _parentPid;
|
||||
private readonly Channel<Envelope> _outbound = Channel.CreateUnbounded<Envelope>(
|
||||
new UnboundedChannelOptions { SingleReader = true });
|
||||
private readonly ConcurrentDictionary<string, LoadedScript> _scripts = new(StringComparer.Ordinal);
|
||||
private readonly ConcurrentDictionary<long, RunningEval> _running = new();
|
||||
private readonly BlockingCollection<EvalRequest> _queue = new(new ConcurrentQueue<EvalRequest>());
|
||||
private readonly ForwardingLogSink _logSink;
|
||||
private readonly ILogger _scriptLogger;
|
||||
private int _maxFrameBytes = ScriptWorkerFrameCodec.DefaultMaxFrameBytes;
|
||||
private int _queueCapacity = 4096;
|
||||
private TimeSpan _hardGrace = TimeSpan.FromSeconds(1);
|
||||
private TimeSpan _heartbeatInterval = TimeSpan.FromSeconds(1);
|
||||
private long _seq;
|
||||
|
||||
public WorkerHost(NamedPipeClientStream pipe, string nonce, int parentPid)
|
||||
{
|
||||
_pipe = pipe;
|
||||
_nonce = nonce;
|
||||
_parentPid = parentPid;
|
||||
_logSink = new ForwardingLogSink();
|
||||
_scriptLogger = new LoggerConfiguration().MinimumLevel.Verbose().WriteTo.Sink(_logSink).CreateLogger();
|
||||
}
|
||||
|
||||
public async Task<int> RunAsync()
|
||||
{
|
||||
try
|
||||
{
|
||||
var hello = await ScriptWorkerFrameCodec.ReadAsync(_pipe, ScriptWorkerFrameCodec.DefaultMaxFrameBytes);
|
||||
if (hello is null) return ScriptWorkerEnvironment.ExitPipeClosed;
|
||||
if (hello.BodyCase != Envelope.BodyOneofCase.Hello)
|
||||
throw new ScriptWorkerProtocolException($"expected Hello, got {hello.BodyCase}");
|
||||
Configure(hello.Hello);
|
||||
|
||||
// The nonce is echoed, not trusted: the host compares it with the one it put in OUR environment, which
|
||||
// proves the process on the other end of its pipe is the one it launched.
|
||||
await ScriptWorkerFrameCodec.WriteAsync(_pipe, Wrap(new Envelope
|
||||
{
|
||||
Ready = new Ready
|
||||
{
|
||||
Nonce = _nonce,
|
||||
Pid = Environment.ProcessId,
|
||||
RuntimeVersion = Environment.Version.ToString(),
|
||||
},
|
||||
}), _maxFrameBytes);
|
||||
|
||||
_logSink.Capacity = (int)Math.Max(1, hello.Hello.MaxLogLines == 0 ? 1000 : hello.Hello.MaxLogLines);
|
||||
StartBackground(Math.Max(1, (int)hello.Hello.EvalThreads));
|
||||
var writer = Task.Run(WriteLoopAsync);
|
||||
var exit = await ReadLoopAsync();
|
||||
_outbound.Writer.TryComplete();
|
||||
try { await writer.WaitAsync(TimeSpan.FromSeconds(1)); } catch { /* exiting regardless */ }
|
||||
return exit;
|
||||
}
|
||||
catch (ScriptWorkerProtocolException ex)
|
||||
{
|
||||
await Console.Error.WriteLineAsync($"ScriptWorker: protocol fault: {ex.Message}");
|
||||
return ScriptWorkerEnvironment.ExitProtocolFault;
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
return ScriptWorkerEnvironment.ExitPipeClosed;
|
||||
}
|
||||
}
|
||||
|
||||
private void Configure(Hello hello)
|
||||
{
|
||||
if (hello.MaxFrameBytes > 0) _maxFrameBytes = (int)Math.Min(hello.MaxFrameBytes, 64u * 1024 * 1024);
|
||||
if (hello.QueueCapacity > 0) _queueCapacity = (int)hello.QueueCapacity;
|
||||
if (hello.HardGraceMs > 0) _hardGrace = TimeSpan.FromMilliseconds(hello.HardGraceMs);
|
||||
if (hello.HeartbeatIntervalMs > 0) _heartbeatInterval = TimeSpan.FromMilliseconds(hello.HeartbeatIntervalMs);
|
||||
}
|
||||
|
||||
private void StartBackground(int evalThreads)
|
||||
{
|
||||
for (var i = 0; i < evalThreads; i++)
|
||||
{
|
||||
new Thread(EvalThreadLoop) { IsBackground = true, Name = $"ScriptWorker.Eval{i}" }.Start();
|
||||
}
|
||||
new Thread(WatchdogLoop) { IsBackground = true, Name = "ScriptWorker.Watchdog" }.Start();
|
||||
_ = Task.Run(HeartbeatLoopAsync);
|
||||
_ = Task.Run(LogFlushLoopAsync);
|
||||
if (_parentPid > 0) _ = Task.Run(ParentWatchLoopAsync);
|
||||
}
|
||||
|
||||
private async Task<int> ReadLoopAsync()
|
||||
{
|
||||
while (true)
|
||||
{
|
||||
Envelope? frame;
|
||||
try
|
||||
{
|
||||
frame = await ScriptWorkerFrameCodec.ReadAsync(_pipe, _maxFrameBytes);
|
||||
}
|
||||
catch (ScriptWorkerProtocolException ex)
|
||||
{
|
||||
await Console.Error.WriteLineAsync($"ScriptWorker: protocol fault: {ex.Message}");
|
||||
return ScriptWorkerEnvironment.ExitProtocolFault;
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
return ScriptWorkerEnvironment.ExitPipeClosed;
|
||||
}
|
||||
if (frame is null) return ScriptWorkerEnvironment.ExitPipeClosed; // host gone: fatal by design
|
||||
|
||||
switch (frame.BodyCase)
|
||||
{
|
||||
case Envelope.BodyOneofCase.Load:
|
||||
{
|
||||
// Compile off the reader, so evaluations for already-loaded keys keep flowing during a load.
|
||||
var load = frame.Load;
|
||||
_ = Task.Run(() => Load(load));
|
||||
break;
|
||||
}
|
||||
case Envelope.BodyOneofCase.Unload:
|
||||
foreach (var key in frame.Unload.Keys)
|
||||
if (_scripts.TryRemove(key, out var gone)) gone.Dispose();
|
||||
break;
|
||||
case Envelope.BodyOneofCase.Eval:
|
||||
foreach (var item in frame.Eval.Items) Admit(item);
|
||||
break;
|
||||
case Envelope.BodyOneofCase.Cancel:
|
||||
if (_running.TryGetValue((long)frame.Cancel.EvalId, out var toCancel))
|
||||
{
|
||||
try { toCancel.Cancellation.Cancel(); } catch (ObjectDisposedException) { /* just finished */ }
|
||||
}
|
||||
break;
|
||||
case Envelope.BodyOneofCase.DiagRequest:
|
||||
Send(new Envelope { DiagReply = Diagnostics() });
|
||||
break;
|
||||
case Envelope.BodyOneofCase.Shutdown:
|
||||
return 0;
|
||||
case Envelope.BodyOneofCase.Fault:
|
||||
await Console.Error.WriteLineAsync($"ScriptWorker: host fault: {frame.Fault.Message}");
|
||||
return ScriptWorkerEnvironment.ExitProtocolFault;
|
||||
default:
|
||||
await Console.Error.WriteLineAsync($"ScriptWorker: unexpected frame {frame.BodyCase}");
|
||||
return ScriptWorkerEnvironment.ExitProtocolFault;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private void Load(LoadScripts load)
|
||||
{
|
||||
var result = new LoadResult { LoadId = load.LoadId };
|
||||
foreach (var script in load.Scripts)
|
||||
{
|
||||
var outcome = new LoadOutcome { Key = script.Key, Ok = true };
|
||||
try
|
||||
{
|
||||
var loaded = LoadedScript.Compile(script);
|
||||
if (_scripts.TryRemove(script.Key, out var previous)) previous.Dispose();
|
||||
_scripts[script.Key] = loaded;
|
||||
}
|
||||
catch (CompilationErrorException ex)
|
||||
{
|
||||
outcome.Ok = false;
|
||||
outcome.Error = new ScriptError { Kind = ScriptError.Types.Kind.Compile, Message = ex.Message };
|
||||
}
|
||||
catch (ScriptSandboxViolationException ex)
|
||||
{
|
||||
outcome.Ok = false;
|
||||
outcome.Error = new ScriptError { Kind = ScriptError.Types.Kind.Sandbox, Message = ex.Message };
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
outcome.Ok = false;
|
||||
outcome.Error = new ScriptError { Kind = ScriptError.Types.Kind.Threw, Message = ex.Message };
|
||||
}
|
||||
result.Outcomes.Add(outcome);
|
||||
}
|
||||
if (ForbiddenAssemblyLoaded() is { } forbidden) DieForbidden(forbidden);
|
||||
Send(new Envelope { LoadResult = result });
|
||||
}
|
||||
|
||||
private void Admit(EvalRequest request)
|
||||
{
|
||||
if (_queue.Count >= _queueCapacity)
|
||||
{
|
||||
SendResult(new EvalResult
|
||||
{
|
||||
EvalId = request.EvalId,
|
||||
Error = new ScriptError { Kind = ScriptError.Types.Kind.Busy, Message = "script worker busy" },
|
||||
});
|
||||
return;
|
||||
}
|
||||
_queue.Add(request);
|
||||
}
|
||||
|
||||
private void EvalThreadLoop()
|
||||
{
|
||||
foreach (var request in _queue.GetConsumingEnumerable())
|
||||
{
|
||||
EvalResult result;
|
||||
try
|
||||
{
|
||||
result = Evaluate(request);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
result = new EvalResult
|
||||
{
|
||||
EvalId = request.EvalId,
|
||||
Error = new ScriptError { Kind = ScriptError.Types.Kind.Threw, Message = ex.Message },
|
||||
};
|
||||
}
|
||||
SendResult(result);
|
||||
}
|
||||
}
|
||||
|
||||
private EvalResult Evaluate(EvalRequest request)
|
||||
{
|
||||
var result = new EvalResult { EvalId = request.EvalId };
|
||||
if (!_scripts.TryGetValue(request.Key, out var script))
|
||||
{
|
||||
result.Error = new ScriptError { Kind = ScriptError.Types.Kind.NotLoaded, Message = $"script '{request.Key}' is not loaded" };
|
||||
return result;
|
||||
}
|
||||
|
||||
var inputs = new Dictionary<string, DataValueSnapshot>(request.Inputs.Count, StringComparer.Ordinal);
|
||||
foreach (var input in request.Inputs)
|
||||
{
|
||||
string path;
|
||||
if (!string.IsNullOrEmpty(input.Path)) path = input.Path;
|
||||
else if (input.Slot < script.InputPaths.Length) path = script.InputPaths[input.Slot];
|
||||
else throw new ScriptWorkerProtocolException($"input slot {input.Slot} is not registered for '{request.Key}'");
|
||||
|
||||
if (input.Unsupported)
|
||||
{
|
||||
result.Error = new ScriptError
|
||||
{
|
||||
Kind = ScriptError.Types.Kind.UnsupportedType,
|
||||
Message = $"input '{path}' has a value of unsupported type {input.UnsupportedType}",
|
||||
};
|
||||
return result;
|
||||
}
|
||||
inputs[path] = new DataValueSnapshot(
|
||||
VariantCodec.Decode(input.Value),
|
||||
input.StatusCode,
|
||||
input.SourceTsTicks == 0 ? null : new DateTime(input.SourceTsTicks, DateTimeKind.Utc),
|
||||
new DateTime(input.ServerTsTicks, DateTimeKind.Utc));
|
||||
}
|
||||
|
||||
// ctx.Now follows the HOST clock (design §4.3) but keeps ticking: a fixed instant would never let a script
|
||||
// that waits on ctx.Now finish.
|
||||
var offset = request.NowUtcTicks == 0
|
||||
? TimeSpan.Zero
|
||||
: new DateTime(request.NowUtcTicks, DateTimeKind.Utc) - DateTime.UtcNow;
|
||||
Func<DateTime> clock = () => DateTime.UtcNow + offset;
|
||||
var logger = _scriptLogger.ForContext(ForwardingLogSink.KeyProperty, request.Key);
|
||||
var soft = TimeSpan.FromMilliseconds(Math.Max(1, request.SoftTimeoutMs));
|
||||
|
||||
// Cancelled by the host's Cancel frame at the soft deadline, never by our own clock (see the proto).
|
||||
using var cts = new CancellationTokenSource();
|
||||
var grace = request.HardGraceMs > 0 ? TimeSpan.FromMilliseconds(request.HardGraceMs) : _hardGrace;
|
||||
var running = new RunningEval(request.Key, Stopwatch.GetTimestamp(), soft + grace + WatchdogMargin, cts);
|
||||
_running[(long)request.EvalId] = running;
|
||||
var before = GC.GetAllocatedBytesForCurrentThread();
|
||||
var sw = Stopwatch.StartNew();
|
||||
try
|
||||
{
|
||||
object? value = script.Run(inputs, logger, clock, cts.Token, out var isPredicate);
|
||||
if (isPredicate)
|
||||
{
|
||||
result.Predicate = (bool)value!;
|
||||
}
|
||||
else if (VariantCodec.TryEncode(value, out var variant, out var unsupported))
|
||||
{
|
||||
result.Value = variant;
|
||||
}
|
||||
else
|
||||
{
|
||||
result.Error = new ScriptError
|
||||
{
|
||||
Kind = ScriptError.Types.Kind.UnsupportedType,
|
||||
Message = $"script returned a value of unsupported type {unsupported}",
|
||||
};
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
result.Error = new ScriptError { Kind = ScriptError.Types.Kind.Threw, Message = ex.Message };
|
||||
}
|
||||
finally
|
||||
{
|
||||
_running.TryRemove((long)request.EvalId, out _);
|
||||
}
|
||||
result.AllocatedBytes = (ulong)Math.Max(0, GC.GetAllocatedBytesForCurrentThread() - before);
|
||||
result.RunMicros = (uint)Math.Min(uint.MaxValue, sw.Elapsed.TotalMicroseconds);
|
||||
return result;
|
||||
}
|
||||
|
||||
private void WatchdogLoop()
|
||||
{
|
||||
while (true)
|
||||
{
|
||||
Thread.Sleep(50);
|
||||
foreach (var (_, run) in _running)
|
||||
{
|
||||
if (Stopwatch.GetElapsedTime(run.StartedTimestamp) > run.FailFastAfter)
|
||||
{
|
||||
// The host should have killed us already; this names the culprit in the exit/dump.
|
||||
Environment.FailFast($"ScriptWorker: script '{run.Key}' exceeded its hard deadline ({run.FailFastAfter.TotalMilliseconds:F0} ms)");
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private async Task HeartbeatLoopAsync()
|
||||
{
|
||||
using var timer = new PeriodicTimer(_heartbeatInterval);
|
||||
while (await timer.WaitForNextTickAsync())
|
||||
{
|
||||
Send(new Envelope
|
||||
{
|
||||
Heartbeat = new Heartbeat
|
||||
{
|
||||
Inflight = (uint)_running.Count,
|
||||
Queued = (uint)_queue.Count,
|
||||
HeapBytes = GC.GetTotalMemory(false),
|
||||
Gen2Count = (uint)GC.CollectionCount(2),
|
||||
},
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
private async Task LogFlushLoopAsync()
|
||||
{
|
||||
using var timer = new PeriodicTimer(TimeSpan.FromMilliseconds(100));
|
||||
while (await timer.WaitForNextTickAsync())
|
||||
{
|
||||
if (_logSink.Drain() is { } batch) Send(new Envelope { Logs = batch });
|
||||
}
|
||||
}
|
||||
|
||||
private async Task ParentWatchLoopAsync()
|
||||
{
|
||||
using var timer = new PeriodicTimer(TimeSpan.FromMilliseconds(500));
|
||||
while (await timer.WaitForNextTickAsync())
|
||||
{
|
||||
if (!ParentAlive())
|
||||
{
|
||||
await Console.Error.WriteLineAsync("ScriptWorker: parent process is gone; exiting.");
|
||||
Environment.Exit(ScriptWorkerEnvironment.ExitParentGone);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
private bool ParentAlive()
|
||||
{
|
||||
try
|
||||
{
|
||||
using var parent = Process.GetProcessById(_parentPid);
|
||||
return !parent.HasExited;
|
||||
}
|
||||
catch (ArgumentException)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
catch (InvalidOperationException)
|
||||
{
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
private async Task WriteLoopAsync()
|
||||
{
|
||||
var reader = _outbound.Reader;
|
||||
var pendingResults = new EvalResults();
|
||||
try
|
||||
{
|
||||
while (await reader.WaitToReadAsync())
|
||||
{
|
||||
// Opportunistic coalescing (design §4.5): drain whatever is queued right now, merging every completed
|
||||
// evaluation into one EvalResults frame; no timer, so an idle pipe adds no latency.
|
||||
while (reader.TryRead(out var envelope))
|
||||
{
|
||||
if (envelope.BodyCase == Envelope.BodyOneofCase.Results)
|
||||
{
|
||||
pendingResults.Items.AddRange(envelope.Results.Items);
|
||||
continue;
|
||||
}
|
||||
await ScriptWorkerFrameCodec.WriteAsync(_pipe, envelope, _maxFrameBytes);
|
||||
}
|
||||
if (pendingResults.Items.Count > 0)
|
||||
{
|
||||
await ScriptWorkerFrameCodec.WriteAsync(_pipe, Wrap(new Envelope { Results = pendingResults }), _maxFrameBytes);
|
||||
pendingResults = new EvalResults();
|
||||
}
|
||||
await _pipe.FlushAsync();
|
||||
}
|
||||
}
|
||||
catch (IOException)
|
||||
{
|
||||
Environment.Exit(ScriptWorkerEnvironment.ExitPipeClosed);
|
||||
}
|
||||
catch (ObjectDisposedException)
|
||||
{
|
||||
Environment.Exit(ScriptWorkerEnvironment.ExitPipeClosed);
|
||||
}
|
||||
}
|
||||
|
||||
private void SendResult(EvalResult result)
|
||||
{
|
||||
var results = new EvalResults();
|
||||
results.Items.Add(result);
|
||||
Send(new Envelope { Results = results });
|
||||
}
|
||||
|
||||
private void Send(Envelope envelope) => _outbound.Writer.TryWrite(Wrap(envelope));
|
||||
|
||||
private Envelope Wrap(Envelope envelope)
|
||||
{
|
||||
envelope.ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion;
|
||||
envelope.Seq = (ulong)Interlocked.Increment(ref _seq);
|
||||
return envelope;
|
||||
}
|
||||
|
||||
private static DiagnosticsReply Diagnostics()
|
||||
{
|
||||
var reply = new DiagnosticsReply
|
||||
{
|
||||
Pid = Environment.ProcessId,
|
||||
WorkingDirectory = Environment.CurrentDirectory,
|
||||
};
|
||||
reply.LoadedAssemblies.AddRange(AppDomain.CurrentDomain.GetAssemblies()
|
||||
.Select(a => a.GetName().Name ?? string.Empty)
|
||||
.Where(n => n.Length > 0));
|
||||
foreach (DictionaryEntry entry in Environment.GetEnvironmentVariables())
|
||||
reply.EnvironmentNames.Add((string)entry.Key);
|
||||
return reply;
|
||||
}
|
||||
|
||||
private static string? ForbiddenAssemblyLoaded()
|
||||
{
|
||||
foreach (var assembly in AppDomain.CurrentDomain.GetAssemblies())
|
||||
{
|
||||
var name = assembly.GetName().Name ?? string.Empty;
|
||||
foreach (var prefix in ScriptWorkerEnvironment.ForbiddenAssemblyPrefixes)
|
||||
if (name.StartsWith(prefix, StringComparison.Ordinal)) return name;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
private static void DieForbidden(string assembly)
|
||||
{
|
||||
Console.Error.WriteLine($"ScriptWorker: forbidden assembly '{assembly}' is loaded; exiting.");
|
||||
Environment.Exit(ScriptWorkerEnvironment.ExitForbiddenAssembly);
|
||||
}
|
||||
|
||||
private sealed record RunningEval(string Key, long StartedTimestamp, TimeSpan FailFastAfter, CancellationTokenSource Cancellation);
|
||||
|
||||
/// <summary>One loaded script: its compiled evaluator and its input slot order.</summary>
|
||||
private sealed class LoadedScript : IDisposable
|
||||
{
|
||||
private readonly ScriptEvaluator<VirtualTagContext, object?>? _value;
|
||||
private readonly ScriptEvaluator<AlarmPredicateContext, bool>? _predicate;
|
||||
|
||||
private LoadedScript(string[] inputPaths, ScriptEvaluator<VirtualTagContext, object?>? value,
|
||||
ScriptEvaluator<AlarmPredicateContext, bool>? predicate)
|
||||
{
|
||||
InputPaths = inputPaths;
|
||||
_value = value;
|
||||
_predicate = predicate;
|
||||
}
|
||||
|
||||
public string[] InputPaths { get; }
|
||||
|
||||
public static LoadedScript Compile(ScriptSource source) => source.Kind switch
|
||||
{
|
||||
ContextKind.VirtualTag or ContextKind.Calculation => new LoadedScript(
|
||||
[.. source.InputPaths], ScriptEvaluator<VirtualTagContext, object?>.Compile(source.Source), null),
|
||||
ContextKind.AlarmPredicate => new LoadedScript(
|
||||
[.. source.InputPaths], null, ScriptEvaluator<AlarmPredicateContext, bool>.Compile(source.Source)),
|
||||
_ => throw new ScriptWorkerProtocolException($"unknown script kind {source.Kind}"),
|
||||
};
|
||||
|
||||
public object? Run(IReadOnlyDictionary<string, DataValueSnapshot> inputs, ILogger logger, Func<DateTime> clock,
|
||||
CancellationToken ct, out bool isPredicate)
|
||||
{
|
||||
if (_predicate is not null)
|
||||
{
|
||||
isPredicate = true;
|
||||
return _predicate.RunAsync(new AlarmPredicateContext(inputs, logger, clock), ct).GetAwaiter().GetResult();
|
||||
}
|
||||
isPredicate = false;
|
||||
// Single-tag mode, as in the host: a cross-tag write is dropped.
|
||||
return _value!.RunAsync(new VirtualTagContext(inputs, static (_, _) => { }, logger, clock), ct)
|
||||
.GetAwaiter().GetResult();
|
||||
}
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
_value?.Dispose();
|
||||
_predicate?.Dispose();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Buffers <c>ctx.Logger</c> events for the host, bounded, counting what it had to drop.</summary>
|
||||
private sealed class ForwardingLogSink : ILogEventSink
|
||||
{
|
||||
public const string KeyProperty = "ScriptWorkerKey";
|
||||
private readonly ConcurrentQueue<LogLine> _lines = new();
|
||||
private int _count;
|
||||
private int _dropped;
|
||||
|
||||
public int Capacity { get; set; } = 1000;
|
||||
|
||||
public void Emit(LogEvent logEvent)
|
||||
{
|
||||
if (Interlocked.Increment(ref _count) > Capacity)
|
||||
{
|
||||
Interlocked.Decrement(ref _count);
|
||||
Interlocked.Increment(ref _dropped);
|
||||
return;
|
||||
}
|
||||
var key = logEvent.Properties.TryGetValue(KeyProperty, out var k) && k is ScalarValue { Value: string s } ? s : string.Empty;
|
||||
_lines.Enqueue(new LogLine
|
||||
{
|
||||
Key = key,
|
||||
Level = (uint)logEvent.Level,
|
||||
Rendered = Truncate(logEvent.RenderMessage(), 8 * 1024),
|
||||
Exception = Truncate(logEvent.Exception?.ToString() ?? string.Empty, 16 * 1024),
|
||||
TsTicks = logEvent.Timestamp.UtcTicks,
|
||||
});
|
||||
}
|
||||
|
||||
public LogBatch? Drain()
|
||||
{
|
||||
var dropped = Interlocked.Exchange(ref _dropped, 0);
|
||||
if (_lines.IsEmpty && dropped == 0) return null;
|
||||
var batch = new LogBatch { Dropped = (uint)dropped };
|
||||
while (_lines.TryDequeue(out var line))
|
||||
{
|
||||
Interlocked.Decrement(ref _count);
|
||||
batch.Lines.Add(line);
|
||||
}
|
||||
return batch;
|
||||
}
|
||||
|
||||
private static string Truncate(string value, int max) => value.Length <= max ? value : value[..max];
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<!-- lmxopcua#780 — the out-of-process, pre-emptible script runner's worker (design §3, §6). A plain console app the
|
||||
host launches per shard over a named pipe. It references only the compile/run core and the frame protocol, so
|
||||
the host's cluster, database, OPC UA and secrets stacks are not in its assembly set; it checks itself against
|
||||
ScriptWorkerEnvironment.ForbiddenAssemblyPrefixes at runtime as well. -->
|
||||
<PropertyGroup>
|
||||
<OutputType>Exe</OutputType>
|
||||
<RootNamespace>ZB.MOM.WW.OtOpcUa.ScriptWorker</RootNamespace>
|
||||
<AssemblyName>ZB.MOM.WW.OtOpcUa.ScriptWorker</AssemblyName>
|
||||
<InvariantGlobalization>true</InvariantGlobalization>
|
||||
<!-- Workstation, non-concurrent GC: the heap hard limit (DOTNET_GCHeapHardLimit) is the worker's memory bound. -->
|
||||
<ServerGarbageCollection>false</ServerGarbageCollection>
|
||||
<ConcurrentGarbageCollection>false</ConcurrentGarbageCollection>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.Scripting\ZB.MOM.WW.OtOpcUa.Core.Scripting.csproj"/>
|
||||
<ProjectReference Include="..\..\Core\ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol\ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.csproj"/>
|
||||
</ItemGroup>
|
||||
|
||||
<!-- Core.Scripting reaches Akka and Grpc.Core.Api transitively through Commons (telemetry meter, log messages),
|
||||
none of which the worker executes. Excluding their assets keeps them out of the worker's deps.json and output
|
||||
folder, so they are not loadable at all, not merely unloaded (design §6 "assembly allow-list by
|
||||
construction"). A code path that did touch them would fail with FileNotFoundException, which the
|
||||
containment tests would catch. -->
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Akka" ExcludeAssets="all" PrivateAssets="all"/>
|
||||
<PackageReference Include="Grpc.Core.Api" ExcludeAssets="all" PrivateAssets="all"/>
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
@@ -0,0 +1,43 @@
|
||||
<Project>
|
||||
<!--
|
||||
lmxopcua#780 — places the ScriptWorker next to a host that launches it: the worker's build output is copied into
|
||||
$(OutDir)script-worker/ after Build, and into $(PublishDir)script-worker/ after Publish. Imported by the Host and by
|
||||
the worker test project. The worker is its own app (own deps.json / runtimeconfig.json), so it lives in its own folder
|
||||
rather than beside the host's assemblies.
|
||||
|
||||
Publish note: the copy is the worker's framework-dependent BUILD output, run through the dotnet muxer. That suits the
|
||||
docker-dev image (aspnet:10.0 runtime). A self-contained host publish (the Windows deploys) needs a self-contained
|
||||
worker publish too; that is part of the installer work tracked on lmxopcua#780.
|
||||
-->
|
||||
<PropertyGroup>
|
||||
<_OtScriptWorkerProject>$(MSBuildThisFileDirectory)../ZB.MOM.WW.OtOpcUa.ScriptWorker.csproj</_OtScriptWorkerProject>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="$(_OtScriptWorkerProject)" ReferenceOutputAssembly="false" Private="false" />
|
||||
</ItemGroup>
|
||||
|
||||
<Target Name="_OtResolveScriptWorkerOutput">
|
||||
<MSBuild Projects="$(_OtScriptWorkerProject)" Targets="GetTargetPath" Properties="Configuration=$(Configuration)">
|
||||
<Output TaskParameter="TargetOutputs" ItemName="_OtScriptWorkerTargetPath" />
|
||||
</MSBuild>
|
||||
<PropertyGroup>
|
||||
<_OtScriptWorkerOutDir>@(_OtScriptWorkerTargetPath->'%(RootDir)%(Directory)')</_OtScriptWorkerOutDir>
|
||||
</PropertyGroup>
|
||||
<ItemGroup>
|
||||
<_OtScriptWorkerFiles Include="$(_OtScriptWorkerOutDir)**/*" />
|
||||
</ItemGroup>
|
||||
</Target>
|
||||
|
||||
<Target Name="_OtCopyScriptWorkerAfterBuild" AfterTargets="Build" DependsOnTargets="_OtResolveScriptWorkerOutput">
|
||||
<Copy SourceFiles="@(_OtScriptWorkerFiles)"
|
||||
DestinationFiles="@(_OtScriptWorkerFiles->'$(OutDir)script-worker/%(RecursiveDir)%(Filename)%(Extension)')"
|
||||
SkipUnchangedFiles="true" />
|
||||
</Target>
|
||||
|
||||
<Target Name="_OtCopyScriptWorkerAfterPublish" AfterTargets="Publish" DependsOnTargets="_OtResolveScriptWorkerOutput">
|
||||
<Copy SourceFiles="@(_OtScriptWorkerFiles)"
|
||||
DestinationFiles="@(_OtScriptWorkerFiles->'$(PublishDir)script-worker/%(RecursiveDir)%(Filename)%(Extension)')"
|
||||
SkipUnchangedFiles="true" />
|
||||
</Target>
|
||||
</Project>
|
||||
+122
@@ -0,0 +1,122 @@
|
||||
using System.Collections.Concurrent;
|
||||
using Serilog;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.Calculation.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// lmxopcua#780 — with <c>Scripting:Runner=OutOfProcess</c> selecting the Calculation seam, the driver hands each
|
||||
/// compiled run to the node's <see cref="IScriptRunner"/>: keyed by the calc tag's RawPath, with the driver's own
|
||||
/// budget. A runner-side failure publishes <c>BadResourceUnavailable</c> (0x80040000); a script failure keeps
|
||||
/// <c>BadInternalError</c>. The passthrough fast path never reaches the runner.
|
||||
/// </summary>
|
||||
public sealed class CalculationDriverScriptRunnerTests
|
||||
{
|
||||
private const uint StatusGood = 0u;
|
||||
private const uint StatusBadInternalError = 0x80020000u;
|
||||
private const uint StatusBadResourceUnavailable = 0x80040000u;
|
||||
private const string Tag = "calc/e/x";
|
||||
|
||||
private sealed class FakeRunner : IScriptRunner
|
||||
{
|
||||
public ConcurrentQueue<ScriptRunOutcome> Outcomes { get; } = new();
|
||||
public ConcurrentQueue<ScriptInvocation> Invocations { get; } = new();
|
||||
public List<ScriptContextKind> Resets { get; } = [];
|
||||
|
||||
public Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default)
|
||||
{
|
||||
Invocations.Enqueue(invocation);
|
||||
return Task.FromResult(Outcomes.TryDequeue(out var next) ? next : ScriptRunOutcome.Ok(null));
|
||||
}
|
||||
|
||||
public void Reset(ScriptContextKind kind) => Resets.Add(kind);
|
||||
|
||||
public IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind) => ["q-" + kind];
|
||||
}
|
||||
|
||||
private static RawTagEntry CalcTag(string source)
|
||||
{
|
||||
var json = $"{{\"scriptId\":\"s\",\"changeTriggered\":true,\"scriptSource\":{System.Text.Json.JsonSerializer.Serialize(source)}}}";
|
||||
return new RawTagEntry(Tag, json, WriteIdempotent: false);
|
||||
}
|
||||
|
||||
private static async Task Settle(CalculationDriver driver)
|
||||
{
|
||||
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
|
||||
while (DateTime.UtcNow < deadline && !driver.IsEvaluationIdle) await Task.Delay(10);
|
||||
driver.IsEvaluationIdle.ShouldBeTrue();
|
||||
}
|
||||
|
||||
private static (CalculationDriver Driver, ConcurrentQueue<DataChangeEventArgs> Published) Build(FakeRunner runner, string source)
|
||||
{
|
||||
var driver = new CalculationDriver(
|
||||
new CalculationDriverOptions { RawTags = [CalcTag(source)], RunTimeout = TimeSpan.FromMilliseconds(1234), ScriptRunner = runner },
|
||||
"calc-drv", new ScriptRootLogger(new LoggerConfiguration().CreateLogger()));
|
||||
var published = new ConcurrentQueue<DataChangeEventArgs>();
|
||||
driver.OnDataChange += (_, e) => published.Enqueue(e);
|
||||
return (driver, published);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_runner_failure_publishes_BadResourceUnavailable_and_a_script_failure_BadInternalError()
|
||||
{
|
||||
var runner = new FakeRunner();
|
||||
runner.Outcomes.Enqueue(ScriptRunOutcome.Ok(30.0));
|
||||
runner.Outcomes.Enqueue(ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason));
|
||||
runner.Outcomes.Enqueue(ScriptRunOutcome.Failed("script threw: boom"));
|
||||
var (driver, published) = Build(runner, "return (double)ctx.GetTag(\"src/a\").Value * 3;");
|
||||
await driver.InitializeAsync("{}", TestContext.Current.CancellationToken);
|
||||
|
||||
driver.OnDependencyValue("src/a", 10.0, StatusGood, DateTime.UtcNow);
|
||||
await Settle(driver);
|
||||
published.TryDequeue(out var good).ShouldBeTrue();
|
||||
good!.Snapshot.StatusCode.ShouldBe(StatusGood);
|
||||
good.Snapshot.Value.ShouldBe(30.0);
|
||||
|
||||
var invocation = runner.Invocations.Single();
|
||||
invocation.Key.ShouldBe(Tag);
|
||||
invocation.Kind.ShouldBe(ScriptContextKind.Calculation);
|
||||
invocation.Timeout.ShouldBe(TimeSpan.FromMilliseconds(1234));
|
||||
invocation.Inputs["src/a"].Value.ShouldBe(10.0);
|
||||
|
||||
driver.OnDependencyValue("src/a", 11.0, StatusGood, DateTime.UtcNow);
|
||||
await Settle(driver);
|
||||
published.TryDequeue(out var unavailable).ShouldBeTrue();
|
||||
unavailable!.Snapshot.StatusCode.ShouldBe(StatusBadResourceUnavailable);
|
||||
unavailable.Snapshot.Value.ShouldBe(30.0, "the last known value rides out with the Bad status");
|
||||
|
||||
driver.OnDependencyValue("src/a", 12.0, StatusGood, DateTime.UtcNow);
|
||||
await Settle(driver);
|
||||
published.TryDequeue(out var scriptFault).ShouldBeTrue();
|
||||
scriptFault!.Snapshot.StatusCode.ShouldBe(StatusBadInternalError);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task The_passthrough_fast_path_never_reaches_the_runner()
|
||||
{
|
||||
var runner = new FakeRunner();
|
||||
var (driver, published) = Build(runner, "return ctx.GetTag(\"src/a\").Value;");
|
||||
await driver.InitializeAsync("{}", TestContext.Current.CancellationToken);
|
||||
|
||||
driver.OnDependencyValue("src/a", 4.0, StatusGood, DateTime.UtcNow);
|
||||
await Settle(driver);
|
||||
published.TryDequeue(out var mirrored).ShouldBeTrue();
|
||||
mirrored!.Snapshot.Value.ShouldBe(4.0);
|
||||
runner.Invocations.ShouldBeEmpty();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void The_deploy_boundary_resets_the_runners_calculation_seam_and_quarantines_are_reported()
|
||||
{
|
||||
var runner = new FakeRunner();
|
||||
using var evaluator = new CalculationEvaluator(Microsoft.Extensions.Logging.Abstractions.NullLogger.Instance,
|
||||
new ScriptRootLogger(new LoggerConfiguration().CreateLogger()), scriptRunner: runner);
|
||||
evaluator.ClearCompiledScripts();
|
||||
evaluator.ResetQuarantines();
|
||||
runner.Resets.ShouldBe([ScriptContextKind.Calculation, ScriptContextKind.Calculation]);
|
||||
evaluator.QuarantinedScripts.ShouldBe(["q-Calculation"]);
|
||||
}
|
||||
}
|
||||
+170
@@ -0,0 +1,170 @@
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Serilog;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Configuration;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Configuration;
|
||||
|
||||
/// <summary>
|
||||
/// lmxopcua#780 — <c>Scripting:Runner</c> binds by its documented keys (design §12), defaults to InProcess (every seam
|
||||
/// gets no runner, so the in-process path is untouched), and the worker options clamp and fail closed.
|
||||
/// </summary>
|
||||
public sealed class ScriptRunnerSelectionTests
|
||||
{
|
||||
private static IConfiguration Config(Dictionary<string, string?> values) =>
|
||||
new ConfigurationBuilder().AddInMemoryCollection(values).Build();
|
||||
|
||||
private static readonly ILogger Silent = new LoggerConfiguration().CreateLogger();
|
||||
|
||||
[Fact]
|
||||
public void The_default_is_InProcess_and_selects_no_runner_for_any_seam()
|
||||
{
|
||||
var bound = ScriptingOptionsBinding.Bind(Config(new()), out var adjustments);
|
||||
adjustments.ShouldBeEmpty();
|
||||
bound.Runner.ShouldBe(ScriptRunnerMode.InProcess);
|
||||
ScriptRunnerSelection.Validate(bound, "/srv/app").ShouldBeEmpty();
|
||||
using var selection = ScriptRunnerSelection.Create(bound, Silent, null);
|
||||
foreach (var seam in Enum.GetValues<ScriptSeam>()) selection.ForSeam(seam).ShouldBeNull(seam.ToString());
|
||||
selection.OutOfProcessRunner.ShouldBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Every_documented_runner_key_binds()
|
||||
{
|
||||
var bound = ScriptingOptionsBinding.Bind(Config(new()
|
||||
{
|
||||
["Scripting:Runner"] = "OutOfProcess",
|
||||
["Scripting:Runner:Seams:0"] = "Calculation",
|
||||
["Scripting:Worker:Shards"] = "3",
|
||||
["Scripting:Worker:HardGrace"] = "00:00:02",
|
||||
["Scripting:Worker:HeapLimitBytes"] = "268435456",
|
||||
["Scripting:Worker:QuarantineAfterKills"] = "4",
|
||||
["Scripting:Worker:UnavailableGrace"] = "00:00:07",
|
||||
["Scripting:Worker:IsolationProbationRuns"] = "9",
|
||||
["Scripting:Worker:PeCacheDir"] = "/var/cache/otopcua-pe",
|
||||
["Scripting:Worker:ExecutablePath"] = "/opt/w/ZB.MOM.WW.OtOpcUa.ScriptWorker.dll",
|
||||
}), out var adjustments);
|
||||
|
||||
adjustments.ShouldBeEmpty();
|
||||
bound.Runner.ShouldBe(ScriptRunnerMode.OutOfProcess);
|
||||
bound.RunnerSeams.ShouldBe([ScriptSeam.Calculation]);
|
||||
bound.Worker.Shards.ShouldBe(3);
|
||||
bound.Worker.HardGrace.ShouldBe(TimeSpan.FromSeconds(2));
|
||||
bound.Worker.HeapLimitBytes.ShouldBe(256L * 1024 * 1024);
|
||||
bound.Worker.QuarantineAfterKills.ShouldBe(4);
|
||||
bound.Worker.UnavailableGrace.ShouldBe(TimeSpan.FromSeconds(7));
|
||||
bound.Worker.IsolationProbationRuns.ShouldBe(9);
|
||||
bound.Worker.PeCacheDir.ShouldBe("/var/cache/otopcua-pe");
|
||||
bound.Worker.ExecutablePath.ShouldBe("/opt/w/ZB.MOM.WW.OtOpcUa.ScriptWorker.dll");
|
||||
bound.IsOutOfProcess(ScriptSeam.Calculation).ShouldBeTrue();
|
||||
bound.IsOutOfProcess(ScriptSeam.VirtualTag).ShouldBeFalse();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_misspelled_runner_mode_fails_startup()
|
||||
=> Should.Throw<InvalidOperationException>(() =>
|
||||
ScriptingOptionsBinding.Bind(Config(new() { ["Scripting:Runner"] = "OutOfProc" }), out _));
|
||||
|
||||
[Fact]
|
||||
public void Worker_values_out_of_range_are_clamped_and_reported()
|
||||
{
|
||||
var bound = ScriptingOptionsBinding.Bind(Config(new()
|
||||
{
|
||||
["Scripting:Worker:Shards"] = "64",
|
||||
["Scripting:Worker:HardGrace"] = "00:00:00.010",
|
||||
["Scripting:Worker:HeapLimitBytes"] = "1024",
|
||||
["Scripting:Worker:QuarantineAfterKills"] = "0",
|
||||
["Scripting:Worker:UnavailableGrace"] = "00:05:00",
|
||||
["Scripting:Worker:IsolationProbationRuns"] = "0",
|
||||
}), out var adjustments);
|
||||
|
||||
bound.Worker.Shards.ShouldBe(ScriptWorkerOptions.MaxShards);
|
||||
bound.Worker.HardGrace.ShouldBe(ScriptWorkerOptions.MinHardGrace);
|
||||
bound.Worker.HeapLimitBytes.ShouldBe(ScriptWorkerOptions.MinHeapLimitBytes);
|
||||
bound.Worker.QuarantineAfterKills.ShouldBe(1);
|
||||
bound.Worker.UnavailableGrace.ShouldBe(ScriptWorkerOptions.MaxUnavailableGrace);
|
||||
bound.Worker.IsolationProbationRuns.ShouldBe(1);
|
||||
adjustments.Count.ShouldBe(6);
|
||||
adjustments.ShouldAllBe(a => a.StartsWith("Scripting:Worker:"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void The_hard_grace_defaults_to_max_of_one_second_and_the_budget()
|
||||
{
|
||||
var worker = new ScriptWorkerOptions();
|
||||
worker.EffectiveHardGrace(TimeSpan.FromMilliseconds(250)).ShouldBe(TimeSpan.FromSeconds(1));
|
||||
worker.EffectiveHardGrace(TimeSpan.FromSeconds(2)).ShouldBe(TimeSpan.FromSeconds(2));
|
||||
new ScriptWorkerOptions { HardGrace = TimeSpan.FromMilliseconds(400) }
|
||||
.EffectiveHardGrace(TimeSpan.FromSeconds(2)).ShouldBe(TimeSpan.FromMilliseconds(400));
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("relative/cache", "must be an absolute path")]
|
||||
[InlineData("{root}/pe-cache", "inside the content root")]
|
||||
[InlineData("{root}", "inside the content root")]
|
||||
public void A_pe_cache_dir_that_is_relative_or_inside_the_content_root_fails_startup(string dir, string why)
|
||||
{
|
||||
var root = Path.Combine(Path.GetTempPath(), "otopcua-root-" + Guid.NewGuid().ToString("N"));
|
||||
var options = new ScriptingOptions
|
||||
{
|
||||
Runner = ScriptRunnerMode.OutOfProcess,
|
||||
Worker = new ScriptWorkerOptions { PeCacheDir = dir.Replace("{root}", root) },
|
||||
};
|
||||
Should.Throw<InvalidOperationException>(() => ScriptRunnerSelection.Validate(options, root)).Message.ShouldContain(why);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_pe_cache_dir_outside_the_content_root_is_accepted()
|
||||
{
|
||||
var root = Path.Combine(Path.GetTempPath(), "otopcua-root");
|
||||
var options = new ScriptingOptions
|
||||
{
|
||||
Runner = ScriptRunnerMode.OutOfProcess,
|
||||
Worker = new ScriptWorkerOptions { PeCacheDir = Path.Combine(Path.GetTempPath(), "otopcua-root-sibling") },
|
||||
};
|
||||
Should.NotThrow(() => ScriptRunnerSelection.Validate(options, root));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Shadow_is_not_implemented_yet_and_runs_in_process_with_a_warning()
|
||||
{
|
||||
var options = new ScriptingOptions { Runner = ScriptRunnerMode.Shadow };
|
||||
ScriptRunnerSelection.Validate(options, "/srv").ShouldHaveSingleItem().ShouldContain("Shadow");
|
||||
using var selection = ScriptRunnerSelection.Create(options, Silent, null);
|
||||
selection.OutOfProcessRunner.ShouldBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void OutOfProcess_selects_only_the_supported_requested_seams()
|
||||
{
|
||||
var all = new ScriptingOptions { Runner = ScriptRunnerMode.OutOfProcess };
|
||||
ScriptRunnerSelection.Validate(all, "/srv").ShouldContain(w => w.Contains("ScriptedAlarm seam cannot run out of process yet"));
|
||||
using (var selection = ScriptRunnerSelection.Create(all, Silent, null))
|
||||
{
|
||||
selection.ForSeam(ScriptSeam.VirtualTag).ShouldNotBeNull();
|
||||
selection.ForSeam(ScriptSeam.Calculation).ShouldBeSameAs(selection.ForSeam(ScriptSeam.VirtualTag), "one node-wide runner");
|
||||
selection.ForSeam(ScriptSeam.ScriptedAlarm).ShouldBeNull();
|
||||
}
|
||||
|
||||
var calcOnly = new ScriptingOptions { Runner = ScriptRunnerMode.OutOfProcess, RunnerSeams = [ScriptSeam.Calculation] };
|
||||
using (var selection = ScriptRunnerSelection.Create(calcOnly, Silent, null))
|
||||
{
|
||||
selection.ForSeam(ScriptSeam.Calculation).ShouldNotBeNull();
|
||||
selection.ForSeam(ScriptSeam.VirtualTag).ShouldBeNull();
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_missing_worker_is_a_startup_warning_not_a_silent_in_process_fallback()
|
||||
{
|
||||
var options = new ScriptingOptions
|
||||
{
|
||||
Runner = ScriptRunnerMode.OutOfProcess,
|
||||
RunnerSeams = [ScriptSeam.VirtualTag],
|
||||
Worker = new ScriptWorkerOptions { ExecutablePath = Path.Combine(Path.GetTempPath(), "missing-" + Guid.NewGuid().ToString("N"), "w.dll") },
|
||||
};
|
||||
ScriptRunnerSelection.Validate(options, "/srv").ShouldHaveSingleItem().ShouldContain("BadResourceUnavailable");
|
||||
}
|
||||
}
|
||||
+109
@@ -0,0 +1,109 @@
|
||||
using System.Collections.Concurrent;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Serilog;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
||||
using ZB.MOM.WW.OtOpcUa.Host.Engines;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Host.Tests.Engines;
|
||||
|
||||
/// <summary>
|
||||
/// lmxopcua#780 — the virtual-tag seam over <see cref="IScriptRunner"/>: with a runner (OutOfProcess) the compiled run
|
||||
/// is the runner's, keyed by virtual-tag id with the evaluator's budget, and its Unavailable outcome becomes a failure
|
||||
/// carrying <see cref="OpcUaStatus.BadResourceUnavailable"/>. Without one (the InProcess default) nothing changes.
|
||||
/// </summary>
|
||||
public sealed class RoslynVirtualTagEvaluatorScriptRunnerTests
|
||||
{
|
||||
private sealed class FakeRunner : IScriptRunner
|
||||
{
|
||||
public ConcurrentQueue<ScriptRunOutcome> Outcomes { get; } = new();
|
||||
public ConcurrentQueue<ScriptInvocation> Invocations { get; } = new();
|
||||
public List<ScriptContextKind> Resets { get; } = [];
|
||||
|
||||
public Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default)
|
||||
{
|
||||
Invocations.Enqueue(invocation);
|
||||
return Task.FromResult(Outcomes.TryDequeue(out var next) ? next : ScriptRunOutcome.Ok(null));
|
||||
}
|
||||
|
||||
public void Reset(ScriptContextKind kind) => Resets.Add(kind);
|
||||
|
||||
public IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind) => ["vt-q"];
|
||||
}
|
||||
|
||||
private static RoslynVirtualTagEvaluator Build(IScriptRunner? runner) =>
|
||||
new(NullLogger<RoslynVirtualTagEvaluator>.Instance, new ScriptRootLogger(new LoggerConfiguration().CreateLogger()),
|
||||
runTimeout: TimeSpan.FromMilliseconds(1500), scriptRunner: runner);
|
||||
|
||||
private static Dictionary<string, DataValueSnapshot> Deps(int a) => new(StringComparer.Ordinal)
|
||||
{
|
||||
["a"] = new DataValueSnapshot(a, 0u, DateTime.UtcNow, DateTime.UtcNow),
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public async Task Compiled_runs_go_to_the_runner_and_its_outcomes_map_onto_the_seam_contract()
|
||||
{
|
||||
var runner = new FakeRunner();
|
||||
runner.Outcomes.Enqueue(ScriptRunOutcome.Ok(42));
|
||||
runner.Outcomes.Enqueue(ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason));
|
||||
runner.Outcomes.Enqueue(ScriptRunOutcome.Failed("script threw: boom"));
|
||||
using var sut = Build(runner);
|
||||
const string script = "return (int)ctx.GetTag(\"a\").Value + 1;";
|
||||
|
||||
var ok = await sut.EvaluateAsync("vt-1", script, Deps(41));
|
||||
ok.Success.ShouldBeTrue();
|
||||
ok.Value.ShouldBe(42);
|
||||
ok.StatusCode.ShouldBeNull();
|
||||
|
||||
var unavailable = await sut.EvaluateAsync("vt-1", script, Deps(41));
|
||||
unavailable.Success.ShouldBeFalse();
|
||||
unavailable.Reason.ShouldBe(ScriptRunOutcome.WorkerRestartedReason);
|
||||
unavailable.StatusCode.ShouldBe(OpcUaStatus.BadResourceUnavailable);
|
||||
|
||||
var failed = await sut.EvaluateAsync("vt-1", script, Deps(41));
|
||||
failed.Reason.ShouldBe("script threw: boom");
|
||||
failed.StatusCode.ShouldBeNull("a script failure keeps BadInternalError");
|
||||
|
||||
var invocation = runner.Invocations.First();
|
||||
invocation.Key.ShouldBe("vt-1");
|
||||
invocation.Kind.ShouldBe(ScriptContextKind.VirtualTag);
|
||||
invocation.Source.ShouldBe(script);
|
||||
invocation.Timeout.ShouldBe(TimeSpan.FromMilliseconds(1500));
|
||||
invocation.Inputs["a"].Value.ShouldBe(41);
|
||||
invocation.ScriptLogger.ShouldNotBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task The_passthrough_fast_path_and_an_empty_expression_never_reach_the_runner()
|
||||
{
|
||||
var runner = new FakeRunner();
|
||||
using var sut = Build(runner);
|
||||
(await sut.EvaluateAsync("vt-p", "return ctx.GetTag(\"a\").Value;", Deps(7))).Value.ShouldBe(7);
|
||||
(await sut.EvaluateAsync("vt-e", " ", Deps(7))).Success.ShouldBeFalse();
|
||||
runner.Invocations.ShouldBeEmpty();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void The_deploy_boundary_resets_the_runners_virtual_tag_seam_and_its_quarantines_are_reported()
|
||||
{
|
||||
var runner = new FakeRunner();
|
||||
using var sut = Build(runner);
|
||||
sut.ClearCompiledScripts();
|
||||
sut.ResetQuarantines();
|
||||
runner.Resets.ShouldBe([ScriptContextKind.VirtualTag, ScriptContextKind.VirtualTag]);
|
||||
sut.QuarantinedScripts.ShouldBe(["vt-q"]);
|
||||
}
|
||||
|
||||
/// <summary>Positive control for the default: no runner means the in-process Roslyn path actually runs the script.</summary>
|
||||
[Fact]
|
||||
public async Task Without_a_runner_the_script_runs_in_process()
|
||||
{
|
||||
using var sut = Build(runner: null);
|
||||
sut.ScriptRunner.ShouldBeNull();
|
||||
(await sut.EvaluateAsync("vt-1", "return (int)ctx.GetTag(\"a\").Value + 1;", Deps(41))).Value.ShouldBe(42);
|
||||
sut.RunGuard.StartedRunCount.ShouldBe(1);
|
||||
}
|
||||
}
|
||||
+64
@@ -0,0 +1,64 @@
|
||||
using System.Collections.Concurrent;
|
||||
using Akka.Actor;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Engines;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.Tests.Harness;
|
||||
using ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Runtime.Tests.VirtualTags;
|
||||
|
||||
/// <summary>
|
||||
/// lmxopcua#780 — when the out-of-process runner, not the script, fails (its worker restarted, no worker ready), the
|
||||
/// virtual tag publishes <see cref="OpcUaStatus.BadResourceUnavailable"/> with its last value, so a client can tell a
|
||||
/// sick node from a broken script. Every other failure keeps the long-standing <see cref="OpcUaStatus.BadInternalError"/>.
|
||||
/// </summary>
|
||||
public sealed class VirtualTagActorRunnerUnavailableTests : RuntimeActorTestBase
|
||||
{
|
||||
private sealed class QueuedEvaluator : IVirtualTagEvaluator
|
||||
{
|
||||
public ConcurrentQueue<VirtualTagEvalResult> Results { get; } = new();
|
||||
|
||||
public VirtualTagEvalResult Evaluate(string virtualTagId, string expression, IReadOnlyDictionary<string, object?> dependencies)
|
||||
=> Results.TryDequeue(out var next) ? next : VirtualTagEvalResult.NoChange;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_runner_unavailable_failure_publishes_BadResourceUnavailable_and_a_script_failure_BadInternalError()
|
||||
{
|
||||
var parent = CreateTestProbe();
|
||||
var evaluator = new QueuedEvaluator();
|
||||
evaluator.Results.Enqueue(VirtualTagEvalResult.Ok(5));
|
||||
evaluator.Results.Enqueue(VirtualTagEvalResult.Unavailable("script worker restarted"));
|
||||
evaluator.Results.Enqueue(VirtualTagEvalResult.Unavailable("script worker restarted"));
|
||||
evaluator.Results.Enqueue(VirtualTagEvalResult.Failure("script threw: boom"));
|
||||
var actor = parent.ChildActorOf(VirtualTagActor.Props("vt-1", "x", evaluator: evaluator, dependencyRefs: ["a"]));
|
||||
|
||||
actor.Tell(new VirtualTagActor.DependencyValueChanged("a", 1, DateTime.UtcNow));
|
||||
parent.ExpectMsg<VirtualTagActor.EvaluationResult>().StatusCode.ShouldBe(OpcUaStatus.Good);
|
||||
|
||||
actor.Tell(new VirtualTagActor.DependencyValueChanged("a", 2, DateTime.UtcNow));
|
||||
var unavailable = parent.ExpectMsg<VirtualTagActor.EvaluationResult>(PresenceBudget);
|
||||
unavailable.StatusCode.ShouldBe(OpcUaStatus.BadResourceUnavailable);
|
||||
unavailable.Value.ShouldBe(5, "the last known value rides out with the Bad status");
|
||||
|
||||
// Once per transition: the same code again publishes nothing.
|
||||
actor.Tell(new VirtualTagActor.DependencyValueChanged("a", 3, DateTime.UtcNow));
|
||||
parent.ExpectNoMsg(TimeSpan.FromMilliseconds(300));
|
||||
|
||||
// A script failure is a different Bad and is published.
|
||||
actor.Tell(new VirtualTagActor.DependencyValueChanged("a", 4, DateTime.UtcNow));
|
||||
var scriptFault = parent.ExpectMsg<VirtualTagActor.EvaluationResult>(PresenceBudget);
|
||||
scriptFault.StatusCode.ShouldBe(OpcUaStatus.BadInternalError);
|
||||
scriptFault.Value.ShouldBe(5);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_failure_without_a_status_keeps_BadInternalError()
|
||||
{
|
||||
VirtualTagEvalResult.Failure("x").StatusCode.ShouldBeNull();
|
||||
VirtualTagEvalResult.Unavailable("x").StatusCode.ShouldBe(OpcUaStatus.BadResourceUnavailable);
|
||||
OpcUaStatus.BadResourceUnavailable.ShouldBe(0x80040000u, "the OPC UA Part 6 code ScadaBridge maps");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,382 @@
|
||||
using System.Diagnostics;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests;
|
||||
|
||||
/// <summary>Worker-backed tests spawn real processes and measure wall-clock bounds, so they never run in parallel.</summary>
|
||||
[CollectionDefinition(Name, DisableParallelization = true)]
|
||||
public sealed class ScriptWorkerCollection
|
||||
{
|
||||
public const string Name = "ScriptWorker";
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// lmxopcua#780 design §13 "Containment": each case runs against the real <c>ScriptWorker</c> process, and each has a
|
||||
/// positive control — a healthy sibling on the SAME shard (every runner here has one shard) that keeps producing
|
||||
/// correct values before and after the event.
|
||||
/// </summary>
|
||||
[Collection(ScriptWorkerCollection.Name)]
|
||||
[Trait("Category", "ScriptWorker")]
|
||||
public sealed class ContainmentTests
|
||||
{
|
||||
private const string Healthy = "return 7;";
|
||||
private const string Runaway = "while (true) { }";
|
||||
private const string Cooperative = "while (!ctx.CancellationToken.IsCancellationRequested) { } return 1;";
|
||||
|
||||
/// <summary>Spins for the number of milliseconds in input <c>ms</c>, so one source serves slow and fast runs.</summary>
|
||||
private const string Spin = "var ms = (int)ctx.GetTag(\"ms\").Value; var end = ctx.Now.AddMilliseconds(ms); while (ctx.Now < end) { } return ms;";
|
||||
|
||||
private static IReadOnlyDictionary<string, DataValueSnapshot> Ms(int ms) => Scripts.Input("ms", ms);
|
||||
|
||||
private static CancellationToken Ct => TestContext.Current.CancellationToken;
|
||||
|
||||
[Fact]
|
||||
public async Task A_healthy_script_runs_out_of_process_and_reads_its_inputs()
|
||||
{
|
||||
await using var runner = Scripts.Runner(Scripts.Options());
|
||||
var outcome = await runner.RunAsync(Scripts.Vt("vt", "return (int)ctx.GetTag(\"x\").Value + 1;",
|
||||
inputs: Scripts.Input("x", 41)), Ct);
|
||||
|
||||
outcome.Status.ShouldBe(ScriptRunStatus.Ok, outcome.Reason);
|
||||
outcome.Value.ShouldBe(42);
|
||||
runner.Shards[0].Current.ShouldNotBeNull().Pid.ShouldNotBe(Environment.ProcessId);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The headline case: <c>while(true){}</c> ignores the token. In process it would spin a host core forever; here
|
||||
/// the caller gets the usual timeout at T, the worker is killed at T + G, the host burns no CPU on it, and the
|
||||
/// sibling on the same shard keeps working. Mutation check: with the hard-deadline kill disabled the worker is
|
||||
/// never killed and the Exited wait times out.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task A_runaway_script_is_killed_at_its_hard_deadline_and_its_sibling_keeps_working()
|
||||
{
|
||||
var softT = TimeSpan.FromMilliseconds(400);
|
||||
var grace = TimeSpan.FromMilliseconds(500);
|
||||
await using var runner = Scripts.Runner(Scripts.Options(hardGrace: grace, quarantineAfterKills: 2));
|
||||
|
||||
(await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7);
|
||||
var worker = runner.Shards[0].Current.ShouldNotBeNull();
|
||||
|
||||
var sw = Stopwatch.StartNew();
|
||||
var outcome = await runner.RunAsync(Scripts.Vt("runaway", Runaway, softT), Ct);
|
||||
var softElapsed = sw.Elapsed;
|
||||
outcome.Status.ShouldBe(ScriptRunStatus.Failed);
|
||||
outcome.Reason!.ShouldStartWith("script timed out after ");
|
||||
softElapsed.ShouldBeLessThan(softT + TimeSpan.FromMilliseconds(400), "the caller is answered at the soft deadline");
|
||||
|
||||
var exit = await worker.Exited.WaitAsync(TimeSpan.FromSeconds(10), Ct);
|
||||
var killElapsed = sw.Elapsed;
|
||||
exit.Reason.ShouldBe(WorkerExitReason.KilledDeadline);
|
||||
exit.CulpritKey.ShouldBe(InProcessScriptRunner.ScopedKey(ScriptContextKind.VirtualTag, "runaway"));
|
||||
killElapsed.ShouldBeGreaterThanOrEqualTo(softT + grace - TimeSpan.FromMilliseconds(50));
|
||||
// Design budget is T + G + 100 ms; the slack here is for a loaded CI box (process reap + exit notification).
|
||||
killElapsed.ShouldBeLessThan(softT + grace + TimeSpan.FromMilliseconds(1500));
|
||||
|
||||
// No orphan in the host: an in-process while(true) would burn one host core for the whole window.
|
||||
var cpuBefore = Process.GetCurrentProcess().TotalProcessorTime;
|
||||
await Task.Delay(TimeSpan.FromMilliseconds(1500), Ct);
|
||||
var cpuUsed = Process.GetCurrentProcess().TotalProcessorTime - cpuBefore;
|
||||
cpuUsed.ShouldBeLessThan(TimeSpan.FromMilliseconds(1000), "the runaway's thread died with its worker");
|
||||
|
||||
// The sibling recovers on a fresh worker.
|
||||
(await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7);
|
||||
runner.Shards[0].Current.ShouldNotBeNull().Pid.ShouldNotBe(worker.Pid);
|
||||
|
||||
// A second kill quarantines (QuarantineAfterKills = 2); the third trigger is refused without running.
|
||||
var second = runner.Shards[0].Current!;
|
||||
(await runner.RunAsync(Scripts.Vt("runaway", Runaway, softT), Ct)).Status.ShouldBe(ScriptRunStatus.Failed);
|
||||
(await second.Exited.WaitAsync(TimeSpan.FromSeconds(10), Ct)).Reason.ShouldBe(WorkerExitReason.KilledDeadline);
|
||||
// The strike is recorded by the shard's exit handler, which runs asynchronously to the Exited completion.
|
||||
await Scripts.Eventually(() => runner.QuarantinedKeys(ScriptContextKind.VirtualTag).Count == 1,
|
||||
TimeSpan.FromSeconds(5), "the second kill strike");
|
||||
var refused = await runner.RunAsync(Scripts.Vt("runaway", Runaway, softT), Ct);
|
||||
refused.Reason!.ShouldStartWith(ScriptRunGuard.QuarantinedReasonPrefix);
|
||||
refused.Reason.ShouldContain("hard deadline");
|
||||
runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBe(["runaway"]);
|
||||
(await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7);
|
||||
|
||||
// Lifted by a source change (the operator fixed it)…
|
||||
(await runner.RunAsync(Scripts.Vt("runaway", "return 1;"), Ct)).Value.ShouldBe(1);
|
||||
runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBeEmpty();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_redeploy_lifts_a_kill_quarantine_for_its_seam_only()
|
||||
{
|
||||
var softT = TimeSpan.FromMilliseconds(300);
|
||||
await using var runner = Scripts.Runner(Scripts.Options(hardGrace: TimeSpan.FromMilliseconds(300), quarantineAfterKills: 1));
|
||||
|
||||
(await runner.RunAsync(Scripts.Vt("runaway", Runaway, softT), Ct)).Status.ShouldBe(ScriptRunStatus.Failed);
|
||||
await Scripts.Eventually(() => runner.QuarantinedKeys(ScriptContextKind.VirtualTag).Count == 1, TimeSpan.FromSeconds(10), "the kill quarantine");
|
||||
|
||||
runner.Reset(ScriptContextKind.Calculation);
|
||||
runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBe(["runaway"], "another seam's redeploy leaves it alone");
|
||||
runner.Reset(ScriptContextKind.VirtualTag);
|
||||
runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBeEmpty();
|
||||
}
|
||||
|
||||
/// <summary>A script that polls its token ends itself at T: no kill, no strike beyond the timeout (design §13).</summary>
|
||||
[Fact]
|
||||
public async Task A_cooperative_script_ends_at_its_soft_deadline_without_a_kill()
|
||||
{
|
||||
var softT = TimeSpan.FromMilliseconds(400);
|
||||
var grace = TimeSpan.FromMilliseconds(1500);
|
||||
await using var runner = Scripts.Runner(Scripts.Options(hardGrace: grace, quarantineAfterKills: 1));
|
||||
(await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7);
|
||||
var worker = runner.Shards[0].Current.ShouldNotBeNull();
|
||||
|
||||
var outcome = await runner.RunAsync(Scripts.Vt("polite", Cooperative, softT), Ct);
|
||||
outcome.Reason!.ShouldStartWith("script timed out after ");
|
||||
|
||||
await Task.Delay(softT + grace + TimeSpan.FromMilliseconds(500), Ct);
|
||||
worker.IsAlive.ShouldBeTrue("a script that ended itself must not cost the shard a restart");
|
||||
runner.Shards[0].Current!.Pid.ShouldBe(worker.Pid);
|
||||
runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBeEmpty("no kill strike (QuarantineAfterKills = 1)");
|
||||
(await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_second_trigger_while_the_first_still_runs_is_refused_not_queued()
|
||||
{
|
||||
await using var runner = Scripts.Runner(Scripts.Options());
|
||||
(await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0);
|
||||
|
||||
var first = runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(1500)), Ct);
|
||||
await Task.Delay(200, Ct);
|
||||
var second = await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct);
|
||||
second.Reason.ShouldBe(ScriptRunGuard.PreviousRunStillRunningReason);
|
||||
(await first).Value.ShouldBe(1500);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// <c>kill -9</c> of the worker: the evaluation in flight is a victim — Unavailable "script worker restarted"
|
||||
/// (BadResourceUnavailable), not a script fault — and, with no deadline crossed, a suspect that proves itself in
|
||||
/// the isolation worker and then returns to its shard.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task An_externally_killed_worker_makes_its_in_flight_script_a_suspect_that_recovers_in_isolation()
|
||||
{
|
||||
await using var runner = Scripts.Runner(Scripts.Options(probation: 2));
|
||||
(await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0);
|
||||
var worker = runner.Shards[0].Current.ShouldNotBeNull();
|
||||
|
||||
var inFlight = runner.RunAsync(Scripts.Vt("spin", Spin, TimeSpan.FromSeconds(20), Ms(5000)), Ct);
|
||||
await Task.Delay(300, Ct);
|
||||
Process.GetProcessById(worker.Pid).Kill();
|
||||
|
||||
var victim = await inFlight.WaitAsync(TimeSpan.FromSeconds(10), Ct);
|
||||
victim.Status.ShouldBe(ScriptRunStatus.Unavailable);
|
||||
victim.Reason.ShouldBe(ScriptRunOutcome.WorkerRestartedReason);
|
||||
(await worker.Exited).Reason.ShouldBe(WorkerExitReason.Crash);
|
||||
runner.SuspectKeys.ShouldBe([InProcessScriptRunner.ScopedKey(ScriptContextKind.VirtualTag, "spin")]);
|
||||
|
||||
(await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0);
|
||||
runner.Isolation.Current.ShouldNotBeNull("a suspect runs in the isolation worker");
|
||||
(await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0);
|
||||
runner.SuspectKeys.ShouldBeEmpty("two clean runs (probation = 2) return it to its shard");
|
||||
runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBeEmpty();
|
||||
|
||||
// The sibling recovers on a fresh shard worker.
|
||||
(await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7);
|
||||
runner.Shards[0].Current!.Pid.ShouldNotBe(worker.Pid);
|
||||
}
|
||||
|
||||
/// <summary>The isolation worker runs one evaluation at a time, so a crash there names exactly one key.</summary>
|
||||
[Fact]
|
||||
public async Task A_crash_in_the_isolation_worker_strikes_its_one_key_and_quarantines_it()
|
||||
{
|
||||
await using var runner = Scripts.Runner(Scripts.Options(quarantineAfterKills: 2));
|
||||
(await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0);
|
||||
|
||||
async Task<ScriptRunOutcome> RunAndKill(Func<WorkerConnection?> worker)
|
||||
{
|
||||
var run = runner.RunAsync(Scripts.Vt("spin", Spin, TimeSpan.FromSeconds(20), Ms(5000)), Ct);
|
||||
var live = await Scripts.Eventually(worker, TimeSpan.FromSeconds(15), "a live worker");
|
||||
await Scripts.Eventually(() => live.InFlightCount == 1, TimeSpan.FromSeconds(15), "the evaluation in flight");
|
||||
await Task.Delay(200, Ct);
|
||||
Process.GetProcessById(live.Pid).Kill();
|
||||
return await run.WaitAsync(TimeSpan.FromSeconds(10), Ct);
|
||||
}
|
||||
|
||||
(await RunAndKill(() => runner.Shards[0].Current)).Reason.ShouldBe(ScriptRunOutcome.WorkerRestartedReason);
|
||||
runner.SuspectKeys.Count.ShouldBe(1);
|
||||
runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBeEmpty("an ambiguous shard crash is not a strike");
|
||||
|
||||
(await RunAndKill(() => runner.Isolation.Current)).Reason.ShouldBe(ScriptRunOutcome.WorkerRestartedReason);
|
||||
runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBeEmpty("one isolation crash is one strike");
|
||||
(await RunAndKill(() => runner.Isolation.Current)).Reason.ShouldBe(ScriptRunOutcome.WorkerRestartedReason);
|
||||
await Scripts.Eventually(() => runner.QuarantinedKeys(ScriptContextKind.VirtualTag).Count == 1,
|
||||
TimeSpan.FromSeconds(5), "the isolated-crash quarantine");
|
||||
|
||||
var refused = await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct);
|
||||
refused.Reason!.ShouldStartWith(ScriptRunGuard.QuarantinedReasonPrefix);
|
||||
refused.Reason.ShouldContain("isolation worker");
|
||||
(await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7);
|
||||
}
|
||||
|
||||
/// <summary>A deadline kill blames exactly one key: the sibling in flight on the same worker is a victim, not a suspect.</summary>
|
||||
[Fact]
|
||||
public async Task A_deadline_kill_blames_only_its_culprit()
|
||||
{
|
||||
await using var runner = Scripts.Runner(Scripts.Options(hardGrace: TimeSpan.FromMilliseconds(300), quarantineAfterKills: 1));
|
||||
(await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0);
|
||||
(await runner.RunAsync(Scripts.Vt("warm", Healthy), Ct)).Value.ShouldBe(7);
|
||||
// Compile the runaway first, so both evaluations are on the worker at once.
|
||||
_ = runner.Shards[0].Current!;
|
||||
|
||||
var victim = runner.RunAsync(Scripts.Vt("spin", Spin, TimeSpan.FromSeconds(20), Ms(5000)), Ct);
|
||||
await Task.Delay(200, Ct);
|
||||
var culprit = await runner.RunAsync(Scripts.Vt("runaway", Runaway, TimeSpan.FromMilliseconds(300)), Ct);
|
||||
culprit.Reason!.ShouldStartWith("script timed out after ");
|
||||
|
||||
var victimOutcome = await victim.WaitAsync(TimeSpan.FromSeconds(15), Ct);
|
||||
victimOutcome.Status.ShouldBe(ScriptRunStatus.Unavailable);
|
||||
victimOutcome.Reason.ShouldBe(ScriptRunOutcome.WorkerRestartedReason);
|
||||
await Scripts.Eventually(() => runner.QuarantinedKeys(ScriptContextKind.VirtualTag).Count == 1, TimeSpan.FromSeconds(5), "the culprit's quarantine");
|
||||
runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBe(["runaway"]);
|
||||
runner.SuspectKeys.ShouldBeEmpty("a deadline kill is unambiguous: nobody else is suspected");
|
||||
(await runner.RunAsync(Scripts.Vt("spin", Spin, inputs: Ms(0)), Ct)).Value.ShouldBe(0);
|
||||
}
|
||||
|
||||
/// <summary>An allocation bomb dies in its worker (heap hard limit); the host's own memory does not move.</summary>
|
||||
[Fact]
|
||||
public async Task An_allocation_bomb_is_contained_in_its_worker()
|
||||
{
|
||||
await using var runner = Scripts.Runner(Scripts.Options(heapLimit: 128L * 1024 * 1024));
|
||||
(await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7);
|
||||
|
||||
GC.Collect();
|
||||
var hostBefore = GC.GetTotalMemory(forceFullCollection: true);
|
||||
var outcome = await runner.RunAsync(Scripts.Vt("bomb",
|
||||
"var hoard = new System.Collections.Generic.List<byte[]>(); while (true) { hoard.Add(new byte[1 << 20]); }",
|
||||
TimeSpan.FromSeconds(10)), Ct);
|
||||
var hostAfter = GC.GetTotalMemory(forceFullCollection: true);
|
||||
|
||||
outcome.Status.ShouldNotBe(ScriptRunStatus.Ok);
|
||||
(hostAfter - hostBefore).ShouldBeLessThan(32L * 1024 * 1024, "the hoard lived in the worker's heap, not the host's");
|
||||
await Scripts.Eventually(() => runner.Shards[0].Current, TimeSpan.FromSeconds(10), "a worker");
|
||||
(await runner.RunAsync(Scripts.Vt("sibling", Healthy), Ct)).Value.ShouldBe(7);
|
||||
}
|
||||
|
||||
/// <summary>Fail closed (design §14 #4): no worker means Bad, never an in-process run.</summary>
|
||||
[Fact]
|
||||
public async Task A_worker_that_cannot_start_reports_unavailable_and_never_runs_in_process()
|
||||
{
|
||||
var missing = Path.Combine(Path.GetTempPath(), "no-such-dir-" + Guid.NewGuid().ToString("N"), OutOfProcessScriptRunner.WorkerAssemblyFileName);
|
||||
await using var runner = Scripts.Runner(Scripts.Options(workerPath: missing, unavailableGrace: TimeSpan.FromSeconds(1)));
|
||||
|
||||
var outcome = await runner.RunAsync(Scripts.Vt("vt", "return 1;"), Ct);
|
||||
outcome.Status.ShouldBe(ScriptRunStatus.Unavailable);
|
||||
outcome.Reason.ShouldBe(ScriptRunOutcome.RunnerUnavailableReason);
|
||||
outcome.ToVirtualTagEvalResult().StatusCode.ShouldBe(ZB.MOM.WW.OtOpcUa.Commons.OpcUa.OpcUaStatus.BadResourceUnavailable);
|
||||
runner.Shards[0].ConsecutiveFailures.ShouldBeGreaterThan(0, "the failed start backs off");
|
||||
}
|
||||
|
||||
/// <summary>A worker whose pipe reaches end of stream exits by itself (the Linux "host died" path, design §3.3).</summary>
|
||||
[Fact]
|
||||
public async Task A_worker_exits_by_itself_when_its_pipe_closes()
|
||||
{
|
||||
await using var runner = Scripts.Runner(Scripts.Options());
|
||||
(await runner.RunAsync(Scripts.Vt("vt", Healthy), Ct)).Value.ShouldBe(7);
|
||||
var worker = runner.Shards[0].Current.ShouldNotBeNull();
|
||||
using var process = Process.GetProcessById(worker.Pid);
|
||||
|
||||
var sw = Stopwatch.StartNew();
|
||||
worker.SeverPipeForTest();
|
||||
await process.WaitForExitAsync(Ct).WaitAsync(TimeSpan.FromSeconds(5), Ct);
|
||||
sw.Elapsed.ShouldBeLessThan(TimeSpan.FromSeconds(2));
|
||||
var exit = await worker.Exited.WaitAsync(TimeSpan.FromSeconds(5), Ct);
|
||||
exit.ExitCode.ShouldBe(ScriptWorkerEnvironment.ExitPipeClosed, "it noticed the EOF; the host did not have to kill it");
|
||||
}
|
||||
|
||||
/// <summary>Design §6: the environment is cleared and repopulated from an allow-list. Positive control first.</summary>
|
||||
[Fact]
|
||||
public async Task The_worker_environment_carries_no_host_secrets()
|
||||
{
|
||||
const string apiKey = "ServerHistorian__ApiKey";
|
||||
const string configDb = "ConnectionStrings__ConfigDb";
|
||||
Environment.SetEnvironmentVariable(apiKey, "sekrit");
|
||||
Environment.SetEnvironmentVariable(configDb, "Server=x;Password=y");
|
||||
try
|
||||
{
|
||||
Environment.GetEnvironmentVariable(apiKey).ShouldBe("sekrit", "positive control: the host has it");
|
||||
await using var runner = Scripts.Runner(Scripts.Options());
|
||||
(await runner.RunAsync(Scripts.Vt("vt", Healthy), Ct)).Value.ShouldBe(7);
|
||||
|
||||
var diag = await runner.Shards[0].Current!.DiagnosticsAsync(TimeSpan.FromSeconds(10));
|
||||
diag.EnvironmentNames.ShouldNotContain(apiKey);
|
||||
diag.EnvironmentNames.ShouldNotContain(configDb);
|
||||
diag.EnvironmentNames.ShouldContain(ScriptWorkerEnvironment.Nonce, "the allow-listed launch variables do arrive");
|
||||
diag.EnvironmentNames.ShouldContain("DOTNET_GCHeapHardLimit");
|
||||
diag.WorkingDirectory.ShouldNotBe(Environment.CurrentDirectory);
|
||||
// An empty per-worker directory under the temp root (macOS reports /var as /private/var, so match the leaf).
|
||||
diag.WorkingDirectory.ShouldContain(Path.Combine("otopcua-scriptworker", "otopcua-sw-"));
|
||||
Directory.EnumerateFileSystemEntries(diag.WorkingDirectory).ShouldBeEmpty();
|
||||
}
|
||||
finally
|
||||
{
|
||||
Environment.SetEnvironmentVariable(apiKey, null);
|
||||
Environment.SetEnvironmentVariable(configDb, null);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Design §6: the host's cluster, database, OPC UA and secrets stacks are not in the worker.</summary>
|
||||
[Fact]
|
||||
public async Task The_worker_loads_none_of_the_host_stacks()
|
||||
{
|
||||
await using var runner = Scripts.Runner(Scripts.Options());
|
||||
(await runner.RunAsync(Scripts.Vt("vt", "return System.Math.Abs(-3);"), Ct)).Value.ShouldBe(3);
|
||||
var diag = await runner.Shards[0].Current!.DiagnosticsAsync(TimeSpan.FromSeconds(10));
|
||||
|
||||
diag.LoadedAssemblies.ShouldContain("ZB.MOM.WW.OtOpcUa.Core.Scripting", "positive control: the compile ran here");
|
||||
diag.LoadedAssemblies.ShouldContain("Microsoft.CodeAnalysis.CSharp");
|
||||
foreach (var prefix in ScriptWorkerEnvironment.ForbiddenAssemblyPrefixes)
|
||||
diag.LoadedAssemblies.ShouldNotContain(a => a.StartsWith(prefix, StringComparison.Ordinal), prefix);
|
||||
|
||||
// Not merely unloaded: Akka is not even deployable beside the worker.
|
||||
var folder = Path.GetDirectoryName(runner.WorkerPath)!;
|
||||
File.Exists(Path.Combine(folder, "ZB.MOM.WW.OtOpcUa.ScriptWorker.dll")).ShouldBeTrue("positive control");
|
||||
File.Exists(Path.Combine(folder, "Akka.dll")).ShouldBeFalse();
|
||||
File.Exists(Path.Combine(folder, "Grpc.Core.Api.dll")).ShouldBeFalse();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Script_log_lines_come_back_through_the_invocations_logger()
|
||||
{
|
||||
var sink = new CapturingSink();
|
||||
await using var runner = Scripts.Runner(Scripts.Options());
|
||||
(await runner.RunAsync(Scripts.Vt("vt", "ctx.Logger.Information(\"hello {N}\", 42); return 1;", logger: sink.CreateLogger()), Ct))
|
||||
.Value.ShouldBe(1);
|
||||
await Scripts.Eventually(() => sink.Rendered.Contains("hello 42"), TimeSpan.FromSeconds(5), "the forwarded log line");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task An_alarm_predicate_runs_out_of_process()
|
||||
{
|
||||
await using var runner = Scripts.Runner(Scripts.Options());
|
||||
var outcome = await runner.RunAsync(new ScriptInvocation("alarm-1", ScriptContextKind.AlarmPredicate,
|
||||
"return (double)ctx.GetTag(\"t\").Value > 50;", Scripts.Input("t", 75.0), TimeSpan.FromSeconds(5)), Ct);
|
||||
outcome.Status.ShouldBe(ScriptRunStatus.Ok, outcome.Reason);
|
||||
outcome.Value.ShouldBe(true);
|
||||
}
|
||||
|
||||
/// <summary>Design §14 #11: an unsupported type is an explicit failure, never a silent conversion.</summary>
|
||||
[Fact]
|
||||
public async Task Unsupported_input_and_result_types_are_explicit_failures()
|
||||
{
|
||||
await using var runner = Scripts.Runner(Scripts.Options());
|
||||
var result = await runner.RunAsync(Scripts.Vt("list", "return new System.Collections.Generic.List<int> { 1 };"), Ct);
|
||||
result.Status.ShouldBe(ScriptRunStatus.Failed);
|
||||
result.Reason!.ShouldStartWith("unsupported type: script returned a value of unsupported type System.Collections.Generic.List");
|
||||
|
||||
var input = await runner.RunAsync(Scripts.Vt("guid", "return 1;", inputs: Scripts.Input("g", Guid.NewGuid())), Ct);
|
||||
input.Status.ShouldBe(ScriptRunStatus.Failed);
|
||||
input.Reason!.ShouldStartWith("unsupported type: input 'g' has a value of unsupported type System.Guid");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
using System.IO.Pipes;
|
||||
using System.Security.Cryptography;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests;
|
||||
|
||||
/// <summary>lmxopcua#780 design §6 "Pipe hardening": only the launched process, echoing its nonce, is accepted.</summary>
|
||||
public sealed class HandshakeTests
|
||||
{
|
||||
private static CancellationToken Ct => TestContext.Current.CancellationToken;
|
||||
|
||||
private static async Task<Exception?> HandshakeWith(Func<Envelope, Envelope> reply, string expectedNonce = "right", int? expectedPid = 1234)
|
||||
{
|
||||
var name = "otopcua-sw-test-" + Convert.ToHexString(RandomNumberGenerator.GetBytes(6));
|
||||
await using var server = new NamedPipeServerStream(name, PipeDirection.InOut, 1, PipeTransmissionMode.Byte,
|
||||
PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly);
|
||||
await using var client = new NamedPipeClientStream(".", name, PipeDirection.InOut, PipeOptions.Asynchronous | PipeOptions.CurrentUserOnly);
|
||||
var accept = server.WaitForConnectionAsync(Ct);
|
||||
await client.ConnectAsync(TimeSpan.FromSeconds(5), Ct);
|
||||
await accept;
|
||||
|
||||
var rogue = Task.Run(async () =>
|
||||
{
|
||||
var hello = await ScriptWorkerFrameCodec.ReadAsync(client, ScriptWorkerFrameCodec.DefaultMaxFrameBytes, Ct);
|
||||
await ScriptWorkerFrameCodec.WriteAsync(client, reply(hello!), ScriptWorkerFrameCodec.DefaultMaxFrameBytes, Ct);
|
||||
await client.FlushAsync(Ct);
|
||||
}, Ct);
|
||||
|
||||
try
|
||||
{
|
||||
using var timeout = CancellationTokenSource.CreateLinkedTokenSource(Ct);
|
||||
timeout.CancelAfter(TimeSpan.FromSeconds(5));
|
||||
await WorkerConnection.HandshakeAsync(server, new Hello { Nonce = expectedNonce }, expectedNonce, expectedPid, timeout.Token);
|
||||
await rogue;
|
||||
return null;
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
return ex;
|
||||
}
|
||||
}
|
||||
|
||||
private static Envelope Ready(string nonce, int pid) => new()
|
||||
{
|
||||
ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion,
|
||||
Ready = new Ready { Nonce = nonce, Pid = pid },
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public async Task The_launched_process_echoing_its_nonce_is_accepted()
|
||||
=> (await HandshakeWith(_ => Ready("right", 1234))).ShouldBeNull("positive control");
|
||||
|
||||
[Fact]
|
||||
public async Task A_wrong_nonce_is_refused()
|
||||
=> (await HandshakeWith(_ => Ready("wrong", 1234))).ShouldBeOfType<ScriptWorkerProtocolException>()
|
||||
.Message.ShouldContain("nonce");
|
||||
|
||||
[Fact]
|
||||
public async Task A_client_that_is_not_the_launched_process_is_refused()
|
||||
=> (await HandshakeWith(_ => Ready("right", 999))).ShouldBeOfType<ScriptWorkerProtocolException>()
|
||||
.Message.ShouldContain("pid");
|
||||
|
||||
[Fact]
|
||||
public async Task Anything_but_ready_first_is_refused()
|
||||
=> (await HandshakeWith(_ => new Envelope
|
||||
{
|
||||
ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion,
|
||||
Heartbeat = new Heartbeat(),
|
||||
})).ShouldBeOfType<ScriptWorkerProtocolException>();
|
||||
|
||||
[Fact]
|
||||
public async Task The_hello_carries_the_nonce_to_echo()
|
||||
{
|
||||
string? seen = null;
|
||||
(await HandshakeWith(hello =>
|
||||
{
|
||||
seen = hello.Hello.Nonce;
|
||||
return Ready(hello.Hello.Nonce, 1234);
|
||||
})).ShouldBeNull();
|
||||
seen.ShouldBe("right");
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,95 @@
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// lmxopcua#780 design §13 "Parity": the same script, through <see cref="InProcessScriptRunner"/> (today's guard)
|
||||
/// and through the worker, gives the same status, reason, value and exact CLR value type.
|
||||
/// </summary>
|
||||
[Collection(ScriptWorkerCollection.Name)]
|
||||
[Trait("Category", "ScriptWorker")]
|
||||
public sealed class ParityTests
|
||||
{
|
||||
private static CancellationToken Ct => TestContext.Current.CancellationToken;
|
||||
|
||||
public static TheoryData<string, string> Cases() => new()
|
||||
{
|
||||
{ "arithmetic", "return 1 + 2;" },
|
||||
{ "string", "return \"a\" + ctx.GetTag(\"x\").Value;" },
|
||||
{ "throw", "throw new System.InvalidOperationException(\"boom\");" },
|
||||
{ "compile error", "return undefinedThing;" },
|
||||
{ "sandbox violation", "return System.IO.File.Exists(\"x\");" },
|
||||
{ "missing input status", "return ctx.GetTag(\"missing\").StatusCode;" },
|
||||
{ "input status", "return ctx.GetTag(\"x\").StatusCode;" },
|
||||
{ "source timestamp", "return ctx.GetTag(\"x\").SourceTimestampUtc;" },
|
||||
{ "clock", "return ctx.Now > new System.DateTime(2000, 1, 1);" },
|
||||
{ "null", "return null;" },
|
||||
{ "array", "return new double[] { 1.5, 2.5 };" },
|
||||
{ "cast of wrong type", "return (short)ctx.GetTag(\"x\").Value;" },
|
||||
};
|
||||
|
||||
private static readonly IReadOnlyDictionary<string, DataValueSnapshot> Inputs =
|
||||
new Dictionary<string, DataValueSnapshot>(StringComparer.Ordinal)
|
||||
{
|
||||
["x"] = new DataValueSnapshot(5, 0x40000000u, new DateTime(2026, 1, 2, 3, 4, 5, DateTimeKind.Utc), DateTime.UtcNow),
|
||||
};
|
||||
|
||||
[Theory]
|
||||
[MemberData(nameof(Cases))]
|
||||
public async Task In_and_out_of_process_agree(string name, string source)
|
||||
{
|
||||
using var inProcess = new InProcessScriptRunner();
|
||||
await using var outOfProcess = Scripts.Runner(Scripts.Options());
|
||||
var invocation = Scripts.Vt(name, source, inputs: Inputs);
|
||||
|
||||
var expected = await inProcess.RunAsync(invocation, Ct);
|
||||
var actual = await outOfProcess.RunAsync(invocation, Ct);
|
||||
|
||||
actual.Status.ShouldBe(expected.Status, $"{name}: {actual.Reason} vs {expected.Reason}");
|
||||
actual.Reason.ShouldBe(expected.Reason);
|
||||
Same(actual.Value, expected.Value, name);
|
||||
}
|
||||
|
||||
public static TheoryData<string, object> Typed() => new()
|
||||
{
|
||||
{ "bool", true }, { "sbyte", (sbyte)-2 }, { "byte", (byte)2 }, { "short", (short)-3 }, { "ushort", (ushort)3 },
|
||||
{ "int", -4 }, { "uint", 4u }, { "long", -5L }, { "ulong", 5UL }, { "float", 1.25f }, { "double", 2.5 },
|
||||
{ "string", "s" }, { "System.DateTime", new DateTime(2026, 11, 9, 0, 0, 0, DateTimeKind.Utc) },
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// Unboxing parity (pins design assumption A-7): <c>(T)ctx.GetTag("x").Value</c> works out of process for every
|
||||
/// tag type, which needs the exact boxed CLR type to survive the wire.
|
||||
/// </summary>
|
||||
[Theory]
|
||||
[MemberData(nameof(Typed))]
|
||||
public async Task Unboxing_to_the_exact_type_matches(string typeName, object value)
|
||||
{
|
||||
using var inProcess = new InProcessScriptRunner();
|
||||
await using var outOfProcess = Scripts.Runner(Scripts.Options());
|
||||
var invocation = Scripts.Vt("unbox-" + typeName, $"var v = ({typeName})ctx.GetTag(\"x\").Value; return v;",
|
||||
inputs: Scripts.Input("x", value));
|
||||
|
||||
var expected = await inProcess.RunAsync(invocation, Ct);
|
||||
var actual = await outOfProcess.RunAsync(invocation, Ct);
|
||||
expected.Status.ShouldBe(ScriptRunStatus.Ok, expected.Reason);
|
||||
actual.Status.ShouldBe(ScriptRunStatus.Ok, actual.Reason);
|
||||
Same(actual.Value, expected.Value, typeName);
|
||||
}
|
||||
|
||||
private static void Same(object? actual, object? expected, string because)
|
||||
{
|
||||
if (expected is null)
|
||||
{
|
||||
actual.ShouldBeNull(because);
|
||||
return;
|
||||
}
|
||||
actual.ShouldNotBeNull(because);
|
||||
actual.GetType().ShouldBe(expected.GetType(), because);
|
||||
if (expected is Array array) ((Array)actual).Cast<object?>().ShouldBe(array.Cast<object?>(), because);
|
||||
else actual.ShouldBe(expected, because);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,198 @@
|
||||
using System.Buffers.Binary;
|
||||
using Google.Protobuf;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.V1;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests;
|
||||
|
||||
/// <summary>lmxopcua#780 design §13 — frame codec and the CLR ↔ Variant mapping.</summary>
|
||||
public sealed class ProtocolTests
|
||||
{
|
||||
public static IEnumerable<object?[]> Values()
|
||||
{
|
||||
object?[] values =
|
||||
[
|
||||
null, true, false, (sbyte)-5, (byte)250, (short)-3, (ushort)7, -42, 42u, long.MinValue, ulong.MaxValue,
|
||||
1.5f, double.NaN, 3.25, "text", string.Empty, new DateTime(2026, 11, 9, 1, 2, 3, DateTimeKind.Utc),
|
||||
new[] { 1, 2, 3 }, new short[] { -1, 1 }, new[] { "a", "b" }, new[] { true, false }, Array.Empty<double>(),
|
||||
new[] { new DateTime(2020, 1, 1, 0, 0, 0, DateTimeKind.Utc) }, (short)0, 0, 0L, 0.0,
|
||||
];
|
||||
return values.Select(v => new[] { v });
|
||||
}
|
||||
|
||||
/// <summary>The exact boxed CLR type survives (design §4.4): <c>(short)ctx.GetTag(x).Value</c> must keep working.</summary>
|
||||
[Theory]
|
||||
[MemberData(nameof(Values))]
|
||||
public void Every_type_tag_round_trips_with_its_exact_clr_type(object? value)
|
||||
{
|
||||
VariantCodec.TryEncode(value, out var variant, out var unsupported).ShouldBeTrue();
|
||||
unsupported.ShouldBeNull();
|
||||
|
||||
// Through the wire, not just the object: a payload the serializer drops must still decode.
|
||||
var wire = Variant.Parser.ParseFrom(variant.ToByteArray());
|
||||
var decoded = VariantCodec.Decode(wire);
|
||||
|
||||
if (value is null)
|
||||
{
|
||||
decoded.ShouldBeNull();
|
||||
return;
|
||||
}
|
||||
decoded.ShouldNotBeNull();
|
||||
decoded.GetType().ShouldBe(value.GetType());
|
||||
if (value is Array expected)
|
||||
((Array)decoded).Cast<object?>().ShouldBe(expected.Cast<object?>());
|
||||
else if (value is double d && double.IsNaN(d))
|
||||
double.IsNaN((double)decoded).ShouldBeTrue();
|
||||
else
|
||||
decoded.ShouldBe(value);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Unsupported_types_are_refused_not_converted()
|
||||
{
|
||||
foreach (var value in new object[] { new List<int> { 1 }, 1.5m, Guid.NewGuid(), new { A = 1 }, new int[1, 1], new object[] { 1 } })
|
||||
{
|
||||
VariantCodec.TryEncode(value, out _, out var type).ShouldBeFalse(value.GetType().Name);
|
||||
type.ShouldNotBeNullOrEmpty();
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Local_datetime_crosses_as_utc()
|
||||
{
|
||||
var local = new DateTime(2026, 6, 1, 12, 0, 0, DateTimeKind.Local);
|
||||
VariantCodec.TryEncode(local, out var variant, out _).ShouldBeTrue();
|
||||
var decoded = (DateTime)VariantCodec.Decode(variant)!;
|
||||
decoded.Kind.ShouldBe(DateTimeKind.Utc);
|
||||
decoded.ShouldBe(local.ToUniversalTime());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void A_payload_that_contradicts_its_tag_is_a_protocol_fault()
|
||||
{
|
||||
Should.Throw<ScriptWorkerProtocolException>(() => VariantCodec.Decode(new Variant { Type = Variant.Types.TypeTag.Int16, S = "x" }));
|
||||
Should.Throw<ScriptWorkerProtocolException>(() => VariantCodec.Decode(new Variant { Type = (Variant.Types.TypeTag)99 }));
|
||||
Should.Throw<OverflowException>(() => VariantCodec.Decode(new Variant { Type = Variant.Types.TypeTag.Int16, I = 1 << 20 }));
|
||||
}
|
||||
|
||||
private static Envelope Sample() => new()
|
||||
{
|
||||
ProtocolVersion = ScriptWorkerFrameCodec.ProtocolVersion,
|
||||
Seq = 7,
|
||||
Heartbeat = new Heartbeat { Inflight = 3, HeapBytes = 1234 },
|
||||
};
|
||||
|
||||
[Fact]
|
||||
public async Task A_frame_round_trips()
|
||||
{
|
||||
using var stream = new MemoryStream();
|
||||
await ScriptWorkerFrameCodec.WriteAsync(stream, Sample(), 1024, TestContext.Current.CancellationToken);
|
||||
stream.Position = 0;
|
||||
var read = await ScriptWorkerFrameCodec.ReadAsync(stream, 1024, TestContext.Current.CancellationToken);
|
||||
read.ShouldBe(Sample());
|
||||
(await ScriptWorkerFrameCodec.ReadAsync(stream, 1024, TestContext.Current.CancellationToken)).ShouldBeNull("a clean end of stream at a frame boundary");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Oversize_frames_are_refused_on_write_and_on_read()
|
||||
{
|
||||
using var stream = new MemoryStream();
|
||||
await Should.ThrowAsync<ScriptWorkerProtocolException>(async () =>
|
||||
await ScriptWorkerFrameCodec.WriteAsync(stream, Sample(), 4, TestContext.Current.CancellationToken));
|
||||
stream.Length.ShouldBe(0, "nothing is written for a refused frame");
|
||||
|
||||
await ScriptWorkerFrameCodec.WriteAsync(stream, Sample(), 1024, TestContext.Current.CancellationToken);
|
||||
stream.Position = 0;
|
||||
await Should.ThrowAsync<ScriptWorkerProtocolException>(async () =>
|
||||
await ScriptWorkerFrameCodec.ReadAsync(stream, 4, TestContext.Current.CancellationToken));
|
||||
}
|
||||
|
||||
public static TheoryData<string, byte[]> BadFrames()
|
||||
{
|
||||
var good = Sample().ToByteArray();
|
||||
var header = new byte[4];
|
||||
BinaryPrimitives.WriteInt32LittleEndian(header, good.Length);
|
||||
var wrongVersion = new Envelope { ProtocolVersion = 99, Heartbeat = new Heartbeat() }.ToByteArray();
|
||||
var noBody = new Envelope { ProtocolVersion = 1, Seq = 1 }.ToByteArray();
|
||||
return new TheoryData<string, byte[]>
|
||||
{
|
||||
{ "zero length", new byte[] { 0, 0, 0, 0 } },
|
||||
{ "negative length", new byte[] { 0xFF, 0xFF, 0xFF, 0xFF } },
|
||||
{ "truncated header", new byte[] { 5, 0 } },
|
||||
{ "truncated body", [.. header, .. good[..^1]] },
|
||||
{ "garbage body", [5, 0, 0, 0, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF] },
|
||||
{ "wrong protocol version", Frame(wrongVersion) },
|
||||
{ "no body", Frame(noBody) },
|
||||
};
|
||||
}
|
||||
|
||||
private static byte[] Frame(byte[] body)
|
||||
{
|
||||
var header = new byte[4];
|
||||
BinaryPrimitives.WriteInt32LittleEndian(header, body.Length);
|
||||
return [.. header, .. body];
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[MemberData(nameof(BadFrames))]
|
||||
public async Task Malformed_frames_are_protocol_faults(string why, byte[] bytes)
|
||||
{
|
||||
using var stream = new MemoryStream(bytes);
|
||||
await Should.ThrowAsync<ScriptWorkerProtocolException>(async () =>
|
||||
await ScriptWorkerFrameCodec.ReadAsync(stream, 1024, TestContext.Current.CancellationToken), why);
|
||||
}
|
||||
|
||||
/// <summary>Random bytes never escape as anything but a protocol fault (or a clean end of stream).</summary>
|
||||
[Fact]
|
||||
public async Task Fuzzed_input_only_ever_yields_a_protocol_fault()
|
||||
{
|
||||
var random = new Random(780);
|
||||
for (var i = 0; i < 3000; i++)
|
||||
{
|
||||
var bytes = new byte[random.Next(0, 64)];
|
||||
random.NextBytes(bytes);
|
||||
if (bytes.Length >= 4 && random.Next(2) == 0)
|
||||
BinaryPrimitives.WriteInt32LittleEndian(bytes, random.Next(1, bytes.Length));
|
||||
using var stream = new MemoryStream(bytes);
|
||||
try
|
||||
{
|
||||
while (await ScriptWorkerFrameCodec.ReadAsync(stream, 1024, TestContext.Current.CancellationToken) is { } envelope)
|
||||
{
|
||||
// A frame that parses must still decode its variants without an unexpected exception type.
|
||||
if (envelope.BodyCase == Envelope.BodyOneofCase.Results)
|
||||
foreach (var r in envelope.Results.Items)
|
||||
try { VariantCodec.Decode(r.Value); } catch (Exception ex) when (ex is ScriptWorkerProtocolException or OverflowException) { }
|
||||
}
|
||||
}
|
||||
catch (ScriptWorkerProtocolException)
|
||||
{
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Backoff_doubles_from_250ms_to_a_30s_cap_with_20_percent_jitter()
|
||||
{
|
||||
WorkerShard.NextBackoff(1, 0.5).ShouldBe(TimeSpan.FromMilliseconds(250));
|
||||
WorkerShard.NextBackoff(2, 0.5).ShouldBe(TimeSpan.FromMilliseconds(500));
|
||||
WorkerShard.NextBackoff(3, 0.5).ShouldBe(TimeSpan.FromSeconds(1));
|
||||
WorkerShard.NextBackoff(20, 0.5).ShouldBe(TimeSpan.FromSeconds(30));
|
||||
WorkerShard.NextBackoff(1, 0).ShouldBe(TimeSpan.FromMilliseconds(200));
|
||||
WorkerShard.NextBackoff(1, 0.9999999).TotalMilliseconds.ShouldBe(300, tolerance: 0.01);
|
||||
WorkerShard.NextBackoff(40, 1).ShouldBe(TimeSpan.FromSeconds(36));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Shard_placement_is_stable_and_in_range()
|
||||
{
|
||||
// FNV-1a, not string.GetHashCode (randomised per process): a script stays on its shard across restarts.
|
||||
OutOfProcessScriptRunner.ShardIndex("1:vt-1", 2).ShouldBe(OutOfProcessScriptRunner.ShardIndex("1:vt-1", 2));
|
||||
var spread = Enumerable.Range(0, 400).Select(i => OutOfProcessScriptRunner.ShardIndex($"1:vt-{i}", 4)).ToArray();
|
||||
spread.ShouldAllBe(i => i >= 0 && i < 4);
|
||||
spread.Distinct().Count().ShouldBe(4);
|
||||
OutOfProcessScriptRunner.ShardIndex("abc", 1000).ShouldBe((int)(0x1A47E90Bu % 1000));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,88 @@
|
||||
using System.Collections.Concurrent;
|
||||
using System.Diagnostics;
|
||||
using Serilog;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.Calculation;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// lmxopcua#780 — the whole path a client sees: a real Calculation driver whose seam runs out of process on the real
|
||||
/// worker. A script failure publishes <c>BadInternalError</c> as today; the worker dying under a script publishes
|
||||
/// <c>BadResourceUnavailable</c> (0x80040000), the ScadaBridge-visible code; and the tag recovers Good afterwards.
|
||||
/// </summary>
|
||||
[Collection(ScriptWorkerCollection.Name)]
|
||||
[Trait("Category", "ScriptWorker")]
|
||||
public sealed class SeamEndToEndTests
|
||||
{
|
||||
private const uint StatusGood = 0u;
|
||||
private const uint StatusBadInternalError = 0x80020000u;
|
||||
private const uint StatusBadResourceUnavailable = 0x80040000u;
|
||||
private const string Tag = "calc/e/spin";
|
||||
|
||||
/// <summary>Spins for <c>src/ms</c> milliseconds, throws when it is negative, and returns it otherwise.</summary>
|
||||
private const string Script =
|
||||
"var ms = (double)ctx.GetTag(\"src/ms\").Value; if (ms < 0) throw new System.InvalidOperationException(\"negative\"); " +
|
||||
"var end = ctx.Now.AddMilliseconds(ms); while (ctx.Now < end) { } return ms;";
|
||||
|
||||
private static CancellationToken Ct => TestContext.Current.CancellationToken;
|
||||
|
||||
private static async Task Settle(CalculationDriver driver)
|
||||
{
|
||||
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(60);
|
||||
while (DateTime.UtcNow < deadline && !driver.IsEvaluationIdle) await Task.Delay(10);
|
||||
driver.IsEvaluationIdle.ShouldBeTrue();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_calculation_tag_publishes_BadResourceUnavailable_only_when_the_worker_died_under_it()
|
||||
{
|
||||
await using var runner = Scripts.Runner(Scripts.Options());
|
||||
var json = $"{{\"scriptId\":\"s\",\"changeTriggered\":true,\"scriptSource\":{System.Text.Json.JsonSerializer.Serialize(Script)}}}";
|
||||
var driver = new CalculationDriver(
|
||||
new CalculationDriverOptions { RawTags = [new RawTagEntry(Tag, json, WriteIdempotent: false)], RunTimeout = TimeSpan.FromSeconds(20), ScriptRunner = runner },
|
||||
"calc-drv", new ScriptRootLogger(new LoggerConfiguration().CreateLogger()));
|
||||
var published = new ConcurrentQueue<DataChangeEventArgs>();
|
||||
driver.OnDataChange += (_, e) => published.Enqueue(e);
|
||||
await driver.InitializeAsync("{}", Ct);
|
||||
|
||||
driver.OnDependencyValue("src/ms", 0.0, StatusGood, DateTime.UtcNow);
|
||||
await Settle(driver);
|
||||
published.TryDequeue(out var good).ShouldBeTrue();
|
||||
good!.Snapshot.StatusCode.ShouldBe(StatusGood);
|
||||
good.Snapshot.Value.ShouldBe(0.0);
|
||||
|
||||
// A script fault: BadInternalError, exactly as in process.
|
||||
driver.OnDependencyValue("src/ms", -1.0, StatusGood, DateTime.UtcNow);
|
||||
await Settle(driver);
|
||||
published.TryDequeue(out var threw).ShouldBeTrue();
|
||||
threw!.Snapshot.StatusCode.ShouldBe(StatusBadInternalError);
|
||||
|
||||
driver.OnDependencyValue("src/ms", 1.0, StatusGood, DateTime.UtcNow);
|
||||
await Settle(driver);
|
||||
published.TryDequeue(out var back).ShouldBeTrue();
|
||||
back!.Snapshot.StatusCode.ShouldBe(StatusGood);
|
||||
|
||||
// The worker dies under a run: the runner, not the script, failed.
|
||||
driver.OnDependencyValue("src/ms", 5000.0, StatusGood, DateTime.UtcNow);
|
||||
var worker = await Scripts.Eventually(() => runner.Shards[0].Current is { InFlightCount: 1 } w ? w : null,
|
||||
TimeSpan.FromSeconds(10), "the run in flight");
|
||||
await Task.Delay(200, Ct);
|
||||
Process.GetProcessById(worker.Pid).Kill();
|
||||
await Settle(driver);
|
||||
published.TryDequeue(out var unavailable).ShouldBeTrue();
|
||||
unavailable!.Snapshot.StatusCode.ShouldBe(StatusBadResourceUnavailable);
|
||||
unavailable.Snapshot.Value.ShouldBe(1.0, "the last good value rides out with the Bad status");
|
||||
|
||||
// Recovery: the next trigger runs (in the isolation worker, as a suspect) and publishes Good.
|
||||
driver.OnDependencyValue("src/ms", 2.0, StatusGood, DateTime.UtcNow);
|
||||
await Settle(driver);
|
||||
published.TryDequeue(out var recovered).ShouldBeTrue();
|
||||
recovered!.Snapshot.StatusCode.ShouldBe(StatusGood);
|
||||
recovered.Snapshot.Value.ShouldBe(2.0);
|
||||
await driver.ShutdownAsync(Ct);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
using System.Collections.Concurrent;
|
||||
using Serilog;
|
||||
using Serilog.Core;
|
||||
using Serilog.Events;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests;
|
||||
|
||||
/// <summary>Collects Serilog events in memory.</summary>
|
||||
internal sealed class CapturingSink : ILogEventSink
|
||||
{
|
||||
public ConcurrentQueue<LogEvent> Events { get; } = new();
|
||||
|
||||
public void Emit(LogEvent logEvent) => Events.Enqueue(logEvent);
|
||||
|
||||
public ILogger CreateLogger() => new LoggerConfiguration().MinimumLevel.Verbose().WriteTo.Sink(this).CreateLogger();
|
||||
|
||||
public IEnumerable<string> Rendered => Events.Select(e => e.RenderMessage());
|
||||
}
|
||||
|
||||
internal static class Scripts
|
||||
{
|
||||
public static readonly IReadOnlyDictionary<string, DataValueSnapshot> NoInputs =
|
||||
new Dictionary<string, DataValueSnapshot>(StringComparer.Ordinal);
|
||||
|
||||
public static IReadOnlyDictionary<string, DataValueSnapshot> Input(string path, object? value, uint status = 0)
|
||||
=> new Dictionary<string, DataValueSnapshot>(StringComparer.Ordinal)
|
||||
{
|
||||
[path] = new DataValueSnapshot(value, status, DateTime.UtcNow, DateTime.UtcNow),
|
||||
};
|
||||
|
||||
public static ScriptInvocation Vt(string key, string source, TimeSpan? timeout = null,
|
||||
IReadOnlyDictionary<string, DataValueSnapshot>? inputs = null, ILogger? logger = null)
|
||||
=> new(key, ScriptContextKind.VirtualTag, source, inputs ?? NoInputs, timeout ?? TimeSpan.FromSeconds(5), logger);
|
||||
|
||||
/// <summary>Options for a single-shard runner with short budgets, so the containment tests run in seconds.</summary>
|
||||
public static ScriptingOptions Options(int shards = 1, TimeSpan? hardGrace = null, int quarantineAfterKills = 2,
|
||||
int probation = 20, TimeSpan? unavailableGrace = null, string? workerPath = null, long heapLimit = 256L * 1024 * 1024)
|
||||
=> new()
|
||||
{
|
||||
Runner = ScriptRunnerMode.OutOfProcess,
|
||||
QuarantineAfterConsecutiveTimeouts = 100,
|
||||
Worker = new ScriptWorkerOptions
|
||||
{
|
||||
Shards = shards,
|
||||
HardGrace = hardGrace ?? TimeSpan.FromMilliseconds(500),
|
||||
QuarantineAfterKills = quarantineAfterKills,
|
||||
IsolationProbationRuns = probation,
|
||||
UnavailableGrace = unavailableGrace ?? TimeSpan.FromSeconds(30),
|
||||
ExecutablePath = workerPath,
|
||||
HeapLimitBytes = heapLimit,
|
||||
},
|
||||
};
|
||||
|
||||
public static OutOfProcessScriptRunner Runner(ScriptingOptions options, ILogger? log = null)
|
||||
=> new(options, log ?? new LoggerConfiguration().CreateLogger(), null,
|
||||
heartbeatInterval: TimeSpan.FromSeconds(1), loadTimeout: TimeSpan.FromSeconds(60));
|
||||
|
||||
public static async Task<T> Eventually<T>(Func<T?> probe, TimeSpan timeout, string what) where T : class
|
||||
{
|
||||
var deadline = DateTime.UtcNow + timeout;
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
if (probe() is { } value) return value;
|
||||
await Task.Delay(20);
|
||||
}
|
||||
throw new TimeoutException($"timed out waiting for {what}");
|
||||
}
|
||||
|
||||
public static async Task Eventually(Func<bool> probe, TimeSpan timeout, string what)
|
||||
{
|
||||
var deadline = DateTime.UtcNow + timeout;
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
if (probe()) return;
|
||||
await Task.Delay(20);
|
||||
}
|
||||
throw new TimeoutException($"timed out waiting for {what}");
|
||||
}
|
||||
}
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<IsPackable>false</IsPackable>
|
||||
<IsTestProject>true</IsTestProject>
|
||||
<RootNamespace>ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests</RootNamespace>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="xunit.v3"/>
|
||||
<PackageReference Include="Shouldly"/>
|
||||
<PackageReference Include="Microsoft.NET.Test.Sdk"/>
|
||||
<PackageReference Include="xunit.runner.visualstudio">
|
||||
<PrivateAssets>all</PrivateAssets>
|
||||
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
|
||||
</PackageReference>
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\..\src\Core\ZB.MOM.WW.OtOpcUa.Core.Scripting\ZB.MOM.WW.OtOpcUa.Core.Scripting.csproj"/>
|
||||
<ProjectReference Include="..\..\..\src\Core\ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol\ZB.MOM.WW.OtOpcUa.Core.Scripting.Protocol.csproj"/>
|
||||
<ProjectReference Include="..\..\..\src\Core\ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess\ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess.csproj"/>
|
||||
<!-- End-to-end seam test: a real Calculation driver over the real worker. -->
|
||||
<ProjectReference Include="..\..\..\src\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Calculation\ZB.MOM.WW.OtOpcUa.Driver.Calculation.csproj"/>
|
||||
</ItemGroup>
|
||||
|
||||
<!-- The real worker, built and placed in script-worker/ beside the tests, exactly as beside the Host. -->
|
||||
<Import Project="..\..\..\src\Server\ZB.MOM.WW.OtOpcUa.ScriptWorker\build\ScriptWorker.Deploy.targets"/>
|
||||
|
||||
</Project>
|
||||
Reference in New Issue
Block a user