dohertj2/lmxopcua
1
0
Fork 0
Code Issues 54 Pull Requests Actions Packages Projects Releases Wiki Activity

docs(scripting): design an out-of-process, pre-emptible script runner (lmxopcua#750)
v2-ci / unit-tests (push) Has been cancelled
Details

Browse Source 
Design only. Documents why today's in-process ScriptRunGuard can abandon but
not stop a runaway script (file:line evidence), and proposes a sharded
worker pool with an isolation worker under a ScriptRunnerActor, a
named-pipe + protobuf frame protocol carrying full input snapshots,
soft/hard deadlines with process kill, kill attribution and quarantine,
worker sandboxing (scrubbed env, pinned assemblies, Job Object, heap
limit), an HMAC'd content-addressed PE cache, performance budgets,
failure-quality semantics, a flag-gated InProcess/Shadow/OutOfProcess
rollout, a test plan, open decisions and an assumptions register.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 17:04:12 -05:00
parent 94926eb0d7
commit 9e4a26f888
1 changed files with 846 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/plans/2026-11-09-out-of-process-script-runner-design.md
+846
View File
@@ -0,0 +1,846 @@
# Out-of-process, pre-emptible script runner — design
**Date:** 2026-11-09 · **Status:** proposed (design only — no code in this change) · **Issue:** #750 · **Related:** #734 (Lane E remediation)
This document designs a script runner that can actually **stop** a user script, instead of abandoning it.
It covers the three script seams that run operator-authored C# on a driver node: virtual tags, Calculation
tags and scripted-alarm predicates.
Claims about today's code carry `file:line` evidence against `origin/master` @ `94926eb0`. Anything not
read from the code is marked **[A-n]** and listed in §15.
---
## 1. Today: in-process execution
### 1.1 One compile pipeline, three seams
All three seams share `Core.Scripting`:
- **Compile.** `ScriptEvaluator<TContext,TResult>.Compile` wraps the user source in a synthesized
`CompiledScript.Run(ScriptGlobals<TContext>)`, compiles it with Roslyn against the `ScriptSandbox` reference
set, runs `ForbiddenTypeAnalyzer`, emits PE bytes to a `MemoryStream` and loads them into a collectible ALC
(`src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ScriptEvaluator.cs:125` emit, `:139` `LoadFromStream`).
- **Cache.** `CompiledScriptCache` is keyed by SHA-256 of the source (`CompiledScriptCache.cs:157`). It is
**in memory only**, so every process start recompiles every script on first use.
- **Execution.** It is synchronous: `ScriptEvaluator.RunAsync` invokes the bound delegate inline and returns
`Task.FromResult` (`ScriptEvaluator.cs:178-195`). The script-visible `ctx.CancellationToken` is an
`AsyncLocal` set for the duration of the call (`ScriptEvaluator.cs:191`,
`Core.Scripting.Abstractions/ScriptContext.cs:96-98`).
- **The context is a pure read cache.** `VirtualTagContext.GetTag` and `AlarmPredicateContext.GetTag` serve
only from a dictionary handed in by the caller (`VirtualTagContext.cs:50`, `AlarmPredicateContext.cs:40`).
`SetVirtualTag` is dropped in both production single-tag evaluators
(`Host/Engines/RoslynVirtualTagEvaluator.cs:129-131`, `Driver.Calculation/CalculationEvaluator.cs:185-188`)
and throws in alarm predicates (`AlarmPredicateContext.cs:50`). Inputs are known statically, because
`DependencyExtractor` rejects non-literal `GetTag` paths at publish.
This matters most for the design (§4.1). **A script needs nothing from the host while it runs**: the
inputs go in, one value comes out, and log lines are a side channel.
| Seam | Where it runs | Budget | Serialisation | Evidence |
|---|---|---|---|---|
| Virtual tag | `VirtualTagActor` message handler calls `IVirtualTagEvaluator.Evaluate` **synchronously** on the actor's dispatcher thread; `RoslynVirtualTagEvaluator` blocks with `.GetAwaiter().GetResult()` | 2 s | per actor (one actor per VT) | `Runtime/VirtualTags/VirtualTagActor.cs:276`; `RoslynVirtualTagEvaluator.cs:70`, `:174-175`; singleton `Host/Program.cs:393-398` |
| Calculation tag | async pump in `CalculationDriver`, one evaluation at a time per driver instance | 2 s (`runTimeoutMs`) | `_evalGate` = `SemaphoreSlim(1,1)` per driver | `CalculationDriver.cs:99`, `:523-556`, `:628`; `CalculationDriverOptions.cs:13`; `CalculationEvaluator.cs:232-233` |
| Scripted alarm | `ScriptedAlarmEngine.ReevaluateAsync` walks the dirty set **serially** under one gate | **250 ms** (the engine is built without `scriptTimeout`, so it takes `TimedScriptEvaluator.DefaultTimeout`) | `_evalGate` per engine | `ScriptedAlarmEngine.cs:204`, `:257`, `:743-765`, `:833-835`; `Runtime/Drivers/DriverHostActor.cs:1251-1252` |
Both value seams skip Roslyn for the mirror shape `return ctx.GetTag("X").Value;` (`PassthroughScript.TryMatch`,
`RoslynVirtualTagEvaluator.cs:111-118`, `CalculationEvaluator.cs:167-176`).
### 1.2 Timeout and cancellation model
Every compiled run in production goes through `ScriptRunGuard.RunAsync` (`Core.Scripting/ScriptRunGuard.cs:170-304`).
There are three guard instances per seam owner: one per `RoslynVirtualTagEvaluator`
(`RoslynVirtualTagEvaluator.cs:71`), one per `CalculationEvaluator` (`CalculationEvaluator.cs:88`) and one
per `ScriptedAlarmEngine` (`ScriptedAlarmEngine.cs:258`).
1. **Single flight per key.** A CAS on `state.InFlight` refuses a second run while the previous one, an
orphan included, still holds the slot. The caller gets `ScriptRunRefusedException` →
Bad `"previous evaluation still running"` (`ScriptRunGuard.cs:188-219`).
2. **The run is pushed to the pool** (`Task.Run`, `:231-249`), so a CPU-bound script cannot run inline and
complete before the timeout is registered. The budget clock starts only when the pool picks the run up
(`:268`).
3. **The wait is `runTask.WaitAsync(timeout, ct)`** (`:273`). On timeout the guard cancels the run token
(`:279`). Only a script that polls `ctx.CancellationToken` can see that.
4. **Quarantine** happens on any of three triggers:
- N consecutive timeouts (`:332-341`, default 3);
- a refusal behind an orphan older than `StuckOrphanQuarantineFactor` × the timeout (`:199-216`,
default 10×);
- an allocation breach, where `GC.GetAllocatedBytesForCurrentThread` is sampled before and after the
delegate (`:235`, `:241`, `:262-263`, `:296-302`; default 64 MB, `ScriptingOptions.cs`).
A quarantine lifts on a source change or a redeploy (`ResetAll`, called from `ClearCompiledScripts` at
`RoslynVirtualTagEvaluator.cs:212-218` and from `ScriptedAlarmEngine.LoadAsync`).
5. **What the operator sees.**
- A VT or Calc failure publishes `BadInternalError` with the last-known value, once per Good→Bad
transition (`VirtualTagActor.cs:384-392`, `CalculationDriver.cs:640-647`).
- An alarm whose predicate did not complete keeps its frozen condition state, annotated
`BadInternalError` (`ScriptedAlarmEngine.cs:895-907`).
`TimedScriptEvaluator` (`TimedScriptEvaluator.cs:80-83`) is the older wrapper. Its own remarks say it is now
used only by single-shot callers (`:18-27`).
### 1.3 Why a runaway script cannot be pre-empted
.NET has no supported way to stop a managed thread. `Thread.Abort` is gone in .NET Core, and the guard's
`WaitAsync` returns control to the **caller**; it does not stop the run. Concretely:
| Failure | What happens today | Evidence |
|---|---|---|
| `while(true){}` (ignores the token) | The pool thread spins **forever**, holding one core and its collectible ALC. The guard bounds this to one orphan per key and then quarantines, but the thread keeps spinning until the process restarts. N such scripts peg N cores. | `ScriptRunGuard.cs:17-22` remarks; `ScriptRunGuardTests.cs:170-205` asserts *exactly one* orphan stays in flight (`InFlightCount.ShouldBe(1)`) |
| Fast allocation (`new byte[int.MaxValue]` in a loop, or a growing list) | The sample is taken **after** the delegate returns, so the process can OOM first. The whole server dies, including OPC UA sessions and every driver. | `ScriptRunGuard.cs:39-43`; `docs/VirtualTags.md:40`; review `docs/reviews/2026-10-21/05-core.md:100-105` |
| Unbounded recursion (a local function or a self-referencing `Func<>`; neither is a class member, so `EnforceSingleRunMember` does not see it) | `StackOverflowException` cannot be caught in .NET and **terminates the host process**. **[A-1]** I did not reproduce this; it follows from runtime semantics, and no analyzer rule targets recursion (`git grep -i "recursion\|StackOverflow"` in `Core.Scripting` finds nothing). | `ScriptEvaluator.cs:228-281` (member check only) |
| Collectible-ALC reclaim of a stuck script | `ClearCompiledScripts` disposes the evaluator, but an ALC cannot be collected while a thread is executing its code, so a stuck orphan pins its assembly until restart. **[A-2]** This is .NET ALC semantics, not observed here. | `CompiledScriptCache.Clear`; `ScriptEvaluator.Dispose` |
| A slow (not runaway) script | VT: blocks a default-dispatcher thread for up to 2 s per evaluation. Calc / alarm: holds the driver-wide or engine-wide gate, so every sibling waits behind it (up to 2 s / 250 ms each). | `RoslynVirtualTagEvaluator.cs:174-175`; `CalculationDriver.cs:539-541`; `ScriptedAlarmEngine.cs:743-765` |
The sandbox (`ForbiddenTypeAnalyzer.cs:62-80`: `System.IO`, `System.Net`, `System.Diagnostics`,
`System.Reflection`, `System.Threading[.Tasks]`, `Environment`, `GC`, …) blocks *capabilities*, and it does
that well. It cannot block *consumption* (CPU, memory, stack). Every review that looked at this reached the
same conclusion:
- `docs/VirtualTags.md:40,54`;
- `docs/ScriptEditor.md` §"Runtime safety";
- `docs/superpowers/plans/2026-10-22-review-remediation.md` Lane E2;
- `docs/reviews/2026-10-21/05-core.md:103` and `03-host-controlplane-security-config.md:38`.
All of them say the fix is an **out-of-process runner**.
Script authoring is gated by the `ConfigEditor` policy, which admits Administrator **or Designer** (CLAUDE.md
§Scripting). The issue says "admin-gated", but the gate also admits Designers. That is the primary control,
and it explains why this work is p3.
---
## 2. Goals and non-goals
**Goals**
- G1. **Pre-emption.** A script that exceeds its budget is stopped within a bounded time. The thread, the
memory and the ALC are all reclaimed.
- G2. **Containment.** No script (OOM, stack overflow, `FailFast`-equivalent, native crash) can take down the
OtOpcUa host process, its OPC UA sessions, its drivers or its Akka membership.
- G3. **Attribution.** The script that caused a kill is identified and quarantined. Healthy scripts that
shared the worker are not.
- G4. **Parity.** Compile-time behaviour is unchanged: the same sandbox, analyzer, diagnostics and Monaco parity.
Runtime semantics are unchanged too: the same `ctx` API, the same input-quality gating, the same Bad / last-value
contract, and the same quarantine lift rules.
- G5. **Bounded overhead** (§8). It must fit the tag counts the repo already tests at.
- G6. **Flag-gated, reversible rollout.** In-process stays the default until the gates pass.
**Non-goals**
- N1. A new sandbox. `ForbiddenTypeAnalyzer` stays the capability control; process isolation is
defence-in-depth, not a replacement.
- N2. Moving the **authoring** surface. The AdminUI only compiles and analyzes (`ScriptAnalysisService`), and it
never executes a script, so central/admin nodes are out of scope.
- N3. Multi-tag cascades (`ctx.SetVirtualTag` taking effect). They are dropped today and stay dropped.
- N4. Cross-node or remote execution. The worker is always local to the driver node that owns the tags.
- N5. Per-run memory *pre-emption*. Even out of process, the limit is per *worker*. A per-run limit stays a
sampled observation (§5.4).
- N6. Running non-.NET or untrusted third-party code.
---
## 3. Process model
### 3.1 Options
| Option | Blast radius of a kill | Process count | Verdict |
|---|---|---|---|
| A. One worker per node | Every script on the node blips | 1 | Simple, but one bad script restarts every script on the node |
| B. One worker per script | One script | = script count (hundreds to thousands) | Infeasible: ~tens of MB per .NET process **[A-3]** |
| C. One worker per seam or driver instance | One seam / driver | 3 to ~10 | Arbitrary; a hot Calc driver still shares fate with its siblings |
| **D. Small sharded pool + an isolation worker** | One shard (1/K of the scripts), then one script | K + 0..1 (default 2 + 1) | **Recommended** |
### 3.2 Recommended: sharded pool with an isolation worker
- **K shard workers per driver node.** `Scripting:Worker:Shards` defaults to 2 and is clamped to 1–8. A
script is placed by `hash(scriptKey) mod K`. The script key is the guard key used today: the VT id, the calc
RawPath, or the alarm id. Placement is deterministic, so a script's compiled PE and warm state stay on one
worker across restarts.
- **One isolation worker** ("penalty box") is spawned on demand. Scripts under suspicion (§5.3) run there,
**one evaluation at a time**. A kill in the isolation worker is therefore unambiguous.
- **Inside a worker:** a bounded evaluation scheduler with `P` dedicated threads, where
`P = max(2, ProcessorCount / K)` **[A-4]**. Single flight per key is kept, as the guard does today. A queue
that is full answers `Busy`, which the host treats as a refusal, not a strike.
- **The passthrough fast path stays in the host.** It never runs user code, and it is probably the largest
share of evaluations **[A-5]**.
### 3.3 Lifecycle and supervision under Akka
The design splits a **control plane** (actors) from a **data plane** (a lock-free client). An Akka `Ask` per
evaluation would add mailbox hops to the hottest path for no benefit.
```
DriverHostActor (driver role)
└─ ScriptRunnerActor node-wide; owns the pool policy, attribution ledger, quarantine table
├─ ScriptWorkerActor #0 owns one OS process + pipe; state machine below
├─ ScriptWorkerActor #1
└─ ScriptWorkerActor iso spawned on first suspect, stopped after IsolationIdle (default 10 min)
IScriptRunner (DI singleton) ── data plane ──► ScriptWorkerConnection (per worker; pipe writer + correlation map)
used by RoslynVirtualTagEvaluator, CalculationEvaluator, ScriptedAlarmEngine
```
- **Ownership.** `ScriptRunnerActor` is created by `DriverHostActor`, next to `VirtualTagHostActor` and the
scripted-alarm host, so it exists only on driver-role nodes and dies with the driver tree. Its supervisor
strategy is `OneForOne`, and a restart of a `ScriptWorkerActor` kills and respawns its process.
- **Process exit is a message, not an exception.** A watcher on `Process.Exited` / `WaitForExitAsync` sends
`WorkerExited(exitCode, signal, inFlight[])` to its `ScriptWorkerActor`, so worker crashes never escalate
through Akka supervision.
- **Worker states:**
`Spawning → Handshaking → Loading → Ready ⇄ Draining → Dead → Backoff → Spawning`
`Ready` is the only state that accepts evaluations. In the other states the connection queues per key, with
latest-wins coalescing (§9).
- **The host process must take its workers down with it.**
- **Windows:** every worker is assigned to a Job Object with `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE`. If the host
dies, the kernel closes the job handle and kills the workers.
- **Linux** (the docker-dev rig): the worker treats **pipe EOF as fatal**, and also polls its parent PID. The
host is not trusted to clean up after itself.
- **CoordinatedShutdown.** In the `before-actor-system-terminate` phase, the host sends `Shutdown`, waits
1 s, then kills.
- **Orphan sweep at start.** The host kills only workers whose launch nonce file sits under *this* host's
instance directory, never "every process with this name". mxaccessgw's `OrphanWorkerTerminator` killed a
live gateway's workers when a second gateway started (scadaproj memory "wonder upgrade program"); this
design avoids that by construction.
- **Redeploy.** `ClearCompiledScripts` / `LoadAsync` become `IScriptRunner.ApplyGeneration(scriptSet)`. That
call sends `LoadScripts` / `Unload` deltas to each shard and clears the host-side quarantine rows for edited
or removed keys, which keeps today's lift rule. No worker restart is needed. Collectible ALCs in the
worker are unloaded as they are today, and a stuck one no longer matters because its worker gets killed.
- **Redundancy.** No change. Each node runs its own scripts today and will keep doing so; workers are not
shared across the pair.
---
## 4. IPC
### 4.1 Shape of the conversation
§1.1 showed that a run needs no host callbacks. Each evaluation is therefore **one request frame and one
reply frame**, with no re-entrant RPC. ScadaBridge's `SandboxWorkerProtocol` has to proxy `call`/`reply`
frames for external systems and databases; this design does not. Log lines from `ctx.Logger` are the only
other worker→host traffic, and they are fire-and-forget.
### 4.2 Transport: named pipe vs gRPC
| | Named pipe + length-prefixed frames | gRPC (h2c over UDS / named pipe) |
|---|---|---|
| Per-call overhead | Two syscalls plus serialize; no HTTP/2 framing **[A-6]** | HTTP/2 stream setup, HPACK and a Kestrel pipeline per call |
| Worker footprint | Plain console app | Needs Kestrel in the worker, or a client stack in it and a server in the host |
| Batching / priority | Free to design: control frames ahead of events, as in mxaccessgw `WorkerFrameProtocol` | Per-RPC; batching means streaming RPCs, which is a frame protocol again |
| Cross-platform | `NamedPipeServerStream` maps to a Unix domain socket on Linux; `PipeOptions.CurrentUserOnly` on both | Yes |
| Prior art in the family | mxaccessgw gateway↔worker (protobuf `WorkerEnvelope`, 4-byte LE length, priority-drained writer); ScadaBridge sandbox (JSON frames over stdio, 1 MiB cap) | OtOpcUa mesh transport (`Commons/Protos/mesh_transport.proto`) — but that is cross-node |
**Recommendation: a named pipe with 4-byte little-endian length-prefixed protobuf envelopes.** It copies
mxaccessgw's `WorkerFrameProtocol` discipline:
- a frame-size cap negotiated at the handshake (default 4 MiB);
- zero-length and oversize frames rejected;
- a monotonically increasing `seq`;
- control frames (heartbeat, shutdown, fault) drained ahead of evaluation traffic;
- coalesced flushes.
`Google.Protobuf` and `Grpc.Tools` are already central pins, and `Commons.csproj:28-30` already compiles
protos, so no new package is needed. `MessagePack` is pinned in `Directory.Packages.props` but referenced by
no project, so it would be a new dependency in practice. Stdio, as ScadaBridge uses, is rejected: a script
cannot write to the console (it is forbidden), but the runtime and the logger can, and one stray byte on
stdout corrupts the stream.
### 4.3 Message schema (sketch)
```proto
syntax = "proto3";
package otopcua.scriptworker.v1;
message Envelope {
uint32 protocol_version = 1;
uint64 seq = 2;
oneof body {
Hello hello = 10; // host→worker: nonce, max_frame_bytes, options
Ready ready = 11; // worker→host: pid, runtime version, sandbox fingerprint
LoadScripts load = 12; // host→worker: add/replace compiled-or-source scripts
LoadResult load_result = 13; // worker→host: per-script compile outcome + diagnostics
Unload unload = 14; // host→worker: drop keys (ALC unload)
EvalBatch eval = 15; // host→worker
EvalResults results = 16; // worker→host (one or more completions)
LogBatch logs = 17; // worker→host, fire-and-forget, bounded
Heartbeat heartbeat = 18; // both ways; carries worker gauges (inflight, heap, gen2 count)
Shutdown shutdown = 19;
Fault fault = 20;
}
}
enum ContextKind { CONTEXT_KIND_UNSPECIFIED = 0; VIRTUAL_TAG = 1; CALCULATION = 2; ALARM_PREDICATE = 3; }
message ScriptSource { string key = 1; bytes source_sha256 = 2; ContextKind kind = 3;
string source = 4; bytes pe_image = 5; repeated string input_paths = 6; }
message LoadScripts { uint64 generation = 1; repeated ScriptSource scripts = 2; }
message EvalBatch { repeated EvalRequest items = 1; }
message EvalRequest {
uint64 eval_id = 1; string key = 2; bytes source_sha256 = 3;
repeated Input inputs = 4; // positional against ScriptSource.input_paths; see 4.4
int64 now_utc_ticks = 5; // ctx.Now — host clock, so tests that pin the clock still work
uint32 soft_timeout_ms = 6;
}
message Input { uint32 slot = 1; Variant value = 2; uint32 status_code = 3;
int64 source_ts_ticks = 4; /* 0 = null */ int64 server_ts_ticks = 5; }
message EvalResult {
uint64 eval_id = 1;
oneof outcome { Variant value = 2; bool predicate = 3; ScriptError error = 4; }
uint64 allocated_bytes = 5; // same GC.GetAllocatedBytesForCurrentThread sample as today
uint32 run_micros = 6;
}
message EvalResults { repeated EvalResult items = 1; }
message ScriptError { Kind kind = 1; string message = 2;
enum Kind { THREW = 0; COMPILE = 1; SANDBOX = 2; SOFT_TIMEOUT = 3; UNSUPPORTED_TYPE = 4; BUSY = 5; } }
message Variant {
TypeTag type = 1; // exact CLR type — see 4.4
oneof v { bool b = 2; sint64 i = 3; uint64 u = 4; double d = 5; float f = 6;
string s = 7; int64 dt_ticks = 8; ArrayValue array = 9; }
enum TypeTag { NULL = 0; BOOLEAN = 1; SBYTE = 2; BYTE = 3; INT16 = 4; UINT16 = 5; INT32 = 6; UINT32 = 7;
INT64 = 8; UINT64 = 9; FLOAT = 10; DOUBLE = 11; STRING = 12; DATETIME_UTC = 13;
ARRAY = 14; }
}
message ArrayValue { Variant.TypeTag element_type = 1; repeated Variant items = 2; }
message LogBatch { repeated LogLine lines = 1; uint32 dropped = 2; }
message LogLine { string key = 1; uint32 level = 2; string message_template = 3;
string rendered = 4; string exception = 5; int64 ts_ticks = 6; }
```
### 4.4 How values and quality cross the boundary
- **Quality crosses verbatim.** `DataValueSnapshot` is `(object? Value, uint StatusCode, DateTime?
SourceTimestampUtc, DateTime ServerTimestampUtc)` (`Core.Abstractions/DataValueSnapshot.cs:13`), and it maps
1:1 onto `Input`. The worker rebuilds the same snapshots, so `ctx.GetTag(p).StatusCode` and
`.SourceTimestampUtc` behave as they do today. The input-quality gating does not move: VT holds evaluation
while an input is Bad (`VirtualTagActor.cs:264-269`), Calc holds on Bad, and alarms short-circuit on
`AreInputsReady`. A request is sent only when today's code would have run the script.
- **The exact boxed CLR type is load-bearing.** A script that writes `(short)ctx.GetTag(x).Value` throws
`InvalidCastException` if the value arrives boxed as an `int`. `Variant.type` therefore carries the CLR type,
not just the OPC UA type: `DriverDataType` has `Int16` / `UInt16`, which have to survive the round trip.
**[A-7]** I have not inventoried which CLR types each driver actually boxes, for example whether a Modbus
`Int16` arrives as `short` or as `int`. The shadow mode in §12 exists partly to find out.
- **Unsupported types.** An input whose CLR type has no tag is not silently converted. The request carries no
value for that slot, and the worker answers `UNSUPPORTED_TYPE`; the host publishes Bad with the type name. On
the result side, a script that returns an unsupported type (a `List<T>`, an anonymous object) also gets
`UNSUPPORTED_TYPE`. That is a **behaviour change**: today the raw `object` goes to
`OpcUaPublishActor.AttributeValueUpdate` (`VirtualTagHostActor.cs:233`), and what the sink does with it was
not traced **[A-8]**. Shadow mode reports every such case before the flag flips.
- **DateTime.** Values are sent as UTC ticks. A `Kind` other than `Utc` is converted at the host edge and
counted on a metric. **[A-9]** I assume drivers already produce UTC.
- **Positional inputs.** `LoadScripts` registers the script's ordered `input_paths`: the
`DependencyExtractor` reads, after the `{{equip}}` resolution done at deploy. Requests then send values by slot,
not by path string, which takes the dominant payload cost out of the hot path. A key with no registered
slots, a fallback that should never happen, sends the paths inline.
### 4.5 Batching
- **Opportunistic, not timed.** The connection's writer drains every request queued at that moment into one
`EvalBatch` and flushes once, as mxaccessgw's drain pass does. An idle system pays no added latency; a burst
is amortised automatically. There is no coalescing *timer*, because it would add latency to every
evaluation.
- **Results are streamed, not batch-atomic.** The worker sends `EvalResults` as items complete, coalescing
whatever finished in the same scheduler pass, so a slow script never holds a fast sibling's reply.
- **The seams must submit concurrently, or batching never happens.** Today both Calc and alarms await one
script at a time (§1.1), and with IPC that serial loop costs N × round-trip. Two seam changes are required:
- **Scripted alarms:** `ReevaluateAsync` submits the whole dirty set at once, then applies results **in the
original order** under `_evalGate`. Predicates are pure functions of a snapshot taken under the gate, so
the state machine sees the same sequence.
- **Calculation:** the pump dequeues everything pending, with each entry's dependency snapshot already
captured at enqueue, submits it as one batch and publishes in dequeue order.
- **Virtual tags become asynchronous.** `IVirtualTagEvaluator` gains `EvaluateAsync`, and `VirtualTagActor`
`PipeTo`s the result to itself. While a run is in flight, further dependency changes update the map and set a
"re-run" flag; when the result lands, the actor evaluates **once more** against the latest map (latest-wins).
This also ends today's blocking of a default-dispatcher thread for up to 2 s
(`RoslynVirtualTagEvaluator.cs:174-175`).
---
## 5. Pre-emption
### 5.1 Two deadlines
| | Soft deadline `T` | Hard deadline `T + G` |
|---|---|---|
| Value | Today's per-seam budget, unchanged: VT 2 s, Calc `runTimeoutMs` (2 s), alarm 250 ms | `G = Scripting:Worker:HardGrace`, default `max(1 s, T)`, clamped to 250 ms–30 s |
| Who acts | The host connection's timer | The host `ScriptWorkerActor` (authoritative) **and** a worker-side watchdog thread |
| Effect | The caller gets the timeout failure it gets today (Bad, same reason prefix `ScriptEvalFailureReasons.TimedOutPrefix`) and the run token is cancelled, so a polling script still ends itself as it does today | The worker process is **killed** (`Process.Kill(entireProcessTree: true)`). The worker watchdog calls `Environment.FailFast` first if it is still scheduling, so the dump and exit code name the key. |
| Strike | Yes (as today) | Yes, plus a kill strike (§5.3) |
The grace period `G` keeps a script that overran by a few milliseconds from costing a whole shard restart, and
the cooperative token from #734 still gets its chance. Making `G = 0` (kill at `T`) is an open decision (§14).
**Single flight between T and T+G.** The key stays in flight, and a trigger is refused as today, which is not
a strike. After the kill there is **no orphan**, so the stuck-orphan rule (`StuckOrphanQuarantineFactor`)
becomes moot out of process; it stays for the in-process mode.
### 5.2 Resource limits on the worker
The limits combine prior art from ScadaBridge (heap limit) and mxaccessgw (process-tree kill):
- **Heap:** `DOTNET_GCHeapHardLimit = Scripting:Worker:HeapLimitBytes`, default 512 MiB **[A-10]**. Breaching
it makes the worker throw `OutOfMemoryException` or fail fast. Either way the worker dies or reports, and the
host survives.
- **Windows Job Object** (per worker):
- `JOB_OBJECT_LIMIT_PROCESS_MEMORY`, set slightly above the heap limit as a backstop for native and JIT
memory;
- `JOB_OBJECT_LIMIT_ACTIVE_PROCESS = 1`, so no child processes are possible;
- `KILL_ON_JOB_CLOSE`;
- UI restrictions.
- **Linux / containers:** the heap limit only. A container's cgroup is the rig's concern. **[A-11]** I assume
the docker-dev containers run without CAP_SYS_ADMIN, so a nested cgroup is not available.
- **Liveness:** a heartbeat every 1 s. Three missed heartbeats while not in a GC pause, which the heartbeat
reports, trigger a kill with reason `heartbeat`. This catches a wedged worker that neither replies nor exits.
### 5.3 Attribution, restart, backoff and quarantine
A kill or crash takes down every in-flight evaluation on that shard. Attribution decides which **one** to
blame.
1. **Hard-deadline kill:** the key that crossed `T+G` is known exactly. It takes a **kill strike**. The other
in-flight keys on that shard are victims: their runs fail with "script worker restarted", which is not a
strike.
2. **Crash with no deadline crossed** (OOM, stack overflow, native fault): attribution is ambiguous. Every
key in flight at exit becomes a **suspect**, and suspects move to the **isolation worker**.
3. **Isolation worker:** evaluations run one at a time, so a crash there attributes to exactly one key, which
takes a kill strike. A suspect that completes `IsolationProbation` (default 20 successful runs or 10 min)
without incident returns to its shard.
4. **Quarantine:** a key is quarantined after `Scripting:Worker:QuarantineAfterKills` kill strikes (default 2)
within 1 h, or by today's rules (N consecutive soft timeouts, allocation breach). The quarantine table
lives in the **host** (`ScriptRunnerActor`), so it survives worker restarts. It is lifted by a source
change or a redeploy, exactly as today, and quarantine reasons keep the `"script quarantined: "` prefix so
every existing consumer and test classifier still matches.
5. **Restart backoff per worker:** exponential, 250 ms → 30 s cap, ×2, with ±20% jitter. The backoff resets
after 60 s of `Ready`. A worker that dies **during** `Loading` (a poison PE or a runtime fault) does not
loop: after 3 consecutive load-phase deaths, the scripts in its generation are loaded one by one in the
isolation worker to find the culprit.
6. **Crash-loop circuit:** if all shards are in backoff for longer than `Scripting:Worker:CircuitOpenAfter`
(default 2 min), the runner reports `Unavailable` (§10) and logs one Error. It keeps retrying at the cap.
---
## 6. Security
Process isolation contains **failures**. Most of the **capability** control stays with the compile-time
sandbox, which is retained. Defence-in-depth adds the following.
- **The compile-time sandbox is unchanged.** It still includes `ScriptSandbox` (with the TPA reference set
filtered to `System.*`, `ScriptSandbox.cs:118-122`), `ForbiddenTypeAnalyzer`, and the injection guard. The
worker runs **exactly** the same `ScriptEvaluator.Compile`, so editor/publish/runtime parity holds.
- **Assembly allow-list by construction.** The worker is its own app (`ZB.MOM.WW.OtOpcUa.ScriptWorker`), with
its own `deps.json` and only these references:
- `Core.Scripting`;
- `Core.Scripting.Abstractions`;
- `Core.Abstractions`;
- Serilog (core only, for `ILogger` on `ctx`);
- Google.Protobuf.
Its trusted-platform list therefore contains the shared framework plus those assemblies. Akka, SqlClient, the
OPC UA stack, driver assemblies and the Secrets library are **not loadable** in the worker, even through
reflection. At startup the worker asserts its loaded-assembly set against a pinned list and exits if the list
differs.
- **No secrets in the environment.** `ProcessStartInfo.Environment` is **cleared** and then repopulated from
an explicit allow-list: `DOTNET_*` runtime knobs, `TEMP`, `PATH` minimal, and the launch nonce. Host env vars
like `ServerHistorian__ApiKey` (CLAUDE.md "Never commit secrets") and the ConfigDb connection string must
never reach the worker. ScadaBridge's sandbox **inherits** the parent environment; do not copy that part.
- **No configuration access.** The worker gets no appsettings path. Every option it needs (budgets, heap,
frame cap) arrives in `Hello`.
- **Pipe hardening (mxaccessgw pattern):**
- a random pipe name;
- `PipeOptions.CurrentUserOnly`, or an explicit ACL when the worker runs as a different account;
- a one-time nonce passed in an env var (never on the command line; mxaccessgw `WorkerBootstrap.md` explains
why), which the worker must echo in `Ready`;
- the host accepts exactly one connection per launch.
- **Filesystem:**
- the working directory is an empty per-worker temp directory;
- the PE cache (§7) is opened read-only by the worker, if the host compiles, or written only under its own
subdirectory;
- on Windows, a dedicated low-privilege account (see §14) gets read access only to the install folder and
its own temp directory.
- **Network:** the analyzer already forbids `System.Net`. Process-level options are a Windows Firewall
outbound-block rule scoped to the worker executable, created by the installer, and on Linux nothing beyond
the container. This is an open decision (§14). The recommendation is the installer firewall rule on Windows,
and to accept the analyzer as the control on Linux.
- **What the host trusts from the worker:** nothing it cannot validate. Frames are size-capped and
schema-checked, an `eval_id` must match an outstanding request, and log lines are length-capped and
rate-limited (§11). A compromised worker can at worst return wrong values for scripts on its own shard,
which the same script could have done anyway.
---
## 7. Compile caching across restarts
The problem: today every process start recompiles every script lazily (`CompiledScriptCache` is memory-only).
Out of process, a **shard restart** has the same cost, and it can happen in the middle of production running.
mem-probe measured the old `CSharpScript` path at roughly 2.4 MiB of working set per retained script
(`tools/mem-probe/MemProbe/Program.cs:1-4`). Compile *time* is not measured anywhere in the repo **[A-12]**;
I assume tens of ms per script warm and seconds for the first compile while Roslyn JITs. For 1,000 scripts,
that is far outside a 2 s restart budget.
**Recommendation: a content-addressed PE cache on node-local disk.**
- **Key:** SHA-256 over these fields:
- the source;
- the `ContextKind`, i.e. `TContext` and `TResult`;
- the Roslyn compiler version;
- a **sandbox fingerprint**: a hash of `ForbiddenTypeAnalyzer`'s lists, the `ScriptSandbox` reference set
and the wrapper template.
An analyzer change therefore invalidates every entry, and a cached PE can never skip a newer sandbox rule.
- **Value:** the emitted PE bytes plus an HMAC-SHA256 tag. The HMAC key is a per-node random key created at
first start and stored next to the cache, readable only by the host account. The worker loads a PE only if
the HMAC verifies, because loading attacker-supplied PE bytes would be code execution outside the analyzer.
- **Location:** `Scripting:Worker:PeCacheDir`. It must be absolute and outside the content root, the same
fail-closed rule the family applies to `Secrets:SqlitePath` (scadaproj CLAUDE.md, Secrets row). Entries
whose key is not in the current generation are pruned at `ApplyGeneration`.
- **Where compilation happens:** **in the worker** (recommended). Roslyn's memory and CPU leave the host. A
pathological source that makes Roslyn itself blow up kills a worker, not the server, and the worker writes
the PE plus HMAC (it holds the key through `Hello`). The alternative is to compile in the host and ship
`pe_image` in `LoadScripts`. That keeps the worker free of Roslyn (smaller, faster start), but compile-time
blowups stay in the host. This is an open decision (§14).
- **Restart path:** `Spawning → Ready` takes about 150 ms of runtime start **[A-13]**. Then `LoadScripts`
sends hash + key + slots only. The worker loads PEs from the cache lazily, on each key's first evaluation,
and a cache miss compiles. Loading a PE into a collectible ALC is about 1 ms **[A-13]**.
- **Negative cache:** Calculation already remembers compile failures by source hash (R07:P4,
`CalculationEvaluator.cs:63-68`). The worker reports compile failures in `LoadResult`, and the host keeps
that negative cache for **all** seams, so a broken script never re-enters Roslyn.
---
## 8. Performance budgets
### 8.1 Reference loads in the repo
| Source | Load |
|---|---|
| `docs/v2/Galaxy.Performance.md` (EventPump sizing, 24 h soak) | 50,000 tags at 1 Hz on one node; compressed CI soak at 1,000 tags |
| `tests/Server/.../NodeManagerLockTimeBenchmarkTests.cs:41-105` | 10,000-tag folder; 200 equipment × 250 tags (50,000 variables) |
| `ScriptRunGuardTests.cs:170-205` | a 1 Hz runaway and a healthy sibling over 30 triggers |
No repo perf test exercises scripts at scale. **[A-14]** I assume a planning envelope of **5,000 compiled
script evaluations per second per node** (for example 5,000 VT + Calc at 1 Hz, or 500 scripts at 10 Hz) and
size for 2× headroom (10k/s). The passthrough fast path is excluded because it stays in the host.
### 8.2 Budgets (acceptance gates for the implementation)
| Metric | Budget | Notes |
|---|---|---|
| Added latency per evaluation, p50 | ≤ 150 µs | Host enqueue → reply decoded, trivial script, idle system |
| Added latency per evaluation, p99 | ≤ 2 ms | At 10k evals/s. Compare the 250 ms alarm budget: under 1% of it |
| Throughput | ≥ 10,000 trivial evals/s/node with K=2 | With opportunistic batching |
| Host CPU for IPC | ≤ 25% of one core at 10k evals/s | Serialisation + pipe I/O, measured by an opt-in perf test |
| Bytes per eval on the wire | ≤ 64 B + 24 B per input | Positional slots, §4.4 |
| Shard restart → first result, 1,000 cached scripts | ≤ 2 s | §7 lazy PE load |
| Worker idle RSS | ≤ 80 MB + scripts **[A-3]** | Per shard |
| Kill latency after `T+G` | ≤ 100 ms | Host timer resolution plus `Process.Kill` |
**Comparison with today:** the in-process cost per compiled run is one `Task.Run` hop plus `WaitAsync`, which
is a few µs to tens of µs and was never measured **[A-15]**. The out-of-process path is therefore one to two
orders of magnitude slower **per call**. That is acceptable only because the budgets above sit two to three
orders of magnitude below the script budgets (250 ms / 2 s), and because batching amortises the syscalls. The
risk case is a large scripted-alarm engine, with its serial gate and its 250 ms budget. That is why §4.5
makes the batch-submit change a **prerequisite**, not an optimisation.
---
## 9. Failure semantics: what quality a tag gets
The principle: **a runner failure must never look like a script result.** In particular, an alarm must never
clear because the worker died.
| Situation | Virtual tag / Calculation tag | Scripted alarm |
|---|---|---|
| Soft timeout (`T`) | `BadInternalError` + last value, once per transition, as today | Condition frozen + `BadInternalError` quality annotation, as today |
| This key killed at `T+G`, or crashed in isolation | Same as soft timeout; reason `"script killed: exceeded hard deadline …"` | Same as soft timeout |
| Victim: in flight on a shard that died for another key | `BadInternalError` + last value, reason `"script worker restarted"`; re-evaluated on the shard's return with the latest inputs (latest-wins) | Frozen + Bad annotation; re-evaluated on return |
| Not in flight while its shard restarts | **Unchanged.** Keeps its last published value and quality. A trigger during the outage is coalesced (latest-wins, one per key) and evaluated when the shard is `Ready` | Same; the condition holds its state |
| Shard not back within `Scripting:Worker:UnavailableGrace` (default 5 s) | Coalesced keys publish Bad + last value, reason `"script runner unavailable"` | Frozen + Bad annotation |
| Circuit open (§5.3.6) | Every compiled-script key on the node is Bad as above; passthrough keys are unaffected | Every compiled predicate is frozen + Bad |
| Quarantined | `"script quarantined: …"` Bad, as today | As today |
| Host process dies | Workers die with it (Job Object / pipe EOF); no change from today's host-death behaviour | — |
**Status code:** `BadInternalError` stays for script-attributable faults (the existing contract). For "the
runner is unavailable, not this script's fault", this design proposes **`BadResourceUnavailable`
(0x80040000)**, so operators and the ScadaBridge consumer can tell a broken script from a sick node. The
alternative is to keep `BadInternalError` everywhere for zero consumer impact. This is an open decision (§14);
if it is taken, the ScadaBridge consumer-impact record must note it (scadaproj `sister-project-history`
skill, `otopcua.md`).
**No silent fallback to in-process.** If the worker cannot start, the runner does **not** run scripts in the
host. That would reintroduce exactly the risk the flag was set to remove. Operators flip the flag back (§12).
This is an open decision (§14).
---
## 10. Observability
All metrics go on `OtOpcUaTelemetry.Meter`, so the existing `ZbTelemetryOptions` allow-list applies.
scadaproj memory: "`ZbTelemetryOptions.Meters` is a silent allowlist"; check that the new names flow end to
end.
| Metric | Type | Tags |
|---|---|---|
| `otopcua.script.worker.up` | gauge | `shard` |
| `otopcua.script.worker.starts` / `.exits` | counter | `shard`, `reason` = `clean` \| `killed_deadline` \| `killed_heartbeat` \| `oom` \| `crash` \| `load_failure` |
| `otopcua.script.worker.inflight` / `.queue_depth` | gauge | `shard` |
| `otopcua.script.worker.heap_bytes` | gauge (from heartbeat) | `shard` |
| `otopcua.script.ipc.roundtrip` | histogram (µs) | `kind` |
| `otopcua.script.eval.duration` | histogram (µs, worker-measured) | `kind` |
| `otopcua.script.compile` | counter | `cache` = `hit` \| `miss` \| `negative` |
| `otopcua.script.quarantined` (existing) | counter | adds `reason` = `killed` \| `isolated_crash` |
| `otopcua.script.run_refused` (existing) | counter | unchanged |
| `otopcua.script.isolation.suspects` | gauge | — |
| `otopcua.script.runner.state` | gauge (0 ready / 1 degraded / 2 unavailable) | — |
**Per-key series:** the existing counters tag the series with `script = key`. Do not add `key` tags to the new
per-eval histograms (cardinality).
**Logs:**
- the worker's own stderr is captured, prefixed with `ScriptWorker[shard]`, into the host log;
- every kill logs one Warning naming the key and the reason, and every quarantine logs one Error, as today;
- `ctx.Logger` lines come back in `LogBatch`. The host re-emits them through `ScriptRootLogger` with the same
`ScriptId` / `VirtualTagId` properties, so `scripts-*.log`, the error mirror, the DPS script-log topic and its
20/min-per-source rate limit all keep working unchanged;
- the worker caps its outbound log buffer (default 1,000 lines / 1 MiB) and reports drops as
`LogBatch.dropped`, so a script logging in a tight loop cannot flood the pipe.
**Health:** expose the runner state as a non-fatal component of the driver-node status (`/health/ready`
stays green while degraded, because OPC UA still serves driver data). Whether `Unavailable` should fail
readiness is an open decision (§14). The AdminUI status dashboard gets a "Script runner" tile with the
per-shard state, restarts in the last hour and the quarantine list. Quarantines already surface through the
`QuarantinedScripts` properties.
**Traces:** one `Activity` per `EvalBatch` (not per evaluation), with `batch.size` and `shard`.
---
## 11. Seam changes in the host (summary)
| Component | Change |
|---|---|
| `Core.Scripting` | New `IScriptRunner` (`EvaluateAsync(ScriptInvocation[])`, `ApplyGeneration`, `QuarantinedKeys`, `ResetAll`). `InProcessScriptRunner` wraps today's `ScriptRunGuard` + `CompiledScriptCache` exactly. |
| New `Core.Scripting.Protocol` | The proto file, the frame codec and the `Variant` ↔ CLR mapping. Shared by host and worker. |
| New `ZB.MOM.WW.OtOpcUa.ScriptWorker` | Console app: handshake, scheduler, watchdog, PE cache, log forwarding. Shipped in the Host's publish output. |
| New `Runtime/Scripting/` | `ScriptRunnerActor`, `ScriptWorkerActor`, `OutOfProcessScriptRunner` (data plane), Job Object interop (Windows-only, behind `OperatingSystem.IsWindows()`). |
| `IVirtualTagEvaluator` / `VirtualTagActor` | Add `EvaluateAsync`; the actor `PipeTo`s with latest-wins re-run (§4.5). |
| `RoslynVirtualTagEvaluator`, `CalculationEvaluator`, `ScriptedAlarmEngine` | Replace direct `ScriptRunGuard` use with `IScriptRunner`. Calc pump and alarm `ReevaluateAsync` submit batches (§4.5). |
| `ScriptingOptions` / `ScriptingOptionsBinding` | New keys (§12), clamped with startup Warnings like the existing ones. |
---
## 12. Migration and rollout
**Configuration** (node-local appsettings, `Scripting` section, consistent with the existing knobs):
| Key | Default | Meaning |
|---|---|---|
| `Scripting:Runner` | `InProcess` | `InProcess` \| `OutOfProcess` \| `Shadow` |
| `Scripting:Runner:Seams` | all three | Optional per-seam subset for `OutOfProcess` / `Shadow` (e.g. `["Calculation"]`) |
| `Scripting:Worker:Shards` | `2` | 1–8 |
| `Scripting:Worker:HardGrace` | `max(1s, T)` | §5.1 |
| `Scripting:Worker:HeapLimitBytes` | `536870912` | §5.2 |
| `Scripting:Worker:QuarantineAfterKills` | `2` | §5.3 |
| `Scripting:Worker:UnavailableGrace` | `5s` | §9 |
| `Scripting:Worker:PeCacheDir` | none (required when not `InProcess`) | §7; absolute, outside the content root, fail-closed |
**Phases** (each one is its own issue and PR):
1. **P0, in-process refactor with no behaviour change.**
- Introduce `IScriptRunner` + `InProcessScriptRunner`.
- Make VT evaluation async (`PipeTo`), and make the alarm engine and the Calc pump batch-submit.
- Gate: every existing Scripting / VirtualTags / Calculation / ScriptedAlarms suite stays green unchanged,
plus new ordering tests for batch-then-apply.
- This phase has value on its own: it ends the dispatcher-thread blocking.
2. **P1, worker + protocol behind the flag.**
- Ship `ScriptWorker`, `OutOfProcessScriptRunner` and the PE cache.
- Default stays `InProcess`. Enable on the docker-dev rig only.
- Gate: §13 runaway / containment tests green on Linux (docker-dev) **and** Windows (windev box, which has
the .NET 10 toolchain).
3. **P2, shadow mode on docker-dev and QA.**
- `Shadow` runs the **in-process** evaluation as the published result, and *also* submits to the worker.
- It compares value, status class and failure kind per evaluation, and counts
`otopcua.script.shadow.mismatch{kind, reason}`. Mismatching keys are logged once each.
- Shadow doubles script CPU and still carries the in-process risk, so it is a time-boxed diagnostic (for
example 1 week on QA), not a mode to run in production.
- Exit criterion: zero unexplained mismatches. This is where [A-7] / [A-8] get answered.
4. **P3, flip on QA, then the other tiers.**
- Set `Scripting:Runner=OutOfProcess` per node, seam subset first (Calculation, then VT, then alarms; alarms
last because of the 250 ms budget).
- Note: remote command execution on the wonder boxes is currently scoped to user-requested deploys (CLAUDE.md
hold), so tier rollout rides normal deploys.
5. **P4, decide the long-term status of `InProcess`.** The recommendation is to keep it for dev and tests and
as the documented rollback, and to retire `StuckOrphanQuarantineFactor` from the docs as "in-process only".
**Rollback:** set the flag back to `InProcess` and restart the node. No data or config migration is involved.
The PE cache directory can be deleted at any time.
**Docs to update when implementing:**
- `docs/VirtualTags.md` §"Known resource limits" and §"Per-evaluation timeout";
- `docs/ScriptEditor.md` §"Runtime safety";
- `docs/ScriptedAlarms.md`;
- `docs/ServiceHosting.md`: the new process, Job Object and firewall rule;
- `docs/security.md`;
- `docs/Telemetry.md`;
- the scadaproj index row, if the status-code decision changes the ScadaBridge-visible contract.
---
## 13. Test plan
**Unit (`Core.Scripting.Tests`, new `ScriptWorker.Tests`):**
- Frame codec:
- round-trip every `Variant.TypeTag` (arrays included);
- reject zero-length, oversize and truncated frames;
- reject an `eval_id` with no outstanding request;
- fuzz the reader with random bytes and assert there is no exception other than a protocol fault.
- **Unboxing parity:** for each tag, a script doing `(T)ctx.GetTag("x").Value` behaves identically in and out
of process. This pins [A-7].
- Attribution ledger:
- a deadline kill blames exactly one key;
- a crash with 3 in flight → 3 suspects → isolation;
- an isolation crash → one strike;
- probation returns a suspect to its shard;
- quarantine after 2 kills;
- a source change lifts the quarantine;
- a redeploy lifts every quarantine.
- Backoff schedule, jitter bounds, the reset-after-healthy rule and the load-phase crash bisection.
- Options binding: clamps plus a Warning per clamped key; `PeCacheDir` relative or inside the content root is a
startup failure.
**Containment (worker-backed, `Category=ScriptWorker`; runs in CI on Linux, gated on windev for Windows).** Each
case has a positive control: a healthy sibling script on the **same shard** that keeps producing correct values
before and after the event. Following scadaproj memory "assert the artifact before the outcome", each assertion
is checked to be capable of failing.
| Case | Assert |
|---|---|
| `while(true){}` ignoring the token | Killed within `T + G + 100 ms`; host `ThreadPool.ThreadCount` and `Process.Threads` unchanged; no orphan (`InFlightCount == 0` after the kill). Mutation check: disable the kill and see the test fail |
| `while(!ctx.CancellationToken.IsCancellationRequested){}` | Ends at `T` by itself; **no** kill; no strike beyond the timeout |
| Allocation bomb | Worker dies of OOM or the heap limit; **host RSS stays below a bound** and the host stays up; suspect → isolation → quarantine |
| Unbounded recursion | Worker dies; host up; quarantined after the isolation reproduction. Also the first in-repo reproduction of [A-1] for the in-process mode, marked `Skip` there because it would kill the test host |
| Worker killed externally (`kill -9`) | Victims Bad with "script worker restarted", then recover with the latest inputs; the alarm condition is **not** cleared |
| Host killed | Worker exits within 2 s (pipe EOF on Linux, Job Object on Windows) |
| Environment | The worker's environment contains no `ServerHistorian__ApiKey` / ConfigDb variables (set them in the test host first: positive control) |
| Pipe | A second client connecting to the pipe is refused; a wrong nonce is refused |
| Loaded assemblies | The worker's loaded set equals the pinned list; Akka / SqlClient / Opc.Ua are absent |
| Crash loop | A poison generation trips the circuit; state `Unavailable`; keys Bad with the runner-unavailable reason; recovery after a fixed redeploy |
**Parity:** run the existing `RoslynVirtualTagEvaluatorTests`, `CalculationDriver*Tests` and
`ScriptedAlarmEngine*Tests` against **both** runners. Use xunit theory or class fixtures over
`IScriptRunner`, with the worker built as a test dependency.
**Performance (`Category=Perf`, opt-in, not in CI):** a Stopwatch-based harness (no new packages) that
measures the §8 table: p50/p99 added latency, throughput at K=1/2/4, host CPU, restart-to-first-result with
1,000 cached PEs, and RSS per shard.
**Live gates:**
- docker-dev rig: a deploy with VTs, Calc tags and scripted alarms; drill a runaway script through the
AdminUI and watch `/uns` go Bad, the quarantine appear, and siblings stay Good;
- QA shadow week (P2);
- a QA flip with the same drill.
---
## 14. Open decisions for the owner
Each has a recommendation, marked ★.
1. **Pool shape.**
- ★ K=2 shards plus an on-demand isolation worker;
- one worker per node, simplest, but every kill blips every script;
- per-seam workers.
2. **Hard grace `G`.**
- ★ `max(1 s, T)`, which keeps cooperative cancellation useful;
- `G = 0` (kill at `T`): strictest, but every soft overrun restarts a shard.
3. **Status code for "runner unavailable / worker restarted".**
- ★ `BadResourceUnavailable`, which separates a sick node from a broken script but is a ScadaBridge-visible
contract change;
- keep `BadInternalError` everywhere, for zero consumer impact.
4. **Fallback when the worker cannot start.**
- ★ fail closed: Bad, and no in-process execution;
- fail open to in-process with an Error, which keeps values flowing but silently reopens the risk.
5. **Worker identity on Windows.**
- ★ a dedicated low-privilege virtual account created by the installer, which needs installer work, an
explicit pipe ACL, and a deploy that touches the box;
- the same account as the host: zero install change, and the Job Object plus the analyzer remain the
controls.
6. **Network isolation.**
- ★ a Windows Firewall outbound-block rule on the worker executable (installer), with the analyzer as the
only control on Linux;
- analyzer only, everywhere.
7. **Compile location.**
- ★ in the worker: Roslyn blowups are contained and the host gets lighter;
- in the host, shipping PE bytes: a lighter worker and faster cold start.
8. **PE cache store.**
- ★ a node-local directory with an HMAC;
- LocalDb (already a dependency; it would also replicate to the pair, which is unnecessary);
- no persistence, recompiling on restart (violates the 2 s budget at scale).
9. **IPC encoding.**
- ★ protobuf (already compiled in `Commons`, and mxaccessgw precedent);
- MessagePack (pinned but unused, so in practice a new dependency).
10. **Scripted alarms in the first flip.**
- ★ last, after Calc and VT, because of the 250 ms budget and the engine refactor;
- all seams at once.
11. **Unsupported input/result CLR types.**
- ★ an explicit Bad `UNSUPPORTED_TYPE`, surfaced in shadow mode first;
- best-effort conversion (to string, `double`), which hides type bugs.
12. **Readiness when the circuit is open.**
- ★ degraded but ready (OPC UA still serves driver data);
- not ready, which on a redundant pair would shift ServiceLevel to the peer.
13. **Long-term status of `InProcess`.**
- ★ keep it as the dev/test default and the rollback;
- delete it after a soak period.
14. **Does this need a real-deployment scale target?** [A-14]'s 5k evals/s/node is a guess. The owner should
supply the expected VT / Calc / alarm counts per node for the target sites, or accept the guess as the
gate.
---
## 15. Assumptions (not verified from code)
| Id | Assumption | How to verify |
|---|---|---|
| A-1 | Unbounded recursion in a script throws `StackOverflowException` and kills the host today | Reproduce in a throwaway process (not the test host) |
| A-2 | A collectible ALC whose code is still executing on an orphan thread cannot be unloaded | .NET docs / a `WeakReference` test with a spinning orphan |
| A-3 | A minimal .NET 10 worker idles at tens of MB RSS | Measure the P1 worker |
| A-4 | `P = max(2, cores/K)` evaluation threads per worker is enough | P1 perf harness |
| A-5 | Passthrough scripts are a large share of evaluations on real deployments | Count `PassthroughScript.TryMatch` hits against compiled runs on QA (add a counter in P0) |
| A-6 | A local pipe round trip costs tens of µs | P1 perf harness |
| A-7 | Drivers box tag values with stable, enumerable CLR primitive types | Shadow-mode type census |
| A-8 | What `OpcUaPublishActor` / the sink does with an unsupported result object today | Trace `AttributeValueUpdate` → `SdkAddressSpaceSink` |
| A-9 | Drivers produce UTC `DateTime` values | Shadow-mode census on `DateTime.Kind` |
| A-10 | 512 MiB is a sane per-worker heap limit (ScadaBridge uses the same) | Size against measured RSS × script count |
| A-11 | docker-dev containers cannot create nested cgroups | Check the compose capabilities |
| A-12 | Warm Roslyn compile is tens of ms per script; cold is seconds | P1 perf harness |
| A-13 | Worker start is ~150 ms; a PE load into a collectible ALC is ~1 ms | P1 perf harness |
| A-14 | 5k compiled evals/s/node is a fair planning envelope | Owner input (decision 14) |
| A-15 | Today's in-process overhead per compiled run is µs to tens of µs | Add the same harness to `InProcessScriptRunner` in P0 for a baseline |
---
## 16. Prior art consulted
- **ScadaBridge `CentralUI.SandboxWorker`** (`~/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.CentralUI/ScriptAnalysis/SandboxWorkerProtocol.cs`).
It runs one disposable process per *test run* over stdio, using 4-byte LE length-prefixed JSON frames with a
1 MiB cap, admission 4, `DOTNET_GCHeapHardLimit` 512 MiB, and `Kill(entireProcessTree)` on the deadline.
Taken from it: the heap limit, the tree kill and the frame cap. Not taken:
- stdio (fragile for a long-lived worker);
- per-run process spawn (too slow for runtime evaluation);
- inheriting the parent environment;
- proxying host calls.
ScadaBridge's *runtime* scripts are still in-process (dedicated scheduler plus watchdog,
`docs/plans/2026-08-15-script-pool-split-design.md`), so this design would be the family's first
out-of-process **runtime** evaluator.
- **mxaccessgw gateway↔worker** (`~/Desktop/MxAccessGateway/docs/WorkerFrameProtocol.md`,
`WorkerProcessLauncher.md`, `WorkerBootstrap.md`). It runs a long-lived supervised worker over a named pipe:
protobuf envelopes, a negotiated frame cap, a priority-drained writer with coalesced flushes, sequence
numbers, a nonce in an env var (never on the command line), startup probe plus retry, and a tree kill on a
failed start. Taken from it: the frame protocol shape, the nonce handshake and the launcher discipline.
Avoided: an orphan terminator that is not scoped to its own instance.