From 9e4a26f88891d382e12deac93993cc749f1da596 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 9 Nov 2026 17:04:12 -0500 Subject: [PATCH] docs(scripting): design an out-of-process, pre-emptible script runner (lmxopcua#750) Design only. Documents why today's in-process ScriptRunGuard can abandon but not stop a runaway script (file:line evidence), and proposes a sharded worker pool with an isolation worker under a ScriptRunnerActor, a named-pipe + protobuf frame protocol carrying full input snapshots, soft/hard deadlines with process kill, kill attribution and quarantine, worker sandboxing (scrubbed env, pinned assemblies, Job Object, heap limit), an HMAC'd content-addressed PE cache, performance budgets, failure-quality semantics, a flag-gated InProcess/Shadow/OutOfProcess rollout, a test plan, open decisions and an assumptions register. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV --- ...-09-out-of-process-script-runner-design.md | 846 ++++++++++++++++++ 1 file changed, 846 insertions(+) create mode 100644 docs/plans/2026-11-09-out-of-process-script-runner-design.md diff --git a/docs/plans/2026-11-09-out-of-process-script-runner-design.md b/docs/plans/2026-11-09-out-of-process-script-runner-design.md new file mode 100644 index 000000000..b89653985 --- /dev/null +++ b/docs/plans/2026-11-09-out-of-process-script-runner-design.md @@ -0,0 +1,846 @@ +# Out-of-process, pre-emptible script runner — design + +**Date:** 2026-11-09 · **Status:** proposed (design only — no code in this change) · **Issue:** #750 · **Related:** #734 (Lane E remediation) + +This document designs a script runner that can actually **stop** a user script, instead of abandoning it. +It covers the three script seams that run operator-authored C# on a driver node: virtual tags, Calculation +tags and scripted-alarm predicates. + +Claims about today's code carry `file:line` evidence against `origin/master` @ `94926eb0`. Anything not +read from the code is marked **[A-n]** and listed in §15. + +--- + +## 1. Today: in-process execution + +### 1.1 One compile pipeline, three seams + +All three seams share `Core.Scripting`: + +- **Compile.** `ScriptEvaluator.Compile` wraps the user source in a synthesized + `CompiledScript.Run(ScriptGlobals)`, compiles it with Roslyn against the `ScriptSandbox` reference + set, runs `ForbiddenTypeAnalyzer`, emits PE bytes to a `MemoryStream` and loads them into a collectible ALC + (`src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ScriptEvaluator.cs:125` emit, `:139` `LoadFromStream`). +- **Cache.** `CompiledScriptCache` is keyed by SHA-256 of the source (`CompiledScriptCache.cs:157`). It is + **in memory only**, so every process start recompiles every script on first use. +- **Execution.** It is synchronous: `ScriptEvaluator.RunAsync` invokes the bound delegate inline and returns + `Task.FromResult` (`ScriptEvaluator.cs:178-195`). The script-visible `ctx.CancellationToken` is an + `AsyncLocal` set for the duration of the call (`ScriptEvaluator.cs:191`, + `Core.Scripting.Abstractions/ScriptContext.cs:96-98`). +- **The context is a pure read cache.** `VirtualTagContext.GetTag` and `AlarmPredicateContext.GetTag` serve + only from a dictionary handed in by the caller (`VirtualTagContext.cs:50`, `AlarmPredicateContext.cs:40`). + `SetVirtualTag` is dropped in both production single-tag evaluators + (`Host/Engines/RoslynVirtualTagEvaluator.cs:129-131`, `Driver.Calculation/CalculationEvaluator.cs:185-188`) + and throws in alarm predicates (`AlarmPredicateContext.cs:50`). Inputs are known statically, because + `DependencyExtractor` rejects non-literal `GetTag` paths at publish. + + This matters most for the design (§4.1). **A script needs nothing from the host while it runs**: the + inputs go in, one value comes out, and log lines are a side channel. + +| Seam | Where it runs | Budget | Serialisation | Evidence | +|---|---|---|---|---| +| Virtual tag | `VirtualTagActor` message handler calls `IVirtualTagEvaluator.Evaluate` **synchronously** on the actor's dispatcher thread; `RoslynVirtualTagEvaluator` blocks with `.GetAwaiter().GetResult()` | 2 s | per actor (one actor per VT) | `Runtime/VirtualTags/VirtualTagActor.cs:276`; `RoslynVirtualTagEvaluator.cs:70`, `:174-175`; singleton `Host/Program.cs:393-398` | +| Calculation tag | async pump in `CalculationDriver`, one evaluation at a time per driver instance | 2 s (`runTimeoutMs`) | `_evalGate` = `SemaphoreSlim(1,1)` per driver | `CalculationDriver.cs:99`, `:523-556`, `:628`; `CalculationDriverOptions.cs:13`; `CalculationEvaluator.cs:232-233` | +| Scripted alarm | `ScriptedAlarmEngine.ReevaluateAsync` walks the dirty set **serially** under one gate | **250 ms** (the engine is built without `scriptTimeout`, so it takes `TimedScriptEvaluator.DefaultTimeout`) | `_evalGate` per engine | `ScriptedAlarmEngine.cs:204`, `:257`, `:743-765`, `:833-835`; `Runtime/Drivers/DriverHostActor.cs:1251-1252` | + +Both value seams skip Roslyn for the mirror shape `return ctx.GetTag("X").Value;` (`PassthroughScript.TryMatch`, +`RoslynVirtualTagEvaluator.cs:111-118`, `CalculationEvaluator.cs:167-176`). + +### 1.2 Timeout and cancellation model + +Every compiled run in production goes through `ScriptRunGuard.RunAsync` (`Core.Scripting/ScriptRunGuard.cs:170-304`). +There are three guard instances per seam owner: one per `RoslynVirtualTagEvaluator` +(`RoslynVirtualTagEvaluator.cs:71`), one per `CalculationEvaluator` (`CalculationEvaluator.cs:88`) and one +per `ScriptedAlarmEngine` (`ScriptedAlarmEngine.cs:258`). + +1. **Single flight per key.** A CAS on `state.InFlight` refuses a second run while the previous one, an + orphan included, still holds the slot. The caller gets `ScriptRunRefusedException` → + Bad `"previous evaluation still running"` (`ScriptRunGuard.cs:188-219`). +2. **The run is pushed to the pool** (`Task.Run`, `:231-249`), so a CPU-bound script cannot run inline and + complete before the timeout is registered. The budget clock starts only when the pool picks the run up + (`:268`). +3. **The wait is `runTask.WaitAsync(timeout, ct)`** (`:273`). On timeout the guard cancels the run token + (`:279`). Only a script that polls `ctx.CancellationToken` can see that. +4. **Quarantine** happens on any of three triggers: + - N consecutive timeouts (`:332-341`, default 3); + - a refusal behind an orphan older than `StuckOrphanQuarantineFactor` × the timeout (`:199-216`, + default 10×); + - an allocation breach, where `GC.GetAllocatedBytesForCurrentThread` is sampled before and after the + delegate (`:235`, `:241`, `:262-263`, `:296-302`; default 64 MB, `ScriptingOptions.cs`). + + A quarantine lifts on a source change or a redeploy (`ResetAll`, called from `ClearCompiledScripts` at + `RoslynVirtualTagEvaluator.cs:212-218` and from `ScriptedAlarmEngine.LoadAsync`). +5. **What the operator sees.** + - A VT or Calc failure publishes `BadInternalError` with the last-known value, once per Good→Bad + transition (`VirtualTagActor.cs:384-392`, `CalculationDriver.cs:640-647`). + - An alarm whose predicate did not complete keeps its frozen condition state, annotated + `BadInternalError` (`ScriptedAlarmEngine.cs:895-907`). + +`TimedScriptEvaluator` (`TimedScriptEvaluator.cs:80-83`) is the older wrapper. Its own remarks say it is now +used only by single-shot callers (`:18-27`). + +### 1.3 Why a runaway script cannot be pre-empted + +.NET has no supported way to stop a managed thread. `Thread.Abort` is gone in .NET Core, and the guard's +`WaitAsync` returns control to the **caller**; it does not stop the run. Concretely: + +| Failure | What happens today | Evidence | +|---|---|---| +| `while(true){}` (ignores the token) | The pool thread spins **forever**, holding one core and its collectible ALC. The guard bounds this to one orphan per key and then quarantines, but the thread keeps spinning until the process restarts. N such scripts peg N cores. | `ScriptRunGuard.cs:17-22` remarks; `ScriptRunGuardTests.cs:170-205` asserts *exactly one* orphan stays in flight (`InFlightCount.ShouldBe(1)`) | +| Fast allocation (`new byte[int.MaxValue]` in a loop, or a growing list) | The sample is taken **after** the delegate returns, so the process can OOM first. The whole server dies, including OPC UA sessions and every driver. | `ScriptRunGuard.cs:39-43`; `docs/VirtualTags.md:40`; review `docs/reviews/2026-10-21/05-core.md:100-105` | +| Unbounded recursion (a local function or a self-referencing `Func<>`; neither is a class member, so `EnforceSingleRunMember` does not see it) | `StackOverflowException` cannot be caught in .NET and **terminates the host process**. **[A-1]** I did not reproduce this; it follows from runtime semantics, and no analyzer rule targets recursion (`git grep -i "recursion\|StackOverflow"` in `Core.Scripting` finds nothing). | `ScriptEvaluator.cs:228-281` (member check only) | +| Collectible-ALC reclaim of a stuck script | `ClearCompiledScripts` disposes the evaluator, but an ALC cannot be collected while a thread is executing its code, so a stuck orphan pins its assembly until restart. **[A-2]** This is .NET ALC semantics, not observed here. | `CompiledScriptCache.Clear`; `ScriptEvaluator.Dispose` | +| A slow (not runaway) script | VT: blocks a default-dispatcher thread for up to 2 s per evaluation. Calc / alarm: holds the driver-wide or engine-wide gate, so every sibling waits behind it (up to 2 s / 250 ms each). | `RoslynVirtualTagEvaluator.cs:174-175`; `CalculationDriver.cs:539-541`; `ScriptedAlarmEngine.cs:743-765` | + +The sandbox (`ForbiddenTypeAnalyzer.cs:62-80`: `System.IO`, `System.Net`, `System.Diagnostics`, +`System.Reflection`, `System.Threading[.Tasks]`, `Environment`, `GC`, …) blocks *capabilities*, and it does +that well. It cannot block *consumption* (CPU, memory, stack). Every review that looked at this reached the +same conclusion: + +- `docs/VirtualTags.md:40,54`; +- `docs/ScriptEditor.md` §"Runtime safety"; +- `docs/superpowers/plans/2026-10-22-review-remediation.md` Lane E2; +- `docs/reviews/2026-10-21/05-core.md:103` and `03-host-controlplane-security-config.md:38`. + +All of them say the fix is an **out-of-process runner**. + +Script authoring is gated by the `ConfigEditor` policy, which admits Administrator **or Designer** (CLAUDE.md +§Scripting). The issue says "admin-gated", but the gate also admits Designers. That is the primary control, +and it explains why this work is p3. + +--- + +## 2. Goals and non-goals + +**Goals** + +- G1. **Pre-emption.** A script that exceeds its budget is stopped within a bounded time. The thread, the + memory and the ALC are all reclaimed. +- G2. **Containment.** No script (OOM, stack overflow, `FailFast`-equivalent, native crash) can take down the + OtOpcUa host process, its OPC UA sessions, its drivers or its Akka membership. +- G3. **Attribution.** The script that caused a kill is identified and quarantined. Healthy scripts that + shared the worker are not. +- G4. **Parity.** Compile-time behaviour is unchanged: the same sandbox, analyzer, diagnostics and Monaco parity. + Runtime semantics are unchanged too: the same `ctx` API, the same input-quality gating, the same Bad / last-value + contract, and the same quarantine lift rules. +- G5. **Bounded overhead** (§8). It must fit the tag counts the repo already tests at. +- G6. **Flag-gated, reversible rollout.** In-process stays the default until the gates pass. + +**Non-goals** + +- N1. A new sandbox. `ForbiddenTypeAnalyzer` stays the capability control; process isolation is + defence-in-depth, not a replacement. +- N2. Moving the **authoring** surface. The AdminUI only compiles and analyzes (`ScriptAnalysisService`), and it + never executes a script, so central/admin nodes are out of scope. +- N3. Multi-tag cascades (`ctx.SetVirtualTag` taking effect). They are dropped today and stay dropped. +- N4. Cross-node or remote execution. The worker is always local to the driver node that owns the tags. +- N5. Per-run memory *pre-emption*. Even out of process, the limit is per *worker*. A per-run limit stays a + sampled observation (§5.4). +- N6. Running non-.NET or untrusted third-party code. + +--- + +## 3. Process model + +### 3.1 Options + +| Option | Blast radius of a kill | Process count | Verdict | +|---|---|---|---| +| A. One worker per node | Every script on the node blips | 1 | Simple, but one bad script restarts every script on the node | +| B. One worker per script | One script | = script count (hundreds to thousands) | Infeasible: ~tens of MB per .NET process **[A-3]** | +| C. One worker per seam or driver instance | One seam / driver | 3 to ~10 | Arbitrary; a hot Calc driver still shares fate with its siblings | +| **D. Small sharded pool + an isolation worker** | One shard (1/K of the scripts), then one script | K + 0..1 (default 2 + 1) | **Recommended** | + +### 3.2 Recommended: sharded pool with an isolation worker + +- **K shard workers per driver node.** `Scripting:Worker:Shards` defaults to 2 and is clamped to 1–8. A + script is placed by `hash(scriptKey) mod K`. The script key is the guard key used today: the VT id, the calc + RawPath, or the alarm id. Placement is deterministic, so a script's compiled PE and warm state stay on one + worker across restarts. +- **One isolation worker** ("penalty box") is spawned on demand. Scripts under suspicion (§5.3) run there, + **one evaluation at a time**. A kill in the isolation worker is therefore unambiguous. +- **Inside a worker:** a bounded evaluation scheduler with `P` dedicated threads, where + `P = max(2, ProcessorCount / K)` **[A-4]**. Single flight per key is kept, as the guard does today. A queue + that is full answers `Busy`, which the host treats as a refusal, not a strike. +- **The passthrough fast path stays in the host.** It never runs user code, and it is probably the largest + share of evaluations **[A-5]**. + +### 3.3 Lifecycle and supervision under Akka + +The design splits a **control plane** (actors) from a **data plane** (a lock-free client). An Akka `Ask` per +evaluation would add mailbox hops to the hottest path for no benefit. + +``` +DriverHostActor (driver role) + └─ ScriptRunnerActor node-wide; owns the pool policy, attribution ledger, quarantine table + ├─ ScriptWorkerActor #0 owns one OS process + pipe; state machine below + ├─ ScriptWorkerActor #1 + └─ ScriptWorkerActor iso spawned on first suspect, stopped after IsolationIdle (default 10 min) + +IScriptRunner (DI singleton) ── data plane ──► ScriptWorkerConnection (per worker; pipe writer + correlation map) + used by RoslynVirtualTagEvaluator, CalculationEvaluator, ScriptedAlarmEngine +``` + +- **Ownership.** `ScriptRunnerActor` is created by `DriverHostActor`, next to `VirtualTagHostActor` and the + scripted-alarm host, so it exists only on driver-role nodes and dies with the driver tree. Its supervisor + strategy is `OneForOne`, and a restart of a `ScriptWorkerActor` kills and respawns its process. +- **Process exit is a message, not an exception.** A watcher on `Process.Exited` / `WaitForExitAsync` sends + `WorkerExited(exitCode, signal, inFlight[])` to its `ScriptWorkerActor`, so worker crashes never escalate + through Akka supervision. +- **Worker states:** + + `Spawning → Handshaking → Loading → Ready ⇄ Draining → Dead → Backoff → Spawning` + + `Ready` is the only state that accepts evaluations. In the other states the connection queues per key, with + latest-wins coalescing (§9). +- **The host process must take its workers down with it.** + - **Windows:** every worker is assigned to a Job Object with `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE`. If the host + dies, the kernel closes the job handle and kills the workers. + - **Linux** (the docker-dev rig): the worker treats **pipe EOF as fatal**, and also polls its parent PID. The + host is not trusted to clean up after itself. +- **CoordinatedShutdown.** In the `before-actor-system-terminate` phase, the host sends `Shutdown`, waits + 1 s, then kills. +- **Orphan sweep at start.** The host kills only workers whose launch nonce file sits under *this* host's + instance directory, never "every process with this name". mxaccessgw's `OrphanWorkerTerminator` killed a + live gateway's workers when a second gateway started (scadaproj memory "wonder upgrade program"); this + design avoids that by construction. +- **Redeploy.** `ClearCompiledScripts` / `LoadAsync` become `IScriptRunner.ApplyGeneration(scriptSet)`. That + call sends `LoadScripts` / `Unload` deltas to each shard and clears the host-side quarantine rows for edited + or removed keys, which keeps today's lift rule. No worker restart is needed. Collectible ALCs in the + worker are unloaded as they are today, and a stuck one no longer matters because its worker gets killed. +- **Redundancy.** No change. Each node runs its own scripts today and will keep doing so; workers are not + shared across the pair. + +--- + +## 4. IPC + +### 4.1 Shape of the conversation + +§1.1 showed that a run needs no host callbacks. Each evaluation is therefore **one request frame and one +reply frame**, with no re-entrant RPC. ScadaBridge's `SandboxWorkerProtocol` has to proxy `call`/`reply` +frames for external systems and databases; this design does not. Log lines from `ctx.Logger` are the only +other worker→host traffic, and they are fire-and-forget. + +### 4.2 Transport: named pipe vs gRPC + +| | Named pipe + length-prefixed frames | gRPC (h2c over UDS / named pipe) | +|---|---|---| +| Per-call overhead | Two syscalls plus serialize; no HTTP/2 framing **[A-6]** | HTTP/2 stream setup, HPACK and a Kestrel pipeline per call | +| Worker footprint | Plain console app | Needs Kestrel in the worker, or a client stack in it and a server in the host | +| Batching / priority | Free to design: control frames ahead of events, as in mxaccessgw `WorkerFrameProtocol` | Per-RPC; batching means streaming RPCs, which is a frame protocol again | +| Cross-platform | `NamedPipeServerStream` maps to a Unix domain socket on Linux; `PipeOptions.CurrentUserOnly` on both | Yes | +| Prior art in the family | mxaccessgw gateway↔worker (protobuf `WorkerEnvelope`, 4-byte LE length, priority-drained writer); ScadaBridge sandbox (JSON frames over stdio, 1 MiB cap) | OtOpcUa mesh transport (`Commons/Protos/mesh_transport.proto`) — but that is cross-node | + +**Recommendation: a named pipe with 4-byte little-endian length-prefixed protobuf envelopes.** It copies +mxaccessgw's `WorkerFrameProtocol` discipline: + +- a frame-size cap negotiated at the handshake (default 4 MiB); +- zero-length and oversize frames rejected; +- a monotonically increasing `seq`; +- control frames (heartbeat, shutdown, fault) drained ahead of evaluation traffic; +- coalesced flushes. + +`Google.Protobuf` and `Grpc.Tools` are already central pins, and `Commons.csproj:28-30` already compiles +protos, so no new package is needed. `MessagePack` is pinned in `Directory.Packages.props` but referenced by +no project, so it would be a new dependency in practice. Stdio, as ScadaBridge uses, is rejected: a script +cannot write to the console (it is forbidden), but the runtime and the logger can, and one stray byte on +stdout corrupts the stream. + +### 4.3 Message schema (sketch) + +```proto +syntax = "proto3"; +package otopcua.scriptworker.v1; + +message Envelope { + uint32 protocol_version = 1; + uint64 seq = 2; + oneof body { + Hello hello = 10; // host→worker: nonce, max_frame_bytes, options + Ready ready = 11; // worker→host: pid, runtime version, sandbox fingerprint + LoadScripts load = 12; // host→worker: add/replace compiled-or-source scripts + LoadResult load_result = 13; // worker→host: per-script compile outcome + diagnostics + Unload unload = 14; // host→worker: drop keys (ALC unload) + EvalBatch eval = 15; // host→worker + EvalResults results = 16; // worker→host (one or more completions) + LogBatch logs = 17; // worker→host, fire-and-forget, bounded + Heartbeat heartbeat = 18; // both ways; carries worker gauges (inflight, heap, gen2 count) + Shutdown shutdown = 19; + Fault fault = 20; + } +} + +enum ContextKind { CONTEXT_KIND_UNSPECIFIED = 0; VIRTUAL_TAG = 1; CALCULATION = 2; ALARM_PREDICATE = 3; } + +message ScriptSource { string key = 1; bytes source_sha256 = 2; ContextKind kind = 3; + string source = 4; bytes pe_image = 5; repeated string input_paths = 6; } +message LoadScripts { uint64 generation = 1; repeated ScriptSource scripts = 2; } + +message EvalBatch { repeated EvalRequest items = 1; } +message EvalRequest { + uint64 eval_id = 1; string key = 2; bytes source_sha256 = 3; + repeated Input inputs = 4; // positional against ScriptSource.input_paths; see 4.4 + int64 now_utc_ticks = 5; // ctx.Now — host clock, so tests that pin the clock still work + uint32 soft_timeout_ms = 6; +} +message Input { uint32 slot = 1; Variant value = 2; uint32 status_code = 3; + int64 source_ts_ticks = 4; /* 0 = null */ int64 server_ts_ticks = 5; } + +message EvalResult { + uint64 eval_id = 1; + oneof outcome { Variant value = 2; bool predicate = 3; ScriptError error = 4; } + uint64 allocated_bytes = 5; // same GC.GetAllocatedBytesForCurrentThread sample as today + uint32 run_micros = 6; +} +message EvalResults { repeated EvalResult items = 1; } +message ScriptError { Kind kind = 1; string message = 2; + enum Kind { THREW = 0; COMPILE = 1; SANDBOX = 2; SOFT_TIMEOUT = 3; UNSUPPORTED_TYPE = 4; BUSY = 5; } } + +message Variant { + TypeTag type = 1; // exact CLR type — see 4.4 + oneof v { bool b = 2; sint64 i = 3; uint64 u = 4; double d = 5; float f = 6; + string s = 7; int64 dt_ticks = 8; ArrayValue array = 9; } + enum TypeTag { NULL = 0; BOOLEAN = 1; SBYTE = 2; BYTE = 3; INT16 = 4; UINT16 = 5; INT32 = 6; UINT32 = 7; + INT64 = 8; UINT64 = 9; FLOAT = 10; DOUBLE = 11; STRING = 12; DATETIME_UTC = 13; + ARRAY = 14; } +} +message ArrayValue { Variant.TypeTag element_type = 1; repeated Variant items = 2; } + +message LogBatch { repeated LogLine lines = 1; uint32 dropped = 2; } +message LogLine { string key = 1; uint32 level = 2; string message_template = 3; + string rendered = 4; string exception = 5; int64 ts_ticks = 6; } +``` + +### 4.4 How values and quality cross the boundary + +- **Quality crosses verbatim.** `DataValueSnapshot` is `(object? Value, uint StatusCode, DateTime? + SourceTimestampUtc, DateTime ServerTimestampUtc)` (`Core.Abstractions/DataValueSnapshot.cs:13`), and it maps + 1:1 onto `Input`. The worker rebuilds the same snapshots, so `ctx.GetTag(p).StatusCode` and + `.SourceTimestampUtc` behave as they do today. The input-quality gating does not move: VT holds evaluation + while an input is Bad (`VirtualTagActor.cs:264-269`), Calc holds on Bad, and alarms short-circuit on + `AreInputsReady`. A request is sent only when today's code would have run the script. +- **The exact boxed CLR type is load-bearing.** A script that writes `(short)ctx.GetTag(x).Value` throws + `InvalidCastException` if the value arrives boxed as an `int`. `Variant.type` therefore carries the CLR type, + not just the OPC UA type: `DriverDataType` has `Int16` / `UInt16`, which have to survive the round trip. + **[A-7]** I have not inventoried which CLR types each driver actually boxes, for example whether a Modbus + `Int16` arrives as `short` or as `int`. The shadow mode in §12 exists partly to find out. +- **Unsupported types.** An input whose CLR type has no tag is not silently converted. The request carries no + value for that slot, and the worker answers `UNSUPPORTED_TYPE`; the host publishes Bad with the type name. On + the result side, a script that returns an unsupported type (a `List`, an anonymous object) also gets + `UNSUPPORTED_TYPE`. That is a **behaviour change**: today the raw `object` goes to + `OpcUaPublishActor.AttributeValueUpdate` (`VirtualTagHostActor.cs:233`), and what the sink does with it was + not traced **[A-8]**. Shadow mode reports every such case before the flag flips. +- **DateTime.** Values are sent as UTC ticks. A `Kind` other than `Utc` is converted at the host edge and + counted on a metric. **[A-9]** I assume drivers already produce UTC. +- **Positional inputs.** `LoadScripts` registers the script's ordered `input_paths`: the + `DependencyExtractor` reads, after the `{{equip}}` resolution done at deploy. Requests then send values by slot, + not by path string, which takes the dominant payload cost out of the hot path. A key with no registered + slots, a fallback that should never happen, sends the paths inline. + +### 4.5 Batching + +- **Opportunistic, not timed.** The connection's writer drains every request queued at that moment into one + `EvalBatch` and flushes once, as mxaccessgw's drain pass does. An idle system pays no added latency; a burst + is amortised automatically. There is no coalescing *timer*, because it would add latency to every + evaluation. +- **Results are streamed, not batch-atomic.** The worker sends `EvalResults` as items complete, coalescing + whatever finished in the same scheduler pass, so a slow script never holds a fast sibling's reply. +- **The seams must submit concurrently, or batching never happens.** Today both Calc and alarms await one + script at a time (§1.1), and with IPC that serial loop costs N × round-trip. Two seam changes are required: + - **Scripted alarms:** `ReevaluateAsync` submits the whole dirty set at once, then applies results **in the + original order** under `_evalGate`. Predicates are pure functions of a snapshot taken under the gate, so + the state machine sees the same sequence. + - **Calculation:** the pump dequeues everything pending, with each entry's dependency snapshot already + captured at enqueue, submits it as one batch and publishes in dequeue order. +- **Virtual tags become asynchronous.** `IVirtualTagEvaluator` gains `EvaluateAsync`, and `VirtualTagActor` + `PipeTo`s the result to itself. While a run is in flight, further dependency changes update the map and set a + "re-run" flag; when the result lands, the actor evaluates **once more** against the latest map (latest-wins). + This also ends today's blocking of a default-dispatcher thread for up to 2 s + (`RoslynVirtualTagEvaluator.cs:174-175`). + +--- + +## 5. Pre-emption + +### 5.1 Two deadlines + +| | Soft deadline `T` | Hard deadline `T + G` | +|---|---|---| +| Value | Today's per-seam budget, unchanged: VT 2 s, Calc `runTimeoutMs` (2 s), alarm 250 ms | `G = Scripting:Worker:HardGrace`, default `max(1 s, T)`, clamped to 250 ms–30 s | +| Who acts | The host connection's timer | The host `ScriptWorkerActor` (authoritative) **and** a worker-side watchdog thread | +| Effect | The caller gets the timeout failure it gets today (Bad, same reason prefix `ScriptEvalFailureReasons.TimedOutPrefix`) and the run token is cancelled, so a polling script still ends itself as it does today | The worker process is **killed** (`Process.Kill(entireProcessTree: true)`). The worker watchdog calls `Environment.FailFast` first if it is still scheduling, so the dump and exit code name the key. | +| Strike | Yes (as today) | Yes, plus a kill strike (§5.3) | + +The grace period `G` keeps a script that overran by a few milliseconds from costing a whole shard restart, and +the cooperative token from #734 still gets its chance. Making `G = 0` (kill at `T`) is an open decision (§14). + +**Single flight between T and T+G.** The key stays in flight, and a trigger is refused as today, which is not +a strike. After the kill there is **no orphan**, so the stuck-orphan rule (`StuckOrphanQuarantineFactor`) +becomes moot out of process; it stays for the in-process mode. + +### 5.2 Resource limits on the worker + +The limits combine prior art from ScadaBridge (heap limit) and mxaccessgw (process-tree kill): + +- **Heap:** `DOTNET_GCHeapHardLimit = Scripting:Worker:HeapLimitBytes`, default 512 MiB **[A-10]**. Breaching + it makes the worker throw `OutOfMemoryException` or fail fast. Either way the worker dies or reports, and the + host survives. +- **Windows Job Object** (per worker): + - `JOB_OBJECT_LIMIT_PROCESS_MEMORY`, set slightly above the heap limit as a backstop for native and JIT + memory; + - `JOB_OBJECT_LIMIT_ACTIVE_PROCESS = 1`, so no child processes are possible; + - `KILL_ON_JOB_CLOSE`; + - UI restrictions. +- **Linux / containers:** the heap limit only. A container's cgroup is the rig's concern. **[A-11]** I assume + the docker-dev containers run without CAP_SYS_ADMIN, so a nested cgroup is not available. +- **Liveness:** a heartbeat every 1 s. Three missed heartbeats while not in a GC pause, which the heartbeat + reports, trigger a kill with reason `heartbeat`. This catches a wedged worker that neither replies nor exits. + +### 5.3 Attribution, restart, backoff and quarantine + +A kill or crash takes down every in-flight evaluation on that shard. Attribution decides which **one** to +blame. + +1. **Hard-deadline kill:** the key that crossed `T+G` is known exactly. It takes a **kill strike**. The other + in-flight keys on that shard are victims: their runs fail with "script worker restarted", which is not a + strike. +2. **Crash with no deadline crossed** (OOM, stack overflow, native fault): attribution is ambiguous. Every + key in flight at exit becomes a **suspect**, and suspects move to the **isolation worker**. +3. **Isolation worker:** evaluations run one at a time, so a crash there attributes to exactly one key, which + takes a kill strike. A suspect that completes `IsolationProbation` (default 20 successful runs or 10 min) + without incident returns to its shard. +4. **Quarantine:** a key is quarantined after `Scripting:Worker:QuarantineAfterKills` kill strikes (default 2) + within 1 h, or by today's rules (N consecutive soft timeouts, allocation breach). The quarantine table + lives in the **host** (`ScriptRunnerActor`), so it survives worker restarts. It is lifted by a source + change or a redeploy, exactly as today, and quarantine reasons keep the `"script quarantined: "` prefix so + every existing consumer and test classifier still matches. +5. **Restart backoff per worker:** exponential, 250 ms → 30 s cap, ×2, with ±20% jitter. The backoff resets + after 60 s of `Ready`. A worker that dies **during** `Loading` (a poison PE or a runtime fault) does not + loop: after 3 consecutive load-phase deaths, the scripts in its generation are loaded one by one in the + isolation worker to find the culprit. +6. **Crash-loop circuit:** if all shards are in backoff for longer than `Scripting:Worker:CircuitOpenAfter` + (default 2 min), the runner reports `Unavailable` (§10) and logs one Error. It keeps retrying at the cap. + +--- + +## 6. Security + +Process isolation contains **failures**. Most of the **capability** control stays with the compile-time +sandbox, which is retained. Defence-in-depth adds the following. + +- **The compile-time sandbox is unchanged.** It still includes `ScriptSandbox` (with the TPA reference set + filtered to `System.*`, `ScriptSandbox.cs:118-122`), `ForbiddenTypeAnalyzer`, and the injection guard. The + worker runs **exactly** the same `ScriptEvaluator.Compile`, so editor/publish/runtime parity holds. +- **Assembly allow-list by construction.** The worker is its own app (`ZB.MOM.WW.OtOpcUa.ScriptWorker`), with + its own `deps.json` and only these references: + - `Core.Scripting`; + - `Core.Scripting.Abstractions`; + - `Core.Abstractions`; + - Serilog (core only, for `ILogger` on `ctx`); + - Google.Protobuf. + + Its trusted-platform list therefore contains the shared framework plus those assemblies. Akka, SqlClient, the + OPC UA stack, driver assemblies and the Secrets library are **not loadable** in the worker, even through + reflection. At startup the worker asserts its loaded-assembly set against a pinned list and exits if the list + differs. +- **No secrets in the environment.** `ProcessStartInfo.Environment` is **cleared** and then repopulated from + an explicit allow-list: `DOTNET_*` runtime knobs, `TEMP`, `PATH` minimal, and the launch nonce. Host env vars + like `ServerHistorian__ApiKey` (CLAUDE.md "Never commit secrets") and the ConfigDb connection string must + never reach the worker. ScadaBridge's sandbox **inherits** the parent environment; do not copy that part. +- **No configuration access.** The worker gets no appsettings path. Every option it needs (budgets, heap, + frame cap) arrives in `Hello`. +- **Pipe hardening (mxaccessgw pattern):** + - a random pipe name; + - `PipeOptions.CurrentUserOnly`, or an explicit ACL when the worker runs as a different account; + - a one-time nonce passed in an env var (never on the command line; mxaccessgw `WorkerBootstrap.md` explains + why), which the worker must echo in `Ready`; + - the host accepts exactly one connection per launch. +- **Filesystem:** + - the working directory is an empty per-worker temp directory; + - the PE cache (§7) is opened read-only by the worker, if the host compiles, or written only under its own + subdirectory; + - on Windows, a dedicated low-privilege account (see §14) gets read access only to the install folder and + its own temp directory. +- **Network:** the analyzer already forbids `System.Net`. Process-level options are a Windows Firewall + outbound-block rule scoped to the worker executable, created by the installer, and on Linux nothing beyond + the container. This is an open decision (§14). The recommendation is the installer firewall rule on Windows, + and to accept the analyzer as the control on Linux. +- **What the host trusts from the worker:** nothing it cannot validate. Frames are size-capped and + schema-checked, an `eval_id` must match an outstanding request, and log lines are length-capped and + rate-limited (§11). A compromised worker can at worst return wrong values for scripts on its own shard, + which the same script could have done anyway. + +--- + +## 7. Compile caching across restarts + +The problem: today every process start recompiles every script lazily (`CompiledScriptCache` is memory-only). +Out of process, a **shard restart** has the same cost, and it can happen in the middle of production running. +mem-probe measured the old `CSharpScript` path at roughly 2.4 MiB of working set per retained script +(`tools/mem-probe/MemProbe/Program.cs:1-4`). Compile *time* is not measured anywhere in the repo **[A-12]**; +I assume tens of ms per script warm and seconds for the first compile while Roslyn JITs. For 1,000 scripts, +that is far outside a 2 s restart budget. + +**Recommendation: a content-addressed PE cache on node-local disk.** + +- **Key:** SHA-256 over these fields: + - the source; + - the `ContextKind`, i.e. `TContext` and `TResult`; + - the Roslyn compiler version; + - a **sandbox fingerprint**: a hash of `ForbiddenTypeAnalyzer`'s lists, the `ScriptSandbox` reference set + and the wrapper template. + + An analyzer change therefore invalidates every entry, and a cached PE can never skip a newer sandbox rule. +- **Value:** the emitted PE bytes plus an HMAC-SHA256 tag. The HMAC key is a per-node random key created at + first start and stored next to the cache, readable only by the host account. The worker loads a PE only if + the HMAC verifies, because loading attacker-supplied PE bytes would be code execution outside the analyzer. +- **Location:** `Scripting:Worker:PeCacheDir`. It must be absolute and outside the content root, the same + fail-closed rule the family applies to `Secrets:SqlitePath` (scadaproj CLAUDE.md, Secrets row). Entries + whose key is not in the current generation are pruned at `ApplyGeneration`. +- **Where compilation happens:** **in the worker** (recommended). Roslyn's memory and CPU leave the host. A + pathological source that makes Roslyn itself blow up kills a worker, not the server, and the worker writes + the PE plus HMAC (it holds the key through `Hello`). The alternative is to compile in the host and ship + `pe_image` in `LoadScripts`. That keeps the worker free of Roslyn (smaller, faster start), but compile-time + blowups stay in the host. This is an open decision (§14). +- **Restart path:** `Spawning → Ready` takes about 150 ms of runtime start **[A-13]**. Then `LoadScripts` + sends hash + key + slots only. The worker loads PEs from the cache lazily, on each key's first evaluation, + and a cache miss compiles. Loading a PE into a collectible ALC is about 1 ms **[A-13]**. +- **Negative cache:** Calculation already remembers compile failures by source hash (R07:P4, + `CalculationEvaluator.cs:63-68`). The worker reports compile failures in `LoadResult`, and the host keeps + that negative cache for **all** seams, so a broken script never re-enters Roslyn. + +--- + +## 8. Performance budgets + +### 8.1 Reference loads in the repo + +| Source | Load | +|---|---| +| `docs/v2/Galaxy.Performance.md` (EventPump sizing, 24 h soak) | 50,000 tags at 1 Hz on one node; compressed CI soak at 1,000 tags | +| `tests/Server/.../NodeManagerLockTimeBenchmarkTests.cs:41-105` | 10,000-tag folder; 200 equipment × 250 tags (50,000 variables) | +| `ScriptRunGuardTests.cs:170-205` | a 1 Hz runaway and a healthy sibling over 30 triggers | + +No repo perf test exercises scripts at scale. **[A-14]** I assume a planning envelope of **5,000 compiled +script evaluations per second per node** (for example 5,000 VT + Calc at 1 Hz, or 500 scripts at 10 Hz) and +size for 2× headroom (10k/s). The passthrough fast path is excluded because it stays in the host. + +### 8.2 Budgets (acceptance gates for the implementation) + +| Metric | Budget | Notes | +|---|---|---| +| Added latency per evaluation, p50 | ≤ 150 µs | Host enqueue → reply decoded, trivial script, idle system | +| Added latency per evaluation, p99 | ≤ 2 ms | At 10k evals/s. Compare the 250 ms alarm budget: under 1% of it | +| Throughput | ≥ 10,000 trivial evals/s/node with K=2 | With opportunistic batching | +| Host CPU for IPC | ≤ 25% of one core at 10k evals/s | Serialisation + pipe I/O, measured by an opt-in perf test | +| Bytes per eval on the wire | ≤ 64 B + 24 B per input | Positional slots, §4.4 | +| Shard restart → first result, 1,000 cached scripts | ≤ 2 s | §7 lazy PE load | +| Worker idle RSS | ≤ 80 MB + scripts **[A-3]** | Per shard | +| Kill latency after `T+G` | ≤ 100 ms | Host timer resolution plus `Process.Kill` | + +**Comparison with today:** the in-process cost per compiled run is one `Task.Run` hop plus `WaitAsync`, which +is a few µs to tens of µs and was never measured **[A-15]**. The out-of-process path is therefore one to two +orders of magnitude slower **per call**. That is acceptable only because the budgets above sit two to three +orders of magnitude below the script budgets (250 ms / 2 s), and because batching amortises the syscalls. The +risk case is a large scripted-alarm engine, with its serial gate and its 250 ms budget. That is why §4.5 +makes the batch-submit change a **prerequisite**, not an optimisation. + +--- + +## 9. Failure semantics: what quality a tag gets + +The principle: **a runner failure must never look like a script result.** In particular, an alarm must never +clear because the worker died. + +| Situation | Virtual tag / Calculation tag | Scripted alarm | +|---|---|---| +| Soft timeout (`T`) | `BadInternalError` + last value, once per transition, as today | Condition frozen + `BadInternalError` quality annotation, as today | +| This key killed at `T+G`, or crashed in isolation | Same as soft timeout; reason `"script killed: exceeded hard deadline …"` | Same as soft timeout | +| Victim: in flight on a shard that died for another key | `BadInternalError` + last value, reason `"script worker restarted"`; re-evaluated on the shard's return with the latest inputs (latest-wins) | Frozen + Bad annotation; re-evaluated on return | +| Not in flight while its shard restarts | **Unchanged.** Keeps its last published value and quality. A trigger during the outage is coalesced (latest-wins, one per key) and evaluated when the shard is `Ready` | Same; the condition holds its state | +| Shard not back within `Scripting:Worker:UnavailableGrace` (default 5 s) | Coalesced keys publish Bad + last value, reason `"script runner unavailable"` | Frozen + Bad annotation | +| Circuit open (§5.3.6) | Every compiled-script key on the node is Bad as above; passthrough keys are unaffected | Every compiled predicate is frozen + Bad | +| Quarantined | `"script quarantined: …"` Bad, as today | As today | +| Host process dies | Workers die with it (Job Object / pipe EOF); no change from today's host-death behaviour | — | + +**Status code:** `BadInternalError` stays for script-attributable faults (the existing contract). For "the +runner is unavailable, not this script's fault", this design proposes **`BadResourceUnavailable` +(0x80040000)**, so operators and the ScadaBridge consumer can tell a broken script from a sick node. The +alternative is to keep `BadInternalError` everywhere for zero consumer impact. This is an open decision (§14); +if it is taken, the ScadaBridge consumer-impact record must note it (scadaproj `sister-project-history` +skill, `otopcua.md`). + +**No silent fallback to in-process.** If the worker cannot start, the runner does **not** run scripts in the +host. That would reintroduce exactly the risk the flag was set to remove. Operators flip the flag back (§12). +This is an open decision (§14). + +--- + +## 10. Observability + +All metrics go on `OtOpcUaTelemetry.Meter`, so the existing `ZbTelemetryOptions` allow-list applies. +scadaproj memory: "`ZbTelemetryOptions.Meters` is a silent allowlist"; check that the new names flow end to +end. + +| Metric | Type | Tags | +|---|---|---| +| `otopcua.script.worker.up` | gauge | `shard` | +| `otopcua.script.worker.starts` / `.exits` | counter | `shard`, `reason` = `clean` \| `killed_deadline` \| `killed_heartbeat` \| `oom` \| `crash` \| `load_failure` | +| `otopcua.script.worker.inflight` / `.queue_depth` | gauge | `shard` | +| `otopcua.script.worker.heap_bytes` | gauge (from heartbeat) | `shard` | +| `otopcua.script.ipc.roundtrip` | histogram (µs) | `kind` | +| `otopcua.script.eval.duration` | histogram (µs, worker-measured) | `kind` | +| `otopcua.script.compile` | counter | `cache` = `hit` \| `miss` \| `negative` | +| `otopcua.script.quarantined` (existing) | counter | adds `reason` = `killed` \| `isolated_crash` | +| `otopcua.script.run_refused` (existing) | counter | unchanged | +| `otopcua.script.isolation.suspects` | gauge | — | +| `otopcua.script.runner.state` | gauge (0 ready / 1 degraded / 2 unavailable) | — | + +**Per-key series:** the existing counters tag the series with `script = key`. Do not add `key` tags to the new +per-eval histograms (cardinality). + +**Logs:** +- the worker's own stderr is captured, prefixed with `ScriptWorker[shard]`, into the host log; +- every kill logs one Warning naming the key and the reason, and every quarantine logs one Error, as today; +- `ctx.Logger` lines come back in `LogBatch`. The host re-emits them through `ScriptRootLogger` with the same + `ScriptId` / `VirtualTagId` properties, so `scripts-*.log`, the error mirror, the DPS script-log topic and its + 20/min-per-source rate limit all keep working unchanged; +- the worker caps its outbound log buffer (default 1,000 lines / 1 MiB) and reports drops as + `LogBatch.dropped`, so a script logging in a tight loop cannot flood the pipe. + +**Health:** expose the runner state as a non-fatal component of the driver-node status (`/health/ready` +stays green while degraded, because OPC UA still serves driver data). Whether `Unavailable` should fail +readiness is an open decision (§14). The AdminUI status dashboard gets a "Script runner" tile with the +per-shard state, restarts in the last hour and the quarantine list. Quarantines already surface through the +`QuarantinedScripts` properties. + +**Traces:** one `Activity` per `EvalBatch` (not per evaluation), with `batch.size` and `shard`. + +--- + +## 11. Seam changes in the host (summary) + +| Component | Change | +|---|---| +| `Core.Scripting` | New `IScriptRunner` (`EvaluateAsync(ScriptInvocation[])`, `ApplyGeneration`, `QuarantinedKeys`, `ResetAll`). `InProcessScriptRunner` wraps today's `ScriptRunGuard` + `CompiledScriptCache` exactly. | +| New `Core.Scripting.Protocol` | The proto file, the frame codec and the `Variant` ↔ CLR mapping. Shared by host and worker. | +| New `ZB.MOM.WW.OtOpcUa.ScriptWorker` | Console app: handshake, scheduler, watchdog, PE cache, log forwarding. Shipped in the Host's publish output. | +| New `Runtime/Scripting/` | `ScriptRunnerActor`, `ScriptWorkerActor`, `OutOfProcessScriptRunner` (data plane), Job Object interop (Windows-only, behind `OperatingSystem.IsWindows()`). | +| `IVirtualTagEvaluator` / `VirtualTagActor` | Add `EvaluateAsync`; the actor `PipeTo`s with latest-wins re-run (§4.5). | +| `RoslynVirtualTagEvaluator`, `CalculationEvaluator`, `ScriptedAlarmEngine` | Replace direct `ScriptRunGuard` use with `IScriptRunner`. Calc pump and alarm `ReevaluateAsync` submit batches (§4.5). | +| `ScriptingOptions` / `ScriptingOptionsBinding` | New keys (§12), clamped with startup Warnings like the existing ones. | + +--- + +## 12. Migration and rollout + +**Configuration** (node-local appsettings, `Scripting` section, consistent with the existing knobs): + +| Key | Default | Meaning | +|---|---|---| +| `Scripting:Runner` | `InProcess` | `InProcess` \| `OutOfProcess` \| `Shadow` | +| `Scripting:Runner:Seams` | all three | Optional per-seam subset for `OutOfProcess` / `Shadow` (e.g. `["Calculation"]`) | +| `Scripting:Worker:Shards` | `2` | 1–8 | +| `Scripting:Worker:HardGrace` | `max(1s, T)` | §5.1 | +| `Scripting:Worker:HeapLimitBytes` | `536870912` | §5.2 | +| `Scripting:Worker:QuarantineAfterKills` | `2` | §5.3 | +| `Scripting:Worker:UnavailableGrace` | `5s` | §9 | +| `Scripting:Worker:PeCacheDir` | none (required when not `InProcess`) | §7; absolute, outside the content root, fail-closed | + +**Phases** (each one is its own issue and PR): + +1. **P0, in-process refactor with no behaviour change.** + - Introduce `IScriptRunner` + `InProcessScriptRunner`. + - Make VT evaluation async (`PipeTo`), and make the alarm engine and the Calc pump batch-submit. + - Gate: every existing Scripting / VirtualTags / Calculation / ScriptedAlarms suite stays green unchanged, + plus new ordering tests for batch-then-apply. + - This phase has value on its own: it ends the dispatcher-thread blocking. +2. **P1, worker + protocol behind the flag.** + - Ship `ScriptWorker`, `OutOfProcessScriptRunner` and the PE cache. + - Default stays `InProcess`. Enable on the docker-dev rig only. + - Gate: §13 runaway / containment tests green on Linux (docker-dev) **and** Windows (windev box, which has + the .NET 10 toolchain). +3. **P2, shadow mode on docker-dev and QA.** + - `Shadow` runs the **in-process** evaluation as the published result, and *also* submits to the worker. + - It compares value, status class and failure kind per evaluation, and counts + `otopcua.script.shadow.mismatch{kind, reason}`. Mismatching keys are logged once each. + - Shadow doubles script CPU and still carries the in-process risk, so it is a time-boxed diagnostic (for + example 1 week on QA), not a mode to run in production. + - Exit criterion: zero unexplained mismatches. This is where [A-7] / [A-8] get answered. +4. **P3, flip on QA, then the other tiers.** + - Set `Scripting:Runner=OutOfProcess` per node, seam subset first (Calculation, then VT, then alarms; alarms + last because of the 250 ms budget). + - Note: remote command execution on the wonder boxes is currently scoped to user-requested deploys (CLAUDE.md + hold), so tier rollout rides normal deploys. +5. **P4, decide the long-term status of `InProcess`.** The recommendation is to keep it for dev and tests and + as the documented rollback, and to retire `StuckOrphanQuarantineFactor` from the docs as "in-process only". + +**Rollback:** set the flag back to `InProcess` and restart the node. No data or config migration is involved. +The PE cache directory can be deleted at any time. + +**Docs to update when implementing:** +- `docs/VirtualTags.md` §"Known resource limits" and §"Per-evaluation timeout"; +- `docs/ScriptEditor.md` §"Runtime safety"; +- `docs/ScriptedAlarms.md`; +- `docs/ServiceHosting.md`: the new process, Job Object and firewall rule; +- `docs/security.md`; +- `docs/Telemetry.md`; +- the scadaproj index row, if the status-code decision changes the ScadaBridge-visible contract. + +--- + +## 13. Test plan + +**Unit (`Core.Scripting.Tests`, new `ScriptWorker.Tests`):** + +- Frame codec: + - round-trip every `Variant.TypeTag` (arrays included); + - reject zero-length, oversize and truncated frames; + - reject an `eval_id` with no outstanding request; + - fuzz the reader with random bytes and assert there is no exception other than a protocol fault. +- **Unboxing parity:** for each tag, a script doing `(T)ctx.GetTag("x").Value` behaves identically in and out + of process. This pins [A-7]. +- Attribution ledger: + - a deadline kill blames exactly one key; + - a crash with 3 in flight → 3 suspects → isolation; + - an isolation crash → one strike; + - probation returns a suspect to its shard; + - quarantine after 2 kills; + - a source change lifts the quarantine; + - a redeploy lifts every quarantine. +- Backoff schedule, jitter bounds, the reset-after-healthy rule and the load-phase crash bisection. +- Options binding: clamps plus a Warning per clamped key; `PeCacheDir` relative or inside the content root is a + startup failure. + +**Containment (worker-backed, `Category=ScriptWorker`; runs in CI on Linux, gated on windev for Windows).** Each +case has a positive control: a healthy sibling script on the **same shard** that keeps producing correct values +before and after the event. Following scadaproj memory "assert the artifact before the outcome", each assertion +is checked to be capable of failing. + +| Case | Assert | +|---|---| +| `while(true){}` ignoring the token | Killed within `T + G + 100 ms`; host `ThreadPool.ThreadCount` and `Process.Threads` unchanged; no orphan (`InFlightCount == 0` after the kill). Mutation check: disable the kill and see the test fail | +| `while(!ctx.CancellationToken.IsCancellationRequested){}` | Ends at `T` by itself; **no** kill; no strike beyond the timeout | +| Allocation bomb | Worker dies of OOM or the heap limit; **host RSS stays below a bound** and the host stays up; suspect → isolation → quarantine | +| Unbounded recursion | Worker dies; host up; quarantined after the isolation reproduction. Also the first in-repo reproduction of [A-1] for the in-process mode, marked `Skip` there because it would kill the test host | +| Worker killed externally (`kill -9`) | Victims Bad with "script worker restarted", then recover with the latest inputs; the alarm condition is **not** cleared | +| Host killed | Worker exits within 2 s (pipe EOF on Linux, Job Object on Windows) | +| Environment | The worker's environment contains no `ServerHistorian__ApiKey` / ConfigDb variables (set them in the test host first: positive control) | +| Pipe | A second client connecting to the pipe is refused; a wrong nonce is refused | +| Loaded assemblies | The worker's loaded set equals the pinned list; Akka / SqlClient / Opc.Ua are absent | +| Crash loop | A poison generation trips the circuit; state `Unavailable`; keys Bad with the runner-unavailable reason; recovery after a fixed redeploy | + +**Parity:** run the existing `RoslynVirtualTagEvaluatorTests`, `CalculationDriver*Tests` and +`ScriptedAlarmEngine*Tests` against **both** runners. Use xunit theory or class fixtures over +`IScriptRunner`, with the worker built as a test dependency. + +**Performance (`Category=Perf`, opt-in, not in CI):** a Stopwatch-based harness (no new packages) that +measures the §8 table: p50/p99 added latency, throughput at K=1/2/4, host CPU, restart-to-first-result with +1,000 cached PEs, and RSS per shard. + +**Live gates:** +- docker-dev rig: a deploy with VTs, Calc tags and scripted alarms; drill a runaway script through the + AdminUI and watch `/uns` go Bad, the quarantine appear, and siblings stay Good; +- QA shadow week (P2); +- a QA flip with the same drill. + +--- + +## 14. Open decisions for the owner + +Each has a recommendation, marked ★. + +1. **Pool shape.** + - ★ K=2 shards plus an on-demand isolation worker; + - one worker per node, simplest, but every kill blips every script; + - per-seam workers. +2. **Hard grace `G`.** + - ★ `max(1 s, T)`, which keeps cooperative cancellation useful; + - `G = 0` (kill at `T`): strictest, but every soft overrun restarts a shard. +3. **Status code for "runner unavailable / worker restarted".** + - ★ `BadResourceUnavailable`, which separates a sick node from a broken script but is a ScadaBridge-visible + contract change; + - keep `BadInternalError` everywhere, for zero consumer impact. +4. **Fallback when the worker cannot start.** + - ★ fail closed: Bad, and no in-process execution; + - fail open to in-process with an Error, which keeps values flowing but silently reopens the risk. +5. **Worker identity on Windows.** + - ★ a dedicated low-privilege virtual account created by the installer, which needs installer work, an + explicit pipe ACL, and a deploy that touches the box; + - the same account as the host: zero install change, and the Job Object plus the analyzer remain the + controls. +6. **Network isolation.** + - ★ a Windows Firewall outbound-block rule on the worker executable (installer), with the analyzer as the + only control on Linux; + - analyzer only, everywhere. +7. **Compile location.** + - ★ in the worker: Roslyn blowups are contained and the host gets lighter; + - in the host, shipping PE bytes: a lighter worker and faster cold start. +8. **PE cache store.** + - ★ a node-local directory with an HMAC; + - LocalDb (already a dependency; it would also replicate to the pair, which is unnecessary); + - no persistence, recompiling on restart (violates the 2 s budget at scale). +9. **IPC encoding.** + - ★ protobuf (already compiled in `Commons`, and mxaccessgw precedent); + - MessagePack (pinned but unused, so in practice a new dependency). +10. **Scripted alarms in the first flip.** + - ★ last, after Calc and VT, because of the 250 ms budget and the engine refactor; + - all seams at once. +11. **Unsupported input/result CLR types.** + - ★ an explicit Bad `UNSUPPORTED_TYPE`, surfaced in shadow mode first; + - best-effort conversion (to string, `double`), which hides type bugs. +12. **Readiness when the circuit is open.** + - ★ degraded but ready (OPC UA still serves driver data); + - not ready, which on a redundant pair would shift ServiceLevel to the peer. +13. **Long-term status of `InProcess`.** + - ★ keep it as the dev/test default and the rollback; + - delete it after a soak period. +14. **Does this need a real-deployment scale target?** [A-14]'s 5k evals/s/node is a guess. The owner should + supply the expected VT / Calc / alarm counts per node for the target sites, or accept the guess as the + gate. + +--- + +## 15. Assumptions (not verified from code) + +| Id | Assumption | How to verify | +|---|---|---| +| A-1 | Unbounded recursion in a script throws `StackOverflowException` and kills the host today | Reproduce in a throwaway process (not the test host) | +| A-2 | A collectible ALC whose code is still executing on an orphan thread cannot be unloaded | .NET docs / a `WeakReference` test with a spinning orphan | +| A-3 | A minimal .NET 10 worker idles at tens of MB RSS | Measure the P1 worker | +| A-4 | `P = max(2, cores/K)` evaluation threads per worker is enough | P1 perf harness | +| A-5 | Passthrough scripts are a large share of evaluations on real deployments | Count `PassthroughScript.TryMatch` hits against compiled runs on QA (add a counter in P0) | +| A-6 | A local pipe round trip costs tens of µs | P1 perf harness | +| A-7 | Drivers box tag values with stable, enumerable CLR primitive types | Shadow-mode type census | +| A-8 | What `OpcUaPublishActor` / the sink does with an unsupported result object today | Trace `AttributeValueUpdate` → `SdkAddressSpaceSink` | +| A-9 | Drivers produce UTC `DateTime` values | Shadow-mode census on `DateTime.Kind` | +| A-10 | 512 MiB is a sane per-worker heap limit (ScadaBridge uses the same) | Size against measured RSS × script count | +| A-11 | docker-dev containers cannot create nested cgroups | Check the compose capabilities | +| A-12 | Warm Roslyn compile is tens of ms per script; cold is seconds | P1 perf harness | +| A-13 | Worker start is ~150 ms; a PE load into a collectible ALC is ~1 ms | P1 perf harness | +| A-14 | 5k compiled evals/s/node is a fair planning envelope | Owner input (decision 14) | +| A-15 | Today's in-process overhead per compiled run is µs to tens of µs | Add the same harness to `InProcessScriptRunner` in P0 for a baseline | + +--- + +## 16. Prior art consulted + +- **ScadaBridge `CentralUI.SandboxWorker`** (`~/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.CentralUI/ScriptAnalysis/SandboxWorkerProtocol.cs`). + It runs one disposable process per *test run* over stdio, using 4-byte LE length-prefixed JSON frames with a + 1 MiB cap, admission 4, `DOTNET_GCHeapHardLimit` 512 MiB, and `Kill(entireProcessTree)` on the deadline. + Taken from it: the heap limit, the tree kill and the frame cap. Not taken: + - stdio (fragile for a long-lived worker); + - per-run process spawn (too slow for runtime evaluation); + - inheriting the parent environment; + - proxying host calls. + + ScadaBridge's *runtime* scripts are still in-process (dedicated scheduler plus watchdog, + `docs/plans/2026-08-15-script-pool-split-design.md`), so this design would be the family's first + out-of-process **runtime** evaluator. +- **mxaccessgw gateway↔worker** (`~/Desktop/MxAccessGateway/docs/WorkerFrameProtocol.md`, + `WorkerProcessLauncher.md`, `WorkerBootstrap.md`). It runs a long-lived supervised worker over a named pipe: + protobuf envelopes, a negotiated frame cap, a priority-drained writer with coalesced flushes, sequence + numbers, a nonce in an env var (never on the command line), startup probe plus retry, and a tree kill on a + failed start. Taken from it: the frame protocol shape, the nonce handshake and the launcher discipline. + Avoided: an orphan terminator that is not scoped to its own instance.