lmxopcua/docs/plans/2026-11-09-out-of-process-script-runner-design.md

Out-of-process, pre-emptible script runner — design

Date: 2026-11-09 · Status: proposed (design only — no code in this change) · Issue: #750 · Related: #734 (Lane E remediation)

This document designs a script runner that can actually stop a user script, instead of abandoning it. It covers the three script seams that run operator-authored C# on a driver node: virtual tags, Calculation tags and scripted-alarm predicates.

Claims about today's code carry file:line evidence against origin/master @ 94926eb0. Anything not read from the code is marked [A-n] and listed in §15.

---

1. Today: in-process execution

1.1 One compile pipeline, three seams

All three seams share Core.Scripting:

• Compile. ScriptEvaluator<TContext,TResult>.Compile wraps the user source in a synthesized CompiledScript.Run(ScriptGlobals<TContext>), compiles it with Roslyn against the ScriptSandbox reference set, runs ForbiddenTypeAnalyzer, emits PE bytes to a MemoryStream and loads them into a collectible ALC (src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ScriptEvaluator.cs:125 emit, :139 LoadFromStream).

• Cache. CompiledScriptCache is keyed by SHA-256 of the source (CompiledScriptCache.cs:157). It is in memory only, so every process start recompiles every script on first use.

• Execution. It is synchronous: ScriptEvaluator.RunAsync invokes the bound delegate inline and returns Task.FromResult (ScriptEvaluator.cs:178-195). The script-visible ctx.CancellationToken is an AsyncLocal set for the duration of the call (ScriptEvaluator.cs:191, Core.Scripting.Abstractions/ScriptContext.cs:96-98).

• The context is a pure read cache. VirtualTagContext.GetTag and AlarmPredicateContext.GetTag serve only from a dictionary handed in by the caller (VirtualTagContext.cs:50, AlarmPredicateContext.cs:40). SetVirtualTag is dropped in both production single-tag evaluators (Host/Engines/RoslynVirtualTagEvaluator.cs:129-131, Driver.Calculation/CalculationEvaluator.cs:185-188) and throws in alarm predicates (AlarmPredicateContext.cs:50). Inputs are known statically, because DependencyExtractor rejects non-literal GetTag paths at publish.

  This matters most for the design (§4.1). A script needs nothing from the host while it runs: the inputs go in, one value comes out, and log lines are a side channel.

Seam	Where it runs	Budget	Serialisation	Evidence
Virtual tag	VirtualTagActor message handler calls IVirtualTagEvaluator.Evaluate synchronously on the actor's dispatcher thread; RoslynVirtualTagEvaluator blocks with .GetAwaiter().GetResult()	2 s	per actor (one actor per VT)	Runtime/VirtualTags/VirtualTagActor.cs:276; RoslynVirtualTagEvaluator.cs:70, :174-175; singleton Host/Program.cs:393-398
Calculation tag	async pump in CalculationDriver, one evaluation at a time per driver instance	2 s (runTimeoutMs)	_evalGate = SemaphoreSlim(1,1) per driver	CalculationDriver.cs:99, :523-556, :628; CalculationDriverOptions.cs:13; CalculationEvaluator.cs:232-233
Scripted alarm	ScriptedAlarmEngine.ReevaluateAsync walks the dirty set serially under one gate	250 ms (the engine is built without scriptTimeout, so it takes TimedScriptEvaluator.DefaultTimeout)	_evalGate per engine	ScriptedAlarmEngine.cs:204, :257, :743-765, :833-835; Runtime/Drivers/DriverHostActor.cs:1251-1252

Both value seams skip Roslyn for the mirror shape return ctx.GetTag("X").Value; (PassthroughScript.TryMatch, RoslynVirtualTagEvaluator.cs:111-118, CalculationEvaluator.cs:167-176).

1.2 Timeout and cancellation model

Every compiled run in production goes through ScriptRunGuard.RunAsync (Core.Scripting/ScriptRunGuard.cs:170-304). There are three guard instances per seam owner: one per RoslynVirtualTagEvaluator (RoslynVirtualTagEvaluator.cs:71), one per CalculationEvaluator (CalculationEvaluator.cs:88) and one per ScriptedAlarmEngine (ScriptedAlarmEngine.cs:258).

1. Single flight per key. A CAS on state.InFlight refuses a second run while the previous one, an orphan included, still holds the slot. The caller gets ScriptRunRefusedException → Bad "previous evaluation still running" (ScriptRunGuard.cs:188-219).

2. The run is pushed to the pool (Task.Run, :231-249), so a CPU-bound script cannot run inline and complete before the timeout is registered. The budget clock starts only when the pool picks the run up (:268).

3. The wait is runTask.WaitAsync(timeout, ct) (:273). On timeout the guard cancels the run token (:279). Only a script that polls ctx.CancellationToken can see that.

4. Quarantine happens on any of three triggers:

   • N consecutive timeouts (:332-341, default 3);
   • a refusal behind an orphan older than StuckOrphanQuarantineFactor × the timeout (:199-216, default 10×);
   • an allocation breach, where GC.GetAllocatedBytesForCurrentThread is sampled before and after the delegate (:235, :241, :262-263, :296-302; default 64 MB, ScriptingOptions.cs).

   A quarantine lifts on a source change or a redeploy (ResetAll, called from ClearCompiledScripts at RoslynVirtualTagEvaluator.cs:212-218 and from ScriptedAlarmEngine.LoadAsync).

5. What the operator sees.

   • A VT or Calc failure publishes BadInternalError with the last-known value, once per Good→Bad transition (VirtualTagActor.cs:384-392, CalculationDriver.cs:640-647).
   • An alarm whose predicate did not complete keeps its frozen condition state, annotated BadInternalError (ScriptedAlarmEngine.cs:895-907).

TimedScriptEvaluator (TimedScriptEvaluator.cs:80-83) is the older wrapper. Its own remarks say it is now used only by single-shot callers (:18-27).

1.3 Why a runaway script cannot be pre-empted

.NET has no supported way to stop a managed thread. Thread.Abort is gone in .NET Core, and the guard's WaitAsync returns control to the caller; it does not stop the run. Concretely:

Failure	What happens today	Evidence
while(true){} (ignores the token)	The pool thread spins forever, holding one core and its collectible ALC. The guard bounds this to one orphan per key and then quarantines, but the thread keeps spinning until the process restarts. N such scripts peg N cores.	ScriptRunGuard.cs:17-22 remarks; ScriptRunGuardTests.cs:170-205 asserts exactly one orphan stays in flight (InFlightCount.ShouldBe(1))
Fast allocation (new byte[int.MaxValue] in a loop, or a growing list)	The sample is taken after the delegate returns, so the process can OOM first. The whole server dies, including OPC UA sessions and every driver.	ScriptRunGuard.cs:39-43; docs/VirtualTags.md:40; review docs/reviews/2026-10-21/05-core.md:100-105
Unbounded recursion (a local function or a self-referencing Func<>; neither is a class member, so EnforceSingleRunMember does not see it)	StackOverflowException cannot be caught in .NET and terminates the host process. [A-1] I did not reproduce this; it follows from runtime semantics, and no analyzer rule targets recursion (git grep -i "recursion|StackOverflow" in Core.Scripting finds nothing). Update (lmxopcua#779): reproduced out of process (exit 134, "Stack overflow.") and fixed in-process: the compile injects ScriptStackGuard.Check() into every script body, so the script now fails with a catchable ScriptStackExhaustedException.	ScriptEvaluator.cs:228-281 (member check only)
Collectible-ALC reclaim of a stuck script	ClearCompiledScripts disposes the evaluator, but an ALC cannot be collected while a thread is executing its code, so a stuck orphan pins its assembly until restart. [A-2] This is .NET ALC semantics, not observed here.	CompiledScriptCache.Clear; ScriptEvaluator.Dispose
A slow (not runaway) script	VT: blocks a default-dispatcher thread for up to 2 s per evaluation. Calc / alarm: holds the driver-wide or engine-wide gate, so every sibling waits behind it (up to 2 s / 250 ms each).	RoslynVirtualTagEvaluator.cs:174-175; CalculationDriver.cs:539-541; ScriptedAlarmEngine.cs:743-765

The sandbox (ForbiddenTypeAnalyzer.cs:62-80: System.IO, System.Net, System.Diagnostics, System.Reflection, System.Threading[.Tasks], Environment, GC, …) blocks capabilities, and it does that well. It cannot block consumption (CPU, memory, stack). Every review that looked at this reached the same conclusion:

• docs/VirtualTags.md:40,54;
• docs/ScriptEditor.md §"Runtime safety";
• docs/superpowers/plans/2026-10-22-review-remediation.md Lane E2;
• docs/reviews/2026-10-21/05-core.md:103 and 03-host-controlplane-security-config.md:38.

All of them say the fix is an out-of-process runner.

Script authoring is gated by the ConfigEditor policy, which admits Administrator or Designer (CLAUDE.md §Scripting). The issue says "admin-gated", but the gate also admits Designers. That is the primary control, and it explains why this work is p3.

---

2. Goals and non-goals

Goals

• G1. Pre-emption. A script that exceeds its budget is stopped within a bounded time. The thread, the memory and the ALC are all reclaimed.
• G2. Containment. No script (OOM, stack overflow, FailFast-equivalent, native crash) can take down the OtOpcUa host process, its OPC UA sessions, its drivers or its Akka membership.
• G3. Attribution. The script that caused a kill is identified and quarantined. Healthy scripts that shared the worker are not.
• G4. Parity. Compile-time behaviour is unchanged: the same sandbox, analyzer, diagnostics and Monaco parity. Runtime semantics are unchanged too: the same ctx API, the same input-quality gating, the same Bad / last-value contract, and the same quarantine lift rules.
• G5. Bounded overhead (§8). It must fit the tag counts the repo already tests at.
• G6. Flag-gated, reversible rollout. In-process stays the default until the gates pass.

Non-goals

• N1. A new sandbox. ForbiddenTypeAnalyzer stays the capability control; process isolation is defence-in-depth, not a replacement.
• N2. Moving the authoring surface. The AdminUI only compiles and analyzes (ScriptAnalysisService), and it never executes a script, so central/admin nodes are out of scope.
• N3. Multi-tag cascades (ctx.SetVirtualTag taking effect). They are dropped today and stay dropped.
• N4. Cross-node or remote execution. The worker is always local to the driver node that owns the tags.
• N5. Per-run memory pre-emption. Even out of process, the limit is per worker. A per-run limit stays a sampled observation (§5.4).
• N6. Running non-.NET or untrusted third-party code.

---

3. Process model

3.1 Options

Option	Blast radius of a kill	Process count	Verdict
A. One worker per node	Every script on the node blips	1	Simple, but one bad script restarts every script on the node
B. One worker per script	One script	= script count (hundreds to thousands)	Infeasible: ~tens of MB per .NET process [A-3]
C. One worker per seam or driver instance	One seam / driver	3 to ~10	Arbitrary; a hot Calc driver still shares fate with its siblings
D. Small sharded pool + an isolation worker	One shard (1/K of the scripts), then one script	K + 0..1 (default 2 + 1)	Recommended

3.2 Recommended: sharded pool with an isolation worker

• K shard workers per driver node. Scripting:Worker:Shards defaults to 2 and is clamped to 1–8. A script is placed by hash(scriptKey) mod K. The script key is the guard key used today: the VT id, the calc RawPath, or the alarm id. Placement is deterministic, so a script's compiled PE and warm state stay on one worker across restarts.
• One isolation worker ("penalty box") is spawned on demand. Scripts under suspicion (§5.3) run there, one evaluation at a time. A kill in the isolation worker is therefore unambiguous.
• Inside a worker: a bounded evaluation scheduler with P dedicated threads, where P = max(2, ProcessorCount / K) [A-4]. Single flight per key is kept, as the guard does today. A queue that is full answers Busy, which the host treats as a refusal, not a strike.
• The passthrough fast path stays in the host. It never runs user code, and it is probably the largest share of evaluations [A-5].

3.3 Lifecycle and supervision under Akka

The design splits a control plane (actors) from a data plane (a lock-free client). An Akka Ask per evaluation would add mailbox hops to the hottest path for no benefit.

DriverHostActor (driver role)
 └─ ScriptRunnerActor            node-wide; owns the pool policy, attribution ledger, quarantine table
     ├─ ScriptWorkerActor #0     owns one OS process + pipe; state machine below
     ├─ ScriptWorkerActor #1
     └─ ScriptWorkerActor iso    spawned on first suspect, stopped after IsolationIdle (default 10 min)

IScriptRunner (DI singleton)  ── data plane ──►  ScriptWorkerConnection (per worker; pipe writer + correlation map)
   used by RoslynVirtualTagEvaluator, CalculationEvaluator, ScriptedAlarmEngine

• Ownership. ScriptRunnerActor is created by DriverHostActor, next to VirtualTagHostActor and the scripted-alarm host, so it exists only on driver-role nodes and dies with the driver tree. Its supervisor strategy is OneForOne, and a restart of a ScriptWorkerActor kills and respawns its process.

• Process exit is a message, not an exception. A watcher on Process.Exited / WaitForExitAsync sends WorkerExited(exitCode, signal, inFlight[]) to its ScriptWorkerActor, so worker crashes never escalate through Akka supervision.

• Worker states:

  Spawning → Handshaking → Loading → Ready ⇄ Draining → Dead → Backoff → Spawning

  Ready is the only state that accepts evaluations. In the other states the connection queues per key, with latest-wins coalescing (§9).

• The host process must take its workers down with it.

  • Windows: every worker is assigned to a Job Object with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE. If the host dies, the kernel closes the job handle and kills the workers.
  • Linux (the docker-dev rig): the worker treats pipe EOF as fatal, and also polls its parent PID. The host is not trusted to clean up after itself.

• CoordinatedShutdown. In the before-actor-system-terminate phase, the host sends Shutdown, waits 1 s, then kills.

• Orphan sweep at start. The host kills only workers whose launch nonce file sits under this host's instance directory, never "every process with this name". mxaccessgw's OrphanWorkerTerminator killed a live gateway's workers when a second gateway started (scadaproj memory "wonder upgrade program"); this design avoids that by construction.

• Redeploy. ClearCompiledScripts / LoadAsync become IScriptRunner.ApplyGeneration(scriptSet). That call sends LoadScripts / Unload deltas to each shard and clears the host-side quarantine rows for edited or removed keys, which keeps today's lift rule. No worker restart is needed. Collectible ALCs in the worker are unloaded as they are today, and a stuck one no longer matters because its worker gets killed.

• Redundancy. No change. Each node runs its own scripts today and will keep doing so; workers are not shared across the pair.

---

4. IPC

4.1 Shape of the conversation

§1.1 showed that a run needs no host callbacks. Each evaluation is therefore one request frame and one reply frame, with no re-entrant RPC. ScadaBridge's SandboxWorkerProtocol has to proxy call/reply frames for external systems and databases; this design does not. Log lines from ctx.Logger are the only other worker→host traffic, and they are fire-and-forget.

4.2 Transport: named pipe vs gRPC

	Named pipe + length-prefixed frames	gRPC (h2c over UDS / named pipe)
Per-call overhead	Two syscalls plus serialize; no HTTP/2 framing [A-6]	HTTP/2 stream setup, HPACK and a Kestrel pipeline per call
Worker footprint	Plain console app	Needs Kestrel in the worker, or a client stack in it and a server in the host
Batching / priority	Free to design: control frames ahead of events, as in mxaccessgw WorkerFrameProtocol	Per-RPC; batching means streaming RPCs, which is a frame protocol again
Cross-platform	NamedPipeServerStream maps to a Unix domain socket on Linux; PipeOptions.CurrentUserOnly on both	Yes
Prior art in the family	mxaccessgw gateway↔worker (protobuf WorkerEnvelope, 4-byte LE length, priority-drained writer); ScadaBridge sandbox (JSON frames over stdio, 1 MiB cap)	OtOpcUa mesh transport (Commons/Protos/mesh_transport.proto) — but that is cross-node

Recommendation: a named pipe with 4-byte little-endian length-prefixed protobuf envelopes. It copies mxaccessgw's WorkerFrameProtocol discipline:

• a frame-size cap negotiated at the handshake (default 4 MiB);
• zero-length and oversize frames rejected;
• a monotonically increasing seq;
• control frames (heartbeat, shutdown, fault) drained ahead of evaluation traffic;
• coalesced flushes.

Google.Protobuf and Grpc.Tools are already central pins, and Commons.csproj:28-30 already compiles protos, so no new package is needed. MessagePack is pinned in Directory.Packages.props but referenced by no project, so it would be a new dependency in practice. Stdio, as ScadaBridge uses, is rejected: a script cannot write to the console (it is forbidden), but the runtime and the logger can, and one stray byte on stdout corrupts the stream.

4.3 Message schema (sketch)

syntax = "proto3";
package otopcua.scriptworker.v1;

message Envelope {
  uint32 protocol_version = 1;
  uint64 seq = 2;
  oneof body {
    Hello hello = 10;            // host→worker: nonce, max_frame_bytes, options
    Ready ready = 11;            // worker→host: pid, runtime version, sandbox fingerprint
    LoadScripts load = 12;       // host→worker: add/replace compiled-or-source scripts
    LoadResult load_result = 13; // worker→host: per-script compile outcome + diagnostics
    Unload unload = 14;          // host→worker: drop keys (ALC unload)
    EvalBatch eval = 15;         // host→worker
    EvalResults results = 16;    // worker→host (one or more completions)
    LogBatch logs = 17;          // worker→host, fire-and-forget, bounded
    Heartbeat heartbeat = 18;    // both ways; carries worker gauges (inflight, heap, gen2 count)
    Shutdown shutdown = 19;
    Fault fault = 20;
  }
}

enum ContextKind { CONTEXT_KIND_UNSPECIFIED = 0; VIRTUAL_TAG = 1; CALCULATION = 2; ALARM_PREDICATE = 3; }

message ScriptSource { string key = 1; bytes source_sha256 = 2; ContextKind kind = 3;
                       string source = 4; bytes pe_image = 5; repeated string input_paths = 6; }
message LoadScripts  { uint64 generation = 1; repeated ScriptSource scripts = 2; }

message EvalBatch    { repeated EvalRequest items = 1; }
message EvalRequest  {
  uint64 eval_id = 1; string key = 2; bytes source_sha256 = 3;
  repeated Input inputs = 4;      // positional against ScriptSource.input_paths; see 4.4
  int64 now_utc_ticks = 5;        // ctx.Now — host clock, so tests that pin the clock still work
  uint32 soft_timeout_ms = 6;
}
message Input { uint32 slot = 1; Variant value = 2; uint32 status_code = 3;
                int64 source_ts_ticks = 4; /* 0 = null */ int64 server_ts_ticks = 5; }

message EvalResult {
  uint64 eval_id = 1;
  oneof outcome { Variant value = 2; bool predicate = 3; ScriptError error = 4; }
  uint64 allocated_bytes = 5;     // same GC.GetAllocatedBytesForCurrentThread sample as today
  uint32 run_micros = 6;
}
message EvalResults { repeated EvalResult items = 1; }
message ScriptError { Kind kind = 1; string message = 2;
  enum Kind { THREW = 0; COMPILE = 1; SANDBOX = 2; SOFT_TIMEOUT = 3; UNSUPPORTED_TYPE = 4; BUSY = 5; } }

message Variant {
  TypeTag type = 1;               // exact CLR type — see 4.4
  oneof v { bool b = 2; sint64 i = 3; uint64 u = 4; double d = 5; float f = 6;
            string s = 7; int64 dt_ticks = 8; ArrayValue array = 9; }
  enum TypeTag { NULL = 0; BOOLEAN = 1; SBYTE = 2; BYTE = 3; INT16 = 4; UINT16 = 5; INT32 = 6; UINT32 = 7;
                 INT64 = 8; UINT64 = 9; FLOAT = 10; DOUBLE = 11; STRING = 12; DATETIME_UTC = 13;
                 ARRAY = 14; }
}
message ArrayValue { Variant.TypeTag element_type = 1; repeated Variant items = 2; }

message LogBatch { repeated LogLine lines = 1; uint32 dropped = 2; }
message LogLine  { string key = 1; uint32 level = 2; string message_template = 3;
                   string rendered = 4; string exception = 5; int64 ts_ticks = 6; }

4.4 How values and quality cross the boundary

• Quality crosses verbatim. DataValueSnapshot is (object? Value, uint StatusCode, DateTime? SourceTimestampUtc, DateTime ServerTimestampUtc) (Core.Abstractions/DataValueSnapshot.cs:13), and it maps 1:1 onto Input. The worker rebuilds the same snapshots, so ctx.GetTag(p).StatusCode and .SourceTimestampUtc behave as they do today. The input-quality gating does not move: VT holds evaluation while an input is Bad (VirtualTagActor.cs:264-269), Calc holds on Bad, and alarms short-circuit on AreInputsReady. A request is sent only when today's code would have run the script.
• The exact boxed CLR type is load-bearing. A script that writes (short)ctx.GetTag(x).Value throws InvalidCastException if the value arrives boxed as an int. Variant.type therefore carries the CLR type, not just the OPC UA type: DriverDataType has Int16 / UInt16, which have to survive the round trip. [A-7] I have not inventoried which CLR types each driver actually boxes, for example whether a Modbus Int16 arrives as short or as int. The shadow mode in §12 exists partly to find out.
• Unsupported types. An input whose CLR type has no tag is not silently converted. The request carries no value for that slot, and the worker answers UNSUPPORTED_TYPE; the host publishes Bad with the type name. On the result side, a script that returns an unsupported type (a List<T>, an anonymous object) also gets UNSUPPORTED_TYPE. That is a behaviour change: today the raw object goes to OpcUaPublishActor.AttributeValueUpdate (VirtualTagHostActor.cs:233), and what the sink does with it was not traced [A-8]. Shadow mode reports every such case before the flag flips.
• DateTime. Values are sent as UTC ticks. A Kind other than Utc is converted at the host edge and counted on a metric. [A-9] I assume drivers already produce UTC.
• Positional inputs. LoadScripts registers the script's ordered input_paths: the DependencyExtractor reads, after the {{equip}} resolution done at deploy. Requests then send values by slot, not by path string, which takes the dominant payload cost out of the hot path. A key with no registered slots, a fallback that should never happen, sends the paths inline.

4.5 Batching

• Opportunistic, not timed. The connection's writer drains every request queued at that moment into one EvalBatch and flushes once, as mxaccessgw's drain pass does. An idle system pays no added latency; a burst is amortised automatically. There is no coalescing timer, because it would add latency to every evaluation.
• Results are streamed, not batch-atomic. The worker sends EvalResults as items complete, coalescing whatever finished in the same scheduler pass, so a slow script never holds a fast sibling's reply.
• The seams must submit concurrently, or batching never happens. Today both Calc and alarms await one script at a time (§1.1), and with IPC that serial loop costs N × round-trip. Two seam changes are required:
  • Scripted alarms: ReevaluateAsync submits the whole dirty set at once, then applies results in the original order under _evalGate. Predicates are pure functions of a snapshot taken under the gate, so the state machine sees the same sequence.
  • Calculation: the pump dequeues everything pending, with each entry's dependency snapshot already captured at enqueue, submits it as one batch and publishes in dequeue order.
• Virtual tags become asynchronous. IVirtualTagEvaluator gains EvaluateAsync, and VirtualTagActor PipeTos the result to itself. While a run is in flight, further dependency changes update the map and set a "re-run" flag; when the result lands, the actor evaluates once more against the latest map (latest-wins). This also ends today's blocking of a default-dispatcher thread for up to 2 s (RoslynVirtualTagEvaluator.cs:174-175).

---

5. Pre-emption

5.1 Two deadlines

	Soft deadline T	Hard deadline T + G
Value	Today's per-seam budget, unchanged: VT 2 s, Calc runTimeoutMs (2 s), alarm 250 ms	G = Scripting:Worker:HardGrace, default max(1 s, T), clamped to 250 ms–30 s
Who acts	The host connection's timer	The host ScriptWorkerActor (authoritative) and a worker-side watchdog thread
Effect	The caller gets the timeout failure it gets today (Bad, same reason prefix ScriptEvalFailureReasons.TimedOutPrefix) and the run token is cancelled, so a polling script still ends itself as it does today	The worker process is killed (Process.Kill(entireProcessTree: true)). The worker watchdog calls Environment.FailFast first if it is still scheduling, so the dump and exit code name the key.
Strike	Yes (as today)	Yes, plus a kill strike (§5.3)

The grace period G keeps a script that overran by a few milliseconds from costing a whole shard restart, and the cooperative token from #734 still gets its chance. Making G = 0 (kill at T) is an open decision (§14).

Single flight between T and T+G. The key stays in flight, and a trigger is refused as today, which is not a strike. After the kill there is no orphan, so the stuck-orphan rule (StuckOrphanQuarantineFactor) becomes moot out of process; it stays for the in-process mode.

5.2 Resource limits on the worker

The limits combine prior art from ScadaBridge (heap limit) and mxaccessgw (process-tree kill):

• Heap: DOTNET_GCHeapHardLimit = Scripting:Worker:HeapLimitBytes, default 512 MiB [A-10]. Breaching it makes the worker throw OutOfMemoryException or fail fast. Either way the worker dies or reports, and the host survives.
• Windows Job Object (per worker):
  • JOB_OBJECT_LIMIT_PROCESS_MEMORY, set slightly above the heap limit as a backstop for native and JIT memory;
  • JOB_OBJECT_LIMIT_ACTIVE_PROCESS = 1, so no child processes are possible;
  • KILL_ON_JOB_CLOSE;
  • UI restrictions.
• Linux / containers: the heap limit only. A container's cgroup is the rig's concern. [A-11] I assume the docker-dev containers run without CAP_SYS_ADMIN, so a nested cgroup is not available.
• Liveness: a heartbeat every 1 s. Three missed heartbeats while not in a GC pause, which the heartbeat reports, trigger a kill with reason heartbeat. This catches a wedged worker that neither replies nor exits.

5.3 Attribution, restart, backoff and quarantine

A kill or crash takes down every in-flight evaluation on that shard. Attribution decides which one to blame.

1. Hard-deadline kill: the key that crossed T+G is known exactly. It takes a kill strike. The other in-flight keys on that shard are victims: their runs fail with "script worker restarted", which is not a strike.
2. Crash with no deadline crossed (OOM, stack overflow, native fault): attribution is ambiguous. Every key in flight at exit becomes a suspect, and suspects move to the isolation worker.
3. Isolation worker: evaluations run one at a time, so a crash there attributes to exactly one key, which takes a kill strike. A suspect that completes IsolationProbation (default 20 successful runs or 10 min) without incident returns to its shard.
4. Quarantine: a key is quarantined after Scripting:Worker:QuarantineAfterKills kill strikes (default 2) within 1 h, or by today's rules (N consecutive soft timeouts, allocation breach). The quarantine table lives in the host (ScriptRunnerActor), so it survives worker restarts. It is lifted by a source change or a redeploy, exactly as today, and quarantine reasons keep the "script quarantined: " prefix so every existing consumer and test classifier still matches.
5. Restart backoff per worker: exponential, 250 ms → 30 s cap, ×2, with ±20% jitter. The backoff resets after 60 s of Ready. A worker that dies during Loading (a poison PE or a runtime fault) does not loop: after 3 consecutive load-phase deaths, the scripts in its generation are loaded one by one in the isolation worker to find the culprit.
6. Crash-loop circuit: if all shards are in backoff for longer than Scripting:Worker:CircuitOpenAfter (default 2 min), the runner reports Unavailable (§10) and logs one Error. It keeps retrying at the cap.

---

6. Security

Process isolation contains failures. Most of the capability control stays with the compile-time sandbox, which is retained. Defence-in-depth adds the following.

• The compile-time sandbox is unchanged. It still includes ScriptSandbox (with the TPA reference set filtered to System.*, ScriptSandbox.cs:118-122), ForbiddenTypeAnalyzer, and the injection guard. The worker runs exactly the same ScriptEvaluator.Compile, so editor/publish/runtime parity holds.

• Assembly allow-list by construction. The worker is its own app (ZB.MOM.WW.OtOpcUa.ScriptWorker), with its own deps.json and only these references:

  • Core.Scripting;
  • Core.Scripting.Abstractions;
  • Core.Abstractions;
  • Serilog (core only, for ILogger on ctx);
  • Google.Protobuf.

  Its trusted-platform list therefore contains the shared framework plus those assemblies. Akka, SqlClient, the OPC UA stack, driver assemblies and the Secrets library are not loadable in the worker, even through reflection. At startup the worker asserts its loaded-assembly set against a pinned list and exits if the list differs.

• No secrets in the environment. ProcessStartInfo.Environment is cleared and then repopulated from an explicit allow-list: DOTNET_* runtime knobs, TEMP, PATH minimal, and the launch nonce. Host env vars like ServerHistorian__ApiKey (CLAUDE.md "Never commit secrets") and the ConfigDb connection string must never reach the worker. ScadaBridge's sandbox inherits the parent environment; do not copy that part.

• No configuration access. The worker gets no appsettings path. Every option it needs (budgets, heap, frame cap) arrives in Hello.

• Pipe hardening (mxaccessgw pattern):

  • a random pipe name;
  • PipeOptions.CurrentUserOnly, or an explicit ACL when the worker runs as a different account;
  • a one-time nonce passed in an env var (never on the command line; mxaccessgw WorkerBootstrap.md explains why), which the worker must echo in Ready;
  • the host accepts exactly one connection per launch.

• Filesystem:

  • the working directory is an empty per-worker temp directory;
  • the PE cache (§7) is opened read-only by the worker, if the host compiles, or written only under its own subdirectory;
  • on Windows, a dedicated low-privilege account (see §14) gets read access only to the install folder and its own temp directory.

• Network: the analyzer already forbids System.Net. Process-level options are a Windows Firewall outbound-block rule scoped to the worker executable, created by the installer, and on Linux nothing beyond the container. This is an open decision (§14). The recommendation is the installer firewall rule on Windows, and to accept the analyzer as the control on Linux.

• What the host trusts from the worker: nothing it cannot validate. Frames are size-capped and schema-checked, an eval_id must match an outstanding request, and log lines are length-capped and rate-limited (§11). A compromised worker can at worst return wrong values for scripts on its own shard, which the same script could have done anyway.

---

7. Compile caching across restarts

The problem: today every process start recompiles every script lazily (CompiledScriptCache is memory-only). Out of process, a shard restart has the same cost, and it can happen in the middle of production running. mem-probe measured the old CSharpScript path at roughly 2.4 MiB of working set per retained script (tools/mem-probe/MemProbe/Program.cs:1-4). Compile time is not measured anywhere in the repo [A-12]; I assume tens of ms per script warm and seconds for the first compile while Roslyn JITs. For 1,000 scripts, that is far outside a 2 s restart budget.

Recommendation: a content-addressed PE cache on node-local disk.

• Key: SHA-256 over these fields:

  • the source;
  • the ContextKind, i.e. TContext and TResult;
  • the Roslyn compiler version;
  • a sandbox fingerprint: a hash of ForbiddenTypeAnalyzer's lists, the ScriptSandbox reference set and the wrapper template.

  An analyzer change therefore invalidates every entry, and a cached PE can never skip a newer sandbox rule.

• Value: the emitted PE bytes plus an HMAC-SHA256 tag. The HMAC key is a per-node random key created at first start and stored next to the cache, readable only by the host account. The worker loads a PE only if the HMAC verifies, because loading attacker-supplied PE bytes would be code execution outside the analyzer.

• Location: Scripting:Worker:PeCacheDir. It must be absolute and outside the content root, the same fail-closed rule the family applies to Secrets:SqlitePath (scadaproj CLAUDE.md, Secrets row). Entries whose key is not in the current generation are pruned at ApplyGeneration.

• Where compilation happens: in the worker (recommended). Roslyn's memory and CPU leave the host. A pathological source that makes Roslyn itself blow up kills a worker, not the server, and the worker writes the PE plus HMAC (it holds the key through Hello). The alternative is to compile in the host and ship pe_image in LoadScripts. That keeps the worker free of Roslyn (smaller, faster start), but compile-time blowups stay in the host. This is an open decision (§14).

• Restart path: Spawning → Ready takes about 150 ms of runtime start [A-13]. Then LoadScripts sends hash + key + slots only. The worker loads PEs from the cache lazily, on each key's first evaluation, and a cache miss compiles. Loading a PE into a collectible ALC is about 1 ms [A-13].

• Negative cache: Calculation already remembers compile failures by source hash (R07:P4, CalculationEvaluator.cs:63-68). The worker reports compile failures in LoadResult, and the host keeps that negative cache for all seams, so a broken script never re-enters Roslyn.

---

8. Performance budgets

8.1 Reference loads in the repo

Source	Load
docs/v2/Galaxy.Performance.md (EventPump sizing, 24 h soak)	50,000 tags at 1 Hz on one node; compressed CI soak at 1,000 tags
tests/Server/.../NodeManagerLockTimeBenchmarkTests.cs:41-105	10,000-tag folder; 200 equipment × 250 tags (50,000 variables)
ScriptRunGuardTests.cs:170-205	a 1 Hz runaway and a healthy sibling over 30 triggers

No repo perf test exercises scripts at scale. [A-14] I assume a planning envelope of 5,000 compiled script evaluations per second per node (for example 5,000 VT + Calc at 1 Hz, or 500 scripts at 10 Hz) and size for 2× headroom (10k/s). The passthrough fast path is excluded because it stays in the host.

8.2 Budgets (acceptance gates for the implementation)

Metric	Budget	Notes
Added latency per evaluation, p50	≤ 150 µs	Host enqueue → reply decoded, trivial script, idle system
Added latency per evaluation, p99	≤ 2 ms	At 10k evals/s. Compare the 250 ms alarm budget: under 1% of it
Throughput	≥ 10,000 trivial evals/s/node with K=2	With opportunistic batching
Host CPU for IPC	≤ 25% of one core at 10k evals/s	Serialisation + pipe I/O, measured by an opt-in perf test
Bytes per eval on the wire	≤ 64 B + 24 B per input	Positional slots, §4.4
Shard restart → first result, 1,000 cached scripts	≤ 2 s	§7 lazy PE load
Worker idle RSS	≤ 80 MB + scripts [A-3]	Per shard
Kill latency after T+G	≤ 100 ms	Host timer resolution plus Process.Kill

Comparison with today: the in-process cost per compiled run is one Task.Run hop plus WaitAsync, which is a few µs to tens of µs and was never measured [A-15]. The out-of-process path is therefore one to two orders of magnitude slower per call. That is acceptable only because the budgets above sit two to three orders of magnitude below the script budgets (250 ms / 2 s), and because batching amortises the syscalls. The risk case is a large scripted-alarm engine, with its serial gate and its 250 ms budget. That is why §4.5 makes the batch-submit change a prerequisite, not an optimisation.

---

9. Failure semantics: what quality a tag gets

The principle: a runner failure must never look like a script result. In particular, an alarm must never clear because the worker died.

Situation	Virtual tag / Calculation tag	Scripted alarm
Soft timeout (T)	BadInternalError + last value, once per transition, as today	Condition frozen + BadInternalError quality annotation, as today
This key killed at T+G, or crashed in isolation	Same as soft timeout; reason "script killed: exceeded hard deadline …"	Same as soft timeout
Victim: in flight on a shard that died for another key	BadInternalError + last value, reason "script worker restarted"; re-evaluated on the shard's return with the latest inputs (latest-wins)	Frozen + Bad annotation; re-evaluated on return
Not in flight while its shard restarts	Unchanged. Keeps its last published value and quality. A trigger during the outage is coalesced (latest-wins, one per key) and evaluated when the shard is Ready	Same; the condition holds its state
Shard not back within Scripting:Worker:UnavailableGrace (default 5 s)	Coalesced keys publish Bad + last value, reason "script runner unavailable"	Frozen + Bad annotation
Circuit open (§5.3.6)	Every compiled-script key on the node is Bad as above; passthrough keys are unaffected	Every compiled predicate is frozen + Bad
Quarantined	"script quarantined: …" Bad, as today	As today
Host process dies	Workers die with it (Job Object / pipe EOF); no change from today's host-death behaviour	—

Status code: BadInternalError stays for script-attributable faults (the existing contract). For "the runner is unavailable, not this script's fault", this design proposes BadResourceUnavailable (0x80040000), so operators and the ScadaBridge consumer can tell a broken script from a sick node. The alternative is to keep BadInternalError everywhere for zero consumer impact. This is an open decision (§14); if it is taken, the ScadaBridge consumer-impact record must note it (scadaproj sister-project-history skill, otopcua.md).

No silent fallback to in-process. If the worker cannot start, the runner does not run scripts in the host. That would reintroduce exactly the risk the flag was set to remove. Operators flip the flag back (§12). This is an open decision (§14).

---

10. Observability

All metrics go on OtOpcUaTelemetry.Meter, so the existing ZbTelemetryOptions allow-list applies. scadaproj memory: "ZbTelemetryOptions.Meters is a silent allowlist"; check that the new names flow end to end.

Metric	Type	Tags
otopcua.script.worker.up	gauge	shard
otopcua.script.worker.starts / .exits	counter	shard, reason = clean | killed_deadline | killed_heartbeat | oom | crash | load_failure
otopcua.script.worker.inflight / .queue_depth	gauge	shard
otopcua.script.worker.heap_bytes	gauge (from heartbeat)	shard
otopcua.script.ipc.roundtrip	histogram (µs)	kind
otopcua.script.eval.duration	histogram (µs, worker-measured)	kind
otopcua.script.compile	counter	cache = hit | miss | negative
otopcua.script.quarantined (existing)	counter	adds reason = killed | isolated_crash
otopcua.script.run_refused (existing)	counter	unchanged
otopcua.script.isolation.suspects	gauge	—
otopcua.script.runner.state	gauge (0 ready / 1 degraded / 2 unavailable)	—

Per-key series: the existing counters tag the series with script = key. Do not add key tags to the new per-eval histograms (cardinality).

Logs:

• the worker's own stderr is captured, prefixed with ScriptWorker[shard], into the host log;
• every kill logs one Warning naming the key and the reason, and every quarantine logs one Error, as today;
• ctx.Logger lines come back in LogBatch. The host re-emits them through ScriptRootLogger with the same ScriptId / VirtualTagId properties, so scripts-*.log, the error mirror, the DPS script-log topic and its 20/min-per-source rate limit all keep working unchanged;
• the worker caps its outbound log buffer (default 1,000 lines / 1 MiB) and reports drops as LogBatch.dropped, so a script logging in a tight loop cannot flood the pipe.

Health: expose the runner state as a non-fatal component of the driver-node status (/health/ready stays green while degraded, because OPC UA still serves driver data). Whether Unavailable should fail readiness is an open decision (§14). The AdminUI status dashboard gets a "Script runner" tile with the per-shard state, restarts in the last hour and the quarantine list. Quarantines already surface through the QuarantinedScripts properties.

Traces: one Activity per EvalBatch (not per evaluation), with batch.size and shard.

---

11. Seam changes in the host (summary)

Component	Change
Core.Scripting	New IScriptRunner (EvaluateAsync(ScriptInvocation[]), ApplyGeneration, QuarantinedKeys, ResetAll). InProcessScriptRunner wraps today's ScriptRunGuard + CompiledScriptCache exactly.
New Core.Scripting.Protocol	The proto file, the frame codec and the Variant ↔ CLR mapping. Shared by host and worker.
New ZB.MOM.WW.OtOpcUa.ScriptWorker	Console app: handshake, scheduler, watchdog, PE cache, log forwarding. Shipped in the Host's publish output.
New Runtime/Scripting/	ScriptRunnerActor, ScriptWorkerActor, OutOfProcessScriptRunner (data plane), Job Object interop (Windows-only, behind OperatingSystem.IsWindows()).
IVirtualTagEvaluator / VirtualTagActor	Add EvaluateAsync; the actor PipeTos with latest-wins re-run (§4.5).
RoslynVirtualTagEvaluator, CalculationEvaluator, ScriptedAlarmEngine	Replace direct ScriptRunGuard use with IScriptRunner. Calc pump and alarm ReevaluateAsync submit batches (§4.5).
ScriptingOptions / ScriptingOptionsBinding	New keys (§12), clamped with startup Warnings like the existing ones.

---

12. Migration and rollout

Configuration (node-local appsettings, Scripting section, consistent with the existing knobs):

Key	Default	Meaning
Scripting:Runner	InProcess	InProcess | OutOfProcess | Shadow
Scripting:Runner:Seams	all three	Optional per-seam subset for OutOfProcess / Shadow (e.g. ["Calculation"])
Scripting:Worker:Shards	2	1–8
Scripting:Worker:HardGrace	max(1s, T)	§5.1
Scripting:Worker:HeapLimitBytes	536870912	§5.2
Scripting:Worker:QuarantineAfterKills	2	§5.3
Scripting:Worker:UnavailableGrace	5s	§9
Scripting:Worker:PeCacheDir	none (required when not InProcess)	§7; absolute, outside the content root, fail-closed

Phases (each one is its own issue and PR):

1. P0, in-process refactor with no behaviour change.
   • Introduce IScriptRunner + InProcessScriptRunner.
   • Make VT evaluation async (PipeTo), and make the alarm engine and the Calc pump batch-submit.
   • Gate: every existing Scripting / VirtualTags / Calculation / ScriptedAlarms suite stays green unchanged, plus new ordering tests for batch-then-apply.
   • This phase has value on its own: it ends the dispatcher-thread blocking.
   • Partly landed in lmxopcua#779: VT evaluation is async (IVirtualTagEvaluator.EvaluateAsync + stash in VirtualTagActor), so the dispatcher-thread blocking is gone. The Calc pump and the alarm engine still run one script at a time; why that was not relaxed in-process is recorded in docs/VirtualTags.md §"Upstream-tag feed". IScriptRunner and batch-submit remain to do.
2. P1, worker + protocol behind the flag.
   • Ship ScriptWorker, OutOfProcessScriptRunner and the PE cache.
   • Default stays InProcess. Enable on the docker-dev rig only.
   • Gate: §13 runaway / containment tests green on Linux (docker-dev) and Windows (windev box, which has the .NET 10 toolchain).
3. P2, shadow mode on docker-dev and QA.
   • Shadow runs the in-process evaluation as the published result, and also submits to the worker.
   • It compares value, status class and failure kind per evaluation, and counts otopcua.script.shadow.mismatch{kind, reason}. Mismatching keys are logged once each.
   • Shadow doubles script CPU and still carries the in-process risk, so it is a time-boxed diagnostic (for example 1 week on QA), not a mode to run in production.
   • Exit criterion: zero unexplained mismatches. This is where [A-7] / [A-8] get answered.
4. P3, flip on QA, then the other tiers.
   • Set Scripting:Runner=OutOfProcess per node, seam subset first (Calculation, then VT, then alarms; alarms last because of the 250 ms budget).
   • Note: remote command execution on the wonder boxes is currently scoped to user-requested deploys (CLAUDE.md hold), so tier rollout rides normal deploys.
5. P4, decide the long-term status of InProcess. The recommendation is to keep it for dev and tests and as the documented rollback, and to retire StuckOrphanQuarantineFactor from the docs as "in-process only".

Rollback: set the flag back to InProcess and restart the node. No data or config migration is involved. The PE cache directory can be deleted at any time.

Docs to update when implementing:

• docs/VirtualTags.md §"Known resource limits" and §"Per-evaluation timeout";
• docs/ScriptEditor.md §"Runtime safety";
• docs/ScriptedAlarms.md;
• docs/ServiceHosting.md: the new process, Job Object and firewall rule;
• docs/security.md;
• docs/Telemetry.md;
• the scadaproj index row, if the status-code decision changes the ScadaBridge-visible contract.

---

13. Test plan

Unit (Core.Scripting.Tests, new ScriptWorker.Tests):

• Frame codec:
  • round-trip every Variant.TypeTag (arrays included);
  • reject zero-length, oversize and truncated frames;
  • reject an eval_id with no outstanding request;
  • fuzz the reader with random bytes and assert there is no exception other than a protocol fault.
• Unboxing parity: for each tag, a script doing (T)ctx.GetTag("x").Value behaves identically in and out of process. This pins [A-7].
• Attribution ledger:
  • a deadline kill blames exactly one key;
  • a crash with 3 in flight → 3 suspects → isolation;
  • an isolation crash → one strike;
  • probation returns a suspect to its shard;
  • quarantine after 2 kills;
  • a source change lifts the quarantine;
  • a redeploy lifts every quarantine.
• Backoff schedule, jitter bounds, the reset-after-healthy rule and the load-phase crash bisection.
• Options binding: clamps plus a Warning per clamped key; PeCacheDir relative or inside the content root is a startup failure.

Containment (worker-backed, Category=ScriptWorker; runs in CI on Linux, gated on windev for Windows). Each case has a positive control: a healthy sibling script on the same shard that keeps producing correct values before and after the event. Following scadaproj memory "assert the artifact before the outcome", each assertion is checked to be capable of failing.

Case	Assert
while(true){} ignoring the token	Killed within T + G + 100 ms; host ThreadPool.ThreadCount and Process.Threads unchanged; no orphan (InFlightCount == 0 after the kill). Mutation check: disable the kill and see the test fail
while(!ctx.CancellationToken.IsCancellationRequested){}	Ends at T by itself; no kill; no strike beyond the timeout
Allocation bomb	Worker dies of OOM or the heap limit; host RSS stays below a bound and the host stays up; suspect → isolation → quarantine
Unbounded recursion	Worker dies; host up; quarantined after the isolation reproduction. Also the first in-repo reproduction of [A-1] for the in-process mode, marked Skip there because it would kill the test host
Worker killed externally (kill -9)	Victims Bad with "script worker restarted", then recover with the latest inputs; the alarm condition is not cleared
Host killed	Worker exits within 2 s (pipe EOF on Linux, Job Object on Windows)
Environment	The worker's environment contains no ServerHistorian__ApiKey / ConfigDb variables (set them in the test host first: positive control)
Pipe	A second client connecting to the pipe is refused; a wrong nonce is refused
Loaded assemblies	The worker's loaded set equals the pinned list; Akka / SqlClient / Opc.Ua are absent
Crash loop	A poison generation trips the circuit; state Unavailable; keys Bad with the runner-unavailable reason; recovery after a fixed redeploy

Parity: run the existing RoslynVirtualTagEvaluatorTests, CalculationDriver*Tests and ScriptedAlarmEngine*Tests against both runners. Use xunit theory or class fixtures over IScriptRunner, with the worker built as a test dependency.

Performance (Category=Perf, opt-in, not in CI): a Stopwatch-based harness (no new packages) that measures the §8 table: p50/p99 added latency, throughput at K=1/2/4, host CPU, restart-to-first-result with 1,000 cached PEs, and RSS per shard.

Live gates:

• docker-dev rig: a deploy with VTs, Calc tags and scripted alarms; drill a runaway script through the AdminUI and watch /uns go Bad, the quarantine appear, and siblings stay Good;
• QA shadow week (P2);
• a QA flip with the same drill.

---

14. Open decisions for the owner

Each has a recommendation, marked ★.

1. Pool shape.
   • ★ K=2 shards plus an on-demand isolation worker;
   • one worker per node, simplest, but every kill blips every script;
   • per-seam workers.
2. Hard grace G.
   • ★ max(1 s, T), which keeps cooperative cancellation useful;
   • G = 0 (kill at T): strictest, but every soft overrun restarts a shard.
3. Status code for "runner unavailable / worker restarted".
   • ★ BadResourceUnavailable, which separates a sick node from a broken script but is a ScadaBridge-visible contract change;
   • keep BadInternalError everywhere, for zero consumer impact.
4. Fallback when the worker cannot start.
   • ★ fail closed: Bad, and no in-process execution;
   • fail open to in-process with an Error, which keeps values flowing but silently reopens the risk.
5. Worker identity on Windows.
   • ★ a dedicated low-privilege virtual account created by the installer, which needs installer work, an explicit pipe ACL, and a deploy that touches the box;
   • the same account as the host: zero install change, and the Job Object plus the analyzer remain the controls.
6. Network isolation.
   • ★ a Windows Firewall outbound-block rule on the worker executable (installer), with the analyzer as the only control on Linux;
   • analyzer only, everywhere.
7. Compile location.
   • ★ in the worker: Roslyn blowups are contained and the host gets lighter;
   • in the host, shipping PE bytes: a lighter worker and faster cold start.
8. PE cache store.
   • ★ a node-local directory with an HMAC;
   • LocalDb (already a dependency; it would also replicate to the pair, which is unnecessary);
   • no persistence, recompiling on restart (violates the 2 s budget at scale).
9. IPC encoding.
   • ★ protobuf (already compiled in Commons, and mxaccessgw precedent);
   • MessagePack (pinned but unused, so in practice a new dependency).
10. Scripted alarms in the first flip.
    • ★ last, after Calc and VT, because of the 250 ms budget and the engine refactor;
    • all seams at once.
11. Unsupported input/result CLR types.
    • ★ an explicit Bad UNSUPPORTED_TYPE, surfaced in shadow mode first;
    • best-effort conversion (to string, double), which hides type bugs.
12. Readiness when the circuit is open.
    • ★ degraded but ready (OPC UA still serves driver data);
    • not ready, which on a redundant pair would shift ServiceLevel to the peer.
13. Long-term status of InProcess.
    • ★ keep it as the dev/test default and the rollback;
    • delete it after a soak period.
14. Does this need a real-deployment scale target? [A-14]'s 5k evals/s/node is a guess. The owner should supply the expected VT / Calc / alarm counts per node for the target sites, or accept the guess as the gate.

---

15. Assumptions (not verified from code)

Id	Assumption	How to verify
A-1	Unbounded recursion in a script throws StackOverflowException and kills the host today	Reproduce in a throwaway process (not the test host)
A-2	A collectible ALC whose code is still executing on an orphan thread cannot be unloaded	.NET docs / a WeakReference test with a spinning orphan
A-3	A minimal .NET 10 worker idles at tens of MB RSS	Measure the P1 worker
A-4	P = max(2, cores/K) evaluation threads per worker is enough	P1 perf harness
A-5	Passthrough scripts are a large share of evaluations on real deployments	Count PassthroughScript.TryMatch hits against compiled runs on QA (add a counter in P0)
A-6	A local pipe round trip costs tens of µs	P1 perf harness
A-7	Drivers box tag values with stable, enumerable CLR primitive types	Shadow-mode type census
A-8	What OpcUaPublishActor / the sink does with an unsupported result object today	Trace AttributeValueUpdate → SdkAddressSpaceSink
A-9	Drivers produce UTC DateTime values	Shadow-mode census on DateTime.Kind
A-10	512 MiB is a sane per-worker heap limit (ScadaBridge uses the same)	Size against measured RSS × script count
A-11	docker-dev containers cannot create nested cgroups	Check the compose capabilities
A-12	Warm Roslyn compile is tens of ms per script; cold is seconds	P1 perf harness
A-13	Worker start is ~150 ms; a PE load into a collectible ALC is ~1 ms	P1 perf harness
A-14	5k compiled evals/s/node is a fair planning envelope	Owner input (decision 14)
A-15	Today's in-process overhead per compiled run is µs to tens of µs	Add the same harness to InProcessScriptRunner in P0 for a baseline

---

16. Prior art consulted

• ScadaBridge CentralUI.SandboxWorker (~/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.CentralUI/ScriptAnalysis/SandboxWorkerProtocol.cs). It runs one disposable process per test run over stdio, using 4-byte LE length-prefixed JSON frames with a 1 MiB cap, admission 4, DOTNET_GCHeapHardLimit 512 MiB, and Kill(entireProcessTree) on the deadline. Taken from it: the heap limit, the tree kill and the frame cap. Not taken:

  • stdio (fragile for a long-lived worker);
  • per-run process spawn (too slow for runtime evaluation);
  • inheriting the parent environment;
  • proxying host calls.

  ScadaBridge's runtime scripts are still in-process (dedicated scheduler plus watchdog, docs/plans/2026-08-15-script-pool-split-design.md), so this design would be the family's first out-of-process runtime evaluator.

• mxaccessgw gateway↔worker (~/Desktop/MxAccessGateway/docs/WorkerFrameProtocol.md, WorkerProcessLauncher.md, WorkerBootstrap.md). It runs a long-lived supervised worker over a named pipe: protobuf envelopes, a negotiated frame cap, a priority-drained writer with coalesced flushes, sequence numbers, a nonce in an env var (never on the command line), startup probe plus retry, and a tree kill on a failed start. Taken from it: the frame protocol shape, the nonce handshake and the launcher discipline. Avoided: an orphan terminator that is not scoped to its own instance.