lmxopcua/docs/plans/2026-11-09-out-of-process-script-runner-design.md

Out-of-process, pre-emptible script runner — design

Date: 2026-11-09 · Status: proposed (design only — no code in this change) · Issue: #750 · Related: #734 (Lane E remediation)

This document designs a script runner that can actually stop a user script, instead of abandoning it. It covers the three script seams that run operator-authored C# on a driver node: virtual tags, Calculation tags and scripted-alarm predicates.

Claims about today's code carry file:line evidence against origin/master @ 94926eb0. Anything not read from the code is marked [A-n] and listed in §15.

---

1. Today: in-process execution

1.1 One compile pipeline, three seams

All three seams share Core.Scripting:

• Compile. ScriptEvaluator<TContext,TResult>.Compile wraps the user source in a synthesized CompiledScript.Run(ScriptGlobals<TContext>), compiles it with Roslyn against the ScriptSandbox reference set, runs ForbiddenTypeAnalyzer, emits PE bytes to a MemoryStream and loads them into a collectible ALC (src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ScriptEvaluator.cs:125 emit, :139 LoadFromStream).

• Cache. CompiledScriptCache is keyed by SHA-256 of the source (CompiledScriptCache.cs:157). It is in memory only, so every process start recompiles every script on first use.

• Execution. It is synchronous: ScriptEvaluator.RunAsync invokes the bound delegate inline and returns Task.FromResult (ScriptEvaluator.cs:178-195). The script-visible ctx.CancellationToken is an AsyncLocal set for the duration of the call (ScriptEvaluator.cs:191, Core.Scripting.Abstractions/ScriptContext.cs:96-98).

• The context is a pure read cache. VirtualTagContext.GetTag and AlarmPredicateContext.GetTag serve only from a dictionary handed in by the caller (VirtualTagContext.cs:50, AlarmPredicateContext.cs:40). SetVirtualTag is dropped in both production single-tag evaluators (Host/Engines/RoslynVirtualTagEvaluator.cs:129-131, Driver.Calculation/CalculationEvaluator.cs:185-188) and throws in alarm predicates (AlarmPredicateContext.cs:50). Inputs are known statically, because DependencyExtractor rejects non-literal GetTag paths at publish.

  This matters most for the design (§4.1). A script needs nothing from the host while it runs: the inputs go in, one value comes out, and log lines are a side channel.

Seam	Where it runs	Budget	Serialisation	Evidence
Virtual tag	VirtualTagActor message handler calls IVirtualTagEvaluator.Evaluate synchronously on the actor's dispatcher thread; RoslynVirtualTagEvaluator blocks with .GetAwaiter().GetResult()	2 s	per actor (one actor per VT)	Runtime/VirtualTags/VirtualTagActor.cs:276; RoslynVirtualTagEvaluator.cs:70, :174-175; singleton Host/Program.cs:393-398
Calculation tag	async pump in CalculationDriver, one evaluation at a time per driver instance	2 s (runTimeoutMs)	_evalGate = SemaphoreSlim(1,1) per driver	CalculationDriver.cs:99, :523-556, :628; CalculationDriverOptions.cs:13; CalculationEvaluator.cs:232-233
Scripted alarm	ScriptedAlarmEngine.ReevaluateAsync walks the dirty set serially under one gate	250 ms (the engine is built without scriptTimeout, so it takes TimedScriptEvaluator.DefaultTimeout)	_evalGate per engine	ScriptedAlarmEngine.cs:204, :257, :743-765, :833-835; Runtime/Drivers/DriverHostActor.cs:1251-1252

Both value seams skip Roslyn for the mirror shape return ctx.GetTag("X").Value; (PassthroughScript.TryMatch, RoslynVirtualTagEvaluator.cs:111-118, CalculationEvaluator.cs:167-176).

1.2 Timeout and cancellation model

Every compiled run in production goes through ScriptRunGuard.RunAsync (Core.Scripting/ScriptRunGuard.cs:170-304). There are three guard instances per seam owner: one per RoslynVirtualTagEvaluator (RoslynVirtualTagEvaluator.cs:71), one per CalculationEvaluator (CalculationEvaluator.cs:88) and one per ScriptedAlarmEngine (ScriptedAlarmEngine.cs:258).

1. Single flight per key. A CAS on state.InFlight refuses a second run while the previous one, an orphan included, still holds the slot. The caller gets ScriptRunRefusedException → Bad "previous evaluation still running" (ScriptRunGuard.cs:188-219).

2. The run is pushed to the pool (Task.Run, :231-249), so a CPU-bound script cannot run inline and complete before the timeout is registered. The budget clock starts only when the pool picks the run up (:268).

3. The wait is runTask.WaitAsync(timeout, ct) (:273). On timeout the guard cancels the run token (:279). Only a script that polls ctx.CancellationToken can see that.

4. Quarantine happens on any of three triggers:

   • N consecutive timeouts (:332-341, default 3);
   • a refusal behind an orphan older than StuckOrphanQuarantineFactor × the timeout (:199-216, default 10×);
   • an allocation breach, where GC.GetAllocatedBytesForCurrentThread is sampled before and after the delegate (:235, :241, :262-263, :296-302; default 64 MB, ScriptingOptions.cs).

   A quarantine lifts on a source change or a redeploy (ResetAll, called from ClearCompiledScripts at RoslynVirtualTagEvaluator.cs:212-218 and from ScriptedAlarmEngine.LoadAsync).

5. What the operator sees.

   • A VT or Calc failure publishes BadInternalError with the last-known value, once per Good→Bad transition (VirtualTagActor.cs:384-392, CalculationDriver.cs:640-647).
   • An alarm whose predicate did not complete keeps its frozen condition state, annotated BadInternalError (ScriptedAlarmEngine.cs:895-907).

TimedScriptEvaluator (TimedScriptEvaluator.cs:80-83) is the older wrapper. Its own remarks say it is now used only by single-shot callers (:18-27).

1.3 Why a runaway script cannot be pre-empted

.NET has no supported way to stop a managed thread. Thread.Abort is gone in .NET Core, and the guard's WaitAsync returns control to the caller; it does not stop the run. Concretely:

Failure	What happens today	Evidence
while(true){} (ignores the token)	The pool thread spins forever, holding one core and its collectible ALC. The guard bounds this to one orphan per key and then quarantines, but the thread keeps spinning until the process restarts. N such scripts peg N cores.	ScriptRunGuard.cs:17-22 remarks; ScriptRunGuardTests.cs:170-205 asserts exactly one orphan stays in flight (InFlightCount.ShouldBe(1))
Fast allocation (new byte[int.MaxValue] in a loop, or a growing list)	The sample is taken after the delegate returns, so the process can OOM first. The whole server dies, including OPC UA sessions and every driver.	ScriptRunGuard.cs:39-43; docs/VirtualTags.md:40; review docs/reviews/2026-10-21/05-core.md:100-105
Unbounded recursion (a local function or a self-referencing Func<>; neither is a class member, so EnforceSingleRunMember does not see it)	StackOverflowException cannot be caught in .NET and terminates the host process. [A-1] I did not reproduce this; it follows from runtime semantics, and no analyzer rule targets recursion (git grep -i "recursion|StackOverflow" in Core.Scripting finds nothing). Update (lmxopcua#779): reproduced out of process (exit 134, "Stack overflow.") and fixed in-process: the compile injects ScriptStackGuard.Check() into every script body, so the script now fails with a catchable ScriptStackExhaustedException.	ScriptEvaluator.cs:228-281 (member check only)
Collectible-ALC reclaim of a stuck script	ClearCompiledScripts disposes the evaluator, but an ALC cannot be collected while a thread is executing its code, so a stuck orphan pins its assembly until restart. [A-2] This is .NET ALC semantics, not observed here.	CompiledScriptCache.Clear; ScriptEvaluator.Dispose
A slow (not runaway) script	VT: blocks a default-dispatcher thread for up to 2 s per evaluation. Calc / alarm: holds the driver-wide or engine-wide gate, so every sibling waits behind it (up to 2 s / 250 ms each).	RoslynVirtualTagEvaluator.cs:174-175; CalculationDriver.cs:539-541; ScriptedAlarmEngine.cs:743-765

The sandbox (ForbiddenTypeAnalyzer.cs:62-80: System.IO, System.Net, System.Diagnostics, System.Reflection, System.Threading[.Tasks], Environment, GC, …) blocks capabilities, and it does that well. It cannot block consumption (CPU, memory, stack). Every review that looked at this reached the same conclusion:

• docs/VirtualTags.md:40,54;
• docs/ScriptEditor.md §"Runtime safety";
• docs/superpowers/plans/2026-10-22-review-remediation.md Lane E2;
• docs/reviews/2026-10-21/05-core.md:103 and 03-host-controlplane-security-config.md:38.

All of them say the fix is an out-of-process runner.

Script authoring is gated by the ConfigEditor policy, which admits Administrator or Designer (CLAUDE.md §Scripting). The issue says "admin-gated", but the gate also admits Designers. That is the primary control, and it explains why this work is p3.

---

2. Goals and non-goals

Goals

• G1. Pre-emption. A script that exceeds its budget is stopped within a bounded time. The thread, the memory and the ALC are all reclaimed.
• G2. Containment. No script (OOM, stack overflow, FailFast-equivalent, native crash) can take down the OtOpcUa host process, its OPC UA sessions, its drivers or its Akka membership.
• G3. Attribution. The script that caused a kill is identified and quarantined. Healthy scripts that shared the worker are not.
• G4. Parity. Compile-time behaviour is unchanged: the same sandbox, analyzer, diagnostics and Monaco parity. Runtime semantics are unchanged too: the same ctx API, the same input-quality gating, the same Bad / last-value contract, and the same quarantine lift rules.
• G5. Bounded overhead (§8). It must fit the tag counts the repo already tests at.
• G6. Flag-gated, reversible rollout. In-process stays the default until the gates pass.

Non-goals

• N1. A new sandbox. ForbiddenTypeAnalyzer stays the capability control; process isolation is defence-in-depth, not a replacement.
• N2. Moving the authoring surface. The AdminUI only compiles and analyzes (ScriptAnalysisService), and it never executes a script, so central/admin nodes are out of scope.
• N3. Multi-tag cascades (ctx.SetVirtualTag taking effect). They are dropped today and stay dropped.
• N4. Cross-node or remote execution. The worker is always local to the driver node that owns the tags.
• N5. Per-run memory pre-emption. Even out of process, the limit is per worker. A per-run limit stays a sampled observation (§5.4).
• N6. Running non-.NET or untrusted third-party code.

---

3. Process model

3.1 Options

Option	Blast radius of a kill	Process count	Verdict
A. One worker per node	Every script on the node blips	1	Simple, but one bad script restarts every script on the node
B. One worker per script	One script	= script count (hundreds to thousands)	Infeasible: ~tens of MB per .NET process [A-3]
C. One worker per seam or driver instance	One seam / driver	3 to ~10	Arbitrary; a hot Calc driver still shares fate with its siblings
D. Small sharded pool + an isolation worker	One shard (1/K of the scripts), then one script	K + 0..1 (default 2 + 1)	Recommended

3.2 Recommended: sharded pool with an isolation worker

• K shard workers per driver node. Scripting:Worker:Shards defaults to 2 and is clamped to 1–8. A script is placed by hash(scriptKey) mod K. The script key is the guard key used today: the VT id, the calc RawPath, or the alarm id. Placement is deterministic, so a script's compiled PE and warm state stay on one worker across restarts.
• One isolation worker ("penalty box") is spawned on demand. Scripts under suspicion (§5.3) run there, one evaluation at a time. A kill in the isolation worker is therefore unambiguous.
• Inside a worker: a bounded evaluation scheduler with P dedicated threads, where P = max(2, ProcessorCount / K) [A-4]. Single flight per key is kept, as the guard does today. A queue that is full answers Busy, which the host treats as a refusal, not a strike.
• The passthrough fast path stays in the host. It never runs user code, and it is probably the largest share of evaluations [A-5].

3.3 Lifecycle and supervision under Akka

The design splits a control plane (actors) from a data plane (a lock-free client). An Akka Ask per evaluation would add mailbox hops to the hottest path for no benefit.

DriverHostActor (driver role)
 └─ ScriptRunnerActor            node-wide; owns the pool policy, attribution ledger, quarantine table
     ├─ ScriptWorkerActor #0     owns one OS process + pipe; state machine below
     ├─ ScriptWorkerActor #1
     └─ ScriptWorkerActor iso    spawned on first suspect, stopped after IsolationIdle (default 10 min)

IScriptRunner (DI singleton)  ── data plane ──►  ScriptWorkerConnection (per worker; pipe writer + correlation map)
   used by RoslynVirtualTagEvaluator, CalculationEvaluator, ScriptedAlarmEngine

• Ownership. ScriptRunnerActor is created by DriverHostActor, next to VirtualTagHostActor and the scripted-alarm host, so it exists only on driver-role nodes and dies with the driver tree. Its supervisor strategy is OneForOne, and a restart of a ScriptWorkerActor kills and respawns its process.

• Process exit is a message, not an exception. A watcher on Process.Exited / WaitForExitAsync sends WorkerExited(exitCode, signal, inFlight[]) to its ScriptWorkerActor, so worker crashes never escalate through Akka supervision.

• Worker states:

  Spawning → Handshaking → Loading → Ready ⇄ Draining → Dead → Backoff → Spawning

  Ready is the only state that accepts evaluations. In the other states the connection queues per key, with latest-wins coalescing (§9).

• The host process must take its workers down with it.

  • Windows: every worker is assigned to a Job Object with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE. If the host dies, the kernel closes the job handle and kills the workers.
  • Linux (the docker-dev rig): the worker treats pipe EOF as fatal, and also polls its parent PID. The host is not trusted to clean up after itself.

• CoordinatedShutdown. In the before-actor-system-terminate phase, the host sends Shutdown, waits 1 s, then kills.

• Orphan sweep at start. The host kills only workers whose launch nonce file sits under this host's instance directory, never "every process with this name". mxaccessgw's OrphanWorkerTerminator killed a live gateway's workers when a second gateway started (scadaproj memory "wonder upgrade program"); this design avoids that by construction.

• Redeploy. ClearCompiledScripts / LoadAsync become IScriptRunner.ApplyGeneration(scriptSet). That call sends LoadScripts / Unload deltas to each shard and clears the host-side quarantine rows for edited or removed keys, which keeps today's lift rule. No worker restart is needed. Collectible ALCs in the worker are unloaded as they are today, and a stuck one no longer matters because its worker gets killed.

• Redundancy. No change. Each node runs its own scripts today and will keep doing so; workers are not shared across the pair.

---

4. IPC

4.1 Shape of the conversation

§1.1 showed that a run needs no host callbacks. Each evaluation is therefore one request frame and one reply frame, with no re-entrant RPC. ScadaBridge's SandboxWorkerProtocol has to proxy call/reply frames for external systems and databases; this design does not. Log lines from ctx.Logger are the only other worker→host traffic, and they are fire-and-forget.

4.2 Transport: named pipe vs gRPC

	Named pipe + length-prefixed frames	gRPC (h2c over UDS / named pipe)
Per-call overhead	Two syscalls plus serialize; no HTTP/2 framing [A-6]	HTTP/2 stream setup, HPACK and a Kestrel pipeline per call
Worker footprint	Plain console app	Needs Kestrel in the worker, or a client stack in it and a server in the host
Batching / priority	Free to design: control frames ahead of events, as in mxaccessgw WorkerFrameProtocol	Per-RPC; batching means streaming RPCs, which is a frame protocol again
Cross-platform	NamedPipeServerStream maps to a Unix domain socket on Linux; PipeOptions.CurrentUserOnly on both	Yes
Prior art in the family	mxaccessgw gateway↔worker (protobuf WorkerEnvelope, 4-byte LE length, priority-drained writer); ScadaBridge sandbox (JSON frames over stdio, 1 MiB cap)	OtOpcUa mesh transport (Commons/Protos/mesh_transport.proto) — but that is cross-node

Recommendation: a named pipe with 4-byte little-endian length-prefixed protobuf envelopes. It copies mxaccessgw's WorkerFrameProtocol discipline:

• a frame-size cap negotiated at the handshake (default 4 MiB);
• zero-length and oversize frames rejected;
• a monotonically increasing seq;
• control frames (heartbeat, shutdown, fault) drained ahead of evaluation traffic;
• coalesced flushes.

Google.Protobuf and Grpc.Tools are already central pins, and Commons.csproj:28-30 already compiles protos, so no new package is needed. MessagePack is pinned in Directory.Packages.props but referenced by no project, so it would be a new dependency in practice. Stdio, as ScadaBridge uses, is rejected: a script cannot write to the console (it is forbidden), but the runtime and the logger can, and one stray byte on stdout corrupts the stream.

4.3 Message schema (sketch)

syntax = "proto3";
package otopcua.scriptworker.v1;

message Envelope {
  uint32 protocol_version = 1;
  uint64 seq = 2;
  oneof body {
    Hello hello = 10;            // host→worker: nonce, max_frame_bytes, options
    Ready ready = 11;            // worker→host: pid, runtime version, sandbox fingerprint
    LoadScripts load = 12;       // host→worker: add/replace compiled-or-source scripts
    LoadResult load_result = 13; // worker→host: per-script compile outcome + diagnostics
    Unload unload = 14;          // host→worker: drop keys (ALC unload)
    EvalBatch eval = 15;         // host→worker
    EvalResults results = 16;    // worker→host (one or more completions)
    LogBatch logs = 17;          // worker→host, fire-and-forget, bounded
    Heartbeat heartbeat = 18;    // both ways; carries worker gauges (inflight, heap, gen2 count)
    Shutdown shutdown = 19;
    Fault fault = 20;
  }
}

enum ContextKind { CONTEXT_KIND_UNSPECIFIED = 0; VIRTUAL_TAG = 1; CALCULATION = 2; ALARM_PREDICATE = 3; }

message ScriptSource { string key = 1; bytes source_sha256 = 2; ContextKind kind = 3;
                       string source = 4; bytes pe_image = 5; repeated string input_paths = 6; }
message LoadScripts  { uint64 generation = 1; repeated ScriptSource scripts = 2; }

message EvalBatch    { repeated EvalRequest items = 1; }
message EvalRequest  {
  uint64 eval_id = 1; string key = 2; bytes source_sha256 = 3;
  repeated Input inputs = 4;      // positional against ScriptSource.input_paths; see 4.4
  int64 now_utc_ticks = 5;        // ctx.Now — host clock, so tests that pin the clock still work
  uint32 soft_timeout_ms = 6;
}
message Input { uint32 slot = 1; Variant value = 2; uint32 status_code = 3;
                int64 source_ts_ticks = 4; /* 0 = null */ int64 server_ts_ticks = 5; }

message EvalResult {
  uint64 eval_id = 1;
  oneof outcome { Variant value = 2; bool predicate = 3; ScriptError error = 4; }
  uint64 allocated_bytes = 5;     // same GC.GetAllocatedBytesForCurrentThread sample as today
  uint32 run_micros = 6;
}
message EvalResults { repeated EvalResult items = 1; }
message ScriptError { Kind kind = 1; string message = 2;
  enum Kind { THREW = 0; COMPILE = 1; SANDBOX = 2; SOFT_TIMEOUT = 3; UNSUPPORTED_TYPE = 4; BUSY = 5; } }

message Variant {
  TypeTag type = 1;               // exact CLR type — see 4.4
  oneof v { bool b = 2; sint64 i = 3; uint64 u = 4; double d = 5; float f = 6;
            string s = 7; int64 dt_ticks = 8; ArrayValue array = 9; }
  enum TypeTag { NULL = 0; BOOLEAN = 1; SBYTE = 2; BYTE = 3; INT16 = 4; UINT16 = 5; INT32 = 6; UINT32 = 7;
                 INT64 = 8; UINT64 = 9; FLOAT = 10; DOUBLE = 11; STRING = 12; DATETIME_UTC = 13;
                 ARRAY = 14; }
}
message ArrayValue { Variant.TypeTag element_type = 1; repeated Variant items = 2; }

message LogBatch { repeated LogLine lines = 1; uint32 dropped = 2; }
message LogLine  { string key = 1; uint32 level = 2; string message_template = 3;
                   string rendered = 4; string exception = 5; int64 ts_ticks = 6; }

4.4 How values and quality cross the boundary

• Quality crosses verbatim. DataValueSnapshot is (object? Value, uint StatusCode, DateTime? SourceTimestampUtc, DateTime ServerTimestampUtc) (Core.Abstractions/DataValueSnapshot.cs:13), and it maps 1:1 onto Input. The worker rebuilds the same snapshots, so ctx.GetTag(p).StatusCode and .SourceTimestampUtc behave as they do today. The input-quality gating does not move: VT holds evaluation while an input is Bad (VirtualTagActor.cs:264-269), Calc holds on Bad, and alarms short-circuit on AreInputsReady. A request is sent only when today's code would have run the script.
• The exact boxed CLR type is load-bearing. A script that writes (short)ctx.GetTag(x).Value throws InvalidCastException if the value arrives boxed as an int. Variant.type therefore carries the CLR type, not just the OPC UA type: DriverDataType has Int16 / UInt16, which have to survive the round trip. [A-7] I have not inventoried which CLR types each driver actually boxes, for example whether a Modbus Int16 arrives as short or as int. The shadow mode in §12 exists partly to find out.
• Unsupported types. An input whose CLR type has no tag is not silently converted. The request carries no value for that slot, and the worker answers UNSUPPORTED_TYPE; the host publishes Bad with the type name. On the result side, a script that returns an unsupported type (a List<T>, an anonymous object) also gets UNSUPPORTED_TYPE. That is a behaviour change: today the raw object goes to OpcUaPublishActor.AttributeValueUpdate (VirtualTagHostActor.cs:233), and what the sink does with it was not traced [A-8]. Shadow mode reports every such case before the flag flips.
• DateTime. Values are sent as UTC ticks. A Kind other than Utc is converted at the host edge and counted on a metric. [A-9] I assume drivers already produce UTC.
• Positional inputs. LoadScripts registers the script's ordered input_paths: the DependencyExtractor reads, after the {{equip}} resolution done at deploy. Requests then send values by slot, not by path string, which takes the dominant payload cost out of the hot path. A key with no registered slots, a fallback that should never happen, sends the paths inline.

4.5 Batching

• Opportunistic, not timed. The connection's writer drains every request queued at that moment into one EvalBatch and flushes once, as mxaccessgw's drain pass does. An idle system pays no added latency; a burst is amortised automatically. There is no coalescing timer, because it would add latency to every evaluation.
• Results are streamed, not batch-atomic. The worker sends EvalResults as items complete, coalescing whatever finished in the same scheduler pass, so a slow script never holds a fast sibling's reply.
• The seams must submit concurrently, or batching never happens. Today both Calc and alarms await one script at a time (§1.1), and with IPC that serial loop costs N × round-trip. Two seam changes are required:
  • Scripted alarms: ReevaluateAsync submits the whole dirty set at once, then applies results in the original order under _evalGate. Predicates are pure functions of a snapshot taken under the gate, so the state machine sees the same sequence.
  • Calculation: the pump dequeues everything pending, with each entry's dependency snapshot already captured at enqueue, submits it as one batch and publishes in dequeue order.
• Virtual tags become asynchronous. IVirtualTagEvaluator gains EvaluateAsync, and VirtualTagActor PipeTos the result to itself. While a run is in flight, further dependency changes update the map and set a "re-run" flag; when the result lands, the actor evaluates once more against the latest map (latest-wins). This also ends today's blocking of a default-dispatcher thread for up to 2 s (RoslynVirtualTagEvaluator.cs:174-175).

---

5. Pre-emption

5.1 Two deadlines

	Soft deadline T	Hard deadline T + G
Value	Today's per-seam budget, unchanged: VT 2 s, Calc runTimeoutMs (2 s), alarm 250 ms	G = Scripting:Worker:HardGrace, default max(1 s, T), clamped to 250 ms–30 s
Who acts	The host connection's timer	The host ScriptWorkerActor (authoritative) and a worker-side watchdog thread
Effect	The caller gets the timeout failure it gets today (Bad, same reason prefix ScriptEvalFailureReasons.TimedOutPrefix) and the run token is cancelled, so a polling script still ends itself as it does today	The worker process is killed (Process.Kill(entireProcessTree: true)). The worker watchdog calls Environment.FailFast first if it is still scheduling, so the dump and exit code name the key.
Strike	Yes (as today)	Yes, plus a kill strike (§5.3)

The grace period G keeps a script that overran by a few milliseconds from costing a whole shard restart, and the cooperative token from #734 still gets its chance. Making G = 0 (kill at T) is an open decision (§14).

Single flight between T and T+G. The key stays in flight, and a trigger is refused as today, which is not a strike. After the kill there is no orphan, so the stuck-orphan rule (StuckOrphanQuarantineFactor) becomes moot out of process; it stays for the in-process mode.

5.2 Resource limits on the worker

The limits combine prior art from ScadaBridge (heap limit) and mxaccessgw (process-tree kill):

• Heap: DOTNET_GCHeapHardLimit = Scripting:Worker:HeapLimitBytes, default 512 MiB [A-10]. Breaching it makes the worker throw OutOfMemoryException or fail fast. Either way the worker dies or reports, and the host survives.
• Windows Job Object (per worker):
  • JOB_OBJECT_LIMIT_PROCESS_MEMORY, set slightly above the heap limit as a backstop for native and JIT memory;
  • JOB_OBJECT_LIMIT_ACTIVE_PROCESS = 1, so no child processes are possible;
  • KILL_ON_JOB_CLOSE;
  • UI restrictions.
• Linux / containers: the heap limit only. A container's cgroup is the rig's concern. [A-11] I assume the docker-dev containers run without CAP_SYS_ADMIN, so a nested cgroup is not available.
• Liveness: a heartbeat every 1 s. Three missed heartbeats while not in a GC pause, which the heartbeat reports, trigger a kill with reason heartbeat. This catches a wedged worker that neither replies nor exits.

5.3 Attribution, restart, backoff and quarantine

A kill or crash takes down every in-flight evaluation on that shard. Attribution decides which one to blame.

1. Hard-deadline kill: the key that crossed T+G is known exactly. It takes a kill strike. The other in-flight keys on that shard are victims: their runs fail with "script worker restarted", which is not a strike.
2. Crash with no deadline crossed (OOM, stack overflow, native fault): attribution is ambiguous. Every key in flight at exit becomes a suspect, and suspects move to the isolation worker.
3. Isolation worker: evaluations run one at a time, so a crash there attributes to exactly one key, which takes a kill strike. A suspect that completes IsolationProbation (default 20 successful runs or 10 min) without incident returns to its shard.
4. Quarantine: a key is quarantined after Scripting:Worker:QuarantineAfterKills kill strikes (default 2) within 1 h, or by today's rules (N consecutive soft timeouts, allocation breach). The quarantine table lives in the host (ScriptRunnerActor), so it survives worker restarts. It is lifted by a source change or a redeploy, exactly as today, and quarantine reasons keep the "script quarantined: " prefix so every existing consumer and test classifier still matches.
5. Restart backoff per worker: exponential, 250 ms → 30 s cap, ×2, with ±20% jitter. The backoff resets after 60 s of Ready. A worker that dies during Loading (a poison PE or a runtime fault) does not loop: after 3 consecutive load-phase deaths, the scripts in its generation are loaded one by one in the isolation worker to find the culprit.
6. Crash-loop circuit: if all shards are in backoff for longer than Scripting:Worker:CircuitOpenAfter (default 2 min), the runner reports Unavailable (§10) and logs one Error. It keeps retrying at the cap.

---

6. Security

Process isolation contains failures. Most of the capability control stays with the compile-time sandbox, which is retained. Defence-in-depth adds the following.

• The compile-time sandbox is unchanged. It still includes ScriptSandbox (with the TPA reference set filtered to System.*, ScriptSandbox.cs:118-122), ForbiddenTypeAnalyzer, and the injection guard. The worker runs exactly the same ScriptEvaluator.Compile, so editor/publish/runtime parity holds.

• Assembly allow-list by construction. The worker is its own app (ZB.MOM.WW.OtOpcUa.ScriptWorker), with its own deps.json and only these references:

  • Core.Scripting;
  • Core.Scripting.Abstractions;
  • Core.Abstractions;
  • Serilog (core only, for ILogger on ctx);
  • Google.Protobuf.

  Its trusted-platform list therefore contains the shared framework plus those assemblies. Akka, SqlClient, the OPC UA stack, driver assemblies and the Secrets library are not loadable in the worker, even through reflection. At startup the worker asserts its loaded-assembly set against a pinned list and exits if the list differs.

• No secrets in the environment. ProcessStartInfo.Environment is cleared and then repopulated from an explicit allow-list: DOTNET_* runtime knobs, TEMP, PATH minimal, and the launch nonce. Host env vars like ServerHistorian__ApiKey (CLAUDE.md "Never commit secrets") and the ConfigDb connection string must never reach the worker. ScadaBridge's sandbox inherits the parent environment; do not copy that part.

• No configuration access. The worker gets no appsettings path. Every option it needs (budgets, heap, frame cap) arrives in Hello.

• Pipe hardening (mxaccessgw pattern):

  • a random pipe name;
  • PipeOptions.CurrentUserOnly, or an explicit ACL when the worker runs as a different account;
  • a one-time nonce passed in an env var (never on the command line; mxaccessgw WorkerBootstrap.md explains why), which the worker must echo in Ready;
  • the host accepts exactly one connection per launch.

• Filesystem:

  • the working directory is an empty per-worker temp directory;
  • the PE cache (§7) is opened read-only by the worker, if the host compiles, or written only under its own subdirectory;
  • on Windows, a dedicated low-privilege account (see §14) gets read access only to the install folder and its own temp directory.

• Network: the analyzer already forbids System.Net. Process-level options are a Windows Firewall outbound-block rule scoped to the worker executable, created by the installer, and on Linux nothing beyond the container. This is an open decision (§14). The recommendation is the installer firewall rule on Windows, and to accept the analyzer as the control on Linux.

• What the host trusts from the worker: nothing it cannot validate. Frames are size-capped and schema-checked, an eval_id must match an outstanding request, and log lines are length-capped and rate-limited (§11). A compromised worker can at worst return wrong values for scripts on its own shard, which the same script could have done anyway.

---

7. Compile caching across restarts

The problem: today every process start recompiles every script lazily (CompiledScriptCache is memory-only). Out of process, a shard restart has the same cost, and it can happen in the middle of production running. mem-probe measured the old CSharpScript path at roughly 2.4 MiB of working set per retained script (tools/mem-probe/MemProbe/Program.cs:1-4). Compile time is not measured anywhere in the repo [A-12]; I assume tens of ms per script warm and seconds for the first compile while Roslyn JITs. For 1,000 scripts, that is far outside a 2 s restart budget.

Recommendation: a content-addressed PE cache on node-local disk.

• Key: SHA-256 over these fields:

  • the source;
  • the ContextKind, i.e. TContext and TResult;
  • the Roslyn compiler version;
  • a sandbox fingerprint: a hash of ForbiddenTypeAnalyzer's lists, the ScriptSandbox reference set and the wrapper template.

  An analyzer change therefore invalidates every entry, and a cached PE can never skip a newer sandbox rule.

• Value: the emitted PE bytes plus an HMAC-SHA256 tag. The HMAC key is a per-node random key created at first start and stored next to the cache, readable only by the host account. The worker loads a PE only if the HMAC verifies, because loading attacker-supplied PE bytes would be code execution outside the analyzer.

• Location: Scripting:Worker:PeCacheDir. It must be absolute and outside the content root, the same fail-closed rule the family applies to Secrets:SqlitePath (scadaproj CLAUDE.md, Secrets row). Entries whose key is not in the current generation are pruned at ApplyGeneration.

• Where compilation happens: in the worker (recommended). Roslyn's memory and CPU leave the host. A pathological source that makes Roslyn itself blow up kills a worker, not the server, and the worker writes the PE plus HMAC (it holds the key through Hello). The alternative is to compile in the host and ship pe_image in LoadScripts. That keeps the worker free of Roslyn (smaller, faster start), but compile-time blowups stay in the host. This is an open decision (§14).

• Restart path: Spawning → Ready takes about 150 ms of runtime start [A-13]. Then LoadScripts sends hash + key + slots only. The worker loads PEs from the cache lazily, on each key's first evaluation, and a cache miss compiles. Loading a PE into a collectible ALC is about 1 ms [A-13].

• Negative cache: Calculation already remembers compile failures by source hash (R07:P4, CalculationEvaluator.cs:63-68). The worker reports compile failures in LoadResult, and the host keeps that negative cache for all seams, so a broken script never re-enters Roslyn.

---

8. Performance budgets

8.1 Reference loads in the repo

Source	Load
docs/v2/Galaxy.Performance.md (EventPump sizing, 24 h soak)	50,000 tags at 1 Hz on one node; compressed CI soak at 1,000 tags
tests/Server/.../NodeManagerLockTimeBenchmarkTests.cs:41-105	10,000-tag folder; 200 equipment × 250 tags (50,000 variables)
ScriptRunGuardTests.cs:170-205	a 1 Hz runaway and a healthy sibling over 30 triggers

No repo perf test exercises scripts at scale. [A-14] I assume a planning envelope of 5,000 compiled script evaluations per second per node (for example 5,000 VT + Calc at 1 Hz, or 500 scripts at 10 Hz) and size for 2× headroom (10k/s). The passthrough fast path is excluded because it stays in the host.

8.2 Budgets (acceptance gates for the implementation)

Metric	Budget	Notes
Added latency per evaluation, p50	≤ 150 µs	Host enqueue → reply decoded, trivial script, idle system
Added latency per evaluation, p99	≤ 2 ms	At 10k evals/s. Compare the 250 ms alarm budget: under 1% of it
Throughput	≥ 10,000 trivial evals/s/node with K=2	With opportunistic batching
Host CPU for IPC	≤ 25% of one core at 10k evals/s	Serialisation + pipe I/O, measured by an opt-in perf test. Since lmxopcua#789 the harness reports it as the out-of-process runner's host CPU minus the in-process runner's in the same phase ("vs InProcess"): the raw figure includes the harness's own pacing loop and pool, which InProcess pays too (§12.2)
Bytes per eval on the wire	≤ 64 B + 24 B per input	Positional slots, §4.4
Shard restart → first result, 1,000 cached scripts	≤ 2 s	§7 lazy PE load
Worker idle RSS	≤ 80 MB + scripts [A-3]	Per shard. Proposed revision (lmxopcua#789, §12.2): ≤ 80 MB + scripts for a worker that has only loaded cached PEs (measured 36–58 MB); ≤ 200 MB + scripts once Roslyn has run in it (measured 159–187 MB), because compiling in the worker (§14 #7) keeps Roslyn's code and metadata resident and no GC setting moves it
Kill latency after T+G	≤ 100 ms	Host timer resolution plus Process.Kill

Comparison with today: the in-process cost per compiled run is one Task.Run hop plus WaitAsync, which is a few µs to tens of µs and was never measured [A-15]. The out-of-process path is therefore one to two orders of magnitude slower per call. That is acceptable only because the budgets above sit two to three orders of magnitude below the script budgets (250 ms / 2 s), and because batching amortises the syscalls. The risk case is a large scripted-alarm engine, with its serial gate and its 250 ms budget. That is why §4.5 makes the batch-submit change a prerequisite, not an optimisation.

---

9. Failure semantics: what quality a tag gets

The principle: a runner failure must never look like a script result. In particular, an alarm must never clear because the worker died.

Situation	Virtual tag / Calculation tag	Scripted alarm
Soft timeout (T)	BadInternalError + last value, once per transition, as today	Condition frozen + BadInternalError quality annotation, as today
This key killed at T+G, or crashed in isolation	Same as soft timeout; reason "script killed: exceeded hard deadline …"	Same as soft timeout
Victim: in flight on a shard that died for another key	BadInternalError + last value, reason "script worker restarted"; re-evaluated on the shard's return with the latest inputs (latest-wins)	Frozen + Bad annotation; re-evaluated on return
Not in flight while its shard restarts	Unchanged. Keeps its last published value and quality. A trigger during the outage is coalesced (latest-wins, one per key) and evaluated when the shard is Ready	Same; the condition holds its state
Shard not back within Scripting:Worker:UnavailableGrace (default 5 s)	Coalesced keys publish Bad + last value, reason "script runner unavailable"	Frozen + Bad annotation
Circuit open (§5.3.6)	Every compiled-script key on the node is Bad as above; passthrough keys are unaffected	Every compiled predicate is frozen + Bad
Quarantined	"script quarantined: …" Bad, as today	As today
Host process dies	Workers die with it (Job Object / pipe EOF); no change from today's host-death behaviour	—

Status code: BadInternalError stays for script-attributable faults (the existing contract). For "the runner is unavailable, not this script's fault", this design proposes BadResourceUnavailable (0x80040000), so operators and the ScadaBridge consumer can tell a broken script from a sick node. The alternative is to keep BadInternalError everywhere for zero consumer impact. This is an open decision (§14); if it is taken, the ScadaBridge consumer-impact record must note it (scadaproj sister-project-history skill, otopcua.md).

No silent fallback to in-process. If the worker cannot start, the runner does not run scripts in the host. That would reintroduce exactly the risk the flag was set to remove. Operators flip the flag back (§12). This is an open decision (§14).

---

10. Observability

All metrics go on OtOpcUaTelemetry.Meter, so the existing ZbTelemetryOptions allow-list applies. scadaproj memory: "ZbTelemetryOptions.Meters is a silent allowlist"; check that the new names flow end to end.

Metric	Type	Tags
otopcua.script.worker.up	gauge	shard
otopcua.script.worker.starts / .exits	counter	shard, reason = clean | killed_deadline | killed_heartbeat | oom | crash | load_failure
otopcua.script.worker.inflight / .queue_depth	gauge	shard
otopcua.script.worker.heap_bytes	gauge (from heartbeat)	shard
otopcua.script.ipc.roundtrip	histogram (µs)	kind
otopcua.script.eval.duration	histogram (µs, worker-measured)	kind
otopcua.script.compile	counter	cache = hit | miss | negative
otopcua.script.quarantined (existing)	counter	adds reason = killed | isolated_crash
otopcua.script.run_refused (existing)	counter	unchanged
otopcua.script.isolation.suspects	gauge	—
otopcua.script.runner.state	gauge (0 ready / 1 degraded / 2 unavailable)	—

Per-key series: the existing counters tag the series with script = key. Do not add key tags to the new per-eval histograms (cardinality).

Logs:

• the worker's own stderr is captured, prefixed with ScriptWorker[shard], into the host log;
• every kill logs one Warning naming the key and the reason, and every quarantine logs one Error, as today;
• ctx.Logger lines come back in LogBatch. The host re-emits them through ScriptRootLogger with the same ScriptId / VirtualTagId properties, so scripts-*.log, the error mirror, the DPS script-log topic and its 20/min-per-source rate limit all keep working unchanged;
• the worker caps its outbound log buffer (default 1,000 lines / 1 MiB) and reports drops as LogBatch.dropped, so a script logging in a tight loop cannot flood the pipe.

Health: expose the runner state as a non-fatal component of the driver-node status (/health/ready stays green while degraded, because OPC UA still serves driver data). Whether Unavailable should fail readiness is an open decision (§14). The AdminUI status dashboard gets a "Script runner" tile with the per-shard state, restarts in the last hour and the quarantine list. Quarantines already surface through the QuarantinedScripts properties.

Traces: one Activity per EvalBatch (not per evaluation), with batch.size and shard.

---

11. Seam changes in the host (summary)

Component	Change
Core.Scripting	New IScriptRunner (EvaluateAsync(ScriptInvocation[]), ApplyGeneration, QuarantinedKeys, ResetAll). InProcessScriptRunner wraps today's ScriptRunGuard + CompiledScriptCache exactly.
New Core.Scripting.Protocol	The proto file, the frame codec and the Variant ↔ CLR mapping. Shared by host and worker.
New ZB.MOM.WW.OtOpcUa.ScriptWorker	Console app: handshake, scheduler, watchdog, PE cache, log forwarding. Shipped in the Host's publish output.
New Runtime/Scripting/	ScriptRunnerActor, ScriptWorkerActor, OutOfProcessScriptRunner (data plane), Job Object interop (Windows-only, behind OperatingSystem.IsWindows()).
IVirtualTagEvaluator / VirtualTagActor	Add EvaluateAsync; the actor PipeTos with latest-wins re-run (§4.5).
RoslynVirtualTagEvaluator, CalculationEvaluator, ScriptedAlarmEngine	Replace direct ScriptRunGuard use with IScriptRunner. Calc pump and alarm ReevaluateAsync submit batches (§4.5).
ScriptingOptions / ScriptingOptionsBinding	New keys (§12), clamped with startup Warnings like the existing ones.

---

12. Migration and rollout

Configuration (node-local appsettings, Scripting section, consistent with the existing knobs):

Key	Default	Meaning
Scripting:Runner	InProcess	InProcess | OutOfProcess | Shadow
Scripting:Runner:Seams	VirtualTag + Calculation	Optional per-seam subset for OutOfProcess / Shadow (e.g. ["Calculation"]). Unset selects VirtualTag and Calculation; ScriptedAlarm only when listed (§14 #10, as built in §12.3)
Scripting:Worker:Shards	2	1–8
Scripting:Worker:HardGrace	max(1s, T)	§5.1
Scripting:Worker:HeapLimitBytes	536870912	§5.2
Scripting:Worker:QuarantineAfterKills	2	§5.3
Scripting:Worker:UnavailableGrace	5s	§9
Scripting:Worker:PeCacheDir	none (required when not InProcess)	§7; absolute, outside the content root, fail-closed
Scripting:Worker:Identity	none (host's token)	Windows only: NT SERVICE\OtOpcUaScriptWorker → restricted worker token + pipe ACL (§14 #5, record §12.5)

Phases (each one is its own issue and PR):

1. P0, in-process refactor with no behaviour change.
   • Introduce IScriptRunner + InProcessScriptRunner.
   • Make VT evaluation async (PipeTo), and make the alarm engine and the Calc pump batch-submit.
   • Gate: every existing Scripting / VirtualTags / Calculation / ScriptedAlarms suite stays green unchanged, plus new ordering tests for batch-then-apply.
   • This phase has value on its own: it ends the dispatcher-thread blocking.
   • Partly landed in lmxopcua#779: VT evaluation is async (IVirtualTagEvaluator.EvaluateAsync + stash in VirtualTagActor), so the dispatcher-thread blocking is gone. The Calc pump and the alarm engine still run one script at a time; why that was not relaxed in-process is recorded in docs/VirtualTags.md §"Upstream-tag feed". IScriptRunner and batch-submit remain to do.
   • Batch submit landed in lmxopcua#780 (fourth P1 slice, §12.3): IScriptRunner.RunBatch, used by the Calc pump, the VirtualTag seam (a coalescer) and the scripted-alarm engine on the out-of-process path. In process the pump and the engine stay one at a time, unchanged.
2. P1, worker + protocol behind the flag.
   • Ship ScriptWorker, OutOfProcessScriptRunner and the PE cache.
   • Default stays InProcess. Enable on the docker-dev rig only.
   • Gate: §13 runaway / containment tests green on Linux (docker-dev) and Windows (windev box, which has the .NET 10 toolchain).
   • Partly landed in lmxopcua#780 (P1 slice): Scripting:Runner / Scripting:Runner:Seams / Scripting:Worker:* (bound, clamped, PeCacheDir path rule fail-closed), IScriptRunner + InProcessScriptRunner, the protobuf frame protocol (Core.Scripting.Protocol), the ScriptWorker app (compile in the worker, P-thread scheduler, watchdog, heartbeats, log forwarding, pipe-EOF / parent-pid exit, forbidden-assembly self-check), OutOfProcessScriptRunner (K shards + isolation worker, soft/hard deadlines, attribution, kill-strike quarantine, backoff, fail closed, Job Object on Windows), the VirtualTag and Calculation seams over it, and BadResourceUnavailable for runner-unavailable / worker-restarted (ScadaBridge#112). The §13 containment cases run on macOS and Windows (windev). Deviations: supervision is a plain object owned by DI, not ScriptRunnerActor/ScriptWorkerActor; the per-seam batch submit (P0) is still one-at-a-time. Remaining: the HMAC'd PE cache, a self-contained worker publish + the low-privilege account / pipe ACL / firewall rule (installer), the runner-state gauge and readiness component, victim re-evaluation on shard return, isolation-worker idle stop, the orphan sweep by nonce file, the scripted-alarm seam, the docker-dev enablement and the §8 perf harness.
   • Second P1 slice (lmxopcua#780): the HMAC'd PE cache (§7; keyed by source + kind + a compile fingerprint, tampered or mismatched entries evicted, pruned by recency at startup instead of by generation — this runner has no generation object), attribution before release (the quarantine race), loading keys counted in attribution (culprit-finding for a compile that crashes its worker, by isolation rather than the §5.3.5 three-strikes bisection — loads are per key here), proactive restart + Recovery / RetryWhen re-evaluation of victims in the VT and Calc seams, isolation idle stop (IsolationIdleTimeout), the crash-loop circuit (CircuitOpenAfter), the §10 gauges and histograms, the degraded-but-ready script-runner health check (§14 #12), and the orphan sweep by launch marker (pid + start time + image, host gone). Still open: self-contained worker publish, installer (account, pipe ACL, firewall), docker-dev live drill, scripted-alarm seam, the §4.5 batch submit, the §8 perf harness.
   • Fourth P1 slice (lmxopcua#780, record in §12.3): the §4.5 batch submit (IScriptRunner.RunBatch, one EvalBatch frame per worker; Calc pump, VT coalescer, alarm dirty set), exact per-item deadlines for work queued in the worker (deferral / start notices), and the scripted-alarm seam behind Scripting:Runner:Seams — alarms run out of process only when listed. Still open: installer (account, pipe ACL, firewall, PeCacheDir ACL), #789's remaining misses, then P2 Shadow.
   • Third P1 slice (lmxopcua#780, record in §12.1): a self-contained worker publish for self-contained (Windows) Host publishes, started through its apphost; the docker-dev live drill (docker-dev/docker-compose.script-runner.yml); the §8 perf harness (Category=Perf, opt-in), which passes the accepted 5k evals/s/node gate; and two runner fixes the harness found (a caller refused by its own finished run; heartbeat starvation under a compile burst). Still open: installer (account, pipe ACL, firewall, PeCacheDir ACL), scripted-alarm seam, §4.5 batch submit, the §8.2 budget misses (lmxopcua#789) and the load-timeout queue issue (lmxopcua#790).
3. P2, shadow mode on docker-dev and QA.
   • Shadow runs the in-process evaluation as the published result, and also submits to the worker.
   • It compares value, status class and failure kind per evaluation, and counts otopcua.script.shadow.mismatch{kind, reason}. Mismatching keys are logged once each.
   • Shadow doubles script CPU and still carries the in-process risk, so it is a time-boxed diagnostic (for example 1 week on QA), not a mode to run in production.
   • Exit criterion: zero unexplained mismatches. This is where [A-7] / [A-8] get answered.
   • Landed in lmxopcua#780 (P2 slice, record in §12.4): ShadowScriptRunner, all three seams, bounded and drop-counted, readiness untouched; docker-dev drill with alarms. The QA shadow week is still to run.
4. P3, flip on QA, then the other tiers.
   • Set Scripting:Runner=OutOfProcess per node, seam subset first (Calculation, then VT, then alarms; alarms last because of the 250 ms budget).
   • Note: remote command execution on the wonder boxes is currently scoped to user-requested deploys (CLAUDE.md hold), so tier rollout rides normal deploys.
5. P4, decide the long-term status of InProcess. The recommendation is to keep it for dev and tests and as the documented rollback, and to retire StuckOrphanQuarantineFactor from the docs as "in-process only".

Rollback: set the flag back to InProcess and restart the node. No data or config migration is involved. The PE cache directory can be deleted at any time.

Docs to update when implementing:

• docs/VirtualTags.md §"Known resource limits" and §"Per-evaluation timeout";
• docs/ScriptEditor.md §"Runtime safety";
• docs/ScriptedAlarms.md;
• docs/ServiceHosting.md: the new process, Job Object and firewall rule;
• docs/security.md;
• docs/Telemetry.md;
• the scadaproj index row, if the status-code decision changes the ScadaBridge-visible contract.

12.1 P1 record: self-contained publish, docker-dev live drill, §8 perf (2026-11-10)

Self-contained worker publish.

• A -r win-x64 --self-contained Host publish now publishes the worker self-contained for the same RID into script-worker/: 236 files, about 97 MB, with its own runtime. It is not trimmed and not single-file.
• The launcher starts the worker's apphost (ZB.MOM.WW.OtOpcUa.ScriptWorker.exe) directly. It does so only when the worker's runtimeconfig.json lists includedFrameworks. Otherwise (framework-dependent) it keeps dotnet exec <dll>.
• A framework-dependent publish is unchanged: 51 files, no includedFrameworks, run through the muxer.
• Gate on windev: the Host was published win-x64 self-contained under C:\build, and the whole ScriptWorker.Tests suite was run against the published apphost (OTOPCUA_SCRIPTWORKER_TEST_PATH).
  • Result: 132/132.
  • It includes A_published_self_contained_worker_runs_as_its_own_process. That test checks that the worker process's MainModule is the .exe, so no muxer is involved.

docker-dev live drill. Image otopcua-host:dev was built from the branch at f50b29ab plus the self-contained change. That build is from before the two runner fixes below. The drill ran with docker-compose.script-runner.yml on central-1/central-2.

Step	Observed
Normal VT + Calc	Workers start on the first compiled evaluation (dotnet exec /app/script-worker/…dll). okvt (HR100+1) stays Good and tracks HR100. Writing HR5 = 21 on the fixture gives Calc HR5x2 42 Good on both nodes, and the sibling VT drillsib (HR5×3) 63 Good.
while (true) { } VT (1 s timer, T = 2 s)	Logged exceeded its hard deadline (4000 ms); killing ScriptWorker[1] twice, 5 s apart. After the second: quarantined: "it exceeded its hard deadline and its worker was killed (2 time(s) in the last hour)…". Both nodes did the same. okvt and Calc share that shard and kept updating Good (26 Good samples, no Bad) because neither was in flight at the kills. run_refused then counts the quarantined trigger.
kill -9 both workers on central-1	A 1.5 s busy VT (drillslow) was in flight on shard 1. It published BadResourceUnavailable at 05:41:22.28, became a suspect and moved to isolation. It was Good again 30 ms later, re-evaluated in isolation. After HR5 = 22 it read 1022. Shards 0 and 1 restarted within about 250 ms. Isolation probation (20 clean runs) returned it to its shard. Calc gave 44 Good and the sibling 66 Good.
Circuit open (worker dll renamed, all workers killed)	/health/ready stayed 200. The overall status went to Degraded, and script-runner showed Degraded → Unavailable (circuit open) after 2 min. Logged Script runner circuit OPEN … once. otopcua_script_runner_state = 2, worker_up = 0 per shard. VTs and Calc (after an HR5 change) published BadResourceUnavailable. Driver tags (HR100) stayed Good, so OPC UA kept serving.
Recovery (dll restored)	The next backoff retry (cap 30 s) started the workers. Logged circuit closed. Calc 46 (HR5 = 23 × 2), sibling 69, drillslow 1023, okvt Good, all without any input change.
Metrics on :9000/metrics	otopcua_script_runner_state, worker_up/inflight/queue_depth/heap_bytes{shard}, worker_starts/exits{reason}, compile{cache}, ipc_roundtrip/eval_duration histograms, quarantined{reason}, isolation_suspects.

On the wire, a Bad publish carries a null Value. This holds for every Bad status in this server, including BadInternalError and BadOutOfRange. The seams keep the last value, but an OPC UA client does not see it. This was noted for ScadaBridge#112.

Rig state afterwards.

• Drill rows were deleted, HR5 was set back to 5, and the config was redeployed. The revision hash is e221b179…, byte-identical to the pre-drill gate-cleanup deployment.
• otopcua-host:dev was re-tagged to the pre-drill image 24f9d7287ed2.
• central-1/central-2 were recreated from the base compose file: InProcess, no Scripting__* env. The four site nodes, sql and traefik were never touched.
• The two central containers were recreated (twice), so their container-local secret stores and their PE cache dirs under /tmp are gone.

Two defects outside the runner were found and filed:

• lmxopcua#787: a disabled VirtualTag is still deployed and evaluated.
• lmxopcua#788: worker start failures log an Error with a stack trace on every retry.

§8 perf harness.

• ScriptRunnerPerfTests, Category=Perf, opt-in with OTOPCUA_SCRIPT_PERF=1.
• The runners are driven directly: trivial one-input script, 2,000 keys over 50 sources.
• Phases:
  • idle sequential;
  • paced open loop at 5k/s and at 10k/s;
  • 64-way closed loop.
• Host CPU is this process's CPU time as a share of one core, harness included. Under the same harness InProcess shows 60–75 % at 5–10k/s, and that is the harness's own floor.

macOS:

Script worker restart / RSS / kill latency (Unix Arm64, 14 logical CPUs, .NET 10.0.14, phase 10 s)

Measure	Result	§8.2 budget
1000 scripts, empty cache, all first results (compile + write)	1,633 ms	—
1000 scripts, fresh runner on a warm cache, all first results	217 ms	≤ 2,000 ms
kill both shards → every script's next correct result	482 ms	≤ 2,000 ms
worker RSS after first eval (max of 2 shards)	188 MB	≤ 80 MB + scripts
worker RSS with 500 scripts loaded per shard (max)	358 MB	—
kill latency after T+G (5 runs, ms)	31 / 35 / 31 / 32 / 33	≤ 100 ms

Script runner perf (Unix Arm64, 14 logical CPUs, .NET 10.0.14, phase 10 s)

Runner	Phase	evals/s	p50 µs	p99 µs	host CPU	ok / not ok
InProcess	idle, sequential	189,154	4	8	446 %	5,000 / 0
InProcess	paced 5,000/s	4,998	32	223	60 %	49,999 / 0
InProcess	paced 10,000/s	9,991	41	364	74 %	99,986 / 0
InProcess	closed loop ×64	789,533	4	15	409 %	7,896,214 / 0
OutOfProcess K=1	idle, sequential	21,641	45	64	264 %	5,000 / 0
OutOfProcess K=1	paced 5,000/s	4,999	149	860	108 %	49,998 / 0
OutOfProcess K=1	paced 10,000/s	9,995	205	1,763	159 %	99,989 / 0
OutOfProcess K=1	closed loop ×64	206,862	258	1,368	526 %	2,068,749 / 0
OutOfProcess K=2	idle, sequential	22,642	41	70	267 %	5,000 / 0
OutOfProcess K=2	paced 5,000/s	4,999	182	722	125 %	49,994 / 0
OutOfProcess K=2	paced 10,000/s	9,996	232	720	237 %	99,998 / 0
OutOfProcess K=2	closed loop ×64	336,519	150	993	544 %	3,365,226 / 0
OutOfProcess K=4	idle, sequential	18,461	50	97	303 %	5,000 / 0
OutOfProcess K=4	paced 5,000/s	4,999	201	455	197 %	49,997 / 0
OutOfProcess K=4	paced 10,000/s	9,997	126	732	176 %	99,989 / 0
OutOfProcess K=4	closed loop ×64	297,094	178	1,015	507 %	2,970,998 / 0

windev (Windows VM; framework-dependent worker built beside the tests):

Script runner perf (Win32NT X64, 36 logical CPUs, .NET 10.0.8, phase 10 s)

Runner	Phase	evals/s	p50 µs	p99 µs	host CPU	ok / not ok
InProcess	idle, sequential	5,876	142	253	204 %	5,000 / 0
InProcess	paced 5,000/s	4,997	136	335	55 %	49,996 / 0
InProcess	paced 10,000/s	9,989	105	274	6 %	99,984 / 0
InProcess	closed loop ×64	177,782	26	10,062	543 %	1,778,824 / 0
OutOfProcess K=1	idle, sequential	1,276	750	1,263	74 %	5,000 / 0
OutOfProcess K=1	paced 5,000/s	4,997	1,073	3,028	68 %	49,990 / 0
OutOfProcess K=1	paced 10,000/s	9,992	1,139	2,840	102 %	99,999 / 0
OutOfProcess K=1	closed loop ×64	38,464	1,514	6,547	350 %	384,727 / 0
OutOfProcess K=2	idle, sequential	1,207	795	1,376	80 %	5,000 / 0
OutOfProcess K=2	paced 5,000/s	4,998	956	1,856	111 %	49,997 / 0
OutOfProcess K=2	paced 10,000/s	9,962	942	1,916	153 %	99,992 / 0
OutOfProcess K=2	closed loop ×64	51,241	1,129	5,894	594 %	512,504 / 0
OutOfProcess K=4	idle, sequential	1,163	828	1,366	61 %	5,000 / 0
OutOfProcess K=4	paced 5,000/s	4,997	804	1,472	172 %	49,991 / 0
OutOfProcess K=4	paced 10,000/s	9,992	757	1,591	233 %	99,992 / 0
OutOfProcess K=4	closed loop ×64	69,503	774	6,321	1,051 %	695,094 / 0

Script worker restart / RSS / kill latency (Win32NT X64, 36 logical CPUs, .NET 10.0.8, phase 10 s)

Measure	Result	§8.2 budget
1000 scripts, empty cache, all first results (compile + write)	11,625 ms	—
1000 scripts, fresh runner on a warm cache, all first results	1,715 ms	≤ 2,000 ms
kill both shards → every script's next correct result	2,028 ms	≤ 2,000 ms
worker RSS after first eval (max of 2 shards)	158 MB	≤ 80 MB + scripts
worker RSS with 500 scripts loaded per shard (max)	232 MB	—
kill latency after T+G (5 runs, ms)	257 / 210 / 200 / 217 / 209	≤ 100 ms

Verdict against §8.2 and the accepted gate:

• Accepted gate (≥ 5,000 evals/s/node at K = 2): PASS on both hosts (macOS 336k/s, windev 51k/s closed loop), with every result correct.
• Throughput ≥ 10k/s: PASS.
• Idle p50 ≤ 150 µs: PASS on macOS (41 µs), MISS on windev (795 µs; InProcess there is already 142 µs).
• p99 ≤ 2 ms at 10k/s: PASS (macOS 0.7 ms, windev 1.9 ms).
• Warm-cache restart of 1,000 scripts ≤ 2 s: PASS (0.2 s / 1.7 s).
• Kill latency ≤ 100 ms: PASS on macOS (about 32 ms), MISS on windev (about 210 ms).
• Host CPU ≤ 25 % of a core at 10k/s: MISS. windev runs K = 2 at about 153 %. On macOS, K = 2 runs about 240 % against InProcess's 74 % under the same harness. A sequential probe measured 156 µs of host CPU per evaluation, and 63 µs with DOTNET_ThreadPool_UnfairSemaphoreSpinLimit=0. Thread-pool spinning is therefore most of it.
• Worker idle RSS ≤ 80 MB: MISS. It is about 190 MB on macOS and 158 MB on windev, because Roslyn is resident since compiles run in the worker.

Both misses → lmxopcua#789 (the §4.5 batching and per-eval allocation work).

Two runner defects found by the harness and fixed in this slice.

1. Refused own run. The single-flight slot was freed by a continuation that was not ordered against the caller's await. A caller that awaited one run and then started the next was sometimes refused with previous evaluation still running: 32 of 200 back-to-back runs, and about 80 % in the perf idle phase. The VT and Calc seams publish that as Bad. The slot is now freed before the caller resumes, idempotently.
   • Pinned by A_caller_that_awaited_its_run_is_never_refused_by_it (1,000 runs).
   • Mutation red.
2. Heartbeat starvation under a compile burst. Each load was compiled synchronously on its own thread-pool work item. On windev, 500 cold loads starved the heartbeat timer and the pipe writer. The host killed the shard (KilledHeartbeat), and all 500 keys became suspects. Compiles now run on min(8, max(1, CPUs/2)) dedicated ScriptWorker.Compile threads.
   • Pinned by A_burst_of_cold_compiles_does_not_starve_the_heartbeat (1,500 cold scripts, 50 ms heartbeat).
   • Red on macOS with the thread-pool version. Green 3/3 on macOS and windev; on windev 1,500 cold compiles take about 18 s.

The host load timeout still counts the time a load waits behind other compiles → lmxopcua#790.

12.2 lmxopcua#789 / #790 / #788 record (2026-11-10)

What cost the host CPU (#789). With DOTNET_ThreadPool_UnfairSemaphoreSpinLimit=0 a scratch probe showed the out-of-process runner at 33 % of a core at 10k evals/s against 25 % in process: the IPC work itself was about 10 µs per evaluation. The rest was thread-pool spin-waiting, triggered by several pool work items per evaluation, and on Unix by the socket engine. Changes:

• Combining frame writer (ScriptWorkerFrameWriter): the thread that queues a frame writes it itself when no write is in progress. This replaces the channel + writer task (one pool wake-up per submit). Consecutive evaluations or results are merged into one batch, frame order is preserved, and there is one write per pass. The old loop also called PipeStream.FlushAsync, which PipeStream does not override, so it queued a pool work item per pass.
• Buffered frame reader (ScriptWorkerFrameReader): one read pulls in every frame available, and frames are parsed from the buffer. It replaces two reads per frame plus a header allocation.
• One timer per evaluation for T and T+G. It replaces the WaitAsync(timeout) timer, the hard-deadline CancellationTokenSource and their two continuations.
• Slot release and deadline cancel run on the reader, before the caller is resumed. The release no longer depends on a continuation that is not ordered against the caller.
• Result dispatcher (ResultDispatcher, one thread per connection, blocking with Monitor.Wait, no spin): callers resume there, not on the pool. They do not resume on the reader either, because the Calculation pump publishes to OPC UA subscribers after its await. On the reader, a slow consumer would hold up heartbeats and get a healthy worker killed. That is pinned by a test, and a mutation that resumed callers on the reader was red.
• Blocking I/O on Unix (ScriptWorkerFrameCodec.UseBlockingIo):
  • The socket under a Unix .NET pipe turns non-blocking for good after one async call. After that even a "blocking" read goes through the epoll thread plus a spinning wait.
  • Both ends now connect, handshake, read (on a dedicated thread) and write synchronously. This was the largest single gain: sequential host CPU per evaluation fell from about 85 µs to about 13 µs.
  • A full pipe blocks only the one thread draining the writer; every other producer just queues. A peer that stops reading is killed by the heartbeat check within 3 s, which fails that write.
  • Windows keeps overlapped async I/O, because a synchronous full-duplex handle serialises reads against writes.
• Kill via TerminateJobObject on Windows. The job admits one process, so no system-wide snapshot is needed.
• Measured and not kept for memory, because none moved the compiled-worker RSS:
  • ReadyToRun worker publish: 195 MB;
  • GCgen0size 4 MB, GCConserveMemory, TieredPGO=0;
  • an aggressive compacting GC after a compile burst. The memory is Roslyn's JIT'd code and metadata, not the heap.

Before / after (§8 harness, OTOPCUA_SCRIPT_PERF=1; macOS figures under 5–25 load average from other agents on the box).

Budget	macOS before	macOS after	windev before	windev after
Host CPU at 10k/s, K=2, raw (InProcess same phase)	228–240 % (71 %)	79 % (69 %)	153 % (6 %)	123 % (7 %)
… vs InProcess (budget ≤ 25 %)	+157	+10 ✓	+147	+115 ✗
Host CPU, idle sequential, K=2	258 %	45 %	80 %	63 %
Idle p50 (≤ 150 µs)	41–43 µs	23 µs ✓	795 µs	507 µs ✗
p99 at 10k/s, K=2 (≤ 2 ms)	720–796 µs	1,024 µs ✓	1,916 µs	1,676 µs ✓
Closed loop ×64, K=2 (gate ≥ 5k/s)	342k/s	442k/s ✓	51k/s	62k/s ✓
Worker RSS after a compile (≤ 80 MB)	188 MB	187 MB ✗	158 MB	159–161 MB ✗
Worker RSS, warm PE cache, no compile	not measured	58 MB ✓	not measured	36 MB ✓
Warm restart, 1,000 cached (≤ 2 s)	245 ms	206–223 ms ✓	1,715 ms	1,778–1,921 ms ✓
Kill both shards → every next result (≤ 2 s)	512 ms	479–502 ms ✓	2,028 ms	2,004–2,087 ms (borderline)
Kill latency after T+G (≤ 100 ms)	31–35 ms	33–43 ms ✓	~210 ms	10–21 ms ✓

Additional measurements:

• In a scratch probe with a steady 10k/s load, K=2, before vs after: 116–158 % → 49–52 % of a core (InProcess 60–65 %). Sequential runs: about 25k → 46k evals/s.
• The first harness timed the kill from before the runaway's call, so it also counted the cold compile of each distinct runaway key (100–200 ms on the VM). The harness now times from the soft-timeout answer. The windev "before" figure of about 210 ms is therefore mostly compile time, not kill time.
• macOS K=1 p99 at 10k/s read 2.3 ms in one loaded run (budget 2 ms). K=2 is the configuration the budget names.

Still missing after this slice (lmxopcua#789 stays open, further tuning deferred by the owner):

• Windows host CPU for IPC: +115 points at 10k/s on the VM. That path still uses overlapped async I/O, so it pays a completion-port and pool hand-off per frame. The next step is either two simplex pipes with synchronous handles and dedicated threads, or a measurement on a non-VM box.
• Windows idle p50: 507 µs. The VM's in-process baseline is already 92 µs.
• Worker RSS after a compile. The proposed revised budget is in §8.2. An alternative is to recycle a worker from the PE cache after a compile burst (blue/green); that is not built.
• Windows kill-both-shards: about 2.0 s, at the budget line.

#790 — load timeout from compile start.

• The worker sends LoadStarted when a compile thread picks a load up. The host arms that load's timer then, instead of each caller timing from when it sent the load.
• A queued load is never blamed. A compile still running LoadTimeout after its start kills the worker and strikes that key only (LoadEntry.TimedOut).
• Hello.compile_threads is a test knob; 0 keeps the worker's default.
• Tests:
  • Compiles_queued_behind_others_longer_than_the_load_timeout_are_not_killed_or_struck: one compile thread, and a burst grown until it outlasts three timeouts (asserted). Red with the timer armed at send.
  • A_compile_still_running_a_load_timeout_after_it_started_is_killed_and_struck: the positive control, with two kills leading to quarantine and the healthy key unblamed. Red with the timer never armed.

#788 — start-failure logging.

• The first failed start of a down period is one Error with the exception.
• Later retries are a Warning without a stack, carrying the attempt count, at most one per 60 s per shard; the rest log at Debug.
• The start that ends the outage is one Information line: started … after N failed start attempt(s) over T.
• Tests: A_shard_that_cannot_start_logs_one_error_then_concise_rate_limited_retries_then_its_recovery, with two intervals. Red when every failure is logged at Error with the exception.

12.3 Fourth P1 slice: batch submit, exact per-item deadlines, the scripted-alarm seam (2026-11-10)

Batch submit (§4.5, P0).

• IScriptRunner.RunBatch(invocations) returns one task per item. Results are per item and streamed.
  • The default implementation is RunAsync per item, started together. InProcessScriptRunner uses it.
  • OutOfProcessScriptRunner sends every item that is ready to run as ONE EvalBatch frame per worker. Ready means admitted, not quarantined, not a suspect, and already loaded in a live worker at this source and slot order.
  • Anything else goes through the full RunAsync path on its own: a first evaluation that needs a compile, a worker that is down, or a suspect bound for the one-at-a-time isolation worker. A batch therefore never waits for a load or a restart as a whole.
• Seams that use it, on the out-of-process path only:
  • Calculation pump. It dequeues everything pending (each entry's dependency snapshot was taken at enqueue), submits one batch, and publishes in dequeue order. A tag queued twice ends the batch, so every queued evaluation still runs, in order, against the snapshot that triggered it. The published sequence is the one-at-a-time pump's.
  • VirtualTag seam. There is one actor per tag, so an upstream burst makes many tags ready at once from different threads. ScriptBatchCoalescer hands those to one RunBatch. It has no timer: an evaluation on an idle seam is submitted at once by its own caller.
  • Scripted alarms. See below.
• In process, the Calc pump and the alarm engine stay one at a time, unchanged. The VirtualTag in-process path does not touch the coalescer.

Per-item deadlines and blame stay exact under batching.

• The host arms T and T + G at submit. That is when the run starts, provided the worker has an idle evaluation thread.
• A batch larger than the worker's scheduler, or work queued behind a runaway, used to wait in the worker with its clock already running. It could then be timed out, take timeout strikes, and even be the key blamed at T + G.
• The worker's scheduler now hands an evaluation straight to an idle thread, or else puts it in a backlog and reports it in EvalResults.deferred_eval_ids.
  • The host stops that evaluation's clock.
  • When a thread picks the evaluation up, its id rides back in started_eval_ids on the result frame of the run that thread just finished. The host then restarts the full T, then G, from the start.
  • The two notices are unordered on the wire, and the host handles either order.
  • The deadlines are absolute timestamps that the timer re-checks, so a stale tick never fires early.
• A run queued behind a runaway is therefore neither timed out nor struck. At the kill it is a victim (script worker restarted, BadResourceUnavailable, re-evaluated on return), not the culprit.
• Tests:
  • A_runaway_in_a_batch_is_blamed_alone_and_siblings_queued_behind_it_are_not_timed_out_or_struck runs one evaluation thread and a batch of a runaway plus three siblings. It is red with the host ignoring the deferral, and red with the worker never deferring.
  • A_deferred_run_is_timed_and_killed_from_when_it_starts is the positive control. It is red with a start notice that does not re-arm the clock.
• This also changes plain RunAsync under saturation: a run that waits for a thread no longer times out on its wait. That matches the in-process guard, whose budget clock starts when the pool picks the run up (§1.2).

Scripted-alarm seam (★10: alarms last).

• Scripting:Runner:Seams unset now means VirtualTag + Calculation. ScriptedAlarm runs out of process only when it is listed.
• ScriptRunnerSelection serves it. The Runtime's DriverHostActor gets it through the new IScriptRunnerProvider and passes it to the engine (a new last parameter, scriptedAlarmRunner).
• Engine changes, all on the runner path only:
  • LoadAsync compiles each predicate to PE bytes only (CompileToImage). A broken predicate still fails the load exactly as in process, and no assembly is loaded into the host. Deviation: Roslyn still runs in the host at load for alarms; the worker compiles again for execution.
  • Startup recovery and every re-evaluation pass snapshot each dirty alarm's inputs under _evalGate, submit all their predicates as one batch, and then apply the results in the serial order under the gate: Part 9 transition, persist before update, emissions. A test shows the transition sequence is identical to an in-process engine's for the same pushes.
• Failure semantics (§9, alarm column):
  • Timed out, killed, refused or quarantined: the condition is frozen with a BadInternalError annotation, as today.
  • The runner's fault (worker restarted, none ready): frozen with BadResourceUnavailable, consistent with VT/Calc. The alarm is re-evaluated by itself when the runner's Recovery completes.
  • A predicate that threw keeps its state with no annotation, as in process.
  • No outcome but a completed bool reaches the state machine, so a dead worker never clears an alarm.
• Real-worker tests:
  • A_runaway_predicate_is_killed_and_quarantined_its_alarm_frozen_Bad_and_a_sibling_keeps_evaluating;
  • A_worker_that_dies_under_a_predicate_freezes_it_BadResourceUnavailable_never_clears_it_and_it_is_re_evaluated_on_return. It is red when the runner's failure is read as false.
• Budget note: since lmxopcua#779 the host passes Scripting:RunTimeout (2 s) to the engine. The passes below are still judged against the stricter 250 ms the design was written for.

Measured (§8 harness, Batch_throughput_and_alarm_pass_latency; macOS Arm64, 14 CPUs, load average about 14 from other work on the box, 10 s phases):

Batch size	RunBatch evals/s	RunAsync ×n evals/s	RunBatch p50 / p99 µs	RunAsync ×n p50 / p99 µs
1	38,296	40,980	24 / 41	23 / 35
10	147,035	118,516	57 / 121	70 / 142
100	369,304	232,967	215 / 1,095	363 / 1,341
500	463,033	256,232	818 / 2,541	1,701 / 3,178

(OutOfProcess K = 2, one batch at a time. The in-process rows show no difference between the two, as expected.)

Alarms per pass	InProcess p50 / p99 ms	OutOfProcess batched p50 / p99 ms	OutOfProcess serial p50 / p99 ms
100	0.4 / 1.5	0.4 / 2.1	3.6 / 4.9
500	2.2 / 4.7	1.7 / 4.8	16.1 / 18.0
1,000	4.4 / 7.7	2.5 / 41.0	28.1 / 33.4

• A pass is push → the last of the N transitions it emits. "Serial" replays the pre-batch one-predicate-at-a-time pass over the same runner. That is the N × round-trip cost §8.2 warned about: about 30 µs per alarm, against 2–4 µs batched.
• Every configuration is far inside 250 ms. The one 41 ms p99 is a single loaded-box outlier.
• The §8.2 harness rerun on the same box after the scheduler change:
  • K = 2 closed loop ×64: 366k evals/s;
  • paced 10k/s: +5 % host CPU vs InProcess (budget ≤ 25 %);
  • idle p50: 25 µs;
  • kill latency: 14–19 ms;
  • warm-cache restart of 1,000 scripts: 220 ms;
  • every result correct.

windev (Windows VM, 36 CPUs, .NET 10.0.8; the full ScriptWorker.Tests suite also passes there, 157/157):

Batch size	RunBatch evals/s	RunAsync ×n evals/s	RunBatch host CPU	RunAsync ×n host CPU
1	1,810	1,921	63 %	65 %
10	13,013	11,220	109 %	157 %
100	48,685	40,699	129 %	224 %
500	86,322	83,116	143 %	289 %

Alarms per pass	InProcess p50 / p99 ms	OutOfProcess batched p50 / p99 ms	OutOfProcess serial p50 / p99 ms
100	8.2 / 39.0	2.9 / 5.1	51.7 / 64.8
500	42.3 / 53.3	7.4 / 35.7	236.5 / 273.3 (MISS)
1,000	36.6 / 84.2	11.7 / 52.7	466.8 / 519.4 (MISS)

• On the VM, where one round trip costs about 0.5 ms, the pre-batch serial pass misses the 250 ms budget from about 500 alarms. The batched pass stays at p99 ≤ 53 ms for 1,000 alarms, below the in-process engine's own figures. That is the measurement behind §8's "batch submit is a prerequisite, not an optimisation", and the reason ★10 put alarms last.
• On Windows, batching halves the host CPU of a 100–500 evaluation burst. That is the overlapped-I/O path that is still #789's main miss.
• No predicate failed to complete in any pass on either host.
• Rerun of the §8.2 harness on windev:
  • K = 2 closed loop ×64: 50k evals/s, gate PASS;
  • paced 10k/s: +99 % host CPU vs InProcess (#789, unchanged in kind);
  • kill latency: 9–19 ms;
  • warm-cache restart of 1,000 scripts: 1.8 s.

12.4 P2 record: Shadow mode, and the docker-dev drill with alarms (2026-11-10)

What Scripting:Runner=Shadow does (as built).

• The seams selected by Scripting:Runner:Seams (same rule as OutOfProcess: unset = VirtualTag + Calculation, ScriptedAlarm only when listed) receive a ShadowScriptRunner (Core.Scripting) as their runner. They recognise it as an IScriptShadow and keep their in-process path byte for byte: no VT coalescer, the one-at-a-time Calc pump, the in-process alarm engine. After each compiled evaluation they call Mirror(invocation, publishedOutcome). The passthrough fast path and host-lifecycle failures (evaluator disposed, shutdown cancel) are not mirrored.
• Mirror copies the inputs (the alarm engine reuses its read cache), drops the script logger (the in-process run already wrote its lines) and queues the shadow run on the pool. It never blocks and never throws.
• Comparison: value (structural for arrays), exact CLR type, status, failure class (compile; budget = timed out / killed / refused / quarantined, which the two runners reach by different routes for the same runaway; threw with its message; unsupported_type). A shadow outcome that is the runner's fault (Unavailable, worker busy) is not compared (shadow_unavailable). ★11: a value the protocol cannot carry is a mismatch with reason unsupported_type.
• Bounded: Scripting:Shadow:MaxInFlight (256) outstanding and one per script; the rest are dropped and counted (behind | key_busy). One per script also keeps the out-of-process single-flight rule from refusing a shadow run and reporting a false mismatch.
• Observability: otopcua.script.shadow.compared{kind, result}, .mismatch{kind, reason, script}, .dropped{kind, reason}, .inflight. One Warning per script and reason, Script shadow mismatch: <kind> <key> (<reason>) — published …; shadow …, re-armed at the seam's deploy boundary and capped at Scripting:Shadow:MismatchLogsPerMinute (20).
• Readiness: ScriptRunnerSelection.Status() is null under Shadow; the script-runner entry is always Healthy and carries the shadow pool's state and the counters as data. The shadow pool's kill/quarantine lines go to the host log with ScriptRunnerRole=Shadow (the console template does not render that property), never to the script log.
• Deviation from the §12 sketch ("mismatching keys are logged once each"): once per key and reason, plus the per-minute cap.

Tests (all mutation-checked; 16 mutants, all red after one survivor got an assertion):

• ShadowScriptRunnerTests (Core.Scripting, fakes): the comparison table; a divergent script counted per script in the metric with one diff-summary line; the rate limit and suppressed-count note; reset re-arming; a shadow runner that hangs, blocks its caller or throws never reaches the caller and is never a mismatch; MaxInFlight drop accounting; inputs copied before Mirror returns.
• ShadowModeTests (real worker): eleven scripts over short / double / string / arrays / DateTime / bool / status / a throw publish exactly what InProcess publishes and match; a deliberately divergent script (it does a culture-sensitive string.Compare("a", "B"): the worker runs InvariantGlobalization so it compares ordinally and answers differently; originally it read AppContext.BaseDirectory, which lmxopcua#792 then denied) and a decimal result are counted against exactly those two scripts (value, unsupported_type). A runaway that only runs away in the worker is killed twice and quarantined there while every published value stays the in-process one, no published evaluation waits more than 250 ms, and its sibling on the killed shard (a victim, not compared) has no mismatch. Alarms: the emission sequence equals an in-process engine's across a kill -9 of the shadow worker, and no shadow failure annotates a condition.
• Host: selection (Shadow hands out the shadow, Status() null), binding/clamps, the health matrix (Healthy whatever the shadow pool's state), the VT seam's mirror.
• Suites (macOS, Release, 0 warnings, AKKA_TEST_TIMEFACTOR=3): Core.Scripting 255/1 skip, ScriptedAlarms 109, VirtualTags 16, Calculation 84, ScriptWorker 160/4 skip, Host 275, Runtime 1002/3 skip, OpcUaServer.IntegrationTests 133. windev (Windows VM, scratch dir under C:\build): ScriptWorker suite 160 passed / 4 skipped, 0 warnings. The first windev run caught a test bug (a DateTime seconds argument past 59 once the slower VM looped longer), fixed before the commit.
• Found while writing them: scripts could reach System.AppContext (SetSwitch / SetData / GetData) → lmxopcua#792, fixed: System.AppContext, System.Console, GCSettings, ProfileOptimization, EncodingProvider, TypeDescriptor, CryptoConfig and the ambient CultureInfo / TimeZoneInfo.ClearCachedData members are now denied by ForbiddenTypeAnalyzer (in-process and in the worker, one analyzer).

docker-dev drill (image otopcua-host:w34-shadow built from this branch; central-1/central-2 only; drill rows w34-* inserted by SQL: VTs w34mul (HR5×3), w34raw (var v = …HR5.Value; return v; — compiled, returns the boxed ushort), w34div (divergent), w34dec (* 1.5m), Calc W34Calc (HR5+1000), alarms w34-al-hi (HR5>50), w34-al-slow (1.5 s spin, then HR5>50), w34-al-run (while(true) when HR5 == 77). HR5 driven by Modbus writes to the fixture.)

(a) Shadow, all three seams (docker-compose.script-runner-shadow.yml):

Step	Observed (both nodes)
HR5 = 21 / 60 / 30	w34mul 63 / 180 / 90, w34raw 21 / 60 / 30, W34Calc 1021 / 1060 / 1030, HR5x2 42 / 120 / 60, all Good; w34-al-hi and w34-al-slow Active at 60 and Inactive at 30
Mismatch counters (leader)	compared{virtual_tag,match} 146, {calculation,match} 11, {alarm,match} 14; mismatch{virtual_tag,value,w34-vt-div} 5 and {virtual_tag,unsupported_type,w34-vt-dec} 4 — nothing else. dropped{key_busy} 5 (startup burst)
Log	exactly two Warnings per node: w34-vt-div (value) — published Ok Int32 1; shadow Ok Int32 2 and w34-vt-dec (unsupported_type) — published Ok Decimal 7.5; shadow Failed [unsupported_type] … System.Decimal
kill -9 both shadow workers on the leader, then HR5 = 61	Published values all correct and Good (183 / 61 / 1061 / 122); workers restarted within ms; /health/ready 200, script-runner Healthy

• [A-7] answered for Modbus: a UInt16 register arrives boxed as ushort and crosses the boundary unchanged (w34raw matched on every evaluation).
• [A-8] answered: in process, a VT that returns a decimal publishes BadNotSupported (0x803D0000) with a null value: the sink rejects it. Out of process it would publish BadInternalError "unsupported type". Either way it is Bad, so the ★11 behaviour change is only the status code.

(b) OutOfProcess with ScriptedAlarm listed (docker-compose.script-runner.yml + Seams__2=ScriptedAlarm):

Step	Observed (both nodes)
HR5 = 60, then 30	w34-al-hi / w34-al-slow Active, then Inactive, Quality Good; otopcua_script_ipc_roundtrip{kind="alarm"} counting, so the predicates ran in the worker
HR5 = 77 (runaway), 76, 77	exceeded its hard deadline (4000 ms); killing ScriptWorker[0] twice, then quarantined: "it exceeded its hard deadline and its worker was killed (2 time(s) in the last hour)…". w34-al-run stays Inactive with Quality BadInternalError; w34-al-hi and w34-al-slow (same shard) stay Active and Good throughout
HR5 = 30, kill -9 the shard-0 worker 0.6 s later, during w34-al-slow's 1.5 s spin	ScriptWorker[0] ended (Crash) with no deadline crossed; 1 script(s) in flight move to the isolation worker: 3:w34-al-slow. Read at once: w34-al-slow Active, Quality BadResourceUnavailable (0x80040000): frozen, not cleared. By the next read, 19 s later, it is Inactive and Good, re-evaluated in the isolation worker with no further input change. w34-al-hi cleared normally; w34-al-run stays BadInternalError (quarantined)
/health/ready	200, script-runner Healthy ("ready") after the restarts

(The kill loop stopped after one worker per node because its own shell matched the pattern; the killed worker was the one running the in-flight predicate, which is what the step needed.)

Rig state afterwards.

• w34-* rows deleted (Script 3, VirtualTag 1, ScriptedAlarm 1, Tag 55, as before), HR5 = 5, redeployed: revision e221b179…, identical to the pre-drill deployment.
• central-1/central-2 recreated from the base compose file in the same compose project: image otopcua-host:dev (24f9d7287ed2, never re-tagged), no Scripting__* env, compose config hash equal to the base file's. The secret stores live on the otopcua-localdb-central-* volumes since an earlier change, so they survived the recreates: central-1's store is byte-identical; central-2's main file differs only by a WAL checkpoint (same size). Both log Mesh PSK secret ot-mesh-psk-* … is absent at startup — central-1 with an unchanged store logs it too, so that is the state the rig was already in (most likely from the earlier drill's store loss), not something this drill did.
• The drill image otopcua-host:w34-shadow was removed afterwards. The site nodes, sql and traefik were not touched.

12.5 Installer record: worker identity, pipe ACL, firewall, PeCacheDir ACL (2026-11-10)

Owner decisions §14 #5 and #6, as built. Operator reference: docs/ServiceHosting.md §"Worker isolation on Windows".

• Identity, deviating in mechanism from "run as a virtual account". Windows starts a process as NT SERVICE\<name> only when the SCM starts that service. A host cannot launch K + 1 children that way, each with its own nonce, pipe and Job Object. A separate local user would need a stored password plus SeAssignPrimaryTokenPrivilege on the host account. Scripting:Worker:Identity (default unset; Windows only; startup refuses it elsewhere or when malformed) therefore starts each worker under a restricted token derived from the host's (RestrictedWorkerLauncher):

  • DISABLE_MAX_PRIVILEGE, so only SeChangeNotifyPrivilege is left;
  • restricting SIDs = the virtual account's SID (computed from the name, S-1-5-80-…) + Everyone + Users + RESTRICTED;
  • token and default DACL = host + SYSTEM + identity + RESTRICTED;
  • created suspended and put in its Job Object before it runs, inheriting exactly its two stdio pipes;
  • TEMP set to its ACL'd work directory, and DOTNET_EnableDiagnostics=0.

  The installer registers a disabled anchor service, so the account exists and resolves by name.

• Found on windev. The runtime would not start under the restricted token (coreclr.dll → ERROR_DLL_INIT_FAILED). A bisection over the token's groups showed that only the user or Administrators SID fixed it, not the logon SID. The cause was USER32's window station and desktop connection, so the host now grants the identity on its own window station and desktop. The logon SID stays out of the restricting set because the host's process DACL gives it PROCESS_VM_READ. The token's own DACL also needed rewriting (the handle needs WRITE_DAC).

• Pipe ACL. On Windows the pipe always has an explicit, protected DACL: the host account and the identity, read/write, nothing else. .NET's CurrentUserOnly grants the token owner, which is Administrators on an elevated host; this ACL does not. The owner is still the token owner, so the worker's CurrentUserOnly client check is unchanged.

• Installer scripts/install/Install-ScriptWorkerIsolation.ps1 + ScriptWorkerIsolation.psm1. It is idempotent, with -WhatIf, -Uninstall and -PlanOnly, and it sets up:

  • the anchor service;
  • PeCacheDir: protected, SYSTEM / Administrators / host full control, no Users, and the key and markers host-only;
  • pe\: the identity gets Modify;
  • script-worker\: the identity gets read+execute;
  • deny ReadData for the identity on the install folder (appsettings, logs, host binaries);
  • outbound block firewall rule OtOpcUa-ScriptWorker-BlockOutbound on the worker apphost;
  • optional host-service env (-ConfigureHostService).

  The runner flip itself is not part of it.

• Tests.

  • scripts/install/tests/Test-ScriptWorkerIsolation.ps1 is the dry-run harness. It runs on any OS and asserts ids, order, targets, SDDL and the rollback (46 assertions). A mutation check (deny → allow, protected → inherited) turns 4 red.
  • ScriptWorkerIdentityTests (Core.Scripting) pin the SID algorithm against NT SERVICE\TrustedInstaller. The PowerShell harness pins the same default SID.
  • WorkerIsolationTests (ScriptWorker) cover the pipe ACL (built and live), the token, and a real worker under the identity running a script with a restricted token. A plain worker's token is unrestricted (positive control).
    • The token case is restricted, privilege-free, and impersonated: it reads System32 and writes its work dir, but cannot read or write the host's temp.
    • They ran green on windev without changing machine state, apart from the session's window station / desktop ACE. No account, firewall rule or persistent ACL was created there.

• Not verified live: the installer on a real node. That is the QA deploy gate on lmxopcua#780's follow-up (see that issue for the checklist).

---

13. Test plan

Unit (Core.Scripting.Tests, new ScriptWorker.Tests):

• Frame codec:
  • round-trip every Variant.TypeTag (arrays included);
  • reject zero-length, oversize and truncated frames;
  • reject an eval_id with no outstanding request;
  • fuzz the reader with random bytes and assert there is no exception other than a protocol fault.
• Unboxing parity: for each tag, a script doing (T)ctx.GetTag("x").Value behaves identically in and out of process. This pins [A-7].
• Attribution ledger:
  • a deadline kill blames exactly one key;
  • a crash with 3 in flight → 3 suspects → isolation;
  • an isolation crash → one strike;
  • probation returns a suspect to its shard;
  • quarantine after 2 kills;
  • a source change lifts the quarantine;
  • a redeploy lifts every quarantine.
• Backoff schedule, jitter bounds, the reset-after-healthy rule and the load-phase crash bisection.
• Options binding: clamps plus a Warning per clamped key; PeCacheDir relative or inside the content root is a startup failure.

Containment (worker-backed, Category=ScriptWorker; runs in CI on Linux, gated on windev for Windows). Each case has a positive control: a healthy sibling script on the same shard that keeps producing correct values before and after the event. Following scadaproj memory "assert the artifact before the outcome", each assertion is checked to be capable of failing.

Case	Assert
while(true){} ignoring the token	Killed within T + G + 100 ms; host ThreadPool.ThreadCount and Process.Threads unchanged; no orphan (InFlightCount == 0 after the kill). Mutation check: disable the kill and see the test fail
while(!ctx.CancellationToken.IsCancellationRequested){}	Ends at T by itself; no kill; no strike beyond the timeout
Allocation bomb	Worker dies of OOM or the heap limit; host RSS stays below a bound and the host stays up; suspect → isolation → quarantine
Unbounded recursion	Worker dies; host up; quarantined after the isolation reproduction. Also the first in-repo reproduction of [A-1] for the in-process mode, marked Skip there because it would kill the test host
Worker killed externally (kill -9)	Victims Bad with "script worker restarted", then recover with the latest inputs; the alarm condition is not cleared
Host killed	Worker exits within 2 s (pipe EOF on Linux, Job Object on Windows)
Environment	The worker's environment contains no ServerHistorian__ApiKey / ConfigDb variables (set them in the test host first: positive control)
Pipe	A second client connecting to the pipe is refused; a wrong nonce is refused
Loaded assemblies	The worker's loaded set equals the pinned list; Akka / SqlClient / Opc.Ua are absent
Crash loop	A poison generation trips the circuit; state Unavailable; keys Bad with the runner-unavailable reason; recovery after a fixed redeploy

Parity: run the existing RoslynVirtualTagEvaluatorTests, CalculationDriver*Tests and ScriptedAlarmEngine*Tests against both runners. Use xunit theory or class fixtures over IScriptRunner, with the worker built as a test dependency.

Performance (Category=Perf, opt-in, not in CI): a Stopwatch-based harness (no new packages) that measures the §8 table: p50/p99 added latency, throughput at K=1/2/4, host CPU, restart-to-first-result with 1,000 cached PEs, and RSS per shard.

Live gates:

• docker-dev rig: a deploy with VTs, Calc tags and scripted alarms; drill a runaway script through the AdminUI and watch /uns go Bad, the quarantine appear, and siblings stay Good;
• QA shadow week (P2);
• a QA flip with the same drill.

---

14. Open decisions for the owner

Each has a recommendation, marked ★.

1. Pool shape.
   • ★ K=2 shards plus an on-demand isolation worker;
   • one worker per node, simplest, but every kill blips every script;
   • per-seam workers.
2. Hard grace G.
   • ★ max(1 s, T), which keeps cooperative cancellation useful;
   • G = 0 (kill at T): strictest, but every soft overrun restarts a shard.
3. Status code for "runner unavailable / worker restarted".
   • ★ BadResourceUnavailable, which separates a sick node from a broken script but is a ScadaBridge-visible contract change;
   • keep BadInternalError everywhere, for zero consumer impact.
4. Fallback when the worker cannot start.
   • ★ fail closed: Bad, and no in-process execution;
   • fail open to in-process with an Error, which keeps values flowing but silently reopens the risk.
5. Worker identity on Windows.
   • ★ a dedicated low-privilege virtual account created by the installer, which needs installer work, an explicit pipe ACL, and a deploy that touches the box;
   • the same account as the host: zero install change, and the Job Object plus the analyzer remain the controls.
6. Network isolation.
   • ★ a Windows Firewall outbound-block rule on the worker executable (installer), with the analyzer as the only control on Linux;
   • analyzer only, everywhere.
7. Compile location.
   • ★ in the worker: Roslyn blowups are contained and the host gets lighter;
   • in the host, shipping PE bytes: a lighter worker and faster cold start.
8. PE cache store.
   • ★ a node-local directory with an HMAC;
   • LocalDb (already a dependency; it would also replicate to the pair, which is unnecessary);
   • no persistence, recompiling on restart (violates the 2 s budget at scale).
9. IPC encoding.
   • ★ protobuf (already compiled in Commons, and mxaccessgw precedent);
   • MessagePack (pinned but unused, so in practice a new dependency).
10. Scripted alarms in the first flip.
    • ★ last, after Calc and VT, because of the 250 ms budget and the engine refactor;
    • all seams at once.
11. Unsupported input/result CLR types.
    • ★ an explicit Bad UNSUPPORTED_TYPE, surfaced in shadow mode first;
    • best-effort conversion (to string, double), which hides type bugs.
12. Readiness when the circuit is open.
    • ★ degraded but ready (OPC UA still serves driver data);
    • not ready, which on a redundant pair would shift ServiceLevel to the peer.
13. Long-term status of InProcess.
    • ★ keep it as the dev/test default and the rollback;
    • delete it after a soak period.
14. Does this need a real-deployment scale target? [A-14]'s 5k evals/s/node is a guess. The owner should supply the expected VT / Calc / alarm counts per node for the target sites, or accept the guess as the gate.

---

15. Assumptions (not verified from code)

Id	Assumption	How to verify
A-1	Unbounded recursion in a script throws StackOverflowException and kills the host today	Reproduce in a throwaway process (not the test host)
A-2	A collectible ALC whose code is still executing on an orphan thread cannot be unloaded	.NET docs / a WeakReference test with a spinning orphan
A-3	A minimal .NET 10 worker idles at tens of MB RSS	Measure the P1 worker
A-4	P = max(2, cores/K) evaluation threads per worker is enough	P1 perf harness
A-5	Passthrough scripts are a large share of evaluations on real deployments	Count PassthroughScript.TryMatch hits against compiled runs on QA (add a counter in P0)
A-6	A local pipe round trip costs tens of µs	P1 perf harness
A-7	Drivers box tag values with stable, enumerable CLR primitive types	Shadow-mode type census
A-8	What OpcUaPublishActor / the sink does with an unsupported result object today	Trace AttributeValueUpdate → SdkAddressSpaceSink
A-9	Drivers produce UTC DateTime values	Shadow-mode census on DateTime.Kind
A-10	512 MiB is a sane per-worker heap limit (ScadaBridge uses the same)	Size against measured RSS × script count
A-11	docker-dev containers cannot create nested cgroups	Check the compose capabilities
A-12	Warm Roslyn compile is tens of ms per script; cold is seconds	P1 perf harness
A-13	Worker start is ~150 ms; a PE load into a collectible ALC is ~1 ms	P1 perf harness
A-14	5k compiled evals/s/node is a fair planning envelope	Owner input (decision 14)
A-15	Today's in-process overhead per compiled run is µs to tens of µs	Add the same harness to InProcessScriptRunner in P0 for a baseline

---

16. Prior art consulted

• ScadaBridge CentralUI.SandboxWorker (~/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.CentralUI/ScriptAnalysis/SandboxWorkerProtocol.cs). It runs one disposable process per test run over stdio, using 4-byte LE length-prefixed JSON frames with a 1 MiB cap, admission 4, DOTNET_GCHeapHardLimit 512 MiB, and Kill(entireProcessTree) on the deadline. Taken from it: the heap limit, the tree kill and the frame cap. Not taken:

  • stdio (fragile for a long-lived worker);
  • per-run process spawn (too slow for runtime evaluation);
  • inheriting the parent environment;
  • proxying host calls.

  ScadaBridge's runtime scripts are still in-process (dedicated scheduler plus watchdog, docs/plans/2026-08-15-script-pool-split-design.md), so this design would be the family's first out-of-process runtime evaluator.

• mxaccessgw gateway↔worker (~/Desktop/MxAccessGateway/docs/WorkerFrameProtocol.md, WorkerProcessLauncher.md, WorkerBootstrap.md). It runs a long-lived supervised worker over a named pipe: protobuf envelopes, a negotiated frame cap, a priority-drained writer with coalesced flushes, sequence numbers, a nonce in an env var (never on the command line), startup probe plus retry, and a tree kill on a failed start. Taken from it: the frame protocol shape, the nonce handshake and the launcher discipline. Avoided: an orphan terminator that is not scoped to its own instance.