lmxopcua/docs/plans/2026-11-09-out-of-process-script-runner-design.md

Out-of-process, pre-emptible script runner — design

Date: 2026-11-09 · Status: proposed (design only — no code in this change) · Issue: #750 · Related: #734 (Lane E remediation)

This document designs a script runner that can actually stop a user script, instead of abandoning it. It covers the three script seams that run operator-authored C# on a driver node: virtual tags, Calculation tags and scripted-alarm predicates.

Claims about today's code carry file:line evidence against origin/master @ 94926eb0. Anything not read from the code is marked [A-n] and listed in §15.

---

1. Today: in-process execution

1.1 One compile pipeline, three seams

All three seams share Core.Scripting:

• Compile. ScriptEvaluator<TContext,TResult>.Compile wraps the user source in a synthesized CompiledScript.Run(ScriptGlobals<TContext>), compiles it with Roslyn against the ScriptSandbox reference set, runs ForbiddenTypeAnalyzer, emits PE bytes to a MemoryStream and loads them into a collectible ALC (src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ScriptEvaluator.cs:125 emit, :139 LoadFromStream).

• Cache. CompiledScriptCache is keyed by SHA-256 of the source (CompiledScriptCache.cs:157). It is in memory only, so every process start recompiles every script on first use.

• Execution. It is synchronous: ScriptEvaluator.RunAsync invokes the bound delegate inline and returns Task.FromResult (ScriptEvaluator.cs:178-195). The script-visible ctx.CancellationToken is an AsyncLocal set for the duration of the call (ScriptEvaluator.cs:191, Core.Scripting.Abstractions/ScriptContext.cs:96-98).

• The context is a pure read cache. VirtualTagContext.GetTag and AlarmPredicateContext.GetTag serve only from a dictionary handed in by the caller (VirtualTagContext.cs:50, AlarmPredicateContext.cs:40). SetVirtualTag is dropped in both production single-tag evaluators (Host/Engines/RoslynVirtualTagEvaluator.cs:129-131, Driver.Calculation/CalculationEvaluator.cs:185-188) and throws in alarm predicates (AlarmPredicateContext.cs:50). Inputs are known statically, because DependencyExtractor rejects non-literal GetTag paths at publish.

  This matters most for the design (§4.1). A script needs nothing from the host while it runs: the inputs go in, one value comes out, and log lines are a side channel.

Seam	Where it runs	Budget	Serialisation	Evidence
Virtual tag	VirtualTagActor message handler calls IVirtualTagEvaluator.Evaluate synchronously on the actor's dispatcher thread; RoslynVirtualTagEvaluator blocks with .GetAwaiter().GetResult()	2 s	per actor (one actor per VT)	Runtime/VirtualTags/VirtualTagActor.cs:276; RoslynVirtualTagEvaluator.cs:70, :174-175; singleton Host/Program.cs:393-398
Calculation tag	async pump in CalculationDriver, one evaluation at a time per driver instance	2 s (runTimeoutMs)	_evalGate = SemaphoreSlim(1,1) per driver	CalculationDriver.cs:99, :523-556, :628; CalculationDriverOptions.cs:13; CalculationEvaluator.cs:232-233
Scripted alarm	ScriptedAlarmEngine.ReevaluateAsync walks the dirty set serially under one gate	250 ms (the engine is built without scriptTimeout, so it takes TimedScriptEvaluator.DefaultTimeout)	_evalGate per engine	ScriptedAlarmEngine.cs:204, :257, :743-765, :833-835; Runtime/Drivers/DriverHostActor.cs:1251-1252

Both value seams skip Roslyn for the mirror shape return ctx.GetTag("X").Value; (PassthroughScript.TryMatch, RoslynVirtualTagEvaluator.cs:111-118, CalculationEvaluator.cs:167-176).

1.2 Timeout and cancellation model

Every compiled run in production goes through ScriptRunGuard.RunAsync (Core.Scripting/ScriptRunGuard.cs:170-304). There are three guard instances per seam owner: one per RoslynVirtualTagEvaluator (RoslynVirtualTagEvaluator.cs:71), one per CalculationEvaluator (CalculationEvaluator.cs:88) and one per ScriptedAlarmEngine (ScriptedAlarmEngine.cs:258).

1. Single flight per key. A CAS on state.InFlight refuses a second run while the previous one, an orphan included, still holds the slot. The caller gets ScriptRunRefusedException → Bad "previous evaluation still running" (ScriptRunGuard.cs:188-219).

2. The run is pushed to the pool (Task.Run, :231-249), so a CPU-bound script cannot run inline and complete before the timeout is registered. The budget clock starts only when the pool picks the run up (:268).

3. The wait is runTask.WaitAsync(timeout, ct) (:273). On timeout the guard cancels the run token (:279). Only a script that polls ctx.CancellationToken can see that.

4. Quarantine happens on any of three triggers:

   • N consecutive timeouts (:332-341, default 3);
   • a refusal behind an orphan older than StuckOrphanQuarantineFactor × the timeout (:199-216, default 10×);
   • an allocation breach, where GC.GetAllocatedBytesForCurrentThread is sampled before and after the delegate (:235, :241, :262-263, :296-302; default 64 MB, ScriptingOptions.cs).

   A quarantine lifts on a source change or a redeploy (ResetAll, called from ClearCompiledScripts at RoslynVirtualTagEvaluator.cs:212-218 and from ScriptedAlarmEngine.LoadAsync).

5. What the operator sees.

   • A VT or Calc failure publishes BadInternalError with the last-known value, once per Good→Bad transition (VirtualTagActor.cs:384-392, CalculationDriver.cs:640-647).
   • An alarm whose predicate did not complete keeps its frozen condition state, annotated BadInternalError (ScriptedAlarmEngine.cs:895-907).

TimedScriptEvaluator (TimedScriptEvaluator.cs:80-83) is the older wrapper. Its own remarks say it is now used only by single-shot callers (:18-27).

1.3 Why a runaway script cannot be pre-empted

.NET has no supported way to stop a managed thread. Thread.Abort is gone in .NET Core, and the guard's WaitAsync returns control to the caller; it does not stop the run. Concretely:

Failure	What happens today	Evidence
while(true){} (ignores the token)	The pool thread spins forever, holding one core and its collectible ALC. The guard bounds this to one orphan per key and then quarantines, but the thread keeps spinning until the process restarts. N such scripts peg N cores.	ScriptRunGuard.cs:17-22 remarks; ScriptRunGuardTests.cs:170-205 asserts exactly one orphan stays in flight (InFlightCount.ShouldBe(1))
Fast allocation (new byte[int.MaxValue] in a loop, or a growing list)	The sample is taken after the delegate returns, so the process can OOM first. The whole server dies, including OPC UA sessions and every driver.	ScriptRunGuard.cs:39-43; docs/VirtualTags.md:40; review docs/reviews/2026-10-21/05-core.md:100-105
Unbounded recursion (a local function or a self-referencing Func<>; neither is a class member, so EnforceSingleRunMember does not see it)	StackOverflowException cannot be caught in .NET and terminates the host process. [A-1] I did not reproduce this; it follows from runtime semantics, and no analyzer rule targets recursion (git grep -i "recursion|StackOverflow" in Core.Scripting finds nothing). Update (lmxopcua#779): reproduced out of process (exit 134, "Stack overflow.") and fixed in-process: the compile injects ScriptStackGuard.Check() into every script body, so the script now fails with a catchable ScriptStackExhaustedException.	ScriptEvaluator.cs:228-281 (member check only)
Collectible-ALC reclaim of a stuck script	ClearCompiledScripts disposes the evaluator, but an ALC cannot be collected while a thread is executing its code, so a stuck orphan pins its assembly until restart. [A-2] This is .NET ALC semantics, not observed here.	CompiledScriptCache.Clear; ScriptEvaluator.Dispose
A slow (not runaway) script	VT: blocks a default-dispatcher thread for up to 2 s per evaluation. Calc / alarm: holds the driver-wide or engine-wide gate, so every sibling waits behind it (up to 2 s / 250 ms each).	RoslynVirtualTagEvaluator.cs:174-175; CalculationDriver.cs:539-541; ScriptedAlarmEngine.cs:743-765

The sandbox (ForbiddenTypeAnalyzer.cs:62-80: System.IO, System.Net, System.Diagnostics, System.Reflection, System.Threading[.Tasks], Environment, GC, …) blocks capabilities, and it does that well. It cannot block consumption (CPU, memory, stack). Every review that looked at this reached the same conclusion:

• docs/VirtualTags.md:40,54;
• docs/ScriptEditor.md §"Runtime safety";
• docs/superpowers/plans/2026-10-22-review-remediation.md Lane E2;
• docs/reviews/2026-10-21/05-core.md:103 and 03-host-controlplane-security-config.md:38.

All of them say the fix is an out-of-process runner.

Script authoring is gated by the ConfigEditor policy, which admits Administrator or Designer (CLAUDE.md §Scripting). The issue says "admin-gated", but the gate also admits Designers. That is the primary control, and it explains why this work is p3.

---

2. Goals and non-goals

Goals

• G1. Pre-emption. A script that exceeds its budget is stopped within a bounded time. The thread, the memory and the ALC are all reclaimed.
• G2. Containment. No script (OOM, stack overflow, FailFast-equivalent, native crash) can take down the OtOpcUa host process, its OPC UA sessions, its drivers or its Akka membership.
• G3. Attribution. The script that caused a kill is identified and quarantined. Healthy scripts that shared the worker are not.
• G4. Parity. Compile-time behaviour is unchanged: the same sandbox, analyzer, diagnostics and Monaco parity. Runtime semantics are unchanged too: the same ctx API, the same input-quality gating, the same Bad / last-value contract, and the same quarantine lift rules.
• G5. Bounded overhead (§8). It must fit the tag counts the repo already tests at.
• G6. Flag-gated, reversible rollout. In-process stays the default until the gates pass.

Non-goals

• N1. A new sandbox. ForbiddenTypeAnalyzer stays the capability control; process isolation is defence-in-depth, not a replacement.
• N2. Moving the authoring surface. The AdminUI only compiles and analyzes (ScriptAnalysisService), and it never executes a script, so central/admin nodes are out of scope.
• N3. Multi-tag cascades (ctx.SetVirtualTag taking effect). They are dropped today and stay dropped.
• N4. Cross-node or remote execution. The worker is always local to the driver node that owns the tags.
• N5. Per-run memory pre-emption. Even out of process, the limit is per worker. A per-run limit stays a sampled observation (§5.4).
• N6. Running non-.NET or untrusted third-party code.

---

3. Process model

3.1 Options

Option	Blast radius of a kill	Process count	Verdict
A. One worker per node	Every script on the node blips	1	Simple, but one bad script restarts every script on the node
B. One worker per script	One script	= script count (hundreds to thousands)	Infeasible: ~tens of MB per .NET process [A-3]
C. One worker per seam or driver instance	One seam / driver	3 to ~10	Arbitrary; a hot Calc driver still shares fate with its siblings
D. Small sharded pool + an isolation worker	One shard (1/K of the scripts), then one script	K + 0..1 (default 2 + 1)	Recommended

3.2 Recommended: sharded pool with an isolation worker

• K shard workers per driver node. Scripting:Worker:Shards defaults to 2 and is clamped to 1–8. A script is placed by hash(scriptKey) mod K. The script key is the guard key used today: the VT id, the calc RawPath, or the alarm id. Placement is deterministic, so a script's compiled PE and warm state stay on one worker across restarts.
• One isolation worker ("penalty box") is spawned on demand. Scripts under suspicion (§5.3) run there, one evaluation at a time. A kill in the isolation worker is therefore unambiguous.
• Inside a worker: a bounded evaluation scheduler with P dedicated threads, where P = max(2, ProcessorCount / K) [A-4]. Single flight per key is kept, as the guard does today. A queue that is full answers Busy, which the host treats as a refusal, not a strike.
• The passthrough fast path stays in the host. It never runs user code, and it is probably the largest share of evaluations [A-5].

3.3 Lifecycle and supervision under Akka

The design splits a control plane (actors) from a data plane (a lock-free client). An Akka Ask per evaluation would add mailbox hops to the hottest path for no benefit.

DriverHostActor (driver role)
 └─ ScriptRunnerActor            node-wide; owns the pool policy, attribution ledger, quarantine table
     ├─ ScriptWorkerActor #0     owns one OS process + pipe; state machine below
     ├─ ScriptWorkerActor #1
     └─ ScriptWorkerActor iso    spawned on first suspect, stopped after IsolationIdle (default 10 min)

IScriptRunner (DI singleton)  ── data plane ──►  ScriptWorkerConnection (per worker; pipe writer + correlation map)
   used by RoslynVirtualTagEvaluator, CalculationEvaluator, ScriptedAlarmEngine

• Ownership. ScriptRunnerActor is created by DriverHostActor, next to VirtualTagHostActor and the scripted-alarm host, so it exists only on driver-role nodes and dies with the driver tree. Its supervisor strategy is OneForOne, and a restart of a ScriptWorkerActor kills and respawns its process.

• Process exit is a message, not an exception. A watcher on Process.Exited / WaitForExitAsync sends WorkerExited(exitCode, signal, inFlight[]) to its ScriptWorkerActor, so worker crashes never escalate through Akka supervision.

• Worker states:

  Spawning → Handshaking → Loading → Ready ⇄ Draining → Dead → Backoff → Spawning

  Ready is the only state that accepts evaluations. In the other states the connection queues per key, with latest-wins coalescing (§9).

• The host process must take its workers down with it.

  • Windows: every worker is assigned to a Job Object with JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE. If the host dies, the kernel closes the job handle and kills the workers.
  • Linux (the docker-dev rig): the worker treats pipe EOF as fatal, and also polls its parent PID. The host is not trusted to clean up after itself.

• CoordinatedShutdown. In the before-actor-system-terminate phase, the host sends Shutdown, waits 1 s, then kills.

• Orphan sweep at start. The host kills only workers whose launch nonce file sits under this host's instance directory, never "every process with this name". mxaccessgw's OrphanWorkerTerminator killed a live gateway's workers when a second gateway started (scadaproj memory "wonder upgrade program"); this design avoids that by construction.

• Redeploy. ClearCompiledScripts / LoadAsync become IScriptRunner.ApplyGeneration(scriptSet). That call sends LoadScripts / Unload deltas to each shard and clears the host-side quarantine rows for edited or removed keys, which keeps today's lift rule. No worker restart is needed. Collectible ALCs in the worker are unloaded as they are today, and a stuck one no longer matters because its worker gets killed.

• Redundancy. No change. Each node runs its own scripts today and will keep doing so; workers are not shared across the pair.

---

4. IPC

4.1 Shape of the conversation

§1.1 showed that a run needs no host callbacks. Each evaluation is therefore one request frame and one reply frame, with no re-entrant RPC. ScadaBridge's SandboxWorkerProtocol has to proxy call/reply frames for external systems and databases; this design does not. Log lines from ctx.Logger are the only other worker→host traffic, and they are fire-and-forget.

4.2 Transport: named pipe vs gRPC

	Named pipe + length-prefixed frames	gRPC (h2c over UDS / named pipe)
Per-call overhead	Two syscalls plus serialize; no HTTP/2 framing [A-6]	HTTP/2 stream setup, HPACK and a Kestrel pipeline per call
Worker footprint	Plain console app	Needs Kestrel in the worker, or a client stack in it and a server in the host
Batching / priority	Free to design: control frames ahead of events, as in mxaccessgw WorkerFrameProtocol	Per-RPC; batching means streaming RPCs, which is a frame protocol again
Cross-platform	NamedPipeServerStream maps to a Unix domain socket on Linux; PipeOptions.CurrentUserOnly on both	Yes
Prior art in the family	mxaccessgw gateway↔worker (protobuf WorkerEnvelope, 4-byte LE length, priority-drained writer); ScadaBridge sandbox (JSON frames over stdio, 1 MiB cap)	OtOpcUa mesh transport (Commons/Protos/mesh_transport.proto) — but that is cross-node

Recommendation: a named pipe with 4-byte little-endian length-prefixed protobuf envelopes. It copies mxaccessgw's WorkerFrameProtocol discipline:

• a frame-size cap negotiated at the handshake (default 4 MiB);
• zero-length and oversize frames rejected;
• a monotonically increasing seq;
• control frames (heartbeat, shutdown, fault) drained ahead of evaluation traffic;
• coalesced flushes.

Google.Protobuf and Grpc.Tools are already central pins, and Commons.csproj:28-30 already compiles protos, so no new package is needed. MessagePack is pinned in Directory.Packages.props but referenced by no project, so it would be a new dependency in practice. Stdio, as ScadaBridge uses, is rejected: a script cannot write to the console (it is forbidden), but the runtime and the logger can, and one stray byte on stdout corrupts the stream.

4.3 Message schema (sketch)

syntax = "proto3";
package otopcua.scriptworker.v1;

message Envelope {
  uint32 protocol_version = 1;
  uint64 seq = 2;
  oneof body {
    Hello hello = 10;            // host→worker: nonce, max_frame_bytes, options
    Ready ready = 11;            // worker→host: pid, runtime version, sandbox fingerprint
    LoadScripts load = 12;       // host→worker: add/replace compiled-or-source scripts
    LoadResult load_result = 13; // worker→host: per-script compile outcome + diagnostics
    Unload unload = 14;          // host→worker: drop keys (ALC unload)
    EvalBatch eval = 15;         // host→worker
    EvalResults results = 16;    // worker→host (one or more completions)
    LogBatch logs = 17;          // worker→host, fire-and-forget, bounded
    Heartbeat heartbeat = 18;    // both ways; carries worker gauges (inflight, heap, gen2 count)
    Shutdown shutdown = 19;
    Fault fault = 20;
  }
}

enum ContextKind { CONTEXT_KIND_UNSPECIFIED = 0; VIRTUAL_TAG = 1; CALCULATION = 2; ALARM_PREDICATE = 3; }

message ScriptSource { string key = 1; bytes source_sha256 = 2; ContextKind kind = 3;
                       string source = 4; bytes pe_image = 5; repeated string input_paths = 6; }
message LoadScripts  { uint64 generation = 1; repeated ScriptSource scripts = 2; }

message EvalBatch    { repeated EvalRequest items = 1; }
message EvalRequest  {
  uint64 eval_id = 1; string key = 2; bytes source_sha256 = 3;
  repeated Input inputs = 4;      // positional against ScriptSource.input_paths; see 4.4
  int64 now_utc_ticks = 5;        // ctx.Now — host clock, so tests that pin the clock still work
  uint32 soft_timeout_ms = 6;
}
message Input { uint32 slot = 1; Variant value = 2; uint32 status_code = 3;
                int64 source_ts_ticks = 4; /* 0 = null */ int64 server_ts_ticks = 5; }

message EvalResult {
  uint64 eval_id = 1;
  oneof outcome { Variant value = 2; bool predicate = 3; ScriptError error = 4; }
  uint64 allocated_bytes = 5;     // same GC.GetAllocatedBytesForCurrentThread sample as today
  uint32 run_micros = 6;
}
message EvalResults { repeated EvalResult items = 1; }
message ScriptError { Kind kind = 1; string message = 2;
  enum Kind { THREW = 0; COMPILE = 1; SANDBOX = 2; SOFT_TIMEOUT = 3; UNSUPPORTED_TYPE = 4; BUSY = 5; } }

message Variant {
  TypeTag type = 1;               // exact CLR type — see 4.4
  oneof v { bool b = 2; sint64 i = 3; uint64 u = 4; double d = 5; float f = 6;
            string s = 7; int64 dt_ticks = 8; ArrayValue array = 9; }
  enum TypeTag { NULL = 0; BOOLEAN = 1; SBYTE = 2; BYTE = 3; INT16 = 4; UINT16 = 5; INT32 = 6; UINT32 = 7;
                 INT64 = 8; UINT64 = 9; FLOAT = 10; DOUBLE = 11; STRING = 12; DATETIME_UTC = 13;
                 ARRAY = 14; }
}
message ArrayValue { Variant.TypeTag element_type = 1; repeated Variant items = 2; }

message LogBatch { repeated LogLine lines = 1; uint32 dropped = 2; }
message LogLine  { string key = 1; uint32 level = 2; string message_template = 3;
                   string rendered = 4; string exception = 5; int64 ts_ticks = 6; }

4.4 How values and quality cross the boundary

• Quality crosses verbatim. DataValueSnapshot is (object? Value, uint StatusCode, DateTime? SourceTimestampUtc, DateTime ServerTimestampUtc) (Core.Abstractions/DataValueSnapshot.cs:13), and it maps 1:1 onto Input. The worker rebuilds the same snapshots, so ctx.GetTag(p).StatusCode and .SourceTimestampUtc behave as they do today. The input-quality gating does not move: VT holds evaluation while an input is Bad (VirtualTagActor.cs:264-269), Calc holds on Bad, and alarms short-circuit on AreInputsReady. A request is sent only when today's code would have run the script.
• The exact boxed CLR type is load-bearing. A script that writes (short)ctx.GetTag(x).Value throws InvalidCastException if the value arrives boxed as an int. Variant.type therefore carries the CLR type, not just the OPC UA type: DriverDataType has Int16 / UInt16, which have to survive the round trip. [A-7] I have not inventoried which CLR types each driver actually boxes, for example whether a Modbus Int16 arrives as short or as int. The shadow mode in §12 exists partly to find out.
• Unsupported types. An input whose CLR type has no tag is not silently converted. The request carries no value for that slot, and the worker answers UNSUPPORTED_TYPE; the host publishes Bad with the type name. On the result side, a script that returns an unsupported type (a List<T>, an anonymous object) also gets UNSUPPORTED_TYPE. That is a behaviour change: today the raw object goes to OpcUaPublishActor.AttributeValueUpdate (VirtualTagHostActor.cs:233), and what the sink does with it was not traced [A-8]. Shadow mode reports every such case before the flag flips.
• DateTime. Values are sent as UTC ticks. A Kind other than Utc is converted at the host edge and counted on a metric. [A-9] I assume drivers already produce UTC.
• Positional inputs. LoadScripts registers the script's ordered input_paths: the DependencyExtractor reads, after the {{equip}} resolution done at deploy. Requests then send values by slot, not by path string, which takes the dominant payload cost out of the hot path. A key with no registered slots, a fallback that should never happen, sends the paths inline.

4.5 Batching

• Opportunistic, not timed. The connection's writer drains every request queued at that moment into one EvalBatch and flushes once, as mxaccessgw's drain pass does. An idle system pays no added latency; a burst is amortised automatically. There is no coalescing timer, because it would add latency to every evaluation.
• Results are streamed, not batch-atomic. The worker sends EvalResults as items complete, coalescing whatever finished in the same scheduler pass, so a slow script never holds a fast sibling's reply.
• The seams must submit concurrently, or batching never happens. Today both Calc and alarms await one script at a time (§1.1), and with IPC that serial loop costs N × round-trip. Two seam changes are required:
  • Scripted alarms: ReevaluateAsync submits the whole dirty set at once, then applies results in the original order under _evalGate. Predicates are pure functions of a snapshot taken under the gate, so the state machine sees the same sequence.
  • Calculation: the pump dequeues everything pending, with each entry's dependency snapshot already captured at enqueue, submits it as one batch and publishes in dequeue order.
• Virtual tags become asynchronous. IVirtualTagEvaluator gains EvaluateAsync, and VirtualTagActor PipeTos the result to itself. While a run is in flight, further dependency changes update the map and set a "re-run" flag; when the result lands, the actor evaluates once more against the latest map (latest-wins). This also ends today's blocking of a default-dispatcher thread for up to 2 s (RoslynVirtualTagEvaluator.cs:174-175).

---

5. Pre-emption

5.1 Two deadlines

	Soft deadline T	Hard deadline T + G
Value	Today's per-seam budget, unchanged: VT 2 s, Calc runTimeoutMs (2 s), alarm 250 ms	G = Scripting:Worker:HardGrace, default max(1 s, T), clamped to 250 ms–30 s
Who acts	The host connection's timer	The host ScriptWorkerActor (authoritative) and a worker-side watchdog thread
Effect	The caller gets the timeout failure it gets today (Bad, same reason prefix ScriptEvalFailureReasons.TimedOutPrefix) and the run token is cancelled, so a polling script still ends itself as it does today	The worker process is killed (Process.Kill(entireProcessTree: true)). The worker watchdog calls Environment.FailFast first if it is still scheduling, so the dump and exit code name the key.
Strike	Yes (as today)	Yes, plus a kill strike (§5.3)

The grace period G keeps a script that overran by a few milliseconds from costing a whole shard restart, and the cooperative token from #734 still gets its chance. Making G = 0 (kill at T) is an open decision (§14).

Single flight between T and T+G. The key stays in flight, and a trigger is refused as today, which is not a strike. After the kill there is no orphan, so the stuck-orphan rule (StuckOrphanQuarantineFactor) becomes moot out of process; it stays for the in-process mode.

5.2 Resource limits on the worker

The limits combine prior art from ScadaBridge (heap limit) and mxaccessgw (process-tree kill):

• Heap: DOTNET_GCHeapHardLimit = Scripting:Worker:HeapLimitBytes, default 512 MiB [A-10]. Breaching it makes the worker throw OutOfMemoryException or fail fast. Either way the worker dies or reports, and the host survives.
• Windows Job Object (per worker):
  • JOB_OBJECT_LIMIT_PROCESS_MEMORY, set slightly above the heap limit as a backstop for native and JIT memory;
  • JOB_OBJECT_LIMIT_ACTIVE_PROCESS = 1, so no child processes are possible;
  • KILL_ON_JOB_CLOSE;
  • UI restrictions.
• Linux / containers: the heap limit only. A container's cgroup is the rig's concern. [A-11] I assume the docker-dev containers run without CAP_SYS_ADMIN, so a nested cgroup is not available.
• Liveness: a heartbeat every 1 s. Three missed heartbeats while not in a GC pause, which the heartbeat reports, trigger a kill with reason heartbeat. This catches a wedged worker that neither replies nor exits.

5.3 Attribution, restart, backoff and quarantine

A kill or crash takes down every in-flight evaluation on that shard. Attribution decides which one to blame.

1. Hard-deadline kill: the key that crossed T+G is known exactly. It takes a kill strike. The other in-flight keys on that shard are victims: their runs fail with "script worker restarted", which is not a strike.
2. Crash with no deadline crossed (OOM, stack overflow, native fault): attribution is ambiguous. Every key in flight at exit becomes a suspect, and suspects move to the isolation worker.
3. Isolation worker: evaluations run one at a time, so a crash there attributes to exactly one key, which takes a kill strike. A suspect that completes IsolationProbation (default 20 successful runs or 10 min) without incident returns to its shard.
4. Quarantine: a key is quarantined after Scripting:Worker:QuarantineAfterKills kill strikes (default 2) within 1 h, or by today's rules (N consecutive soft timeouts, allocation breach). The quarantine table lives in the host (ScriptRunnerActor), so it survives worker restarts. It is lifted by a source change or a redeploy, exactly as today, and quarantine reasons keep the "script quarantined: " prefix so every existing consumer and test classifier still matches.
5. Restart backoff per worker: exponential, 250 ms → 30 s cap, ×2, with ±20% jitter. The backoff resets after 60 s of Ready. A worker that dies during Loading (a poison PE or a runtime fault) does not loop: after 3 consecutive load-phase deaths, the scripts in its generation are loaded one by one in the isolation worker to find the culprit.
6. Crash-loop circuit: if all shards are in backoff for longer than Scripting:Worker:CircuitOpenAfter (default 2 min), the runner reports Unavailable (§10) and logs one Error. It keeps retrying at the cap.

---

6. Security

Process isolation contains failures. Most of the capability control stays with the compile-time sandbox, which is retained. Defence-in-depth adds the following.

• The compile-time sandbox is unchanged. It still includes ScriptSandbox (with the TPA reference set filtered to System.*, ScriptSandbox.cs:118-122), ForbiddenTypeAnalyzer, and the injection guard. The worker runs exactly the same ScriptEvaluator.Compile, so editor/publish/runtime parity holds.

• Assembly allow-list by construction. The worker is its own app (ZB.MOM.WW.OtOpcUa.ScriptWorker), with its own deps.json and only these references:

  • Core.Scripting;
  • Core.Scripting.Abstractions;
  • Core.Abstractions;
  • Serilog (core only, for ILogger on ctx);
  • Google.Protobuf.

  Its trusted-platform list therefore contains the shared framework plus those assemblies. Akka, SqlClient, the OPC UA stack, driver assemblies and the Secrets library are not loadable in the worker, even through reflection. At startup the worker asserts its loaded-assembly set against a pinned list and exits if the list differs.

• No secrets in the environment. ProcessStartInfo.Environment is cleared and then repopulated from an explicit allow-list: DOTNET_* runtime knobs, TEMP, PATH minimal, and the launch nonce. Host env vars like ServerHistorian__ApiKey (CLAUDE.md "Never commit secrets") and the ConfigDb connection string must never reach the worker. ScadaBridge's sandbox inherits the parent environment; do not copy that part.

• No configuration access. The worker gets no appsettings path. Every option it needs (budgets, heap, frame cap) arrives in Hello.

• Pipe hardening (mxaccessgw pattern):

  • a random pipe name;
  • PipeOptions.CurrentUserOnly, or an explicit ACL when the worker runs as a different account;
  • a one-time nonce passed in an env var (never on the command line; mxaccessgw WorkerBootstrap.md explains why), which the worker must echo in Ready;
  • the host accepts exactly one connection per launch.

• Filesystem:

  • the working directory is an empty per-worker temp directory;
  • the PE cache (§7) is opened read-only by the worker, if the host compiles, or written only under its own subdirectory;
  • on Windows, a dedicated low-privilege account (see §14) gets read access only to the install folder and its own temp directory.

• Network: the analyzer already forbids System.Net. Process-level options are a Windows Firewall outbound-block rule scoped to the worker executable, created by the installer, and on Linux nothing beyond the container. This is an open decision (§14). The recommendation is the installer firewall rule on Windows, and to accept the analyzer as the control on Linux.

• What the host trusts from the worker: nothing it cannot validate. Frames are size-capped and schema-checked, an eval_id must match an outstanding request, and log lines are length-capped and rate-limited (§11). A compromised worker can at worst return wrong values for scripts on its own shard, which the same script could have done anyway.

---

7. Compile caching across restarts

The problem: today every process start recompiles every script lazily (CompiledScriptCache is memory-only). Out of process, a shard restart has the same cost, and it can happen in the middle of production running. mem-probe measured the old CSharpScript path at roughly 2.4 MiB of working set per retained script (tools/mem-probe/MemProbe/Program.cs:1-4). Compile time is not measured anywhere in the repo [A-12]; I assume tens of ms per script warm and seconds for the first compile while Roslyn JITs. For 1,000 scripts, that is far outside a 2 s restart budget.

Recommendation: a content-addressed PE cache on node-local disk.

• Key: SHA-256 over these fields:

  • the source;
  • the ContextKind, i.e. TContext and TResult;
  • the Roslyn compiler version;
  • a sandbox fingerprint: a hash of ForbiddenTypeAnalyzer's lists, the ScriptSandbox reference set and the wrapper template.

  An analyzer change therefore invalidates every entry, and a cached PE can never skip a newer sandbox rule.

• Value: the emitted PE bytes plus an HMAC-SHA256 tag. The HMAC key is a per-node random key created at first start and stored next to the cache, readable only by the host account. The worker loads a PE only if the HMAC verifies, because loading attacker-supplied PE bytes would be code execution outside the analyzer.

• Location: Scripting:Worker:PeCacheDir. It must be absolute and outside the content root, the same fail-closed rule the family applies to Secrets:SqlitePath (scadaproj CLAUDE.md, Secrets row). Entries whose key is not in the current generation are pruned at ApplyGeneration.

• Where compilation happens: in the worker (recommended). Roslyn's memory and CPU leave the host. A pathological source that makes Roslyn itself blow up kills a worker, not the server, and the worker writes the PE plus HMAC (it holds the key through Hello). The alternative is to compile in the host and ship pe_image in LoadScripts. That keeps the worker free of Roslyn (smaller, faster start), but compile-time blowups stay in the host. This is an open decision (§14).

• Restart path: Spawning → Ready takes about 150 ms of runtime start [A-13]. Then LoadScripts sends hash + key + slots only. The worker loads PEs from the cache lazily, on each key's first evaluation, and a cache miss compiles. Loading a PE into a collectible ALC is about 1 ms [A-13].

• Negative cache: Calculation already remembers compile failures by source hash (R07:P4, CalculationEvaluator.cs:63-68). The worker reports compile failures in LoadResult, and the host keeps that negative cache for all seams, so a broken script never re-enters Roslyn.

---

8. Performance budgets

8.1 Reference loads in the repo

Source	Load
docs/v2/Galaxy.Performance.md (EventPump sizing, 24 h soak)	50,000 tags at 1 Hz on one node; compressed CI soak at 1,000 tags
tests/Server/.../NodeManagerLockTimeBenchmarkTests.cs:41-105	10,000-tag folder; 200 equipment × 250 tags (50,000 variables)
ScriptRunGuardTests.cs:170-205	a 1 Hz runaway and a healthy sibling over 30 triggers

No repo perf test exercises scripts at scale. [A-14] I assume a planning envelope of 5,000 compiled script evaluations per second per node (for example 5,000 VT + Calc at 1 Hz, or 500 scripts at 10 Hz) and size for 2× headroom (10k/s). The passthrough fast path is excluded because it stays in the host.

8.2 Budgets (acceptance gates for the implementation)

Metric	Budget	Notes
Added latency per evaluation, p50	≤ 150 µs (Linux, macOS); ≤ 500 µs on a Windows VM	Host enqueue → reply decoded, trivial script, idle system. Windows-VM budget accepted by the owner 2026-11-12 (lmxopcua#789, §14 #15): windev measured 355–388 µs, because a blocking hand-off costs about 10× macOS there (§12.8). 500 µs is 0.2 % of the 250 ms alarm budget
Added latency per evaluation, p99	≤ 2 ms	At 10k evals/s. Compare the 250 ms alarm budget: under 1% of it
Throughput	≥ 10,000 trivial evals/s/node with K=2	With opportunistic batching
Host CPU for IPC	≤ 25% of one core at 10k evals/s (Linux, macOS); ≤ 50 points batched at 10k/s on a Windows VM	Serialisation + pipe I/O, measured by an opt-in perf test. Since lmxopcua#789 the harness reports it as the out-of-process runner's host CPU minus the in-process runner's in the same phase ("vs InProcess"): the raw figure includes the harness's own pacing loop and pool, which InProcess pays too (§12.2). On a Windows VM the gate is the paced 10,000/s, batched row (windev measured +36 to +45; one-at-a-time +71 to +84 is reported but not gated). At 5k/s, the planning envelope [A-14], out-of-process is already below InProcess
Bytes per eval on the wire	≤ 64 B + 24 B per input	Positional slots, §4.4
Shard restart → first result, 1,000 cached scripts	≤ 2 s	§7 lazy PE load
Worker idle RSS	≤ 200 MB + scripts	Per shard. Revised by the owner 2026-11-11 (lmxopcua#789) from the original ≤ 80 MB + scripts [A-3]. Compiling in the worker (§14 #7) keeps Roslyn's code and metadata resident once it has run (measured 159–187 MB), and no GC setting moves it (§12.2). A worker that has only loaded cached PEs sits far below that (measured 36–58 MB). The perf harness reports both against 200 MB
Kill latency after T+G	≤ 100 ms	Host timer resolution plus Process.Kill

Comparison with today: the in-process cost per compiled run is one Task.Run hop plus WaitAsync, which is a few µs to tens of µs and was never measured [A-15]. The out-of-process path is therefore one to two orders of magnitude slower per call. That is acceptable only because the budgets above sit two to three orders of magnitude below the script budgets (250 ms / 2 s), and because batching amortises the syscalls. The risk case is a large scripted-alarm engine, with its serial gate and its 250 ms budget. That is why §4.5 makes the batch-submit change a prerequisite, not an optimisation.

---

9. Failure semantics: what quality a tag gets

The principle: a runner failure must never look like a script result. In particular, an alarm must never clear because the worker died.

Situation	Virtual tag / Calculation tag	Scripted alarm
Soft timeout (T)	BadInternalError + last value, once per transition, as today	Condition frozen + BadInternalError quality annotation, as today
This key killed at T+G, or crashed in isolation	Same as soft timeout; reason "script killed: exceeded hard deadline …"	Same as soft timeout
Victim: in flight on a shard that died for another key	BadInternalError + last value, reason "script worker restarted"; re-evaluated on the shard's return with the latest inputs (latest-wins)	Frozen + Bad annotation; re-evaluated on return
Not in flight while its shard restarts	Unchanged. Keeps its last published value and quality. A trigger during the outage is coalesced (latest-wins, one per key) and evaluated when the shard is Ready	Same; the condition holds its state
Shard not back within Scripting:Worker:UnavailableGrace (default 5 s)	Coalesced keys publish Bad + last value, reason "script runner unavailable"	Frozen + Bad annotation
Circuit open (§5.3.6)	Every compiled-script key on the node is Bad as above; passthrough keys are unaffected	Every compiled predicate is frozen + Bad
Quarantined	"script quarantined: …" Bad, as today	As today
Host process dies	Workers die with it (Job Object / pipe EOF); no change from today's host-death behaviour	—

Status code: BadInternalError stays for script-attributable faults (the existing contract). For "the runner is unavailable, not this script's fault", this design proposes BadResourceUnavailable (0x80040000), so operators and the ScadaBridge consumer can tell a broken script from a sick node. The alternative is to keep BadInternalError everywhere for zero consumer impact. This is an open decision (§14); if it is taken, the ScadaBridge consumer-impact record must note it (scadaproj sister-project-history skill, otopcua.md).

No silent fallback to in-process. If the worker cannot start, the runner does not run scripts in the host. That would reintroduce exactly the risk the flag was set to remove. Operators flip the flag back (§12). This is an open decision (§14).

---

10. Observability

All metrics go on OtOpcUaTelemetry.Meter, so the existing ZbTelemetryOptions allow-list applies. scadaproj memory: "ZbTelemetryOptions.Meters is a silent allowlist"; check that the new names flow end to end.

Metric	Type	Tags
otopcua.script.worker.up	gauge	shard
otopcua.script.worker.starts / .exits	counter	shard, reason = clean | killed_deadline | killed_heartbeat | oom | crash | load_failure
otopcua.script.worker.inflight / .queue_depth	gauge	shard
otopcua.script.worker.heap_bytes	gauge (from heartbeat)	shard
otopcua.script.ipc.roundtrip	histogram (µs)	kind
otopcua.script.eval.duration	histogram (µs, worker-measured)	kind
otopcua.script.compile	counter	cache = hit | miss | negative
otopcua.script.quarantined (existing)	counter	adds reason = killed | isolated_crash
otopcua.script.run_refused (existing)	counter	unchanged
otopcua.script.isolation.suspects	gauge	—
otopcua.script.runner.state	gauge (0 ready / 1 degraded / 2 unavailable)	—

Per-key series: the existing counters tag the series with script = key. Do not add key tags to the new per-eval histograms (cardinality).

Logs:

• the worker's own stderr is captured, prefixed with ScriptWorker[shard], into the host log;
• every kill logs one Warning naming the key and the reason, and every quarantine logs one Error, as today;
• ctx.Logger lines come back in LogBatch. The host re-emits them through ScriptRootLogger with the same ScriptId / VirtualTagId properties, so scripts-*.log, the error mirror, the DPS script-log topic and its 20/min-per-source rate limit all keep working unchanged;
• the worker caps its outbound log buffer (default 1,000 lines / 1 MiB) and reports drops as LogBatch.dropped, so a script logging in a tight loop cannot flood the pipe.

Health: expose the runner state as a non-fatal component of the driver-node status (/health/ready stays green while degraded, because OPC UA still serves driver data). Whether Unavailable should fail readiness is an open decision (§14). The AdminUI status dashboard gets a "Script runner" tile with the per-shard state, restarts in the last hour and the quarantine list. Quarantines already surface through the QuarantinedScripts properties.

Traces: one Activity per EvalBatch (not per evaluation), with batch.size and shard.

---

11. Seam changes in the host (summary)

Component	Change
Core.Scripting	New IScriptRunner (EvaluateAsync(ScriptInvocation[]), ApplyGeneration, QuarantinedKeys, ResetAll). InProcessScriptRunner wraps today's ScriptRunGuard + CompiledScriptCache exactly.
New Core.Scripting.Protocol	The proto file, the frame codec and the Variant ↔ CLR mapping. Shared by host and worker.
New ZB.MOM.WW.OtOpcUa.ScriptWorker	Console app: handshake, scheduler, watchdog, PE cache, log forwarding. Shipped in the Host's publish output.
New Runtime/Scripting/	ScriptRunnerActor, ScriptWorkerActor, OutOfProcessScriptRunner (data plane), Job Object interop (Windows-only, behind OperatingSystem.IsWindows()).
IVirtualTagEvaluator / VirtualTagActor	Add EvaluateAsync; the actor PipeTos with latest-wins re-run (§4.5).
RoslynVirtualTagEvaluator, CalculationEvaluator, ScriptedAlarmEngine	Replace direct ScriptRunGuard use with IScriptRunner. Calc pump and alarm ReevaluateAsync submit batches (§4.5).
ScriptingOptions / ScriptingOptionsBinding	New keys (§12), clamped with startup Warnings like the existing ones.

---

12. Migration and rollout

Configuration (node-local appsettings, Scripting section, consistent with the existing knobs):

Key	Default	Meaning
Scripting:Runner	InProcess	InProcess | OutOfProcess | Shadow
Scripting:Runner:Seams	VirtualTag + Calculation	Optional per-seam subset for OutOfProcess / Shadow (e.g. ["Calculation"]). Unset selects VirtualTag and Calculation; ScriptedAlarm only when listed (§14 #10, as built in §12.3)
Scripting:Worker:Shards	2	1–8
Scripting:Worker:HardGrace	max(1s, T)	§5.1
Scripting:Worker:HeapLimitBytes	536870912	§5.2
Scripting:Worker:QuarantineAfterKills	2	§5.3
Scripting:Worker:UnavailableGrace	5s	§9
Scripting:Worker:PeCacheDir	none (required when not InProcess)	§7; absolute, outside the content root, fail-closed
Scripting:Worker:Identity	none (host's token)	Windows only: NT SERVICE\OtOpcUaScriptWorker → restricted worker token + pipe ACL (§14 #5, record §12.5)

Phases (each one is its own issue and PR):

1. P0, in-process refactor with no behaviour change.
   • Introduce IScriptRunner + InProcessScriptRunner.
   • Make VT evaluation async (PipeTo), and make the alarm engine and the Calc pump batch-submit.
   • Gate: every existing Scripting / VirtualTags / Calculation / ScriptedAlarms suite stays green unchanged, plus new ordering tests for batch-then-apply.
   • This phase has value on its own: it ends the dispatcher-thread blocking.
   • Partly landed in lmxopcua#779: VT evaluation is async (IVirtualTagEvaluator.EvaluateAsync + stash in VirtualTagActor), so the dispatcher-thread blocking is gone. The Calc pump and the alarm engine still run one script at a time; why that was not relaxed in-process is recorded in docs/VirtualTags.md §"Upstream-tag feed". IScriptRunner and batch-submit remain to do.
   • Batch submit landed in lmxopcua#780 (fourth P1 slice, §12.3): IScriptRunner.RunBatch, used by the Calc pump, the VirtualTag seam (a coalescer) and the scripted-alarm engine on the out-of-process path. In process the pump and the engine stay one at a time, unchanged.
2. P1, worker + protocol behind the flag.
   • Ship ScriptWorker, OutOfProcessScriptRunner and the PE cache.
   • Default stays InProcess. Enable on the docker-dev rig only.
   • Gate: §13 runaway / containment tests green on Linux (docker-dev) and Windows (windev box, which has the .NET 10 toolchain).
   • Partly landed in lmxopcua#780 (P1 slice): Scripting:Runner / Scripting:Runner:Seams / Scripting:Worker:* (bound, clamped, PeCacheDir path rule fail-closed), IScriptRunner + InProcessScriptRunner, the protobuf frame protocol (Core.Scripting.Protocol), the ScriptWorker app (compile in the worker, P-thread scheduler, watchdog, heartbeats, log forwarding, pipe-EOF / parent-pid exit, forbidden-assembly self-check), OutOfProcessScriptRunner (K shards + isolation worker, soft/hard deadlines, attribution, kill-strike quarantine, backoff, fail closed, Job Object on Windows), the VirtualTag and Calculation seams over it, and BadResourceUnavailable for runner-unavailable / worker-restarted (ScadaBridge#112). The §13 containment cases run on macOS and Windows (windev). Deviations: supervision is a plain object owned by DI, not ScriptRunnerActor/ScriptWorkerActor; the per-seam batch submit (P0) is still one-at-a-time. Remaining: the HMAC'd PE cache, a self-contained worker publish + the low-privilege account / pipe ACL / firewall rule (installer), the runner-state gauge and readiness component, victim re-evaluation on shard return, isolation-worker idle stop, the orphan sweep by nonce file, the scripted-alarm seam, the docker-dev enablement and the §8 perf harness.
   • Second P1 slice (lmxopcua#780): the HMAC'd PE cache (§7; keyed by source + kind + a compile fingerprint, tampered or mismatched entries evicted, pruned by recency at startup instead of by generation — this runner has no generation object), attribution before release (the quarantine race), loading keys counted in attribution (culprit-finding for a compile that crashes its worker, by isolation rather than the §5.3.5 three-strikes bisection — loads are per key here), proactive restart + Recovery / RetryWhen re-evaluation of victims in the VT and Calc seams, isolation idle stop (IsolationIdleTimeout), the crash-loop circuit (CircuitOpenAfter), the §10 gauges and histograms, the degraded-but-ready script-runner health check (§14 #12), and the orphan sweep by launch marker (pid + start time + image, host gone). Still open: self-contained worker publish, installer (account, pipe ACL, firewall), docker-dev live drill, scripted-alarm seam, the §4.5 batch submit, the §8 perf harness.
   • Fourth P1 slice (lmxopcua#780, record in §12.3): the §4.5 batch submit (IScriptRunner.RunBatch, one EvalBatch frame per worker; Calc pump, VT coalescer, alarm dirty set), exact per-item deadlines for work queued in the worker (deferral / start notices), and the scripted-alarm seam behind Scripting:Runner:Seams — alarms run out of process only when listed. Still open: installer (account, pipe ACL, firewall, PeCacheDir ACL), #789's remaining misses, then P2 Shadow.
   • Third P1 slice (lmxopcua#780, record in §12.1): a self-contained worker publish for self-contained (Windows) Host publishes, started through its apphost; the docker-dev live drill (docker-dev/docker-compose.script-runner.yml); the §8 perf harness (Category=Perf, opt-in), which passes the accepted 5k evals/s/node gate; and two runner fixes the harness found (a caller refused by its own finished run; heartbeat starvation under a compile burst). Still open: installer (account, pipe ACL, firewall, PeCacheDir ACL), scripted-alarm seam, §4.5 batch submit, the §8.2 budget misses (lmxopcua#789) and the load-timeout queue issue (lmxopcua#790).
3. P2, shadow mode on docker-dev and QA.
   • Shadow runs the in-process evaluation as the published result, and also submits to the worker.
   • It compares value, status class and failure kind per evaluation, and counts otopcua.script.shadow.mismatch{kind, reason}. Mismatching keys are logged once each.
   • Shadow doubles script CPU and still carries the in-process risk, so it is a time-boxed diagnostic (for example 1 week on QA), not a mode to run in production.
   • Exit criterion: zero unexplained mismatches. This is where [A-7] / [A-8] get answered.
   • Landed in lmxopcua#780 (P2 slice, record in §12.4): ShadowScriptRunner, all three seams, bounded and drop-counted, readiness untouched; docker-dev drill with alarms.
   • Owner decision 2026-11-12: no QA shadow week. P2 ends with the docker-dev drill; Shadow stays available as an opt-in diagnostic but is not a rollout gate. QA goes straight to P3.
4. P3, flip on QA, then the other tiers.
   • Set Scripting:Runner=OutOfProcess per node, seam subset first (Calculation, then VT, then alarms; alarms last because of the 250 ms budget).
   • Note: remote command execution on the wonder boxes is currently scoped to user-requested deploys (CLAUDE.md hold), so tier rollout rides normal deploys.
5. P4, decide the long-term status of InProcess. The recommendation is to keep it for dev and tests and as the documented rollback, and to retire StuckOrphanQuarantineFactor from the docs as "in-process only".
   • Decided (owner, §14, 2026-11-09) and recorded 2026-11-12: InProcess stays the compiled default (dev, tests, docker-dev) and the documented rollback (set the flag back and restart the node). StuckOrphanQuarantineFactor is documented as in-process only in docs/ScriptEditor.md. The QA flip with the live runaway drill passed (record: docs/deploy/qa-upgrade-b8b4c18f-to-fea7de0a.md, section "Canary, live isolation check and runaway drill").

Rollback: set the flag back to InProcess and restart the node. No data or config migration is involved. The PE cache directory can be deleted at any time.

Docs to update when implementing:

• docs/VirtualTags.md §"Known resource limits" and §"Per-evaluation timeout";
• docs/ScriptEditor.md §"Runtime safety";
• docs/ScriptedAlarms.md;
• docs/ServiceHosting.md: the new process, Job Object and firewall rule;
• docs/security.md;
• docs/Telemetry.md;
• the scadaproj index row, if the status-code decision changes the ScadaBridge-visible contract.

12.1 P1 record: self-contained publish, docker-dev live drill, §8 perf (2026-11-10)

Self-contained worker publish.

• A -r win-x64 --self-contained Host publish now publishes the worker self-contained for the same RID into script-worker/: 236 files, about 97 MB, with its own runtime. It is not trimmed and not single-file.
• The launcher starts the worker's apphost (ZB.MOM.WW.OtOpcUa.ScriptWorker.exe) directly. It does so only when the worker's runtimeconfig.json lists includedFrameworks. Otherwise (framework-dependent) it keeps dotnet exec <dll>.
• A framework-dependent publish is unchanged: 51 files, no includedFrameworks, run through the muxer.
• Gate on windev: the Host was published win-x64 self-contained under C:\build, and the whole ScriptWorker.Tests suite was run against the published apphost (OTOPCUA_SCRIPTWORKER_TEST_PATH).
  • Result: 132/132.
  • It includes A_published_self_contained_worker_runs_as_its_own_process. That test checks that the worker process's MainModule is the .exe, so no muxer is involved.

docker-dev live drill. Image otopcua-host:dev was built from the branch at f50b29ab plus the self-contained change. That build is from before the two runner fixes below. The drill ran with docker-compose.script-runner.yml on central-1/central-2.

Step	Observed
Normal VT + Calc	Workers start on the first compiled evaluation (dotnet exec /app/script-worker/…dll). okvt (HR100+1) stays Good and tracks HR100. Writing HR5 = 21 on the fixture gives Calc HR5x2 42 Good on both nodes, and the sibling VT drillsib (HR5×3) 63 Good.
while (true) { } VT (1 s timer, T = 2 s)	Logged exceeded its hard deadline (4000 ms); killing ScriptWorker[1] twice, 5 s apart. After the second: quarantined: "it exceeded its hard deadline and its worker was killed (2 time(s) in the last hour)…". Both nodes did the same. okvt and Calc share that shard and kept updating Good (26 Good samples, no Bad) because neither was in flight at the kills. run_refused then counts the quarantined trigger.
kill -9 both workers on central-1	A 1.5 s busy VT (drillslow) was in flight on shard 1. It published BadResourceUnavailable at 05:41:22.28, became a suspect and moved to isolation. It was Good again 30 ms later, re-evaluated in isolation. After HR5 = 22 it read 1022. Shards 0 and 1 restarted within about 250 ms. Isolation probation (20 clean runs) returned it to its shard. Calc gave 44 Good and the sibling 66 Good.
Circuit open (worker dll renamed, all workers killed)	/health/ready stayed 200. The overall status went to Degraded, and script-runner showed Degraded → Unavailable (circuit open) after 2 min. Logged Script runner circuit OPEN … once. otopcua_script_runner_state = 2, worker_up = 0 per shard. VTs and Calc (after an HR5 change) published BadResourceUnavailable. Driver tags (HR100) stayed Good, so OPC UA kept serving.
Recovery (dll restored)	The next backoff retry (cap 30 s) started the workers. Logged circuit closed. Calc 46 (HR5 = 23 × 2), sibling 69, drillslow 1023, okvt Good, all without any input change.
Metrics on :9000/metrics	otopcua_script_runner_state, worker_up/inflight/queue_depth/heap_bytes{shard}, worker_starts/exits{reason}, compile{cache}, ipc_roundtrip/eval_duration histograms, quarantined{reason}, isolation_suspects.

On the wire, a Bad publish carries a null Value. This holds for every Bad status in this server, including BadInternalError and BadOutOfRange. The seams keep the last value, but an OPC UA client does not see it. This was noted for ScadaBridge#112.

Rig state afterwards.

• Drill rows were deleted, HR5 was set back to 5, and the config was redeployed. The revision hash is e221b179…, byte-identical to the pre-drill gate-cleanup deployment.
• otopcua-host:dev was re-tagged to the pre-drill image 24f9d7287ed2.
• central-1/central-2 were recreated from the base compose file: InProcess, no Scripting__* env. The four site nodes, sql and traefik were never touched.
• The two central containers were recreated (twice), so their container-local secret stores and their PE cache dirs under /tmp are gone.

Two defects outside the runner were found and filed:

• lmxopcua#787: a disabled VirtualTag is still deployed and evaluated.
• lmxopcua#788: worker start failures log an Error with a stack trace on every retry.

§8 perf harness.

• ScriptRunnerPerfTests, Category=Perf, opt-in with OTOPCUA_SCRIPT_PERF=1.
• The runners are driven directly: trivial one-input script, 2,000 keys over 50 sources.
• Phases:
  • idle sequential;
  • paced open loop at 5k/s and at 10k/s;
  • 64-way closed loop.
• Host CPU is this process's CPU time as a share of one core, harness included. Under the same harness InProcess shows 60–75 % at 5–10k/s, and that is the harness's own floor.

macOS:

Script worker restart / RSS / kill latency (Unix Arm64, 14 logical CPUs, .NET 10.0.14, phase 10 s)

Measure	Result	§8.2 budget
1000 scripts, empty cache, all first results (compile + write)	1,633 ms	—
1000 scripts, fresh runner on a warm cache, all first results	217 ms	≤ 2,000 ms
kill both shards → every script's next correct result	482 ms	≤ 2,000 ms
worker RSS after first eval (max of 2 shards)	188 MB	≤ 80 MB + scripts
worker RSS with 500 scripts loaded per shard (max)	358 MB	—
kill latency after T+G (5 runs, ms)	31 / 35 / 31 / 32 / 33	≤ 100 ms

Script runner perf (Unix Arm64, 14 logical CPUs, .NET 10.0.14, phase 10 s)

Runner	Phase	evals/s	p50 µs	p99 µs	host CPU	ok / not ok
InProcess	idle, sequential	189,154	4	8	446 %	5,000 / 0
InProcess	paced 5,000/s	4,998	32	223	60 %	49,999 / 0
InProcess	paced 10,000/s	9,991	41	364	74 %	99,986 / 0
InProcess	closed loop ×64	789,533	4	15	409 %	7,896,214 / 0
OutOfProcess K=1	idle, sequential	21,641	45	64	264 %	5,000 / 0
OutOfProcess K=1	paced 5,000/s	4,999	149	860	108 %	49,998 / 0
OutOfProcess K=1	paced 10,000/s	9,995	205	1,763	159 %	99,989 / 0
OutOfProcess K=1	closed loop ×64	206,862	258	1,368	526 %	2,068,749 / 0
OutOfProcess K=2	idle, sequential	22,642	41	70	267 %	5,000 / 0
OutOfProcess K=2	paced 5,000/s	4,999	182	722	125 %	49,994 / 0
OutOfProcess K=2	paced 10,000/s	9,996	232	720	237 %	99,998 / 0
OutOfProcess K=2	closed loop ×64	336,519	150	993	544 %	3,365,226 / 0
OutOfProcess K=4	idle, sequential	18,461	50	97	303 %	5,000 / 0
OutOfProcess K=4	paced 5,000/s	4,999	201	455	197 %	49,997 / 0
OutOfProcess K=4	paced 10,000/s	9,997	126	732	176 %	99,989 / 0
OutOfProcess K=4	closed loop ×64	297,094	178	1,015	507 %	2,970,998 / 0

windev (Windows VM; framework-dependent worker built beside the tests):

Script runner perf (Win32NT X64, 36 logical CPUs, .NET 10.0.8, phase 10 s)

Runner	Phase	evals/s	p50 µs	p99 µs	host CPU	ok / not ok
InProcess	idle, sequential	5,876	142	253	204 %	5,000 / 0
InProcess	paced 5,000/s	4,997	136	335	55 %	49,996 / 0
InProcess	paced 10,000/s	9,989	105	274	6 %	99,984 / 0
InProcess	closed loop ×64	177,782	26	10,062	543 %	1,778,824 / 0
OutOfProcess K=1	idle, sequential	1,276	750	1,263	74 %	5,000 / 0
OutOfProcess K=1	paced 5,000/s	4,997	1,073	3,028	68 %	49,990 / 0
OutOfProcess K=1	paced 10,000/s	9,992	1,139	2,840	102 %	99,999 / 0
OutOfProcess K=1	closed loop ×64	38,464	1,514	6,547	350 %	384,727 / 0
OutOfProcess K=2	idle, sequential	1,207	795	1,376	80 %	5,000 / 0
OutOfProcess K=2	paced 5,000/s	4,998	956	1,856	111 %	49,997 / 0
OutOfProcess K=2	paced 10,000/s	9,962	942	1,916	153 %	99,992 / 0
OutOfProcess K=2	closed loop ×64	51,241	1,129	5,894	594 %	512,504 / 0
OutOfProcess K=4	idle, sequential	1,163	828	1,366	61 %	5,000 / 0
OutOfProcess K=4	paced 5,000/s	4,997	804	1,472	172 %	49,991 / 0
OutOfProcess K=4	paced 10,000/s	9,992	757	1,591	233 %	99,992 / 0
OutOfProcess K=4	closed loop ×64	69,503	774	6,321	1,051 %	695,094 / 0

Script worker restart / RSS / kill latency (Win32NT X64, 36 logical CPUs, .NET 10.0.8, phase 10 s)

Measure	Result	§8.2 budget
1000 scripts, empty cache, all first results (compile + write)	11,625 ms	—
1000 scripts, fresh runner on a warm cache, all first results	1,715 ms	≤ 2,000 ms
kill both shards → every script's next correct result	2,028 ms	≤ 2,000 ms
worker RSS after first eval (max of 2 shards)	158 MB	≤ 80 MB + scripts
worker RSS with 500 scripts loaded per shard (max)	232 MB	—
kill latency after T+G (5 runs, ms)	257 / 210 / 200 / 217 / 209	≤ 100 ms

Verdict against §8.2 and the accepted gate:

• Accepted gate (≥ 5,000 evals/s/node at K = 2): PASS on both hosts (macOS 336k/s, windev 51k/s closed loop), with every result correct.
• Throughput ≥ 10k/s: PASS.
• Idle p50 ≤ 150 µs: PASS on macOS (41 µs), MISS on windev (795 µs; InProcess there is already 142 µs).
• p99 ≤ 2 ms at 10k/s: PASS (macOS 0.7 ms, windev 1.9 ms).
• Warm-cache restart of 1,000 scripts ≤ 2 s: PASS (0.2 s / 1.7 s).
• Kill latency ≤ 100 ms: PASS on macOS (about 32 ms), MISS on windev (about 210 ms).
• Host CPU ≤ 25 % of a core at 10k/s: MISS. windev runs K = 2 at about 153 %. On macOS, K = 2 runs about 240 % against InProcess's 74 % under the same harness. A sequential probe measured 156 µs of host CPU per evaluation, and 63 µs with DOTNET_ThreadPool_UnfairSemaphoreSpinLimit=0. Thread-pool spinning is therefore most of it.
• Worker idle RSS ≤ 80 MB: MISS. It is about 190 MB on macOS and 158 MB on windev, because Roslyn is resident since compiles run in the worker.

Both misses → lmxopcua#789 (the §4.5 batching and per-eval allocation work).

Two runner defects found by the harness and fixed in this slice.

1. Refused own run. The single-flight slot was freed by a continuation that was not ordered against the caller's await. A caller that awaited one run and then started the next was sometimes refused with previous evaluation still running: 32 of 200 back-to-back runs, and about 80 % in the perf idle phase. The VT and Calc seams publish that as Bad. The slot is now freed before the caller resumes, idempotently.
   • Pinned by A_caller_that_awaited_its_run_is_never_refused_by_it (1,000 runs).
   • Mutation red.
2. Heartbeat starvation under a compile burst. Each load was compiled synchronously on its own thread-pool work item. On windev, 500 cold loads starved the heartbeat timer and the pipe writer. The host killed the shard (KilledHeartbeat), and all 500 keys became suspects. Compiles now run on min(8, max(1, CPUs/2)) dedicated ScriptWorker.Compile threads.
   • Pinned by A_burst_of_cold_compiles_does_not_starve_the_heartbeat (1,500 cold scripts, 50 ms heartbeat).
   • Red on macOS with the thread-pool version. Green 3/3 on macOS and windev; on windev 1,500 cold compiles take about 18 s.

The host load timeout still counts the time a load waits behind other compiles → lmxopcua#790.

windev re-measure, 2026-11-11 (lmxopcua#789, after the §12.2–§12.3 changes; budget revision applied).

• windev is a VM: VMware20,1 on an Intel Xeon E5-2697 v4 @ 2.30 GHz, 36 logical CPUs, 32 GB RAM, Windows, .NET 10.0.8. The box was otherwise idle (about 4 % total CPU before the run). The issue's "re-measure on a non-VM box" request is therefore still open; these are VM figures.
• Run from C:\build\otopcua-842 (a git archive of the branch head) with -c Release, OTOPCUA_SCRIPT_PERF=1, 10 s phases, filtered to Throughput_and_latency_against_the_accepted_gate and Restart_rss_and_kill_latency. The restart test ran twice; both figures are given.
• Framework-dependent worker built beside the tests (dotnet exec), as in the earlier windev rows.

Measure (K = 2 unless stated)	windev 2026-11-11	§8.2 budget	Verdict
Added latency p50, idle sequential	581 µs (K=1 490, K=4 537; InProcess 134)	≤ 150 µs	MISS
p99 at paced 10k/s	3,220 µs (K=1 5,380, K=4 1,358)	≤ 2 ms	MISS at K=2 (was 1,676 µs in §12.2)
Host CPU at 10k/s, vs InProcess	+131 % (raw 141 %, InProcess 10 %)	≤ 25 %	MISS (not tuned in this task)
Closed loop ×64	54,152 evals/s, 0 not ok	gate ≥ 5,000	PASS
Kill latency after T+G (5 runs × 2)	16 / 11 / 10 / 11 / 12 and 16 / 12 / 8 / 11 / 12 ms	≤ 100 ms	PASS
Kill both shards → every script's next correct result (1,000 cached)	2,135 ms and 2,117 ms	≤ 2,000 ms	MISS by about 6 %
Warm restart, 1,000 cached, fresh runner	1,847 ms and 1,829 ms	≤ 2,000 ms	PASS
Worker RSS after first eval, compiled (max of 2 shards)	160 MB (both runs)	≤ 200 MB + scripts	PASS
Worker RSS after first eval, warm PE cache, no compile	36 MB (both runs)	≤ 200 MB + scripts	PASS
Worker RSS, 500 scripts compiled per shard	236 / 262 MB	— (200 MB + scripts)	—
Worker RSS, 500 scripts loaded from the cache per shard	93 / 96 MB	—	—
1,000 scripts, empty cache, all first results	12,380 / 12,800 ms	—	—

Notes:

• Under the revised budget, RSS passes. Kill latency stays fixed since §12.2: the earlier 210 ms was compile time.
• Kill-both-shards is about 120 ms over the line. It includes the 250 ms first restart backoff and the warm-cache load of 1,000 PEs. The warm-cache restart alone takes about 1.83 s on this VM, so the kill path adds only about 300 ms on top.
• Idle p50 and host CPU are the overlapped-I/O path that §12.2 left open; CPU tuning is deliberately not part of this re-measure. The K=2 p99 at 10k/s reads worse than §12.2's 1,676 µs. K=4 is 1,358 µs in the same run, so this is likely VM scheduling noise on a single run, but it was not re-run to confirm.

Windows ScriptWorker.Tests run, 2026-11-11 (lmxopcua#842). --filter Category=ScriptWorker, Debug, on the same VM. This is the first Windows run of the #827 soft-clock tests, which freeze the worker with NtSuspendProcess / NtResumeProcess.

• From C:\build\otopcua-842: 107 passed, 0 failed, 2 skipped (109), 13 m 37 s, 0 build warnings.
  • All 13 WorkerSoftClockTests cases passed, including the 5 that suspend a real worker process: stalled > T, stalled past T+G, late-starting runaway, Shadow under a stall (two freezes), and a notice that arrives after the transport deadline. No code change was needed.
  • Skipped by design: A_published_self_contained_worker_runs_as_its_own_process (needs OTOPCUA_SCRIPTWORKER_TEST_PATH, §12.1) and the off-Windows identity refusal.
• From C:\Users\dohertj2\otopcua-842: 106 / 1 failed / 2. The failure is WorkerIsolationTests.A_worker_under_the_identity_runs_scripts_with_a_restricted_token_and_a_plain_one_does_not: "The application to execute does not exist: …\script-worker\ZB.MOM.WW.OtOpcUa.ScriptWorker.dll", exit 0x80008081.
  • Cause: the test environment, not the code. The restricted worker token cannot read the worker dll inside a user profile, whose ACL is SYSTEM, Administrators and the owner only. That is the access §6 intends.
  • In production the installer grants the identity read+execute on script-worker\ (§12.5).
  • Run the Windows suite from a directory the BUILTIN\Users group can read, such as C:\build. The §12.5 green run used that.

12.2 lmxopcua#789 / #790 / #788 record (2026-11-10)

What cost the host CPU (#789). With DOTNET_ThreadPool_UnfairSemaphoreSpinLimit=0 a scratch probe showed the out-of-process runner at 33 % of a core at 10k evals/s against 25 % in process: the IPC work itself was about 10 µs per evaluation. The rest was thread-pool spin-waiting, triggered by several pool work items per evaluation, and on Unix by the socket engine. Changes:

• Combining frame writer (ScriptWorkerFrameWriter): the thread that queues a frame writes it itself when no write is in progress. This replaces the channel + writer task (one pool wake-up per submit). Consecutive evaluations or results are merged into one batch, frame order is preserved, and there is one write per pass. The old loop also called PipeStream.FlushAsync, which PipeStream does not override, so it queued a pool work item per pass.
• Buffered frame reader (ScriptWorkerFrameReader): one read pulls in every frame available, and frames are parsed from the buffer. It replaces two reads per frame plus a header allocation.
• One timer per evaluation for T and T+G. It replaces the WaitAsync(timeout) timer, the hard-deadline CancellationTokenSource and their two continuations.
• Slot release and deadline cancel run on the reader, before the caller is resumed. The release no longer depends on a continuation that is not ordered against the caller.
• Result dispatcher (ResultDispatcher, one thread per connection, blocking with Monitor.Wait, no spin): callers resume there, not on the pool. They do not resume on the reader either, because the Calculation pump publishes to OPC UA subscribers after its await. On the reader, a slow consumer would hold up heartbeats and get a healthy worker killed. That is pinned by a test, and a mutation that resumed callers on the reader was red.
• Blocking I/O on Unix (ScriptWorkerFrameCodec.UseBlockingIo):
  • The socket under a Unix .NET pipe turns non-blocking for good after one async call. After that even a "blocking" read goes through the epoll thread plus a spinning wait.
  • Both ends now connect, handshake, read (on a dedicated thread) and write synchronously. This was the largest single gain: sequential host CPU per evaluation fell from about 85 µs to about 13 µs.
  • A full pipe blocks only the one thread draining the writer; every other producer just queues. A peer that stops reading is killed by the heartbeat check within 3 s, which fails that write.
  • Windows keeps overlapped async I/O, because a synchronous full-duplex handle serialises reads against writes. (Superseded in §12.6: Windows now blocks too, through WindowsBlockingPipeStream.)
• Kill via TerminateJobObject on Windows. The job admits one process, so no system-wide snapshot is needed.
• Measured and not kept for memory, because none moved the compiled-worker RSS:
  • ReadyToRun worker publish: 195 MB;
  • GCgen0size 4 MB, GCConserveMemory, TieredPGO=0;
  • an aggressive compacting GC after a compile burst. The memory is Roslyn's JIT'd code and metadata, not the heap.

Before / after (§8 harness, OTOPCUA_SCRIPT_PERF=1; macOS figures under 5–25 load average from other agents on the box).

Budget	macOS before	macOS after	windev before	windev after
Host CPU at 10k/s, K=2, raw (InProcess same phase)	228–240 % (71 %)	79 % (69 %)	153 % (6 %)	123 % (7 %)
… vs InProcess (budget ≤ 25 %)	+157	+10 ✓	+147	+115 ✗
Host CPU, idle sequential, K=2	258 %	45 %	80 %	63 %
Idle p50 (≤ 150 µs)	41–43 µs	23 µs ✓	795 µs	507 µs ✗
p99 at 10k/s, K=2 (≤ 2 ms)	720–796 µs	1,024 µs ✓	1,916 µs	1,676 µs ✓
Closed loop ×64, K=2 (gate ≥ 5k/s)	342k/s	442k/s ✓	51k/s	62k/s ✓
Worker RSS after a compile (≤ 80 MB)	188 MB	187 MB ✗	158 MB	159–161 MB ✗
Worker RSS, warm PE cache, no compile	not measured	58 MB ✓	not measured	36 MB ✓
Warm restart, 1,000 cached (≤ 2 s)	245 ms	206–223 ms ✓	1,715 ms	1,778–1,921 ms ✓
Kill both shards → every next result (≤ 2 s)	512 ms	479–502 ms ✓	2,028 ms	2,004–2,087 ms (borderline)
Kill latency after T+G (≤ 100 ms)	31–35 ms	33–43 ms ✓	~210 ms	10–21 ms ✓

Additional measurements:

• In a scratch probe with a steady 10k/s load, K=2, before vs after: 116–158 % → 49–52 % of a core (InProcess 60–65 %). Sequential runs: about 25k → 46k evals/s.
• The first harness timed the kill from before the runaway's call, so it also counted the cold compile of each distinct runaway key (100–200 ms on the VM). The harness now times from the soft-timeout answer. The windev "before" figure of about 210 ms is therefore mostly compile time, not kill time.
• macOS K=1 p99 at 10k/s read 2.3 ms in one loaded run (budget 2 ms). K=2 is the configuration the budget names.

Still missing after this slice (lmxopcua#789 stays open, further tuning deferred by the owner):

• Windows host CPU for IPC: +115 points at 10k/s on the VM. That path still uses overlapped async I/O, so it pays a completion-port and pool hand-off per frame. The next step is either two simplex pipes with synchronous handles and dedicated threads, or a measurement on a non-VM box.
• Windows idle p50: 507 µs. The VM's in-process baseline is already 92 µs.
• Worker RSS after a compile. The proposed revised budget is in §8.2. An alternative is to recycle a worker from the PE cache after a compile burst (blue/green); that is not built.
• Windows kill-both-shards: about 2.0 s, at the budget line.

#790 — load timeout from compile start.

• The worker sends LoadStarted when a compile thread picks a load up. The host arms that load's timer then, instead of each caller timing from when it sent the load.
• A queued load is never blamed. A compile still running LoadTimeout after its start kills the worker and strikes that key only (LoadEntry.TimedOut).
• Hello.compile_threads is a test knob; 0 keeps the worker's default.
• Tests:
  • Compiles_queued_behind_others_longer_than_the_load_timeout_are_not_killed_or_struck: one compile thread, and a burst grown until it outlasts three timeouts (asserted). Red with the timer armed at send.
  • A_compile_still_running_a_load_timeout_after_it_started_is_killed_and_struck: the positive control, with two kills leading to quarantine and the healthy key unblamed. Red with the timer never armed.

#788 — start-failure logging.

• The first failed start of a down period is one Error with the exception.
• Later retries are a Warning without a stack, carrying the attempt count, at most one per 60 s per shard; the rest log at Debug.
• The start that ends the outage is one Information line: started … after N failed start attempt(s) over T.
• Tests: A_shard_that_cannot_start_logs_one_error_then_concise_rate_limited_retries_then_its_recovery, with two intervals. Red when every failure is logged at Error with the exception.

12.3 Fourth P1 slice: batch submit, exact per-item deadlines, the scripted-alarm seam (2026-11-10)

Batch submit (§4.5, P0).

• IScriptRunner.RunBatch(invocations) returns one task per item. Results are per item and streamed.
  • The default implementation is RunAsync per item, started together. InProcessScriptRunner uses it.
  • OutOfProcessScriptRunner sends every item that is ready to run as ONE EvalBatch frame per worker. Ready means admitted, not quarantined, not a suspect, and already loaded in a live worker at this source and slot order.
  • Anything else goes through the full RunAsync path on its own: a first evaluation that needs a compile, a worker that is down, or a suspect bound for the one-at-a-time isolation worker. A batch therefore never waits for a load or a restart as a whole.
• Seams that use it, on the out-of-process path only:
  • Calculation pump. It dequeues everything pending (each entry's dependency snapshot was taken at enqueue), submits one batch, and publishes in dequeue order. A tag queued twice ends the batch, so every queued evaluation still runs, in order, against the snapshot that triggered it. The published sequence is the one-at-a-time pump's.
  • VirtualTag seam. There is one actor per tag, so an upstream burst makes many tags ready at once from different threads. ScriptBatchCoalescer hands those to one RunBatch. It has no timer: an evaluation on an idle seam is submitted at once by its own caller.
  • Scripted alarms. See below.
• In process, the Calc pump and the alarm engine stay one at a time, unchanged. The VirtualTag in-process path does not touch the coalescer.

Per-item deadlines and blame stay exact under batching.

• The host arms T and T + G at submit. That is when the run starts, provided the worker has an idle evaluation thread.
• A batch larger than the worker's scheduler, or work queued behind a runaway, used to wait in the worker with its clock already running. It could then be timed out, take timeout strikes, and even be the key blamed at T + G.
• The worker's scheduler now hands an evaluation straight to an idle thread, or else puts it in a backlog and reports it in EvalResults.deferred_eval_ids.
  • The host stops that evaluation's clock.
  • When a thread picks the evaluation up, its id rides back in started_eval_ids on the result frame of the run that thread just finished. The host then restarts the full T, then G, from the start.
  • The two notices are unordered on the wire, and the host handles either order.
  • The deadlines are absolute timestamps that the timer re-checks, so a stale tick never fires early.
• A run queued behind a runaway is therefore neither timed out nor struck. At the kill it is a victim (script worker restarted, BadResourceUnavailable, re-evaluated on return), not the culprit.
• Tests:
  • A_runaway_in_a_batch_is_blamed_alone_and_siblings_queued_behind_it_are_not_timed_out_or_struck runs one evaluation thread and a batch of a runaway plus three siblings. It is red with the host ignoring the deferral, and red with the worker never deferring.
  • A_deferred_run_is_timed_and_killed_from_when_it_starts is the positive control. It is red with a start notice that does not re-arm the clock.
• This also changes plain RunAsync under saturation: a run that waits for a thread no longer times out on its wait. That matches the in-process guard, whose budget clock starts when the pool picks the run up (§1.2).

Scripted-alarm seam (★10: alarms last).

• Scripting:Runner:Seams unset now means VirtualTag + Calculation. ScriptedAlarm runs out of process only when it is listed.
• ScriptRunnerSelection serves it. The Runtime's DriverHostActor gets it through the new IScriptRunnerProvider and passes it to the engine (a new last parameter, scriptedAlarmRunner).
• Engine changes, all on the runner path only:
  • LoadAsync compiles each predicate to PE bytes only (CompileToImage). A broken predicate still fails the load exactly as in process, and no assembly is loaded into the host. Deviation: Roslyn still runs in the host at load for alarms; the worker compiles again for execution.
  • Startup recovery and every re-evaluation pass snapshot each dirty alarm's inputs under _evalGate, submit all their predicates as one batch, and then apply the results in the serial order under the gate: Part 9 transition, persist before update, emissions. A test shows the transition sequence is identical to an in-process engine's for the same pushes.
• Failure semantics (§9, alarm column):
  • Timed out, killed, refused or quarantined: the condition is frozen with a BadInternalError annotation, as today.
  • The runner's fault (worker restarted, none ready): frozen with BadResourceUnavailable, consistent with VT/Calc. The alarm is re-evaluated by itself when the runner's Recovery completes.
  • A predicate that threw keeps its state with no annotation, as in process.
  • No outcome but a completed bool reaches the state machine, so a dead worker never clears an alarm.
• Real-worker tests:
  • A_runaway_predicate_is_killed_and_quarantined_its_alarm_frozen_Bad_and_a_sibling_keeps_evaluating;
  • A_worker_that_dies_under_a_predicate_freezes_it_BadResourceUnavailable_never_clears_it_and_it_is_re_evaluated_on_return. It is red when the runner's failure is read as false.
• Budget note: since lmxopcua#779 the host passes Scripting:RunTimeout (2 s) to the engine. The passes below are still judged against the stricter 250 ms the design was written for.

Measured (§8 harness, Batch_throughput_and_alarm_pass_latency; macOS Arm64, 14 CPUs, load average about 14 from other work on the box, 10 s phases):

Batch size	RunBatch evals/s	RunAsync ×n evals/s	RunBatch p50 / p99 µs	RunAsync ×n p50 / p99 µs
1	38,296	40,980	24 / 41	23 / 35
10	147,035	118,516	57 / 121	70 / 142
100	369,304	232,967	215 / 1,095	363 / 1,341
500	463,033	256,232	818 / 2,541	1,701 / 3,178

(OutOfProcess K = 2, one batch at a time. The in-process rows show no difference between the two, as expected.)

Alarms per pass	InProcess p50 / p99 ms	OutOfProcess batched p50 / p99 ms	OutOfProcess serial p50 / p99 ms
100	0.4 / 1.5	0.4 / 2.1	3.6 / 4.9
500	2.2 / 4.7	1.7 / 4.8	16.1 / 18.0
1,000	4.4 / 7.7	2.5 / 41.0	28.1 / 33.4

• A pass is push → the last of the N transitions it emits. "Serial" replays the pre-batch one-predicate-at-a-time pass over the same runner. That is the N × round-trip cost §8.2 warned about: about 30 µs per alarm, against 2–4 µs batched.
• Every configuration is far inside 250 ms. The one 41 ms p99 is a single loaded-box outlier.
• The §8.2 harness rerun on the same box after the scheduler change:
  • K = 2 closed loop ×64: 366k evals/s;
  • paced 10k/s: +5 % host CPU vs InProcess (budget ≤ 25 %);
  • idle p50: 25 µs;
  • kill latency: 14–19 ms;
  • warm-cache restart of 1,000 scripts: 220 ms;
  • every result correct.

windev (Windows VM, 36 CPUs, .NET 10.0.8; the full ScriptWorker.Tests suite also passes there, 157/157):

Batch size	RunBatch evals/s	RunAsync ×n evals/s	RunBatch host CPU	RunAsync ×n host CPU
1	1,810	1,921	63 %	65 %
10	13,013	11,220	109 %	157 %
100	48,685	40,699	129 %	224 %
500	86,322	83,116	143 %	289 %

Alarms per pass	InProcess p50 / p99 ms	OutOfProcess batched p50 / p99 ms	OutOfProcess serial p50 / p99 ms
100	8.2 / 39.0	2.9 / 5.1	51.7 / 64.8
500	42.3 / 53.3	7.4 / 35.7	236.5 / 273.3 (MISS)
1,000	36.6 / 84.2	11.7 / 52.7	466.8 / 519.4 (MISS)

• On the VM, where one round trip costs about 0.5 ms, the pre-batch serial pass misses the 250 ms budget from about 500 alarms. The batched pass stays at p99 ≤ 53 ms for 1,000 alarms, below the in-process engine's own figures. That is the measurement behind §8's "batch submit is a prerequisite, not an optimisation", and the reason ★10 put alarms last.
• On Windows, batching halves the host CPU of a 100–500 evaluation burst. That is the overlapped-I/O path that is still #789's main miss.
• No predicate failed to complete in any pass on either host.
• Rerun of the §8.2 harness on windev:
  • K = 2 closed loop ×64: 50k evals/s, gate PASS;
  • paced 10k/s: +99 % host CPU vs InProcess (#789, unchanged in kind);
  • kill latency: 9–19 ms;
  • warm-cache restart of 1,000 scripts: 1.8 s.

12.4 P2 record: Shadow mode, and the docker-dev drill with alarms (2026-11-10)

What Scripting:Runner=Shadow does (as built).

• The seams selected by Scripting:Runner:Seams (same rule as OutOfProcess: unset = VirtualTag + Calculation, ScriptedAlarm only when listed) receive a ShadowScriptRunner (Core.Scripting) as their runner. They recognise it as an IScriptShadow and keep their in-process path byte for byte: no VT coalescer, the one-at-a-time Calc pump, the in-process alarm engine. After each compiled evaluation they call Mirror(invocation, publishedOutcome). The passthrough fast path and host-lifecycle failures (evaluator disposed, shutdown cancel) are not mirrored.
• Mirror copies the inputs (the alarm engine reuses its read cache), drops the script logger (the in-process run already wrote its lines) and queues the shadow run on the pool. It never blocks and never throws.
• Comparison: value (structural for arrays), exact CLR type, status, failure class (compile; budget = timed out / killed / refused / quarantined, which the two runners reach by different routes for the same runaway; threw with its message; unsupported_type). A shadow outcome that is the runner's fault (Unavailable, worker busy) is not compared (shadow_unavailable). ★11: a value the protocol cannot carry is a mismatch with reason unsupported_type.
• Bounded: Scripting:Shadow:MaxInFlight (256) outstanding and one per script; the rest are dropped and counted (behind | key_busy). One per script also keeps the out-of-process single-flight rule from refusing a shadow run and reporting a false mismatch.
• Observability: otopcua.script.shadow.compared{kind, result}, .mismatch{kind, reason, script}, .dropped{kind, reason}, .inflight. One Warning per script and reason, Script shadow mismatch: <kind> <key> (<reason>) — published …; shadow …, re-armed at the seam's deploy boundary and capped at Scripting:Shadow:MismatchLogsPerMinute (20).
• Readiness: ScriptRunnerSelection.Status() is null under Shadow; the script-runner entry is always Healthy and carries the shadow pool's state and the counters as data. The shadow pool's kill/quarantine lines go to the host log with ScriptRunnerRole=Shadow (the console template does not render that property), never to the script log.
• Deviation from the §12 sketch ("mismatching keys are logged once each"): once per key and reason, plus the per-minute cap.

Tests (all mutation-checked; 16 mutants, all red after one survivor got an assertion):

• ShadowScriptRunnerTests (Core.Scripting, fakes): the comparison table; a divergent script counted per script in the metric with one diff-summary line; the rate limit and suppressed-count note; reset re-arming; a shadow runner that hangs, blocks its caller or throws never reaches the caller and is never a mismatch; MaxInFlight drop accounting; inputs copied before Mirror returns.
• ShadowModeTests (real worker): eleven scripts over short / double / string / arrays / DateTime / bool / status / a throw publish exactly what InProcess publishes and match; a deliberately divergent script (it does a culture-sensitive string.Compare("a", "B"): the worker runs InvariantGlobalization so it compares ordinally and answers differently; originally it read AppContext.BaseDirectory, which lmxopcua#792 then denied) and a decimal result are counted against exactly those two scripts (value, unsupported_type). A runaway that only runs away in the worker is killed twice and quarantined there while every published value stays the in-process one, no published evaluation waits more than 250 ms, and its sibling on the killed shard (a victim, not compared) has no mismatch. Alarms: the emission sequence equals an in-process engine's across a kill -9 of the shadow worker, and no shadow failure annotates a condition.
• Host: selection (Shadow hands out the shadow, Status() null), binding/clamps, the health matrix (Healthy whatever the shadow pool's state), the VT seam's mirror.
• Suites (macOS, Release, 0 warnings, AKKA_TEST_TIMEFACTOR=3): Core.Scripting 255/1 skip, ScriptedAlarms 109, VirtualTags 16, Calculation 84, ScriptWorker 160/4 skip, Host 275, Runtime 1002/3 skip, OpcUaServer.IntegrationTests 133. windev (Windows VM, scratch dir under C:\build): ScriptWorker suite 160 passed / 4 skipped, 0 warnings. The first windev run caught a test bug (a DateTime seconds argument past 59 once the slower VM looped longer), fixed before the commit.
• Found while writing them: scripts could reach System.AppContext (SetSwitch / SetData / GetData) → lmxopcua#792, fixed: System.AppContext, System.Console, GCSettings, ProfileOptimization, EncodingProvider, TypeDescriptor, CryptoConfig and the ambient CultureInfo / TimeZoneInfo.ClearCachedData members are now denied by ForbiddenTypeAnalyzer (in-process and in the worker, one analyzer).

docker-dev drill (image otopcua-host:w34-shadow built from this branch; central-1/central-2 only; drill rows w34-* inserted by SQL: VTs w34mul (HR5×3), w34raw (var v = …HR5.Value; return v; — compiled, returns the boxed ushort), w34div (divergent), w34dec (* 1.5m), Calc W34Calc (HR5+1000), alarms w34-al-hi (HR5>50), w34-al-slow (1.5 s spin, then HR5>50), w34-al-run (while(true) when HR5 == 77). HR5 driven by Modbus writes to the fixture.)

(a) Shadow, all three seams (docker-compose.script-runner-shadow.yml):

Step	Observed (both nodes)
HR5 = 21 / 60 / 30	w34mul 63 / 180 / 90, w34raw 21 / 60 / 30, W34Calc 1021 / 1060 / 1030, HR5x2 42 / 120 / 60, all Good; w34-al-hi and w34-al-slow Active at 60 and Inactive at 30
Mismatch counters (leader)	compared{virtual_tag,match} 146, {calculation,match} 11, {alarm,match} 14; mismatch{virtual_tag,value,w34-vt-div} 5 and {virtual_tag,unsupported_type,w34-vt-dec} 4 — nothing else. dropped{key_busy} 5 (startup burst)
Log	exactly two Warnings per node: w34-vt-div (value) — published Ok Int32 1; shadow Ok Int32 2 and w34-vt-dec (unsupported_type) — published Ok Decimal 7.5; shadow Failed [unsupported_type] … System.Decimal
kill -9 both shadow workers on the leader, then HR5 = 61	Published values all correct and Good (183 / 61 / 1061 / 122); workers restarted within ms; /health/ready 200, script-runner Healthy

• [A-7] answered for Modbus: a UInt16 register arrives boxed as ushort and crosses the boundary unchanged (w34raw matched on every evaluation).
• [A-8] answered: in process, a VT that returns a decimal publishes BadNotSupported (0x803D0000) with a null value: the sink rejects it. Out of process it would publish BadInternalError "unsupported type". Either way it is Bad, so the ★11 behaviour change is only the status code.

(b) OutOfProcess with ScriptedAlarm listed (docker-compose.script-runner.yml + Seams__2=ScriptedAlarm):

Step	Observed (both nodes)
HR5 = 60, then 30	w34-al-hi / w34-al-slow Active, then Inactive, Quality Good; otopcua_script_ipc_roundtrip{kind="alarm"} counting, so the predicates ran in the worker
HR5 = 77 (runaway), 76, 77	exceeded its hard deadline (4000 ms); killing ScriptWorker[0] twice, then quarantined: "it exceeded its hard deadline and its worker was killed (2 time(s) in the last hour)…". w34-al-run stays Inactive with Quality BadInternalError; w34-al-hi and w34-al-slow (same shard) stay Active and Good throughout
HR5 = 30, kill -9 the shard-0 worker 0.6 s later, during w34-al-slow's 1.5 s spin	ScriptWorker[0] ended (Crash) with no deadline crossed; 1 script(s) in flight move to the isolation worker: 3:w34-al-slow. Read at once: w34-al-slow Active, Quality BadResourceUnavailable (0x80040000): frozen, not cleared. By the next read, 19 s later, it is Inactive and Good, re-evaluated in the isolation worker with no further input change. w34-al-hi cleared normally; w34-al-run stays BadInternalError (quarantined)
/health/ready	200, script-runner Healthy ("ready") after the restarts

(The kill loop stopped after one worker per node because its own shell matched the pattern; the killed worker was the one running the in-flight predicate, which is what the step needed.)

Rig state afterwards.

• w34-* rows deleted (Script 3, VirtualTag 1, ScriptedAlarm 1, Tag 55, as before), HR5 = 5, redeployed: revision e221b179…, identical to the pre-drill deployment.
• central-1/central-2 recreated from the base compose file in the same compose project: image otopcua-host:dev (24f9d7287ed2, never re-tagged), no Scripting__* env, compose config hash equal to the base file's. The secret stores live on the otopcua-localdb-central-* volumes since an earlier change, so they survived the recreates: central-1's store is byte-identical; central-2's main file differs only by a WAL checkpoint (same size). Both log Mesh PSK secret ot-mesh-psk-* … is absent at startup — central-1 with an unchanged store logs it too, so that is the state the rig was already in (most likely from the earlier drill's store loss), not something this drill did.
• The drill image otopcua-host:w34-shadow was removed afterwards. The site nodes, sql and traefik were not touched.

12.5 Installer record: worker identity, pipe ACL, firewall, PeCacheDir ACL (2026-11-10)

Owner decisions §14 #5 and #6, as built. Operator reference: docs/ServiceHosting.md §"Worker isolation on Windows".

• Identity, deviating in mechanism from "run as a virtual account". Windows starts a process as NT SERVICE\<name> only when the SCM starts that service. A host cannot launch K + 1 children that way, each with its own nonce, pipe and Job Object. A separate local user would need a stored password plus SeAssignPrimaryTokenPrivilege on the host account. Scripting:Worker:Identity (default unset; Windows only; startup refuses it elsewhere or when malformed) therefore starts each worker under a restricted token derived from the host's (RestrictedWorkerLauncher):

  • DISABLE_MAX_PRIVILEGE, so only SeChangeNotifyPrivilege is left;
  • restricting SIDs = the virtual account's SID (computed from the name, S-1-5-80-…) + Everyone + Users + RESTRICTED;
  • token and default DACL = host + SYSTEM + identity + RESTRICTED;
  • created suspended and put in its Job Object before it runs, inheriting exactly its two stdio pipes;
  • TEMP set to its ACL'd work directory, and DOTNET_EnableDiagnostics=0.

  The installer registers a disabled anchor service, so the account exists and resolves by name.

• Found on windev. The runtime would not start under the restricted token (coreclr.dll → ERROR_DLL_INIT_FAILED). A bisection over the token's groups showed that only the user or Administrators SID fixed it, not the logon SID. The cause was USER32's window station and desktop connection, so the host now grants the identity on its own window station and desktop. The logon SID stays out of the restricting set because the host's process DACL gives it PROCESS_VM_READ. The token's own DACL also needed rewriting (the handle needs WRITE_DAC).

• Pipe ACL. On Windows the pipe always has an explicit, protected DACL: the host account and the identity, read/write, nothing else. .NET's CurrentUserOnly grants the token owner, which is Administrators on an elevated host; this ACL does not. The owner is still the token owner, so the worker's CurrentUserOnly client check is unchanged.

• Installer scripts/install/Install-ScriptWorkerIsolation.ps1 + ScriptWorkerIsolation.psm1. It is idempotent, with -WhatIf, -Uninstall and -PlanOnly, and it sets up:

  • the anchor service;
  • PeCacheDir: protected, SYSTEM / Administrators / host full control, no Users, and the key and markers host-only;
  • pe\: the identity gets Modify;
  • script-worker\: the identity gets read+execute;
  • deny ReadData for the identity on the install folder (appsettings, logs, host binaries);
  • outbound block firewall rule OtOpcUa-ScriptWorker-BlockOutbound on the worker apphost;
  • optional host-service env (-ConfigureHostService).

  The runner flip itself is not part of it.

• Tests.

  • scripts/install/tests/Test-ScriptWorkerIsolation.ps1 is the dry-run harness. It runs on any OS and asserts ids, order, targets, SDDL and the rollback (46 assertions). A mutation check (deny → allow, protected → inherited) turns 4 red.
  • ScriptWorkerIdentityTests (Core.Scripting) pin the SID algorithm against NT SERVICE\TrustedInstaller. The PowerShell harness pins the same default SID.
  • WorkerIsolationTests (ScriptWorker) cover the pipe ACL (built and live), the token, and a real worker under the identity running a script with a restricted token. A plain worker's token is unrestricted (positive control).
    • The token case is restricted, privilege-free, and impersonated: it reads System32 and writes its work dir, but cannot read or write the host's temp.
    • They ran green on windev without changing machine state, apart from the session's window station / desktop ACE. No account, firewall rule or persistent ACL was created there.

• Not verified live: the installer on a real node. That is the QA deploy gate on lmxopcua#780's follow-up (see that issue for the checklist).

• lmxopcua#890 (2026-11-12): sensitive siblings. The QA deploy found the deny covered InstallRoot (app\) only, while the wonder layout keeps pki\ (application certificate private key) and data\ (secrets.db, LocalDb) beside it, inheriting BUILTIN\Users:RX from E:\. Users is a restricting SID, so a worker could read them. The installer now:

  • takes -DenyPaths (the deployment folder around InstallRoot plus each key/secret folder) and gives each the same inherited deny as InstallRoot;
  • widens the deny from ReadData to read and write (0xD0157: ReadData, WriteData, AppendData, WriteEA, DeleteChild, WriteAttributes, Delete, WRITE_DAC, WRITE_OWNER). Traverse/Execute, ReadAttributes, ReadControl and Synchronize stay allowed, so the loader still reaches script-worker\. The write half closes a planted trust-list certificate, since a data drive lets Users create files in subfolders;
  • records the list under the anchor service's key, so a re-run drops stale denies and -Uninstall removes exactly the recorded ones;
  • WARNs about protected-DACL items under a denied folder that the inherited deny cannot reach.

  Proof: InstallerLayoutIsolationTests on windev, a scratch wonder layout under a real restricted token. Before the install and after the uninstall, the token reads every file and plants a trust-list file. Installed, it cannot do either, still reads script-worker\, and a real worker launched from the layout runs a script. With the -DenyPaths denies mutated to a no-op mask, the test goes red on pki\own\private\*.pfx, which reproduces #890.

12.6 lmxopcua#789 host CPU on Windows: blocking pipe I/O off the completion port (2026-11-11)

Already in place before this slice (checked against the issue's candidates): one timer per evaluation for T and T + G (§12.2); the source SHA-256, its hex form and the negative-cache key are computed once per key's source (KeyState.Sha), not per RunAsync; batch submit (§12.3) is used by the Calculation pump, the VirtualTag coalescer and the alarm engine on the out-of-process path. The §8 paced harness phases call RunAsync one at a time on purpose (one trigger per evaluation); Batch_throughput_and_alarm_pass_latency measures the batched path.

What changed.

• WindowsBlockingPipeStream (Protocol). A .NET pipe opened Asynchronous completes even its synchronous Read/Write through the thread pool's completion port, so every frame on Windows cost a completion packet plus a pool work item. The stream issues its own overlapped ReadFile/WriteFile on the pipe handle with the OVERLAPPED.hEvent low-order bit set, which tells the kernel not to queue a completion packet, and waits on that event on the calling thread. Reads and writes stay concurrent (a handle opened for synchronous I/O would serialise them). Dispose cancels a waiting read or write (CancelIoEx) and closes the pipe; a peer that goes away ends a read as end of stream and fails a write with IOException, as PipeStream does.
• ScriptWorkerFrameCodec.BlockingStream(pipe) picks that stream on Windows and the pipe itself on Unix. Both ends now use the dedicated blocking reader thread and the synchronous combining writer on every OS. UseBlockingIo and the two async read loops are gone; HandshakeAsync stays for its tests.
• 1 MiB pipe buffer quota each way (WorkerPipeSecurity.BufferBytes). With a quota of 0 an overlapped write stays pending until the peer has read the bytes, so the blocking writer waited out a peer wake-up per frame. The first cut without it halved the windev closed loop (54k → 26–27k evals/s); with it the closed loop is back.
• Tests: BlockingPipeStreamTests (duplex with reads blocked on both ends, a 1 MB frame, peer gone, self-dispose wakes the reader).

Measured on windev (same VM as §12.1, idle, Release, Throughput_and_latency_against_the_accepted_gate, 10 s phases, base = f109e5f2, runs alternated base / new):

K = 2	base (3 runs)	new (2 runs)	§8.2 budget
Host CPU at paced 10k/s, raw	116 / 118 / 120 %	61 / 84 %	—
… vs InProcess (InProcess 6–9 %)	+108 / +112 / +112	+53 / +78	≤ 25 ✗
Host CPU at paced 5k/s, vs InProcess	+35 / +37 / +38	−26 / −38	—
Idle p50 (sequential)	491 / 499 / 560 µs	363 / 386 µs	≤ 150 µs ✗
Idle host CPU (sequential)	57 / 61 / 66 %	27 / 37 %	—
p99 at 10k/s	2,108 / 2,264 / 2,373 µs	878 / 944 µs	≤ 2 ms ✓ (was ✗)
Closed loop ×64	51k / 54k / 55k evals/s	54k / 55k evals/s	gate ≥ 5k ✓

• The first cut (no buffer quota), two runs: 10k/s raw 118 / 127 %, idle p50 354 / 362 µs, closed loop 26–27k/s.
• Attribution of what is left. With DOTNET_ThreadPool_UnfairSemaphoreSpinLimit=0 the new build reads 71 % raw at 10k/s (+65) and the base 93 % (+85). So pool spinning is no longer what the remainder is made of: it is about 60 µs of host CPU per evaluation in syscalls and thread wake-ups on this VM (reader wake, dispatcher wake, and the harness's own ContinueWith pool hop per result), against about 13 µs on macOS. A non-VM Windows box is still unmeasured.

macOS (Unix path unchanged; 2 runs each, alternated, load average 3–14): K = 2 at 10k/s +9 / +12 vs base +16 / +5 (InProcess 46–57 %); idle p50 26 µs both; closed loop 365–374k vs 364–379k evals/s. No change, as expected.

Tests. ScriptWorker Category=ScriptWorker on windev: 107 passed, 2 skipped by design (109), including all WorkerSoftClockTests suspend cases, the deferred-start / runaway-in-batch deadline tests, the kill and quarantine tests and the restricted-token worker. On macOS: 104 passed, 5 skipped (109). Unit classes BlockingPipeStreamTests, ProtocolTests and HandshakeTests: 67 / 67 on both.

Review follow-ups (same slice). Blocking I/O moves two failure modes onto dedicated threads and producers:

• Reader faults. The host reader runs completion work inline (slot release, load bookkeeping). The frame read and its handling are now caught separately: only the read's I/O exceptions mean the pipe ended. Anything the handling throws kills that worker as ProtocolFault, so its outstanding runs answer "worker restarted" (BadResourceUnavailable) and the shard restarts. Before, an exception escaping a dedicated thread would end the host process, and an InvalidOperationException was swallowed as a clean pipe end, leaving the worker unread until a Crash 2 s later. The worker's reader has the same catch-all (exit ProtocolFault).
• A worker that talks but does not listen. A full pipe blocks the producer that is draining the writer: a seam submitting, a deploy loading scripts, a Cancel. Heartbeats keep the silence check quiet, and with only loads queued nothing else fires. The heartbeat monitor now kills a worker whose pipe write has been blocked for longer than max(10 s, silence limit) (ScriptWorkerFrameWriter.WriteInProgressFor, KilledUnresponsive). That fails the write and releases the producer. The one known way a healthy worker's reader could wedge is closed too: Cancel now runs a script's ctx.CancellationToken callbacks on the pool, not on the worker's reader. StopAsync sends Shutdown off the caller, so its 1 s budget holds on a full pipe.
• WindowsBlockingPipeStream refuses the async surface, because Stream's base versions serialise a write behind a waiting read.
• Tests: PipeFaultTests covers a throwing completion callback (the worker ends ProtocolFault, the sibling in flight gets BadResourceUnavailable, and the shard serves again) and a worker that stops reading while it heartbeats (killed KilledUnresponsive within the bound, and the blocked deploy producer returns). The second uses the Hello.test_stall_reader_on_diag seam. BlockingPipeStreamTests adds dispose with a write blocked on a full pipe, and with a read and a write both pending. The protocol, handshake and pipe classes now carry Category=ScriptWorker. Mutation checks: with no stall kill the stall test is red (timeout); with a catch-all that does not fault, the callback test is red (Crash).
• Re-run: Category=ScriptWorker on windev: 179 passed, 2 skipped by design (181). On macOS: 176 passed, 5 skipped (181). windev perf after the fixes (alternated new / base / new): 10k/s vs InProcess +55 / +68 (base +95 in the same session); idle p50 391 / 365 µs (base 517); closed loop 51k / 54k evals/s (base 57k). No regression against the figures above.

Still missed: Windows host CPU vs InProcess (+53 to +78 against ≤ 25) and Windows idle p50 (363–386 µs against ≤ 150 µs; InProcess alone is about 105 µs on this VM). Next candidates: drop the dispatcher wake when the caller's continuation is known to be cheap (needs a seam-level guarantee, the reason it exists is a slow consumer), or measure on non-VM Windows hardware before tuning further. (The first candidate was measured and dropped: §12.7.)

12.7 lmxopcua#789: dropping the result-dispatcher hop, measured and not kept (2026-11-11)

Question. A worker result wakes three things in turn: the host reader, then the ResultDispatcher thread, then the caller's continuation. Can the dispatcher wake be removed without the caller's code running on the reader?

Where callers resume today.

• Every production consumer resumes synchronously on whichever thread completes PendingEval.Outcome:
  • AwaitSubmittedAsync, the Calculation evaluator, the VirtualTag evaluator, the alarm engine and ShadowScriptRunner all use ConfigureAwait(false). AwaitSubmittedAsync goes through Outcome.WaitAsync(ct) when the token can be cancelled, which also resumes inline in practice.
  • ScriptBatchCoalescer.Link and VirtualTagActor use ContinueWith(ExecuteSynchronously).
  • The coalescer's completion source has no RunContinuationsAsynchronously.
• So that thread runs the runner's mapping (OutOfProcessScriptRunner.Complete) and then the seam's own code. The Calculation pump publishes inline after its await. The alarm engine runs its state machine and persists.
• The runner's mapping is not safe on the reader:
  • A soft-timeout answer calls connection.Cancel, a pipe write that blocks on a full pipe.
  • OutOfProcessScriptRunner.Complete can quarantine a key and log, and can log the return from isolation.
• The dispatcher carries only results and the worker's soft-deadline notice (SoftExpired). Host-clock soft timeouts and transport timeouts already complete Outcome on the timer thread (PendingEval.OnTimer), so their Cancel write never touches the reader.
• One seam would tolerate running on the reader: VirtualTag. After the coalescer's Settle and the evaluator's ToVirtualTagEvalResult, VirtualTagActor only Tells itself; ApplyEvaluation runs later on the Akka dispatcher. A per-submission "resume on the reader" flag set only by that seam could complete real results inline. Soft-deadline notices and worker deaths would still go through the dispatcher, and the runner's rare quarantine and isolation log lines would move to the pool. That cheapness is a property of code in three assemblies (Core.Scripting, Host, Runtime), and nothing would enforce it. Its value was measured before building it (the "inline" rows below).

Variants, selected by a temporary environment switch in WorkerConnection. None was committed.

• pool: no dispatcher. PendingEval completes Outcome with RunContinuationsAsynchronously, so the mapping and the caller resume on the thread pool. This is the pre-§12.2 path, now with blocking pipe I/O. It is safe and costs the same number of wakes as the dispatcher, with a pool thread in place of the dedicated one.
• inline: no dispatcher. Outcome completes on the reader, so the mapping and the caller's continuation run there. It is not shippable for every seam: it is the mutation that §12.2's test caught. It was measured as the most that removing the hop can buy, which is also the most the VirtualTag opt-in above could buy.
• The harness's paced consumer is ContinueWith(…, TaskScheduler.Default) with no ExecuteSynchronously. That is a pool hop of its own per result, and it hides the dispatcher in the 10k/s figure: it is one of the hops §12.6 counts in the ~60 µs. So the last session repeated inline against the dispatcher with that continuation made ExecuteSynchronously (temporary switch OTOPCUA_SCRIPT_PERF_SYNC=1, not committed). That is the shape of the VirtualTag seam (a cheap continuation, no pool hop).

windev (same VM as §12.6, idle, Release, Throughput_and_latency_against_the_accepted_gate, 10 s phases, master = 31be927f, K = 2, three sessions; within a session the runs alternated):

K = 2	dispatcher	pool	inline
Session 1, harness as committed: 10k/s vs InProcess	+79 / +67 / +55	+59 / +64 / +72	—
… idle p50	366 / 328 / 366 µs	386 / 405 / 377 µs	—
… idle host CPU	40 / 36 / 33 %	50 / 49 / 53 %	—
… closed loop ×64	51k / 58k / 59k	88k / 82k / 86k	—
Session 2, harness as committed: 10k/s raw (vs InProcess)	58 / 88 / 65 % (+52 / +81 / +59)	—	71 / 85 / 66 % (+52 / +70 / +56)
… idle p50	376 / 390 / 318 µs	—	316 / 319 / 296 µs
… idle host CPU	31 / 38 / 32 %	—	32 / 29 / 25 %
Session 3, paced continuation synchronous: 10k/s raw (vs InProcess)	57 / 51 / 60 % (+46 / +22 / +32)	—	50 / 49 / 46 % (+41 / +37 / +35)
… idle p50	388 / 392 / 366 µs	—	321 / 334 / 344 µs
… p99 at 10k/s	803 / 755 / 1,180 µs	—	668 / 716 / 732 µs

In session 3 the InProcess baseline at 10k/s read 11 / 29 / 27 % in the dispatcher runs against 9 / 12 / 11 % in the inline runs. That makes the "vs InProcess" column noisy, so the raw host CPU is the figure to compare.

macOS (2 runs per variant, alternated; load average 3–19 from other sessions on the box):

• Harness as committed:
  • dispatcher: idle p50 25–28 µs, idle host CPU 51–63 %, 10k/s vs InProcess −5 to +19.
  • pool: idle p50 35 / 25 µs, idle host CPU 168 / 201 % (thread-pool spin, as in §12.2), 10k/s −4 / +17.
  • inline: idle p50 20 / 25 µs, idle CPU 53 / 44 %, 10k/s +11 / +13.
• Paced continuation synchronous:
  • dispatcher: 10k/s raw 61 / 59 %, idle CPU 55 / 59 %.
  • inline: 10k/s raw 53 / 52 %, idle CPU 46 / 46 %.

Reading.

• pool is not kept.
  • Its 10k/s figure sits inside the dispatcher's spread.
  • It is worse where the dispatcher was meant to help. On windev, in session 1, idle p50 is +20 µs (median) or +36 µs (mean), and idle CPU is about +15 points. On macOS, idle CPU is three times higher.
  • Its one gain is the closed loop: ~85k against ~56k evals/s on windev. The dispatcher's ~56k is already 11 times the accepted gate.
• The most any dispatcher removal can buy is small.
  • Against the harness as committed, inline does not move 10k/s host CPU. Its spread overlaps the dispatcher's.
  • With a cheap synchronous continuation (session 3), inline saves about 8 points of one core at 10k/s raw on windev (mean 48 against 56 %; the ranges are 46–50 and 51–60) and about 7 on macOS.
  • Idle p50 drops by about 50 µs on windev, consistently but with three runs per variant (sessions 2–3: 296–344 against 318–392 µs). On macOS it drops by a few µs.
  • Even at that bound Windows misses both budgets by a wide margin: about +35 to +41 against ≤ 25 at 10k/s, and an idle p50 above 300 µs against ≤ 150 µs (InProcess alone is about 105 µs on this VM).
• The VirtualTag opt-in was therefore not built. It would deliver at most that bound, and only on the VirtualTag seam. In exchange, the reader would come to rely on an unenforced "cheap continuation" property spread across three assemblies. A later edit after any of those awaits would run on the reader unnoticed, which is exactly the heartbeat-starvation failure §12.2 and §12.6 had to design out. If the owner wants the ~8 points regardless, the shape is the one above, plus a test that makes the VirtualTag continuation block and expects the heartbeat to survive.
• Kept: nothing. The code is unchanged, and the dispatcher stays as §12.2 describes it. No tests were re-run, since no code changed. The experiment builds were only measured.

Still missed (unchanged from §12.6):

• Windows host CPU at 10k/s: +52 to +81 against ≤ 25 with the harness as committed. The harness's own pool hop accounts for about 15 points of raw host CPU (session 3's dispatcher runs, mean 56 %, against sessions 1–2, mean 72 %).
• Windows idle p50: about 320–390 µs against ≤ 150 µs.

What is left after this slice: the reader's own wake per frame and the syscalls behind it (about 60 µs per evaluation on this VM against about 13 µs on macOS, §12.6), and the worker side of the round trip (its reader, the dedicated evaluation threads and its writer), which this slice did not touch. A non-VM Windows box is still unmeasured.

12.8 lmxopcua#789: where the Windows cost goes, spin-before-block measured, nothing kept (2026-11-12)

Question. Which §8.2 budgets still miss on windev, what dominates each, and does any targeted change move them? master = a28adba2 (runner code identical to 31be927f). Same VM as §12.6–§12.7 (VMware20,1, Xeon E5-2697 v4, 36 vCPUs, Windows 10 19045, .NET 10.0.8), idle apart from other agents' builds (machine CPU 4–11 % around each run). The mxaccessgw service on the box was not touched.

Where each budget stands (Throughput_and_latency_against_the_accepted_gate, Release, 10 s phases, four runs; and Restart_rss_and_kill_latency, two runs):

Budget (§8.2)	windev	Verdict
Idle p50 ≤ 150 µs, K = 2	382 / 386 / 388 / 355 µs (InProcess 132–143 µs)	MISS
Host CPU vs InProcess ≤ 25 at paced 10k/s, K = 2	+71 / +76 / +84 / +82 (raw 77–89 %, InProcess 4–7 %)	MISS
p99 at paced 10k/s ≤ 2 ms	818 / 901 / 1,348 / 841 µs	PASS
Closed loop ×64, gate ≥ 5k/s	58k / 50k / 49k / 56k evals/s, 0 not ok	PASS
Worker RSS ≤ 200 MB + scripts (compiled / warm PE cache)	161 / 160 MB; 36 / 36 MB	PASS
Kill latency after T + G ≤ 100 ms	10–19 ms	PASS
Warm restart, 1,000 cached ≤ 2 s	1,804 / 1,778 ms	PASS
Kill both shards → every next result ≤ 2 s	1,949 / 2,100 ms	borderline (as in §12.1)

At paced 5k/s (the accepted planning envelope, [A-14]) the out-of-process host CPU is 22–43 % raw, below InProcess's 61–70 % in the same phase. The 10k/s phase is where it misses.

The floor on this VM. A scratch microbenchmark (not committed) timed the primitives the runner is built from, 20,000 round trips each:

Round trip (2 wake-ups)	windev p50	windev CPU / round trip	macOS (M-series, loaded) p50 / CPU
Two threads, AutoResetEvent	52 µs	37 µs	5.5 µs / 6.3 µs
Two threads, Monitor.Wait/Pulse	55 µs	38 µs	5.7 µs / 6.7 µs
Two processes, 8-byte ping-pong over WindowsBlockingPipeStream	84–91 µs	host 35–56 µs, peer 41–50 µs	—
… same, both ends spin 20 µs before blocking	28 µs	host 31 µs, peer 34 µs	—

So one blocking hand-off costs ~26 µs of latency and ~18 µs of CPU on this VM, about ten times macOS. An idle evaluation makes four of them in a row (thread accounting below): host writer → worker reader, worker reader → evaluation thread, evaluation thread → host reader, host reader → result dispatcher. The bare cross-process round trip alone (84–91 µs) plus the one in-worker hand-off that pre-emption needs (§3.2) is ~115 µs before any protocol work, and InProcess on this VM is already 132–143 µs. 150 µs is not reachable here without spinning.

Where the host CPU goes at 10k/s. Two scratch tools (not committed): per-thread CPU and context switches from NtQuerySystemInformation around one phase, and an ETW CPU-sampling trace (kernel profile at 0.5 ms with stacks, CLR rundown; analysed with TraceEvent).

• Per thread, K = 2, paced 10k/s, host 78.8 µs per evaluation: pool threads 42.9 µs (the harness's pacing loop, which submits, plus its per-result ContinueWith hop; 0.57 context switches per evaluation), the two readers 20.2 µs (0.35), the two dispatchers 11.4 µs (0.36). One reader wake costs ~57 µs of reader CPU including the frame's handling.
• ETW, host process: 48 % of samples in ntoskrnl. By syscall: NtWriteFile 14.6 % (one submission frame per RunAsync), thread-pool wake NtSetIoCompletion 7.8 % plus pool idling (NtRemoveIoCompletion, NtDelayExecution) 5.2 %, NtSetEvent 5.2 %, NtReadFile 4.4 %, blocking waits 5.7 %, TimerQueue.SetTimer 1.0 %. Inclusive: RunAsync 36.7 % (of which Submit 24.4 %, the pipe write 15.7 %, the deadline timer 1.6 %), the reader loop 22.8 %, the dispatcher 13.6 %. Managed code that is not a syscall or a wake is a few percent per frame.
• Idle sequential: exactly 2.01 context switches per evaluation on the host (reader 1.00, dispatcher 1.00) and 2.07 in the worker (reader 1.00, evaluation thread 1.00) — the four hand-offs above.

What is left is syscalls and wake-ups, each several times dearer on this VM than on macOS. The issue's candidates are all in place (§12.6), and the dispatcher hop was measured in §12.7.

Spin before block, measured, not kept. A temporary environment switch (not committed) made each hand-off spin before it blocked: the pipe reads (on the OVERLAPPED status, no syscall per poll), the result dispatcher, and one evaluation thread per worker. Per-thread accounting, one run each, same method:

Spin (µs): host pipe / worker pipe / dispatcher / eval	idle p50	host µs/eval idle	host µs/eval at 10k/s
none	358	128	78.8
– / 50 / – / –	357	122	70.3
50 / 50 / – / –	343	175	85.6
50 / 50 / 50 / –	341	225	99.7
50 / 50 / 50 / 50	328	216	87.8
400 everywhere	118	253 (host 191 % of a core while idle-sequential)	260
– / 400 / – / 400 (worker only)	221	106	82.8 (worker ~390 % of a core)

• A spin shorter than the ~330 µs cycle never catches anything in the sequential phase: each hand-off waits for the whole rest of the round trip.
• Spinning longer than the cycle meets 150 µs (118 µs), but at about two host cores and four worker cores busy for a single stream, and +250 points at 10k/s. That trades one budget for a much worse miss of the other.
• Worker-only spinning cuts idle p50 to 221 µs and does nothing for host CPU.
• The one cheap host gain, a worker-side pipe spin of 50 µs (−8 µs/eval), sits inside run-to-run noise.

The production submission shape. The paced phase calls RunAsync once per evaluation on purpose. The seams do not: the Calculation pump and the alarm engine call RunBatch per pass, and the VirtualTag coalescer batches whatever became due together (§12.3). The harness now has a row for that shape, paced 10,000/s, batched: the same rate, keys and per-result continuation, with each 1 ms tick's evaluations submitted in one RunBatch call. It runs after the original four phases, so their rows stay comparable with every earlier record.

• windev, three runs: host CPU +36 / +41 / +45 (raw 44–52 %, against 83–89 % unbatched). One submission frame (one NtWriteFile) per tick and shard instead of per evaluation halves the host cost. Still above 25.
• Its latency is measured from the batch's submit, so it includes queueing behind siblings: p50 678–741 µs, p99 3.9–6.1 ms. InProcess through the same phase has a p99 of 1.2–4.9 ms on this VM, so most of that comes from the harness shape, not the transport. This row has no §8.2 latency budget.
• macOS, one run (load 6–9): +40 batched against +32 unbatched, so no gain there. The Unix path has no per-write completion cost to save.

Worker-side findings (not budgeted; for the record):

• BlockingCollection.Take (its SemaphoreSlim spin, yield and wait) is 35 % of worker CPU at 10k/s, and SemaphoreSlim.Release → NtSetEvent is 8 % (the reader waking an evaluation thread). Evaluating the script is 7 %.
• A worker's .NET Tiered Compilation Worker runs at ~100 % of a core for tens of seconds after the harness warms 2,000 keys, because every key's script is its own assembly and tiers up on its own. It is transient and costs worker CPU, not host CPU, but it competes for cores on a busy node right after a deploy.

Kept: the harness row only (ScriptRunnerPerfTests.PacedAsync(…, batched: true)). No runner change: nothing measured moved a budget without making another one much worse.

Still missed on windev: idle p50 (355–388 µs against 150) and host CPU at 10k/s (+71 to +84 unbatched, +36 to +45 batched, against 25). macOS still meets both (§12.6–§12.7). The owner's decision is §14 #15.

---

13. Test plan

Unit (Core.Scripting.Tests, new ScriptWorker.Tests):

• Frame codec:
  • round-trip every Variant.TypeTag (arrays included);
  • reject zero-length, oversize and truncated frames;
  • reject an eval_id with no outstanding request;
  • fuzz the reader with random bytes and assert there is no exception other than a protocol fault.
• Unboxing parity: for each tag, a script doing (T)ctx.GetTag("x").Value behaves identically in and out of process. This pins [A-7].
• Attribution ledger:
  • a deadline kill blames exactly one key;
  • a crash with 3 in flight → 3 suspects → isolation;
  • an isolation crash → one strike;
  • probation returns a suspect to its shard;
  • quarantine after 2 kills;
  • a source change lifts the quarantine;
  • a redeploy lifts every quarantine.
• Backoff schedule, jitter bounds, the reset-after-healthy rule and the load-phase crash bisection.
• Options binding: clamps plus a Warning per clamped key; PeCacheDir relative or inside the content root is a startup failure.

Containment (worker-backed, Category=ScriptWorker; runs in CI on Linux, gated on windev for Windows). Each case has a positive control: a healthy sibling script on the same shard that keeps producing correct values before and after the event. Following scadaproj memory "assert the artifact before the outcome", each assertion is checked to be capable of failing.

Case	Assert
while(true){} ignoring the token	Killed within T + G + 100 ms; host ThreadPool.ThreadCount and Process.Threads unchanged; no orphan (InFlightCount == 0 after the kill). Mutation check: disable the kill and see the test fail
while(!ctx.CancellationToken.IsCancellationRequested){}	Ends at T by itself; no kill; no strike beyond the timeout
Allocation bomb	Worker dies of OOM or the heap limit; host RSS stays below a bound and the host stays up; suspect → isolation → quarantine
Unbounded recursion	Worker dies; host up; quarantined after the isolation reproduction. Also the first in-repo reproduction of [A-1] for the in-process mode, marked Skip there because it would kill the test host
Worker killed externally (kill -9)	Victims Bad with "script worker restarted", then recover with the latest inputs; the alarm condition is not cleared
Host killed	Worker exits within 2 s (pipe EOF on Linux, Job Object on Windows)
Environment	The worker's environment contains no ServerHistorian__ApiKey / ConfigDb variables (set them in the test host first: positive control)
Pipe	A second client connecting to the pipe is refused; a wrong nonce is refused
Loaded assemblies	The worker's loaded set equals the pinned list; Akka / SqlClient / Opc.Ua are absent
Crash loop	A poison generation trips the circuit; state Unavailable; keys Bad with the runner-unavailable reason; recovery after a fixed redeploy

Parity: run the existing RoslynVirtualTagEvaluatorTests, CalculationDriver*Tests and ScriptedAlarmEngine*Tests against both runners. Use xunit theory or class fixtures over IScriptRunner, with the worker built as a test dependency.

Performance (Category=Perf, opt-in, not in CI): a Stopwatch-based harness (no new packages) that measures the §8 table: p50/p99 added latency, throughput at K=1/2/4, host CPU, restart-to-first-result with 1,000 cached PEs, and RSS per shard.

Live gates:

• docker-dev rig: a deploy with VTs, Calc tags and scripted alarms; drill a runaway script through the AdminUI and watch /uns go Bad, the quarantine appear, and siblings stay Good;
• QA shadow week (P2) — dropped by the owner 2026-11-12;
• a QA flip with the same drill. Done 2026-11-12 on vq03/vq04 (record above).

---

14. Open decisions for the owner

Each has a recommendation, marked ★.

1. Pool shape.
   • ★ K=2 shards plus an on-demand isolation worker;
   • one worker per node, simplest, but every kill blips every script;
   • per-seam workers.
2. Hard grace G.
   • ★ max(1 s, T), which keeps cooperative cancellation useful;
   • G = 0 (kill at T): strictest, but every soft overrun restarts a shard.
3. Status code for "runner unavailable / worker restarted".
   • ★ BadResourceUnavailable, which separates a sick node from a broken script but is a ScadaBridge-visible contract change;
   • keep BadInternalError everywhere, for zero consumer impact.
4. Fallback when the worker cannot start.
   • ★ fail closed: Bad, and no in-process execution;
   • fail open to in-process with an Error, which keeps values flowing but silently reopens the risk.
5. Worker identity on Windows.
   • ★ a dedicated low-privilege virtual account created by the installer, which needs installer work, an explicit pipe ACL, and a deploy that touches the box;
   • the same account as the host: zero install change, and the Job Object plus the analyzer remain the controls.
6. Network isolation.
   • ★ a Windows Firewall outbound-block rule on the worker executable (installer), with the analyzer as the only control on Linux;
   • analyzer only, everywhere.
7. Compile location.
   • ★ in the worker: Roslyn blowups are contained and the host gets lighter;
   • in the host, shipping PE bytes: a lighter worker and faster cold start.
8. PE cache store.
   • ★ a node-local directory with an HMAC;
   • LocalDb (already a dependency; it would also replicate to the pair, which is unnecessary);
   • no persistence, recompiling on restart (violates the 2 s budget at scale).
9. IPC encoding.
   • ★ protobuf (already compiled in Commons, and mxaccessgw precedent);
   • MessagePack (pinned but unused, so in practice a new dependency).
10. Scripted alarms in the first flip.
    • ★ last, after Calc and VT, because of the 250 ms budget and the engine refactor;
    • all seams at once.
11. Unsupported input/result CLR types.
    • ★ an explicit Bad UNSUPPORTED_TYPE, surfaced in shadow mode first;
    • best-effort conversion (to string, double), which hides type bugs.
12. Readiness when the circuit is open.
    • ★ degraded but ready (OPC UA still serves driver data);
    • not ready, which on a redundant pair would shift ServiceLevel to the peer.
13. Long-term status of InProcess.
    • ★ keep it as the dev/test default and the rollback;
    • delete it after a soak period.
14. Does this need a real-deployment scale target? [A-14]'s 5k evals/s/node is a guess. The owner should supply the expected VT / Calc / alarm counts per node for the target sites, or accept the guess as the gate.
15. DECIDED 2026-11-12: the owner accepts the measured windev values. §8.2 now states per-platform budgets: Windows VM idle p50 ≤ 500 µs and ≤ 50 points batched at 10k/s. No bare-metal measurement is required, and no transport change will be made. lmxopcua#789 is closed. Windows idle-latency and host-CPU budgets (lmxopcua#789, §12.8). On the windev VM one blocking hand-off costs ~26 µs of latency and ~18 µs of CPU, ten times macOS, and an evaluation needs four of them. Idle p50 is 355–388 µs (InProcess alone 132–143 µs). Host CPU at 10k/s is +71 to +84 points one evaluation at a time and +36 to +45 batched, against ≤ 150 µs and ≤ 25.
    • ★ Measure on bare-metal Windows, then state the two budgets per platform. Keep 150 µs / 25 points where they are met (Linux and macOS; Windows hardware if it meets them). If a VM is the target, set the Windows figures from that box, measured with the batched row as the gate. For example, idle p50 ≤ 500 µs (0.2 % of the 250 ms alarm budget) and ≤ 50 points batched at 10k/s, with 5k/s, the planning envelope [A-14], already below InProcess. No code change.
    • A shared-memory ring transport: frames in a mapped section, with an event signalled only when the peer is asleep, and a bounded spin. It would remove most NtWriteFile/NtReadFile calls and wakes under load. Idle latency still needs ≥ 2 cross-process wakes unless a thread spins, and §12.8 measured spinning at about two host cores for one stream. It is a new transport, with a new ACL'd kernel object, new fault modes and new tests. Worth it only if the target load from #14 makes ~0.4–0.9 of a host core at 10k/s matter.
    • Spin before blocking (§12.8): rejected. It meets 150 µs only by keeping cores busy.

---

15. Assumptions (not verified from code)

Id	Assumption	How to verify
A-1	Unbounded recursion in a script throws StackOverflowException and kills the host today	Reproduce in a throwaway process (not the test host)
A-2	A collectible ALC whose code is still executing on an orphan thread cannot be unloaded	.NET docs / a WeakReference test with a spinning orphan
A-3	A minimal .NET 10 worker idles at tens of MB RSS	Measured (§12.1/§12.2): true only for a worker that has never compiled (36–58 MB); a worker that has run Roslyn idles at 159–187 MB, so §8.2 was revised to 200 MB
A-4	P = max(2, cores/K) evaluation threads per worker is enough	P1 perf harness
A-5	Passthrough scripts are a large share of evaluations on real deployments	Count PassthroughScript.TryMatch hits against compiled runs on QA (add a counter in P0)
A-6	A local pipe round trip costs tens of µs	P1 perf harness
A-7	Drivers box tag values with stable, enumerable CLR primitive types	Shadow-mode type census
A-8	What OpcUaPublishActor / the sink does with an unsupported result object today	Trace AttributeValueUpdate → SdkAddressSpaceSink
A-9	Drivers produce UTC DateTime values	Shadow-mode census on DateTime.Kind
A-10	512 MiB is a sane per-worker heap limit (ScadaBridge uses the same)	Size against measured RSS × script count
A-11	docker-dev containers cannot create nested cgroups	Check the compose capabilities
A-12	Warm Roslyn compile is tens of ms per script; cold is seconds	P1 perf harness
A-13	Worker start is ~150 ms; a PE load into a collectible ALC is ~1 ms	P1 perf harness
A-14	5k compiled evals/s/node is a fair planning envelope	Owner input (decision 14)
A-15	Today's in-process overhead per compiled run is µs to tens of µs	Add the same harness to InProcessScriptRunner in P0 for a baseline

---

16. Prior art consulted

• ScadaBridge CentralUI.SandboxWorker (~/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.CentralUI/ScriptAnalysis/SandboxWorkerProtocol.cs). It runs one disposable process per test run over stdio, using 4-byte LE length-prefixed JSON frames with a 1 MiB cap, admission 4, DOTNET_GCHeapHardLimit 512 MiB, and Kill(entireProcessTree) on the deadline. Taken from it: the heap limit, the tree kill and the frame cap. Not taken:

  • stdio (fragile for a long-lived worker);
  • per-run process spawn (too slow for runtime evaluation);
  • inheriting the parent environment;
  • proxying host calls.

  ScadaBridge's runtime scripts are still in-process (dedicated scheduler plus watchdog, docs/plans/2026-08-15-script-pool-split-design.md), so this design would be the family's first out-of-process runtime evaluator.

• mxaccessgw gateway↔worker (~/Desktop/MxAccessGateway/docs/WorkerFrameProtocol.md, WorkerProcessLauncher.md, WorkerBootstrap.md). It runs a long-lived supervised worker over a named pipe: protobuf envelopes, a negotiated frame cap, a priority-drained writer with coalesced flushes, sequence numbers, a nonce in an env var (never on the command line), startup probe plus retry, and a tree kill on a failed start. Taken from it: the frame protocol shape, the nonce handshake and the launcher discipline. Avoided: an orphan terminator that is not scoped to its own instance.