lmxopcua/docs/plans/2026-11-09-out-of-process-script-runner-design.md

# Out-of-process, pre-emptible script runner — design

**Date:** 2026-11-09 · **Status:** proposed (design only — no code in this change) · **Issue:** #750 · **Related:** #734 (Lane E remediation)

This document designs a script runner that can actually **stop** a user script, instead of abandoning it.
It covers the three script seams that run operator-authored C# on a driver node: virtual tags, Calculation
tags and scripted-alarm predicates.

Claims about today's code carry `file:line` evidence against `origin/master` @ `94926eb0`. Anything not
read from the code is marked **[A-n]** and listed in §15.

---

## 1. Today: in-process execution

### 1.1 One compile pipeline, three seams

All three seams share `Core.Scripting`:

- **Compile.** `ScriptEvaluator<TContext,TResult>.Compile` wraps the user source in a synthesized
  `CompiledScript.Run(ScriptGlobals<TContext>)`, compiles it with Roslyn against the `ScriptSandbox` reference
  set, runs `ForbiddenTypeAnalyzer`, emits PE bytes to a `MemoryStream` and loads them into a collectible ALC
  (`src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting/ScriptEvaluator.cs:125` emit, `:139` `LoadFromStream`).
- **Cache.** `CompiledScriptCache` is keyed by SHA-256 of the source (`CompiledScriptCache.cs:157`). It is
  **in memory only**, so every process start recompiles every script on first use.
- **Execution.** It is synchronous: `ScriptEvaluator.RunAsync` invokes the bound delegate inline and returns
  `Task.FromResult` (`ScriptEvaluator.cs:178-195`). The script-visible `ctx.CancellationToken` is an
  `AsyncLocal` set for the duration of the call (`ScriptEvaluator.cs:191`,
  `Core.Scripting.Abstractions/ScriptContext.cs:96-98`).
- **The context is a pure read cache.** `VirtualTagContext.GetTag` and `AlarmPredicateContext.GetTag` serve
  only from a dictionary handed in by the caller (`VirtualTagContext.cs:50`, `AlarmPredicateContext.cs:40`).
  `SetVirtualTag` is dropped in both production single-tag evaluators
  (`Host/Engines/RoslynVirtualTagEvaluator.cs:129-131`, `Driver.Calculation/CalculationEvaluator.cs:185-188`)
  and throws in alarm predicates (`AlarmPredicateContext.cs:50`). Inputs are known statically, because
  `DependencyExtractor` rejects non-literal `GetTag` paths at publish.

  This matters most for the design (§4.1). **A script needs nothing from the host while it runs**: the
  inputs go in, one value comes out, and log lines are a side channel.

| Seam | Where it runs | Budget | Serialisation | Evidence |
|---|---|---|---|---|
| Virtual tag | `VirtualTagActor` message handler calls `IVirtualTagEvaluator.Evaluate` **synchronously** on the actor's dispatcher thread; `RoslynVirtualTagEvaluator` blocks with `.GetAwaiter().GetResult()` | 2 s | per actor (one actor per VT) | `Runtime/VirtualTags/VirtualTagActor.cs:276`; `RoslynVirtualTagEvaluator.cs:70`, `:174-175`; singleton `Host/Program.cs:393-398` |
| Calculation tag | async pump in `CalculationDriver`, one evaluation at a time per driver instance | 2 s (`runTimeoutMs`) | `_evalGate` = `SemaphoreSlim(1,1)` per driver | `CalculationDriver.cs:99`, `:523-556`, `:628`; `CalculationDriverOptions.cs:13`; `CalculationEvaluator.cs:232-233` |
| Scripted alarm | `ScriptedAlarmEngine.ReevaluateAsync` walks the dirty set **serially** under one gate | **250 ms** (the engine is built without `scriptTimeout`, so it takes `TimedScriptEvaluator.DefaultTimeout`) | `_evalGate` per engine | `ScriptedAlarmEngine.cs:204`, `:257`, `:743-765`, `:833-835`; `Runtime/Drivers/DriverHostActor.cs:1251-1252` |

Both value seams skip Roslyn for the mirror shape `return ctx.GetTag("X").Value;` (`PassthroughScript.TryMatch`,
`RoslynVirtualTagEvaluator.cs:111-118`, `CalculationEvaluator.cs:167-176`).

### 1.2 Timeout and cancellation model

Every compiled run in production goes through `ScriptRunGuard.RunAsync` (`Core.Scripting/ScriptRunGuard.cs:170-304`).
There are three guard instances per seam owner: one per `RoslynVirtualTagEvaluator`
(`RoslynVirtualTagEvaluator.cs:71`), one per `CalculationEvaluator` (`CalculationEvaluator.cs:88`) and one
per `ScriptedAlarmEngine` (`ScriptedAlarmEngine.cs:258`).

1. **Single flight per key.** A CAS on `state.InFlight` refuses a second run while the previous one, an
   orphan included, still holds the slot. The caller gets `ScriptRunRefusedException` →
   Bad `"previous evaluation still running"` (`ScriptRunGuard.cs:188-219`).
2. **The run is pushed to the pool** (`Task.Run`, `:231-249`), so a CPU-bound script cannot run inline and
   complete before the timeout is registered. The budget clock starts only when the pool picks the run up
   (`:268`).
3. **The wait is `runTask.WaitAsync(timeout, ct)`** (`:273`). On timeout the guard cancels the run token
   (`:279`). Only a script that polls `ctx.CancellationToken` can see that.
4. **Quarantine** happens on any of three triggers:
   - N consecutive timeouts (`:332-341`, default 3);
   - a refusal behind an orphan older than `StuckOrphanQuarantineFactor` × the timeout (`:199-216`,
     default 10×);
   - an allocation breach, where `GC.GetAllocatedBytesForCurrentThread` is sampled before and after the
     delegate (`:235`, `:241`, `:262-263`, `:296-302`; default 64 MB, `ScriptingOptions.cs`).

   A quarantine lifts on a source change or a redeploy (`ResetAll`, called from `ClearCompiledScripts` at
   `RoslynVirtualTagEvaluator.cs:212-218` and from `ScriptedAlarmEngine.LoadAsync`).
5. **What the operator sees.**
   - A VT or Calc failure publishes `BadInternalError` with the last-known value, once per Good→Bad
     transition (`VirtualTagActor.cs:384-392`, `CalculationDriver.cs:640-647`).
   - An alarm whose predicate did not complete keeps its frozen condition state, annotated
     `BadInternalError` (`ScriptedAlarmEngine.cs:895-907`).

`TimedScriptEvaluator` (`TimedScriptEvaluator.cs:80-83`) is the older wrapper. Its own remarks say it is now
used only by single-shot callers (`:18-27`).

### 1.3 Why a runaway script cannot be pre-empted

.NET has no supported way to stop a managed thread. `Thread.Abort` is gone in .NET Core, and the guard's
`WaitAsync` returns control to the **caller**; it does not stop the run. Concretely:

| Failure | What happens today | Evidence |
|---|---|---|
| `while(true){}` (ignores the token) | The pool thread spins **forever**, holding one core and its collectible ALC. The guard bounds this to one orphan per key and then quarantines, but the thread keeps spinning until the process restarts. N such scripts peg N cores. | `ScriptRunGuard.cs:17-22` remarks; `ScriptRunGuardTests.cs:170-205` asserts *exactly one* orphan stays in flight (`InFlightCount.ShouldBe(1)`) |
| Fast allocation (`new byte[int.MaxValue]` in a loop, or a growing list) | The sample is taken **after** the delegate returns, so the process can OOM first. The whole server dies, including OPC UA sessions and every driver. | `ScriptRunGuard.cs:39-43`; `docs/VirtualTags.md:40`; review `docs/reviews/2026-10-21/05-core.md:100-105` |
| Unbounded recursion (a local function or a self-referencing `Func<>`; neither is a class member, so `EnforceSingleRunMember` does not see it) | `StackOverflowException` cannot be caught in .NET and **terminates the host process**. **[A-1]** I did not reproduce this; it follows from runtime semantics, and no analyzer rule targets recursion (`git grep -i "recursion\|StackOverflow"` in `Core.Scripting` finds nothing). **Update (lmxopcua#779):** reproduced out of process (exit 134, "Stack overflow.") and fixed in-process: the compile injects `ScriptStackGuard.Check()` into every script body, so the script now fails with a catchable `ScriptStackExhaustedException`. | `ScriptEvaluator.cs:228-281` (member check only) |
| Collectible-ALC reclaim of a stuck script | `ClearCompiledScripts` disposes the evaluator, but an ALC cannot be collected while a thread is executing its code, so a stuck orphan pins its assembly until restart. **[A-2]** This is .NET ALC semantics, not observed here. | `CompiledScriptCache.Clear`; `ScriptEvaluator.Dispose` |
| A slow (not runaway) script | VT: blocks a default-dispatcher thread for up to 2 s per evaluation. Calc / alarm: holds the driver-wide or engine-wide gate, so every sibling waits behind it (up to 2 s / 250 ms each). | `RoslynVirtualTagEvaluator.cs:174-175`; `CalculationDriver.cs:539-541`; `ScriptedAlarmEngine.cs:743-765` |

The sandbox (`ForbiddenTypeAnalyzer.cs:62-80`: `System.IO`, `System.Net`, `System.Diagnostics`,
`System.Reflection`, `System.Threading[.Tasks]`, `Environment`, `GC`, …) blocks *capabilities*, and it does
that well. It cannot block *consumption* (CPU, memory, stack). Every review that looked at this reached the
same conclusion:

- `docs/VirtualTags.md:40,54`;
- `docs/ScriptEditor.md` §"Runtime safety";
- `docs/superpowers/plans/2026-10-22-review-remediation.md` Lane E2;
- `docs/reviews/2026-10-21/05-core.md:103` and `03-host-controlplane-security-config.md:38`.

All of them say the fix is an **out-of-process runner**.

Script authoring is gated by the `ConfigEditor` policy, which admits Administrator **or Designer** (CLAUDE.md
§Scripting). The issue says "admin-gated", but the gate also admits Designers. That is the primary control,
and it explains why this work is p3.

---

## 2. Goals and non-goals

**Goals**

- G1. **Pre-emption.** A script that exceeds its budget is stopped within a bounded time. The thread, the
  memory and the ALC are all reclaimed.
- G2. **Containment.** No script (OOM, stack overflow, `FailFast`-equivalent, native crash) can take down the
  OtOpcUa host process, its OPC UA sessions, its drivers or its Akka membership.
- G3. **Attribution.** The script that caused a kill is identified and quarantined. Healthy scripts that
  shared the worker are not.
- G4. **Parity.** Compile-time behaviour is unchanged: the same sandbox, analyzer, diagnostics and Monaco parity.
  Runtime semantics are unchanged too: the same `ctx` API, the same input-quality gating, the same Bad / last-value
  contract, and the same quarantine lift rules.
- G5. **Bounded overhead** (§8). It must fit the tag counts the repo already tests at.
- G6. **Flag-gated, reversible rollout.** In-process stays the default until the gates pass.

**Non-goals**

- N1. A new sandbox. `ForbiddenTypeAnalyzer` stays the capability control; process isolation is
  defence-in-depth, not a replacement.
- N2. Moving the **authoring** surface. The AdminUI only compiles and analyzes (`ScriptAnalysisService`), and it
  never executes a script, so central/admin nodes are out of scope.
- N3. Multi-tag cascades (`ctx.SetVirtualTag` taking effect). They are dropped today and stay dropped.
- N4. Cross-node or remote execution. The worker is always local to the driver node that owns the tags.
- N5. Per-run memory *pre-emption*. Even out of process, the limit is per *worker*. A per-run limit stays a
  sampled observation (§5.4).
- N6. Running non-.NET or untrusted third-party code.

---

## 3. Process model

### 3.1 Options

| Option | Blast radius of a kill | Process count | Verdict |
|---|---|---|---|
| A. One worker per node | Every script on the node blips | 1 | Simple, but one bad script restarts every script on the node |
| B. One worker per script | One script | = script count (hundreds to thousands) | Infeasible: ~tens of MB per .NET process **[A-3]** |
| C. One worker per seam or driver instance | One seam / driver | 3 to ~10 | Arbitrary; a hot Calc driver still shares fate with its siblings |
| **D. Small sharded pool + an isolation worker** | One shard (1/K of the scripts), then one script | K + 0..1 (default 2 + 1) | **Recommended** |

### 3.2 Recommended: sharded pool with an isolation worker

- **K shard workers per driver node.** `Scripting:Worker:Shards` defaults to 2 and is clamped to 1–8. A
  script is placed by `hash(scriptKey) mod K`. The script key is the guard key used today: the VT id, the calc
  RawPath, or the alarm id. Placement is deterministic, so a script's compiled PE and warm state stay on one
  worker across restarts.
- **One isolation worker** ("penalty box") is spawned on demand. Scripts under suspicion (§5.3) run there,
  **one evaluation at a time**. A kill in the isolation worker is therefore unambiguous.
- **Inside a worker:** a bounded evaluation scheduler with `P` dedicated threads, where
  `P = max(2, ProcessorCount / K)` **[A-4]**. Single flight per key is kept, as the guard does today. A queue
  that is full answers `Busy`, which the host treats as a refusal, not a strike.
- **The passthrough fast path stays in the host.** It never runs user code, and it is probably the largest
  share of evaluations **[A-5]**.

### 3.3 Lifecycle and supervision under Akka

The design splits a **control plane** (actors) from a **data plane** (a lock-free client). An Akka `Ask` per
evaluation would add mailbox hops to the hottest path for no benefit.

```
DriverHostActor (driver role)
 └─ ScriptRunnerActor            node-wide; owns the pool policy, attribution ledger, quarantine table
     ├─ ScriptWorkerActor #0     owns one OS process + pipe; state machine below
     ├─ ScriptWorkerActor #1
     └─ ScriptWorkerActor iso    spawned on first suspect, stopped after IsolationIdle (default 10 min)

IScriptRunner (DI singleton)  ── data plane ──►  ScriptWorkerConnection (per worker; pipe writer + correlation map)
   used by RoslynVirtualTagEvaluator, CalculationEvaluator, ScriptedAlarmEngine
```

- **Ownership.** `ScriptRunnerActor` is created by `DriverHostActor`, next to `VirtualTagHostActor` and the
  scripted-alarm host, so it exists only on driver-role nodes and dies with the driver tree. Its supervisor
  strategy is `OneForOne`, and a restart of a `ScriptWorkerActor` kills and respawns its process.
- **Process exit is a message, not an exception.** A watcher on `Process.Exited` / `WaitForExitAsync` sends
  `WorkerExited(exitCode, signal, inFlight[])` to its `ScriptWorkerActor`, so worker crashes never escalate
  through Akka supervision.
- **Worker states:**

  `Spawning → Handshaking → Loading → Ready ⇄ Draining → Dead → Backoff → Spawning`

  `Ready` is the only state that accepts evaluations. In the other states the connection queues per key, with
  latest-wins coalescing (§9).
- **The host process must take its workers down with it.**
  - **Windows:** every worker is assigned to a Job Object with `JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE`. If the host
    dies, the kernel closes the job handle and kills the workers.
  - **Linux** (the docker-dev rig): the worker treats **pipe EOF as fatal**, and also polls its parent PID. The
    host is not trusted to clean up after itself.
- **CoordinatedShutdown.** In the `before-actor-system-terminate` phase, the host sends `Shutdown`, waits
  1 s, then kills.
- **Orphan sweep at start.** The host kills only workers whose launch nonce file sits under *this* host's
  instance directory, never "every process with this name". mxaccessgw's `OrphanWorkerTerminator` killed a
  live gateway's workers when a second gateway started (scadaproj memory "wonder upgrade program"); this
  design avoids that by construction.
- **Redeploy.** `ClearCompiledScripts` / `LoadAsync` become `IScriptRunner.ApplyGeneration(scriptSet)`. That
  call sends `LoadScripts` / `Unload` deltas to each shard and clears the host-side quarantine rows for edited
  or removed keys, which keeps today's lift rule. No worker restart is needed. Collectible ALCs in the
  worker are unloaded as they are today, and a stuck one no longer matters because its worker gets killed.
- **Redundancy.** No change. Each node runs its own scripts today and will keep doing so; workers are not
  shared across the pair.

---

## 4. IPC

### 4.1 Shape of the conversation

§1.1 showed that a run needs no host callbacks. Each evaluation is therefore **one request frame and one
reply frame**, with no re-entrant RPC. ScadaBridge's `SandboxWorkerProtocol` has to proxy `call`/`reply`
frames for external systems and databases; this design does not. Log lines from `ctx.Logger` are the only
other worker→host traffic, and they are fire-and-forget.

### 4.2 Transport: named pipe vs gRPC

| | Named pipe + length-prefixed frames | gRPC (h2c over UDS / named pipe) |
|---|---|---|
| Per-call overhead | Two syscalls plus serialize; no HTTP/2 framing **[A-6]** | HTTP/2 stream setup, HPACK and a Kestrel pipeline per call |
| Worker footprint | Plain console app | Needs Kestrel in the worker, or a client stack in it and a server in the host |
| Batching / priority | Free to design: control frames ahead of events, as in mxaccessgw `WorkerFrameProtocol` | Per-RPC; batching means streaming RPCs, which is a frame protocol again |
| Cross-platform | `NamedPipeServerStream` maps to a Unix domain socket on Linux; `PipeOptions.CurrentUserOnly` on both | Yes |
| Prior art in the family | mxaccessgw gateway↔worker (protobuf `WorkerEnvelope`, 4-byte LE length, priority-drained writer); ScadaBridge sandbox (JSON frames over stdio, 1 MiB cap) | OtOpcUa mesh transport (`Commons/Protos/mesh_transport.proto`) — but that is cross-node |

**Recommendation: a named pipe with 4-byte little-endian length-prefixed protobuf envelopes.** It copies
mxaccessgw's `WorkerFrameProtocol` discipline:

- a frame-size cap negotiated at the handshake (default 4 MiB);
- zero-length and oversize frames rejected;
- a monotonically increasing `seq`;
- control frames (heartbeat, shutdown, fault) drained ahead of evaluation traffic;
- coalesced flushes.

`Google.Protobuf` and `Grpc.Tools` are already central pins, and `Commons.csproj:28-30` already compiles
protos, so no new package is needed. `MessagePack` is pinned in `Directory.Packages.props` but referenced by
no project, so it would be a new dependency in practice. Stdio, as ScadaBridge uses, is rejected: a script
cannot write to the console (it is forbidden), but the runtime and the logger can, and one stray byte on
stdout corrupts the stream.

### 4.3 Message schema (sketch)

```proto
syntax = "proto3";
package otopcua.scriptworker.v1;

message Envelope {
  uint32 protocol_version = 1;
  uint64 seq = 2;
  oneof body {
    Hello hello = 10;            // host→worker: nonce, max_frame_bytes, options
    Ready ready = 11;            // worker→host: pid, runtime version, sandbox fingerprint
    LoadScripts load = 12;       // host→worker: add/replace compiled-or-source scripts
    LoadResult load_result = 13; // worker→host: per-script compile outcome + diagnostics
    Unload unload = 14;          // host→worker: drop keys (ALC unload)
    EvalBatch eval = 15;         // host→worker
    EvalResults results = 16;    // worker→host (one or more completions)
    LogBatch logs = 17;          // worker→host, fire-and-forget, bounded
    Heartbeat heartbeat = 18;    // both ways; carries worker gauges (inflight, heap, gen2 count)
    Shutdown shutdown = 19;
    Fault fault = 20;
  }
}

enum ContextKind { CONTEXT_KIND_UNSPECIFIED = 0; VIRTUAL_TAG = 1; CALCULATION = 2; ALARM_PREDICATE = 3; }

message ScriptSource { string key = 1; bytes source_sha256 = 2; ContextKind kind = 3;
                       string source = 4; bytes pe_image = 5; repeated string input_paths = 6; }
message LoadScripts  { uint64 generation = 1; repeated ScriptSource scripts = 2; }

message EvalBatch    { repeated EvalRequest items = 1; }
message EvalRequest  {
  uint64 eval_id = 1; string key = 2; bytes source_sha256 = 3;
  repeated Input inputs = 4;      // positional against ScriptSource.input_paths; see 4.4
  int64 now_utc_ticks = 5;        // ctx.Now — host clock, so tests that pin the clock still work
  uint32 soft_timeout_ms = 6;
}
message Input { uint32 slot = 1; Variant value = 2; uint32 status_code = 3;
                int64 source_ts_ticks = 4; /* 0 = null */ int64 server_ts_ticks = 5; }

message EvalResult {
  uint64 eval_id = 1;
  oneof outcome { Variant value = 2; bool predicate = 3; ScriptError error = 4; }
  uint64 allocated_bytes = 5;     // same GC.GetAllocatedBytesForCurrentThread sample as today
  uint32 run_micros = 6;
}
message EvalResults { repeated EvalResult items = 1; }
message ScriptError { Kind kind = 1; string message = 2;
  enum Kind { THREW = 0; COMPILE = 1; SANDBOX = 2; SOFT_TIMEOUT = 3; UNSUPPORTED_TYPE = 4; BUSY = 5; } }

message Variant {
  TypeTag type = 1;               // exact CLR type — see 4.4
  oneof v { bool b = 2; sint64 i = 3; uint64 u = 4; double d = 5; float f = 6;
            string s = 7; int64 dt_ticks = 8; ArrayValue array = 9; }
  enum TypeTag { NULL = 0; BOOLEAN = 1; SBYTE = 2; BYTE = 3; INT16 = 4; UINT16 = 5; INT32 = 6; UINT32 = 7;
                 INT64 = 8; UINT64 = 9; FLOAT = 10; DOUBLE = 11; STRING = 12; DATETIME_UTC = 13;
                 ARRAY = 14; }
}
message ArrayValue { Variant.TypeTag element_type = 1; repeated Variant items = 2; }

message LogBatch { repeated LogLine lines = 1; uint32 dropped = 2; }
message LogLine  { string key = 1; uint32 level = 2; string message_template = 3;
                   string rendered = 4; string exception = 5; int64 ts_ticks = 6; }
```

### 4.4 How values and quality cross the boundary

- **Quality crosses verbatim.** `DataValueSnapshot` is `(object? Value, uint StatusCode, DateTime?
  SourceTimestampUtc, DateTime ServerTimestampUtc)` (`Core.Abstractions/DataValueSnapshot.cs:13`), and it maps
  1:1 onto `Input`. The worker rebuilds the same snapshots, so `ctx.GetTag(p).StatusCode` and
  `.SourceTimestampUtc` behave as they do today. The input-quality gating does not move: VT holds evaluation
  while an input is Bad (`VirtualTagActor.cs:264-269`), Calc holds on Bad, and alarms short-circuit on
  `AreInputsReady`. A request is sent only when today's code would have run the script.
- **The exact boxed CLR type is load-bearing.** A script that writes `(short)ctx.GetTag(x).Value` throws
  `InvalidCastException` if the value arrives boxed as an `int`. `Variant.type` therefore carries the CLR type,
  not just the OPC UA type: `DriverDataType` has `Int16` / `UInt16`, which have to survive the round trip.
  **[A-7]** I have not inventoried which CLR types each driver actually boxes, for example whether a Modbus
  `Int16` arrives as `short` or as `int`. The shadow mode in §12 exists partly to find out.
- **Unsupported types.** An input whose CLR type has no tag is not silently converted. The request carries no
  value for that slot, and the worker answers `UNSUPPORTED_TYPE`; the host publishes Bad with the type name. On
  the result side, a script that returns an unsupported type (a `List<T>`, an anonymous object) also gets
  `UNSUPPORTED_TYPE`. That is a **behaviour change**: today the raw `object` goes to
  `OpcUaPublishActor.AttributeValueUpdate` (`VirtualTagHostActor.cs:233`), and what the sink does with it was
  not traced **[A-8]**. Shadow mode reports every such case before the flag flips.
- **DateTime.** Values are sent as UTC ticks. A `Kind` other than `Utc` is converted at the host edge and
  counted on a metric. **[A-9]** I assume drivers already produce UTC.
- **Positional inputs.** `LoadScripts` registers the script's ordered `input_paths`: the
  `DependencyExtractor` reads, after the `{{equip}}` resolution done at deploy. Requests then send values by slot,
  not by path string, which takes the dominant payload cost out of the hot path. A key with no registered
  slots, a fallback that should never happen, sends the paths inline.

### 4.5 Batching

- **Opportunistic, not timed.** The connection's writer drains every request queued at that moment into one
  `EvalBatch` and flushes once, as mxaccessgw's drain pass does. An idle system pays no added latency; a burst
  is amortised automatically. There is no coalescing *timer*, because it would add latency to every
  evaluation.
- **Results are streamed, not batch-atomic.** The worker sends `EvalResults` as items complete, coalescing
  whatever finished in the same scheduler pass, so a slow script never holds a fast sibling's reply.
- **The seams must submit concurrently, or batching never happens.** Today both Calc and alarms await one
  script at a time (§1.1), and with IPC that serial loop costs N × round-trip. Two seam changes are required:
  - **Scripted alarms:** `ReevaluateAsync` submits the whole dirty set at once, then applies results **in the
    original order** under `_evalGate`. Predicates are pure functions of a snapshot taken under the gate, so
    the state machine sees the same sequence.
  - **Calculation:** the pump dequeues everything pending, with each entry's dependency snapshot already
    captured at enqueue, submits it as one batch and publishes in dequeue order.
- **Virtual tags become asynchronous.** `IVirtualTagEvaluator` gains `EvaluateAsync`, and `VirtualTagActor`
  `PipeTo`s the result to itself. While a run is in flight, further dependency changes update the map and set a
  "re-run" flag; when the result lands, the actor evaluates **once more** against the latest map (latest-wins).
  This also ends today's blocking of a default-dispatcher thread for up to 2 s
  (`RoslynVirtualTagEvaluator.cs:174-175`).

---

## 5. Pre-emption

### 5.1 Two deadlines

| | Soft deadline `T` | Hard deadline `T + G` |
|---|---|---|
| Value | Today's per-seam budget, unchanged: VT 2 s, Calc `runTimeoutMs` (2 s), alarm 250 ms | `G = Scripting:Worker:HardGrace`, default `max(1 s, T)`, clamped to 250 ms–30 s |
| Who acts | The host connection's timer | The host `ScriptWorkerActor` (authoritative) **and** a worker-side watchdog thread |
| Effect | The caller gets the timeout failure it gets today (Bad, same reason prefix `ScriptEvalFailureReasons.TimedOutPrefix`) and the run token is cancelled, so a polling script still ends itself as it does today | The worker process is **killed** (`Process.Kill(entireProcessTree: true)`). The worker watchdog calls `Environment.FailFast` first if it is still scheduling, so the dump and exit code name the key. |
| Strike | Yes (as today) | Yes, plus a kill strike (§5.3) |

The grace period `G` keeps a script that overran by a few milliseconds from costing a whole shard restart, and
the cooperative token from #734 still gets its chance. Making `G = 0` (kill at `T`) is an open decision (§14).

**Single flight between T and T+G.** The key stays in flight, and a trigger is refused as today, which is not
a strike. After the kill there is **no orphan**, so the stuck-orphan rule (`StuckOrphanQuarantineFactor`)
becomes moot out of process; it stays for the in-process mode.

### 5.2 Resource limits on the worker

The limits combine prior art from ScadaBridge (heap limit) and mxaccessgw (process-tree kill):

- **Heap:** `DOTNET_GCHeapHardLimit = Scripting:Worker:HeapLimitBytes`, default 512 MiB **[A-10]**. Breaching
  it makes the worker throw `OutOfMemoryException` or fail fast. Either way the worker dies or reports, and the
  host survives.
- **Windows Job Object** (per worker):
  - `JOB_OBJECT_LIMIT_PROCESS_MEMORY`, set slightly above the heap limit as a backstop for native and JIT
    memory;
  - `JOB_OBJECT_LIMIT_ACTIVE_PROCESS = 1`, so no child processes are possible;
  - `KILL_ON_JOB_CLOSE`;
  - UI restrictions.
- **Linux / containers:** the heap limit only. A container's cgroup is the rig's concern. **[A-11]** I assume
  the docker-dev containers run without CAP_SYS_ADMIN, so a nested cgroup is not available.
- **Liveness:** a heartbeat every 1 s. Three missed heartbeats while not in a GC pause, which the heartbeat
  reports, trigger a kill with reason `heartbeat`. This catches a wedged worker that neither replies nor exits.

### 5.3 Attribution, restart, backoff and quarantine

A kill or crash takes down every in-flight evaluation on that shard. Attribution decides which **one** to
blame.

1. **Hard-deadline kill:** the key that crossed `T+G` is known exactly. It takes a **kill strike**. The other
   in-flight keys on that shard are victims: their runs fail with "script worker restarted", which is not a
   strike.
2. **Crash with no deadline crossed** (OOM, stack overflow, native fault): attribution is ambiguous. Every
   key in flight at exit becomes a **suspect**, and suspects move to the **isolation worker**.
3. **Isolation worker:** evaluations run one at a time, so a crash there attributes to exactly one key, which
   takes a kill strike. A suspect that completes `IsolationProbation` (default 20 successful runs or 10 min)
   without incident returns to its shard.
4. **Quarantine:** a key is quarantined after `Scripting:Worker:QuarantineAfterKills` kill strikes (default 2)
   within 1 h, or by today's rules (N consecutive soft timeouts, allocation breach). The quarantine table
   lives in the **host** (`ScriptRunnerActor`), so it survives worker restarts. It is lifted by a source
   change or a redeploy, exactly as today, and quarantine reasons keep the `"script quarantined: "` prefix so
   every existing consumer and test classifier still matches.
5. **Restart backoff per worker:** exponential, 250 ms → 30 s cap, ×2, with ±20% jitter. The backoff resets
   after 60 s of `Ready`. A worker that dies **during** `Loading` (a poison PE or a runtime fault) does not
   loop: after 3 consecutive load-phase deaths, the scripts in its generation are loaded one by one in the
   isolation worker to find the culprit.
6. **Crash-loop circuit:** if all shards are in backoff for longer than `Scripting:Worker:CircuitOpenAfter`
   (default 2 min), the runner reports `Unavailable` (§10) and logs one Error. It keeps retrying at the cap.

---

## 6. Security

Process isolation contains **failures**. Most of the **capability** control stays with the compile-time
sandbox, which is retained. Defence-in-depth adds the following.

- **The compile-time sandbox is unchanged.** It still includes `ScriptSandbox` (with the TPA reference set
  filtered to `System.*`, `ScriptSandbox.cs:118-122`), `ForbiddenTypeAnalyzer`, and the injection guard. The
  worker runs **exactly** the same `ScriptEvaluator.Compile`, so editor/publish/runtime parity holds.
- **Assembly allow-list by construction.** The worker is its own app (`ZB.MOM.WW.OtOpcUa.ScriptWorker`), with
  its own `deps.json` and only these references:
  - `Core.Scripting`;
  - `Core.Scripting.Abstractions`;
  - `Core.Abstractions`;
  - Serilog (core only, for `ILogger` on `ctx`);
  - Google.Protobuf.

  Its trusted-platform list therefore contains the shared framework plus those assemblies. Akka, SqlClient, the
  OPC UA stack, driver assemblies and the Secrets library are **not loadable** in the worker, even through
  reflection. At startup the worker asserts its loaded-assembly set against a pinned list and exits if the list
  differs.
- **No secrets in the environment.** `ProcessStartInfo.Environment` is **cleared** and then repopulated from
  an explicit allow-list: `DOTNET_*` runtime knobs, `TEMP`, `PATH` minimal, and the launch nonce. Host env vars
  like `ServerHistorian__ApiKey` (CLAUDE.md "Never commit secrets") and the ConfigDb connection string must
  never reach the worker. ScadaBridge's sandbox **inherits** the parent environment; do not copy that part.
- **No configuration access.** The worker gets no appsettings path. Every option it needs (budgets, heap,
  frame cap) arrives in `Hello`.
- **Pipe hardening (mxaccessgw pattern):**
  - a random pipe name;
  - `PipeOptions.CurrentUserOnly`, or an explicit ACL when the worker runs as a different account;
  - a one-time nonce passed in an env var (never on the command line; mxaccessgw `WorkerBootstrap.md` explains
    why), which the worker must echo in `Ready`;
  - the host accepts exactly one connection per launch.
- **Filesystem:**
  - the working directory is an empty per-worker temp directory;
  - the PE cache (§7) is opened read-only by the worker, if the host compiles, or written only under its own
    subdirectory;
  - on Windows, a dedicated low-privilege account (see §14) gets read access only to the install folder and
    its own temp directory.
- **Network:** the analyzer already forbids `System.Net`. Process-level options are a Windows Firewall
  outbound-block rule scoped to the worker executable, created by the installer, and on Linux nothing beyond
  the container. This is an open decision (§14). The recommendation is the installer firewall rule on Windows,
  and to accept the analyzer as the control on Linux.
- **What the host trusts from the worker:** nothing it cannot validate. Frames are size-capped and
  schema-checked, an `eval_id` must match an outstanding request, and log lines are length-capped and
  rate-limited (§11). A compromised worker can at worst return wrong values for scripts on its own shard,
  which the same script could have done anyway.

---

## 7. Compile caching across restarts

The problem: today every process start recompiles every script lazily (`CompiledScriptCache` is memory-only).
Out of process, a **shard restart** has the same cost, and it can happen in the middle of production running.
mem-probe measured the old `CSharpScript` path at roughly 2.4 MiB of working set per retained script
(`tools/mem-probe/MemProbe/Program.cs:1-4`). Compile *time* is not measured anywhere in the repo **[A-12]**;
I assume tens of ms per script warm and seconds for the first compile while Roslyn JITs. For 1,000 scripts,
that is far outside a 2 s restart budget.

**Recommendation: a content-addressed PE cache on node-local disk.**

- **Key:** SHA-256 over these fields:
  - the source;
  - the `ContextKind`, i.e. `TContext` and `TResult`;
  - the Roslyn compiler version;
  - a **sandbox fingerprint**: a hash of `ForbiddenTypeAnalyzer`'s lists, the `ScriptSandbox` reference set
    and the wrapper template.

  An analyzer change therefore invalidates every entry, and a cached PE can never skip a newer sandbox rule.
- **Value:** the emitted PE bytes plus an HMAC-SHA256 tag. The HMAC key is a per-node random key created at
  first start and stored next to the cache, readable only by the host account. The worker loads a PE only if
  the HMAC verifies, because loading attacker-supplied PE bytes would be code execution outside the analyzer.
- **Location:** `Scripting:Worker:PeCacheDir`. It must be absolute and outside the content root, the same
  fail-closed rule the family applies to `Secrets:SqlitePath` (scadaproj CLAUDE.md, Secrets row). Entries
  whose key is not in the current generation are pruned at `ApplyGeneration`.
- **Where compilation happens:** **in the worker** (recommended). Roslyn's memory and CPU leave the host. A
  pathological source that makes Roslyn itself blow up kills a worker, not the server, and the worker writes
  the PE plus HMAC (it holds the key through `Hello`). The alternative is to compile in the host and ship
  `pe_image` in `LoadScripts`. That keeps the worker free of Roslyn (smaller, faster start), but compile-time
  blowups stay in the host. This is an open decision (§14).
- **Restart path:** `Spawning → Ready` takes about 150 ms of runtime start **[A-13]**. Then `LoadScripts`
  sends hash + key + slots only. The worker loads PEs from the cache lazily, on each key's first evaluation,
  and a cache miss compiles. Loading a PE into a collectible ALC is about 1 ms **[A-13]**.
- **Negative cache:** Calculation already remembers compile failures by source hash (R07:P4,
  `CalculationEvaluator.cs:63-68`). The worker reports compile failures in `LoadResult`, and the host keeps
  that negative cache for **all** seams, so a broken script never re-enters Roslyn.

---

## 8. Performance budgets

### 8.1 Reference loads in the repo

| Source | Load |
|---|---|
| `docs/v2/Galaxy.Performance.md` (EventPump sizing, 24 h soak) | 50,000 tags at 1 Hz on one node; compressed CI soak at 1,000 tags |
| `tests/Server/.../NodeManagerLockTimeBenchmarkTests.cs:41-105` | 10,000-tag folder; 200 equipment × 250 tags (50,000 variables) |
| `ScriptRunGuardTests.cs:170-205` | a 1 Hz runaway and a healthy sibling over 30 triggers |

No repo perf test exercises scripts at scale. **[A-14]** I assume a planning envelope of **5,000 compiled
script evaluations per second per node** (for example 5,000 VT + Calc at 1 Hz, or 500 scripts at 10 Hz) and
size for 2× headroom (10k/s). The passthrough fast path is excluded because it stays in the host.

### 8.2 Budgets (acceptance gates for the implementation)

| Metric | Budget | Notes |
|---|---|---|
| Added latency per evaluation, p50 | ≤ 150 µs | Host enqueue → reply decoded, trivial script, idle system |
| Added latency per evaluation, p99 | ≤ 2 ms | At 10k evals/s. Compare the 250 ms alarm budget: under 1% of it |
| Throughput | ≥ 10,000 trivial evals/s/node with K=2 | With opportunistic batching |
| Host CPU for IPC | ≤ 25% of one core at 10k evals/s | Serialisation + pipe I/O, measured by an opt-in perf test. Since lmxopcua#789 the harness reports it as the out-of-process runner's host CPU minus the in-process runner's in the same phase ("vs InProcess"): the raw figure includes the harness's own pacing loop and pool, which InProcess pays too (§12.2) |
| Bytes per eval on the wire | ≤ 64 B + 24 B per input | Positional slots, §4.4 |
| Shard restart → first result, 1,000 cached scripts | ≤ 2 s | §7 lazy PE load |
| Worker idle RSS | ≤ 80 MB + scripts **[A-3]** | Per shard. **Proposed revision (lmxopcua#789, §12.2):** ≤ 80 MB + scripts for a worker that has only loaded cached PEs (measured 36–58 MB); ≤ 200 MB + scripts once Roslyn has run in it (measured 159–187 MB), because compiling in the worker (§14 #7) keeps Roslyn's code and metadata resident and no GC setting moves it |
| Kill latency after `T+G` | ≤ 100 ms | Host timer resolution plus `Process.Kill` |

**Comparison with today:** the in-process cost per compiled run is one `Task.Run` hop plus `WaitAsync`, which
is a few µs to tens of µs and was never measured **[A-15]**. The out-of-process path is therefore one to two
orders of magnitude slower **per call**. That is acceptable only because the budgets above sit two to three
orders of magnitude below the script budgets (250 ms / 2 s), and because batching amortises the syscalls. The
risk case is a large scripted-alarm engine, with its serial gate and its 250 ms budget. That is why §4.5
makes the batch-submit change a **prerequisite**, not an optimisation.

---

## 9. Failure semantics: what quality a tag gets

The principle: **a runner failure must never look like a script result.** In particular, an alarm must never
clear because the worker died.

| Situation | Virtual tag / Calculation tag | Scripted alarm |
|---|---|---|
| Soft timeout (`T`) | `BadInternalError` + last value, once per transition, as today | Condition frozen + `BadInternalError` quality annotation, as today |
| This key killed at `T+G`, or crashed in isolation | Same as soft timeout; reason `"script killed: exceeded hard deadline …"` | Same as soft timeout |
| Victim: in flight on a shard that died for another key | `BadInternalError` + last value, reason `"script worker restarted"`; re-evaluated on the shard's return with the latest inputs (latest-wins) | Frozen + Bad annotation; re-evaluated on return |
| Not in flight while its shard restarts | **Unchanged.** Keeps its last published value and quality. A trigger during the outage is coalesced (latest-wins, one per key) and evaluated when the shard is `Ready` | Same; the condition holds its state |
| Shard not back within `Scripting:Worker:UnavailableGrace` (default 5 s) | Coalesced keys publish Bad + last value, reason `"script runner unavailable"` | Frozen + Bad annotation |
| Circuit open (§5.3.6) | Every compiled-script key on the node is Bad as above; passthrough keys are unaffected | Every compiled predicate is frozen + Bad |
| Quarantined | `"script quarantined: …"` Bad, as today | As today |
| Host process dies | Workers die with it (Job Object / pipe EOF); no change from today's host-death behaviour | — |

**Status code:** `BadInternalError` stays for script-attributable faults (the existing contract). For "the
runner is unavailable, not this script's fault", this design proposes **`BadResourceUnavailable`
(0x80040000)**, so operators and the ScadaBridge consumer can tell a broken script from a sick node. The
alternative is to keep `BadInternalError` everywhere for zero consumer impact. This is an open decision (§14);
if it is taken, the ScadaBridge consumer-impact record must note it (scadaproj `sister-project-history`
skill, `otopcua.md`).

**No silent fallback to in-process.** If the worker cannot start, the runner does **not** run scripts in the
host. That would reintroduce exactly the risk the flag was set to remove. Operators flip the flag back (§12).
This is an open decision (§14).

---

## 10. Observability

All metrics go on `OtOpcUaTelemetry.Meter`, so the existing `ZbTelemetryOptions` allow-list applies.
scadaproj memory: "`ZbTelemetryOptions.Meters` is a silent allowlist"; check that the new names flow end to
end.

| Metric | Type | Tags |
|---|---|---|
| `otopcua.script.worker.up` | gauge | `shard` |
| `otopcua.script.worker.starts` / `.exits` | counter | `shard`, `reason` = `clean` \| `killed_deadline` \| `killed_heartbeat` \| `oom` \| `crash` \| `load_failure` |
| `otopcua.script.worker.inflight` / `.queue_depth` | gauge | `shard` |
| `otopcua.script.worker.heap_bytes` | gauge (from heartbeat) | `shard` |
| `otopcua.script.ipc.roundtrip` | histogram (µs) | `kind` |
| `otopcua.script.eval.duration` | histogram (µs, worker-measured) | `kind` |
| `otopcua.script.compile` | counter | `cache` = `hit` \| `miss` \| `negative` |
| `otopcua.script.quarantined` (existing) | counter | adds `reason` = `killed` \| `isolated_crash` |
| `otopcua.script.run_refused` (existing) | counter | unchanged |
| `otopcua.script.isolation.suspects` | gauge | — |
| `otopcua.script.runner.state` | gauge (0 ready / 1 degraded / 2 unavailable) | — |

**Per-key series:** the existing counters tag the series with `script = key`. Do not add `key` tags to the new
per-eval histograms (cardinality).

**Logs:**
- the worker's own stderr is captured, prefixed with `ScriptWorker[shard]`, into the host log;
- every kill logs one Warning naming the key and the reason, and every quarantine logs one Error, as today;
- `ctx.Logger` lines come back in `LogBatch`. The host re-emits them through `ScriptRootLogger` with the same
  `ScriptId` / `VirtualTagId` properties, so `scripts-*.log`, the error mirror, the DPS script-log topic and its
  20/min-per-source rate limit all keep working unchanged;
- the worker caps its outbound log buffer (default 1,000 lines / 1 MiB) and reports drops as
  `LogBatch.dropped`, so a script logging in a tight loop cannot flood the pipe.

**Health:** expose the runner state as a non-fatal component of the driver-node status (`/health/ready`
stays green while degraded, because OPC UA still serves driver data). Whether `Unavailable` should fail
readiness is an open decision (§14). The AdminUI status dashboard gets a "Script runner" tile with the
per-shard state, restarts in the last hour and the quarantine list. Quarantines already surface through the
`QuarantinedScripts` properties.

**Traces:** one `Activity` per `EvalBatch` (not per evaluation), with `batch.size` and `shard`.

---

## 11. Seam changes in the host (summary)

| Component | Change |
|---|---|
| `Core.Scripting` | New `IScriptRunner` (`EvaluateAsync(ScriptInvocation[])`, `ApplyGeneration`, `QuarantinedKeys`, `ResetAll`). `InProcessScriptRunner` wraps today's `ScriptRunGuard` + `CompiledScriptCache` exactly. |
| New `Core.Scripting.Protocol` | The proto file, the frame codec and the `Variant` ↔ CLR mapping. Shared by host and worker. |
| New `ZB.MOM.WW.OtOpcUa.ScriptWorker` | Console app: handshake, scheduler, watchdog, PE cache, log forwarding. Shipped in the Host's publish output. |
| New `Runtime/Scripting/` | `ScriptRunnerActor`, `ScriptWorkerActor`, `OutOfProcessScriptRunner` (data plane), Job Object interop (Windows-only, behind `OperatingSystem.IsWindows()`). |
| `IVirtualTagEvaluator` / `VirtualTagActor` | Add `EvaluateAsync`; the actor `PipeTo`s with latest-wins re-run (§4.5). |
| `RoslynVirtualTagEvaluator`, `CalculationEvaluator`, `ScriptedAlarmEngine` | Replace direct `ScriptRunGuard` use with `IScriptRunner`. Calc pump and alarm `ReevaluateAsync` submit batches (§4.5). |
| `ScriptingOptions` / `ScriptingOptionsBinding` | New keys (§12), clamped with startup Warnings like the existing ones. |

---

## 12. Migration and rollout

**Configuration** (node-local appsettings, `Scripting` section, consistent with the existing knobs):

| Key | Default | Meaning |
|---|---|---|
| `Scripting:Runner` | `InProcess` | `InProcess` \| `OutOfProcess` \| `Shadow` |
| `Scripting:Runner:Seams` | VirtualTag + Calculation | Optional per-seam subset for `OutOfProcess` / `Shadow` (e.g. `["Calculation"]`). Unset selects VirtualTag and Calculation; `ScriptedAlarm` only when listed (§14 #10, as built in §12.3) |
| `Scripting:Worker:Shards` | `2` | 1–8 |
| `Scripting:Worker:HardGrace` | `max(1s, T)` | §5.1 |
| `Scripting:Worker:HeapLimitBytes` | `536870912` | §5.2 |
| `Scripting:Worker:QuarantineAfterKills` | `2` | §5.3 |
| `Scripting:Worker:UnavailableGrace` | `5s` | §9 |
| `Scripting:Worker:PeCacheDir` | none (required when not `InProcess`) | §7; absolute, outside the content root, fail-closed |
| `Scripting:Worker:Identity` | none (host's token) | Windows only: `NT SERVICE\OtOpcUaScriptWorker` → restricted worker token + pipe ACL (§14 #5, record §12.5) |

**Phases** (each one is its own issue and PR):

1. **P0, in-process refactor with no behaviour change.**
   - Introduce `IScriptRunner` + `InProcessScriptRunner`.
   - Make VT evaluation async (`PipeTo`), and make the alarm engine and the Calc pump batch-submit.
   - Gate: every existing Scripting / VirtualTags / Calculation / ScriptedAlarms suite stays green unchanged,
     plus new ordering tests for batch-then-apply.
   - This phase has value on its own: it ends the dispatcher-thread blocking.
   - **Partly landed in lmxopcua#779:** VT evaluation is async (`IVirtualTagEvaluator.EvaluateAsync` + stash in
     `VirtualTagActor`), so the dispatcher-thread blocking is gone. The Calc pump and the alarm engine still run one
     script at a time; why that was not relaxed in-process is recorded in `docs/VirtualTags.md` §"Upstream-tag feed".
     `IScriptRunner` and batch-submit remain to do.
   - **Batch submit landed in lmxopcua#780 (fourth P1 slice, §12.3):** `IScriptRunner.RunBatch`, used by the Calc pump, the
     VirtualTag seam (a coalescer) and the scripted-alarm engine on the out-of-process path. In process the pump and the
     engine stay one at a time, unchanged.
2. **P1, worker + protocol behind the flag.**
   - Ship `ScriptWorker`, `OutOfProcessScriptRunner` and the PE cache.
   - Default stays `InProcess`. Enable on the docker-dev rig only.
   - Gate: §13 runaway / containment tests green on Linux (docker-dev) **and** Windows (windev box, which has
     the .NET 10 toolchain).
   - **Partly landed in lmxopcua#780 (P1 slice):** `Scripting:Runner` / `Scripting:Runner:Seams` / `Scripting:Worker:*`
     (bound, clamped, `PeCacheDir` path rule fail-closed), `IScriptRunner` + `InProcessScriptRunner`, the protobuf frame
     protocol (`Core.Scripting.Protocol`), the `ScriptWorker` app (compile in the worker, P-thread scheduler, watchdog,
     heartbeats, log forwarding, pipe-EOF / parent-pid exit, forbidden-assembly self-check), `OutOfProcessScriptRunner`
     (K shards + isolation worker, soft/hard deadlines, attribution, kill-strike quarantine, backoff, fail closed,
     Job Object on Windows), the VirtualTag and Calculation seams over it, and `BadResourceUnavailable` for
     runner-unavailable / worker-restarted (ScadaBridge#112). The §13 containment cases run on macOS and Windows
     (windev). Deviations: supervision is a plain object owned by DI, not `ScriptRunnerActor`/`ScriptWorkerActor`;
     the per-seam batch submit (P0) is still one-at-a-time. Remaining: the HMAC'd PE cache, a self-contained worker
     publish + the low-privilege account / pipe ACL / firewall rule (installer), the runner-state gauge and readiness
     component, victim re-evaluation on shard return, isolation-worker idle stop, the orphan sweep by nonce file, the
     scripted-alarm seam, the docker-dev enablement and the §8 perf harness.
   - **Second P1 slice (lmxopcua#780):** the HMAC'd PE cache (§7; keyed by source + kind + a compile fingerprint, tampered
     or mismatched entries evicted, pruned by recency at startup instead of by generation — this runner has no
     generation object), attribution before release (the quarantine race), loading keys counted in attribution
     (culprit-finding for a compile that crashes its worker, by isolation rather than the §5.3.5 three-strikes
     bisection — loads are per key here), proactive restart + `Recovery` / `RetryWhen` re-evaluation of victims in the
     VT and Calc seams, isolation idle stop (`IsolationIdleTimeout`), the crash-loop circuit (`CircuitOpenAfter`), the
     §10 gauges and histograms, the degraded-but-ready `script-runner` health check (§14 #12), and the orphan sweep by
     launch marker (pid + start time + image, host gone). Still open: self-contained worker publish, installer (account,
     pipe ACL, firewall), docker-dev live drill, scripted-alarm seam, the §4.5 batch submit, the §8 perf harness.
   - **Fourth P1 slice (lmxopcua#780, record in §12.3):** the §4.5 batch submit (`IScriptRunner.RunBatch`, one `EvalBatch`
     frame per worker; Calc pump, VT coalescer, alarm dirty set), exact per-item deadlines for work queued in the worker
     (deferral / start notices), and the scripted-alarm seam behind `Scripting:Runner:Seams` — alarms run out of process
     only when listed. Still open: installer (account, pipe ACL, firewall, PeCacheDir ACL), #789's remaining misses, then
     P2 Shadow.
   - **Third P1 slice (lmxopcua#780, record in §12.1):** a self-contained worker publish for self-contained (Windows)
     Host publishes, started through its apphost; the docker-dev live drill (`docker-dev/docker-compose.script-runner.yml`);
     the §8 perf harness (`Category=Perf`, opt-in), which passes the accepted 5k evals/s/node gate; and two runner fixes
     the harness found (a caller refused by its own finished run; heartbeat starvation under a compile burst). Still open:
     installer (account, pipe ACL, firewall, PeCacheDir ACL), scripted-alarm seam, §4.5 batch submit, the §8.2 budget
     misses (lmxopcua#789) and the load-timeout queue issue (lmxopcua#790).
3. **P2, shadow mode on docker-dev and QA.**
   - `Shadow` runs the **in-process** evaluation as the published result, and *also* submits to the worker.
   - It compares value, status class and failure kind per evaluation, and counts
     `otopcua.script.shadow.mismatch{kind, reason}`. Mismatching keys are logged once each.
   - Shadow doubles script CPU and still carries the in-process risk, so it is a time-boxed diagnostic (for
     example 1 week on QA), not a mode to run in production.
   - Exit criterion: zero unexplained mismatches. This is where [A-7] / [A-8] get answered.
   - **Landed in lmxopcua#780 (P2 slice, record in §12.4):** `ShadowScriptRunner`, all three seams, bounded and
     drop-counted, readiness untouched; docker-dev drill with alarms. The QA shadow week is still to run.
4. **P3, flip on QA, then the other tiers.**
   - Set `Scripting:Runner=OutOfProcess` per node, seam subset first (Calculation, then VT, then alarms; alarms
     last because of the 250 ms budget).
   - Note: remote command execution on the wonder boxes is currently scoped to user-requested deploys (CLAUDE.md
     hold), so tier rollout rides normal deploys.
5. **P4, decide the long-term status of `InProcess`.** The recommendation is to keep it for dev and tests and
   as the documented rollback, and to retire `StuckOrphanQuarantineFactor` from the docs as "in-process only".

**Rollback:** set the flag back to `InProcess` and restart the node. No data or config migration is involved.
The PE cache directory can be deleted at any time.

**Docs to update when implementing:**
- `docs/VirtualTags.md` §"Known resource limits" and §"Per-evaluation timeout";
- `docs/ScriptEditor.md` §"Runtime safety";
- `docs/ScriptedAlarms.md`;
- `docs/ServiceHosting.md`: the new process, Job Object and firewall rule;
- `docs/security.md`;
- `docs/Telemetry.md`;
- the scadaproj index row, if the status-code decision changes the ScadaBridge-visible contract.

### 12.1 P1 record: self-contained publish, docker-dev live drill, §8 perf (2026-11-10)

**Self-contained worker publish.**
- A `-r win-x64 --self-contained` Host publish now publishes the worker self-contained for the same RID into
  `script-worker/`: 236 files, about 97 MB, with its own runtime. It is not trimmed and not single-file.
- The launcher starts the worker's apphost (`ZB.MOM.WW.OtOpcUa.ScriptWorker.exe`) directly. It does so only when the
  worker's `runtimeconfig.json` lists `includedFrameworks`. Otherwise (framework-dependent) it keeps `dotnet exec <dll>`.
- A framework-dependent publish is unchanged: 51 files, no `includedFrameworks`, run through the muxer.
- Gate on windev: the Host was published win-x64 self-contained under `C:\build`, and the whole
  `ScriptWorker.Tests` suite was run against the published apphost (`OTOPCUA_SCRIPTWORKER_TEST_PATH`).
  - Result: **132/132**.
  - It includes `A_published_self_contained_worker_runs_as_its_own_process`. That test checks that the worker
    process's `MainModule` is the `.exe`, so no muxer is involved.

**docker-dev live drill.** Image `otopcua-host:dev` was built from the branch at `f50b29ab` plus the self-contained
change. That build is from before the two runner fixes below. The drill ran with `docker-compose.script-runner.yml` on
`central-1`/`central-2`.

| Step | Observed |
|---|---|
| Normal VT + Calc | Workers start on the first compiled evaluation (`dotnet exec /app/script-worker/…dll`). `okvt` (HR100+1) stays Good and tracks HR100. Writing HR5 = 21 on the fixture gives Calc `HR5x2` **42 Good** on both nodes, and the sibling VT `drillsib` (HR5×3) **63 Good**. |
| `while (true) { }` VT (1 s timer, T = 2 s) | Logged `exceeded its hard deadline (4000 ms); killing ScriptWorker[1]` twice, 5 s apart. After the second: `quarantined: "it exceeded its hard deadline and its worker was killed (2 time(s) in the last hour)…"`. Both nodes did the same. `okvt` and Calc share that shard and kept updating Good (26 Good samples, no Bad) because neither was in flight at the kills. `run_refused` then counts the quarantined trigger. |
| `kill -9` both workers on central-1 | A 1.5 s busy VT (`drillslow`) was in flight on shard 1. It published **BadResourceUnavailable** at 05:41:22.28, became a suspect and moved to isolation. It was Good again 30 ms later, re-evaluated in isolation. After HR5 = 22 it read 1022. Shards 0 and 1 restarted within about 250 ms. Isolation probation (20 clean runs) returned it to its shard. Calc gave 44 Good and the sibling 66 Good. |
| Circuit open (worker dll renamed, all workers killed) | `/health/ready` stayed **200**. The overall status went to `Degraded`, and `script-runner` showed `Degraded` → `Unavailable (circuit open)` after 2 min. Logged `Script runner circuit OPEN …` once. `otopcua_script_runner_state` = **2**, `worker_up` = 0 per shard. VTs and Calc (after an HR5 change) published **BadResourceUnavailable**. Driver tags (HR100) stayed Good, so OPC UA kept serving. |
| Recovery (dll restored) | The next backoff retry (cap 30 s) started the workers. Logged `circuit closed`. Calc **46** (HR5 = 23 × 2), sibling **69**, `drillslow` 1023, `okvt` Good, all without any input change. |
| Metrics on `:9000/metrics` | `otopcua_script_runner_state`, `worker_up`/`inflight`/`queue_depth`/`heap_bytes{shard}`, `worker_starts`/`exits{reason}`, `compile{cache}`, `ipc_roundtrip`/`eval_duration` histograms, `quarantined{reason}`, `isolation_suspects`. |

On the wire, a Bad publish carries a null Value. This holds for every Bad status in this server, including
BadInternalError and BadOutOfRange. The seams keep the last value, but an OPC UA client does not see it. This was
noted for ScadaBridge#112.

**Rig state afterwards.**
- Drill rows were deleted, HR5 was set back to 5, and the config was redeployed. The revision hash is `e221b179…`,
  byte-identical to the pre-drill `gate-cleanup` deployment.
- `otopcua-host:dev` was re-tagged to the pre-drill image `24f9d7287ed2`.
- `central-1`/`central-2` were recreated from the base compose file: InProcess, no `Scripting__*` env. The four site
  nodes, sql and traefik were never touched.
- The two central containers were recreated (twice), so their container-local secret stores and their PE cache dirs
  under /tmp are gone.

Two defects outside the runner were found and filed:
- lmxopcua#787: a disabled VirtualTag is still deployed and evaluated.
- lmxopcua#788: worker start failures log an Error with a stack trace on every retry.

**§8 perf harness.**
- `ScriptRunnerPerfTests`, `Category=Perf`, opt-in with `OTOPCUA_SCRIPT_PERF=1`.
- The runners are driven directly: trivial one-input script, 2,000 keys over 50 sources.
- Phases:
  - idle sequential;
  - paced open loop at 5k/s and at 10k/s;
  - 64-way closed loop.
- Host CPU is this process's CPU time as a share of one core, harness included. Under the same harness InProcess shows
  60–75 % at 5–10k/s, and that is the harness's own floor.

macOS:

**Script worker restart / RSS / kill latency (Unix Arm64, 14 logical CPUs, .NET 10.0.14, phase 10 s)**

| Measure | Result | §8.2 budget |
|---|---:|---|
| 1000 scripts, empty cache, all first results (compile + write) | 1,633 ms | — |
| 1000 scripts, fresh runner on a warm cache, all first results | 217 ms | ≤ 2,000 ms |
| kill both shards → every script's next correct result | 482 ms | ≤ 2,000 ms |
| worker RSS after first eval (max of 2 shards) | 188 MB | ≤ 80 MB + scripts |
| worker RSS with 500 scripts loaded per shard (max) | 358 MB | — |
| kill latency after T+G (5 runs, ms) | 31 / 35 / 31 / 32 / 33 | ≤ 100 ms |

**Script runner perf (Unix Arm64, 14 logical CPUs, .NET 10.0.14, phase 10 s)**

| Runner | Phase | evals/s | p50 µs | p99 µs | host CPU | ok / not ok |
|---|---|---:|---:|---:|---:|---|
| InProcess | idle, sequential |   189,154 |        4 |         8 |    446 % | 5,000 / 0 |
| InProcess | paced 5,000/s |     4,998 |       32 |       223 |     60 % | 49,999 / 0 |
| InProcess | paced 10,000/s |     9,991 |       41 |       364 |     74 % | 99,986 / 0 |
| InProcess | closed loop ×64 |   789,533 |        4 |        15 |    409 % | 7,896,214 / 0 |
| OutOfProcess K=1 | idle, sequential |    21,641 |       45 |        64 |    264 % | 5,000 / 0 |
| OutOfProcess K=1 | paced 5,000/s |     4,999 |      149 |       860 |    108 % | 49,998 / 0 |
| OutOfProcess K=1 | paced 10,000/s |     9,995 |      205 |     1,763 |    159 % | 99,989 / 0 |
| OutOfProcess K=1 | closed loop ×64 |   206,862 |      258 |     1,368 |    526 % | 2,068,749 / 0 |
| OutOfProcess K=2 | idle, sequential |    22,642 |       41 |        70 |    267 % | 5,000 / 0 |
| OutOfProcess K=2 | paced 5,000/s |     4,999 |      182 |       722 |    125 % | 49,994 / 0 |
| OutOfProcess K=2 | paced 10,000/s |     9,996 |      232 |       720 |    237 % | 99,998 / 0 |
| OutOfProcess K=2 | closed loop ×64 |   336,519 |      150 |       993 |    544 % | 3,365,226 / 0 |
| OutOfProcess K=4 | idle, sequential |    18,461 |       50 |        97 |    303 % | 5,000 / 0 |
| OutOfProcess K=4 | paced 5,000/s |     4,999 |      201 |       455 |    197 % | 49,997 / 0 |
| OutOfProcess K=4 | paced 10,000/s |     9,997 |      126 |       732 |    176 % | 99,989 / 0 |
| OutOfProcess K=4 | closed loop ×64 |   297,094 |      178 |     1,015 |    507 % | 2,970,998 / 0 |

windev (Windows VM; framework-dependent worker built beside the tests):

**Script runner perf (Win32NT X64, 36 logical CPUs, .NET 10.0.8, phase 10 s)**

| Runner | Phase | evals/s | p50 µs | p99 µs | host CPU | ok / not ok |
|---|---|---:|---:|---:|---:|---|
| InProcess | idle, sequential |     5,876 |      142 |       253 |    204 % | 5,000 / 0 |
| InProcess | paced 5,000/s |     4,997 |      136 |       335 |     55 % | 49,996 / 0 |
| InProcess | paced 10,000/s |     9,989 |      105 |       274 |      6 % | 99,984 / 0 |
| InProcess | closed loop ×64 |   177,782 |       26 |    10,062 |    543 % | 1,778,824 / 0 |
| OutOfProcess K=1 | idle, sequential |     1,276 |      750 |     1,263 |     74 % | 5,000 / 0 |
| OutOfProcess K=1 | paced 5,000/s |     4,997 |    1,073 |     3,028 |     68 % | 49,990 / 0 |
| OutOfProcess K=1 | paced 10,000/s |     9,992 |    1,139 |     2,840 |    102 % | 99,999 / 0 |
| OutOfProcess K=1 | closed loop ×64 |    38,464 |    1,514 |     6,547 |    350 % | 384,727 / 0 |
| OutOfProcess K=2 | idle, sequential |     1,207 |      795 |     1,376 |     80 % | 5,000 / 0 |
| OutOfProcess K=2 | paced 5,000/s |     4,998 |      956 |     1,856 |    111 % | 49,997 / 0 |
| OutOfProcess K=2 | paced 10,000/s |     9,962 |      942 |     1,916 |    153 % | 99,992 / 0 |
| OutOfProcess K=2 | closed loop ×64 |    51,241 |    1,129 |     5,894 |    594 % | 512,504 / 0 |
| OutOfProcess K=4 | idle, sequential |     1,163 |      828 |     1,366 |     61 % | 5,000 / 0 |
| OutOfProcess K=4 | paced 5,000/s |     4,997 |      804 |     1,472 |    172 % | 49,991 / 0 |
| OutOfProcess K=4 | paced 10,000/s |     9,992 |      757 |     1,591 |    233 % | 99,992 / 0 |
| OutOfProcess K=4 | closed loop ×64 |    69,503 |      774 |     6,321 |  1,051 % | 695,094 / 0 |


**Script worker restart / RSS / kill latency (Win32NT X64, 36 logical CPUs, .NET 10.0.8, phase 10 s)**

| Measure | Result | §8.2 budget |
|---|---:|---|
| 1000 scripts, empty cache, all first results (compile + write) | 11,625 ms | — |
| 1000 scripts, fresh runner on a warm cache, all first results | 1,715 ms | ≤ 2,000 ms |
| kill both shards → every script's next correct result | 2,028 ms | ≤ 2,000 ms |
| worker RSS after first eval (max of 2 shards) | 158 MB | ≤ 80 MB + scripts |
| worker RSS with 500 scripts loaded per shard (max) | 232 MB | — |
| kill latency after T+G (5 runs, ms) | 257 / 210 / 200 / 217 / 209 | ≤ 100 ms |

Verdict against §8.2 and the accepted gate:
- **Accepted gate (≥ 5,000 evals/s/node at K = 2): PASS** on both hosts (macOS 336k/s, windev 51k/s closed loop), with every result correct.
- Throughput ≥ 10k/s: PASS.
- Idle p50 ≤ 150 µs: PASS on macOS (41 µs), MISS on windev (795 µs; InProcess there is already 142 µs).
- p99 ≤ 2 ms at 10k/s: PASS (macOS 0.7 ms, windev 1.9 ms).
- Warm-cache restart of 1,000 scripts ≤ 2 s: PASS (0.2 s / 1.7 s).
- Kill latency ≤ 100 ms: PASS on macOS (about 32 ms), MISS on windev (about 210 ms).
- **Host CPU ≤ 25 % of a core at 10k/s: MISS.** windev runs K = 2 at about 153 %. On macOS, K = 2 runs about 240 % against InProcess's 74 % under the same
  harness. A sequential probe measured 156 µs of host CPU per evaluation, and 63 µs with
  `DOTNET_ThreadPool_UnfairSemaphoreSpinLimit=0`. Thread-pool spinning is therefore most of it.
- **Worker idle RSS ≤ 80 MB: MISS.** It is about 190 MB on macOS and 158 MB on windev, because Roslyn is resident since compiles run in the worker.

Both misses → lmxopcua#789 (the §4.5 batching and per-eval allocation work).

**Two runner defects found by the harness and fixed in this slice.**
1. **Refused own run.** The single-flight slot was freed by a continuation that was not ordered against the caller's
   await. A caller that awaited one run and then started the next was sometimes refused with
   `previous evaluation still running`: 32 of 200 back-to-back runs, and about 80 % in the perf idle phase. The VT and
   Calc seams publish that as Bad. The slot is now freed before the caller resumes, idempotently.
   - Pinned by `A_caller_that_awaited_its_run_is_never_refused_by_it` (1,000 runs).
   - Mutation red.
2. **Heartbeat starvation under a compile burst.** Each load was compiled synchronously on its own thread-pool work item.
   On windev, 500 cold loads starved the heartbeat timer and the pipe writer. The host killed the shard
   (`KilledHeartbeat`), and all 500 keys became suspects. Compiles now run on `min(8, max(1, CPUs/2))` dedicated
   `ScriptWorker.Compile` threads.
   - Pinned by `A_burst_of_cold_compiles_does_not_starve_the_heartbeat` (1,500 cold scripts, 50 ms heartbeat).
   - Red on macOS with the thread-pool version. Green 3/3 on macOS and windev; on windev 1,500 cold compiles take
     about 18 s.

The host load timeout still counts the time a load waits behind other compiles → lmxopcua#790.

### 12.2 lmxopcua#789 / #790 / #788 record (2026-11-10)

**What cost the host CPU (#789).** With `DOTNET_ThreadPool_UnfairSemaphoreSpinLimit=0` a scratch probe showed the
out-of-process runner at 33 % of a core at 10k evals/s against 25 % in process: the IPC work itself was about
10 µs per evaluation. The rest was thread-pool spin-waiting, triggered by several pool work items per evaluation, and
on Unix by the socket engine. Changes:
- **Combining frame writer** (`ScriptWorkerFrameWriter`): the thread that queues a frame writes it itself when no
  write is in progress. This replaces the channel + writer task (one pool wake-up per submit). Consecutive
  evaluations or results are merged into one batch, frame order is preserved, and there is one write per pass. The
  old loop also called `PipeStream.FlushAsync`, which PipeStream does not override, so it queued a pool work item
  per pass.
- **Buffered frame reader** (`ScriptWorkerFrameReader`): one read pulls in every frame available, and frames are
  parsed from the buffer. It replaces two reads per frame plus a header allocation.
- **One timer per evaluation** for T and T+G. It replaces the `WaitAsync(timeout)` timer, the hard-deadline
  `CancellationTokenSource` and their two continuations.
- **Slot release and deadline cancel run on the reader, before the caller is resumed.** The release no longer
  depends on a continuation that is not ordered against the caller.
- **Result dispatcher** (`ResultDispatcher`, one thread per connection, blocking with `Monitor.Wait`, no spin):
  callers resume there, not on the pool. They do not resume on the reader either, because the Calculation pump
  publishes to OPC UA subscribers after its await. On the reader, a slow consumer would hold up heartbeats and get a
  healthy worker killed. That is pinned by a test, and a mutation that resumed callers on the reader was red.
- **Blocking I/O on Unix** (`ScriptWorkerFrameCodec.UseBlockingIo`):
  - The socket under a Unix .NET pipe turns non-blocking for good after one async call. After that even a
    "blocking" read goes through the epoll thread plus a spinning wait.
  - Both ends now connect, handshake, read (on a dedicated thread) and write synchronously. This was the largest
    single gain: sequential host CPU per evaluation fell from about 85 µs to about 13 µs.
  - A full pipe blocks only the one thread draining the writer; every other producer just queues. A peer that stops
    reading is killed by the heartbeat check within 3 s, which fails that write.
  - Windows keeps overlapped async I/O, because a synchronous full-duplex handle serialises reads against writes.
- **Kill via `TerminateJobObject`** on Windows. The job admits one process, so no system-wide snapshot is needed.
- Measured and **not** kept for memory, because none moved the compiled-worker RSS:
  - ReadyToRun worker publish: 195 MB;
  - `GCgen0size` 4 MB, `GCConserveMemory`, `TieredPGO=0`;
  - an aggressive compacting GC after a compile burst.
  The memory is Roslyn's JIT'd code and metadata, not the heap.

**Before / after (§8 harness, `OTOPCUA_SCRIPT_PERF=1`; macOS figures under 5–25 load average from other agents on
the box).**

| Budget | macOS before | macOS after | windev before | windev after |
|---|---:|---:|---:|---:|
| Host CPU at 10k/s, K=2, raw (InProcess same phase) | 228–240 % (71 %) | 79 % (69 %) | 153 % (6 %) | 123 % (7 %) |
| … vs InProcess (budget ≤ 25 %) | +157 | **+10 ✓** | +147 | +115 ✗ |
| Host CPU, idle sequential, K=2 | 258 % | 45 % | 80 % | 63 % |
| Idle p50 (≤ 150 µs) | 41–43 µs | 23 µs ✓ | 795 µs | 507 µs ✗ |
| p99 at 10k/s, K=2 (≤ 2 ms) | 720–796 µs | 1,024 µs ✓ | 1,916 µs | 1,676 µs ✓ |
| Closed loop ×64, K=2 (gate ≥ 5k/s) | 342k/s | 442k/s ✓ | 51k/s | 62k/s ✓ |
| Worker RSS after a compile (≤ 80 MB) | 188 MB | 187 MB ✗ | 158 MB | 159–161 MB ✗ |
| Worker RSS, warm PE cache, no compile | not measured | 58 MB ✓ | not measured | 36 MB ✓ |
| Warm restart, 1,000 cached (≤ 2 s) | 245 ms | 206–223 ms ✓ | 1,715 ms | 1,778–1,921 ms ✓ |
| Kill both shards → every next result (≤ 2 s) | 512 ms | 479–502 ms ✓ | 2,028 ms | 2,004–2,087 ms (borderline) |
| Kill latency after T+G (≤ 100 ms) | 31–35 ms | 33–43 ms ✓ | ~210 ms | **10–21 ms ✓** |

Additional measurements:
- In a scratch probe with a steady 10k/s load, K=2, before vs after: 116–158 % → 49–52 % of a core (InProcess
  60–65 %). Sequential runs: about 25k → 46k evals/s.
- The first harness timed the kill from before the runaway's call, so it also counted the cold compile of each
  distinct runaway key (100–200 ms on the VM). The harness now times from the soft-timeout answer. The windev
  "before" figure of about 210 ms is therefore mostly compile time, not kill time.
- macOS K=1 p99 at 10k/s read 2.3 ms in one loaded run (budget 2 ms). K=2 is the configuration the budget names.

**Still missing after this slice (lmxopcua#789 stays open, further tuning deferred by the owner):**
- Windows host CPU for IPC: +115 points at 10k/s on the VM. That path still uses overlapped async I/O, so it pays a
  completion-port and pool hand-off per frame. The next step is either two simplex pipes with synchronous handles and
  dedicated threads, or a measurement on a non-VM box.
- Windows idle p50: 507 µs. The VM's in-process baseline is already 92 µs.
- Worker RSS after a compile. The proposed revised budget is in §8.2. An alternative is to recycle a worker from the
  PE cache after a compile burst (blue/green); that is not built.
- Windows kill-both-shards: about 2.0 s, at the budget line.

**#790 — load timeout from compile start.**
- The worker sends `LoadStarted` when a compile thread picks a load up. The host arms that load's timer then,
  instead of each caller timing from when it sent the load.
- A queued load is never blamed. A compile still running `LoadTimeout` after its start kills the worker and strikes
  that key only (`LoadEntry.TimedOut`).
- `Hello.compile_threads` is a test knob; 0 keeps the worker's default.
- Tests:
  - `Compiles_queued_behind_others_longer_than_the_load_timeout_are_not_killed_or_struck`: one compile thread, and
    a burst grown until it outlasts three timeouts (asserted). Red with the timer armed at send.
  - `A_compile_still_running_a_load_timeout_after_it_started_is_killed_and_struck`: the positive control, with two
    kills leading to quarantine and the healthy key unblamed. Red with the timer never armed.

**#788 — start-failure logging.**
- The first failed start of a down period is one Error with the exception.
- Later retries are a Warning without a stack, carrying the attempt count, at most one per 60 s per shard; the rest
  log at Debug.
- The start that ends the outage is one Information line: `started … after N failed start attempt(s) over T`.
- Tests: `A_shard_that_cannot_start_logs_one_error_then_concise_rate_limited_retries_then_its_recovery`, with two
  intervals. Red when every failure is logged at Error with the exception.

### 12.3 Fourth P1 slice: batch submit, exact per-item deadlines, the scripted-alarm seam (2026-11-10)

**Batch submit (§4.5, P0).**
- `IScriptRunner.RunBatch(invocations)` returns one task per item. Results are per item and streamed.
  - The default implementation is `RunAsync` per item, started together. `InProcessScriptRunner` uses it.
  - `OutOfProcessScriptRunner` sends every item that is ready to run as ONE `EvalBatch` frame per worker. Ready means
    admitted, not quarantined, not a suspect, and already loaded in a live worker at this source and slot order.
  - Anything else goes through the full `RunAsync` path on its own: a first evaluation that needs a compile, a worker
    that is down, or a suspect bound for the one-at-a-time isolation worker. A batch therefore never waits for a load
    or a restart as a whole.
- Seams that use it, on the out-of-process path only:
  - **Calculation pump.** It dequeues everything pending (each entry's dependency snapshot was taken at enqueue),
    submits one batch, and publishes in dequeue order. A tag queued twice ends the batch, so every queued evaluation
    still runs, in order, against the snapshot that triggered it. The published sequence is the one-at-a-time pump's.
  - **VirtualTag seam.** There is one actor per tag, so an upstream burst makes many tags ready at once from different
    threads. `ScriptBatchCoalescer` hands those to one `RunBatch`. It has no timer: an evaluation on an idle seam is
    submitted at once by its own caller.
  - **Scripted alarms.** See below.
- In process, the Calc pump and the alarm engine stay one at a time, unchanged. The VirtualTag in-process path does
  not touch the coalescer.

**Per-item deadlines and blame stay exact under batching.**
- The host arms T and T + G at submit. That is when the run starts, provided the worker has an idle evaluation thread.
- A batch larger than the worker's scheduler, or work queued behind a runaway, used to wait in the worker with its
  clock already running. It could then be timed out, take timeout strikes, and even be the key blamed at T + G.
- The worker's scheduler now hands an evaluation straight to an idle thread, or else puts it in a backlog and reports
  it in `EvalResults.deferred_eval_ids`.
  - The host stops that evaluation's clock.
  - When a thread picks the evaluation up, its id rides back in `started_eval_ids` on the result frame of the run that
    thread just finished. The host then restarts the full T, then G, from the start.
  - The two notices are unordered on the wire, and the host handles either order.
  - The deadlines are absolute timestamps that the timer re-checks, so a stale tick never fires early.
- A run queued behind a runaway is therefore neither timed out nor struck. At the kill it is a victim
  (`script worker restarted`, `BadResourceUnavailable`, re-evaluated on return), not the culprit.
- Tests:
  - `A_runaway_in_a_batch_is_blamed_alone_and_siblings_queued_behind_it_are_not_timed_out_or_struck` runs one evaluation
    thread and a batch of a runaway plus three siblings. It is red with the host ignoring the deferral, and red with the
    worker never deferring.
  - `A_deferred_run_is_timed_and_killed_from_when_it_starts` is the positive control. It is red with a start notice that
    does not re-arm the clock.
- This also changes plain `RunAsync` under saturation: a run that waits for a thread no longer times out on its wait.
  That matches the in-process guard, whose budget clock starts when the pool picks the run up (§1.2).

**Scripted-alarm seam (★10: alarms last).**
- `Scripting:Runner:Seams` unset now means VirtualTag + Calculation. **`ScriptedAlarm` runs out of process only when it
  is listed.**
- `ScriptRunnerSelection` serves it. The Runtime's `DriverHostActor` gets it through the new `IScriptRunnerProvider`
  and passes it to the engine (a new last parameter, `scriptedAlarmRunner`).
- Engine changes, all on the runner path only:
  - `LoadAsync` compiles each predicate to PE bytes only (`CompileToImage`). A broken predicate still fails the load
    exactly as in process, and no assembly is loaded into the host. Deviation: Roslyn still runs in the host at load
    for alarms; the worker compiles again for execution.
  - Startup recovery and every re-evaluation pass snapshot each dirty alarm's inputs under `_evalGate`, submit all
    their predicates as one batch, and then apply the results in the serial order under the gate: Part 9 transition,
    persist before update, emissions. A test shows the transition sequence is identical to an in-process engine's
    for the same pushes.
- Failure semantics (§9, alarm column):
  - Timed out, killed, refused or quarantined: the condition is frozen with a `BadInternalError` annotation, as today.
  - The runner's fault (worker restarted, none ready): frozen with **`BadResourceUnavailable`**, consistent with
    VT/Calc. The alarm is re-evaluated by itself when the runner's `Recovery` completes.
  - A predicate that threw keeps its state with no annotation, as in process.
  - No outcome but a completed `bool` reaches the state machine, so a dead worker never clears an alarm.
- Real-worker tests:
  - `A_runaway_predicate_is_killed_and_quarantined_its_alarm_frozen_Bad_and_a_sibling_keeps_evaluating`;
  - `A_worker_that_dies_under_a_predicate_freezes_it_BadResourceUnavailable_never_clears_it_and_it_is_re_evaluated_on_return`.
    It is red when the runner's failure is read as `false`.
- Budget note: since lmxopcua#779 the host passes `Scripting:RunTimeout` (2 s) to the engine. The passes below are
  still judged against the stricter 250 ms the design was written for.

**Measured (§8 harness, `Batch_throughput_and_alarm_pass_latency`; macOS Arm64, 14 CPUs, load average about 14 from
other work on the box, 10 s phases):**

| Batch size | RunBatch evals/s | RunAsync ×n evals/s | RunBatch p50 / p99 µs | RunAsync ×n p50 / p99 µs |
|---:|---:|---:|---:|---:|
| 1 | 38,296 | 40,980 | 24 / 41 | 23 / 35 |
| 10 | 147,035 | 118,516 | 57 / 121 | 70 / 142 |
| 100 | 369,304 | 232,967 | 215 / 1,095 | 363 / 1,341 |
| 500 | 463,033 | 256,232 | 818 / 2,541 | 1,701 / 3,178 |

(OutOfProcess K = 2, one batch at a time. The in-process rows show no difference between the two, as expected.)

| Alarms per pass | InProcess p50 / p99 ms | OutOfProcess batched p50 / p99 ms | OutOfProcess serial p50 / p99 ms |
|---:|---:|---:|---:|
| 100 | 0.4 / 1.5 | 0.4 / 2.1 | 3.6 / 4.9 |
| 500 | 2.2 / 4.7 | 1.7 / 4.8 | 16.1 / 18.0 |
| 1,000 | 4.4 / 7.7 | 2.5 / 41.0 | 28.1 / 33.4 |

- A pass is push → the last of the N transitions it emits. "Serial" replays the pre-batch one-predicate-at-a-time pass
  over the same runner. That is the N × round-trip cost §8.2 warned about: about 30 µs per alarm, against 2–4 µs
  batched.
- Every configuration is far inside 250 ms. The one 41 ms p99 is a single loaded-box outlier.
- The §8.2 harness rerun on the same box after the scheduler change:
  - K = 2 closed loop ×64: 366k evals/s;
  - paced 10k/s: +5 % host CPU vs InProcess (budget ≤ 25 %);
  - idle p50: 25 µs;
  - kill latency: 14–19 ms;
  - warm-cache restart of 1,000 scripts: 220 ms;
  - every result correct.

**windev** (Windows VM, 36 CPUs, .NET 10.0.8; the full `ScriptWorker.Tests` suite also passes there, 157/157):

| Batch size | RunBatch evals/s | RunAsync ×n evals/s | RunBatch host CPU | RunAsync ×n host CPU |
|---:|---:|---:|---:|---:|
| 1 | 1,810 | 1,921 | 63 % | 65 % |
| 10 | 13,013 | 11,220 | 109 % | 157 % |
| 100 | 48,685 | 40,699 | 129 % | 224 % |
| 500 | 86,322 | 83,116 | 143 % | 289 % |

| Alarms per pass | InProcess p50 / p99 ms | OutOfProcess batched p50 / p99 ms | OutOfProcess serial p50 / p99 ms |
|---:|---:|---:|---:|
| 100 | 8.2 / 39.0 | 2.9 / 5.1 | 51.7 / 64.8 |
| 500 | 42.3 / 53.3 | 7.4 / 35.7 | **236.5 / 273.3 (MISS)** |
| 1,000 | 36.6 / 84.2 | 11.7 / 52.7 | **466.8 / 519.4 (MISS)** |

- On the VM, where one round trip costs about 0.5 ms, the pre-batch serial pass **misses the 250 ms budget from about
  500 alarms**. The batched pass stays at p99 ≤ 53 ms for 1,000 alarms, below the in-process engine's own figures.
  That is the measurement behind §8's "batch submit is a prerequisite, not an optimisation", and the reason ★10 put
  alarms last.
- On Windows, batching halves the host CPU of a 100–500 evaluation burst. That is the overlapped-I/O path that is still
  #789's main miss.
- No predicate failed to complete in any pass on either host.
- Rerun of the §8.2 harness on windev:
  - K = 2 closed loop ×64: 50k evals/s, gate PASS;
  - paced 10k/s: +99 % host CPU vs InProcess (#789, unchanged in kind);
  - kill latency: 9–19 ms;
  - warm-cache restart of 1,000 scripts: 1.8 s.

### 12.4 P2 record: Shadow mode, and the docker-dev drill with alarms (2026-11-10)

**What `Scripting:Runner=Shadow` does (as built).**
- The seams selected by `Scripting:Runner:Seams` (same rule as OutOfProcess: unset = VirtualTag + Calculation,
  `ScriptedAlarm` only when listed) receive a `ShadowScriptRunner` (`Core.Scripting`) as their runner. They recognise it
  as an `IScriptShadow` and keep their **in-process path byte for byte**: no VT coalescer, the one-at-a-time Calc pump,
  the in-process alarm engine. After each compiled evaluation they call `Mirror(invocation, publishedOutcome)`. The
  passthrough fast path and host-lifecycle failures (evaluator disposed, shutdown cancel) are not mirrored.
- `Mirror` copies the inputs (the alarm engine reuses its read cache), drops the script logger (the in-process run
  already wrote its lines) and queues the shadow run on the pool. It never blocks and never throws.
- **Comparison:** value (structural for arrays), exact CLR type, status, failure class (`compile`; `budget` = timed
  out / killed / refused / quarantined, which the two runners reach by different routes for the same runaway; `threw`
  with its message; `unsupported_type`). A shadow outcome that is the runner's fault (`Unavailable`, worker busy) is
  not compared (`shadow_unavailable`). ★11: a value the protocol cannot carry is a mismatch with reason
  `unsupported_type`.
- **Bounded:** `Scripting:Shadow:MaxInFlight` (256) outstanding and one per script; the rest are dropped and counted
  (`behind` | `key_busy`). One per script also keeps the out-of-process single-flight rule from refusing a shadow run
  and reporting a false mismatch.
- **Observability:** `otopcua.script.shadow.compared{kind, result}`, `.mismatch{kind, reason, script}`, `.dropped{kind,
  reason}`, `.inflight`. One Warning per script and reason, `Script shadow mismatch: <kind> <key> (<reason>) — published
  …; shadow …`, re-armed at the seam's deploy boundary and capped at `Scripting:Shadow:MismatchLogsPerMinute` (20).
- **Readiness:** `ScriptRunnerSelection.Status()` is null under Shadow; the `script-runner` entry is always Healthy and
  carries the shadow pool's state and the counters as data. The shadow pool's kill/quarantine lines go to the host log
  with `ScriptRunnerRole=Shadow` (the console template does not render that property), never to the script log.
- Deviation from the §12 sketch ("mismatching keys are logged once each"): once per key **and reason**, plus the
  per-minute cap.

**Tests** (all mutation-checked; 16 mutants, all red after one survivor got an assertion):
- `ShadowScriptRunnerTests` (Core.Scripting, fakes): the comparison table; a divergent script counted per script in the
  metric with one diff-summary line; the rate limit and suppressed-count note; reset re-arming; a shadow runner that
  hangs, blocks its caller or throws never reaches the caller and is never a mismatch; MaxInFlight drop accounting;
  inputs copied before `Mirror` returns.
- `ShadowModeTests` (real worker): eleven scripts over short / double / string / arrays / DateTime / bool / status /
  a throw publish exactly what InProcess publishes and match; a deliberately divergent script (it does a culture-sensitive
  `string.Compare("a", "B")`: the worker runs `InvariantGlobalization` so it compares ordinally and answers differently;
  originally it read `AppContext.BaseDirectory`, which lmxopcua#792 then denied) and a `decimal` result are counted against
  exactly those two scripts (`value`, `unsupported_type`). A runaway that only runs away in the worker is killed twice
  and quarantined **there** while every published value stays the in-process one, no published evaluation waits more
  than 250 ms, and its sibling on the killed shard (a victim, not compared) has no mismatch. Alarms: the emission
  sequence equals an in-process engine's across a `kill -9` of the shadow worker, and no shadow failure annotates a
  condition.
- Host: selection (Shadow hands out the shadow, `Status()` null), binding/clamps, the health matrix (Healthy whatever
  the shadow pool's state), the VT seam's mirror.
- Suites (macOS, Release, 0 warnings, `AKKA_TEST_TIMEFACTOR=3`): Core.Scripting 255/1 skip, ScriptedAlarms 109,
  VirtualTags 16, Calculation 84, ScriptWorker 160/4 skip, Host 275, Runtime 1002/3 skip, OpcUaServer.IntegrationTests
  133. windev (Windows VM, scratch dir under `C:\build`): ScriptWorker suite 160 passed / 4 skipped, 0 warnings. The first
  windev run caught a test bug (a `DateTime` seconds argument past 59 once the slower VM looped longer), fixed before the commit.
- Found while writing them: scripts could reach `System.AppContext` (`SetSwitch` / `SetData` / `GetData`) →
  lmxopcua#792, fixed: `System.AppContext`, `System.Console`, `GCSettings`, `ProfileOptimization`, `EncodingProvider`,
  `TypeDescriptor`, `CryptoConfig` and the ambient `CultureInfo` / `TimeZoneInfo.ClearCachedData` members are now denied
  by `ForbiddenTypeAnalyzer` (in-process and in the worker, one analyzer).

**docker-dev drill** (image `otopcua-host:w34-shadow` built from this branch; `central-1`/`central-2` only; drill rows
`w34-*` inserted by SQL: VTs `w34mul` (HR5×3), `w34raw` (`var v = …HR5.Value; return v;` — compiled, returns the boxed
`ushort`), `w34div` (divergent), `w34dec` (`* 1.5m`), Calc `W34Calc` (HR5+1000), alarms `w34-al-hi` (HR5>50),
`w34-al-slow` (1.5 s spin, then HR5>50), `w34-al-run` (`while(true)` when HR5 == 77). HR5 driven by Modbus writes to the
fixture.)

(a) **Shadow, all three seams** (`docker-compose.script-runner-shadow.yml`):

| Step | Observed (both nodes) |
|---|---|
| HR5 = 21 / 60 / 30 | `w34mul` 63 / 180 / 90, `w34raw` 21 / 60 / 30, `W34Calc` 1021 / 1060 / 1030, `HR5x2` 42 / 120 / 60, all Good; `w34-al-hi` and `w34-al-slow` Active at 60 and Inactive at 30 |
| Mismatch counters (leader) | `compared{virtual_tag,match}` 146, `{calculation,match}` 11, `{alarm,match}` 14; `mismatch{virtual_tag,value,w34-vt-div}` 5 and `{virtual_tag,unsupported_type,w34-vt-dec}` 4 — **nothing else**. `dropped{key_busy}` 5 (startup burst) |
| Log | exactly two Warnings per node: `w34-vt-div (value) — published Ok Int32 1; shadow Ok Int32 2` and `w34-vt-dec (unsupported_type) — published Ok Decimal 7.5; shadow Failed [unsupported_type] … System.Decimal` |
| `kill -9` both shadow workers on the leader, then HR5 = 61 | Published values all correct and Good (183 / 61 / 1061 / 122); workers restarted within ms; `/health/ready` 200, `script-runner` Healthy |

- **[A-7] answered for Modbus:** a UInt16 register arrives boxed as `ushort` and crosses the boundary unchanged (`w34raw`
  matched on every evaluation).
- **[A-8] answered:** in process, a VT that returns a `decimal` publishes **`BadNotSupported`** (`0x803D0000`) with a null
  value: the sink rejects it. Out of process it would publish `BadInternalError` "unsupported type". Either way it is Bad,
  so the ★11 behaviour change is only the status code.

(b) **OutOfProcess with `ScriptedAlarm` listed** (`docker-compose.script-runner.yml` + `Seams__2=ScriptedAlarm`):

| Step | Observed (both nodes) |
|---|---|
| HR5 = 60, then 30 | `w34-al-hi` / `w34-al-slow` Active, then Inactive, Quality Good; `otopcua_script_ipc_roundtrip{kind="alarm"}` counting, so the predicates ran in the worker |
| HR5 = 77 (runaway), 76, 77 | `exceeded its hard deadline (4000 ms); killing ScriptWorker[0]` twice, then `quarantined: "it exceeded its hard deadline and its worker was killed (2 time(s) in the last hour)…"`. `w34-al-run` stays Inactive with Quality **BadInternalError**; `w34-al-hi` and `w34-al-slow` (same shard) stay Active and Good throughout |
| HR5 = 30, `kill -9` the shard-0 worker 0.6 s later, during `w34-al-slow`'s 1.5 s spin | `ScriptWorker[0] ended (Crash) with no deadline crossed; 1 script(s) in flight move to the isolation worker: 3:w34-al-slow`. Read at once: `w34-al-slow` **Active, Quality BadResourceUnavailable** (`0x80040000`): frozen, not cleared. By the next read, 19 s later, it is Inactive and Good, re-evaluated in the isolation worker with no further input change. `w34-al-hi` cleared normally; `w34-al-run` stays BadInternalError (quarantined) |
| `/health/ready` | 200, `script-runner` Healthy ("ready") after the restarts |

(The kill loop stopped after one worker per node because its own shell matched the pattern; the killed worker was the
one running the in-flight predicate, which is what the step needed.)

**Rig state afterwards.**
- `w34-*` rows deleted (Script 3, VirtualTag 1, ScriptedAlarm 1, Tag 55, as before), HR5 = 5, redeployed: revision
  `e221b179…`, identical to the pre-drill deployment.
- `central-1`/`central-2` recreated from the base compose file in the same compose project: image `otopcua-host:dev`
  (`24f9d7287ed2`, never re-tagged), no `Scripting__*` env, compose config hash equal to the base file's. The secret
  stores live on the `otopcua-localdb-central-*` volumes since an earlier change, so they **survived** the recreates:
  central-1's store is byte-identical; central-2's main file differs only by a WAL checkpoint (same size). Both log
  `Mesh PSK secret ot-mesh-psk-* … is absent` at startup — central-1 with an unchanged store logs it too, so that is
  the state the rig was already in (most likely from the earlier drill's store loss), not something this drill did.
- The drill image `otopcua-host:w34-shadow` was removed afterwards. The site nodes, sql and traefik were not touched.

### 12.5 Installer record: worker identity, pipe ACL, firewall, PeCacheDir ACL (2026-11-10)

Owner decisions §14 #5 and #6, as built. Operator reference: `docs/ServiceHosting.md` §"Worker isolation on Windows".

- **Identity, deviating in mechanism from "run as a virtual account".** Windows starts a process as `NT SERVICE\<name>` only
  when the SCM starts that service. A host cannot launch K + 1 children that way, each with its own nonce, pipe and Job
  Object. A separate local user would need a stored password plus `SeAssignPrimaryTokenPrivilege` on the host account.
  `Scripting:Worker:Identity` (default unset; Windows only; startup refuses it elsewhere or when malformed) therefore starts
  each worker under a **restricted token** derived from the host's (`RestrictedWorkerLauncher`):
  - `DISABLE_MAX_PRIVILEGE`, so only `SeChangeNotifyPrivilege` is left;
  - restricting SIDs = the virtual account's SID (computed from the name, `S-1-5-80-…`) + Everyone + Users + RESTRICTED;
  - token and default DACL = host + SYSTEM + identity + RESTRICTED;
  - created suspended and put in its Job Object before it runs, inheriting exactly its two stdio pipes;
  - `TEMP` set to its ACL'd work directory, and `DOTNET_EnableDiagnostics=0`.

  The installer registers a disabled anchor service, so the account exists and resolves by name.
- **Found on windev.** The runtime would not start under the restricted token (`coreclr.dll` → `ERROR_DLL_INIT_FAILED`). A
  bisection over the token's groups showed that only the user or Administrators SID fixed it, not the logon SID. The cause
  was USER32's window station and desktop connection, so the host now grants the identity on its own window station and
  desktop. The logon SID stays out of the restricting set because the host's process DACL gives it `PROCESS_VM_READ`. The
  token's own DACL also needed rewriting (the handle needs `WRITE_DAC`).
- **Pipe ACL.** On Windows the pipe always has an explicit, protected DACL: the host account and the identity, read/write,
  nothing else. .NET's `CurrentUserOnly` grants the token *owner*, which is Administrators on an elevated host; this ACL
  does not. The owner is still the token owner, so the worker's `CurrentUserOnly` client check is unchanged.
- **Installer** `scripts/install/Install-ScriptWorkerIsolation.ps1` + `ScriptWorkerIsolation.psm1`. It is idempotent, with
  `-WhatIf`, `-Uninstall` and `-PlanOnly`, and it sets up:
  - the anchor service;
  - `PeCacheDir`: protected, SYSTEM / Administrators / host full control, no Users, and the key and markers host-only;
  - `pe\`: the identity gets Modify;
  - `script-worker\`: the identity gets read+execute;
  - **deny** ReadData for the identity on the install folder (appsettings, logs, host binaries);
  - outbound **block** firewall rule `OtOpcUa-ScriptWorker-BlockOutbound` on the worker apphost;
  - optional host-service env (`-ConfigureHostService`).

  The runner flip itself is not part of it.
- **Tests.**
  - `scripts/install/tests/Test-ScriptWorkerIsolation.ps1` is the dry-run harness. It runs on any OS and asserts ids,
    order, targets, SDDL and the rollback (46 assertions). A mutation check (deny → allow, protected → inherited) turns
    4 red.
  - `ScriptWorkerIdentityTests` (Core.Scripting) pin the SID algorithm against `NT SERVICE\TrustedInstaller`. The
    PowerShell harness pins the same default SID.
  - `WorkerIsolationTests` (ScriptWorker) cover the pipe ACL (built and live), the token, and a real worker under the
    identity running a script with a restricted token. A plain worker's token is unrestricted (positive control).
    - The token case is restricted, privilege-free, and impersonated: it reads System32 and writes its work dir, but
      cannot read or write the host's temp.
    - They ran green on windev without changing machine state, apart from the session's window station / desktop ACE.
      No account, firewall rule or persistent ACL was created there.
- **Not verified live:** the installer on a real node. That is the QA deploy gate on lmxopcua#780's follow-up (see
  that issue for the checklist).

---

## 13. Test plan

**Unit (`Core.Scripting.Tests`, new `ScriptWorker.Tests`):**

- Frame codec:
  - round-trip every `Variant.TypeTag` (arrays included);
  - reject zero-length, oversize and truncated frames;
  - reject an `eval_id` with no outstanding request;
  - fuzz the reader with random bytes and assert there is no exception other than a protocol fault.
- **Unboxing parity:** for each tag, a script doing `(T)ctx.GetTag("x").Value` behaves identically in and out
  of process. This pins [A-7].
- Attribution ledger:
  - a deadline kill blames exactly one key;
  - a crash with 3 in flight → 3 suspects → isolation;
  - an isolation crash → one strike;
  - probation returns a suspect to its shard;
  - quarantine after 2 kills;
  - a source change lifts the quarantine;
  - a redeploy lifts every quarantine.
- Backoff schedule, jitter bounds, the reset-after-healthy rule and the load-phase crash bisection.
- Options binding: clamps plus a Warning per clamped key; `PeCacheDir` relative or inside the content root is a
  startup failure.

**Containment (worker-backed, `Category=ScriptWorker`; runs in CI on Linux, gated on windev for Windows).** Each
case has a positive control: a healthy sibling script on the **same shard** that keeps producing correct values
before and after the event. Following scadaproj memory "assert the artifact before the outcome", each assertion
is checked to be capable of failing.

| Case | Assert |
|---|---|
| `while(true){}` ignoring the token | Killed within `T + G + 100 ms`; host `ThreadPool.ThreadCount` and `Process.Threads` unchanged; no orphan (`InFlightCount == 0` after the kill). Mutation check: disable the kill and see the test fail |
| `while(!ctx.CancellationToken.IsCancellationRequested){}` | Ends at `T` by itself; **no** kill; no strike beyond the timeout |
| Allocation bomb | Worker dies of OOM or the heap limit; **host RSS stays below a bound** and the host stays up; suspect → isolation → quarantine |
| Unbounded recursion | Worker dies; host up; quarantined after the isolation reproduction. Also the first in-repo reproduction of [A-1] for the in-process mode, marked `Skip` there because it would kill the test host |
| Worker killed externally (`kill -9`) | Victims Bad with "script worker restarted", then recover with the latest inputs; the alarm condition is **not** cleared |
| Host killed | Worker exits within 2 s (pipe EOF on Linux, Job Object on Windows) |
| Environment | The worker's environment contains no `ServerHistorian__ApiKey` / ConfigDb variables (set them in the test host first: positive control) |
| Pipe | A second client connecting to the pipe is refused; a wrong nonce is refused |
| Loaded assemblies | The worker's loaded set equals the pinned list; Akka / SqlClient / Opc.Ua are absent |
| Crash loop | A poison generation trips the circuit; state `Unavailable`; keys Bad with the runner-unavailable reason; recovery after a fixed redeploy |

**Parity:** run the existing `RoslynVirtualTagEvaluatorTests`, `CalculationDriver*Tests` and
`ScriptedAlarmEngine*Tests` against **both** runners. Use xunit theory or class fixtures over
`IScriptRunner`, with the worker built as a test dependency.

**Performance (`Category=Perf`, opt-in, not in CI):** a Stopwatch-based harness (no new packages) that
measures the §8 table: p50/p99 added latency, throughput at K=1/2/4, host CPU, restart-to-first-result with
1,000 cached PEs, and RSS per shard.

**Live gates:**
- docker-dev rig: a deploy with VTs, Calc tags and scripted alarms; drill a runaway script through the
  AdminUI and watch `/uns` go Bad, the quarantine appear, and siblings stay Good;
- QA shadow week (P2);
- a QA flip with the same drill.

---

## 14. Open decisions for the owner

Each has a recommendation, marked ★.

1. **Pool shape.**
   - ★ K=2 shards plus an on-demand isolation worker;
   - one worker per node, simplest, but every kill blips every script;
   - per-seam workers.
2. **Hard grace `G`.**
   - ★ `max(1 s, T)`, which keeps cooperative cancellation useful;
   - `G = 0` (kill at `T`): strictest, but every soft overrun restarts a shard.
3. **Status code for "runner unavailable / worker restarted".**
   - ★ `BadResourceUnavailable`, which separates a sick node from a broken script but is a ScadaBridge-visible
     contract change;
   - keep `BadInternalError` everywhere, for zero consumer impact.
4. **Fallback when the worker cannot start.**
   - ★ fail closed: Bad, and no in-process execution;
   - fail open to in-process with an Error, which keeps values flowing but silently reopens the risk.
5. **Worker identity on Windows.**
   - ★ a dedicated low-privilege virtual account created by the installer, which needs installer work, an
     explicit pipe ACL, and a deploy that touches the box;
   - the same account as the host: zero install change, and the Job Object plus the analyzer remain the
     controls.
6. **Network isolation.**
   - ★ a Windows Firewall outbound-block rule on the worker executable (installer), with the analyzer as the
     only control on Linux;
   - analyzer only, everywhere.
7. **Compile location.**
   - ★ in the worker: Roslyn blowups are contained and the host gets lighter;
   - in the host, shipping PE bytes: a lighter worker and faster cold start.
8. **PE cache store.**
   - ★ a node-local directory with an HMAC;
   - LocalDb (already a dependency; it would also replicate to the pair, which is unnecessary);
   - no persistence, recompiling on restart (violates the 2 s budget at scale).
9. **IPC encoding.**
   - ★ protobuf (already compiled in `Commons`, and mxaccessgw precedent);
   - MessagePack (pinned but unused, so in practice a new dependency).
10. **Scripted alarms in the first flip.**
    - ★ last, after Calc and VT, because of the 250 ms budget and the engine refactor;
    - all seams at once.
11. **Unsupported input/result CLR types.**
    - ★ an explicit Bad `UNSUPPORTED_TYPE`, surfaced in shadow mode first;
    - best-effort conversion (to string, `double`), which hides type bugs.
12. **Readiness when the circuit is open.**
    - ★ degraded but ready (OPC UA still serves driver data);
    - not ready, which on a redundant pair would shift ServiceLevel to the peer.
13. **Long-term status of `InProcess`.**
    - ★ keep it as the dev/test default and the rollback;
    - delete it after a soak period.
14. **Does this need a real-deployment scale target?** [A-14]'s 5k evals/s/node is a guess. The owner should
    supply the expected VT / Calc / alarm counts per node for the target sites, or accept the guess as the
    gate.

---

## 15. Assumptions (not verified from code)

| Id | Assumption | How to verify |
|---|---|---|
| A-1 | Unbounded recursion in a script throws `StackOverflowException` and kills the host today | Reproduce in a throwaway process (not the test host) |
| A-2 | A collectible ALC whose code is still executing on an orphan thread cannot be unloaded | .NET docs / a `WeakReference` test with a spinning orphan |
| A-3 | A minimal .NET 10 worker idles at tens of MB RSS | Measure the P1 worker |
| A-4 | `P = max(2, cores/K)` evaluation threads per worker is enough | P1 perf harness |
| A-5 | Passthrough scripts are a large share of evaluations on real deployments | Count `PassthroughScript.TryMatch` hits against compiled runs on QA (add a counter in P0) |
| A-6 | A local pipe round trip costs tens of µs | P1 perf harness |
| A-7 | Drivers box tag values with stable, enumerable CLR primitive types | Shadow-mode type census |
| A-8 | What `OpcUaPublishActor` / the sink does with an unsupported result object today | Trace `AttributeValueUpdate` → `SdkAddressSpaceSink` |
| A-9 | Drivers produce UTC `DateTime` values | Shadow-mode census on `DateTime.Kind` |
| A-10 | 512 MiB is a sane per-worker heap limit (ScadaBridge uses the same) | Size against measured RSS × script count |
| A-11 | docker-dev containers cannot create nested cgroups | Check the compose capabilities |
| A-12 | Warm Roslyn compile is tens of ms per script; cold is seconds | P1 perf harness |
| A-13 | Worker start is ~150 ms; a PE load into a collectible ALC is ~1 ms | P1 perf harness |
| A-14 | 5k compiled evals/s/node is a fair planning envelope | Owner input (decision 14) |
| A-15 | Today's in-process overhead per compiled run is µs to tens of µs | Add the same harness to `InProcessScriptRunner` in P0 for a baseline |

---

## 16. Prior art consulted

- **ScadaBridge `CentralUI.SandboxWorker`** (`~/Desktop/ScadaBridge/src/ZB.MOM.WW.ScadaBridge.CentralUI/ScriptAnalysis/SandboxWorkerProtocol.cs`).
  It runs one disposable process per *test run* over stdio, using 4-byte LE length-prefixed JSON frames with a
  1 MiB cap, admission 4, `DOTNET_GCHeapHardLimit` 512 MiB, and `Kill(entireProcessTree)` on the deadline.
  Taken from it: the heap limit, the tree kill and the frame cap. Not taken:
  - stdio (fragile for a long-lived worker);
  - per-run process spawn (too slow for runtime evaluation);
  - inheriting the parent environment;
  - proxying host calls.

  ScadaBridge's *runtime* scripts are still in-process (dedicated scheduler plus watchdog,
  `docs/plans/2026-08-15-script-pool-split-design.md`), so this design would be the family's first
  out-of-process **runtime** evaluator.
- **mxaccessgw gateway↔worker** (`~/Desktop/MxAccessGateway/docs/WorkerFrameProtocol.md`,
  `WorkerProcessLauncher.md`, `WorkerBootstrap.md`). It runs a long-lived supervised worker over a named pipe:
  protobuf envelopes, a negotiated frame cap, a priority-drained writer with coalesced flushes, sequence
  numbers, a nonce in an env var (never on the command line), startup probe plus retry, and a tree kill on a
  failed start. Taken from it: the frame protocol shape, the nonce handshake and the launcher discipline.
  Avoided: an orphan terminator that is not scoped to its own instance.