feat(scripting): out-of-process pre-emptible script runner behind Scripting:Runner (lmxopcua#780 P1)

Opt-in (Scripting:Runner=OutOfProcess; InProcess stays the default and its code path is unchanged). Implements
the P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md with the owner's §14 decisions:

- Options: Scripting:Runner (InProcess | Shadow | OutOfProcess), Scripting:Runner:Seams, Scripting:Worker:*
  (Shards 2, HardGrace max(1 s,T), HeapLimitBytes 512 MiB, QuarantineAfterKills 2, UnavailableGrace 5 s,
  IsolationProbationRuns 20, PeCacheDir validated absolute + outside the content root (fail closed),
  ExecutablePath). Shadow is not implemented yet: runs InProcess with a startup Warning.
- Core.Scripting: IScriptRunner seam + InProcessScriptRunner (today's guard, the parity baseline).
- Core.Scripting.Protocol: protobuf envelopes, 4-byte LE framing (zero/oversize/truncated/garbage refused),
  Variant carrying the exact CLR type; unsupported types are explicit failures.
- ScriptWorker app: compiles in the worker, P-thread scheduler, host-driven soft cancel, fail-fast
  watchdog, heartbeats, ctx.Logger forwarding, exits on pipe EOF / parent gone, forbidden-assembly
  self-check; Akka and Grpc.Core.Api excluded from its folder. Shipped in script-worker/ (Build + Publish).
- Core.Scripting.OutOfProcess: K shards + isolation worker, soft T / hard T+G kill, attribution
  (deadline kill = one key; crash = suspects -> isolation; isolation crash = strike), kill-strike
  quarantine, backoff 250 ms -> 30 s, fail closed, cleared allow-listed env, nonce + pid handshake on a
  current-user pipe, Windows Job Object.
- VirtualTag + Calculation seams use the runner when selected; a runner-side failure (worker restarted /
  runner unavailable) publishes BadResourceUnavailable (0x80040000) — ScadaBridge#112. Script faults keep
  BadInternalError. The scripted-alarm seam stays in process (alarms last).

Tests: new ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests (Category=ScriptWorker, real processes; green on macOS and
on windev/Windows), seam tests in Host/Runtime/Calculation suites; mutation-checked.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV

docs/ScriptEditor.md

@@ -211,6 +211,11 @@ scripted-alarm predicates) runs scripts through the shared `ScriptRunGuard`:
   `Scripting:MaxAllocBytesPerRun` (64 MB) is quarantined. This is sampled after the
   run returns, so a run that OOMs the process first is not stopped — that needs an
   out-of-process runner (→ #750).
+- **Out of process (opt-in, lmxopcua#780).** `Scripting:Runner=OutOfProcess` moves the
+  virtual-tag and Calculation seams' compiled runs into `ScriptWorker` processes, where a
+  run past `T + Scripting:Worker:HardGrace` is stopped by killing its worker and an OOM
+  kills a worker instead of the server. A run whose worker died under it reads
+  `BadResourceUnavailable`. See `docs/VirtualTags.md` §"Out-of-process runner".
 - **Recursion.** Unbounded recursion fails the script with
   `ScriptStackExhaustedException` (Bad, script log) instead of overflowing the
   stack, which would terminate the whole server. The runtime compile inserts a

docs/ServiceHosting.md

@@ -88,6 +88,16 @@ and supply `ServerHistorian__ApiKey` via the environment on the OtOpcUa host sid
 `historian:write` + `historian:tags:write`. See [Historian.md](Historian.md) for the full config-key and
 deployment-prerequisite reference. (The retired Wonderware TCP sidecar: [Historian.Wonderware.md](drivers/Historian.Wonderware.md).)
 
+## Script worker processes (`Scripting:Runner=OutOfProcess`, lmxopcua#780)
+
+Off by default. When enabled, a driver node launches up to `Scripting:Worker:Shards` + 1 child processes
+(`ZB.MOM.WW.OtOpcUa.ScriptWorker`, from `script-worker/` beside the host) on its first compiled script evaluation and
+kills them with it: a Job Object with kill-on-close on Windows, pipe end of stream plus a parent-pid poll elsewhere,
+and a Shutdown frame then a kill on host stop. Their stderr lands in the host log prefixed `ScriptWorker[<shard>]`.
+The worker currently runs under the host's account through the dotnet muxer; the dedicated low-privilege account,
+pipe ACL and firewall outbound rule (design §14 #5/#6) are installer work still open on lmxopcua#780. See
+`docs/VirtualTags.md` §"Out-of-process runner".
+
 ## Install / Uninstall
 
 - `scripts/install/Install-Services.ps1 -Roles admin,driver` — installs `OtOpcUaHost`.

docs/VirtualTags.md

@@ -37,11 +37,54 @@ User scripts are compiled via `Microsoft.CodeAnalysis.CSharp` (regular compiler,
 
 #### Known resource limits (accepted trade-offs)
 
-The sandbox cannot pre-empt a script. **Memory:** each run's own-thread allocation is sampled (`GC.GetAllocatedBytesForCurrentThread`) and a run over `Scripting:MaxAllocBytesPerRun` (default 64 MB) quarantines the script — but the sample is taken after the run returns, so a script that allocates fast enough (`new byte[int.MaxValue]` in a loop) can still OOM the process first. **CPU:** a tight loop keeps its pool thread after the timeout fires; `ScriptRunGuard` bounds that to one orphan per script by refusing further runs while it spins (Bad `"previous evaluation still running"`) and quarantines after three consecutive timeouts (a refusal is not a strike) — or immediately on a refusal behind an orphan that has run longer than `Scripting:StuckOrphanQuarantineFactor` (default 10) × the timeout, which is what quarantines a never-returning `while(true)`. Both are mitigations; pre-emptive CPU/memory limits need an out-of-process runner (→ #750). **Stack:** unbounded recursion used to terminate the whole server — a `StackOverflowException` cannot be caught, and lmxopcua#779 reproduced it out of process (local function, mutual recursion, self-referencing `Func<>`, anonymous self-application, recursion through LINQ: the child process died with "Stack overflow." / SIGABRT). `ScriptEvaluator.Compile` now rewrites the script so the body and every lambda, anonymous method and local function starts with `ScriptStackGuard.Check()` (`RuntimeHelpers.TryEnsureSufficientExecutionStack`), so the recursion fails the script with `ScriptStackExhaustedException` — Bad quality and a script-log entry like any throwing script — while there is still stack left. A runtime check rather than an analyzer rule because recursion is not decidable from syntax (anonymous self-application never names itself); bounded recursion keeps working. Limits: the depth reached depends on the evaluating thread's stack size; a recursion that never re-enters script code is not seen (the sandbox exposes no such API); a script can catch the exception, which only unwinds it. Pinned by `ScriptRecursionContainmentTests` (one child process per shape) and `ScriptStackGuardInjectionTests`. Script authoring is gated by the AdminUI `ConfigEditor` policy — **`Administrator` or `Designer`**, not Administrator alone — and that same policy gates deploy, so a Designer can author a script and put it into production; that gate is the primary control (AdminUI-001 deliberately admitted Designers to the script pages and `/api/script-analysis/*`; lmxopcua#779 corrected this sentence, which used to say Admin-only). See `docs/ScriptEditor.md` §"Runtime safety".
+The sandbox cannot pre-empt a script. **Memory:** each run's own-thread allocation is sampled (`GC.GetAllocatedBytesForCurrentThread`) and a run over `Scripting:MaxAllocBytesPerRun` (default 64 MB) quarantines the script — but the sample is taken after the run returns, so a script that allocates fast enough (`new byte[int.MaxValue]` in a loop) can still OOM the process first. **CPU:** a tight loop keeps its pool thread after the timeout fires; `ScriptRunGuard` bounds that to one orphan per script by refusing further runs while it spins (Bad `"previous evaluation still running"`) and quarantines after three consecutive timeouts (a refusal is not a strike) — or immediately on a refusal behind an orphan that has run longer than `Scripting:StuckOrphanQuarantineFactor` (default 10) × the timeout, which is what quarantines a never-returning `while(true)`. Both are mitigations; pre-emptive CPU/memory limits need the out-of-process runner (→ #750 design, #780; opt-in via `Scripting:Runner=OutOfProcess`, below). **Stack:** unbounded recursion used to terminate the whole server — a `StackOverflowException` cannot be caught, and lmxopcua#779 reproduced it out of process (local function, mutual recursion, self-referencing `Func<>`, anonymous self-application, recursion through LINQ: the child process died with "Stack overflow." / SIGABRT). `ScriptEvaluator.Compile` now rewrites the script so the body and every lambda, anonymous method and local function starts with `ScriptStackGuard.Check()` (`RuntimeHelpers.TryEnsureSufficientExecutionStack`), so the recursion fails the script with `ScriptStackExhaustedException` — Bad quality and a script-log entry like any throwing script — while there is still stack left. A runtime check rather than an analyzer rule because recursion is not decidable from syntax (anonymous self-application never names itself); bounded recursion keeps working. Limits: the depth reached depends on the evaluating thread's stack size; a recursion that never re-enters script code is not seen (the sandbox exposes no such API); a script can catch the exception, which only unwinds it. Pinned by `ScriptRecursionContainmentTests` (one child process per shape) and `ScriptStackGuardInjectionTests`. Script authoring is gated by the AdminUI `ConfigEditor` policy — **`Administrator` or `Designer`**, not Administrator alone — and that same policy gates deploy, so a Designer can author a script and put it into production; that gate is the primary control (AdminUI-001 deliberately admitted Designers to the script pages and `/api/script-analysis/*`; lmxopcua#779 corrected this sentence, which used to say Admin-only). See `docs/ScriptEditor.md` §"Runtime safety".
 
 `System.Threading.Tasks` is denied (Core.Scripting-003), which prevents `Task.Run` / `Parallel` fan-out that would spawn background work outliving the timeout.
 
 
+#### Out-of-process runner (`Scripting:Runner`, lmxopcua#780)
+
+`Scripting:Runner` selects where compiled scripts run: **`InProcess`** (the default and the rollback — everything
+above, unchanged), **`OutOfProcess`**, or `Shadow` (design P2; not implemented yet, so it runs `InProcess` with a
+startup Warning). `Scripting:Runner:Seams` narrows `OutOfProcess` to a subset (`VirtualTag`, `Calculation`; the
+`ScriptedAlarm` seam stays in process for now — alarms go last, design §14 #10). Under `OutOfProcess` the virtual-tag
+and Calculation evaluators hand each compiled run to `OutOfProcessScriptRunner`
+(`Core.Scripting.OutOfProcess`), which runs it in a pool of `ZB.MOM.WW.OtOpcUa.ScriptWorker` processes:
+
+- **K shards** (`Scripting:Worker:Shards`, default 2) placed by a stable FNV-1a hash of the key, plus an on-demand
+  **isolation worker** that runs suspects one at a time. No worker starts until the first compiled evaluation, so a
+  node without scripts never launches one. The passthrough fast path never leaves the host.
+- **Two deadlines.** At the soft deadline `T` (the seam's budget, unchanged) the caller gets today's
+  `script timed out after …` failure and the run token is cancelled inside the worker; at `T + G`
+  (`Scripting:Worker:HardGrace`, default `max(1 s, T)`) the worker is **killed** (`Process.Kill(entireProcessTree)`,
+  plus a worker-side fail-fast watchdog) — the thread, its memory and its ALC go with it.
+- **Attribution and quarantine** live in the host and survive restarts: a deadline kill strikes exactly its key
+  (other evaluations in flight are victims, not strikes); a crash with no deadline crossed makes every key in flight
+  a suspect that must complete `Scripting:Worker:IsolationProbationRuns` (20) clean runs in the isolation worker,
+  where a crash strikes exactly one key. `Scripting:Worker:QuarantineAfterKills` (2) strikes within an hour, or the
+  existing consecutive-timeout and allocation rules, quarantine with the usual `"script quarantined: "` prefix; a
+  source change or a redeploy lifts it.
+- **Status codes.** Script faults keep `BadInternalError`. When the runner, not the script, failed — the worker
+  restarted under the run (`"script worker restarted"`) or no worker became ready within
+  `Scripting:Worker:UnavailableGrace` (5 s, `"script runner unavailable"`) — the tag publishes
+  **`BadResourceUnavailable` (0x80040000)** with its last value. This is ScadaBridge-visible (ScadaBridge#112).
+  The runner **fails closed**: no worker means Bad, never an in-process run.
+- **Containment.** The worker gets a cleared, allow-listed environment (no `ServerHistorian__ApiKey`, no
+  connection strings), an empty temp working directory, `DOTNET_GCHeapHardLimit = Scripting:Worker:HeapLimitBytes`
+  (512 MiB), a Windows Job Object (kill-on-close, one process, a commit backstop), a random current-user-only pipe
+  with a launch nonce in its environment, and no Akka / gRPC assemblies in its folder. It exits on pipe end of stream
+  or when its parent pid disappears. Results cross as protobuf `Variant`s carrying the exact CLR type, so
+  `(short)ctx.GetTag(x).Value` behaves as in process; an input or result type with no tag (a `List<T>`, a `Guid`)
+  is an explicit `unsupported type: …` failure rather than a conversion — a behaviour change from in process.
+- **Deployment.** The Host build copies the worker into `script-worker/` beside `OtOpcUa.Host.dll`
+  (`ScriptWorker.Deploy.targets`, Build and Publish); `Scripting:Worker:ExecutablePath` overrides it. It is run through
+  the dotnet muxer, so it needs a framework-dependent host (the docker-dev image). Not yet done (lmxopcua#780 P1
+  remainder): the PE cache (`Scripting:Worker:PeCacheDir` is validated — absolute, outside the content root — but not
+  used, so a worker restart recompiles), a self-contained worker for the Windows installs, the low-privilege worker
+  account and firewall rule, the runner-state gauge / readiness component, and the scripted-alarm seam.
+
+Pinned by `tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests` (`Category=ScriptWorker`: real worker processes).
+
 #### Quarantine: what a client sees (status code)
 
 A quarantined script's output reads **`BadInternalError`** on OPC UA, the same code as a script that

docs/plans/2026-11-09-out-of-process-script-runner-design.md

@@ -665,6 +665,18 @@ per-shard state, restarts in the last hour and the quarantine list. Quarantines
    - Default stays `InProcess`. Enable on the docker-dev rig only.
    - Gate: §13 runaway / containment tests green on Linux (docker-dev) **and** Windows (windev box, which has
      the .NET 10 toolchain).
+   - **Partly landed in lmxopcua#780 (P1 slice):** `Scripting:Runner` / `Scripting:Runner:Seams` / `Scripting:Worker:*`
+     (bound, clamped, `PeCacheDir` path rule fail-closed), `IScriptRunner` + `InProcessScriptRunner`, the protobuf frame
+     protocol (`Core.Scripting.Protocol`), the `ScriptWorker` app (compile in the worker, P-thread scheduler, watchdog,
+     heartbeats, log forwarding, pipe-EOF / parent-pid exit, forbidden-assembly self-check), `OutOfProcessScriptRunner`
+     (K shards + isolation worker, soft/hard deadlines, attribution, kill-strike quarantine, backoff, fail closed,
+     Job Object on Windows), the VirtualTag and Calculation seams over it, and `BadResourceUnavailable` for
+     runner-unavailable / worker-restarted (ScadaBridge#112). The §13 containment cases run on macOS and Windows
+     (windev). Deviations: supervision is a plain object owned by DI, not `ScriptRunnerActor`/`ScriptWorkerActor`;
+     the per-seam batch submit (P0) is still one-at-a-time. Remaining: the HMAC'd PE cache, a self-contained worker
+     publish + the low-privilege account / pipe ACL / firewall rule (installer), the runner-state gauge and readiness
+     component, victim re-evaluation on shard return, isolation-worker idle stop, the orphan sweep by nonce file, the
+     scripted-alarm seam, the docker-dev enablement and the §8 perf harness.
 3. **P2, shadow mode on docker-dev and QA.**
    - `Shadow` runs the **in-process** evaluation as the published result, and *also* submits to the worker.
    - It compares value, status class and failure kind per evaluation, and counts