feat(scripting): PE cache, quarantine-race fix, recovery, readiness for the out-of-process runner (lmxopcua#780 P1)

Second P1 slice of docs/plans/2026-11-09-out-of-process-script-runner-design.md; still opt-in behind
Scripting:Runner=OutOfProcess (InProcess unchanged).

- Compiled-PE cache (design §7, §14 #8): Scripting:Worker:PeCacheDir holds <dir>/pe/<id>.pe entries authenticated
  with HMAC-SHA256 under a per-node random key (<dir>/pe-cache.key, created by the host, 0600/0700 on Unix), handed to
  the worker in Hello. Entry id = SHA-256(compile fingerprint + kind + source); the fingerprint covers the Roslyn
  version, the Core.Scripting build (analyzer, wrapper, stack guard), the sandbox reference set and the runtime. Edited,
  truncated, other-key or renamed entries are rejected, deleted and recompiled; the worker never loads unverified bytes.
  Pruned by recency at startup (30 d / 50 000). ScriptEvaluator splits into CompileToImage + FromImage. Measured on
  40 scripts, restart to every first result: macOS 820 ms no cache / 840 ms empty / 85 ms warm;
  windev (Windows) 9.5 s / 9.4 s / 1.0 s.
- Quarantine race: WorkerConnection runs exit attribution synchronously BEFORE completing the evaluations/loads it held
  (and before Exited), and the shard passes it straight to the runner — a kill strike is in place before the killed
  key's slot is released.
- Supervision: a dead worker restarts by itself after its backoff; Unavailable outcomes carry ScriptRunOutcome.Recovery
  (-> VirtualTagEvalResult.RetryWhen) and the VT actor / Calculation driver re-evaluate once against current inputs when
  it completes. Isolation worker stops after Scripting:Worker:IsolationIdleTimeout (10 min). Keys compiling when a
  worker dies are suspects too (isolation finds a poison compile and quarantines it). Startup orphan sweep by launch
  marker: kills only a process whose marker's host is gone and whose pid + start time + image match.
- Crash-loop circuit (Scripting:Worker:CircuitOpenAfter, 2 min): runner Unavailable, one Error, evaluations answered
  at once while shards keep retrying at the cap.
- Observability (§10): otopcua.script.runner.state, worker.up/inflight/queue_depth/heap_bytes (shard),
  ipc.roundtrip + eval.duration (kind), compile{cache}, isolation.suspects, exits reason=idle. `script-runner` health
  check on the ready tier, at worst Degraded (§14 #12: degraded but still ready).

Left on #780: self-contained worker publish, installer (account, pipe ACL, firewall), docker-dev live drill,
scripted-alarm seam, the §4.5 per-seam batch submit, the §8 perf harness.

Tests: SupervisionTests, PeCacheStoreTests/PeCacheWorkerTests (real workers), ScriptRunnerHealthCheckTests, VT + Calc
recovery seam tests; 14 mutations each turned their test red.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV

docs/ServiceHosting.md

@@ -94,7 +94,11 @@ Off by default. When enabled, a driver node launches up to `Scripting:Worker:Sha
 (`ZB.MOM.WW.OtOpcUa.ScriptWorker`, from `script-worker/` beside the host) on its first compiled script evaluation and
 kills them with it: a Job Object with kill-on-close on Windows, pipe end of stream plus a parent-pid poll elsewhere,
 and a Shutdown frame then a kill on host stop. Their stderr lands in the host log prefixed `ScriptWorker[<shard>]`.
-The worker currently runs under the host's account through the dotnet muxer; the dedicated low-privilege account,
+At startup the host kills workers a dead predecessor of the same install left behind, matched by launch marker
+(pid + start time + image, and only when the marker's host is gone). `Scripting:Worker:PeCacheDir`, when set, holds the
+compiled-PE cache, its HMAC key and the launch markers: put it outside the install folder (startup refuses a relative path
+or one inside the content root) on a directory only the service account can read. The `script-runner` readiness component
+is at worst Degraded. The worker currently runs under the host's account through the dotnet muxer; the dedicated low-privilege account,
 pipe ACL and firewall outbound rule (design §14 #5/#6) are installer work still open on lmxopcua#780. See
 `docs/VirtualTags.md` §"Out-of-process runner".
 

docs/VirtualTags.md

@@ -76,12 +76,49 @@ and Calculation evaluators hand each compiled run to `OutOfProcessScriptRunner`
   or when its parent pid disappears. Results cross as protobuf `Variant`s carrying the exact CLR type, so
   `(short)ctx.GetTag(x).Value` behaves as in process; an input or result type with no tag (a `List<T>`, a `Guid`)
   is an explicit `unsupported type: …` failure rather than a conversion — a behaviour change from in process.
+- **Attribution happens before release.** The worker's exit path runs attribution (strike, quarantine, suspect
+  marks) synchronously **before** it completes any evaluation the worker held, so a script killed for its deadline is
+  already quarantined when its single-flight slot is released — no trigger can re-run it in the restart gap. A worker
+  that dies while a script is **compiling** makes that script a suspect too; it then compiles alone in the isolation
+  worker, where a second death quarantines it (`"its compile crashed the isolation worker"`), so a poison source cannot
+  crash-loop its shard.
+- **Recovery.** A worker that dies restarts by itself after its backoff (250 ms doubling to 30 s, ±20 %), not on the
+  next trigger. An `Unavailable` outcome carries a recovery task (`ScriptRunOutcome.Recovery` →
+  `VirtualTagEvalResult.RetryWhen`); the VirtualTag actor and the Calculation driver re-evaluate once, against their
+  **current** inputs, when it completes — so a victim with a static input does not stay `BadResourceUnavailable`. The
+  isolation worker stops after `Scripting:Worker:IsolationIdleTimeout` (10 min) with nothing to run and starts again for
+  the next suspect.
+- **Crash-loop circuit.** When every shard has been down for `Scripting:Worker:CircuitOpenAfter` (2 min) the runner is
+  `Unavailable`: one Error is logged, evaluations answer `script runner unavailable` at once instead of each waiting out
+  `UnavailableGrace`, and the shards keep retrying at the backoff cap.
+- **Compiled-PE cache** (`Scripting:Worker:PeCacheDir`, optional; absolute and outside the content root, or startup
+  fails). Workers write each compiled image to `<dir>/pe/<id>.pe`, authenticated with HMAC-SHA256 under a per-node
+  random key (`<dir>/pe-cache.key`, created by the host; 0600 / directory 0700 on Linux and macOS, the directory ACL on
+  Windows), and a restarted worker loads the image instead of re-entering Roslyn. The id is SHA-256 over the source, the
+  context kind and a compile fingerprint (Roslyn version, the `Core.Scripting` build — analyzer, wrapper, stack guard —,
+  the sandbox reference set, the runtime), so a compiler or analyzer change misses every old entry. An entry that fails
+  verification (edited, truncated, another key, copied under another id) is deleted and the source recompiled; the
+  worker never loads bytes that did not verify. The key reaches the worker in `Hello`, never on disk or in its
+  environment. Entries unused for 30 days, and the least recently used beyond 50 000, are pruned at startup; the
+  directory can be deleted at any time. Measured on macOS (40 scripts, restart to every first result): 820 ms with no
+  cache, 840 ms with an empty one, **85 ms warm**; on windev (Windows) 9.5 s / 9.4 s / **1.0 s**. An unusable directory is a Warning and a cache-less runner.
+- **Orphan sweep.** Every launched worker gets a marker (`<PeCacheDir>/workers/<pid>.worker`, or a per-install
+  directory under temp) naming its pid, start time and image, and the host's pid and start time. At startup the runner
+  kills only a process whose marker's host is gone **and** whose pid, start time and image all still match — never
+  "every process with this name", never a live host's worker (mxaccessgw's `OrphanWorkerTerminator` lesson).
+- **Observability.** On `OtOpcUaTelemetry.Meter`: `otopcua.script.runner.state` (0 ready / 1 degraded / 2 circuit
+  open), `otopcua.script.worker.up` / `.inflight` / `.queue_depth` / `.heap_bytes` (tag `shard`; queue and heap from
+  the worker heartbeat), `otopcua.script.worker.starts` / `.exits` (`reason` adds `idle`), `otopcua.script.ipc.roundtrip`
+  and `otopcua.script.eval.duration` (µs, tag `kind`; no per-key tag), `otopcua.script.compile` (`cache` = `hit` |
+  `miss` | `rejected` | `uncached` | `negative`) and `otopcua.script.isolation.suspects`. The `script-runner` health check
+  is on the **ready** tier and is at worst **Degraded** (still 200): a sick runner never fails readiness, so a pair
+  does not move ServiceLevel to its peer over a script problem (design §14 #12).
 - **Deployment.** The Host build copies the worker into `script-worker/` beside `OtOpcUa.Host.dll`
   (`ScriptWorker.Deploy.targets`, Build and Publish); `Scripting:Worker:ExecutablePath` overrides it. It is run through
-  the dotnet muxer, so it needs a framework-dependent host (the docker-dev image). Not yet done (lmxopcua#780 P1
-  remainder): the PE cache (`Scripting:Worker:PeCacheDir` is validated — absolute, outside the content root — but not
-  used, so a worker restart recompiles), a self-contained worker for the Windows installs, the low-privilege worker
-  account and firewall rule, the runner-state gauge / readiness component, and the scripted-alarm seam.
+  the dotnet muxer, so it needs a framework-dependent host (the docker-dev image). Not yet done (lmxopcua#780): a
+  self-contained worker for the Windows installs, the low-privilege worker account, pipe ACL and firewall rule
+  (installer), the docker-dev live drill, the scripted-alarm seam, the per-seam batch submit (design §4.5) and the §8
+  perf harness.
 
 Pinned by `tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests` (`Category=ScriptWorker`: real worker processes).
 

docs/plans/2026-11-09-out-of-process-script-runner-design.md

@@ -677,6 +677,15 @@ per-shard state, restarts in the last hour and the quarantine list. Quarantines
      publish + the low-privilege account / pipe ACL / firewall rule (installer), the runner-state gauge and readiness
      component, victim re-evaluation on shard return, isolation-worker idle stop, the orphan sweep by nonce file, the
      scripted-alarm seam, the docker-dev enablement and the §8 perf harness.
+   - **Second P1 slice (lmxopcua#780):** the HMAC'd PE cache (§7; keyed by source + kind + a compile fingerprint, tampered
+     or mismatched entries evicted, pruned by recency at startup instead of by generation — this runner has no
+     generation object), attribution before release (the quarantine race), loading keys counted in attribution
+     (culprit-finding for a compile that crashes its worker, by isolation rather than the §5.3.5 three-strikes
+     bisection — loads are per key here), proactive restart + `Recovery` / `RetryWhen` re-evaluation of victims in the
+     VT and Calc seams, isolation idle stop (`IsolationIdleTimeout`), the crash-loop circuit (`CircuitOpenAfter`), the
+     §10 gauges and histograms, the degraded-but-ready `script-runner` health check (§14 #12), and the orphan sweep by
+     launch marker (pid + start time + image, host gone). Still open: self-contained worker publish, installer (account,
+     pipe ACL, firewall), docker-dev live drill, scripted-alarm seam, the §4.5 batch submit, the §8 perf harness.
 3. **P2, shadow mode on docker-dev and QA.**
    - `Shadow` runs the **in-process** evaluation as the published result, and *also* submits to the worker.
    - It compares value, status class and failure kind per evaluation, and counts