perf(scripting): cut out-of-process runner host CPU; compile timeout from compile start; quiet start-failure logs (lmxopcua#789, #790, #788)

#789 (partial, further tuning deferred): combining inline frame writer, buffered
frame reader, one timer per evaluation for T and T+G, slot release on the reader
before the caller resumes, a non-spinning per-connection result dispatcher, and
blocking pipe I/O on Unix (both ends) so the socket never enters the socket
engine; TerminateJobObject kill on Windows. macOS host CPU at 10k evals/s K=2:
228-240 % -> 79 % (InProcess 69 %, IPC share +10 vs budget 25); windev 153 % ->
123 % (+115, still a miss). Harness reports the "vs InProcess" share, warm-cache
worker RSS, and times kill latency from the soft-timeout answer.

#790: the worker sends LoadStarted; the compile timeout is armed then, so a load
queued behind other compiles is never killed or struck.

#788: one Error with stack per down period, rate-limited Warnings with attempt
count, one Information on recovery.

Design doc §8.2 notes + §12.2 record.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV

docs/plans/2026-11-09-out-of-process-script-runner-design.md

@@ -533,10 +533,10 @@ size for 2× headroom (10k/s). The passthrough fast path is excluded because it
 | Added latency per evaluation, p50 | ≤ 150 µs | Host enqueue → reply decoded, trivial script, idle system |
 | Added latency per evaluation, p99 | ≤ 2 ms | At 10k evals/s. Compare the 250 ms alarm budget: under 1% of it |
 | Throughput | ≥ 10,000 trivial evals/s/node with K=2 | With opportunistic batching |
-| Host CPU for IPC | ≤ 25% of one core at 10k evals/s | Serialisation + pipe I/O, measured by an opt-in perf test |
+| Host CPU for IPC | ≤ 25% of one core at 10k evals/s | Serialisation + pipe I/O, measured by an opt-in perf test. Since lmxopcua#789 the harness reports it as the out-of-process runner's host CPU minus the in-process runner's in the same phase ("vs InProcess"): the raw figure includes the harness's own pacing loop and pool, which InProcess pays too (§12.2) |
 | Bytes per eval on the wire | ≤ 64 B + 24 B per input | Positional slots, §4.4 |
 | Shard restart → first result, 1,000 cached scripts | ≤ 2 s | §7 lazy PE load |
-| Worker idle RSS | ≤ 80 MB + scripts **[A-3]** | Per shard |
+| Worker idle RSS | ≤ 80 MB + scripts **[A-3]** | Per shard. **Proposed revision (lmxopcua#789, §12.2):** ≤ 80 MB + scripts for a worker that has only loaded cached PEs (measured 36–58 MB); ≤ 200 MB + scripts once Roslyn has run in it (measured 159–187 MB), because compiling in the worker (§14 #7) keeps Roslyn's code and metadata resident and no GC setting moves it |
 | Kill latency after `T+G` | ≤ 100 ms | Host timer resolution plus `Process.Kill` |
 
 **Comparison with today:** the in-process cost per compiled run is one `Task.Run` hop plus `WaitAsync`, which
@@ -873,6 +873,96 @@ Both misses → lmxopcua#789 (the §4.5 batching and per-eval allocation work).
 
 The host load timeout still counts the time a load waits behind other compiles → lmxopcua#790.
 
+### 12.2 lmxopcua#789 / #790 / #788 record (2026-11-10)
+
+**What cost the host CPU (#789).** With `DOTNET_ThreadPool_UnfairSemaphoreSpinLimit=0` a scratch probe showed the
+out-of-process runner at 33 % of a core at 10k evals/s against 25 % in process: the IPC work itself was about
+10 µs per evaluation. The rest was thread-pool spin-waiting, triggered by several pool work items per evaluation, and
+on Unix by the socket engine. Changes:
+- **Combining frame writer** (`ScriptWorkerFrameWriter`): the thread that queues a frame writes it itself when no
+  write is in progress. This replaces the channel + writer task (one pool wake-up per submit). Consecutive
+  evaluations or results are merged into one batch, frame order is preserved, and there is one write per pass. The
+  old loop also called `PipeStream.FlushAsync`, which PipeStream does not override, so it queued a pool work item
+  per pass.
+- **Buffered frame reader** (`ScriptWorkerFrameReader`): one read pulls in every frame available, and frames are
+  parsed from the buffer. It replaces two reads per frame plus a header allocation.
+- **One timer per evaluation** for T and T+G. It replaces the `WaitAsync(timeout)` timer, the hard-deadline
+  `CancellationTokenSource` and their two continuations.
+- **Slot release and deadline cancel run on the reader, before the caller is resumed.** The release no longer
+  depends on a continuation that is not ordered against the caller.
+- **Result dispatcher** (`ResultDispatcher`, one thread per connection, blocking with `Monitor.Wait`, no spin):
+  callers resume there, not on the pool. They do not resume on the reader either, because the Calculation pump
+  publishes to OPC UA subscribers after its await. On the reader, a slow consumer would hold up heartbeats and get a
+  healthy worker killed. That is pinned by a test, and a mutation that resumed callers on the reader was red.
+- **Blocking I/O on Unix** (`ScriptWorkerFrameCodec.UseBlockingIo`):
+  - The socket under a Unix .NET pipe turns non-blocking for good after one async call. After that even a
+    "blocking" read goes through the epoll thread plus a spinning wait.
+  - Both ends now connect, handshake, read (on a dedicated thread) and write synchronously. This was the largest
+    single gain: sequential host CPU per evaluation fell from about 85 µs to about 13 µs.
+  - A full pipe blocks only the one thread draining the writer; every other producer just queues. A peer that stops
+    reading is killed by the heartbeat check within 3 s, which fails that write.
+  - Windows keeps overlapped async I/O, because a synchronous full-duplex handle serialises reads against writes.
+- **Kill via `TerminateJobObject`** on Windows. The job admits one process, so no system-wide snapshot is needed.
+- Measured and **not** kept for memory, because none moved the compiled-worker RSS:
+  - ReadyToRun worker publish: 195 MB;
+  - `GCgen0size` 4 MB, `GCConserveMemory`, `TieredPGO=0`;
+  - an aggressive compacting GC after a compile burst.
+  The memory is Roslyn's JIT'd code and metadata, not the heap.
+
+**Before / after (§8 harness, `OTOPCUA_SCRIPT_PERF=1`; macOS figures under 5–25 load average from other agents on
+the box).**
+
+| Budget | macOS before | macOS after | windev before | windev after |
+|---|---:|---:|---:|---:|
+| Host CPU at 10k/s, K=2, raw (InProcess same phase) | 228–240 % (71 %) | 79 % (69 %) | 153 % (6 %) | 123 % (7 %) |
+| … vs InProcess (budget ≤ 25 %) | +157 | **+10 ✓** | +147 | +115 ✗ |
+| Host CPU, idle sequential, K=2 | 258 % | 45 % | 80 % | 63 % |
+| Idle p50 (≤ 150 µs) | 41–43 µs | 23 µs ✓ | 795 µs | 507 µs ✗ |
+| p99 at 10k/s, K=2 (≤ 2 ms) | 720–796 µs | 1,024 µs ✓ | 1,916 µs | 1,676 µs ✓ |
+| Closed loop ×64, K=2 (gate ≥ 5k/s) | 342k/s | 442k/s ✓ | 51k/s | 62k/s ✓ |
+| Worker RSS after a compile (≤ 80 MB) | 188 MB | 187 MB ✗ | 158 MB | 159–161 MB ✗ |
+| Worker RSS, warm PE cache, no compile | not measured | 58 MB ✓ | not measured | 36 MB ✓ |
+| Warm restart, 1,000 cached (≤ 2 s) | 245 ms | 206–223 ms ✓ | 1,715 ms | 1,778–1,921 ms ✓ |
+| Kill both shards → every next result (≤ 2 s) | 512 ms | 479–502 ms ✓ | 2,028 ms | 2,004–2,087 ms (borderline) |
+| Kill latency after T+G (≤ 100 ms) | 31–35 ms | 33–43 ms ✓ | ~210 ms | **10–21 ms ✓** |
+
+Additional measurements:
+- In a scratch probe with a steady 10k/s load, K=2, before vs after: 116–158 % → 49–52 % of a core (InProcess
+  60–65 %). Sequential runs: about 25k → 46k evals/s.
+- The first harness timed the kill from before the runaway's call, so it also counted the cold compile of each
+  distinct runaway key (100–200 ms on the VM). The harness now times from the soft-timeout answer. The windev
+  "before" figure of about 210 ms is therefore mostly compile time, not kill time.
+- macOS K=1 p99 at 10k/s read 2.3 ms in one loaded run (budget 2 ms). K=2 is the configuration the budget names.
+
+**Still missing after this slice (lmxopcua#789 stays open, further tuning deferred by the owner):**
+- Windows host CPU for IPC: +115 points at 10k/s on the VM. That path still uses overlapped async I/O, so it pays a
+  completion-port and pool hand-off per frame. The next step is either two simplex pipes with synchronous handles and
+  dedicated threads, or a measurement on a non-VM box.
+- Windows idle p50: 507 µs. The VM's in-process baseline is already 92 µs.
+- Worker RSS after a compile. The proposed revised budget is in §8.2. An alternative is to recycle a worker from the
+  PE cache after a compile burst (blue/green); that is not built.
+- Windows kill-both-shards: about 2.0 s, at the budget line.
+
+**#790 — load timeout from compile start.**
+- The worker sends `LoadStarted` when a compile thread picks a load up. The host arms that load's timer then,
+  instead of each caller timing from when it sent the load.
+- A queued load is never blamed. A compile still running `LoadTimeout` after its start kills the worker and strikes
+  that key only (`LoadEntry.TimedOut`).
+- `Hello.compile_threads` is a test knob; 0 keeps the worker's default.
+- Tests:
+  - `Compiles_queued_behind_others_longer_than_the_load_timeout_are_not_killed_or_struck`: one compile thread, and
+    a burst grown until it outlasts three timeouts (asserted). Red with the timer armed at send.
+  - `A_compile_still_running_a_load_timeout_after_it_started_is_killed_and_struck`: the positive control, with two
+    kills leading to quarantine and the healthy key unblamed. Red with the timer never armed.
+
+**#788 — start-failure logging.**
+- The first failed start of a down period is one Error with the exception.
+- Later retries are a Warning without a stack, carrying the attempt count, at most one per 60 s per shard; the rest
+  log at Debug.
+- The start that ends the outage is one Information line: `started … after N failed start attempt(s) over T`.
+- Tests: `A_shard_that_cannot_start_logs_one_error_then_concise_rate_limited_retries_then_its_recovery`, with two
+  intervals. Red when every failure is logged at Error with the exception.
+
 ---
 
 ## 13. Test plan