feat(scripting): batch submit, exact per-item deadlines, scripted-alarm seam on the out-of-process runner (lmxopcua#780)
v2-ci / unit-tests (push) Waiting to run

- IScriptRunner.RunBatch: one EvalBatch frame per worker for every evaluation ready to run; per-item, streamed results.
  The default (InProcessScriptRunner) is RunAsync per item. Used by the Calculation pump (dequeue all, split at a
  repeated tag, publish in dequeue order), the VirtualTag seam (ScriptBatchCoalescer, no timer) and the scripted-alarm
  engine (whole dirty set). In process the pump and the engine stay one at a time, unchanged.
- Exact per-item deadlines: the worker hands work to an idle evaluation thread or defers it (EvalResults
  deferred_eval_ids); the host stops that run's clock and restarts T then G when it starts (started_eval_ids). A run
  queued behind a runaway is neither timed out nor struck and is a victim, not the culprit, at the kill.
- Scripted-alarm seam behind Scripting:Runner:Seams; unset Seams now means VirtualTag + Calculation, so alarms run out of
  process only when listed. Load still compiles each predicate (to PE bytes only) so a broken one fails the load; passes
  batch and apply in the serial order under the gate. Script faults freeze the condition BadInternalError; runner faults
  freeze it BadResourceUnavailable and re-evaluate on recovery; a dead worker never clears an alarm. IScriptRunnerProvider
  carries the runner to DriverHostActor.
- Perf harness: batch throughput (RunBatch vs RunAsync x n) and alarm-pass latency (in process / batched / serial);
  design §12.3 records macOS + windev. On windev the serial out-of-process alarm pass misses 250 ms from ~500 alarms;
  batched p99 <= 53 ms at 1,000.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-10 06:37:36 -05:00
parent 11a5c1d4e2
commit 0796928504
33 changed files with 2168 additions and 117 deletions
+4 -2
View File
@@ -212,10 +212,12 @@ scripted-alarm predicates) runs scripts through the shared `ScriptRunGuard`:
run returns, so a run that OOMs the process first is not stopped — that needs an
out-of-process runner (→ #750).
- **Out of process (opt-in, lmxopcua#780).** `Scripting:Runner=OutOfProcess` moves the
virtual-tag and Calculation seams' compiled runs into `ScriptWorker` processes, where a
virtual-tag and Calculation seams' compiled runs — and the scripted-alarm predicates,
when `Scripting:Runner:Seams` lists `ScriptedAlarm` — into `ScriptWorker` processes, where a
run past `T + Scripting:Worker:HardGrace` is stopped by killing its worker and an OOM
kills a worker instead of the server. A run whose worker died under it reads
`BadResourceUnavailable`. See `docs/VirtualTags.md` §"Out-of-process runner".
`BadResourceUnavailable` (an alarm is frozen with that annotation, never cleared). See
`docs/VirtualTags.md` §"Out-of-process runner".
- **Recursion.** Unbounded recursion fails the script with
`ScriptStackExhaustedException` (Bad, script log) instead of overflowing the
stack, which would terminate the whole server. The runtime compile inserts a
+2
View File
@@ -49,6 +49,8 @@ new ScriptedAlarmDefinition(
Alarm predicates reuse the same Roslyn sandbox as virtual tags — `ScriptEvaluator<AlarmPredicateContext, bool>` compiles the source, `ScriptRunGuard` runs it with the node's `Scripting:RunTimeout` budget (default 2 s — the same budget virtual tags get; before lmxopcua#779 the host built the engine without a budget and every predicate silently ran on `TimedScriptEvaluator.DefaultTimeout`, 250 ms), and `DependencyExtractor` statically harvests the tag paths the script reads. The sandbox rules (forbidden types, cancellation, logging sinks) are documented in [VirtualTags.md](VirtualTags.md); ScriptedAlarms does not redefine them. The known resource limits — unbounded script-side memory and the orphan-thread CPU-budget caveat — are documented in that file as well; per-publish assembly accretion was resolved by the Core.Scripting-008 collectible-`AssemblyLoadContext` rewrite and no longer requires periodic server restarts.
**Out of process (opt-in, lmxopcua#780).** With `Scripting:Runner=OutOfProcess` **and** `ScriptedAlarm` listed in `Scripting:Runner:Seams` (an unset list leaves alarms in process — they go last in the rollout), the predicates run in the pre-emptible `ScriptWorker` pool. `LoadAsync` still compiles every predicate, to PE bytes only, so a broken predicate fails the load exactly as before; the worker compiles and runs it. Each re-evaluation pass (and the startup-recovery pass) submits every dirty alarm's predicate in **one batch** and applies the results — Part 9 transition, persist-before-update, emissions — in the serial order under the engine's gate, so the transition sequence is the in-process one and a large engine pays one round trip per pass, not one per alarm. Failure semantics (design §9): a predicate that times out, is killed at its hard deadline or is quarantined freezes its condition with a `BadInternalError` quality annotation, as in process; one whose worker restarted under it, or that found no worker ready, freezes with **`BadResourceUnavailable`** (`0x80040000`) and is re-evaluated by itself when the worker is back. No outcome but a completed `bool` reaches the state machine, so a dead worker never clears an alarm. Quarantines (`ScriptedAlarmEngine.QuarantinedAlarmIds`) lift on a reload. See [VirtualTags.md](VirtualTags.md) §"Out-of-process runner".
`AlarmPredicateContext` (`AlarmPredicateContext.cs`) is the script's `ScriptContext` subclass:
- `GetTag(path)` returns a `DataValueSnapshot` from the engine-maintained read cache. Missing path → `DataValueSnapshot(null, 0x80340000u, null, now)` (`BadNodeIdUnknown`). An empty path returns the same.
+27 -6
View File
@@ -46,10 +46,11 @@ The sandbox cannot pre-empt a script. **Memory:** each run's own-thread allocati
`Scripting:Runner` selects where compiled scripts run: **`InProcess`** (the default and the rollback — everything
above, unchanged), **`OutOfProcess`**, or `Shadow` (design P2; not implemented yet, so it runs `InProcess` with a
startup Warning). `Scripting:Runner:Seams` narrows `OutOfProcess` to a subset (`VirtualTag`, `Calculation`; the
`ScriptedAlarm` seam stays in process for now — alarms go last, design §14 #10). Under `OutOfProcess` the virtual-tag
and Calculation evaluators hand each compiled run to `OutOfProcessScriptRunner`
(`Core.Scripting.OutOfProcess`), which runs it in a pool of `ZB.MOM.WW.OtOpcUa.ScriptWorker` processes:
startup Warning). `Scripting:Runner:Seams` selects the seams (`VirtualTag`, `Calculation`, `ScriptedAlarm`). Unset or
empty means `VirtualTag` + `Calculation`: the **`ScriptedAlarm` seam runs out of process only when it is listed
explicitly** (alarms go last, design §14 #10). Under `OutOfProcess` the selected seams hand their compiled runs to
`OutOfProcessScriptRunner` (`Core.Scripting.OutOfProcess`), which runs them in a pool of `ZB.MOM.WW.OtOpcUa.ScriptWorker`
processes:
- **K shards** (`Scripting:Worker:Shards`, default 2) placed by a stable FNV-1a hash of the key, plus an on-demand
**isolation worker** that runs suspects one at a time. No worker starts until the first compiled evaluation, so a
@@ -122,13 +123,33 @@ and Calculation evaluators hand each compiled run to `OutOfProcessScriptRunner`
directly: `OutOfProcessScriptRunner.ResolveWorkerEntry` picks the apphost only when the worker's `runtimeconfig.json` lists
`includedFrameworks`. Gated on windev against a real `-r win-x64` publish (`OTOPCUA_SCRIPTWORKER_TEST_PATH` points the
worker suite at the published apphost). Not yet done (lmxopcua#780): the low-privilege worker account, pipe ACL, firewall
rule and PeCacheDir ACL (installer), the scripted-alarm seam and the per-seam batch submit (design §4.5).
rule and PeCacheDir ACL (installer).
- **docker-dev.** `docker-dev/docker-compose.script-runner.yml` turns `OutOfProcess` on for the central pair's
VirtualTag and Calculation seams. The 2026-11-10 live drill (runaway kill and quarantine, `kill -9` recovery, open circuit
with readiness still 200) is recorded in the design doc's rollout section.
- **Perf harness.** `ScriptRunnerPerfTests` (`Category=Perf`, skipped unless `OTOPCUA_SCRIPT_PERF=1`) measures InProcess
against OutOfProcess at K = 1/2/4 against the accepted 5,000 evals/s/node gate and the design §8.2 budgets. The numbers
are in the design doc's rollout section (§12.1). The gate passes; the host-CPU and RSS budgets miss (lmxopcua#789).
are in the design doc's rollout section (§12.1, §12.2). The gate passes; the host-CPU and RSS budgets miss (lmxopcua#789).
It also measures batch submit (RunBatch against the same evaluations as concurrent RunAsync calls) and the
scripted-alarm pass latency (push → last transition, in process / out of process batched / out of process serial)
against the 250 ms alarm budget — design §12.3.
- **Batch submit (design §4.5).** `IScriptRunner.RunBatch` submits several evaluations at once: every item that is ready
to run (its key loaded in a live worker, not a suspect) goes out in ONE `EvalBatch` frame per worker, and results are
per item and streamed. The Calculation pump dequeues everything pending and submits it as one batch, publishing in
dequeue order (a tag queued twice ends the batch, so every queued evaluation still runs against the snapshot that
triggered it); the virtual-tag seam coalesces evaluations that become ready together (one actor per tag) into one
batch, with no timer; the scripted-alarm engine submits a pass's whole dirty set. In process the pump and the engine
stay one-at-a-time, unchanged. **Per-item deadlines and blame stay exact:** an evaluation that arrives while every
evaluation thread is busy is reported back as *deferred*, the host stops its clock, and restarts T and G when the worker
reports it *started* — so an item queued behind a runaway sibling is neither timed out nor struck, and at the kill it
is a victim (`script worker restarted`), not the culprit.
- **Scripted alarms** (only when `Scripting:Runner:Seams` lists `ScriptedAlarm`). The engine compiles each predicate at
load only to validate it (same Roslyn, sandbox and diagnostics, so a broken predicate still fails the load; no
assembly is loaded into the host), the worker compiles and runs it, and each re-evaluation pass submits its dirty set
as one batch and applies the results in the serial order under the engine's gate — the Part 9 sequence is the
in-process one. A predicate that did not complete freezes its condition (a dead worker never clears an alarm): timed
out, killed or quarantined → `BadInternalError` annotation, as in process; the runner's fault (worker restarted, none
ready) → `BadResourceUnavailable` annotation, and the alarm is re-evaluated by itself when the worker is back.
- **Compile threads.** A worker compiles on `min(8, max(1, CPUs/2))` dedicated threads, never on the thread pool, so a
burst of cold loads cannot starve its heartbeat or its pipe writer. The host load timeout still counts the time a load
spends in that queue (lmxopcua#790).
@@ -640,7 +640,7 @@ per-shard state, restarts in the last hour and the quarantine list. Quarantines
| Key | Default | Meaning |
|---|---|---|
| `Scripting:Runner` | `InProcess` | `InProcess` \| `OutOfProcess` \| `Shadow` |
| `Scripting:Runner:Seams` | all three | Optional per-seam subset for `OutOfProcess` / `Shadow` (e.g. `["Calculation"]`) |
| `Scripting:Runner:Seams` | VirtualTag + Calculation | Optional per-seam subset for `OutOfProcess` / `Shadow` (e.g. `["Calculation"]`). Unset selects VirtualTag and Calculation; `ScriptedAlarm` only when listed (§14 #10, as built in §12.3) |
| `Scripting:Worker:Shards` | `2` | 1–8 |
| `Scripting:Worker:HardGrace` | `max(1s, T)` | §5.1 |
| `Scripting:Worker:HeapLimitBytes` | `536870912` | §5.2 |
@@ -660,6 +660,9 @@ per-shard state, restarts in the last hour and the quarantine list. Quarantines
`VirtualTagActor`), so the dispatcher-thread blocking is gone. The Calc pump and the alarm engine still run one
script at a time; why that was not relaxed in-process is recorded in `docs/VirtualTags.md` §"Upstream-tag feed".
`IScriptRunner` and batch-submit remain to do.
- **Batch submit landed in lmxopcua#780 (fourth P1 slice, §12.3):** `IScriptRunner.RunBatch`, used by the Calc pump, the
VirtualTag seam (a coalescer) and the scripted-alarm engine on the out-of-process path. In process the pump and the
engine stay one at a time, unchanged.
2. **P1, worker + protocol behind the flag.**
- Ship `ScriptWorker`, `OutOfProcessScriptRunner` and the PE cache.
- Default stays `InProcess`. Enable on the docker-dev rig only.
@@ -686,6 +689,11 @@ per-shard state, restarts in the last hour and the quarantine list. Quarantines
§10 gauges and histograms, the degraded-but-ready `script-runner` health check (§14 #12), and the orphan sweep by
launch marker (pid + start time + image, host gone). Still open: self-contained worker publish, installer (account,
pipe ACL, firewall), docker-dev live drill, scripted-alarm seam, the §4.5 batch submit, the §8 perf harness.
- **Fourth P1 slice (lmxopcua#780, record in §12.3):** the §4.5 batch submit (`IScriptRunner.RunBatch`, one `EvalBatch`
frame per worker; Calc pump, VT coalescer, alarm dirty set), exact per-item deadlines for work queued in the worker
(deferral / start notices), and the scripted-alarm seam behind `Scripting:Runner:Seams` — alarms run out of process
only when listed. Still open: installer (account, pipe ACL, firewall, PeCacheDir ACL), #789's remaining misses, then
P2 Shadow.
- **Third P1 slice (lmxopcua#780, record in §12.1):** a self-contained worker publish for self-contained (Windows)
Host publishes, started through its apphost; the docker-dev live drill (`docker-dev/docker-compose.script-runner.yml`);
the §8 perf harness (`Category=Perf`, opt-in), which passes the accepted 5k evals/s/node gate; and two runner fixes
@@ -963,6 +971,133 @@ Additional measurements:
- Tests: `A_shard_that_cannot_start_logs_one_error_then_concise_rate_limited_retries_then_its_recovery`, with two
intervals. Red when every failure is logged at Error with the exception.
### 12.3 Fourth P1 slice: batch submit, exact per-item deadlines, the scripted-alarm seam (2026-11-10)
**Batch submit (§4.5, P0).**
- `IScriptRunner.RunBatch(invocations)` returns one task per item. Results are per item and streamed.
- The default implementation is `RunAsync` per item, started together. `InProcessScriptRunner` uses it.
- `OutOfProcessScriptRunner` sends every item that is ready to run as ONE `EvalBatch` frame per worker. Ready means
admitted, not quarantined, not a suspect, and already loaded in a live worker at this source and slot order.
- Anything else goes through the full `RunAsync` path on its own: a first evaluation that needs a compile, a worker
that is down, or a suspect bound for the one-at-a-time isolation worker. A batch therefore never waits for a load
or a restart as a whole.
- Seams that use it, on the out-of-process path only:
- **Calculation pump.** It dequeues everything pending (each entry's dependency snapshot was taken at enqueue),
submits one batch, and publishes in dequeue order. A tag queued twice ends the batch, so every queued evaluation
still runs, in order, against the snapshot that triggered it. The published sequence is the one-at-a-time pump's.
- **VirtualTag seam.** There is one actor per tag, so an upstream burst makes many tags ready at once from different
threads. `ScriptBatchCoalescer` hands those to one `RunBatch`. It has no timer: an evaluation on an idle seam is
submitted at once by its own caller.
- **Scripted alarms.** See below.
- In process, the Calc pump and the alarm engine stay one at a time, unchanged. The VirtualTag in-process path does
not touch the coalescer.
**Per-item deadlines and blame stay exact under batching.**
- The host arms T and T + G at submit. That is when the run starts, provided the worker has an idle evaluation thread.
- A batch larger than the worker's scheduler, or work queued behind a runaway, used to wait in the worker with its
clock already running. It could then be timed out, take timeout strikes, and even be the key blamed at T + G.
- The worker's scheduler now hands an evaluation straight to an idle thread, or else puts it in a backlog and reports
it in `EvalResults.deferred_eval_ids`.
- The host stops that evaluation's clock.
- When a thread picks the evaluation up, its id rides back in `started_eval_ids` on the result frame of the run that
thread just finished. The host then restarts the full T, then G, from the start.
- The two notices are unordered on the wire, and the host handles either order.
- The deadlines are absolute timestamps that the timer re-checks, so a stale tick never fires early.
- A run queued behind a runaway is therefore neither timed out nor struck. At the kill it is a victim
(`script worker restarted`, `BadResourceUnavailable`, re-evaluated on return), not the culprit.
- Tests:
- `A_runaway_in_a_batch_is_blamed_alone_and_siblings_queued_behind_it_are_not_timed_out_or_struck` runs one evaluation
thread and a batch of a runaway plus three siblings. It is red with the host ignoring the deferral, and red with the
worker never deferring.
- `A_deferred_run_is_timed_and_killed_from_when_it_starts` is the positive control. It is red with a start notice that
does not re-arm the clock.
- This also changes plain `RunAsync` under saturation: a run that waits for a thread no longer times out on its wait.
That matches the in-process guard, whose budget clock starts when the pool picks the run up (§1.2).
**Scripted-alarm seam (★10: alarms last).**
- `Scripting:Runner:Seams` unset now means VirtualTag + Calculation. **`ScriptedAlarm` runs out of process only when it
is listed.**
- `ScriptRunnerSelection` serves it. The Runtime's `DriverHostActor` gets it through the new `IScriptRunnerProvider`
and passes it to the engine (a new last parameter, `scriptedAlarmRunner`).
- Engine changes, all on the runner path only:
- `LoadAsync` compiles each predicate to PE bytes only (`CompileToImage`). A broken predicate still fails the load
exactly as in process, and no assembly is loaded into the host. Deviation: Roslyn still runs in the host at load
for alarms; the worker compiles again for execution.
- Startup recovery and every re-evaluation pass snapshot each dirty alarm's inputs under `_evalGate`, submit all
their predicates as one batch, and then apply the results in the serial order under the gate: Part 9 transition,
persist before update, emissions. A test shows the transition sequence is identical to an in-process engine's
for the same pushes.
- Failure semantics (§9, alarm column):
- Timed out, killed, refused or quarantined: the condition is frozen with a `BadInternalError` annotation, as today.
- The runner's fault (worker restarted, none ready): frozen with **`BadResourceUnavailable`**, consistent with
VT/Calc. The alarm is re-evaluated by itself when the runner's `Recovery` completes.
- A predicate that threw keeps its state with no annotation, as in process.
- No outcome but a completed `bool` reaches the state machine, so a dead worker never clears an alarm.
- Real-worker tests:
- `A_runaway_predicate_is_killed_and_quarantined_its_alarm_frozen_Bad_and_a_sibling_keeps_evaluating`;
- `A_worker_that_dies_under_a_predicate_freezes_it_BadResourceUnavailable_never_clears_it_and_it_is_re_evaluated_on_return`.
It is red when the runner's failure is read as `false`.
- Budget note: since lmxopcua#779 the host passes `Scripting:RunTimeout` (2 s) to the engine. The passes below are
still judged against the stricter 250 ms the design was written for.
**Measured (§8 harness, `Batch_throughput_and_alarm_pass_latency`; macOS Arm64, 14 CPUs, load average about 14 from
other work on the box, 10 s phases):**
| Batch size | RunBatch evals/s | RunAsync ×n evals/s | RunBatch p50 / p99 µs | RunAsync ×n p50 / p99 µs |
|---:|---:|---:|---:|---:|
| 1 | 38,296 | 40,980 | 24 / 41 | 23 / 35 |
| 10 | 147,035 | 118,516 | 57 / 121 | 70 / 142 |
| 100 | 369,304 | 232,967 | 215 / 1,095 | 363 / 1,341 |
| 500 | 463,033 | 256,232 | 818 / 2,541 | 1,701 / 3,178 |
(OutOfProcess K = 2, one batch at a time. The in-process rows show no difference between the two, as expected.)
| Alarms per pass | InProcess p50 / p99 ms | OutOfProcess batched p50 / p99 ms | OutOfProcess serial p50 / p99 ms |
|---:|---:|---:|---:|
| 100 | 0.4 / 1.5 | 0.4 / 2.1 | 3.6 / 4.9 |
| 500 | 2.2 / 4.7 | 1.7 / 4.8 | 16.1 / 18.0 |
| 1,000 | 4.4 / 7.7 | 2.5 / 41.0 | 28.1 / 33.4 |
- A pass is push → the last of the N transitions it emits. "Serial" replays the pre-batch one-predicate-at-a-time pass
over the same runner. That is the N × round-trip cost §8.2 warned about: about 30 µs per alarm, against 2–4 µs
batched.
- Every configuration is far inside 250 ms. The one 41 ms p99 is a single loaded-box outlier.
- The §8.2 harness rerun on the same box after the scheduler change:
- K = 2 closed loop ×64: 366k evals/s;
- paced 10k/s: +5 % host CPU vs InProcess (budget ≤ 25 %);
- idle p50: 25 µs;
- kill latency: 14–19 ms;
- warm-cache restart of 1,000 scripts: 220 ms;
- every result correct.
**windev** (Windows VM, 36 CPUs, .NET 10.0.8; the full `ScriptWorker.Tests` suite also passes there, 157/157):
| Batch size | RunBatch evals/s | RunAsync ×n evals/s | RunBatch host CPU | RunAsync ×n host CPU |
|---:|---:|---:|---:|---:|
| 1 | 1,810 | 1,921 | 63 % | 65 % |
| 10 | 13,013 | 11,220 | 109 % | 157 % |
| 100 | 48,685 | 40,699 | 129 % | 224 % |
| 500 | 86,322 | 83,116 | 143 % | 289 % |
| Alarms per pass | InProcess p50 / p99 ms | OutOfProcess batched p50 / p99 ms | OutOfProcess serial p50 / p99 ms |
|---:|---:|---:|---:|
| 100 | 8.2 / 39.0 | 2.9 / 5.1 | 51.7 / 64.8 |
| 500 | 42.3 / 53.3 | 7.4 / 35.7 | **236.5 / 273.3 (MISS)** |
| 1,000 | 36.6 / 84.2 | 11.7 / 52.7 | **466.8 / 519.4 (MISS)** |
- On the VM, where one round trip costs about 0.5 ms, the pre-batch serial pass **misses the 250 ms budget from about
500 alarms**. The batched pass stays at p99 ≤ 53 ms for 1,000 alarms, below the in-process engine's own figures.
That is the measurement behind §8's "batch submit is a prerequisite, not an optimisation", and the reason ★10 put
alarms last.
- On Windows, batching halves the host CPU of a 100–500 evaluation burst. That is the overlapped-I/O path that is still
#789's main miss.
- No predicate failed to complete in any pass on either host.
- Rerun of the §8.2 harness on windev:
- K = 2 closed loop ×64: 50k evals/s, gate PASS;
- paced 10k/s: +99 % host CPU vs InProcess (#789, unchanged in kind);
- kill latency: 9–19 ms;
- warm-cache restart of 1,000 scripts: 1.8 s.
---
## 13. Test plan