feat(scripting): batch submit, exact per-item deadlines, scripted-alarm seam on the out-of-process runner (lmxopcua#780)
v2-ci / unit-tests (push) Waiting to run

- IScriptRunner.RunBatch: one EvalBatch frame per worker for every evaluation ready to run; per-item, streamed results.
  The default (InProcessScriptRunner) is RunAsync per item. Used by the Calculation pump (dequeue all, split at a
  repeated tag, publish in dequeue order), the VirtualTag seam (ScriptBatchCoalescer, no timer) and the scripted-alarm
  engine (whole dirty set). In process the pump and the engine stay one at a time, unchanged.
- Exact per-item deadlines: the worker hands work to an idle evaluation thread or defers it (EvalResults
  deferred_eval_ids); the host stops that run's clock and restarts T then G when it starts (started_eval_ids). A run
  queued behind a runaway is neither timed out nor struck and is a victim, not the culprit, at the kill.
- Scripted-alarm seam behind Scripting:Runner:Seams; unset Seams now means VirtualTag + Calculation, so alarms run out of
  process only when listed. Load still compiles each predicate (to PE bytes only) so a broken one fails the load; passes
  batch and apply in the serial order under the gate. Script faults freeze the condition BadInternalError; runner faults
  freeze it BadResourceUnavailable and re-evaluate on recovery; a dead worker never clears an alarm. IScriptRunnerProvider
  carries the runner to DriverHostActor.
- Perf harness: batch throughput (RunBatch vs RunAsync x n) and alarm-pass latency (in process / batched / serial);
  design §12.3 records macOS + windev. On windev the serial out-of-process alarm pass misses 250 ms from ~500 alarms;
  batched p99 <= 53 ms at 1,000.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-10 06:37:36 -05:00
parent 11a5c1d4e2
commit 0796928504
33 changed files with 2168 additions and 117 deletions
+4 -2
View File
@@ -390,10 +390,12 @@ endpoints under `/api/script-analysis/*`, gated by the `ConfigEditor` policy
See `docs/ScriptEditor.md` for the full guide. Scripts may use the `{{equip}}` token for equipment-relative tag paths that resolve per-equipment at deploy time — see the "Equipment-relative tag paths" section in `docs/ScriptEditor.md`.
**Out-of-process script runner (lmxopcua#780, opt-in).** `Scripting:Runner` = `InProcess` (default, unchanged) | `OutOfProcess` | `Shadow` (not yet).
`OutOfProcess` runs the VirtualTag + Calculation seams' compiled scripts in `ZB.MOM.WW.OtOpcUa.ScriptWorker` processes
`OutOfProcess` runs the VirtualTag + Calculation seams' compiled scripts (and the scripted-alarm predicates when `Scripting:Runner:Seams`
lists `ScriptedAlarm` — never by default) in `ZB.MOM.WW.OtOpcUa.ScriptWorker` processes
(`src/Core/ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess`, protocol in `Core.Scripting.Protocol`), shipped in `script-worker/`
beside the host (self-contained + started via its apphost on a `-r win-x64 --self-contained` publish, through the muxer otherwise); a runner-side failure publishes `BadResourceUnavailable` (ScadaBridge-visible, ScadaBridge#112) and the seam re-evaluates
when the worker is back. Optional `Scripting:Worker:PeCacheDir` = HMAC'd compiled-PE cache (restart without Roslyn); the `script-runner`
when the worker is back. `IScriptRunner.RunBatch` sends ready evaluations as one frame per worker (Calc pump, VT coalescer,
alarm dirty set); a run queued behind busy worker threads has its deadlines restarted when it starts, so it is never blamed for a sibling. Optional `Scripting:Worker:PeCacheDir` = HMAC'd compiled-PE cache (restart without Roslyn); the `script-runner`
readiness check is at worst Degraded (never fails ready). The worker tests
are `tests/Server/ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests` (`Category=ScriptWorker`, real processes; `Category=Perf` harness opt-in via
`OTOPCUA_SCRIPT_PERF=1`). docker-dev overlay: `docker-dev/docker-compose.script-runner.yml`. See `docs/VirtualTags.md`
+2 -1
View File
@@ -1,5 +1,6 @@
# docker-dev/docker-compose.script-runner.yml — optional overlay: run the central pair's VirtualTag + Calculation
# scripts OUT OF PROCESS (lmxopcua#780, Scripting:Runner=OutOfProcess). The scripted-alarm seam stays in process.
# scripts OUT OF PROCESS (lmxopcua#780, Scripting:Runner=OutOfProcess). The scripted-alarm seam stays in process; to
# move it too (last in the rollout, design §14 #10) add `Scripting__Runner__Seams__2: "ScriptedAlarm"` below.
#
# Up (recreates only the two central nodes with the extra env; the image must be built from a branch that has the
# runner, i.e. `docker compose -f docker-dev/docker-compose.yml build central-1` first):
+4 -2
View File
@@ -212,10 +212,12 @@ scripted-alarm predicates) runs scripts through the shared `ScriptRunGuard`:
run returns, so a run that OOMs the process first is not stopped — that needs an
out-of-process runner (→ #750).
- **Out of process (opt-in, lmxopcua#780).** `Scripting:Runner=OutOfProcess` moves the
virtual-tag and Calculation seams' compiled runs into `ScriptWorker` processes, where a
virtual-tag and Calculation seams' compiled runs — and the scripted-alarm predicates,
when `Scripting:Runner:Seams` lists `ScriptedAlarm` — into `ScriptWorker` processes, where a
run past `T + Scripting:Worker:HardGrace` is stopped by killing its worker and an OOM
kills a worker instead of the server. A run whose worker died under it reads
`BadResourceUnavailable`. See `docs/VirtualTags.md` §"Out-of-process runner".
`BadResourceUnavailable` (an alarm is frozen with that annotation, never cleared). See
`docs/VirtualTags.md` §"Out-of-process runner".
- **Recursion.** Unbounded recursion fails the script with
`ScriptStackExhaustedException` (Bad, script log) instead of overflowing the
stack, which would terminate the whole server. The runtime compile inserts a
+2
View File
@@ -49,6 +49,8 @@ new ScriptedAlarmDefinition(
Alarm predicates reuse the same Roslyn sandbox as virtual tags — `ScriptEvaluator<AlarmPredicateContext, bool>` compiles the source, `ScriptRunGuard` runs it with the node's `Scripting:RunTimeout` budget (default 2 s — the same budget virtual tags get; before lmxopcua#779 the host built the engine without a budget and every predicate silently ran on `TimedScriptEvaluator.DefaultTimeout`, 250 ms), and `DependencyExtractor` statically harvests the tag paths the script reads. The sandbox rules (forbidden types, cancellation, logging sinks) are documented in [VirtualTags.md](VirtualTags.md); ScriptedAlarms does not redefine them. The known resource limits — unbounded script-side memory and the orphan-thread CPU-budget caveat — are documented in that file as well; per-publish assembly accretion was resolved by the Core.Scripting-008 collectible-`AssemblyLoadContext` rewrite and no longer requires periodic server restarts.
**Out of process (opt-in, lmxopcua#780).** With `Scripting:Runner=OutOfProcess` **and** `ScriptedAlarm` listed in `Scripting:Runner:Seams` (an unset list leaves alarms in process — they go last in the rollout), the predicates run in the pre-emptible `ScriptWorker` pool. `LoadAsync` still compiles every predicate, to PE bytes only, so a broken predicate fails the load exactly as before; the worker compiles and runs it. Each re-evaluation pass (and the startup-recovery pass) submits every dirty alarm's predicate in **one batch** and applies the results — Part 9 transition, persist-before-update, emissions — in the serial order under the engine's gate, so the transition sequence is the in-process one and a large engine pays one round trip per pass, not one per alarm. Failure semantics (design §9): a predicate that times out, is killed at its hard deadline or is quarantined freezes its condition with a `BadInternalError` quality annotation, as in process; one whose worker restarted under it, or that found no worker ready, freezes with **`BadResourceUnavailable`** (`0x80040000`) and is re-evaluated by itself when the worker is back. No outcome but a completed `bool` reaches the state machine, so a dead worker never clears an alarm. Quarantines (`ScriptedAlarmEngine.QuarantinedAlarmIds`) lift on a reload. See [VirtualTags.md](VirtualTags.md) §"Out-of-process runner".
`AlarmPredicateContext` (`AlarmPredicateContext.cs`) is the script's `ScriptContext` subclass:
- `GetTag(path)` returns a `DataValueSnapshot` from the engine-maintained read cache. Missing path → `DataValueSnapshot(null, 0x80340000u, null, now)` (`BadNodeIdUnknown`). An empty path returns the same.
+27 -6
View File
@@ -46,10 +46,11 @@ The sandbox cannot pre-empt a script. **Memory:** each run's own-thread allocati
`Scripting:Runner` selects where compiled scripts run: **`InProcess`** (the default and the rollback — everything
above, unchanged), **`OutOfProcess`**, or `Shadow` (design P2; not implemented yet, so it runs `InProcess` with a
startup Warning). `Scripting:Runner:Seams` narrows `OutOfProcess` to a subset (`VirtualTag`, `Calculation`; the
`ScriptedAlarm` seam stays in process for now — alarms go last, design §14 #10). Under `OutOfProcess` the virtual-tag
and Calculation evaluators hand each compiled run to `OutOfProcessScriptRunner`
(`Core.Scripting.OutOfProcess`), which runs it in a pool of `ZB.MOM.WW.OtOpcUa.ScriptWorker` processes:
startup Warning). `Scripting:Runner:Seams` selects the seams (`VirtualTag`, `Calculation`, `ScriptedAlarm`). Unset or
empty means `VirtualTag` + `Calculation`: the **`ScriptedAlarm` seam runs out of process only when it is listed
explicitly** (alarms go last, design §14 #10). Under `OutOfProcess` the selected seams hand their compiled runs to
`OutOfProcessScriptRunner` (`Core.Scripting.OutOfProcess`), which runs them in a pool of `ZB.MOM.WW.OtOpcUa.ScriptWorker`
processes:
- **K shards** (`Scripting:Worker:Shards`, default 2) placed by a stable FNV-1a hash of the key, plus an on-demand
**isolation worker** that runs suspects one at a time. No worker starts until the first compiled evaluation, so a
@@ -122,13 +123,33 @@ and Calculation evaluators hand each compiled run to `OutOfProcessScriptRunner`
directly: `OutOfProcessScriptRunner.ResolveWorkerEntry` picks the apphost only when the worker's `runtimeconfig.json` lists
`includedFrameworks`. Gated on windev against a real `-r win-x64` publish (`OTOPCUA_SCRIPTWORKER_TEST_PATH` points the
worker suite at the published apphost). Not yet done (lmxopcua#780): the low-privilege worker account, pipe ACL, firewall
rule and PeCacheDir ACL (installer), the scripted-alarm seam and the per-seam batch submit (design §4.5).
rule and PeCacheDir ACL (installer).
- **docker-dev.** `docker-dev/docker-compose.script-runner.yml` turns `OutOfProcess` on for the central pair's
VirtualTag and Calculation seams. The 2026-11-10 live drill (runaway kill and quarantine, `kill -9` recovery, open circuit
with readiness still 200) is recorded in the design doc's rollout section.
- **Perf harness.** `ScriptRunnerPerfTests` (`Category=Perf`, skipped unless `OTOPCUA_SCRIPT_PERF=1`) measures InProcess
against OutOfProcess at K = 1/2/4 against the accepted 5,000 evals/s/node gate and the design §8.2 budgets. The numbers
are in the design doc's rollout section (§12.1). The gate passes; the host-CPU and RSS budgets miss (lmxopcua#789).
are in the design doc's rollout section (§12.1, §12.2). The gate passes; the host-CPU and RSS budgets miss (lmxopcua#789).
It also measures batch submit (RunBatch against the same evaluations as concurrent RunAsync calls) and the
scripted-alarm pass latency (push → last transition, in process / out of process batched / out of process serial)
against the 250 ms alarm budget — design §12.3.
- **Batch submit (design §4.5).** `IScriptRunner.RunBatch` submits several evaluations at once: every item that is ready
to run (its key loaded in a live worker, not a suspect) goes out in ONE `EvalBatch` frame per worker, and results are
per item and streamed. The Calculation pump dequeues everything pending and submits it as one batch, publishing in
dequeue order (a tag queued twice ends the batch, so every queued evaluation still runs against the snapshot that
triggered it); the virtual-tag seam coalesces evaluations that become ready together (one actor per tag) into one
batch, with no timer; the scripted-alarm engine submits a pass's whole dirty set. In process the pump and the engine
stay one-at-a-time, unchanged. **Per-item deadlines and blame stay exact:** an evaluation that arrives while every
evaluation thread is busy is reported back as *deferred*, the host stops its clock, and restarts T and G when the worker
reports it *started* — so an item queued behind a runaway sibling is neither timed out nor struck, and at the kill it
is a victim (`script worker restarted`), not the culprit.
- **Scripted alarms** (only when `Scripting:Runner:Seams` lists `ScriptedAlarm`). The engine compiles each predicate at
load only to validate it (same Roslyn, sandbox and diagnostics, so a broken predicate still fails the load; no
assembly is loaded into the host), the worker compiles and runs it, and each re-evaluation pass submits its dirty set
as one batch and applies the results in the serial order under the engine's gate — the Part 9 sequence is the
in-process one. A predicate that did not complete freezes its condition (a dead worker never clears an alarm): timed
out, killed or quarantined → `BadInternalError` annotation, as in process; the runner's fault (worker restarted, none
ready) → `BadResourceUnavailable` annotation, and the alarm is re-evaluated by itself when the worker is back.
- **Compile threads.** A worker compiles on `min(8, max(1, CPUs/2))` dedicated threads, never on the thread pool, so a
burst of cold loads cannot starve its heartbeat or its pipe writer. The host load timeout still counts the time a load
spends in that queue (lmxopcua#790).
@@ -640,7 +640,7 @@ per-shard state, restarts in the last hour and the quarantine list. Quarantines
| Key | Default | Meaning |
|---|---|---|
| `Scripting:Runner` | `InProcess` | `InProcess` \| `OutOfProcess` \| `Shadow` |
| `Scripting:Runner:Seams` | all three | Optional per-seam subset for `OutOfProcess` / `Shadow` (e.g. `["Calculation"]`) |
| `Scripting:Runner:Seams` | VirtualTag + Calculation | Optional per-seam subset for `OutOfProcess` / `Shadow` (e.g. `["Calculation"]`). Unset selects VirtualTag and Calculation; `ScriptedAlarm` only when listed (§14 #10, as built in §12.3) |
| `Scripting:Worker:Shards` | `2` | 1–8 |
| `Scripting:Worker:HardGrace` | `max(1s, T)` | §5.1 |
| `Scripting:Worker:HeapLimitBytes` | `536870912` | §5.2 |
@@ -660,6 +660,9 @@ per-shard state, restarts in the last hour and the quarantine list. Quarantines
`VirtualTagActor`), so the dispatcher-thread blocking is gone. The Calc pump and the alarm engine still run one
script at a time; why that was not relaxed in-process is recorded in `docs/VirtualTags.md` §"Upstream-tag feed".
`IScriptRunner` and batch-submit remain to do.
- **Batch submit landed in lmxopcua#780 (fourth P1 slice, §12.3):** `IScriptRunner.RunBatch`, used by the Calc pump, the
VirtualTag seam (a coalescer) and the scripted-alarm engine on the out-of-process path. In process the pump and the
engine stay one at a time, unchanged.
2. **P1, worker + protocol behind the flag.**
- Ship `ScriptWorker`, `OutOfProcessScriptRunner` and the PE cache.
- Default stays `InProcess`. Enable on the docker-dev rig only.
@@ -686,6 +689,11 @@ per-shard state, restarts in the last hour and the quarantine list. Quarantines
§10 gauges and histograms, the degraded-but-ready `script-runner` health check (§14 #12), and the orphan sweep by
launch marker (pid + start time + image, host gone). Still open: self-contained worker publish, installer (account,
pipe ACL, firewall), docker-dev live drill, scripted-alarm seam, the §4.5 batch submit, the §8 perf harness.
- **Fourth P1 slice (lmxopcua#780, record in §12.3):** the §4.5 batch submit (`IScriptRunner.RunBatch`, one `EvalBatch`
frame per worker; Calc pump, VT coalescer, alarm dirty set), exact per-item deadlines for work queued in the worker
(deferral / start notices), and the scripted-alarm seam behind `Scripting:Runner:Seams` — alarms run out of process
only when listed. Still open: installer (account, pipe ACL, firewall, PeCacheDir ACL), #789's remaining misses, then
P2 Shadow.
- **Third P1 slice (lmxopcua#780, record in §12.1):** a self-contained worker publish for self-contained (Windows)
Host publishes, started through its apphost; the docker-dev live drill (`docker-dev/docker-compose.script-runner.yml`);
the §8 perf harness (`Category=Perf`, opt-in), which passes the accepted 5k evals/s/node gate; and two runner fixes
@@ -963,6 +971,133 @@ Additional measurements:
- Tests: `A_shard_that_cannot_start_logs_one_error_then_concise_rate_limited_retries_then_its_recovery`, with two
intervals. Red when every failure is logged at Error with the exception.
### 12.3 Fourth P1 slice: batch submit, exact per-item deadlines, the scripted-alarm seam (2026-11-10)
**Batch submit (§4.5, P0).**
- `IScriptRunner.RunBatch(invocations)` returns one task per item. Results are per item and streamed.
- The default implementation is `RunAsync` per item, started together. `InProcessScriptRunner` uses it.
- `OutOfProcessScriptRunner` sends every item that is ready to run as ONE `EvalBatch` frame per worker. Ready means
admitted, not quarantined, not a suspect, and already loaded in a live worker at this source and slot order.
- Anything else goes through the full `RunAsync` path on its own: a first evaluation that needs a compile, a worker
that is down, or a suspect bound for the one-at-a-time isolation worker. A batch therefore never waits for a load
or a restart as a whole.
- Seams that use it, on the out-of-process path only:
- **Calculation pump.** It dequeues everything pending (each entry's dependency snapshot was taken at enqueue),
submits one batch, and publishes in dequeue order. A tag queued twice ends the batch, so every queued evaluation
still runs, in order, against the snapshot that triggered it. The published sequence is the one-at-a-time pump's.
- **VirtualTag seam.** There is one actor per tag, so an upstream burst makes many tags ready at once from different
threads. `ScriptBatchCoalescer` hands those to one `RunBatch`. It has no timer: an evaluation on an idle seam is
submitted at once by its own caller.
- **Scripted alarms.** See below.
- In process, the Calc pump and the alarm engine stay one at a time, unchanged. The VirtualTag in-process path does
not touch the coalescer.
**Per-item deadlines and blame stay exact under batching.**
- The host arms T and T + G at submit. That is when the run starts, provided the worker has an idle evaluation thread.
- A batch larger than the worker's scheduler, or work queued behind a runaway, used to wait in the worker with its
clock already running. It could then be timed out, take timeout strikes, and even be the key blamed at T + G.
- The worker's scheduler now hands an evaluation straight to an idle thread, or else puts it in a backlog and reports
it in `EvalResults.deferred_eval_ids`.
- The host stops that evaluation's clock.
- When a thread picks the evaluation up, its id rides back in `started_eval_ids` on the result frame of the run that
thread just finished. The host then restarts the full T, then G, from the start.
- The two notices are unordered on the wire, and the host handles either order.
- The deadlines are absolute timestamps that the timer re-checks, so a stale tick never fires early.
- A run queued behind a runaway is therefore neither timed out nor struck. At the kill it is a victim
(`script worker restarted`, `BadResourceUnavailable`, re-evaluated on return), not the culprit.
- Tests:
- `A_runaway_in_a_batch_is_blamed_alone_and_siblings_queued_behind_it_are_not_timed_out_or_struck` runs one evaluation
thread and a batch of a runaway plus three siblings. It is red with the host ignoring the deferral, and red with the
worker never deferring.
- `A_deferred_run_is_timed_and_killed_from_when_it_starts` is the positive control. It is red with a start notice that
does not re-arm the clock.
- This also changes plain `RunAsync` under saturation: a run that waits for a thread no longer times out on its wait.
That matches the in-process guard, whose budget clock starts when the pool picks the run up (§1.2).
**Scripted-alarm seam (★10: alarms last).**
- `Scripting:Runner:Seams` unset now means VirtualTag + Calculation. **`ScriptedAlarm` runs out of process only when it
is listed.**
- `ScriptRunnerSelection` serves it. The Runtime's `DriverHostActor` gets it through the new `IScriptRunnerProvider`
and passes it to the engine (a new last parameter, `scriptedAlarmRunner`).
- Engine changes, all on the runner path only:
- `LoadAsync` compiles each predicate to PE bytes only (`CompileToImage`). A broken predicate still fails the load
exactly as in process, and no assembly is loaded into the host. Deviation: Roslyn still runs in the host at load
for alarms; the worker compiles again for execution.
- Startup recovery and every re-evaluation pass snapshot each dirty alarm's inputs under `_evalGate`, submit all
their predicates as one batch, and then apply the results in the serial order under the gate: Part 9 transition,
persist before update, emissions. A test shows the transition sequence is identical to an in-process engine's
for the same pushes.
- Failure semantics (§9, alarm column):
- Timed out, killed, refused or quarantined: the condition is frozen with a `BadInternalError` annotation, as today.
- The runner's fault (worker restarted, none ready): frozen with **`BadResourceUnavailable`**, consistent with
VT/Calc. The alarm is re-evaluated by itself when the runner's `Recovery` completes.
- A predicate that threw keeps its state with no annotation, as in process.
- No outcome but a completed `bool` reaches the state machine, so a dead worker never clears an alarm.
- Real-worker tests:
- `A_runaway_predicate_is_killed_and_quarantined_its_alarm_frozen_Bad_and_a_sibling_keeps_evaluating`;
- `A_worker_that_dies_under_a_predicate_freezes_it_BadResourceUnavailable_never_clears_it_and_it_is_re_evaluated_on_return`.
It is red when the runner's failure is read as `false`.
- Budget note: since lmxopcua#779 the host passes `Scripting:RunTimeout` (2 s) to the engine. The passes below are
still judged against the stricter 250 ms the design was written for.
**Measured (§8 harness, `Batch_throughput_and_alarm_pass_latency`; macOS Arm64, 14 CPUs, load average about 14 from
other work on the box, 10 s phases):**
| Batch size | RunBatch evals/s | RunAsync ×n evals/s | RunBatch p50 / p99 µs | RunAsync ×n p50 / p99 µs |
|---:|---:|---:|---:|---:|
| 1 | 38,296 | 40,980 | 24 / 41 | 23 / 35 |
| 10 | 147,035 | 118,516 | 57 / 121 | 70 / 142 |
| 100 | 369,304 | 232,967 | 215 / 1,095 | 363 / 1,341 |
| 500 | 463,033 | 256,232 | 818 / 2,541 | 1,701 / 3,178 |
(OutOfProcess K = 2, one batch at a time. The in-process rows show no difference between the two, as expected.)
| Alarms per pass | InProcess p50 / p99 ms | OutOfProcess batched p50 / p99 ms | OutOfProcess serial p50 / p99 ms |
|---:|---:|---:|---:|
| 100 | 0.4 / 1.5 | 0.4 / 2.1 | 3.6 / 4.9 |
| 500 | 2.2 / 4.7 | 1.7 / 4.8 | 16.1 / 18.0 |
| 1,000 | 4.4 / 7.7 | 2.5 / 41.0 | 28.1 / 33.4 |
- A pass is push → the last of the N transitions it emits. "Serial" replays the pre-batch one-predicate-at-a-time pass
over the same runner. That is the N × round-trip cost §8.2 warned about: about 30 µs per alarm, against 2–4 µs
batched.
- Every configuration is far inside 250 ms. The one 41 ms p99 is a single loaded-box outlier.
- The §8.2 harness rerun on the same box after the scheduler change:
- K = 2 closed loop ×64: 366k evals/s;
- paced 10k/s: +5 % host CPU vs InProcess (budget ≤ 25 %);
- idle p50: 25 µs;
- kill latency: 14–19 ms;
- warm-cache restart of 1,000 scripts: 220 ms;
- every result correct.
**windev** (Windows VM, 36 CPUs, .NET 10.0.8; the full `ScriptWorker.Tests` suite also passes there, 157/157):
| Batch size | RunBatch evals/s | RunAsync ×n evals/s | RunBatch host CPU | RunAsync ×n host CPU |
|---:|---:|---:|---:|---:|
| 1 | 1,810 | 1,921 | 63 % | 65 % |
| 10 | 13,013 | 11,220 | 109 % | 157 % |
| 100 | 48,685 | 40,699 | 129 % | 224 % |
| 500 | 86,322 | 83,116 | 143 % | 289 % |
| Alarms per pass | InProcess p50 / p99 ms | OutOfProcess batched p50 / p99 ms | OutOfProcess serial p50 / p99 ms |
|---:|---:|---:|---:|
| 100 | 8.2 / 39.0 | 2.9 / 5.1 | 51.7 / 64.8 |
| 500 | 42.3 / 53.3 | 7.4 / 35.7 | **236.5 / 273.3 (MISS)** |
| 1,000 | 36.6 / 84.2 | 11.7 / 52.7 | **466.8 / 519.4 (MISS)** |
- On the VM, where one round trip costs about 0.5 ms, the pre-batch serial pass **misses the 250 ms budget from about
500 alarms**. The batched pass stays at p99 ≤ 53 ms for 1,000 alarms, below the in-process engine's own figures.
That is the measurement behind §8's "batch submit is a prerequisite, not an optimisation", and the reason ★10 put
alarms last.
- On Windows, batching halves the host CPU of a 100–500 evaluation burst. That is the overlapped-I/O path that is still
#789's main miss.
- No predicate failed to complete in any pass on either host.
- Rerun of the §8.2 harness on windev:
- K = 2 closed loop ×64: 50k evals/s, gate PASS;
- paced 10k/s: +99 % host CPU vs InProcess (#789, unchanged in kind);
- kill latency: 9–19 ms;
- warm-cache restart of 1,000 scripts: 1.8 s.
---
## 13. Test plan
@@ -81,6 +81,17 @@ public sealed class ScriptedAlarmEngine : IDisposable
/// </summary>
private readonly ScriptRunGuard _guard;
/// <summary>
/// lmxopcua#780 — the out-of-process runner when <c>Scripting:Runner=OutOfProcess</c> lists the ScriptedAlarm seam;
/// null (the default) keeps the in-process guard path, unchanged. With a runner the predicates compile and run in
/// the pre-emptible worker pool, and a re-evaluation pass submits its whole dirty set as one batch.
/// </summary>
private readonly IScriptRunner? _runner;
// Alarms whose predicate came back Unavailable (the runner, not the script, failed) and that are waiting for the
// runner's Recovery to re-evaluate them once. Guarded by _dirtyLock.
private readonly HashSet<string> _recoveryArmed = new(StringComparer.Ordinal);
// ConcurrentDictionary, not a plain Dictionary: every mutation happens under
// _evalGate, but four read paths (GetState, GetAllStates, LoadedAlarmIds,
// RunShelvingCheck) touch _alarms from arbitrary threads (Admin UI request
@@ -243,6 +254,8 @@ public sealed class ScriptedAlarmEngine : IDisposable
/// default used to be <c>TimedScriptEvaluator.DefaultTimeout</c> (250 ms), which is what every production engine
/// silently ran with because the host never passed a value.</param>
/// <param name="scripting">Run-guard budget + quarantine threshold; null ⇒ <see cref="ScriptingOptions"/> defaults.</param>
/// <param name="scriptRunner">lmxopcua#780 — the out-of-process runner when the ScriptedAlarm seam runs out of
/// process; null (the default) keeps the in-process guard path, unchanged.</param>
public ScriptedAlarmEngine(
ITagUpstreamSource upstream,
IAlarmStateStore store,
@@ -250,8 +263,10 @@ public sealed class ScriptedAlarmEngine : IDisposable
ILogger engineLogger,
Func<DateTime>? clock = null,
TimeSpan? scriptTimeout = null,
ScriptingOptions? scripting = null)
ScriptingOptions? scripting = null,
IScriptRunner? scriptRunner = null)
{
_runner = scriptRunner;
_upstream = upstream ?? throw new ArgumentNullException(nameof(upstream));
_store = store ?? throw new ArgumentNullException(nameof(store));
_loggerFactory = loggerFactory ?? throw new ArgumentNullException(nameof(loggerFactory));
@@ -268,6 +283,25 @@ public sealed class ScriptedAlarmEngine : IDisposable
/// <summary>The wall-clock budget each predicate run is given (diagnostics / tests).</summary>
public TimeSpan ScriptTimeout => _scriptTimeout;
/// <summary>The out-of-process runner predicates run through, or null in process (diagnostics / tests).</summary>
public IScriptRunner? ScriptRunner => _runner;
/// <summary>
/// The alarm ids whose predicate is quarantined — by the in-process guard, or by the out-of-process runner when
/// one is in use (diagnostics / tests).
/// </summary>
public IReadOnlyCollection<string> QuarantinedAlarmIds => _runner is null
? _guard.QuarantinedKeys
: [.. _guard.QuarantinedKeys, .. _runner.QuarantinedKeys(ScriptContextKind.AlarmPredicate)];
/// <summary>
/// OPC UA <c>BadResourceUnavailable</c> — the annotation for a predicate the out-of-process RUNNER could not run
/// (its worker restarted under it, or no worker was ready): the condition is frozen exactly as for a script
/// fault, but the quality tells the operator the node is sick, not the script (design §9, §14 #3; the same split
/// the virtual-tag and Calculation seams make).
/// </summary>
private const uint StatusBadResourceUnavailable = 0x80040000u;
/// <summary>
/// OPC UA <c>BadInternalError</c> — the quality a condition is annotated with while its predicate
/// cannot be evaluated (timed out, refused because the previous run is still executing, or
@@ -331,6 +365,8 @@ public sealed class ScriptedAlarmEngine : IDisposable
// A (re)load is a deploy: lift every run-guard quarantine (an edited predicate already lifts
// its own through the guard's source identity; this covers "redeployed to retry").
_guard.ResetAll();
_runner?.Reset(ScriptContextKind.AlarmPredicate);
lock (_dirtyLock) _recoveryArmed.Clear();
var compileFailures = new List<string>();
foreach (var def in definitions)
@@ -350,7 +386,13 @@ public sealed class ScriptedAlarmEngine : IDisposable
// The compiled script is run through _guard (timeout + quarantine) at evaluation time —
// the per-alarm TimedScriptEvaluator this used to wrap gave the timeout but could not
// see that the run it abandoned was still executing.
var evaluator = _compileCache.GetOrCompile(def.PredicateScriptSource);
//
// Out of process (lmxopcua#780) the worker compiles and runs the predicate; here the source is only
// compiled to PE bytes and dropped — same Roslyn, sandbox and diagnostics, so a broken predicate still
// fails the load exactly as in process — and no assembly is loaded into the host.
ScriptEvaluator<AlarmPredicateContext, bool>? evaluator = null;
if (_runner is null) evaluator = _compileCache.GetOrCompile(def.PredicateScriptSource);
else _ = ScriptEvaluator<AlarmPredicateContext, bool>.CompileToImage(def.PredicateScriptSource);
var logger = _loggerFactory.Create(def.AlarmId);
var templateTokens = MessageTemplate.ExtractTokenPaths(def.MessageTemplate);
@@ -396,14 +438,32 @@ public sealed class ScriptedAlarmEngine : IDisposable
// Any predicate emissions queue into `pending` and fire after the gate
// is released — so a startup-recovery activation event can call back into
// the engine without deadlocking.
foreach (var (alarmId, state) in _alarms)
if (_runner is null)
{
var persisted = await _store.LoadAsync(alarmId, ct).ConfigureAwait(false);
var seed = persisted ?? state.Condition;
var afterPredicate = await EvaluatePredicateToStateAsync(state, seed, nowUtc: _clock(), ct, pending)
.ConfigureAwait(false);
_alarms[alarmId] = state with { Condition = afterPredicate };
await _store.SaveAsync(afterPredicate, ct).ConfigureAwait(false);
foreach (var (alarmId, state) in _alarms)
{
var persisted = await _store.LoadAsync(alarmId, ct).ConfigureAwait(false);
var seed = persisted ?? state.Condition;
var afterPredicate = await EvaluatePredicateToStateAsync(state, seed, nowUtc: _clock(), ct, pending)
.ConfigureAwait(false);
_alarms[alarmId] = state with { Condition = afterPredicate };
await _store.SaveAsync(afterPredicate, ct).ConfigureAwait(false);
}
}
else
{
// Out of process: every predicate goes in one batch, then each alarm is applied and saved in the
// same order the serial loop above uses.
var seeds = new List<(AlarmState State, AlarmConditionState Seed)>(_alarms.Count);
foreach (var (alarmId, state) in _alarms)
seeds.Add((state, await _store.LoadAsync(alarmId, ct).ConfigureAwait(false) ?? state.Condition));
var after = await EvaluatePredicateBatchToStatesAsync(seeds, _clock(), ct, pending).ConfigureAwait(false);
for (var i = 0; i < seeds.Count; i++)
{
var state = seeds[i].State;
_alarms[state.Definition.AlarmId] = state with { Condition = after[i] };
await _store.SaveAsync(after[i], ct).ConfigureAwait(false);
}
}
_loaded = true;
@@ -614,7 +674,12 @@ public sealed class ScriptedAlarmEngine : IDisposable
if (_disposed) return; // a disposed engine neither caches nor queues — nothing reads it again
_valueCache[path] = value;
if (!_alarmsReferencing.TryGetValue(path, out var alarmIds)) return;
MarkDirty(alarmIds);
}
/// <summary>Marks <paramref name="alarmIds"/> dirty and starts the drainer if none is running.</summary>
private void MarkDirty(IEnumerable<string> alarmIds)
{
bool startDrain;
lock (_dirtyLock)
{
@@ -754,17 +819,25 @@ public sealed class ScriptedAlarmEngine : IDisposable
// disposing store or mutating _alarms after clear is unsafe.
if (_disposed) return;
foreach (var id in alarmIds)
if (_runner is not null)
{
if (!_alarms.TryGetValue(id, out var state)) continue;
var newState = await EvaluatePredicateToStateAsync(
state, state.Condition, _clock(), ct, pending).ConfigureAwait(false);
if (!ReferenceEquals(newState, state.Condition))
// Out of process (lmxopcua#780, design §4.5): the whole dirty set in one batch, applied in order.
await ReevaluateBatchAsync(alarmIds, ct, pending).ConfigureAwait(false);
}
else
{
foreach (var id in alarmIds)
{
// Persist before updating in-memory so a store failure leaves
// both sides at the prior state.
await _store.SaveAsync(newState, ct).ConfigureAwait(false);
_alarms[id] = state with { Condition = newState };
if (!_alarms.TryGetValue(id, out var state)) continue;
var newState = await EvaluatePredicateToStateAsync(
state, state.Condition, _clock(), ct, pending).ConfigureAwait(false);
if (!ReferenceEquals(newState, state.Condition))
{
// Persist before updating in-memory so a store failure leaves
// both sides at the prior state.
await _store.SaveAsync(newState, ct).ConfigureAwait(false);
_alarms[id] = state with { Condition = newState };
}
}
}
}
@@ -837,7 +910,7 @@ public sealed class ScriptedAlarmEngine : IDisposable
try
{
predicateTrue = await _guard.RunAsync(
state.Definition.AlarmId, state.Definition.PredicateScriptSource, state.Evaluator, context,
state.Definition.AlarmId, state.Definition.PredicateScriptSource, state.Evaluator!, context,
_scriptTimeout, ct)
.ConfigureAwait(false);
}
@@ -890,6 +963,162 @@ public sealed class ScriptedAlarmEngine : IDisposable
return result.State;
}
/// <summary>
/// The out-of-process re-evaluation pass (lmxopcua#780, design §4.5): every dirty alarm's predicate goes to the
/// runner in ONE batch, then the results are applied — Part 9 transition, persist-before-update, emissions — in
/// the original order, under <c>_evalGate</c>. Predicates are pure functions of the input snapshot taken here,
/// under the gate, so the state machine sees exactly the sequence the serial loop produces. The batch costs one
/// round trip for the whole set instead of one per alarm, which is what keeps a large engine inside its budget.
/// </summary>
private async Task ReevaluateBatchAsync(IReadOnlyList<string> alarmIds, CancellationToken ct, List<ScriptedAlarmEvent> pending)
{
var items = new List<(AlarmState State, AlarmConditionState Seed)>(alarmIds.Count);
foreach (var id in alarmIds)
if (_alarms.TryGetValue(id, out var state)) items.Add((state, state.Condition));
if (items.Count == 0) return;
var after = await EvaluatePredicateBatchToStatesAsync(items, _clock(), ct, pending).ConfigureAwait(false);
for (var i = 0; i < items.Count; i++)
{
var (state, seed) = items[i];
if (ReferenceEquals(after[i], seed)) continue;
// Persist before updating in-memory so a store failure leaves both sides at the prior state.
await _store.SaveAsync(after[i], ct).ConfigureAwait(false);
_alarms[state.Definition.AlarmId] = state with { Condition = after[i] };
}
}
/// <summary>
/// <see cref="EvaluatePredicateToStateAsync"/> for a batch, through <see cref="_runner"/>: the same readiness gate,
/// quality tracking and Part 9 application per alarm, in order, with every predicate that has to run submitted
/// together. Failure semantics (design §9): a script fault, timeout, kill or quarantine freezes the condition with a
/// <c>BadInternalError</c> annotation, exactly as in process; a runner failure (worker restarted, none ready) freezes
/// it with <c>BadResourceUnavailable</c> and re-evaluates it once the runner can run it again. A dead worker never
/// clears an alarm: no outcome but a completed <c>bool</c> reaches the state machine.
/// </summary>
private async Task<AlarmConditionState[]> EvaluatePredicateBatchToStatesAsync(
IReadOnlyList<(AlarmState State, AlarmConditionState Seed)> items, DateTime nowUtc, CancellationToken ct,
List<ScriptedAlarmEvent> pendingEmissions)
{
var runner = _runner!;
var after = new AlarmConditionState[items.Count];
var scratches = new AlarmScratch[items.Count];
var worst = new uint[items.Count];
var slot = new int[items.Count];
var invocations = new List<ScriptInvocation>(items.Count);
// Pass 1, no side effects: snapshot each alarm's inputs and decide whether its predicate runs.
for (var i = 0; i < items.Count; i++)
{
var state = items[i].State;
var scratch = scratches[i] = _scratchByAlarmId.GetOrAdd(
state.Definition.AlarmId, _ => new AlarmScratch(state.Inputs, state.Logger, _clock));
RefillReadCache(scratch.ReadCache, state.Inputs);
worst[i] = WorstInputStatus(scratch.ReadCache);
if (!AreInputsReady(scratch.ReadCache))
{
slot[i] = -1;
continue;
}
slot[i] = invocations.Count;
// A copy: the runner may still read the inputs after the caller has its answer (a soft timeout), and the
// scratch is refilled by the next pass.
invocations.Add(new ScriptInvocation(state.Definition.AlarmId, ScriptContextKind.AlarmPredicate,
state.Definition.PredicateScriptSource,
new Dictionary<string, DataValueSnapshot>(scratch.ReadCache, StringComparer.Ordinal), _scriptTimeout, state.Logger));
}
var outcomes = invocations.Count == 0 ? [] : runner.RunBatch(invocations, ct);
PredicateBatchSubmittedForTest?.Invoke(invocations.Count);
// Pass 2, in order: exactly what the serial path does with each alarm.
for (var i = 0; i < items.Count; i++)
{
var (state, seed) = items[i];
var scratch = scratches[i];
if (slot[i] < 0)
{
if (TrackQualityBucket(state.Definition.AlarmId, worst[i]))
pendingEmissions.Add(BuildQualityEmission(state, seed, worst[i]));
after[i] = seed;
continue;
}
var outcome = await outcomes[slot[i]].ConfigureAwait(false);
after[i] = ApplyPredicateOutcome(state, seed, scratch, worst[i], outcome, nowUtc, ct, pendingEmissions);
}
return after;
}
/// <summary>One runner outcome onto one alarm (the second half of <see cref="EvaluatePredicateToStateAsync"/>).</summary>
private AlarmConditionState ApplyPredicateOutcome(AlarmState state, AlarmConditionState seed, AlarmScratch scratch,
uint worstStatus, ScriptRunOutcome outcome, DateTime nowUtc, CancellationToken ct, List<ScriptedAlarmEvent> pendingEmissions)
{
switch (outcome.Status)
{
case ScriptRunStatus.Unavailable:
// Not the script's fault: frozen, annotated BadResourceUnavailable, and evaluated again (against the
// inputs current then) when its worker is back.
ArmRecovery(state.Definition.AlarmId, outcome.Recovery);
return PredicateDidNotComplete(state, seed, scratch, outcome.Reason ?? ScriptRunOutcome.RunnerUnavailableReason,
pendingEmissions, StatusBadResourceUnavailable);
case ScriptRunStatus.Failed:
var reason = outcome.Reason ?? "evaluator failure";
if (ct.IsCancellationRequested) throw new OperationCanceledException(ct);
if (reason.StartsWith(ScriptThrewPrefix, StringComparison.Ordinal))
{
// As in process: a predicate that threw logs an Error and keeps its state, with no quality annotation.
scratch.LastNonCompletion = null;
state.Logger.Error("Alarm predicate threw — state unchanged: {Reason}", reason[ScriptThrewPrefix.Length..]);
return seed;
}
// Timed out, killed, refused, quarantined, unsupported type, did not compile: frozen + BadInternalError.
return PredicateDidNotComplete(state, seed, scratch, reason, pendingEmissions);
}
if (outcome.Value is not bool predicateTrue)
{
scratch.LastNonCompletion = null;
state.Logger.Error("Alarm predicate returned {Type} instead of a bool — state unchanged", outcome.Value?.GetType().Name ?? "null");
return seed;
}
scratch.LastNonCompletion = null;
// From here on, identical to the serial path: the bucket is tracked only once the predicate completed (I3).
var qualityBucketChanged = TrackQualityBucket(state.Definition.AlarmId, worstStatus);
var result = Part9StateMachine.ApplyPredicate(seed, predicateTrue, nowUtc);
var transition = result.Emission != EmissionKind.None
? BuildEmission(state, result.State, result.Emission, worstStatus)
: null;
if (transition is not null) pendingEmissions.Add(transition);
else if (qualityBucketChanged) pendingEmissions.Add(BuildQualityEmission(state, result.State, worstStatus));
return result.State;
}
/// <summary>The prefix the runners put on a predicate that threw (the in-process path catches the exception instead).</summary>
private const string ScriptThrewPrefix = "script threw: ";
/// <summary>
/// Re-evaluate <paramref name="alarmId"/> once when <paramref name="recovery"/> completes — the runner's worker is
/// back (design §9 "re-evaluated on return"). At most one wait per alarm.
/// </summary>
private void ArmRecovery(string alarmId, Task? recovery)
{
if (recovery is null) return;
lock (_dirtyLock)
{
if (_disposed || !_recoveryArmed.Add(alarmId)) return;
}
_ = recovery.ContinueWith(_ =>
{
lock (_dirtyLock) _recoveryArmed.Remove(alarmId);
if (!_disposed) MarkDirty([alarmId]);
}, CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
}
/// <summary>Test hook: the number of predicates each out-of-process batch pass submitted.</summary>
internal Action<int>? PredicateBatchSubmittedForTest { get; set; }
/// <summary>
/// The predicate did not return (timed out, refused because the previous run is still executing, or
/// quarantined): hold the condition at <paramref name="seed"/>, log the reason <b>once per change of
@@ -900,15 +1129,15 @@ public sealed class ScriptedAlarmEngine : IDisposable
/// </summary>
private AlarmConditionState PredicateDidNotComplete(
AlarmState state, AlarmConditionState seed, AlarmScratch scratch, string reason,
List<ScriptedAlarmEvent> pendingEmissions)
List<ScriptedAlarmEvent> pendingEmissions, uint status = StatusBadInternalError)
{
if (!string.Equals(scratch.LastNonCompletion, reason, StringComparison.Ordinal))
{
scratch.LastNonCompletion = reason;
state.Logger.Warning("Alarm predicate did not complete ({Reason}) — state unchanged, quality Bad", reason);
}
if (TrackQualityBucket(state.Definition.AlarmId, StatusBadInternalError))
pendingEmissions.Add(BuildQualityEmission(state, seed, StatusBadInternalError));
if (TrackQualityBucket(state.Definition.AlarmId, status))
pendingEmissions.Add(BuildQualityEmission(state, seed, status));
return seed;
}
@@ -1339,7 +1568,7 @@ public sealed class ScriptedAlarmEngine : IDisposable
private sealed record AlarmState(
ScriptedAlarmDefinition Definition,
ScriptEvaluator<AlarmPredicateContext, bool> Evaluator,
ScriptEvaluator<AlarmPredicateContext, bool>? Evaluator, // null out of process: the worker holds the compiled form
IReadOnlySet<string> Inputs,
IReadOnlyList<string> TemplateTokens,
ILogger Logger,
@@ -22,6 +22,7 @@
<ItemGroup>
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms.Tests"/>
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests"/>
</ItemGroup>
</Project>
@@ -155,7 +155,7 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
internal OutOfProcessScriptRunner(ScriptingOptions options, ILogger? log, ILogger? scriptRoot,
TimeSpan heartbeatInterval, TimeSpan loadTimeout, TimeSpan? maintenanceInterval = null, string? markerDirectory = null,
int compileThreads = 0)
int compileThreads = 0, int evalThreads = 0)
{
_compileThreads = compileThreads;
ArgumentNullException.ThrowIfNull(options);
@@ -192,7 +192,9 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
var swept = WorkerLaunchMarker.Sweep(_markerDirectory, _log);
if (swept.Count > 0) _log.Warning("Killed {Count} orphaned ScriptWorker process(es) at startup: {Pids}", swept.Count, swept);
var evalThreads = Math.Max(2, Environment.ProcessorCount / _worker.Shards);
// Design §3.2 [A-4]: P = max(2, cores / K). A positive evalThreads is a test knob (a small scheduler makes the
// queue-behind-a-busy-thread case deterministic).
if (evalThreads <= 0) evalThreads = Math.Max(2, Environment.ProcessorCount / _worker.Shards);
_shards = new WorkerShard[_worker.Shards];
for (var i = 0; i < _shards.Length; i++)
{
@@ -243,6 +245,9 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
/// <summary>Test hook: runs inside the continuation that releases a key's single-flight slot, just before it does.</summary>
internal Action<string>? SlotReleasingForTest { get; set; }
/// <summary>Test hook: runs once per batch frame <see cref="RunBatch"/> submits, with the number of evaluations in it.</summary>
internal Action<int>? BatchSubmittedForTest { get; set; }
/// <summary>Scoped keys currently treated as suspects (tests / diagnostics).</summary>
internal IReadOnlyCollection<string> SuspectKeys =>
_keys.Where(kv => kv.Value.Suspect).Select(kv => kv.Key).ToArray();
@@ -414,21 +419,147 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
// next is never refused by the run it already has (the §8 harness found 32 of 200 refused when the release
// was an unordered continuation). On a worker death, attribution (strike, quarantine, suspect) has already
// run by then.
var heldGate = gate;
var released = 0;
void ReleaseSlot()
{
if (Interlocked.Exchange(ref released, 1) != 0) return;
SlotReleasingForTest?.Invoke(key);
Interlocked.Exchange(ref state.InFlight, 0);
heldGate?.Release();
}
var submitted = Stopwatch.GetTimestamp();
var pending = connection.Submit(key, BuildRequest(key, entry, invocation), invocation.Timeout,
_worker.EffectiveHardGrace(invocation.Timeout), ReleaseSlot);
_worker.EffectiveHardGrace(invocation.Timeout), SlotRelease(key, state, gate));
handedOff = true;
return await AwaitSubmittedAsync(key, state, invocation, shard, connection, pending, submitted, ct).ConfigureAwait(false);
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
return ScriptRunOutcome.Failed("evaluation cancelled");
}
finally
{
if (!handedOff)
{
Interlocked.Exchange(ref state.InFlight, 0);
gate?.Release();
}
}
// Circuit open (design §5.3.6): answer at once instead of making every trigger wait out the grace. A zero grace
// still kicks a start that is due, so the shards keep retrying at the backoff cap. Only computed when the
// shard has no live worker (or for the isolation gate).
TimeSpan Grace() => State == ScriptRunnerState.Unavailable ? TimeSpan.Zero : _worker.UnavailableGrace;
}
/// <inheritdoc />
/// <remarks>
/// Every item that can go straight to a live worker — its key admitted, not quarantined, not a suspect, its script
/// already loaded there at this source and slot order — is submitted with its siblings for the same worker as ONE
/// <c>EvalBatch</c> frame. Anything else (a first evaluation that needs a compile, a worker that is down, a suspect
/// bound for the one-at-a-time isolation worker) takes the full <see cref="RunAsync"/> path on its own, so a batch
/// never waits for a load or a restart as a whole. Per-item deadlines are armed at submit and, for an item the worker
/// has to queue behind busy evaluation threads, restarted when it starts — so an item waiting behind a runaway
/// sibling is neither timed out, struck nor blamed for the kill (it is that kill's victim, reported
/// <see cref="ScriptRunOutcome.WorkerRestartedReason"/>, design §9).
/// </remarks>
public IReadOnlyList<Task<ScriptRunOutcome>> RunBatch(IReadOnlyList<ScriptInvocation> invocations, CancellationToken ct = default)
{
ArgumentNullException.ThrowIfNull(invocations);
var outcomes = new Task<ScriptRunOutcome>[invocations.Count];
Dictionary<WorkerConnection, List<(int Index, BatchItem Item)>>? groups = null;
for (var i = 0; i < invocations.Count; i++)
{
var invocation = invocations[i];
ArgumentNullException.ThrowIfNull(invocation);
switch (TryPrepareBatchItem(invocation, out var item, out var answered))
{
case BatchPreparation.Submit:
groups ??= [];
if (!groups.TryGetValue(item!.Connection, out var group)) groups[item.Connection] = group = [];
group.Add((i, item));
break;
case BatchPreparation.Answered:
outcomes[i] = Task.FromResult(answered!);
break;
default:
outcomes[i] = RunAsync(invocation, ct);
break;
}
}
if (groups is null) return outcomes;
foreach (var (connection, group) in groups)
{
var submissions = new BatchSubmission[group.Count];
for (var j = 0; j < group.Count; j++)
{
var item = group[j].Item;
submissions[j] = new BatchSubmission(item.Key, BuildRequest(item.Key, item.Entry, item.Invocation),
item.Invocation.Timeout, _worker.EffectiveHardGrace(item.Invocation.Timeout), SlotRelease(item.Key, item.State, null));
}
var submitted = Stopwatch.GetTimestamp();
BatchSubmittedForTest?.Invoke(submissions.Length);
var pendings = connection.SubmitBatch(submissions);
for (var j = 0; j < group.Count; j++)
{
var (index, item) = group[j];
outcomes[index] = AwaitSubmittedAsync(item.Key, item.State, item.Invocation, item.Shard, connection, pendings[j], submitted, ct);
}
}
return outcomes;
}
/// <summary>
/// The synchronous half of a batch item: everything <see cref="RunAsync"/> decides before it submits, for the case
/// where nothing has to be waited for. Takes the key's single-flight slot when it returns
/// <see cref="BatchPreparation.Submit"/>; never when it returns anything else.
/// </summary>
private BatchPreparation TryPrepareBatchItem(ScriptInvocation invocation, out BatchItem? item, out ScriptRunOutcome? answered)
{
item = null;
answered = null;
if (Volatile.Read(ref _disposed) != 0) return BatchPreparation.Full;
var key = InProcessScriptRunner.ScopedKey(invocation.Kind, invocation.Key);
var state = Admit(key, invocation.Kind, invocation.Source);
if (state.QuarantineReason is not null || _compileFailures.ContainsKey(state.FailureKey)) return BatchPreparation.Full;
var shard = state.Suspect ? _isolation : _shards[ShardIndex(key, _shards.Length)];
if (shard.IsolationGate is not null || shard.Current is not { } connection) return BatchPreparation.Full;
if (connection.TryGetLoaded(key, state.ShaHex, SlotOrder(state, invocation.Inputs)) is not { } entry) return BatchPreparation.Full;
if (Interlocked.CompareExchange(ref state.InFlight, 1, 0) != 0)
{
ScriptRunGuard.RefusedCounter.Add(1, new KeyValuePair<string, object?>("script", key));
answered = ScriptRunOutcome.Failed(ScriptRunGuard.PreviousRunStillRunningReason);
return BatchPreparation.Answered;
}
state.ScriptLogger = invocation.ScriptLogger;
NoteLoad(entry);
item = new BatchItem(key, state, invocation, shard, connection, entry);
return BatchPreparation.Submit;
}
/// <summary>
/// The slot release of one submitted evaluation: from submit on, the key's single-flight slot (and the isolation
/// gate) belong to the evaluation itself and are released when the worker answers or dies — not when the caller
/// stops waiting at the soft deadline. That keeps a key single-flight between T and T + G, and the isolation
/// worker strictly one-at-a-time. The release runs inside the connection BEFORE the caller's await resumes, so a
/// caller that awaited one run and starts the next is never refused by the run it already has (the §8 harness found
/// 32 of 200 refused when the release was an unordered continuation). On a worker death, attribution (strike,
/// quarantine, suspect) has already run by then.
/// </summary>
private Action SlotRelease(string key, KeyState state, SemaphoreSlim? heldGate)
{
var released = 0;
return () =>
{
if (Interlocked.Exchange(ref released, 1) != 0) return;
SlotReleasingForTest?.Invoke(key);
Interlocked.Exchange(ref state.InFlight, 0);
heldGate?.Release();
};
}
/// <summary>Waits for one submitted evaluation and turns its result into the caller's outcome.</summary>
private async Task<ScriptRunOutcome> AwaitSubmittedAsync(string key, KeyState state, ScriptInvocation invocation,
WorkerShard shard, WorkerConnection connection, PendingEval pending, long submitted, CancellationToken ct)
{
try
{
var result = ct.CanBeCanceled
? await pending.Outcome.WaitAsync(ct).ConfigureAwait(false)
: await pending.Outcome.ConfigureAwait(false);
@@ -452,19 +583,6 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
{
return ScriptRunOutcome.Failed("evaluation cancelled");
}
finally
{
if (!handedOff)
{
Interlocked.Exchange(ref state.InFlight, 0);
gate?.Release();
}
}
// Circuit open (design §5.3.6): answer at once instead of making every trigger wait out the grace. A zero grace
// still kicks a start that is due, so the shards keep retrying at the backoff cap. Only computed when the
// shard has no live worker (or for the isolation gate).
TimeSpan Grace() => State == ScriptRunnerState.Unavailable ? TimeSpan.Zero : _worker.UnavailableGrace;
}
/// <summary>
@@ -816,6 +934,21 @@ public sealed class OutOfProcessScriptRunner : IScriptRunner, IAsyncDisposable,
=> LiveRunners.Keys.SelectMany(r => r.AllShards.Select(s => s.Status()))
.Select(s => new Measurement<int>(value(s), new KeyValuePair<string, object?>("shard", s.Label)));
private enum BatchPreparation
{
/// <summary>Run it through the full <see cref="RunAsync"/> path on its own.</summary>
Full,
/// <summary>Answered without running (refused).</summary>
Answered,
/// <summary>Submit it with the batch; its slot is taken.</summary>
Submit,
}
private sealed record BatchItem(string Key, KeyState State, ScriptInvocation Invocation, WorkerShard Shard,
WorkerConnection Connection, WorkerConnection.LoadEntry Entry);
private sealed class KeyState(string source, ScriptContextKind kind)
{
private byte[]? _sha;
@@ -53,6 +53,14 @@ internal sealed record WorkerExit(WorkerExitReason Reason, string? CulpritKey, i
}
}
/// <summary>One evaluation of a <see cref="WorkerConnection.SubmitBatch"/>.</summary>
/// <param name="Key">The scoped key.</param>
/// <param name="Request">The request (its id is assigned on submit).</param>
/// <param name="SoftTimeout">T.</param>
/// <param name="HardGrace">G.</param>
/// <param name="OnFinished">Runs once when the worker has answered or ended, before the caller resumes (the slot release).</param>
internal sealed record BatchSubmission(string Key, EvalRequest Request, TimeSpan SoftTimeout, TimeSpan HardGrace, Action? OnFinished);
/// <summary>What the host needs to launch one worker.</summary>
/// <param name="Shard">The shard label (<c>0</c>, <c>1</c>, … or <c>iso</c>).</param>
/// <param name="WorkerPath">The worker entry assembly (<c>.dll</c>, run through the dotnet muxer) or apphost.</param>
@@ -522,6 +530,51 @@ internal sealed class WorkerConnection : IAsyncDisposable
return pending;
}
/// <summary>
/// Submits several evaluations as ONE <see cref="EvalBatch"/> frame (lmxopcua#780 batch submit, design §4.5). Each
/// item gets its own deadlines and slot release exactly as through <see cref="Submit(string, EvalRequest, TimeSpan?, TimeSpan, Action?)"/>;
/// an item the worker has to queue behind busy evaluation threads has its clock stopped until it starts.
/// </summary>
/// <param name="items">The evaluations (their ids are assigned here).</param>
/// <returns>One pending evaluation per item, in order.</returns>
public PendingEval[] SubmitBatch(IReadOnlyList<BatchSubmission> items)
{
ArgumentNullException.ThrowIfNull(items);
var pendings = new PendingEval[items.Count];
var batch = new EvalBatch();
Interlocked.Exchange(ref _lastUsedTimestamp, Stopwatch.GetTimestamp());
for (var i = 0; i < items.Count; i++)
{
var item = items[i];
item.Request.EvalId = (ulong)Interlocked.Increment(ref _nextEvalId);
var pending = new PendingEval(item.Key, item.Request.EvalId, item.OnFinished, _dispatcher);
pending.Arm(item.SoftTimeout, item.HardGrace, OnHardDeadline);
_pending[item.Request.EvalId] = pending;
batch.Items.Add(item.Request);
pendings[i] = pending;
}
if (items.Count > 0 && (!Send(new Envelope { Eval = batch }) || _exited.Task.IsCompleted))
{
foreach (var pending in pendings)
if (_pending.TryRemove(pending.EvalId, out _)) pending.Complete(null);
}
return pendings;
}
/// <summary>
/// The compiled form of <paramref name="key"/> in this worker when it is loaded at <paramref name="sourceHashHex"/>
/// with <paramref name="inputPaths"/> and compiled cleanly — the batch fast path, which never waits for a load.
/// </summary>
/// <param name="key">The scoped key.</param>
/// <param name="sourceHashHex">The source hash (hex).</param>
/// <param name="inputPaths">The slot order.</param>
/// <returns>The entry, or null when the key needs a (re)load or its load failed.</returns>
public LoadEntry? TryGetLoaded(string key, string sourceHashHex, IReadOnlyList<string> inputPaths)
=> _loaded.TryGetValue(key, out var existing) && existing.SourceHash == sourceHashHex && existing.SameSlots(inputPaths)
&& existing.Ready.IsCompleted && existing.Outcome is { Ok: true } && !existing.TimedOut
? existing
: null;
private void OnHardDeadline(PendingEval pending)
{
_log.Warning("Script {ScriptKey} exceeded its hard deadline ({Deadline} ms); killing ScriptWorker[{Shard}] pid {Pid}",
@@ -677,8 +730,20 @@ internal sealed class WorkerConnection : IAsyncDisposable
// An eval id with no outstanding request is a protocol violation (design §6 "what the host
// trusts"), except for one that was already completed by a kill race — ignored then.
if (_pending.TryRemove(result.EvalId, out var pending)) pending.Complete(result);
else if (result.EvalId == 0 || result.EvalId > (ulong)Interlocked.Read(ref _nextEvalId))
throw new ScriptWorkerProtocolException($"result for unknown eval id {result.EvalId}");
else CheckIssued(result.EvalId, "result");
}
// Batch submit (lmxopcua#780): a run that waited behind busy evaluation threads has its deadlines
// restarted when it actually starts, so it is never timed out, struck or killed for a sibling's time.
// The two notices are unordered with respect to each other (see PendingEval.Start).
foreach (var id in frame.Results.StartedEvalIds)
{
if (_pending.TryGetValue(id, out var started)) started.Start();
else CheckIssued(id, "start notice");
}
foreach (var id in frame.Results.DeferredEvalIds)
{
if (_pending.TryGetValue(id, out var deferred)) deferred.Defer();
else CheckIssued(id, "deferral notice");
}
break;
case Envelope.BodyOneofCase.LoadStarted:
@@ -710,6 +775,13 @@ internal sealed class WorkerConnection : IAsyncDisposable
}
}
/// <summary>An id the host never issued is a protocol violation; one it issued and already completed is a benign race.</summary>
private void CheckIssued(ulong evalId, string what)
{
if (evalId == 0 || evalId > (ulong)Interlocked.Read(ref _nextEvalId))
throw new ScriptWorkerProtocolException($"{what} for unknown eval id {evalId}");
}
private async Task PipeEndedAsync(WorkerExitReason? fault)
{
// The pipe is gone: the worker is dead or dying. Make sure, then report.
@@ -917,6 +989,11 @@ internal sealed class WorkerConnection : IAsyncDisposable
/// <c>WaitAsync(timeout)</c> timer, the hard-deadline <c>CancellationTokenSource</c> and their continuations were a
/// measurable share of the host's CPU): it fires at the soft deadline T and is re-armed for the grace G.
/// </summary>
/// <remarks>
/// The deadlines are absolute timestamps the timer callback re-checks, so a deferral (the worker queued the run behind
/// busy evaluation threads, lmxopcua#780 batch submit) can stop the clock and a start notice can restart it — the full
/// T, then G, from the start — without a stale tick firing early.
/// </remarks>
internal sealed class PendingEval
{
/// <summary>What <see cref="Outcome"/> yields when the soft deadline passed first. Never on the wire.</summary>
@@ -926,9 +1003,15 @@ internal sealed class PendingEval
private readonly TaskCompletionSource<EvalResult?> _outcome;
private readonly ResultDispatcher? _dispatcher;
private readonly Action? _onFinished;
private readonly object _clock = new();
private Timer? _timer;
private Action<PendingEval>? _onHardDeadline;
private TimeSpan _soft;
private TimeSpan _grace;
private long _softDue;
private long _hardDue;
private bool _deferred;
private bool _started;
private int _phase;
private int _done;
@@ -954,6 +1037,12 @@ internal sealed class PendingEval
/// <summary>T + G, for the kill log line.</summary>
public TimeSpan HardDeadline { get; private set; }
/// <summary>Whether the worker reported this run as queued and has not reported it started (tests / diagnostics).</summary>
internal bool IsDeferred
{
get { lock (_clock) return _deferred; }
}
/// <summary>The worker's result, or null when the worker ended first.</summary>
public Task<EvalResult?> Completion => _completion.Task;
@@ -966,27 +1055,96 @@ internal sealed class PendingEval
/// <summary>Starts the deadline timer. Called once, before the request is sent.</summary>
internal void Arm(TimeSpan soft, TimeSpan grace, Action<PendingEval> onHardDeadline)
{
_soft = soft;
_grace = grace;
_onHardDeadline = onHardDeadline;
HardDeadline = soft + grace;
var now = Stopwatch.GetTimestamp();
_softDue = now + Ticks(soft);
_hardDue = _softDue + Ticks(grace);
// Created disarmed and assigned before it can fire, so the soft tick can always re-arm it for the grace.
_timer = new Timer(static s => ((PendingEval)s!).OnTimer(), this, Timeout.Infinite, Timeout.Infinite);
_timer.Change(soft, Timeout.InfiniteTimeSpan);
}
/// <summary>
/// The worker queued this run behind busy evaluation threads: stop the clock until it starts (it is not running,
/// so it cannot overrun). Ignored once a start notice has arrived — the two are unordered on the wire.
/// </summary>
internal void Defer()
{
lock (_clock)
{
if (_timer is null || _started || _deferred || Volatile.Read(ref _done) != 0) return;
_deferred = true;
TryChange(Timeout.InfiniteTimeSpan);
}
}
/// <summary>
/// The worker started this run: a deferred run's deadlines restart from now (T, or T + G when its soft deadline
/// had already been reported). A run that was never deferred keeps the clock it has had since submit.
/// </summary>
internal void Start()
{
lock (_clock)
{
_started = true;
if (_timer is null || !_deferred || Volatile.Read(ref _done) != 0) return;
_deferred = false;
var now = Stopwatch.GetTimestamp();
if (Volatile.Read(ref _phase) == 0)
{
_softDue = now + Ticks(_soft);
_hardDue = _softDue + Ticks(_grace);
TryChange(_soft);
}
else
{
// Its caller already has the soft timeout (a deferral notice that arrived after T): the run still gets
// the whole T + G from its start before it can be blamed for a kill.
_hardDue = now + Ticks(_soft + _grace);
TryChange(_soft + _grace);
}
}
}
private void OnTimer()
{
if (Volatile.Read(ref _done) != 0) return;
if (Interlocked.CompareExchange(ref _phase, 1, 0) == 0)
bool soft = false, hard = false;
lock (_clock)
{
_outcome.TrySetResult(SoftTimedOut);
try { _timer?.Change(_grace, Timeout.InfiniteTimeSpan); }
catch (ObjectDisposedException) { /* answered meanwhile */ }
return;
if (_deferred || Volatile.Read(ref _done) != 0) return;
var now = Stopwatch.GetTimestamp();
if (Volatile.Read(ref _phase) == 0)
{
// A tick armed before a deferral/start can land early: re-arm for what is left.
if (now < _softDue) { TryChange(Stopwatch.GetElapsedTime(now, _softDue)); return; }
Interlocked.Exchange(ref _phase, 1);
soft = true;
TryChange(_hardDue > now ? Stopwatch.GetElapsedTime(now, _hardDue) : TimeSpan.Zero);
}
else if (Volatile.Read(ref _phase) == 1)
{
if (now < _hardDue) { TryChange(Stopwatch.GetElapsedTime(now, _hardDue)); return; }
Interlocked.Exchange(ref _phase, 2);
hard = true;
}
}
if (Interlocked.CompareExchange(ref _phase, 2, 1) == 1 && Volatile.Read(ref _done) == 0) _onHardDeadline?.Invoke(this);
// Outside the lock: a caller with a dispatcher resumes synchronously inside TrySetResult.
if (soft) _outcome.TrySetResult(SoftTimedOut);
if (hard && Volatile.Read(ref _done) == 0) _onHardDeadline?.Invoke(this);
}
private void TryChange(TimeSpan due)
{
try { _timer?.Change(due < TimeSpan.Zero ? TimeSpan.Zero : due, Timeout.InfiniteTimeSpan); }
catch (ObjectDisposedException) { /* answered meanwhile */ }
}
private static long Ticks(TimeSpan span) => (long)(span.TotalSeconds * Stopwatch.Frequency);
/// <summary>
/// The worker answered (or ended, with null): cancel the deadlines and release the key's slot HERE, on the
/// reader, then hand the caller's resumption to the dispatcher (or the pool).
@@ -134,8 +134,17 @@ message EvalResult {
uint32 run_micros = 6;
}
// Per-item deadlines are exact for queued work too (lmxopcua#780 batch submit): the host arms an evaluation's soft and
// hard deadlines when it submits, which is when the run starts as long as the worker has an idle evaluation thread. An
// evaluation that arrives while every thread is busy (a batch larger than the scheduler, or work queued behind a
// runaway) is reported in `deferred_eval_ids`; the host then stops its clock, and restarts it — the full T, then G —
// when the evaluation appears in `started_eval_ids`, which an evaluation thread sends alongside the result of the run it
// just finished. So a run waiting behind a sibling is never timed out, struck or killed for the sibling's time.
// Either list may be the only content of a frame. A host that predates these fields ignores them.
message EvalResults {
repeated EvalResult items = 1;
repeated uint64 started_eval_ids = 2;
repeated uint64 deferred_eval_ids = 3;
}
message ScriptError {
@@ -233,7 +233,10 @@ public sealed class ScriptWorkerFrameWriter
break;
case Envelope.BodyOneofCase.Results:
if (evals is not null) { Append(new Envelope { Eval = evals }); evals = null; }
(results ??= new EvalResults()).Items.AddRange(envelope.Results.Items);
results ??= new EvalResults();
results.Items.AddRange(envelope.Results.Items);
results.StartedEvalIds.AddRange(envelope.Results.StartedEvalIds);
results.DeferredEvalIds.AddRange(envelope.Results.DeferredEvalIds);
break;
default:
if (evals is not null) { Append(new Envelope { Eval = evals }); evals = null; }
@@ -28,6 +28,31 @@ public interface IScriptRunner
/// <returns>The outcome; never faults for a script failure.</returns>
Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default);
/// <summary>
/// Submits several evaluations at once (design §4.5): the out-of-process runner sends every one that is ready to
/// run as ONE request frame per worker, instead of one frame and one submission per evaluation. Results are
/// per item and streamed — each task completes when its own evaluation does, so a slow item never holds a fast
/// sibling's result — and every item keeps exactly the semantics of <see cref="RunAsync"/>: its own budget, its
/// own single-flight slot, and its own blame (a runaway item is killed and struck alone; an item queued behind
/// it is neither timed out nor struck for its time).
/// </summary>
/// <remarks>
/// The default implementation — what <see cref="InProcessScriptRunner"/> uses — is <see cref="RunAsync"/> per
/// item, started together. A key that appears twice in one batch is refused for its second item exactly as two
/// concurrent <see cref="RunAsync"/> calls would be; callers that may repeat a key split the batch there.
/// Never throws for script faults; the returned tasks never fault.
/// </remarks>
/// <param name="invocations">What to run, in order.</param>
/// <param name="ct">Caller cancellation (shutdown); wins over every item's budget.</param>
/// <returns>One task per invocation, in the same order.</returns>
IReadOnlyList<Task<ScriptRunOutcome>> RunBatch(IReadOnlyList<ScriptInvocation> invocations, CancellationToken ct = default)
{
ArgumentNullException.ThrowIfNull(invocations);
var outcomes = new Task<ScriptRunOutcome>[invocations.Count];
for (var i = 0; i < outcomes.Length; i++) outcomes[i] = RunAsync(invocations[i], ct);
return outcomes;
}
/// <summary>
/// The deploy boundary for one seam: lifts every quarantine and forgets every remembered compile failure of
/// <paramref name="kind"/>, and drops its compiled scripts.
@@ -50,7 +75,7 @@ public enum ScriptContextKind
/// <summary>A Calculation tag: <c>VirtualTagContext</c> → <c>object?</c>.</summary>
Calculation = 2,
/// <summary>A scripted-alarm predicate: <c>AlarmPredicateContext</c> → <c>bool</c> (not run out of process yet).</summary>
/// <summary>A scripted-alarm predicate: <c>AlarmPredicateContext</c> → <c>bool</c>.</summary>
AlarmPredicate = 3,
}
@@ -142,3 +167,16 @@ public static class ScriptRunOutcomeExtensions
};
}
}
/// <summary>
/// Which <see cref="IScriptRunner"/> each script seam uses on this node (lmxopcua#780), for components that cannot
/// reference the Host's selection directly (the Runtime's scripted-alarm host). Null for a seam means its in-process
/// guard path.
/// </summary>
public interface IScriptRunnerProvider
{
/// <summary>The runner <paramref name="seam"/> must use, or null for its in-process path.</summary>
/// <param name="seam">The seam.</param>
/// <returns>The runner, or null.</returns>
IScriptRunner? ForSeam(ScriptSeam seam);
}
@@ -0,0 +1,106 @@
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting;
/// <summary>
/// Opportunistic batching for a seam whose evaluations are submitted independently — one virtual-tag actor each —
/// but often become ready together (one upstream burst fans out to many tags): evaluations submitted while another
/// submission is in progress are handed to <see cref="IScriptRunner.RunBatch"/> together, in arrival order, by the
/// submitter already holding the flush (lmxopcua#780, design §4.5).
/// </summary>
/// <remarks>
/// <para>There is no timer: an evaluation on an idle seam is submitted at once by its own caller, so batching never
/// adds latency. Per-item semantics are <see cref="IScriptRunner.RunBatch"/>'s — each item keeps its own budget, slot
/// and blame.</para>
/// <para>Each caller's task completes synchronously on the thread that completes its run (the runner's result
/// dispatcher), exactly as awaiting <see cref="IScriptRunner.RunAsync"/> directly does, so there is no extra thread
/// hop.</para>
/// <para>A key submitted twice concurrently is refused for the second exactly as two concurrent
/// <see cref="IScriptRunner.RunAsync"/> calls would be; the virtual-tag actor never has two evaluations in flight.</para>
/// </remarks>
public sealed class ScriptBatchCoalescer
{
private readonly IScriptRunner _runner;
private readonly object _lock = new();
private List<(ScriptInvocation Invocation, TaskCompletionSource<ScriptRunOutcome> Done)> _queue = [];
private bool _flushing;
/// <summary>Initializes the coalescer.</summary>
/// <param name="runner">The runner every batch goes to.</param>
public ScriptBatchCoalescer(IScriptRunner runner)
{
_runner = runner ?? throw new ArgumentNullException(nameof(runner));
}
/// <summary>Test hook: runs once per <see cref="IScriptRunner.RunBatch"/> call, with the batch size.</summary>
internal Action<int>? BatchSubmittedForTest { get; set; }
/// <summary>Submits one evaluation, together with any submitted concurrently.</summary>
/// <param name="invocation">What to run.</param>
/// <returns>The outcome; never faults.</returns>
public Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation)
{
ArgumentNullException.ThrowIfNull(invocation);
// Synchronous continuations: the caller resumes where its run completed, as with a direct await of RunAsync.
var done = new TaskCompletionSource<ScriptRunOutcome>(TaskCreationOptions.None);
lock (_lock)
{
_queue.Add((invocation, done));
if (_flushing) return done.Task;
_flushing = true;
}
Flush();
return done.Task;
}
private void Flush()
{
while (true)
{
List<(ScriptInvocation Invocation, TaskCompletionSource<ScriptRunOutcome> Done)> batch;
lock (_lock)
{
if (_queue.Count == 0)
{
_flushing = false;
return;
}
batch = _queue;
_queue = [];
}
IReadOnlyList<Task<ScriptRunOutcome>> outcomes;
try
{
var invocations = new ScriptInvocation[batch.Count];
for (var i = 0; i < invocations.Length; i++) invocations[i] = batch[i].Invocation;
BatchSubmittedForTest?.Invoke(invocations.Length);
outcomes = _runner.RunBatch(invocations);
}
catch (Exception ex)
{
// A defect in the runner must not strand the callers (nor the flush flag).
foreach (var (_, done) in batch) done.TrySetResult(ScriptRunOutcome.Failed($"script runner threw: {ex.Message}"));
continue;
}
for (var i = 0; i < batch.Count; i++) Link(outcomes[i], batch[i].Done);
}
}
private static void Link(Task<ScriptRunOutcome> outcome, TaskCompletionSource<ScriptRunOutcome> done)
{
if (outcome.IsCompleted)
{
Settle(outcome, done);
return;
}
outcome.ContinueWith(static (t, s) => Settle(t, (TaskCompletionSource<ScriptRunOutcome>)s!), done,
CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
}
private static void Settle(Task<ScriptRunOutcome> outcome, TaskCompletionSource<ScriptRunOutcome> done)
{
if (outcome.IsCompletedSuccessfully) done.TrySetResult(outcome.Result);
else done.TrySetResult(ScriptRunOutcome.Failed(
$"script runner threw: {outcome.Exception?.GetBaseException().Message ?? "cancelled"}"));
}
}
@@ -32,7 +32,10 @@ public enum ScriptSeam
/// <summary>Calculation tags (<c>CalculationEvaluator</c>).</summary>
Calculation = 1,
/// <summary>Scripted-alarm predicates (<c>ScriptedAlarmEngine</c>). Last in the rollout (design §14 #10).</summary>
/// <summary>
/// Scripted-alarm predicates (<c>ScriptedAlarmEngine</c>). Last in the rollout (design §14 #10): out of process only
/// when <c>Scripting:Runner:Seams</c> lists it — an unset list does not select it.
/// </summary>
ScriptedAlarm = 2,
}
@@ -80,7 +80,8 @@ public sealed class ScriptingOptions
/// <summary>
/// The seams <see cref="Runner"/> applies to when it is not <see cref="ScriptRunnerMode.InProcess"/>
/// (<c>Scripting:Runner:Seams</c>, e.g. <c>["Calculation"]</c>). Null or empty means all three. Bound by
/// (<c>Scripting:Runner:Seams</c>, e.g. <c>["Calculation"]</c>). Null or empty means VirtualTag and Calculation:
/// scripted alarms run out of process only when listed explicitly (design §14 #10, alarms last). Bound by
/// <c>ScriptingOptionsBinding</c> from the documented key, because <c>Scripting:Runner</c> is itself a value.
/// </summary>
public IReadOnlyList<ScriptSeam>? RunnerSeams { get; set; }
@@ -91,10 +92,10 @@ public sealed class ScriptingOptions
/// <summary>Whether <paramref name="seam"/> is selected for out-of-process execution by these options.</summary>
/// <param name="seam">The seam.</param>
/// <returns><c>true</c> when <see cref="Runner"/> is <see cref="ScriptRunnerMode.OutOfProcess"/> and the seam is in
/// <see cref="RunnerSeams"/> (or that list is empty).</returns>
/// <see cref="RunnerSeams"/> (or that list is empty and the seam is not <see cref="ScriptSeam.ScriptedAlarm"/>).</returns>
public bool IsOutOfProcess(ScriptSeam seam)
=> Runner == ScriptRunnerMode.OutOfProcess
&& (RunnerSeams is not { Count: > 0 } seams || seams.Contains(seam));
&& (RunnerSeams is { Count: > 0 } seams ? seams.Contains(seam) : seam != ScriptSeam.ScriptedAlarm);
/// <summary>Upper bound <see cref="Sanitized"/> applies to <see cref="StuckOrphanQuarantineFactor"/>.</summary>
public const double MaxStuckOrphanQuarantineFactor = 1000;
@@ -26,6 +26,8 @@
<ItemGroup>
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.Core.Scripting.Tests"/>
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests"/>
<InternalsVisibleTo Include="ZB.MOM.WW.OtOpcUa.Host.Tests"/>
</ItemGroup>
</Project>
@@ -522,6 +522,14 @@ public sealed class CalculationDriver : IDriver, IDependencyConsumer, ISubscriba
/// false timeouts under pool starvation).</summary>
private async Task RunEvaluationPumpAsync()
{
// lmxopcua#780 (design §4.5): with an out-of-process runner a run costs a pipe round trip, so the pump submits
// everything queued as one batch and publishes in dequeue order. In process it stays one at a time, unchanged.
if (_evaluator.SubmitsBatches)
{
await RunEvaluationBatchPumpAsync().ConfigureAwait(false);
return;
}
while (true)
{
PendingEvaluation next;
@@ -557,6 +565,73 @@ public sealed class CalculationDriver : IDriver, IDependencyConsumer, ISubscriba
}
}
/// <summary>
/// The pump on the out-of-process path (lmxopcua#780, design §4.5): dequeue everything pending — each entry's
/// dependency snapshot was captured at enqueue — submit it as ONE batch, and publish in dequeue order. A calc tag
/// queued more than once (a burst of changes) ends the batch at its second entry, so every queued evaluation still
/// runs, in order, against the snapshot that triggered it, exactly as the one-at-a-time pump would run them.
/// </summary>
private async Task RunEvaluationBatchPumpAsync()
{
while (true)
{
List<PendingEvaluation> batch;
lock (_lock)
{
if (!_running) _pending.Clear();
if (_pending.Count == 0)
{
_pumpScheduled = false;
return;
}
batch = [];
var inBatch = new HashSet<string>(StringComparer.Ordinal);
while (_pending.Count > 0 && inBatch.Add(_pending.Peek().Definition.RawPath)) batch.Add(_pending.Dequeue());
}
if (!await TryEnterEvalGateAsync().ConfigureAwait(false)) return; // disposed
try
{
await RunEvaluationBatchAsync(batch).ConfigureAwait(false);
}
catch (Exception ex)
{
// Defect path only (the evaluator never throws for script faults); must not strand the queue.
_logger.LogError(ex, "Calculation {Driver}: batch evaluation of {Count} tag(s) failed", _driverInstanceId, batch.Count);
}
finally
{
_evalGate.Release();
}
}
}
/// <summary>Runs one batch and publishes every outcome in dequeue order. Under <see cref="_evalGate"/>, not <see cref="_lock"/>.</summary>
/// <param name="batch">The prepared evaluations; no calc tag twice.</param>
private async Task RunEvaluationBatchAsync(IReadOnlyList<PendingEvaluation> batch)
{
// Hold-on-Bad entries never reach the runner; the rest are submitted together.
var requests = new List<(string, string, IReadOnlyDictionary<string, DataValueSnapshot>)>(batch.Count);
var slot = new int[batch.Count];
for (var i = 0; i < batch.Count; i++)
{
if (!InputQuality.IsUsable(batch[i].WorstInputStatus)) { slot[i] = -1; continue; }
slot[i] = requests.Count;
requests.Add((batch[i].Definition.RawPath, batch[i].Definition.ScriptSource, batch[i].Dependencies));
}
var results = _evaluator.EvaluateBatch(requests, _evalCts.Token);
BatchSubmittedForTest?.Invoke(requests.Count);
for (var i = 0; i < batch.Count; i++)
{
if (slot[i] < 0) HoldOnBadInput(batch[i]);
else ApplyEvaluationResult(batch[i], await results[slot[i]].ConfigureAwait(false));
}
}
/// <summary>Test hook: the number of compiled evaluations each batch pass submitted.</summary>
internal Action<int>? BatchSubmittedForTest { get; set; }
/// <summary>Timer-trigger entry point — prepares and evaluates a single calc tag (the scheduler's
/// per-group in-flight guard is what bounds outstanding timer work; it stays set until this task
/// completes).</summary>
@@ -610,16 +685,7 @@ public sealed class CalculationDriver : IDriver, IDependencyConsumer, ISubscriba
// sub-code. Uncertain IS usable: the script runs and the output inherits the Uncertain code.
if (!InputQuality.IsUsable(pending.WorstInputStatus))
{
(object? Value, uint StatusCode)? badPublish;
lock (_lock)
{
if (!_stateByRawPath.TryGetValue(def.RawPath, out var badState)) return;
badPublish = MarkInputsBad(badState, pending.WorstInputStatus, pending.TimestampUtc);
}
// No script log: an input dying is not a script fault, and the script log is the per-failure
// SCRIPT diagnostic. Here the status code is the diagnostic.
if (badPublish is { } bad) Publish(def.RawPath, bad.Value, bad.StatusCode, pending.TimestampUtc);
HoldOnBadInput(pending);
return;
}
@@ -627,7 +693,32 @@ public sealed class CalculationDriver : IDriver, IDependencyConsumer, ISubscriba
// then the input's real ones rather than a fabricated Good stamped at evaluation time.
var result = await _evaluator.EvaluateAsync(def.RawPath, def.ScriptSource, pending.Dependencies, _evalCts.Token)
.ConfigureAwait(false);
ApplyEvaluationResult(pending, result);
}
/// <summary>A Bad input holds the script: publish the input's own status (once per transition), no script log.</summary>
/// <param name="pending">The prepared evaluation whose worst input is Bad.</param>
private void HoldOnBadInput(PendingEvaluation pending)
{
var def = pending.Definition;
(object? Value, uint StatusCode)? badPublish;
lock (_lock)
{
if (!_stateByRawPath.TryGetValue(def.RawPath, out var badState)) return;
badPublish = MarkInputsBad(badState, pending.WorstInputStatus, pending.TimestampUtc);
}
// No script log: an input dying is not a script fault, and the script log is the per-failure
// SCRIPT diagnostic. Here the status code is the diagnostic.
if (badPublish is { } bad) Publish(def.RawPath, bad.Value, bad.StatusCode, pending.TimestampUtc);
}
/// <summary>Record one evaluation's outcome and publish it. Under <see cref="_evalGate"/>, not <see cref="_lock"/>.</summary>
/// <param name="pending">The prepared evaluation.</param>
/// <param name="result">Its outcome.</param>
private void ApplyEvaluationResult(PendingEvaluation pending, VirtualTagEvalResult result)
{
var def = pending.Definition;
string? failureToLog = null;
(object? Value, uint StatusCode)? toPublish;
lock (_lock)
@@ -283,6 +283,62 @@ public sealed class CalculationEvaluator : IScriptCacheOwner, IScriptQuarantineO
}
}
/// <summary>
/// Whether compiled runs go to an <see cref="IScriptRunner"/> (the out-of-process seam), so the driver's pump can
/// submit everything it has dequeued as one batch (lmxopcua#780, design §4.5). False on the in-process path, whose
/// pump stays one evaluation at a time, unchanged.
/// </summary>
public bool SubmitsBatches => _runner is not null && !_disposed;
/// <summary>
/// Evaluates several calc tags at once (design §4.5): the passthrough fast path and the trivial failures are answered
/// inline exactly as <see cref="EvaluateAsync"/> answers them, and every compiled run is handed to the runner in ONE
/// <see cref="IScriptRunner.RunBatch"/>. Results are per item; each item keeps <see cref="EvaluateAsync"/>'s
/// outcome, budget and blame. Without a runner it is <see cref="EvaluateAsync"/> per item, in order.
/// </summary>
/// <param name="requests">The evaluations: calc-tag id, script source, dependency snapshots. A calc tag must not appear twice.</param>
/// <param name="ct">Caller cancellation (driver shutdown).</param>
/// <returns>One task per request, in order; none faults.</returns>
public IReadOnlyList<Task<VirtualTagEvalResult>> EvaluateBatch(
IReadOnlyList<(string CalcTagId, string ScriptSource, IReadOnlyDictionary<string, DataValueSnapshot> Dependencies)> requests,
CancellationToken ct = default)
{
ArgumentNullException.ThrowIfNull(requests);
var results = new Task<VirtualTagEvalResult>[requests.Count];
if (_runner is null || _disposed)
{
for (var i = 0; i < requests.Count; i++)
results[i] = EvaluateAsync(requests[i].CalcTagId, requests[i].ScriptSource, requests[i].Dependencies, ct);
return results;
}
List<ScriptInvocation>? invocations = null;
List<int>? indexes = null;
for (var i = 0; i < requests.Count; i++)
{
var (id, source, deps) = requests[i];
// Exactly the fast answers EvaluateAsync gives before it reaches the runner.
if (string.IsNullOrWhiteSpace(source) || PassthroughScript.TryMatch(source, out _))
{
results[i] = EvaluateAsync(id, source, deps, ct);
continue;
}
var scriptLog = _scriptRoot
.ForContext(ScriptLoggerFactory.ScriptIdProperty, id)
.ForContext(ScriptLoggerFactory.VirtualTagIdProperty, id);
(invocations ??= []).Add(new ScriptInvocation(id, ScriptContextKind.Calculation, source, deps, _runTimeout, scriptLog));
(indexes ??= []).Add(i);
}
if (invocations is null) return results;
var outcomes = _runner.RunBatch(invocations, ct);
for (var j = 0; j < outcomes.Count; j++) results[indexes![j]] = ToResultAsync(outcomes[j]);
return results;
static async Task<VirtualTagEvalResult> ToResultAsync(Task<ScriptRunOutcome> outcome)
=> (await outcome.ConfigureAwait(false)).ToVirtualTagEvalResult();
}
/// <summary>Record a compile failure for <paramref name="sourceKey"/> and log it ONCE — the next trigger
/// of the same source returns the remembered reason without a Roslyn round-trip or a second Warning.</summary>
private VirtualTagEvalResult RememberCompileFailure(string sourceKey, string calcTagId, string reason, Exception ex, string what)
@@ -12,10 +12,18 @@ namespace ZB.MOM.WW.OtOpcUa.Host.Configuration;
/// no scripts (or an admin-only node) never launches one. Disposed with the host's service provider, which stops the
/// workers.
/// </summary>
public sealed class ScriptRunnerSelection : IAsyncDisposable, IDisposable
public sealed class ScriptRunnerSelection : IScriptRunnerProvider, IAsyncDisposable, IDisposable
{
/// <summary>Seams the out-of-process runner can serve in this build. Scripted alarms go last (design §14 #10).</summary>
public static readonly IReadOnlyList<ScriptSeam> SupportedOutOfProcessSeams = [ScriptSeam.VirtualTag, ScriptSeam.Calculation];
/// <summary>Seams the out-of-process runner can serve in this build.</summary>
public static readonly IReadOnlyList<ScriptSeam> SupportedOutOfProcessSeams =
[ScriptSeam.VirtualTag, ScriptSeam.Calculation, ScriptSeam.ScriptedAlarm];
/// <summary>
/// The seams an unset or empty <c>Scripting:Runner:Seams</c> selects. Scripted alarms are not among them: they go
/// out of process only when listed explicitly, last in the rollout (design §14 #10 — the alarm budget and the
/// engine refactor).
/// </summary>
public static readonly IReadOnlyList<ScriptSeam> DefaultOutOfProcessSeams = [ScriptSeam.VirtualTag, ScriptSeam.Calculation];
private readonly OutOfProcessScriptRunner? _runner;
private readonly HashSet<ScriptSeam> _seams;
@@ -94,5 +102,5 @@ public sealed class ScriptRunnerSelection : IAsyncDisposable, IDisposable
public void Dispose() => _runner?.Dispose();
private static IEnumerable<ScriptSeam> RequestedSeams(ScriptingOptions options)
=> options.RunnerSeams is { Count: > 0 } seams ? seams.Distinct() : Enum.GetValues<ScriptSeam>();
=> options.RunnerSeams is { Count: > 0 } seams ? seams.Distinct() : DefaultOutOfProcessSeams;
}
@@ -54,6 +54,7 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac
private readonly TimeSpan _runTimeout;
private readonly ScriptRunGuard _guard;
private readonly IScriptRunner? _runner;
private readonly ScriptBatchCoalescer? _batches;
private bool _disposed;
/// <summary>Initializes a new RoslynVirtualTagEvaluator with the given loggers and optional timeout.</summary>
@@ -79,6 +80,7 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac
_runTimeout = runTimeout ?? (scripting ?? new ScriptingOptions()).Sanitized(out _).RunTimeout;
_guard = new ScriptRunGuard(scripting, _scriptRoot);
_runner = scriptRunner;
_batches = scriptRunner is null ? null : new ScriptBatchCoalescer(scriptRunner);
}
/// <summary>The out-of-process runner this evaluator delegates compiled runs to, or null (in process).</summary>
@@ -87,6 +89,9 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac
/// <summary>The run guard every compiled evaluation goes through (diagnostics / tests).</summary>
public ScriptRunGuard RunGuard => _guard;
/// <summary>The coalescer the out-of-process path submits through, or null in process (tests).</summary>
internal ScriptBatchCoalescer? Batches => _batches;
// Value-only overload. Every dependency reaches the script stamped Good with an
// evaluation-time timestamp — bare values carry nothing better. The runtime calls the snapshot
// overload instead; this one remains for callers that genuinely hold only
@@ -161,9 +166,11 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac
// lmxopcua#780 — OutOfProcess: the compiled run goes to the pre-emptible worker pool, which compiles in the
// worker and owns timeouts, kills and quarantine for this seam. An Unavailable outcome (worker restarted, no
// worker ready) publishes BadResourceUnavailable; everything else keeps today's Bad contract.
if (_runner is not null)
// Every tag has its own actor, so tags an upstream burst made ready together arrive here concurrently; the
// coalescer submits those as one batch (design §4.5) and an evaluation on an idle seam at once.
if (_batches is not null)
{
var outcome = await _runner.RunAsync(new ScriptInvocation(
var outcome = await _batches.RunAsync(new ScriptInvocation(
virtualTagId, ScriptContextKind.VirtualTag, expression, readCache, _runTimeout, scriptLog)).ConfigureAwait(false);
return outcome.ToVirtualTagEvalResult();
}
@@ -385,6 +385,9 @@ if (hasDriver)
Log.Information("Scripting runner: {RunnerMode}", scriptingOptions.Runner);
builder.Services.AddSingleton(sp => ZB.MOM.WW.OtOpcUa.Host.Configuration.ScriptRunnerSelection.Create(
sp.GetRequiredService<ScriptingOptions>(), Log.Logger, sp.GetService<ScriptRootLogger>()?.Logger));
// The Runtime's scripted-alarm host resolves its runner through this (it cannot reference the Host).
builder.Services.AddSingleton<IScriptRunnerProvider>(sp =>
sp.GetRequiredService<ZB.MOM.WW.OtOpcUa.Host.Configuration.ScriptRunnerSelection>());
// lmxopcua#777: the ONE operator-write timeout (DriverWrites:Timeout) every write bound is derived from — the
// driver child's cancel + abandon, its queue budget, DriverHostActor's Ask and the inbound-write gateway's Ask.
@@ -206,6 +206,10 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
/// <summary>Node-wide script run-guard options for the scripted-alarm engine; null ⇒ compiled defaults.</summary>
private readonly ScriptingOptions? _scripting;
// lmxopcua#780 — the out-of-process runner when Scripting:Runner:Seams lists ScriptedAlarm; null keeps the engine's
// in-process guard path.
private readonly IScriptRunner? _scriptedAlarmRunner;
/// <summary>
/// Driver children this host has stopped (<see cref="StopChild"/>, an operator Restart) and is watching
/// until they terminate. A value burst from one of them is DROPPED in <see cref="ForwardBatchToMux"/>:
@@ -771,7 +775,8 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
ScriptingOptions? scripting = null,
NativeAlarmAckPreflightView? ackPreflightView = null,
DriverWriteBounds? writeBounds = null,
TimeProvider? writeClock = null) =>
TimeProvider? writeClock = null,
IScriptRunner? scriptedAlarmRunner = null) =>
// WARNING: this forwarding list is POSITIONAL, and Props.Create compiles it into an
// expression tree. Six IActorRef? parameters and several interface-typed ones mean a
// mis-ordered argument is usually type-compatible and therefore compiles clean, then binds
@@ -785,7 +790,7 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
alarmStateStore, telemetryHub, primaryRoleSource, historizationRoleView,
sharesHistorizationScopeWithLocal, applyWedgeTimeout, reprovisionBackoff, commsRecoveryAuditDelay,
sharesAlarmQueueWithLocal, applyOutcomeTimeout, captureWhilePrimaryUnreachable, historyUpdateRoleView,
scripting, ackPreflightView, writeBounds, writeClock));
scripting, ackPreflightView, writeBounds, writeClock, scriptedAlarmRunner));
/// <summary>Initializes a new DriverHostActor with the specified dependencies.</summary>
/// <param name="dbFactory">Database context factory for configuration database access.</param>
@@ -916,8 +921,10 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
ScriptingOptions? scripting = null,
NativeAlarmAckPreflightView? ackPreflightView = null,
DriverWriteBounds? writeBounds = null,
TimeProvider? writeClock = null)
TimeProvider? writeClock = null,
IScriptRunner? scriptedAlarmRunner = null)
{
_scriptedAlarmRunner = scriptedAlarmRunner;
_writeBounds = writeBounds ?? DriverWriteBounds.Default;
_writeClock = writeClock ?? TimeProvider.System;
_ackPreflightView = ackPreflightView;
@@ -1242,13 +1249,14 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
var store = _alarmStateStore;
var scriptLogger = _scriptRootLogger.Logger;
var scripting = _scripting;
var alarmRunner = _scriptedAlarmRunner;
_scriptedAlarmHost = Context.ActorOf(
ScriptedAlarmHostActor.Props(
_opcUaPublishActor, _dependencyMux,
() =>
{
var upstream = new DependencyMuxTagUpstreamSource();
return (upstream, CreateScriptedAlarmEngine(upstream, store, scriptLogger, scripting));
return (upstream, CreateScriptedAlarmEngine(upstream, store, scriptLogger, scripting, alarmRunner));
},
_localNode, telemetryHub: _telemetryHub),
"scripted-alarm-host");
@@ -1265,15 +1273,16 @@ public sealed class DriverHostActor : ReceiveActor, IWithTimers, IWithUnboundedS
/// <param name="store">The condition-state store.</param>
/// <param name="scriptLogger">The root script logger.</param>
/// <param name="scripting">The node-wide <c>Scripting</c> options; null ⇒ defaults.</param>
/// <param name="scriptRunner">The out-of-process runner for the ScriptedAlarm seam; null ⇒ in process.</param>
/// <returns>A new, unloaded engine.</returns>
internal static ScriptedAlarmEngine CreateScriptedAlarmEngine(
ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms.ITagUpstreamSource upstream, IAlarmStateStore store,
Serilog.ILogger scriptLogger, ScriptingOptions? scripting)
Serilog.ILogger scriptLogger, ScriptingOptions? scripting, IScriptRunner? scriptRunner = null)
{
var budget = (scripting ?? new ScriptingOptions()).Sanitized(out _).RunTimeout;
return new ScriptedAlarmEngine(
upstream, store, new ScriptLoggerFactory(scriptLogger), scriptLogger,
scriptTimeout: budget, scripting: scripting);
scriptTimeout: budget, scripting: scripting, scriptRunner: scriptRunner);
}
private void Bootstrap()
@@ -664,7 +664,11 @@ public static class ServiceCollectionExtensions
// The one configured write timeout (DriverWrites:Timeout, bound + validated in Program.cs) the
// children's write bounds AND this host's write Ask are derived from (lmxopcua#777). Unbound in
// test graphs ⇒ null ⇒ DriverWriteBounds.Default.
writeBounds: resolver.GetService<DriverWriteBounds>()),
writeBounds: resolver.GetService<DriverWriteBounds>(),
// lmxopcua#780: the out-of-process runner when Scripting:Runner:Seams lists ScriptedAlarm. Null under
// the InProcess default, when the seam is not listed, and in test graphs that register no provider.
scriptedAlarmRunner: resolver.GetService<ZB.MOM.WW.OtOpcUa.Core.Scripting.IScriptRunnerProvider>()
?.ForSeam(ZB.MOM.WW.OtOpcUa.Core.Scripting.ScriptSeam.ScriptedAlarm)),
DriverHostActorName);
registry.Register<DriverHostActorKey>(driverHost);
@@ -34,8 +34,22 @@ internal sealed class WorkerHost
private readonly int _parentPid;
private readonly ConcurrentDictionary<string, LoadedScript> _scripts = new(StringComparer.Ordinal);
private readonly ConcurrentDictionary<long, RunningEval> _running = new();
/// <summary>
/// Hand-off to an IDLE evaluation thread: an evaluation is put here only when a thread is known to be waiting
/// for it (<see cref="_idle"/>), so whatever is put here starts at once. Everything else waits in
/// <see cref="_backlog"/> and is reported to the host as deferred (lmxopcua#780 batch submit: per-item deadlines
/// stay exact, see the proto's <c>EvalResults</c>).
/// </summary>
private readonly BlockingCollection<EvalRequest> _queue = new(new ConcurrentQueue<EvalRequest>());
/// <summary>Evaluations that arrived while every evaluation thread was busy, in arrival order. Under <see cref="_schedLock"/>.</summary>
private readonly Queue<EvalRequest> _backlog = new();
private readonly object _schedLock = new();
/// <summary>Evaluation threads committed to taking from <see cref="_queue"/>, minus hand-offs not yet taken. Under <see cref="_schedLock"/>.</summary>
private int _idle;
/// <summary>
/// Loads waiting for a compile thread. Compiles are long, synchronous and CPU-bound, so they run on
/// <see cref="CompileThreads"/> dedicated threads, never on the thread pool: a burst of cold loads (a node start
@@ -225,8 +239,20 @@ internal sealed class WorkerHost
if (_scripts.TryRemove(key, out var gone)) gone.Dispose();
return null;
case Envelope.BodyOneofCase.Eval:
foreach (var item in frame.Eval.Items) Admit(item);
{
List<ulong>? deferred = null;
foreach (var item in frame.Eval.Items)
if (!Admit(item)) (deferred ??= []).Add(item.EvalId);
// One notice per incoming frame. It can reach the host after the deferred run's own start notice (an
// evaluation thread may pick the run up before this is queued); the host treats the two as unordered.
if (deferred is not null)
{
var notice = new EvalResults();
notice.DeferredEvalIds.AddRange(deferred);
Send(new Envelope { Results = notice });
}
return null;
}
case Envelope.BodyOneofCase.Cancel:
if (_running.TryGetValue((long)frame.Cancel.EvalId, out var toCancel))
{
@@ -288,38 +314,72 @@ internal sealed class WorkerHost
Send(new Envelope { LoadResult = result });
}
private void Admit(EvalRequest request)
/// <summary>
/// Schedules one evaluation: straight to an idle evaluation thread when there is one (it starts now, so the
/// host's clock, started at submit, is right), otherwise into the backlog.
/// </summary>
/// <param name="request">The evaluation.</param>
/// <returns>False when it was deferred to the backlog (the host must stop its clock until it starts).</returns>
private bool Admit(EvalRequest request)
{
if (_queue.Count >= _queueCapacity)
lock (_schedLock)
{
SendResult(new EvalResult
if (_backlog.Count + _queue.Count < _queueCapacity)
{
EvalId = request.EvalId,
Error = new ScriptError { Kind = ScriptError.Types.Kind.Busy, Message = "script worker busy" },
});
return;
if (_idle > 0 && _backlog.Count == 0)
{
_idle--;
_queue.Add(request);
return true;
}
_backlog.Enqueue(request);
return false;
}
}
_queue.Add(request);
// Full: answered at once, outside the lock (a send may write to the pipe on this thread).
SendResult(new EvalResult
{
EvalId = request.EvalId,
Error = new ScriptError { Kind = ScriptError.Types.Kind.Busy, Message = "script worker busy" },
});
return true;
}
private void EvalThreadLoop()
{
foreach (var request in _queue.GetConsumingEnumerable())
EvalResult? finished = null;
while (true)
{
EvalResult result;
// A thread that just finished a run takes the oldest deferred evaluation, and its start notice rides on
// the finished run's result frame. With nothing deferred it goes idle and waits for a hand-off.
EvalRequest? request = null;
lock (_schedLock)
{
if (finished is not null && _backlog.TryDequeue(out var next)) request = next;
else _idle++;
}
if (finished is not null)
{
var frame = new EvalResults();
frame.Items.Add(finished);
if (request is not null) frame.StartedEvalIds.Add(request.EvalId);
Send(new Envelope { Results = frame });
finished = null;
}
request ??= _queue.Take();
try
{
result = Evaluate(request);
finished = Evaluate(request);
}
catch (Exception ex)
{
result = new EvalResult
finished = new EvalResult
{
EvalId = request.EvalId,
Error = new ScriptError { Kind = ScriptError.Types.Kind.Threw, Message = ex.Message },
};
}
SendResult(result);
}
}
@@ -431,7 +491,7 @@ internal sealed class WorkerHost
Heartbeat = new Heartbeat
{
Inflight = (uint)_running.Count,
Queued = (uint)_queue.Count,
Queued = (uint)QueuedCount(),
HeapBytes = GC.GetTotalMemory(false),
Gen2Count = (uint)GC.CollectionCount(2),
},
@@ -489,6 +549,11 @@ internal sealed class WorkerHost
Environment.Exit(ScriptWorkerEnvironment.ExitPipeClosed);
}
private int QueuedCount()
{
lock (_schedLock) return _backlog.Count + _queue.Count;
}
private void SendResult(EvalResult result)
{
var results = new EvalResults();
@@ -0,0 +1,192 @@
using System.Collections.Concurrent;
using Serilog;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
namespace ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms.Tests;
/// <summary>
/// lmxopcua#780 — the scripted-alarm seam on an <see cref="IScriptRunner"/> (design §4.5, §9). With a runner the engine
/// submits a pass's whole dirty set as ONE batch and applies the results in the serial order, so the Part 9 sequence
/// is the in-process one; and a predicate that did not complete freezes its condition with an annotation that says
/// whose fault it was: <c>BadInternalError</c> for the script (timeout, kill, quarantine), <c>BadResourceUnavailable</c>
/// for the runner (worker restarted, none ready) — re-evaluated by itself when the runner is back. A dead worker never
/// clears an alarm.
/// </summary>
[Trait("Category", "Unit")]
public sealed class ScriptedAlarmEngineScriptRunnerTests
{
private const uint StatusBadInternalError = 0x80020000u;
private const uint StatusBadResourceUnavailable = 0x80040000u;
/// <summary>Answers each predicate through <see cref="Answer"/> and records every batch.</summary>
private sealed class FakeRunner : IScriptRunner
{
public Func<ScriptInvocation, ScriptRunOutcome> Answer { get; set; } = _ => ScriptRunOutcome.Ok(false);
public ConcurrentQueue<IReadOnlyList<string>> Batches { get; } = new();
public List<ScriptContextKind> Resets { get; } = [];
public int SingleRuns;
public Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default)
{
Interlocked.Increment(ref SingleRuns);
return Task.FromResult(Answer(invocation));
}
public IReadOnlyList<Task<ScriptRunOutcome>> RunBatch(IReadOnlyList<ScriptInvocation> invocations, CancellationToken ct = default)
{
invocations.ShouldAllBe(i => i.Kind == ScriptContextKind.AlarmPredicate);
Batches.Enqueue(invocations.Select(i => i.Key).ToArray());
return invocations.Select(i => Task.FromResult(Answer(i))).ToArray();
}
public void Reset(ScriptContextKind kind) => Resets.Add(kind);
public IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind) => [];
}
private static ScriptedAlarmDefinition Alarm(string id, string predicate) =>
new(AlarmId: id, EquipmentPath: "Plant/Line1/Reactor", AlarmName: id, Kind: AlarmKind.AlarmCondition,
Severity: AlarmSeverity.High, MessageTemplate: "condition", PredicateScriptSource: predicate);
private static ScriptedAlarmEngine Build(FakeUpstream up, IScriptRunner? runner)
{
var logger = new LoggerConfiguration().CreateLogger();
return new ScriptedAlarmEngine(up, new InMemoryAlarmStateStore(), new ScriptLoggerFactory(logger), logger,
scriptTimeout: TimeSpan.FromMilliseconds(250), scriptRunner: runner);
}
/// <summary>The fake's stand-in for the five predicates: <c>Temp &gt; threshold</c>, threshold from the alarm id.</summary>
private static ScriptRunOutcome Threshold(ScriptInvocation invocation)
=> ScriptRunOutcome.Ok((int)invocation.Inputs["Temp"].Value! > int.Parse(invocation.Key[1..]));
private static readonly ScriptedAlarmDefinition[] Five =
[.. new[] { 10, 20, 30, 40, 50 }.Select(t => Alarm($"a{t}", $"return (int)ctx.GetTag(\"Temp\").Value > {t};"))];
private static async Task WaitFor(Func<bool> probe, string what)
{
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(10);
while (DateTime.UtcNow < deadline)
{
if (probe()) return;
await Task.Delay(10);
}
throw new TimeoutException(what);
}
[Fact]
public async Task A_dirty_set_is_one_batch_and_the_transitions_come_out_exactly_as_in_process()
{
var ct = TestContext.Current.CancellationToken;
var runner = new FakeRunner { Answer = Threshold };
var outOfProcess = new FakeUpstream();
var inProcess = new FakeUpstream();
outOfProcess.Set("Temp", 0);
inProcess.Set("Temp", 0);
using var oop = Build(outOfProcess, runner);
using var ip = Build(inProcess, null);
var oopEvents = new ConcurrentQueue<(string, EmissionKind)>();
var ipEvents = new ConcurrentQueue<(string, EmissionKind)>();
oop.OnEvent += (_, e) => oopEvents.Enqueue((e.AlarmId, e.Emission));
ip.OnEvent += (_, e) => ipEvents.Enqueue((e.AlarmId, e.Emission));
await oop.LoadAsync(Five, ct);
await ip.LoadAsync(Five, ct);
runner.Batches.Count.ShouldBe(1, "startup recovery evaluates every alarm in one batch");
runner.Resets.ShouldBe([ScriptContextKind.AlarmPredicate], "a load is the seam's deploy boundary");
foreach (var temp in new[] { 35, 60, 15, 0 })
{
var before = runner.Batches.Count;
outOfProcess.Push("Temp", temp);
inProcess.Push("Temp", temp);
await WaitFor(() => runner.Batches.Count > before && oop.GetAllStates().Count == 5, "the out-of-process pass");
await WaitFor(() => ipEvents.Count == oopEvents.Count, "the in-process pass to catch up");
}
runner.Batches.Skip(1).ShouldAllBe(b => b.Count == 5, "each push re-evaluates the whole dirty set in one batch");
runner.SingleRuns.ShouldBe(0);
oopEvents.ToArray().ShouldBe(ipEvents.ToArray(), "the same Part 9 sequence, in the same order");
oopEvents.Count(e => e.Item2 == EmissionKind.Activated).ShouldBe(5);
oop.GetAllStates().Select(s => (s.AlarmId, s.Active)).OrderBy(x => x.AlarmId)
.ShouldBe(ip.GetAllStates().Select(s => (s.AlarmId, s.Active)).OrderBy(x => x.AlarmId));
}
[Fact]
public async Task A_predicate_that_did_not_complete_is_frozen_and_annotated_by_whose_fault_it_was()
{
var ct = TestContext.Current.CancellationToken;
var runner = new FakeRunner { Answer = _ => ScriptRunOutcome.Ok(true) };
var up = new FakeUpstream();
up.Set("Temp", 1);
using var eng = Build(up, runner);
var events = new ConcurrentQueue<ScriptedAlarmEvent>();
eng.OnEvent += (_, e) => events.Enqueue(e);
await eng.LoadAsync([Alarm("a", "return (int)ctx.GetTag(\"Temp\").Value > 0;")], ct);
eng.GetState("a")!.Active.ShouldBe(AlarmActiveState.Active);
events.Clear();
// The runner, not the script: frozen ACTIVE (a dead worker never clears an alarm), BadResourceUnavailable.
var recovered = new TaskCompletionSource();
runner.Answer = _ => ScriptRunOutcome.Unavailable(ScriptRunOutcome.WorkerRestartedReason) with { Recovery = recovered.Task };
up.Push("Temp", 2);
await WaitFor(() => !events.IsEmpty, "the runner-failure annotation");
var annotation = events.Single();
annotation.Emission.ShouldBe(EmissionKind.QualityChanged);
annotation.WorstInputStatusCode.ShouldBe(StatusBadResourceUnavailable);
eng.GetState("a")!.Active.ShouldBe(AlarmActiveState.Active, "not cleared because the worker died");
// The worker is back: re-evaluated by itself, against the current inputs, with no new push.
events.Clear();
runner.Answer = _ => ScriptRunOutcome.Ok(false);
var batches = runner.Batches.Count;
recovered.SetResult();
await WaitFor(() => runner.Batches.Count > batches && eng.GetState("a")!.Active == AlarmActiveState.Inactive, "the recovery re-evaluation");
await WaitFor(() => events.Any(e => e.Emission == EmissionKind.Cleared), "the clear");
// The completed run restored the quality on the transition itself (no separate annotation).
events.Single().WorstInputStatusCode.ShouldBe(0u);
// The script: a timeout (or kill, or quarantine) freezes it with BadInternalError, exactly as in process.
events.Clear();
runner.Answer = _ => ScriptRunOutcome.Failed("script timed out after 0.3s");
up.Push("Temp", 3);
await WaitFor(() => !events.IsEmpty, "the script-failure annotation");
events.Single().WorstInputStatusCode.ShouldBe(StatusBadInternalError);
eng.GetState("a")!.Active.ShouldBe(AlarmActiveState.Inactive);
// A predicate that threw keeps its state with no annotation (as in process: its run completed).
events.Clear();
runner.Answer = _ => ScriptRunOutcome.Ok(true);
up.Push("Temp", 4);
await WaitFor(() => eng.GetState("a")!.Active == AlarmActiveState.Active, "a completed run");
events.Clear();
runner.Answer = _ => ScriptRunOutcome.Failed("script threw: boom");
var before = runner.Batches.Count;
up.Push("Temp", 5);
await WaitFor(() => runner.Batches.Count > before, "the throwing pass");
await Task.Delay(100, ct);
events.ShouldBeEmpty();
eng.GetState("a")!.Active.ShouldBe(AlarmActiveState.Active);
}
[Fact]
public async Task Alarms_whose_inputs_are_not_ready_never_reach_the_runner()
{
var ct = TestContext.Current.CancellationToken;
var runner = new FakeRunner { Answer = _ => ScriptRunOutcome.Ok(true) };
var up = new FakeUpstream();
up.Set("A", 1);
up.Set("B", 0, 0x80000000u); // Bad
using var eng = Build(up, runner);
await eng.LoadAsync([Alarm("good", "return (int)ctx.GetTag(\"A\").Value > 0;"), Alarm("bad", "return (int)ctx.GetTag(\"B\").Value > 0;")], ct);
runner.Batches.Single().ShouldBe(["good"], "a Bad input holds the predicate, as in process");
eng.GetState("bad")!.Active.ShouldBe(AlarmActiveState.Inactive);
eng.GetState("good")!.Active.ShouldBe(AlarmActiveState.Active);
}
}
@@ -0,0 +1,103 @@
using System.Collections.Concurrent;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
namespace ZB.MOM.WW.OtOpcUa.Core.Scripting.Tests;
/// <summary>
/// lmxopcua#780 (design §4.5) — the virtual-tag seam's opportunistic batching: an evaluation on an idle seam is
/// submitted at once by its own caller (no timer, no added latency), and evaluations submitted while a submission
/// is in progress go to the runner together, in arrival order, each caller getting its own outcome.
/// </summary>
[Trait("Category", "Unit")]
public sealed class ScriptBatchCoalescerTests
{
private static readonly IReadOnlyDictionary<string, DataValueSnapshot> NoInputs = new Dictionary<string, DataValueSnapshot>();
private static ScriptInvocation Vt(string key) => new(key, ScriptContextKind.VirtualTag, "return 1;", NoInputs, TimeSpan.FromSeconds(1));
/// <summary>Answers each key with itself; the first <see cref="RunBatch"/> blocks until released.</summary>
private sealed class BlockingRunner : IScriptRunner
{
private int _calls;
public ManualResetEventSlim Entered { get; } = new();
public ManualResetEventSlim Release { get; } = new();
public ConcurrentQueue<string[]> Batches { get; } = new();
public bool BlockFirst { get; init; } = true;
public Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default)
=> throw new InvalidOperationException("the coalescer must submit through RunBatch");
public IReadOnlyList<Task<ScriptRunOutcome>> RunBatch(IReadOnlyList<ScriptInvocation> invocations, CancellationToken ct = default)
{
Batches.Enqueue(invocations.Select(i => i.Key).ToArray());
if (BlockFirst && Interlocked.Increment(ref _calls) == 1)
{
Entered.Set();
Release.Wait(TimeSpan.FromSeconds(10)).ShouldBeTrue();
}
return invocations.Select(i => Task.FromResult(ScriptRunOutcome.Ok(i.Key))).ToArray();
}
public void Reset(ScriptContextKind kind) { }
public IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind) => [];
}
[Fact]
public async Task An_evaluation_on_an_idle_seam_is_submitted_at_once_by_its_own_caller()
{
var runner = new BlockingRunner { BlockFirst = false };
var coalescer = new ScriptBatchCoalescer(runner);
var outcome = coalescer.RunAsync(Vt("only"));
outcome.IsCompletedSuccessfully.ShouldBeTrue("submitted synchronously, no timer");
(await outcome).Value.ShouldBe("only");
runner.Batches.ToArray().ShouldBe([["only"]]);
}
[Fact]
public async Task Evaluations_submitted_while_a_submission_is_in_progress_go_together_in_arrival_order()
{
var runner = new BlockingRunner();
var coalescer = new ScriptBatchCoalescer(runner);
var first = Task.Run(() => coalescer.RunAsync(Vt("first")), TestContext.Current.CancellationToken);
runner.Entered.Wait(TimeSpan.FromSeconds(10), TestContext.Current.CancellationToken).ShouldBeTrue();
var queued = Enumerable.Range(0, 10).Select(i => coalescer.RunAsync(Vt($"k{i}"))).ToArray();
queued.ShouldAllBe(t => !t.IsCompleted, "queued behind the submission in progress");
runner.Release.Set();
(await first).Value.ShouldBe("first");
var outcomes = await Task.WhenAll(queued);
outcomes.Select(o => o.Value).ShouldBe(Enumerable.Range(0, 10).Select(i => (object?)$"k{i}"));
runner.Batches.ToArray().ShouldBe([["first"], [.. Enumerable.Range(0, 10).Select(i => $"k{i}")]]);
}
[Fact]
public async Task A_runner_that_throws_does_not_strand_its_callers()
{
var coalescer = new ScriptBatchCoalescer(new ThrowingRunner());
var outcome = await coalescer.RunAsync(Vt("x"));
outcome.Status.ShouldBe(ScriptRunStatus.Failed);
outcome.Reason!.ShouldContain("boom");
(await coalescer.RunAsync(Vt("y"))).Status.ShouldBe(ScriptRunStatus.Failed, "the flush flag was released");
}
private sealed class ThrowingRunner : IScriptRunner
{
public Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default) => throw new NotSupportedException();
public IReadOnlyList<Task<ScriptRunOutcome>> RunBatch(IReadOnlyList<ScriptInvocation> invocations, CancellationToken ct = default)
=> throw new InvalidOperationException("boom");
public void Reset(ScriptContextKind kind) { }
public IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind) => [];
}
}
@@ -0,0 +1,110 @@
using System.Collections.Concurrent;
using Serilog;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
namespace ZB.MOM.WW.OtOpcUa.Driver.Calculation.Tests;
/// <summary>
/// lmxopcua#780 batch submit (design §4.5) in the Calculation pump: with an out-of-process runner the pump hands
/// everything it dequeued to ONE <see cref="IScriptRunner.RunBatch"/> and publishes in dequeue order. A tag queued
/// twice ends the batch, so every queued evaluation still runs, in order, against the snapshot that triggered it —
/// the published sequence is exactly the one-at-a-time pump's.
/// </summary>
public sealed class CalculationDriverBatchPumpTests
{
private const uint StatusGood = 0u;
/// <summary>
/// Answers <c>src/a × n</c> for tag <c>calc/tn</c>. The first batch is held until <see cref="Release"/>, so the
/// test can queue more triggers behind it.
/// </summary>
private sealed class GatedRunner : IScriptRunner
{
private readonly TaskCompletionSource _gate = new(TaskCreationOptions.RunContinuationsAsynchronously);
private int _batches;
public ConcurrentQueue<string[]> Batches { get; } = new();
public int SingleRuns;
public void Release() => _gate.SetResult();
public Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default)
{
Interlocked.Increment(ref SingleRuns);
return Task.FromResult(Answer(invocation));
}
public IReadOnlyList<Task<ScriptRunOutcome>> RunBatch(IReadOnlyList<ScriptInvocation> invocations, CancellationToken ct = default)
{
Batches.Enqueue(invocations.Select(i => i.Key).ToArray());
var held = Interlocked.Increment(ref _batches) == 1;
return invocations.Select(i => held ? Later(i) : Task.FromResult(Answer(i))).ToArray();
}
private async Task<ScriptRunOutcome> Later(ScriptInvocation invocation)
{
await _gate.Task.ConfigureAwait(false);
return Answer(invocation);
}
private static ScriptRunOutcome Answer(ScriptInvocation invocation)
=> ScriptRunOutcome.Ok((double)invocation.Inputs["src/a"].Value! * int.Parse(invocation.Key["calc/t".Length..]));
public void Reset(ScriptContextKind kind) { }
public IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind) => [];
}
private static RawTagEntry CalcTag(string rawPath, int factor)
{
var source = $"return (double)ctx.GetTag(\"src/a\").Value * {factor};";
var json = $"{{\"scriptId\":\"s{factor}\",\"changeTriggered\":true,\"scriptSource\":{System.Text.Json.JsonSerializer.Serialize(source)}}}";
return new RawTagEntry(rawPath, json, WriteIdempotent: false);
}
private static async Task Settle(CalculationDriver driver)
{
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(30);
while (DateTime.UtcNow < deadline && !driver.IsEvaluationIdle) await Task.Delay(10);
driver.IsEvaluationIdle.ShouldBeTrue();
}
[Fact]
public async Task The_pump_submits_what_it_dequeued_as_one_batch_and_publishes_in_dequeue_order()
{
var runner = new GatedRunner();
var driver = new CalculationDriver(
new CalculationDriverOptions { RawTags = [CalcTag("calc/t1", 1), CalcTag("calc/t2", 2), CalcTag("calc/t3", 3)], ScriptRunner = runner },
"calc-drv", new ScriptRootLogger(new LoggerConfiguration().CreateLogger()));
var published = new ConcurrentQueue<(string Tag, object? Value, uint Status)>();
driver.OnDataChange += (_, e) => published.Enqueue((e.FullReference, e.Snapshot.Value, e.Snapshot.StatusCode));
var sizes = new ConcurrentQueue<int>();
driver.BatchSubmittedForTest = sizes.Enqueue;
await driver.InitializeAsync("{}", TestContext.Current.CancellationToken);
driver.OnDependencyValue("src/a", 1.0, StatusGood, DateTime.UtcNow);
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(10);
while (DateTime.UtcNow < deadline && runner.Batches.IsEmpty) await Task.Delay(5, TestContext.Current.CancellationToken);
// Two more triggers queue behind the held batch: six evaluations, each tag twice.
driver.OnDependencyValue("src/a", 2.0, StatusGood, DateTime.UtcNow);
driver.OnDependencyValue("src/a", 3.0, StatusGood, DateTime.UtcNow);
runner.Release();
await Settle(driver);
runner.SingleRuns.ShouldBe(0);
runner.Batches.ToArray().ShouldBe(
[["calc/t1", "calc/t2", "calc/t3"], ["calc/t1", "calc/t2", "calc/t3"], ["calc/t1", "calc/t2", "calc/t3"]],
"a tag queued twice ends the batch; every queued evaluation still runs");
sizes.ToArray().ShouldBe([3, 3, 3]);
published.ToArray().ShouldBe(
[
("calc/t1", 1.0, StatusGood), ("calc/t2", 2.0, StatusGood), ("calc/t3", 3.0, StatusGood),
("calc/t1", 2.0, StatusGood), ("calc/t2", 4.0, StatusGood), ("calc/t3", 6.0, StatusGood),
("calc/t1", 3.0, StatusGood), ("calc/t2", 6.0, StatusGood), ("calc/t3", 9.0, StatusGood),
], "dequeue order, each against the snapshot that triggered it");
}
}
@@ -139,13 +139,32 @@ public sealed class ScriptRunnerSelectionTests
[Fact]
public void OutOfProcess_selects_only_the_supported_requested_seams()
{
var all = new ScriptingOptions { Runner = ScriptRunnerMode.OutOfProcess };
ScriptRunnerSelection.Validate(all, "/srv").ShouldContain(w => w.Contains("ScriptedAlarm seam cannot run out of process yet"));
using (var selection = ScriptRunnerSelection.Create(all, Silent, null))
// Unset Seams: VirtualTag + Calculation. Scripted alarms stay in process unless listed (design §14 #10).
var unset = new ScriptingOptions { Runner = ScriptRunnerMode.OutOfProcess };
ScriptRunnerSelection.Validate(unset, "/srv").ShouldNotContain(w => w.Contains("cannot run out of process"));
unset.IsOutOfProcess(ScriptSeam.ScriptedAlarm).ShouldBeFalse();
unset.IsOutOfProcess(ScriptSeam.VirtualTag).ShouldBeTrue();
using (var selection = ScriptRunnerSelection.Create(unset, Silent, null))
{
selection.ForSeam(ScriptSeam.VirtualTag).ShouldNotBeNull();
selection.ForSeam(ScriptSeam.Calculation).ShouldBeSameAs(selection.ForSeam(ScriptSeam.VirtualTag), "one node-wide runner");
selection.ForSeam(ScriptSeam.ScriptedAlarm).ShouldBeNull();
selection.ForSeam(ScriptSeam.ScriptedAlarm).ShouldBeNull("alarms go out of process only when listed explicitly");
}
// Listed explicitly: the alarm seam shares the node's one runner.
var withAlarms = new ScriptingOptions
{
Runner = ScriptRunnerMode.OutOfProcess,
RunnerSeams = [ScriptSeam.VirtualTag, ScriptSeam.ScriptedAlarm],
};
withAlarms.IsOutOfProcess(ScriptSeam.ScriptedAlarm).ShouldBeTrue();
ScriptRunnerSelection.Validate(withAlarms, "/srv").ShouldNotContain(w => w.Contains("cannot run out of process"));
using (var selection = ScriptRunnerSelection.Create(withAlarms, Silent, null))
{
selection.ForSeam(ScriptSeam.ScriptedAlarm).ShouldNotBeNull();
selection.ForSeam(ScriptSeam.ScriptedAlarm).ShouldBeSameAs(selection.ForSeam(ScriptSeam.VirtualTag), "one node-wide runner");
selection.ForSeam(ScriptSeam.Calculation).ShouldBeNull();
((IScriptRunnerProvider)selection).ForSeam(ScriptSeam.ScriptedAlarm).ShouldBeSameAs(selection.ForSeam(ScriptSeam.ScriptedAlarm));
}
var calcOnly = new ScriptingOptions { Runner = ScriptRunnerMode.OutOfProcess, RunnerSeams = [ScriptSeam.Calculation] };
@@ -0,0 +1,152 @@
using System.Collections.Concurrent;
using System.Diagnostics;
using Serilog;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests;
/// <summary>A minimal <see cref="ITagUpstreamSource"/>: set or push a value; push notifies subscribers synchronously.</summary>
internal sealed class TestUpstream : ITagUpstreamSource
{
private readonly ConcurrentDictionary<string, DataValueSnapshot> _values = new(StringComparer.Ordinal);
private readonly ConcurrentDictionary<string, List<Action<string, DataValueSnapshot>>> _subs = new(StringComparer.Ordinal);
public void Set(string path, object? value, uint status = 0u)
=> _values[path] = new DataValueSnapshot(value, status, DateTime.UtcNow, DateTime.UtcNow);
public void Push(string path, object? value, uint status = 0u)
{
Set(path, value, status);
if (!_subs.TryGetValue(path, out var list)) return;
Action<string, DataValueSnapshot>[] observers;
lock (list) observers = [.. list];
foreach (var observer in observers) observer(path, _values[path]);
}
public DataValueSnapshot ReadTag(string path)
=> _values.TryGetValue(path, out var v) ? v : new DataValueSnapshot(null, 0x80320000u, null, DateTime.UtcNow);
public IDisposable SubscribeTag(string path, Action<string, DataValueSnapshot> observer)
{
var list = _subs.GetOrAdd(path, _ => []);
lock (list) list.Add(observer);
return new Unsubscribe(() => { lock (list) list.Remove(observer); });
}
private sealed class Unsubscribe(Action dispose) : IDisposable
{
public void Dispose() => dispose();
}
}
/// <summary>
/// lmxopcua#780 — the scripted-alarm seam over the REAL worker (design §9, alarm column): a runaway predicate is killed
/// at T + G and quarantined, its condition frozen with a <c>BadInternalError</c> annotation and never transitioned; a
/// sibling alarm on the same shard keeps evaluating; and a worker that dies under an in-flight predicate freezes its
/// condition <c>BadResourceUnavailable</c> — it never clears it — and the alarm is re-evaluated by itself on return.
/// </summary>
[Collection(ScriptWorkerCollection.Name)]
[Trait("Category", "ScriptWorker")]
public sealed class AlarmSeamTests
{
private const uint StatusBadInternalError = 0x80020000u;
private const uint StatusBadResourceUnavailable = 0x80040000u;
private static CancellationToken Ct => TestContext.Current.CancellationToken;
private static ScriptedAlarmDefinition Alarm(string id, string predicate) =>
new(AlarmId: id, EquipmentPath: "Plant/Line1/Reactor", AlarmName: id, Kind: AlarmKind.AlarmCondition,
Severity: AlarmSeverity.High, MessageTemplate: "condition", PredicateScriptSource: predicate);
private static ScriptedAlarmEngine Engine(TestUpstream up, IScriptRunner runner, TimeSpan budget)
{
var logger = new LoggerConfiguration().CreateLogger();
return new ScriptedAlarmEngine(up, new InMemoryAlarmStateStore(), new ScriptLoggerFactory(logger), logger,
scriptTimeout: budget, scriptRunner: runner);
}
[Fact]
public async Task A_runaway_predicate_is_killed_and_quarantined_its_alarm_frozen_Bad_and_a_sibling_keeps_evaluating()
{
var budget = TimeSpan.FromMilliseconds(300);
var log = new CapturingSink();
await using var runner = Scripts.Runner(Scripts.Options(hardGrace: TimeSpan.FromMilliseconds(500), quarantineAfterKills: 2), log.CreateLogger());
var up = new TestUpstream();
up.Set("x", 0);
using var engine = Engine(up, runner, budget);
var events = new ConcurrentQueue<ScriptedAlarmEvent>();
engine.OnEvent += (_, e) => events.Enqueue(e);
await engine.LoadAsync(
[
Alarm("runaway", "if ((int)ctx.GetTag(\"x\").Value > 5) { while (true) { } } return false;"),
Alarm("sibling", "return (int)ctx.GetTag(\"x\").Value > 1;"),
], Ct);
var first = runner.Shards[0].Current.ShouldNotBeNull();
up.Push("x", 10);
await Scripts.Eventually(() => events.Any(e => e is { AlarmId: "sibling", Emission: EmissionKind.Activated }), TimeSpan.FromSeconds(10), "the sibling's activation");
await Scripts.Eventually(() => events.Any(e => e is { AlarmId: "runaway", Emission: EmissionKind.QualityChanged }), TimeSpan.FromSeconds(10), "the runaway's annotation");
events.Single(e => e.AlarmId == "runaway").WorstInputStatusCode.ShouldBe(StatusBadInternalError);
(await first.Exited.WaitAsync(TimeSpan.FromSeconds(10), Ct)).CulpritKey
.ShouldBe(InProcessScriptRunner.ScopedKey(ScriptContextKind.AlarmPredicate, "runaway"));
// The second kill quarantines it; the sibling (same shard, worker restarted twice) keeps evaluating. Re-pushed:
// Exited completes before the dead worker's evaluations are released, so the first push after it can find the
// runaway's slot still held and be refused (not a strike, exactly as in process) — the next push runs it.
var deadline = DateTime.UtcNow + TimeSpan.FromSeconds(60);
for (var x = 11; !engine.QuarantinedAlarmIds.Contains("runaway"); x++)
{
if (DateTime.UtcNow > deadline)
throw new TimeoutException("no kill quarantine; events: " +
string.Join(", ", events.Select(e => $"{e.AlarmId}:{e.Emission}:{e.WorstInputStatusCode:X8}")) +
Environment.NewLine + string.Join(Environment.NewLine, log.Rendered));
up.Push("x", x);
await Task.Delay(TimeSpan.FromSeconds(2), Ct);
}
engine.QuarantinedAlarmIds.ShouldBe(["runaway"]);
up.Push("x", 0);
await Scripts.Eventually(() => events.Any(e => e is { AlarmId: "sibling", Emission: EmissionKind.Cleared }), TimeSpan.FromSeconds(60), "the sibling's clear");
engine.GetState("runaway")!.Active.ShouldBe(AlarmActiveState.Inactive, "a predicate that never returned never transitions");
events.Where(e => e.AlarmId == "runaway").ShouldAllBe(e => e.Emission == EmissionKind.QualityChanged);
// A redeploy (reload) lifts the quarantine, as in process.
await engine.LoadAsync([Alarm("runaway", "return (int)ctx.GetTag(\"x\").Value > 5;"), Alarm("sibling", "return (int)ctx.GetTag(\"x\").Value > 1;")], Ct);
engine.QuarantinedAlarmIds.ShouldBeEmpty();
}
[Fact]
public async Task A_worker_that_dies_under_a_predicate_freezes_it_BadResourceUnavailable_never_clears_it_and_it_is_re_evaluated_on_return()
{
await using var runner = Scripts.Runner(Scripts.Options(probation: 1));
var up = new TestUpstream();
up.Set("ms", 1);
using var engine = Engine(up, runner, TimeSpan.FromSeconds(5));
var events = new ConcurrentQueue<ScriptedAlarmEvent>();
engine.OnEvent += (_, e) => events.Enqueue(e);
await engine.LoadAsync([Alarm("slow",
"var ms = (int)ctx.GetTag(\"ms\").Value; var end = ctx.Now.AddMilliseconds(ms); while (ctx.Now < end) { } return ms > 0;")], Ct);
engine.GetState("slow")!.Active.ShouldBe(AlarmActiveState.Active);
var worker = runner.Shards[0].Current.ShouldNotBeNull();
events.Clear();
up.Push("ms", 1500);
await Task.Delay(300, Ct); // the predicate is spinning in the worker
Process.GetProcessById(worker.Pid).Kill();
await Scripts.Eventually(() => events.Any(e => e.Emission == EmissionKind.QualityChanged), TimeSpan.FromSeconds(10), "the runner-failure annotation");
events.First().WorstInputStatusCode.ShouldBe(StatusBadResourceUnavailable);
engine.GetState("slow")!.Active.ShouldBe(AlarmActiveState.Active, "a dead worker never clears an alarm");
// No further input change: the runner's recovery re-evaluates it (in the isolation worker) and the quality returns.
await Scripts.Eventually(() => events.Any(e => e is { Emission: EmissionKind.QualityChanged, WorstInputStatusCode: 0u }),
TimeSpan.FromSeconds(60), "the re-evaluation on return");
engine.GetState("slow")!.Active.ShouldBe(AlarmActiveState.Active);
events.ShouldNotContain(e => e.Emission == EmissionKind.Cleared);
}
}
@@ -0,0 +1,172 @@
using System.Collections.Concurrent;
using System.Diagnostics;
using Serilog;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
namespace ZB.MOM.WW.OtOpcUa.ScriptWorker.Tests;
/// <summary>
/// lmxopcua#780 batch submit (design §4.5) against the real worker: <see cref="IScriptRunner.RunBatch"/> sends one frame
/// per worker and answers per item, and every item keeps its own deadlines and blame — in particular an item that has to
/// wait in the worker behind a runaway sibling is neither timed out, nor struck, nor blamed for the kill.
/// </summary>
[Collection(ScriptWorkerCollection.Name)]
[Trait("Category", "ScriptWorker")]
public sealed class BatchSubmitTests
{
/// <summary>Runs away when input <c>spin</c> is true, otherwise returns <c>n + 1</c>.</summary>
private const string MaybeRunaway = "if ((bool)ctx.GetTag(\"spin\").Value) { while (true) { } } return (int)ctx.GetTag(\"n\").Value + 1;";
/// <summary>Spins for input <c>ms</c> milliseconds, then returns it.</summary>
private const string Spin = "var ms = (int)ctx.GetTag(\"ms\").Value; var end = ctx.Now.AddMilliseconds(ms); while (ctx.Now < end) { } return ms;";
private static CancellationToken Ct => TestContext.Current.CancellationToken;
private static IReadOnlyDictionary<string, DataValueSnapshot> Inputs(params (string Path, object Value)[] values)
{
var map = new Dictionary<string, DataValueSnapshot>(StringComparer.Ordinal);
foreach (var (path, value) in values) map[path] = new DataValueSnapshot(value, 0, DateTime.UtcNow, DateTime.UtcNow);
return map;
}
private static ScriptInvocation Item(string key, bool spin, int n, TimeSpan? timeout = null)
=> Scripts.Vt(key, MaybeRunaway, timeout, Inputs(("spin", spin), ("n", n)));
private static OutOfProcessScriptRunner Runner(ScriptingOptions options, int evalThreads)
=> new(options, new LoggerConfiguration().CreateLogger(), null,
heartbeatInterval: TimeSpan.FromSeconds(1), loadTimeout: TimeSpan.FromSeconds(60), evalThreads: evalThreads);
[Fact]
public async Task A_batch_of_loaded_scripts_goes_out_as_one_frame_and_every_item_gets_its_own_result()
{
await using var runner = Runner(Scripts.Options(), evalThreads: 4);
// First evaluations load (compile) each key; those take the full path one by one.
var warm = await Task.WhenAll(runner.RunBatch(Enumerable.Range(0, 40).Select(i => Item($"k{i}", false, i)).ToArray(), Ct));
warm.ShouldAllBe(o => o.Status == ScriptRunStatus.Ok);
var frames = new ConcurrentQueue<int>();
runner.BatchSubmittedForTest = frames.Enqueue;
var outcomes = await Task.WhenAll(runner.RunBatch(Enumerable.Range(0, 40).Select(i => Item($"k{i}", false, i * 10)).ToArray(), Ct));
frames.ToArray().ShouldBe([40], "every loaded item rides one EvalBatch frame");
for (var i = 0; i < 40; i++)
{
outcomes[i].Status.ShouldBe(ScriptRunStatus.Ok, outcomes[i].Reason);
outcomes[i].Value.ShouldBe(i * 10 + 1, $"item {i} got its own result");
}
}
[Fact]
public async Task A_batch_answers_exactly_what_the_same_evaluations_answer_one_by_one()
{
// Parity: Ok, a throw, a compile error, a duplicate key (refused) and a fast item, batched vs RunAsync.
ScriptInvocation[] Items() =>
[
Scripts.Vt("ok", "return (int)ctx.GetTag(\"n\").Value * 2;", inputs: Inputs(("n", 21))),
Scripts.Vt("throws", "throw new System.InvalidOperationException(\"boom\");"),
Scripts.Vt("broken", "return undefinedThing;"),
Scripts.Vt("fast", "return 5;"),
];
await using var runner = Runner(Scripts.Options(), evalThreads: 2);
var single = new List<ScriptRunOutcome>();
foreach (var item in Items()) single.Add(await runner.RunAsync(item, Ct));
var batched = await Task.WhenAll(runner.RunBatch(Items(), Ct));
batched.Select(o => (o.Status, o.Value, o.Reason)).ShouldBe(single.Select(o => (o.Status, o.Value, o.Reason)));
single[0].Value.ShouldBe(42);
single[1].Reason!.ShouldStartWith("script threw: ");
single[2].Reason!.ShouldStartWith("compile error: ");
// In process, the default RunBatch is RunAsync per item: same answers.
using var inProcess = new InProcessScriptRunner();
var inProcessBatched = await Task.WhenAll(((IScriptRunner)inProcess).RunBatch(Items(), Ct));
inProcessBatched.Select(o => (o.Status, o.Value)).ShouldBe(single.Select(o => (o.Status, o.Value)));
// A key twice in one batch: the second is refused, exactly like two concurrent RunAsync calls.
var twice = await Task.WhenAll(runner.RunBatch([Scripts.Vt("dup", Spin, inputs: Inputs(("ms", 200))), Scripts.Vt("dup", Spin, inputs: Inputs(("ms", 1)))], Ct));
twice.Count(o => o.Reason == ScriptRunGuard.PreviousRunStillRunningReason).ShouldBe(1);
}
/// <summary>
/// The blame rule (design §5.3.1) under batching: with ONE evaluation thread, a runaway at the head of a batch holds
/// the thread and its siblings wait in the worker. The siblings are the kill's victims — Unavailable "worker
/// restarted", no timeout, no strike — and only the runaway is killed and quarantined. Mutation check: with the
/// host ignoring the worker's deferral notice, the siblings' clocks run from submit, they time out at T with the
/// runaway, and at T + G any of the four can be the one blamed.
/// </summary>
[Fact]
public async Task A_runaway_in_a_batch_is_blamed_alone_and_siblings_queued_behind_it_are_not_timed_out_or_struck()
{
var softT = TimeSpan.FromMilliseconds(400);
var grace = TimeSpan.FromMilliseconds(500);
await using var runner = Runner(Scripts.Options(hardGrace: grace, quarantineAfterKills: 1), evalThreads: 1);
var keys = new[] { "runaway", "sib1", "sib2", "sib3" };
foreach (var key in keys) (await runner.RunAsync(Item(key, false, 1, softT), Ct)).Value.ShouldBe(2);
var worker = runner.Shards[0].Current.ShouldNotBeNull();
var frames = new ConcurrentQueue<int>();
runner.BatchSubmittedForTest = frames.Enqueue;
var sw = Stopwatch.StartNew();
var tasks = runner.RunBatch([Item("runaway", true, 0, softT), Item("sib1", false, 10, softT), Item("sib2", false, 20, softT), Item("sib3", false, 30, softT)], Ct);
frames.ToArray().ShouldBe([4]);
var runaway = await tasks[0];
runaway.Status.ShouldBe(ScriptRunStatus.Failed);
runaway.Reason!.ShouldStartWith("script timed out after ");
// Nothing has answered the queued siblings at the runaway's soft deadline.
tasks.Skip(1).ShouldAllBe(t => !t.IsCompleted, "a run waiting behind a busy thread is not timed out at its submit-relative T");
var exit = await worker.Exited.WaitAsync(TimeSpan.FromSeconds(10), Ct);
exit.Reason.ShouldBe(WorkerExitReason.KilledDeadline);
exit.CulpritKey.ShouldBe(InProcessScriptRunner.ScopedKey(ScriptContextKind.VirtualTag, "runaway"));
sw.Elapsed.ShouldBeGreaterThanOrEqualTo(softT + grace - TimeSpan.FromMilliseconds(50));
var siblings = await Task.WhenAll(tasks.Skip(1));
foreach (var sibling in siblings)
{
sibling.Status.ShouldBe(ScriptRunStatus.Unavailable, "a victim of another key's kill, not a script failure");
sibling.Reason.ShouldBe(ScriptRunOutcome.WorkerRestartedReason);
}
await Scripts.Eventually(() => runner.QuarantinedKeys(ScriptContextKind.VirtualTag).Count > 0, TimeSpan.FromSeconds(5), "the kill quarantine");
runner.QuarantinedKeys(ScriptContextKind.VirtualTag).ShouldBe(["runaway"], "the siblings took no strike");
// The siblings run again on the restarted worker, with their own inputs.
var again = await Task.WhenAll(runner.RunBatch([Item("sib1", false, 10, softT), Item("sib2", false, 20, softT), Item("sib3", false, 30, softT)], Ct));
again.Select(o => o.Value).ShouldBe([11, 21, 31]);
}
/// <summary>
/// The positive control for the deferral: a run that waited behind a sibling still gets its deadlines — the full T,
/// then G — counted from when it STARTED. Mutation check: a start notice that does not re-arm the clock leaves the
/// deferred runaway with no deadline at all; it is never answered or killed.
/// </summary>
[Fact]
public async Task A_deferred_run_is_timed_and_killed_from_when_it_starts()
{
var softT = TimeSpan.FromMilliseconds(400);
var grace = TimeSpan.FromMilliseconds(500);
var head = TimeSpan.FromMilliseconds(300);
await using var runner = Runner(Scripts.Options(hardGrace: grace, quarantineAfterKills: 5), evalThreads: 1);
(await runner.RunAsync(Scripts.Vt("head", Spin, softT, Inputs(("ms", 1))), Ct)).Value.ShouldBe(1);
(await runner.RunAsync(Item("tail", false, 1, softT), Ct)).Value.ShouldBe(2);
var worker = runner.Shards[0].Current.ShouldNotBeNull();
var sw = Stopwatch.StartNew();
var tasks = runner.RunBatch([Scripts.Vt("head", Spin, softT, Inputs(("ms", (int)head.TotalMilliseconds))), Item("tail", true, 0, softT)], Ct);
(await tasks[0]).Value.ShouldBe((int)head.TotalMilliseconds, "the head ran inside its budget");
var tail = await tasks[1];
var answered = sw.Elapsed;
tail.Reason!.ShouldStartWith("script timed out after ");
answered.ShouldBeGreaterThanOrEqualTo(head + softT - TimeSpan.FromMilliseconds(50), "T counts from the run's start, after the head");
var exit = await worker.Exited.WaitAsync(TimeSpan.FromSeconds(10), Ct);
exit.CulpritKey.ShouldBe(InProcessScriptRunner.ScopedKey(ScriptContextKind.VirtualTag, "tail"));
sw.Elapsed.ShouldBeGreaterThanOrEqualTo(head + softT + grace - TimeSpan.FromMilliseconds(50), "G counts from the run's start too");
}
}
@@ -3,6 +3,7 @@ using System.Text;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms;
using ZB.MOM.WW.OtOpcUa.Core.Scripting;
using ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess;
@@ -261,6 +262,217 @@ public sealed class ScriptRunnerPerfTests
k2Closed.EvalsPerSecond.ShouldBeGreaterThanOrEqualTo(AcceptedGateEvalsPerSecond, "the accepted 5k evals/s/node gate at K=2");
}
/// <summary>
/// lmxopcua#780 batch submit (design §4.5): a closed loop of one batch at a time, <paramref name="batchSize"/>
/// distinct loaded keys per batch, either through <see cref="IScriptRunner.RunBatch"/> or as the same evaluations
/// submitted as concurrent <see cref="IScriptRunner.RunAsync"/> calls. Latency is per batch (submit → last result).
/// </summary>
private static async Task<PhaseResult> BatchLoopAsync(IScriptRunner runner, int batchSize, bool viaBatch)
{
var deadline = Stopwatch.GetTimestamp() + (long)(Phase.TotalSeconds * Stopwatch.Frequency);
var samples = new List<double>(100_000);
long ok = 0, notOk = 0;
string? firstFailure = null;
var offset = 0;
var x = 0;
var cpu0 = Process.GetCurrentProcess().TotalProcessorTime;
var wall = Stopwatch.StartNew();
while (Stopwatch.GetTimestamp() < deadline)
{
var invocations = new ScriptInvocation[batchSize];
var keys = new int[batchSize];
for (var i = 0; i < batchSize; i++)
{
keys[i] = (offset + i) % Keys;
invocations[i] = Invocation(keys[i], x);
}
offset = (offset + batchSize) % Keys;
var t0 = Stopwatch.GetTimestamp();
var outcomes = await Task.WhenAll(viaBatch
? runner.RunBatch(invocations, Ct)
: invocations.Select(inv => runner.RunAsync(inv, Ct)).ToArray());
samples.Add(Stopwatch.GetElapsedTime(t0).TotalMicroseconds);
for (var i = 0; i < batchSize; i++)
{
var o = outcomes[i];
if (o.Status == ScriptRunStatus.Ok && Equals(o.Value, x + keys[i] % DistinctSources)) ok++;
else
{
notOk++;
firstFailure ??= $"{o.Status}: {o.Reason} (value {o.Value})";
}
}
x++;
}
var elapsed = wall.Elapsed;
var cpu = (Process.GetCurrentProcess().TotalProcessorTime - cpu0).TotalMilliseconds / elapsed.TotalMilliseconds * 100;
samples.Sort();
return new PhaseResult((ok + notOk) / elapsed.TotalSeconds, Percentile(samples, 0.50), Percentile(samples, 0.99), cpu, ok, notOk, firstFailure);
}
/// <summary>
/// A runner that turns a batch back into one evaluation after another — the serial pass the alarm engine and the
/// Calc pump made before batch submit, which design §8.2 warns costs N round trips (the comparison row).
/// </summary>
private sealed class SerialRunner(IScriptRunner inner) : IScriptRunner
{
public Task<ScriptRunOutcome> RunAsync(ScriptInvocation invocation, CancellationToken ct = default) => inner.RunAsync(invocation, ct);
public IReadOnlyList<Task<ScriptRunOutcome>> RunBatch(IReadOnlyList<ScriptInvocation> invocations, CancellationToken ct = default)
{
var results = new Task<ScriptRunOutcome>[invocations.Count];
Task previous = Task.CompletedTask;
for (var i = 0; i < results.Length; i++) previous = results[i] = After(previous, invocations[i], ct);
return results;
}
private async Task<ScriptRunOutcome> After(Task previous, ScriptInvocation invocation, CancellationToken ct)
{
await previous.ConfigureAwait(false);
return await inner.RunAsync(invocation, ct).ConfigureAwait(false);
}
public void Reset(ScriptContextKind kind) => inner.Reset(kind);
public IReadOnlyCollection<string> QuarantinedKeys(ScriptContextKind kind) => inner.QuarantinedKeys(kind);
}
/// <summary>
/// Scripted-alarm pass latency: <paramref name="alarms"/> alarms on one input, every push flips all of them, and a
/// sample is the time from the push to the last of the N transitions the pass emits (the whole dirty set through
/// the engine's gate). The design's alarm budget is 250 ms (§1.1; the host now passes <c>Scripting:RunTimeout</c>,
/// 2 s by default, lmxopcua#779), so the pass is judged against the stricter 250 ms.
/// </summary>
private static async Task<(double P50Ms, double P99Ms, double MaxMs, double CpuPercent, int NotCompleted)> AlarmPassLatencyAsync(
IScriptRunner? runner, int alarms, int pushes)
{
var up = new TestUpstream();
up.Set("x", 0);
var logger = new Serilog.LoggerConfiguration().CreateLogger();
using var engine = new ScriptedAlarmEngine(up, new InMemoryAlarmStateStore(), new ScriptLoggerFactory(logger), logger,
scriptTimeout: TimeSpan.FromMilliseconds(250), scriptRunner: runner);
var transitions = 0;
var notCompleted = 0;
var target = int.MaxValue;
var done = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
engine.OnEvent += (_, e) =>
{
// A Bad annotation means a predicate did not complete in that pass (timed out, refused): counted, reported.
if (e.Emission == EmissionKind.QualityChanged && e.WorstInputStatusCode != 0) Interlocked.Increment(ref notCompleted);
if (e.Emission is not (EmissionKind.Activated or EmissionKind.Cleared)) return;
if (Interlocked.Increment(ref transitions) == Volatile.Read(ref target)) done.TrySetResult();
};
await engine.LoadAsync(Enumerable.Range(0, alarms).Select(i => new ScriptedAlarmDefinition(
$"alarm-{i}", "Plant/Perf", $"alarm-{i}", AlarmKind.AlarmCondition, AlarmSeverity.High, "perf",
$"return (int)ctx.GetTag(\"x\").Value > {-(i % DistinctSources)};")).ToArray(), Ct);
var samples = new List<double>(pushes);
var cpu0 = Process.GetCurrentProcess().TotalProcessorTime;
var wall = Stopwatch.StartNew();
for (var p = 0; p < pushes + 2; p++)
{
// Alternate x = -1000 (every predicate false) and x = 1000 (every predicate true). The pass is expected to
// transition exactly the alarms not already in the target state — counted from the engine, so an alarm
// that did not complete an earlier pass cannot leave the count short. The first two passes are warm-up.
var activate = p % 2 == 1;
var expected = engine.GetAllStates().Count(c => (c.Active == AlarmActiveState.Active) != activate);
Interlocked.Exchange(ref transitions, 0);
done = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
Volatile.Write(ref target, expected);
var t0 = Stopwatch.GetTimestamp();
up.Push("x", activate ? 1000 : -1000);
if (expected == 0) continue;
try
{
await done.Task.WaitAsync(TimeSpan.FromSeconds(60), Ct);
}
catch (TimeoutException)
{
throw new TimeoutException($"alarm pass {p} ({(runner is null ? "in process" : runner.GetType().Name)}, {alarms} alarms): " +
$"{Volatile.Read(ref transitions)} of {expected} transitions in 60 s, " +
$"{Volatile.Read(ref notCompleted)} predicate(s) did not complete so far");
}
if (p >= 2) samples.Add(Stopwatch.GetElapsedTime(t0).TotalMilliseconds);
}
var cpu = (Process.GetCurrentProcess().TotalProcessorTime - cpu0).TotalMilliseconds / wall.Elapsed.TotalMilliseconds * 100;
samples.Sort();
return (Percentile(samples, 0.50), Percentile(samples, 0.99), samples[^1], cpu, notCompleted);
}
/// <summary>
/// Batch throughput (RunBatch vs the same evaluations as concurrent RunAsync calls, K = 2) and scripted-alarm pass
/// latency (in process, out of process batched, and out of process serial). Report-only, except that every
/// result must be correct.
/// </summary>
[Fact]
public async Task Batch_throughput_and_alarm_pass_latency()
{
RequireOptIn();
var report = new StringBuilder()
.AppendLine($"### Batch submit throughput ({Env()})")
.AppendLine()
.AppendLine("| Runner | batch | submitted as | evals/s | batch p50 µs | batch p99 µs | host CPU | ok / not ok |")
.AppendLine("|---|---:|---|---:|---:|---:|---:|---|");
foreach (var (name, create) in new (string, Func<IScriptRunner>)[]
{
("InProcess", () => new InProcessScriptRunner(new ScriptingOptions())),
("OutOfProcess K=2", () => Scripts.Runner(PerfOptions(2), _runnerLog.CreateLogger())),
})
{
var runner = create();
try
{
await WarmAsync(runner);
foreach (var size in new[] { 1, 10, 100, 500 })
{
foreach (var viaBatch in new[] { true, false })
{
var r = await BatchLoopAsync(runner, size, viaBatch);
r.NotOk.ShouldBe(0, $"{name} batch {size}: {r.FirstFailure}{Environment.NewLine}{RunnerLog()}");
report.AppendLine($"| {name} | {size} | {(viaBatch ? "RunBatch" : "RunAsync ×n")} | {r.EvalsPerSecond,9:N0} | " +
$"{r.P50Us,8:N0} | {r.P99Us,9:N0} | {r.HostCpuPercent,6:N0} % | {r.Ok:N0} / {r.NotOk:N0} |");
}
}
}
finally
{
switch (runner)
{
case IAsyncDisposable a: await a.DisposeAsync(); break;
case IDisposable d: d.Dispose(); break;
}
}
}
report.AppendLine()
.AppendLine($"### Scripted-alarm pass latency, push → last transition ({Env()})")
.AppendLine()
.AppendLine("| Engine | alarms | pass p50 ms | pass p99 ms | max ms | per alarm (p50) µs | host CPU | not completed | 250 ms |")
.AppendLine("|---|---:|---:|---:|---:|---:|---:|---:|---|");
const int pushes = 40;
await using (var shared = Scripts.Runner(PerfOptions(2), _runnerLog.CreateLogger()))
{
foreach (var alarms in new[] { 100, 500, 1000 })
{
foreach (var (name, runner) in new (string, IScriptRunner?)[]
{
("InProcess (serial)", null),
("OutOfProcess K=2, batched", shared),
("OutOfProcess K=2, serial", new SerialRunner(shared)),
})
{
var (p50, p99, max, cpu, notCompleted) = await AlarmPassLatencyAsync(runner, alarms, pushes);
report.AppendLine($"| {name} | {alarms:N0} | {p50,7:N1} | {p99,7:N1} | {max,7:N1} | {p50 * 1000 / alarms,6:N0} | " +
$"{cpu,5:N0} % | {notCompleted:N0} | {(p99 <= 250 ? "PASS" : "MISS")} |");
}
}
}
report.AppendLine()
.AppendLine("Each push flips every alarm; a pass is the engine's whole dirty set under its gate. \"serial\" replays the " +
"pre-batch one-predicate-at-a-time pass over the same runner.");
Report(report);
}
private static ScriptingOptions PerfOptions(int shards, string? peCacheDir = null)
{
var options = Scripts.Options(shards: shards, hardGrace: TimeSpan.FromSeconds(2), peCacheDir: peCacheDir,
@@ -22,6 +22,8 @@
<ProjectReference Include="..\..\..\src\Core\ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess\ZB.MOM.WW.OtOpcUa.Core.Scripting.OutOfProcess.csproj"/>
<!-- End-to-end seam test: a real Calculation driver over the real worker. -->
<ProjectReference Include="..\..\..\src\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Calculation\ZB.MOM.WW.OtOpcUa.Driver.Calculation.csproj"/>
<!-- The scripted-alarm seam over the real worker (lmxopcua#780). -->
<ProjectReference Include="..\..\..\src\Core\ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms\ZB.MOM.WW.OtOpcUa.Core.ScriptedAlarms.csproj"/>
</ItemGroup>
<!-- The real worker, built and placed in script-worker/ beside the tests, exactly as beside the Host. -->