Scripting: alarm scripts get 250 ms default timeout; dispatcher-thread blocking; unbounded recursion can crash the server #779

New Issue

2026-11-09T17:05:00-05:00

dohertj2 commented

2026-11-09 17:05:00 -05:00

Immediate in-process scripting defects found while designing #750 (fixable before/independent of the out-of-process runner — the design's phase P0):

Scripted alarms run with the 250 ms default timeout, not the 2 s used for virtual and Calculation tags, because the alarm engine is constructed without a timeout (DriverHostActor.cs:~1251). Pass the configured script timeout (or make the 250 ms deliberate and documented).
Virtual tags block an Akka dispatcher thread for up to 2 s (RoslynVirtualTagEvaluator.cs:~174-175); Calculation and alarm scripts run serially behind a shared lock, so one slow script stalls its siblings. P0 of the design moves evaluation off the dispatcher and lets independent scripts proceed.
Unbounded recursion is not blocked by the compile-time sandbox and a StackOverflowException cannot be caught in .NET — a single script can likely take the whole server down. Add a recursion/depth guard to the sandbox analyser (or run evaluation on a thread with a bounded stack and fail the script). Verify with a test first.
Docs say script authoring is admin-gated, but the editor policy also admits Designers — reconcile docs vs policy.

Immediate in-process scripting defects found while designing #750 (fixable before/independent of the out-of-process runner — the design's phase P0): - **Scripted alarms run with the 250 ms default timeout**, not the 2 s used for virtual and Calculation tags, because the alarm engine is constructed without a timeout (DriverHostActor.cs:~1251). Pass the configured script timeout (or make the 250 ms deliberate and documented). - **Virtual tags block an Akka dispatcher thread for up to 2 s** (RoslynVirtualTagEvaluator.cs:~174-175); Calculation and alarm scripts run serially behind a shared lock, so one slow script stalls its siblings. P0 of the design moves evaluation off the dispatcher and lets independent scripts proceed. - **Unbounded recursion** is not blocked by the compile-time sandbox and a StackOverflowException cannot be caught in .NET — a single script can likely take the whole server down. Add a recursion/depth guard to the sandbox analyser (or run evaluation on a thread with a bounded stack and fail the script). Verify with a test first. - Docs say script authoring is admin-gated, but the editor policy also admits Designers — reconcile docs vs policy.

dohertj2 added the type:bug priority:p2 labels 2026-11-09 17:05:00 -05:00

dohertj2 commented

2026-11-09 18:02:27 -05:00

Fixed on master 7d989fb5 (4 commits):

d683b831 scripted-alarm predicates got the generic 250 ms Phase-7 default by omission (virtual tags moved to 2 s in F8b). New node-wide Scripting:RunTimeout (default 2 s, clamped 0–60 s with a startup warning); virtual-tag evaluator and alarm engine both use it (DriverHostActor.CreateScriptedAlarmEngine). Calculation keeps its per-driver runTimeoutMs. Note: a never-returning predicate is now quarantined after ~20 s instead of ~2.5 s.
a84d3870 virtual tags no longer block a dispatcher thread: IVirtualTagEvaluator.EvaluateAsync (default interface method); Roslyn evaluator awaits the run guard and moves first-use compile to the pool; VirtualTagActor stashes inputs while a run is in flight, so trigger order and per-trigger inputs are unchanged. Calculation/alarm one-at-a-time serialisation deliberately kept (reasoning in docs/VirtualTags.md + #750 design note).
9e8b5b0e unbounded recursion crashed the process (reproduced: 7 shapes → child exit 134 / stack overflow). Compile step now injects ScriptStackGuard.Check() (TryEnsureSufficientExecutionStack) into the script body and every lambda/local function/anonymous method → catchable ScriptStackExhaustedException, script goes Bad, process stays up. Bounded recursion and expression-tree lambdas unaffected.
7d989fb5 docs aligned to the code: script authoring + deploy are ConfigEditor (Administrator OR Designer), per AdminUI-001.

Evidence (rebased on master, Release 0 warnings/0 errors, AKKA_TEST_TIMEFACTOR=3): Runtime 981 (+3 skip), Core.Scripting 220 (+1 skip, the child half of the containment test), Core.ScriptedAlarms 106, Core.VirtualTags 16, Driver.Calculation 79, Host 247, OpcUaServer 1026 (+1 skip), OpcUaServer.IntegrationTests 132, Driver.Sql 588. Agent also ran them at timefactor 1 with identical results. Every new test was confirmed red with its fix reverted. (One Runtime.Tests failure appeared in a concurrent full-suite run and did not reproduce in 2 isolated reruns — name not captured; treat as #769-class load flake.)

Fixed on master `7d989fb5` (4 commits): 1. `d683b831` scripted-alarm predicates got the generic 250 ms Phase-7 default by omission (virtual tags moved to 2 s in F8b). New node-wide `Scripting:RunTimeout` (default 2 s, clamped 0–60 s with a startup warning); virtual-tag evaluator and alarm engine both use it (`DriverHostActor.CreateScriptedAlarmEngine`). Calculation keeps its per-driver `runTimeoutMs`. Note: a never-returning predicate is now quarantined after ~20 s instead of ~2.5 s. 2. `a84d3870` virtual tags no longer block a dispatcher thread: `IVirtualTagEvaluator.EvaluateAsync` (default interface method); Roslyn evaluator awaits the run guard and moves first-use compile to the pool; `VirtualTagActor` stashes inputs while a run is in flight, so trigger order and per-trigger inputs are unchanged. Calculation/alarm one-at-a-time serialisation deliberately kept (reasoning in docs/VirtualTags.md + #750 design note). 3. `9e8b5b0e` unbounded recursion crashed the process (reproduced: 7 shapes → child exit 134 / stack overflow). Compile step now injects `ScriptStackGuard.Check()` (`TryEnsureSufficientExecutionStack`) into the script body and every lambda/local function/anonymous method → catchable `ScriptStackExhaustedException`, script goes Bad, process stays up. Bounded recursion and expression-tree lambdas unaffected. 4. `7d989fb5` docs aligned to the code: script authoring + deploy are ConfigEditor (Administrator OR Designer), per AdminUI-001. Evidence (rebased on master, Release 0 warnings/0 errors, AKKA_TEST_TIMEFACTOR=3): Runtime 981 (+3 skip), Core.Scripting 220 (+1 skip, the child half of the containment test), Core.ScriptedAlarms 106, Core.VirtualTags 16, Driver.Calculation 79, Host 247, OpcUaServer 1026 (+1 skip), OpcUaServer.IntegrationTests 132, Driver.Sql 588. Agent also ran them at timefactor 1 with identical results. Every new test was confirmed red with its fix reverted. (One Runtime.Tests failure appeared in a concurrent full-suite run and did not reproduce in 2 isolated reruns — name not captured; treat as #769-class load flake.)

dohertj2 closed this issue

2026-11-09 18:02:27 -05:00

dohertj2 referenced this issue from a commit

2026-11-09 21:28:16 -05:00

fix(virtualtags): consume VirtualTag.TimerIntervalMs; add VirtualTagNoEffectiveTrigger gate (lmxopcua#783)

dohertj2 referenced this issue

2026-11-09 21:28:24 -05:00

Virtual tags: TimerIntervalMs is ignored at runtime, so a timer-only virtual tag never evaluates (no deploy warning) #783

dohertj2 referenced this issue

2026-11-10 04:49:27 -05:00

VirtualTag → VirtualTag dependencies may never update: VT outputs don't go through DependencyMuxActor #791

Sign in to join this conversation.