fix(virtual-tags): evaluate scripts off the Akka dispatcher thread (lmxopcua#779)

VirtualTagActor called the blocking IVirtualTagEvaluator.Evaluate, and
RoslynVirtualTagEvaluator blocked on the run guard with
.GetAwaiter().GetResult(), so every script run parked a dispatcher thread for
up to its 2 s budget (a first-use Roslyn compile did too). This is the #750 P0
refactor for virtual tags.

- IVirtualTagEvaluator.EvaluateAsync (default: a completed task over the
  snapshot overload, so the null evaluator and fakes are unchanged).
- RoslynVirtualTagEvaluator implements it without blocking: passthrough and
  compile failures complete synchronously, a first-use compile goes to the
  pool, the run is awaited. The blocking overload now delegates to it.
- VirtualTagActor: a completed task is applied in the same turn, as before;
  an incomplete one switches to an "evaluating" behaviour that stashes every
  dependency/reassert message until a private completion message arrives,
  then applies the result and unstashes. Each trigger is still evaluated
  against its own inputs, results are applied/published in trigger order and
  the dedup/Bad-transition state sees the same sequence. The dependency map is
  not touched while a run reads it.

Calculation and scripted-alarm serialisation is NOT relaxed: neither blocks a
thread, and running independent scripts in parallel is not semantics-neutral
(pool consumption bound, cross-tag publish order, timer/pump exclusion for
Calc; state-machine/persist boundary, transition timestamps and the
store-failure path for alarms). Recorded in docs/VirtualTags.md and the #750
design's P0 entry.

Tests: VirtualTagActorAsyncEvaluationTests — on a one-thread dispatcher a slow
script no longer starves a sibling actor and the slow actor answers Identify
(fails when the actor calls the blocking Evaluate); triggers during a run are
evaluated afterwards in order against their own inputs; a reassert during a
run is applied after it; a faulted evaluation takes the evaluator-threw path
(the ordering/reassert tests fail with the stash removed).
RoslynVirtualTagEvaluatorAsyncTests — a slow script and a cold compile return
an incomplete task (fail without the override); cheap outcomes and timeouts
match the blocking overload.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV

src/Server/ZB.MOM.WW.OtOpcUa.Host/Engines/RoslynVirtualTagEvaluator.cs

@@ -11,8 +11,9 @@ namespace ZB.MOM.WW.OtOpcUa.Host.Engines;
 /// F8b — production <see cref="IVirtualTagEvaluator"/> binding. Compiles each unique
 /// expression once via <see cref="ScriptEvaluator{TContext, TResult}"/> (Roslyn-backed
 /// sandbox) and caches the resulting evaluator keyed by source. Subsequent evaluations are
-/// in-process method invocations on the dependency dictionary — fast enough to run inline
-/// inside the actor's message handler.
+/// in-process method invocations on the dependency dictionary. The actor calls
+/// <see cref="EvaluateAsync"/>, which awaits the run instead of blocking its dispatcher thread
+/// (lmxopcua#779).
 ///
 /// Single-tag mode: cross-tag <c>ctx.SetVirtualTag</c> writes are dropped (logged) because
 /// fan-out between actors is owned by <c>DependencyMuxActor</c>, not by the eval engine.
@@ -98,8 +99,22 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac
     // actor holds evaluation entirely while any input is Bad, so what a script observes here is Good
     // or Uncertain.
     /// <inheritdoc />
+    /// <remarks>Blocks the caller for up to the run budget. The runtime calls <see cref="EvaluateAsync"/> instead;
+    /// this overload remains for single-shot callers.</remarks>
     public VirtualTagEvalResult Evaluate(
         string virtualTagId, string expression, IReadOnlyDictionary<string, DataValueSnapshot> dependencies)
+        => EvaluateAsync(virtualTagId, expression, dependencies).GetAwaiter().GetResult();
+
+    /// <inheritdoc />
+    /// <remarks>
+    ///     lmxopcua#779 / #750 P0 — the virtual-tag actor used to call the synchronous overload, which blocked on the
+    ///     run guard with <c>.GetAwaiter().GetResult()</c> and parked an Akka dispatcher thread for up to the whole
+    ///     budget (2 s) per evaluation. This path never blocks: the passthrough fast path and compile failures
+    ///     complete synchronously (cheap), a first-use compile is pushed to the pool, and the script run is awaited.
+    ///     The outcomes are exactly those of the synchronous overload, which now delegates here.
+    /// </remarks>
+    public async Task<VirtualTagEvalResult> EvaluateAsync(
+        string virtualTagId, string expression, IReadOnlyDictionary<string, DataValueSnapshot> dependencies)
     {
         if (_disposed) return VirtualTagEvalResult.Failure("evaluator disposed");
         if (string.IsNullOrWhiteSpace(expression)) return VirtualTagEvalResult.Failure("empty expression");
@@ -141,7 +156,11 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac
             ScriptEvaluator<VirtualTagContext, object?> evaluator;
             try
             {
-                evaluator = _cache.GetOrCompile(expression);
+                // A first-use Roslyn compile costs tens to hundreds of milliseconds: never on the caller's thread
+                // (an Akka dispatcher thread). A cached fetch is a dictionary hit and stays inline.
+                evaluator = _cache.Contains(expression)
+                    ? _cache.GetOrCompile(expression)
+                    : await Task.Run(() => _cache.GetOrCompile(expression)).ConfigureAwait(false);
             }
             catch (CompilationErrorException ex)
             {
@@ -172,8 +191,8 @@ public sealed class RoslynVirtualTagEvaluator : IVirtualTagEvaluator, IScriptCac
                 // it), so the wall-clock budget is what returns control — and the guard is what stops the
                 // abandoned run from being joined by another on the next trigger (one orphan per tag, then
                 // quarantine), which is the leak the old per-evaluation TimedScriptEvaluator allowed.
-                var raw = _guard.RunAsync(virtualTagId, expression, evaluator, context, _runTimeout)
-                    .GetAwaiter().GetResult();
+                var raw = await _guard.RunAsync(virtualTagId, expression, evaluator, context, _runTimeout)
+                    .ConfigureAwait(false);
                 return VirtualTagEvalResult.Ok(raw);
             }
             catch (ScriptTimeoutException)

src/Server/ZB.MOM.WW.OtOpcUa.Runtime/VirtualTags/VirtualTagActor.cs

@@ -27,9 +27,29 @@ namespace ZB.MOM.WW.OtOpcUa.Runtime.VirtualTags;
 ///     multi-dep script does not flash Bad during deploy warm-up. Recovery to a Good value clears the
 ///     Bad state and is force-published even when the value equals the pre-failure value.
 /// </para>
+/// <para>
+///     <b>Evaluation never blocks the dispatcher (lmxopcua#779, #750 P0).</b> The evaluator is called through
+///     <see cref="IVirtualTagEvaluator.EvaluateAsync"/>. A synchronously completed result (passthrough, the null
+///     evaluator, test fakes) is applied inline in the same message turn, exactly as before. An incomplete one — a
+///     Roslyn script run, up to its 2 s budget — switches the actor into an <i>evaluating</i> behaviour that
+///     <b>stashes</b> every dependency / reassert message, and the result comes back as a private completion
+///     message. Applying it unstashes everything in arrival order. So each trigger is still evaluated against
+///     the dependency map as it stood at that trigger, results are applied and published in trigger order, and
+///     the dedup / Bad-transition state sees exactly the sequence it saw when the call blocked — only the
+///     dispatcher thread is no longer held while the script runs. The dependency map is not touched while a run
+///     is in flight (the messages that would touch it are stashed), which is what makes handing it to the
+///     script on another thread safe.
+/// </para>
 /// </summary>
-public sealed class VirtualTagActor : ReceiveActor
+public sealed class VirtualTagActor : ReceiveActor, IWithUnboundedStash
 {
+    /// <summary>The asynchronous evaluation started by <see cref="Evaluate"/> finished. Private to the actor.</summary>
+    private sealed record EvaluationCompleted(
+        Task<VirtualTagEvalResult> Evaluation, uint WorstInputStatus, DateTime TimestampUtc);
+
+    /// <inheritdoc />
+    public IStash Stash { get; set; } = null!;
+
     public const string ScriptLogsTopic = "script-logs";
 
     /// <summary>A dependency's value changed. <paramref name="StatusCode"/> carries the source
@@ -168,6 +188,9 @@ public sealed class VirtualTagActor : ReceiveActor
         Receive<DependencyValueChanged>(OnDependencyChanged);
         Receive<DependencyValueBatch>(OnDependencyBatch);
         Receive<ReassertValue>(_ => OnReassertValue());
+        // Only reachable for a completion that outlived its behaviour (a supervision restart mid-evaluation
+        // resets the behaviour stack): the incarnation it belonged to is gone, so it is dropped.
+        Receive<EvaluationCompleted>(_ => { });
     }
 
     /// <inheritdoc />
@@ -270,18 +293,66 @@ public sealed class VirtualTagActor : ReceiveActor
             return;
         }
 
-        VirtualTagEvalResult result;
+        Task<VirtualTagEvalResult> evaluation;
         try
         {
-            result = _evaluator.Evaluate(_virtualTagId, _expression, _dependencies);
+            evaluation = _evaluator.EvaluateAsync(_virtualTagId, _expression, _dependencies);
         }
         catch (Exception ex)
         {
-            _log.Warning(ex, "VirtualTag {Id}: evaluator threw", _virtualTagId);
-            OnEvaluationFailed($"evaluator threw: {ex.Message}", "Error", timestampUtc);
+            OnEvaluatorThrew(ex, timestampUtc);
             return;
         }
 
+        if (evaluation.IsCompleted)
+        {
+            // Synchronous evaluators (passthrough, null evaluator, fakes) and cheap failures: same turn, as before.
+            ApplyEvaluation(evaluation, worst, timestampUtc);
+            return;
+        }
+
+        // A real script run is in flight: free the dispatcher thread, hold every later input until it lands.
+        var self = Self;
+        evaluation.ContinueWith(
+            t => self.Tell(new EvaluationCompleted(t, worst, timestampUtc)),
+            CancellationToken.None, TaskContinuationOptions.ExecuteSynchronously, TaskScheduler.Default);
+        BecomeStacked(Evaluating);
+    }
+
+    /// <summary>Behaviour while an evaluation is in flight: everything but its completion is stashed, so inputs and
+    /// reasserts are processed after it, in arrival order, against the state it leaves behind.</summary>
+    private void Evaluating()
+    {
+        Receive<EvaluationCompleted>(msg =>
+        {
+            UnbecomeStacked();
+            ApplyEvaluation(msg.Evaluation, msg.WorstInputStatus, msg.TimestampUtc);
+            Stash.UnstashAll();
+        });
+        ReceiveAny(_ => Stash.Stash());
+    }
+
+    private void OnEvaluatorThrew(Exception ex, DateTime timestampUtc)
+    {
+        _log.Warning(ex, "VirtualTag {Id}: evaluator threw", _virtualTagId);
+        OnEvaluationFailed($"evaluator threw: {ex.Message}", "Error", timestampUtc);
+    }
+
+    /// <summary>Applies one completed evaluation: the failure tail, or the (value, status) dedup and the publish.</summary>
+    /// <param name="evaluation">The completed evaluation task.</param>
+    /// <param name="worst">The worst input status at the trigger.</param>
+    /// <param name="timestampUtc">The trigger's timestamp.</param>
+    private void ApplyEvaluation(Task<VirtualTagEvalResult> evaluation, uint worst, DateTime timestampUtc)
+    {
+        if (!evaluation.IsCompletedSuccessfully)
+        {
+            var ex = evaluation.Exception?.GetBaseException()
+                     ?? new OperationCanceledException("virtual-tag evaluation was cancelled");
+            OnEvaluatorThrew(ex, timestampUtc);
+            return;
+        }
+
+        var result = evaluation.Result;
         if (!result.Success)
         {
             OnEvaluationFailed(result.Reason ?? "evaluator failure", "Warning", timestampUtc);