fix(scripting): only timeouts are quarantine strikes; alarm Bad annotation once per bucket change (fix round 1 I1, I3)

- I1 ScriptRunGuard: a run refused because the previous one is still
  executing is a Bad publish plus otopcua.script.run_refused, and no longer
  a quarantine strike. Only a timeout (and an allocation breach) counts;
  a refusal neither adds to nor resets the timeout streak; a completed run
  (value or throw) resets it. On a 1 Hz trigger with a 2 s budget a single
  4 s overrun is one timeout plus two refusals and must not quarantine.
  Tests updated across the guard, the alarm engine, the Calculation
  evaluator and the VT evaluator, plus
  Refusals_neither_count_toward_nor_reset_the_timeout_streak.
- I3 ScriptedAlarmEngine: the quality bucket is tracked only after the
  predicate completes (or in the inputs-not-ready path), so the tracked
  bucket is the last one annotated. A refused or quarantined predicate
  used to re-track the input bucket Good before the attempt and then flip
  it Bad, emitting QualityChanged on every push. The run-guard tests now
  count emissions (exactly 1 across N pushes, refused and quarantined
  cases), with a healthy sibling's dispatch count as the positive control
  that every push was processed. Both were red before the fix.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
This commit is contained in:
Joseph Doherty
2026-10-22 13:26:12 -04:00
parent 909b061d3a
commit 6aa5edaf30
10 changed files with 161 additions and 67 deletions
@@ -10,7 +10,7 @@ namespace ZB.MOM.WW.OtOpcUa.Host.IntegrationTests;
/// <summary>
/// <see cref="RoslynVirtualTagEvaluator"/> runs every compiled expression through a
/// <see cref="ScriptRunGuard"/> keyed by virtual-tag id: a runaway expression leaks one orphaned pool
/// thread, not one per trigger, the tag is quarantined after three consecutive non-completions (surfacing
/// thread, not one per trigger, later triggers are refused, the tag is quarantined after three consecutive timeouts (surfacing
/// as a <c>Failure</c> the actor publishes Bad for), and the quarantine lifts at the deploy boundary.
/// </summary>
public sealed class RoslynVirtualTagEvaluatorRunGuardTests
@@ -23,7 +23,7 @@ public sealed class RoslynVirtualTagEvaluatorRunGuardTests
private static readonly Dictionary<string, object?> NoDeps = new();
[Fact]
public void Runaway_expression_is_refused_then_quarantined_and_a_sibling_keeps_evaluating()
public void Runaway_expression_is_refused_while_its_orphan_runs_and_a_sibling_keeps_evaluating()
{
using var sut = new RoslynVirtualTagEvaluator(
NullLogger<RoslynVirtualTagEvaluator>.Instance, NoOpScriptRoot(), TimeSpan.FromMilliseconds(60));
@@ -33,12 +33,10 @@ public sealed class RoslynVirtualTagEvaluatorRunGuardTests
first.Reason.ShouldNotBeNull();
first.Reason.ShouldContain("timed out");
// Refusals, not strikes (fix round 1, I1): the tag stays Bad but is not quarantined by them.
sut.Evaluate("vt-loop", SpinScript, NoDeps).Reason.ShouldBe(ScriptRunGuard.PreviousRunStillRunningReason);
var third = sut.Evaluate("vt-loop", SpinScript, NoDeps);
third.Reason.ShouldNotBeNull();
third.Reason.ShouldStartWith(ScriptRunGuard.QuarantinedReasonPrefix);
sut.QuarantinedScripts.ShouldBe(new[] { "vt-loop" });
sut.Evaluate("vt-loop", SpinScript, NoDeps).Reason.ShouldBe(ScriptRunGuard.PreviousRunStillRunningReason);
sut.QuarantinedScripts.ShouldBeEmpty();
sut.RunGuard.StartedRunCount.ShouldBe(1, "one orphan, not one per trigger");
var sibling = sut.Evaluate("vt-ok", "return 42;", NoDeps);