fix(scripting): only timeouts are quarantine strikes; alarm Bad annotation once per bucket change (fix round 1 I1, I3)

- I1 ScriptRunGuard: a run refused because the previous one is still
  executing is a Bad publish plus otopcua.script.run_refused, and no longer
  a quarantine strike. Only a timeout (and an allocation breach) counts;
  a refusal neither adds to nor resets the timeout streak; a completed run
  (value or throw) resets it. On a 1 Hz trigger with a 2 s budget a single
  4 s overrun is one timeout plus two refusals and must not quarantine.
  Tests updated across the guard, the alarm engine, the Calculation
  evaluator and the VT evaluator, plus
  Refusals_neither_count_toward_nor_reset_the_timeout_streak.
- I3 ScriptedAlarmEngine: the quality bucket is tracked only after the
  predicate completes (or in the inputs-not-ready path), so the tracked
  bucket is the last one annotated. A refused or quarantined predicate
  used to re-track the input bucket Good before the attempt and then flip
  it Bad, emitting QualityChanged on every push. The run-guard tests now
  count emissions (exactly 1 across N pushes, refused and quarantined
  cases), with a healthy sibling's dispatch count as the positive control
  that every push was processed. Both were red before the fix.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
This commit is contained in:
Joseph Doherty
2026-10-22 13:26:12 -04:00
parent 909b061d3a
commit 6aa5edaf30
10 changed files with 161 additions and 67 deletions
@@ -10,8 +10,8 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Calculation.Tests;
/// <see cref="CalculationEvaluator"/> runs every compiled script through a <see cref="ScriptRunGuard"/>
/// keyed by calc-tag RawPath. A timed-out run is abandoned on its pool thread; before the guard, every
/// later trigger of the same tag stacked another orphan behind it (one leaked thread per trigger). Now
/// the next trigger is refused ("previous evaluation still running"), the third consecutive
/// non-completion quarantines the tag, and a healthy sibling on the same evaluator keeps computing.
/// every later trigger is refused ("previous evaluation still running" — Bad, but not a quarantine
/// strike; only consecutive timeouts quarantine), and a healthy sibling on the same evaluator keeps computing.
/// </summary>
public sealed class CalculationEvaluatorRunGuardTests
{
@@ -24,7 +24,7 @@ public sealed class CalculationEvaluatorRunGuardTests
private static readonly Dictionary<string, object?> NoDeps = new();
[Fact]
public async Task A_trigger_during_the_previous_orphan_is_refused_then_the_tag_is_quarantined_and_a_sibling_keeps_computing()
public async Task Triggers_during_the_previous_orphan_are_refused_not_quarantined_and_a_sibling_keeps_computing()
{
using var sut = new CalculationEvaluator(
NullLogger<CalculationEvaluator>.Instance, NoOpScriptRoot(), TimeSpan.FromMilliseconds(60));
@@ -35,15 +35,16 @@ public sealed class CalculationEvaluatorRunGuardTests
first.Reason.ShouldNotBeNull();
first.Reason.ShouldContain("timed out");
var second = await sut.EvaluateAsync("calc/loop", SpinScript, Snap(NoDeps), ct);
second.Success.ShouldBeFalse();
second.Reason.ShouldBe(ScriptRunGuard.PreviousRunStillRunningReason);
var third = await sut.EvaluateAsync("calc/loop", SpinScript, Snap(NoDeps), ct);
third.Success.ShouldBeFalse();
third.Reason.ShouldNotBeNull();
third.Reason.ShouldStartWith(ScriptRunGuard.QuarantinedReasonPrefix);
sut.QuarantinedScripts.ShouldBe(new[] { "calc/loop" });
// Every later trigger while the orphan runs is a Bad "previous evaluation still running" — a
// refusal is not a strike, so the tag is NOT quarantined by them (fix round 1, I1).
for (var i = 0; i < 3; i++)
{
var refused = await sut.EvaluateAsync("calc/loop", SpinScript, Snap(NoDeps), ct);
refused.Success.ShouldBeFalse();
refused.Reason.ShouldBe(ScriptRunGuard.PreviousRunStillRunningReason);
}
sut.QuarantinedScripts.ShouldBeEmpty();
sut.RunGuard.RefusedWhileRunningCount.ShouldBe(3);
// Exactly one run was ever dispatched for the runaway tag — the leak is one orphan, not one per trigger.
sut.RunGuard.StartedRunCount.ShouldBe(1);