Commit Graph

5 Commits

Author SHA1 Message Date
Joseph Doherty 11a5c1d4e2 test: deflake load-sensitive tests (lmxopcua#769)
v2-ci / unit-tests (push) Has been cancelled
- NodeManagerLockTimeBenchmark: wall-clock budget enforced only with OTOPCUA_BENCH=1; default run keeps the state-correctness asserts and logs timings.
- OpcUaClientDriverWire host-status: signal (semaphore) wait that only accepts a non-Running state observed after the server is disposed.
- AbCip PollLatencyComparison: retry once on a fixture connection reset; p95 criterion enforced only with AB_LAT_ENFORCE=1.
- ScriptRunGuardTests: orphan scripts held by a ctx.Released signal instead of a 250 ms wall-clock spin that could finish inside the 60 ms budget under load; stuck-orphan wall-clock rule disabled in the tests that pin the strike counting.
- DependencyMuxReplayTests: split the inside-grace positive control (1 h grace) from the release test (control asserted before unregister), removing the 200 ms grace race.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-10 05:39:22 -05:00
Joseph Doherty 850c5405d0 fix(scripting): name the script on the mirrored quarantine line; keep BadInternalError (lmxopcua#784)
v2-ci / unit-tests (push) Has been cancelled
(a) ScriptRunGuard logs its quarantine Error through the seam's script root
logger, which carries no ScriptName, so ScriptLogCompanionSink mirrored it to
the main log as "[Script] unknown emitted Error ..." and rate-limited every
quarantine on the node under one shared "unknown" key. The guard now binds the
script instance key as ScriptName (ForContext) on the quarantine Error and the
subscriber-threw Warning, so the mirror names the script and limits each one
separately. Test drives two real quarantines through the companion sink with a
1-per-window limiter: both mirrored, each named, none "unknown". Mutation-
checked: dropping the ForContext fails it.

(b) Evaluated a quarantine-specific OPC UA status (BadOutOfService) and kept
BadInternalError: it is the documented code for any script-attributable fault
on virtual tags, Calculation tags and scripted-alarm quality (and the
out-of-process runner design keeps it), BadOutOfService reads as a deliberately
disabled point, a new code is a ScadaBridge-visible contract change, and the
once-per-Good->Bad dedup means a timeout-driven quarantine would rarely show a
different code anyway. Documented in docs/VirtualTags.md ("Quarantine: what a
client sees"), with pointers from Calculation.md, ScriptedAlarms.md and the
OpcUaStatus.BadInternalError doc comment, plus where a quarantine IS visible.

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
2026-11-09 21:30:17 -05:00
Joseph Doherty a85d1c069a fix(scripting): quarantine a refusal behind a stuck orphan (fix round 2)
Ruling refinement on I1. Only timeouts are strikes, so a never-returning
while(true), which times out once and is then only refused, was never
quarantined, and that defeated E1. A refusal now quarantines immediately
when the orphan it was refused behind has been executing for longer than
ScriptingOptions.StuckOrphanQuarantineFactor x the script's timeout
(default 10, so 20 s at the 2 s default; 0 or less disables it). It uses
the same quarantine path: one Error, otopcua.script.quarantined with
reason=stuck, Bad, lifted on source change or redeploy. A young orphan
(a short overrun) is still a plain refusal.

The guard stamps the run's start (Stopwatch timestamp) when the pool
picks it up and clears it when the run really finishes.

Tests: the 30-trigger while(true) regression (paced) now ends
quarantined with reason "stuck", still with exactly one orphan and a
healthy sibling. Positive control: a 2x overrun is refused, not
quarantined, and evaluates again once the orphan completes. A factor of
0 disables the rule. Docs (ScriptEditor "Runtime safety", VirtualTags)
and the seam remarks are updated.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:59:10 -04:00
Joseph Doherty 6aa5edaf30 fix(scripting): only timeouts are quarantine strikes; alarm Bad annotation once per bucket change (fix round 1 I1, I3)
- I1 ScriptRunGuard: a run refused because the previous one is still
  executing is a Bad publish plus otopcua.script.run_refused, and no longer
  a quarantine strike. Only a timeout (and an allocation breach) counts;
  a refusal neither adds to nor resets the timeout streak; a completed run
  (value or throw) resets it. On a 1 Hz trigger with a 2 s budget a single
  4 s overrun is one timeout plus two refusals and must not quarantine.
  Tests updated across the guard, the alarm engine, the Calculation
  evaluator and the VT evaluator, plus
  Refusals_neither_count_toward_nor_reset_the_timeout_streak.
- I3 ScriptedAlarmEngine: the quality bucket is tracked only after the
  predicate completes (or in the inputs-not-ready path), so the tracked
  bucket is the last one annotated. A refused or quarantined predicate
  used to re-track the input bucket Good before the attempt and then flip
  it Bad, emitting QualityChanged on every push. The run-guard tests now
  count emissions (exactly 1 across N pushes, refused and quarantined
  cases), with a healthy sibling's dispatch count as the positive control
  that every push was processed. Both were red before the fix.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 13:26:12 -04:00
Joseph Doherty eb9269c337 feat(scripting): ScriptRunGuard quarantines runaway scripts across every seam (E1)
A timed-out script is abandoned, not stopped, and every seam re-ran it on the
next trigger, leaking one pool thread per trigger (R03:1.1, R05:2, R07:L1).
A shared ScriptRunGuard, keyed per script instance, now:

- refuses a run while that script's previous orphan is still executing
  (Bad "previous evaluation still running" + otopcua.script.run_refused);
- quarantines after 3 consecutive non-completions or an allocation-budget
  breach (sampled with GC.GetAllocatedBytesForCurrentThread; default 64 MB):
  no further dispatch, Bad quality, one Error, otopcua.script.quarantined,
  cleared on source change / redeploy / reset;
- starts the budget clock when the pool picks the run up, so starvation is
  no longer charged to the script (R07:P5), and cancels the evaluator token
  on timeout.

Wired into RoslynVirtualTagEvaluator, CalculationEvaluator (new async
EvaluateAsync; the driver's pump and timer ticks now await the run behind a
SemaphoreSlim gate) and ScriptedAlarmEngine (a non-completing predicate
holds state and annotates the condition BadInternalError). ScriptingOptions
(Scripting:MaxAllocBytesPerRun, Scripting:QuarantineAfterConsecutiveTimeouts)
carries the knobs; the Calculation driver config accepts the same two keys.

Tests: guard unit suite (incl. the 30-trigger runaway leaving one orphan with
a healthy sibling as positive control), per-seam refusal/quarantine/reset
tests. The S12 race test now uses one vtag id per loop: twelve concurrent
evaluations of ONE id is a shape production never produces and the guard
refuses by design.

Refs #734

Claude-Session: https://claude.ai/code/session_01Ub9hGmYLQvn9XETzw7CMsT
2026-10-22 06:39:43 -04:00