dohertj2/lmxopcua

Flaky tests under heavy machine load (4 seen during the #815–#833 batch) #843

New Issue

Closed

opened 2026-11-11 03:27:28 -05:00 by dohertj2 · 1 comment

dohertj2 commented

2026-11-11 03:27:28 -05:00

Owner

All pass reliably in isolation; each failed during parallel agent work at load average 140–280.

ScriptWorker.Tests A_burst_of_cold_compiles_does_not_starve_the_heartbeat (~1/10 under overload).
Core.Scripting.Tests ScriptRunGuardTests.Runaway_script_leaves_one_orphan_is_quarantined_once_stuck_and_a_sibling_keeps_evaluating (2/4 full-suite runs on base 5bbe1e3c; expects refused-young within 600 ms of a 60 ms timeout).
Host.IntegrationTests EquipmentNamespaceMaterializationTests.Deployed_artifact_round_trips_both_namespaces_and_seals_on_the_cluster (48 s vs 3 s normally).
Host.IntegrationTests RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_leaks_a_raw_disposed_failure (passed 5/5 rerun).
Capture failing output with trx and apply the presence-vs-absence rule (#500) or widen wall-clock budgets with self-diagnosing failures, as #811 did.

All pass reliably in isolation; each failed during parallel agent work at load average 140–280. - `ScriptWorker.Tests` `A_burst_of_cold_compiles_does_not_starve_the_heartbeat` (~1/10 under overload). - `Core.Scripting.Tests` `ScriptRunGuardTests.Runaway_script_leaves_one_orphan_is_quarantined_once_stuck_and_a_sibling_keeps_evaluating` (2/4 full-suite runs on base 5bbe1e3c; expects refused-young within 600 ms of a 60 ms timeout). - `Host.IntegrationTests` `EquipmentNamespaceMaterializationTests.Deployed_artifact_round_trips_both_namespaces_and_seals_on_the_cluster` (48 s vs 3 s normally). - `Host.IntegrationTests` `RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_leaks_a_raw_disposed_failure` (passed 5/5 rerun). Capture failing output with trx and apply the presence-vs-absence rule (#500) or widen wall-clock budgets with self-diagnosing failures, as #811 did.

dohertj2 added the type:test priority:p3 labels 2026-11-11 03:27:28 -05:00

dohertj2 referenced this issue from a commit

2026-11-11 05:24:24 -05:00

test(scripting): stuck-orphan quarantine test trusts the guard's young/stuck verdict, not the test's stopwatch (#843)

dohertj2 referenced this issue from a commit

2026-11-11 05:24:24 -05:00

test(scriptworker): cold-burst heartbeat test kills at production's 3 s silence limit, self-diagnosing (#843)

dohertj2 referenced this issue from a commit

2026-11-11 05:24:24 -05:00

fix(scripting): the run guard frees a script's slot before its run completes, not in a continuation (#843)

dohertj2 referenced this issue from a commit

2026-11-11 05:24:24 -05:00

test(host): deploy-seal waits in EquipmentNamespaceMaterializationTests report where the deploy stood on timeout (#843)

dohertj2 referenced this issue from a commit

2026-11-11 05:24:24 -05:00

Merge #843 #840 #841 load flakes, ScriptRunGuard slot-release race, DurationGood conformance, CI QuestDB pin

dohertj2 commented

2026-11-11 05:24:39 -05:00

Author

Owner

Fixed (1696e348, 4cd648b0, e89ac2df, 997c78df). Pushed to master in d1b6cad2. Reproduced under 28 CPU burners (load 100–430):

ScriptRunGuardTests.Runaway… 4/10 → 0/10 (test labelled refusals with its own stopwatch);
A_burst_of_cold_compiles… 10/10 → 0/10 (uses production's 3 s silence limit; self-diagnosing);
Evaluate_racing_ClearCompiledScripts… was a real product race: ScriptRunGuard freed a script's slot in a ContinueWith, so the next run of an already-finished script could be refused and published Bad. Slot now released inside the run; regression test fails 49/50 on the old ordering;
Deployed_artifact_round_trips… never reproduced (0/16); timeout now reports deployment/node state and both cluster views.

Fixed (`1696e348`, `4cd648b0`, `e89ac2df`, `997c78df`). Pushed to master in `d1b6cad2`. Reproduced under 28 CPU burners (load 100–430): - `ScriptRunGuardTests.Runaway…` 4/10 → 0/10 (test labelled refusals with its own stopwatch); - `A_burst_of_cold_compiles…` 10/10 → 0/10 (uses production's 3 s silence limit; self-diagnosing); - `Evaluate_racing_ClearCompiledScripts…` was a **real product race**: `ScriptRunGuard` freed a script's slot in a ContinueWith, so the next run of an already-finished script could be refused and published Bad. Slot now released inside the run; regression test fails 49/50 on the old ordering; - `Deployed_artifact_round_trips…` never reproduced (0/16); timeout now reports deployment/node state and both cluster views.

dohertj2 closed this issue

2026-11-11 05:24:39 -05:00

dohertj2 referenced this issue

2026-11-11 07:55:31 -05:00

Flaky: GalaxyDriverFixRound2Tests.A_fetch_completing_after_init_populates_host_statuses (10 s timeout under load) #856

dohertj2 referenced this issue

2026-11-11 10:01:34 -05:00

Galaxy.Tests: remaining short fixed waits fail under heavy parallel load #858

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#843