ScriptWorker ShadowModeTests: one unexplained wrong sibling value under heavy CPU load #811

New Issue

2026-11-10T21:39:41-05:00

dohertj2 commented

2026-11-10 21:39:41 -05:00

Seen once while fixing #798 (commit 304d1f1e). In the first loaded run of the reworked ShadowModeTests runaway test (16 yes processes, load avg ~64) the sibling script's published value was wrong; the output was truncated before the value/reason, so the cause is unknown. Not reproduced in 20+ further loaded runs (16/24/48 load processes, load avg up to 358) or 20 unloaded runs.

Hypothesis (unverified): the test's 300 ms in-process run budget expiring under load. If it recurs, capture the failing value's status/reason; if it is a budget timeout, raise the test's run budget (which also sets the worker soft deadline) rather than weakening any assertion. Watch CI for it.

Seen once while fixing #798 (commit 304d1f1e). In the first loaded run of the reworked ShadowModeTests runaway test (16 `yes` processes, load avg ~64) the sibling script's published value was wrong; the output was truncated before the value/reason, so the cause is unknown. Not reproduced in 20+ further loaded runs (16/24/48 load processes, load avg up to 358) or 20 unloaded runs. Hypothesis (unverified): the test's 300 ms in-process run budget expiring under load. If it recurs, capture the failing value's status/reason; if it is a budget timeout, raise the test's run budget (which also sets the worker soft deadline) rather than weakening any assertion. Watch CI for it.

dohertj2 added the type:test priority:p3 labels 2026-11-10 21:39:41 -05:00

dohertj2 referenced this issue

2026-11-10 22:32:14 -05:00

ScriptWorker shadow isolation timing test includes cold compile and fails the 250ms bound on baseline #798

dohertj2 referenced this issue

2026-11-11 00:05:59 -05:00

Out-of-process script soft deadline includes pipe round trip and worker scheduling delay #827

dohertj2 referenced this issue

2026-11-11 00:05:59 -05:00

Runtime.Tests: DriverHostActorSourceRemovedTests late-burst test flaked once #828

dohertj2 referenced this issue from a commit

2026-11-11 00:48:23 -05:00

test(scriptworker): self-diagnosing ShadowModeTests runaway failures; 2 s run budget (#811)

dohertj2 commented

2026-11-11 00:49:03 -05:00

Fixed on master (merge 83e8cf76; db057548, test-only). Reproduced: 3/8 failures at load avg ~320–390 (32 yes + nice -n 20). Every failure was the sibling's shadow run in the worker missing the test's 300 ms soft deadline (shadow Failed [budget] script timed out after 0.3s) — the published value was correct in every run. Test run budget raised to the 2 s production default (named RunawayRunBudget); no assertion weakened; failures are now self-diagnosing (value/status/reason/timing, guard counters, thread pool, GC, logs). After the fix 10/10 at load avg up to 588. Matching the original single failure to this cause is a strong inference, not proof. Design follow-up: #827 (soft deadline includes transport/scheduling delay). Runtime flake → #828.

Fixed on master (merge 83e8cf76; db057548, test-only). Reproduced: 3/8 failures at load avg ~320–390 (32 `yes` + `nice -n 20`). Every failure was the sibling's **shadow** run in the worker missing the test's 300 ms soft deadline (`shadow Failed [budget] script timed out after 0.3s`) — the published value was correct in every run. Test run budget raised to the 2 s production default (named `RunawayRunBudget`); no assertion weakened; failures are now self-diagnosing (value/status/reason/timing, guard counters, thread pool, GC, logs). After the fix 10/10 at load avg up to 588. Matching the original single failure to this cause is a strong inference, not proof. Design follow-up: #827 (soft deadline includes transport/scheduling delay). Runtime flake → #828.

dohertj2 closed this issue

2026-11-11 00:49:03 -05:00

dohertj2 referenced this issue

2026-11-11 03:27:28 -05:00

Flaky tests under heavy machine load (4 seen during the #815–#833 batch) #843

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#811