Galaxy.Tests: remaining short fixed waits fail under heavy parallel load #858

New Issue

2026-11-11T10:01:34-05:00

dohertj2 commented

2026-11-11 10:01:34 -05:00

Found while verifying #856. Under extreme load (14 parallel full Galaxy.Tests runs + 14 CPU burners) 11/70 runs failed in tests with short fixed waits: GatewayGalaxyAlarmFeedTests.Decodes_active_alarm_snapshot_then_live_transition (2 s, 6×), DeployWatcherTests (2 tests, 4×), GalaxyDriverSubscribeTests.SubscribeAsync_TwoSubscriptions_SameTag_FanOutOnePerSubscription, GalaxyDriverRawPathKeyingTests, EventPumpStreamFaultTests, ReconnectSupervisorTests. Never seen in normal serial runs (0/40 under 28 burners). Same class as #843: lengthen presence waits, use the self-diagnosing WaitUntilAsync overload added in #856, never assert absence by a short wait. Load scripts: a parallel dotnet test runner loop.

Found while verifying #856. Under extreme load (14 parallel full Galaxy.Tests runs + 14 CPU burners) 11/70 runs failed in tests with short fixed waits: `GatewayGalaxyAlarmFeedTests.Decodes_active_alarm_snapshot_then_live_transition` (2 s, 6×), `DeployWatcherTests` (2 tests, 4×), `GalaxyDriverSubscribeTests.SubscribeAsync_TwoSubscriptions_SameTag_FanOutOnePerSubscription`, `GalaxyDriverRawPathKeyingTests`, `EventPumpStreamFaultTests`, `ReconnectSupervisorTests`. Never seen in normal serial runs (0/40 under 28 burners). Same class as #843: lengthen presence waits, use the self-diagnosing `WaitUntilAsync` overload added in #856, never assert absence by a short wait. Load scripts: a parallel `dotnet test` runner loop.

dohertj2 added the type:test priority:p3 labels 2026-11-11 10:01:34 -05:00

dohertj2 referenced this issue

2026-11-11 11:28:48 -05:00

Flaky: GalaxyDriverFixRound2Tests.A_fetch_completing_after_init_populates_host_statuses (10 s timeout under load) #856

dohertj2 commented

2026-11-11 11:28:49 -05:00

Fixed (4fdc3708, test-only). Pushed to master in e857135e. Every failure was a 1–3 s wait on a Task.Run loop that hadn't been scheduled yet under thread-pool starvation; no production race (checked each). Shared TestWait (30 s, self-diagnosing), absence checks now use a positive control, fixed sleeps replaced by real signals. Before 16/70 runs failed (14 parallel + burners + ForceMinWorkerThreads=1), after 0/350.

Fixed (`4fdc3708`, test-only). Pushed to master in `e857135e`. Every failure was a 1–3 s wait on a `Task.Run` loop that hadn't been scheduled yet under thread-pool starvation; no production race (checked each). Shared `TestWait` (30 s, self-diagnosing), absence checks now use a positive control, fixed sleeps replaced by real signals. Before 16/70 runs failed (14 parallel + burners + `ForceMinWorkerThreads=1`), after 0/350.

dohertj2 closed this issue

2026-11-11 11:28:49 -05:00

dohertj2 referenced this issue from a commit

2026-11-11 11:28:50 -05:00

test(galaxy): presence waits survive a starved/loaded runner; absence asserted behind positive controls (#858)

dohertj2 referenced this issue from a commit

2026-11-11 11:28:50 -05:00

Merge #858 Galaxy.Tests: generous self-diagnosing presence waits, positive-control absence checks

dohertj2 referenced this issue from a commit

2026-11-11 11:28:50 -05:00

fix(galaxy): #859 review — read path window, atomic promote, session epoch, per-call windows

Sign in to join this conversation.