Flaky under load: Galaxy EventPumpBoundedChannelTests.Drops_newest_when_channel_fills_and_records_metric #503

New Issue

2026-07-25T22:44:54-04:00

dohertj2 commented

2026-07-25 22:44:54 -04:00

Found while running the full unit-test sweep on merged master (28c28667). Pre-existing and unrelated to that merge — EventPumpBoundedChannelTests.cs is untouched by it, and the file's last change (560b327e) predates the work.

Test: ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests.Runtime.EventPumpBoundedChannelTests.Drops_newest_when_channel_fills_and_records_metric

Failure: Shouldly.ShouldAssertException : counters.Received — the drop metric did not reach the expected count.

Reproduction (measured):

25 consecutive runs of Driver.Galaxy.Tests in isolation -> 0 failures.
12 runs with Runtime.Tests + AdminUI.Tests + Core.Tests running concurrently -> 1 failure (run 11).
Also failed once during a full 41-project sweep.

So it needs CPU contention, like the #500 family did. Recipe that catches it:

for i in $(seq 1 12); do
  dotnet test tests/Server/…Runtime.Tests --no-build > /dev/null 2>&1 &
  dotnet test tests/Server/…AdminUI.Tests --no-build > /dev/null 2>&1 &
  dotnet test tests/Core/…Core.Tests     --no-build > /dev/null 2>&1 &
  dotnet test tests/Drivers/…Galaxy.Tests --no-build \
    --logger "trx;LogFileName=/tmp/gal/run$i.trx" 2>&1 | grep -E "Failed!|Passed!"
  wait
done

Always capture trx — a summary-only run loses the test name and costs a full re-run to recover.

Likely shape (not yet confirmed — I did not diagnose this one). The test drives a bounded channel to overflow and asserts a drop count. Counts like that are usually observed after a producer/consumer race, so the assertion can be evaluated before the drops have been recorded, or the consumer can drain fast enough under different scheduling that fewer drops occur than the test assumes. Worth checking which of those it is before changing anything — #500 had two hypotheses that looked obvious and were both wrong under measurement.

See #500 for the presence-vs-absence assertion rule, which resolved eight tests of a similar character in Runtime.Tests.

Found while running the full unit-test sweep on merged master (`28c28667`). **Pre-existing and unrelated to that merge** — `EventPumpBoundedChannelTests.cs` is untouched by it, and the file's last change (`560b327e`) predates the work. **Test:** `ZB.MOM.WW.OtOpcUa.Driver.Galaxy.Tests.Runtime.EventPumpBoundedChannelTests.Drops_newest_when_channel_fills_and_records_metric` **Failure:** `Shouldly.ShouldAssertException : counters.Received` — the drop metric did not reach the expected count. **Reproduction (measured):** - 25 consecutive runs of `Driver.Galaxy.Tests` **in isolation** -> 0 failures. - 12 runs **with `Runtime.Tests` + `AdminUI.Tests` + `Core.Tests` running concurrently** -> **1 failure** (run 11). - Also failed once during a full 41-project sweep. So it needs CPU contention, like the #500 family did. Recipe that catches it: ```bash for i in $(seq 1 12); do dotnet test tests/Server/…Runtime.Tests --no-build > /dev/null 2>&1 & dotnet test tests/Server/…AdminUI.Tests --no-build > /dev/null 2>&1 & dotnet test tests/Core/…Core.Tests --no-build > /dev/null 2>&1 & dotnet test tests/Drivers/…Galaxy.Tests --no-build \ --logger "trx;LogFileName=/tmp/gal/run$i.trx" 2>&1 | grep -E "Failed!|Passed!" wait done ``` Always capture `trx` — a summary-only run loses the test name and costs a full re-run to recover. **Likely shape (not yet confirmed — I did not diagnose this one).** The test drives a bounded channel to overflow and asserts a *drop* count. Counts like that are usually observed after a producer/consumer race, so the assertion can be evaluated before the drops have been recorded, or the consumer can drain fast enough under different scheduling that fewer drops occur than the test assumes. Worth checking which of those it is before changing anything — #500 had two hypotheses that looked obvious and were both wrong under measurement. See #500 for the presence-vs-absence assertion rule, which resolved eight tests of a similar character in `Runtime.Tests`.

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#503