test(storeandforward): isolate the process-global queue-depth gauge — register row 32

QueueDepthGaugeTests failed intermittently whenever a sibling test class started a
StoreAndForwardService concurrently. `scadabridge.store_and_forward.queue.depth` is an
ObservableGauge on ScadaBridgeTelemetry's static meter reading ONE process-global
provider slot; every StartAsync writes it, last registrant wins. A concurrent StartAsync
clobbered the gauge mid-assertion and the test read the sibling's depth (observed:
expected 0, actual 6 — 2 runs in 5 with parallel collections enabled).

The global slot cannot be swapped for a per-test meter: three of the six facts assert
the slot-ownership semantics themselves (compare-and-clear identity, stop/restart
re-seed), so the slot IS their subject. The correct isolation is mutual exclusion —
QueueDepthGaugeTests, StoreAndForwardServiceTests and CachedCallAttemptEmissionTests
(the three classes that call StartAsync) now share the QueueDepthGaugeCollection, which
xUnit runs serially; the other eight classes stay parallel.

With real isolation in place the assembly-wide `parallelizeTestCollections: false`
blanket switch is no longer needed and is flipped back on, so the isolation is actually
exercised rather than papered over.

QueueDepthGaugeCollectionGuardTests is a staleness self-test: it scans this project's
sources (anchored by [CallerFilePath], not the output dir) for files that both construct
and start a service, and fails if one omits the collection attribute — so a new test
class cannot silently reintroduce the flake. Negative-controlled: dropping the attribute
from StoreAndForwardServiceTests makes the guard fail with the offending file name, and
the original gauge flake reappears in the same run.

Verification: 8 consecutive full-suite runs green with parallel collections ON (135
tests), plus 5 more after the negative control was reverted. Test-only change; no src/
involvement.
This commit is contained in:
Joseph Doherty
2026-08-15 12:44:14 -04:00
parent ef876b38c5
commit 65b19516c8
5 changed files with 143 additions and 1 deletions
@@ -9,7 +9,13 @@ namespace ZB.MOM.WW.ScadaBridge.StoreAndForward.Tests;
/// <summary>
/// WP-10/12/13/14: Tests for the StoreAndForwardService retry engine and management.
///
/// Register row 32: several facts here call <c>StartAsync</c>, which writes the
/// process-global queue-depth provider slot, so this class joins
/// <see cref="QueueDepthGaugeCollection"/> to stay serialized against
/// <see cref="QueueDepthGaugeTests"/>.
/// </summary>
[Collection(QueueDepthGaugeCollection.Name)]
public class StoreAndForwardServiceTests : IAsyncLifetime, IDisposable
{
private readonly TestLocalDb _localDb;