AuditLog.Tests flaked under parallel full-solution runs. Three distinct
mechanisms, all load-sensitive and all green in isolation.
1. MsSqlMigrationFixture create+migrate saturation (the register's headline).
Database NAMING was already isolated (a guid per fixture), but nothing
bounded how many fixtures ran CREATE DATABASE + the ~100-migration chain at
once. xunit parallelises processor-count collections PER PROCESS, and a
solution-level `dotnet test` starts several MSSQL-backed processes together,
so the dev container saw dozens of simultaneous fresh-database migrations.
That is resource saturation, not object-level locking, and it pushed
individual DDL statements past EF's default 30s command timeout — every
failure had the identical fingerprint the register recorded:
MsSqlMigrationFixture..ctor -> Migrator.MigrateAsync -> SqlException
"Execution Timeout Expired" (error 258), which xunit surfaces as
"Class fixture type ... threw in its constructor" against every test in the
class.
Fix: a machine-wide counting semaphore over the create+migrate window, built
from sp_getapplock on master (ScadaBridgeTestMigrate/0..N-1, default 4 slots,
SCADABRIDGE_MSSQL_TEST_MIGRATE_SLOTS to override). An xunit collection cannot
do this — the contention is cross-PROCESS, so the arbiter has to live on the
shared server. The gate connection is unpooled (a pooled connection defers the
implicit release of a session-scoped app lock until reuse) and releases
explicitly; a crashed process drops its connection and SQL releases the lock,
so the gate cannot wedge the suite, and a deadline miss proceeds ungated
rather than failing. Migration commands also get a 300s timeout and
CREATE/DROP DATABASE 120s: under a bounded queue the server is busy but
progressing, so waiting is the correct response. Plus a once-per-process
best-effort sweep of ScadaBridgeAuditMigTest_* databases older than 2h so a
killed run does not leave strays, and the connectivity probe now retries
(3 attempts) — a busy server misread as an absent one would silently turn
every MSSQL test into a Skip, which reads green and is worse than a red.
2. PartitionPurgeTests leaked AuditLogPurgeActors. Each ticks every 100ms and
switches out the Jan-2026 partition; they were only stopped implicitly by
TestKit Dispose -> Shutdown, which is timeout-bounded and does not fail when
it expires. Under load a previous test's actor was still ticking against the
shared per-class fixture DB, so PerChannelOverride_... found its seeded
Jan-15-2026 row already gone (Expected: 1, Actual: 0).
3. NotifyDispatcherAuditTrailTests leaked NotificationOutboxActors, same shape.
The dispatcher claims EVERY due row, not just the test's siteId, so a leaked
actor delivered the next test's notification through its own exhausted
QueuedOutcomeAdapter — a transient outcome, leaving the row in Retrying
(Expected: Delivered, Actual: Retrying) and audit counts short.
Both fixed by stopping the actor inside the test that created it
(Watch/Stop/ExpectTerminated), removing the dependency on shutdown timing.
Evidence. Reproducer: 10 concurrent ConfigurationDatabase.Tests processes
against the one dev SQL Server — before, 10/10 processes failed (17-77 failures
each, 100% the ctor->MigrateAsync timeout); after, 3 consecutive runs 10/10
green, 395/395 each. Realistic shape (3x AuditLog.Tests + 1x
ConfigurationDatabase.Tests concurrent): first post-fix run still showed the
NotifyDispatcher leak, then 3 consecutive clean runs. Standalone: AuditLog.Tests
383/383, ConfigurationDatabase.Tests 395/395 (12s, unchanged from baseline),
SiteCallAudit.Tests 66/66. No orphan databases or stuck app locks left behind.
docs/test_infra/test_infra_db.md documents the ephemeral per-test-class
databases, the throttle and its env override, the timeouts, and the stray sweep.
Joseph Doherty
da23f2dcae