test(mssql): close register row 82 — throttle cross-process create+migrate, stop leaked test actors
AuditLog.Tests flaked under parallel full-solution runs. Three distinct mechanisms, all load-sensitive and all green in isolation. 1. MsSqlMigrationFixture create+migrate saturation (the register's headline). Database NAMING was already isolated (a guid per fixture), but nothing bounded how many fixtures ran CREATE DATABASE + the ~100-migration chain at once. xunit parallelises processor-count collections PER PROCESS, and a solution-level `dotnet test` starts several MSSQL-backed processes together, so the dev container saw dozens of simultaneous fresh-database migrations. That is resource saturation, not object-level locking, and it pushed individual DDL statements past EF's default 30s command timeout — every failure had the identical fingerprint the register recorded: MsSqlMigrationFixture..ctor -> Migrator.MigrateAsync -> SqlException "Execution Timeout Expired" (error 258), which xunit surfaces as "Class fixture type ... threw in its constructor" against every test in the class. Fix: a machine-wide counting semaphore over the create+migrate window, built from sp_getapplock on master (ScadaBridgeTestMigrate/0..N-1, default 4 slots, SCADABRIDGE_MSSQL_TEST_MIGRATE_SLOTS to override). An xunit collection cannot do this — the contention is cross-PROCESS, so the arbiter has to live on the shared server. The gate connection is unpooled (a pooled connection defers the implicit release of a session-scoped app lock until reuse) and releases explicitly; a crashed process drops its connection and SQL releases the lock, so the gate cannot wedge the suite, and a deadline miss proceeds ungated rather than failing. Migration commands also get a 300s timeout and CREATE/DROP DATABASE 120s: under a bounded queue the server is busy but progressing, so waiting is the correct response. Plus a once-per-process best-effort sweep of ScadaBridgeAuditMigTest_* databases older than 2h so a killed run does not leave strays, and the connectivity probe now retries (3 attempts) — a busy server misread as an absent one would silently turn every MSSQL test into a Skip, which reads green and is worse than a red. 2. PartitionPurgeTests leaked AuditLogPurgeActors. Each ticks every 100ms and switches out the Jan-2026 partition; they were only stopped implicitly by TestKit Dispose -> Shutdown, which is timeout-bounded and does not fail when it expires. Under load a previous test's actor was still ticking against the shared per-class fixture DB, so PerChannelOverride_... found its seeded Jan-15-2026 row already gone (Expected: 1, Actual: 0). 3. NotifyDispatcherAuditTrailTests leaked NotificationOutboxActors, same shape. The dispatcher claims EVERY due row, not just the test's siteId, so a leaked actor delivered the next test's notification through its own exhausted QueuedOutcomeAdapter — a transient outcome, leaving the row in Retrying (Expected: Delivered, Actual: Retrying) and audit counts short. Both fixed by stopping the actor inside the test that created it (Watch/Stop/ExpectTerminated), removing the dependency on shutdown timing. Evidence. Reproducer: 10 concurrent ConfigurationDatabase.Tests processes against the one dev SQL Server — before, 10/10 processes failed (17-77 failures each, 100% the ctor->MigrateAsync timeout); after, 3 consecutive runs 10/10 green, 395/395 each. Realistic shape (3x AuditLog.Tests + 1x ConfigurationDatabase.Tests concurrent): first post-fix run still showed the NotifyDispatcher leak, then 3 consecutive clean runs. Standalone: AuditLog.Tests 383/383, ConfigurationDatabase.Tests 395/395 (12s, unchanged from baseline), SiteCallAudit.Tests 66/66. No orphan databases or stuck app locks left behind. docs/test_infra/test_infra_db.md documents the ephemeral per-test-class databases, the throttle and its env override, the timeouts, and the stray sweep.
This commit is contained in:
@@ -26,6 +26,40 @@ The test database uses Microsoft SQL Server 2022 Developer Edition running in Do
|
||||
|
||||
Both databases are created by `infra/mssql/setup.sql`. EF Core migrations populate the `ScadaBridgeConfig` schema. The `ScadaBridgeMachineData` database is seeded with sample tables and stored procedures by `infra/mssql/machinedata_seed.sql`.
|
||||
|
||||
### Ephemeral per-test-class databases
|
||||
|
||||
`MsSqlMigrationFixture` (`tests/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase.Tests/Migrations/`) creates a
|
||||
throwaway `ScadaBridgeAuditMigTest_<guid>` database per test class, applies the full EF migration chain
|
||||
to it, and drops it on teardown. Both `ConfigurationDatabase.Tests` and `AuditLog.Tests` use it — the
|
||||
latter by project reference, so there is one fixture implementation, not a copy. Seeing a handful of
|
||||
these databases come and go while tests run is normal; seeing them survive a run is not (see below).
|
||||
|
||||
Three behaviours are worth knowing when you look at this server during a test run:
|
||||
|
||||
- **Concurrent create+migrate is throttled machine-wide.** Fixtures take a slot from a counting
|
||||
semaphore built on `sp_getapplock` in `master` (resources `ScadaBridgeTestMigrate/0..N-1`) and hold it
|
||||
across `CREATE DATABASE` + the migration chain. The default is 4 slots; override with
|
||||
`SCADABRIDGE_MSSQL_TEST_MIGRATE_SLOTS`. An xunit collection attribute cannot do this job — a
|
||||
solution-level `dotnet test` runs several MSSQL-backed test *processes* at once, so the throttle has
|
||||
to live on the shared server. Without it, dozens of simultaneous fresh-database migrations saturated
|
||||
the container and individual DDL statements blew past EF's default 30-second command timeout,
|
||||
failing whole test classes at fixture construction (deferred-work register row 82).
|
||||
- **Migration commands use a 300s timeout**, and CREATE/DROP DATABASE use 120s. Under a bounded queue
|
||||
the server is busy but progressing, so the correct response to a slow statement is to wait.
|
||||
- **Strays are swept.** Each test process drops, once at start and best-effort, any
|
||||
`ScadaBridgeAuditMigTest_*` database older than 2 hours, so a run killed before teardown does not
|
||||
leave databases accumulating. To clear them by hand:
|
||||
|
||||
```bash
|
||||
docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
|
||||
-S localhost -U sa -P 'ScadaBridge_Dev1#' -C \
|
||||
-Q "SELECT name, create_date FROM sys.databases WHERE name LIKE 'ScadaBridgeAuditMigTest_%'"
|
||||
```
|
||||
|
||||
Overriding the server the fixture targets (CI, a non-default port) is done with
|
||||
`SCADABRIDGE_MSSQL_TEST_CONN`. When no server answers, the fixture reports `Available = false` and the
|
||||
tests report as **Skipped**, not passed.
|
||||
|
||||
## Data Persistence
|
||||
|
||||
SQL data is stored in the named Docker volume `scadabridge-mssql-data`. Data survives container restarts and `docker compose down`. To reset the database completely:
|
||||
|
||||
Reference in New Issue
Block a user