merge(agent-c): rows 91/95/96 — compile-cache tests on key-scoped delta assertions; Watch/Stop/ExpectTerminated at all 23 fast-tick actor creation sites in the three remaining AuditLog.Tests classes; per-suite xunit maxParallelThreads:4 for the four MSSQL-fixture suites (matches the row-82 semaphore slots) + Test Parallelism Policy doc
This commit is contained in:
@@ -46,6 +46,64 @@ Each service has a dedicated document with configuration details, verification s
|
||||
- [test_infra_playwright.md](test_infra_playwright.md) — Playwright browser server (Central UI testing)
|
||||
- Traefik LB — see `docker/README.md` and `docker/traefik/` (runs with the cluster, not in `infra/`)
|
||||
|
||||
## Test Parallelism Policy
|
||||
|
||||
There are **two independent parallelism axes**, and they are bounded by two different mechanisms.
|
||||
Conflating them is what made the row-82/91/96 flake family hard to read: a fix on one axis leaves the
|
||||
other untouched.
|
||||
|
||||
| Axis | What runs at once | Bounded by |
|
||||
|------|-------------------|-----------|
|
||||
| **In-process** — xunit collection threads | Test collections *within one test assembly*. Default is `Environment.ProcessorCount` (14 on the current dev box). | `xunit.runner.json` → `maxParallelThreads`, per suite. This section. |
|
||||
| **Cross-process** — `dotnet test` on the solution | One test *process* per test project, several at a time. xunit cannot see across them. | For the MS SQL migration path only: the machine-wide `sp_getapplock` counting semaphore in `MsSqlMigrationFixture` (4 slots, `SCADABRIDGE_MSSQL_TEST_MIGRATE_SLOTS`) — see [test_infra_db.md](test_infra_db.md). Nothing bounds it in general. |
|
||||
|
||||
### Which suites carry a runner config, and why
|
||||
|
||||
The cap is **per suite, not global**: pure unit suites (Commons, ScriptAnalysis, TemplateEngine,
|
||||
Transport, …) stay uncapped for speed. A suite gets a config only when a test *class* costs a real
|
||||
external or OS-level resource, so that `ProcessorCount` of them at once is a resource question rather
|
||||
than a CPU question.
|
||||
|
||||
| Suite | Setting | Why |
|
||||
|-------|---------|-----|
|
||||
| `ConfigurationDatabase.Tests` | `maxParallelThreads: 4` | Owns `MsSqlMigrationFixture`: every MSSQL-backed test **class** does a `CREATE DATABASE` plus the full ~100-migration chain against the one shared dev SQL Server. 4 matches the `sp_getapplock` slot count so the in-process and cross-process limits agree. |
|
||||
| `AuditLog.Tests` | `maxParallelThreads: 4` | Same fixture (by project reference), plus most classes stand up an Akka TestKit `ActorSystem` with fast-ticking actors. |
|
||||
| `SiteCallAudit.Tests` | `maxParallelThreads: 4` | Same fixture (by project reference) plus a TestKit `ActorSystem`. |
|
||||
| `Host.Tests` | `maxParallelThreads: 4` | Classes boot real ASP.NET Core hosts and real Akka `ActorSystem`s — Akka.Remote binds real sockets, Kestrel binds real ports, gRPC channels dial them. |
|
||||
| `IntegrationTests` | `parallelizeAssembly: false`, `parallelizeTestCollections: false` | Pre-existing. Fully serialized; a thread cap would be redundant. |
|
||||
| `CentralUI.PlaywrightTests` | `maxParallelThreads: 1` | Pre-existing, and the strictest setting in the repo — the suite drives a browser against a **live rig**, so shared UI state, not CPU, is the constraint. |
|
||||
| `StoreAndForward.Tests` | `parallelizeTestCollections: true` | Pre-existing; that is the default, so this file is a no-op pin. Uncapped by choice — SQLite files are per-test. |
|
||||
|
||||
### Rules
|
||||
|
||||
- **Cap threads, do not serialize.** `maxParallelThreads` throttles how many collections run at once;
|
||||
`parallelizeTestCollections: false` runs them one at a time forever. Reach for the second only when a
|
||||
suite has a genuinely global resource it cannot partition (`IntegrationTests` and the Playwright suite
|
||||
are the only two, and both predate this policy).
|
||||
- **A cap is not a substitute for test isolation.** Rows 82/91/95 were all *correctness* bugs — leaked
|
||||
ticking actors, absolute assertions on process-global counters — that a cap only makes rarer. Fix the
|
||||
test; the cap is for resource pressure.
|
||||
- **The json must be copied to the output directory.** The .NET SDK does not do this automatically:
|
||||
each capped `.csproj` carries
|
||||
`<Content Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" />`. Without it the file
|
||||
is silently ignored and the suite runs uncapped.
|
||||
- **Default for a new test project: no runner config.** Add one only when the project acquires a
|
||||
per-class external resource (an MSSQL fixture, a real host/port, a browser). Record the reason in the
|
||||
`.csproj` comment next to the `Content` item.
|
||||
|
||||
### Measured cost
|
||||
|
||||
The cap is close to free, because these suites are latency-bound on an external resource rather than
|
||||
CPU-bound (CPU utilisation stayed at 14–25 % for the two slowest). Measured standalone on a 14-core dev
|
||||
box, uncapped vs. `maxParallelThreads: 4`:
|
||||
|
||||
| Suite | Tests | Uncapped | Capped at 4 |
|
||||
|-------|-------|----------|-------------|
|
||||
| `AuditLog.Tests` | 383 | 1 m 52 s | 1 m 51 s |
|
||||
| `Host.Tests` | 520 | 4 m 5 s | 4 m 8 s |
|
||||
| `ConfigurationDatabase.Tests` | 396 | 10 s | 12 s |
|
||||
| `SiteCallAudit.Tests` | 66 | 8 s | 8 s |
|
||||
|
||||
## Remote Test Infrastructure
|
||||
|
||||
In addition to the local Docker services, the following remote services are available for testing against real AVEVA System Platform hardware.
|
||||
|
||||
@@ -43,7 +43,10 @@ Three behaviours are worth knowing when you look at this server during a test ru
|
||||
solution-level `dotnet test` runs several MSSQL-backed test *processes* at once, so the throttle has
|
||||
to live on the shared server. Without it, dozens of simultaneous fresh-database migrations saturated
|
||||
the container and individual DDL statements blew past EF's default 30-second command timeout,
|
||||
failing whole test classes at fixture construction (deferred-work register row 82).
|
||||
failing whole test classes at fixture construction (deferred-work register row 82). The
|
||||
*in-process* half of the same problem is bounded separately, by a `maxParallelThreads: 4` cap in
|
||||
each MSSQL-backed suite's `xunit.runner.json` — deliberately the same number, so the two limits
|
||||
agree. See [test_infra.md → Test Parallelism Policy](test_infra.md#test-parallelism-policy).
|
||||
- **Migration commands use a 300s timeout**, and CREATE/DROP DATABASE use 120s. Under a bounded queue
|
||||
the server is busy but progressing, so the correct response to a slow statement is to wait.
|
||||
- **Strays are swept.** Each test process drops, once at start and best-effort, any
|
||||
|
||||
Reference in New Issue
Block a user