merge(agent-c): rows 91/95/96 — compile-cache tests on key-scoped delta assertions; Watch/Stop/ExpectTerminated at all 23 fast-tick actor creation sites in the three remaining AuditLog.Tests classes; per-suite xunit maxParallelThreads:4 for the four MSSQL-fixture suites (matches the row-82 semaphore slots) + Test Parallelism Policy doc

This commit is contained in:
Joseph Doherty
2026-08-17 05:28:51 -04:00
15 changed files with 428 additions and 54 deletions
+58
View File
@@ -46,6 +46,64 @@ Each service has a dedicated document with configuration details, verification s
- [test_infra_playwright.md](test_infra_playwright.md) — Playwright browser server (Central UI testing)
- Traefik LB — see `docker/README.md` and `docker/traefik/` (runs with the cluster, not in `infra/`)
## Test Parallelism Policy
There are **two independent parallelism axes**, and they are bounded by two different mechanisms.
Conflating them is what made the row-82/91/96 flake family hard to read: a fix on one axis leaves the
other untouched.
| Axis | What runs at once | Bounded by |
|------|-------------------|-----------|
| **In-process** — xunit collection threads | Test collections *within one test assembly*. Default is `Environment.ProcessorCount` (14 on the current dev box). | `xunit.runner.json` → `maxParallelThreads`, per suite. This section. |
| **Cross-process** — `dotnet test` on the solution | One test *process* per test project, several at a time. xunit cannot see across them. | For the MS SQL migration path only: the machine-wide `sp_getapplock` counting semaphore in `MsSqlMigrationFixture` (4 slots, `SCADABRIDGE_MSSQL_TEST_MIGRATE_SLOTS`) — see [test_infra_db.md](test_infra_db.md). Nothing bounds it in general. |
### Which suites carry a runner config, and why
The cap is **per suite, not global**: pure unit suites (Commons, ScriptAnalysis, TemplateEngine,
Transport, …) stay uncapped for speed. A suite gets a config only when a test *class* costs a real
external or OS-level resource, so that `ProcessorCount` of them at once is a resource question rather
than a CPU question.
| Suite | Setting | Why |
|-------|---------|-----|
| `ConfigurationDatabase.Tests` | `maxParallelThreads: 4` | Owns `MsSqlMigrationFixture`: every MSSQL-backed test **class** does a `CREATE DATABASE` plus the full ~100-migration chain against the one shared dev SQL Server. 4 matches the `sp_getapplock` slot count so the in-process and cross-process limits agree. |
| `AuditLog.Tests` | `maxParallelThreads: 4` | Same fixture (by project reference), plus most classes stand up an Akka TestKit `ActorSystem` with fast-ticking actors. |
| `SiteCallAudit.Tests` | `maxParallelThreads: 4` | Same fixture (by project reference) plus a TestKit `ActorSystem`. |
| `Host.Tests` | `maxParallelThreads: 4` | Classes boot real ASP.NET Core hosts and real Akka `ActorSystem`s — Akka.Remote binds real sockets, Kestrel binds real ports, gRPC channels dial them. |
| `IntegrationTests` | `parallelizeAssembly: false`, `parallelizeTestCollections: false` | Pre-existing. Fully serialized; a thread cap would be redundant. |
| `CentralUI.PlaywrightTests` | `maxParallelThreads: 1` | Pre-existing, and the strictest setting in the repo — the suite drives a browser against a **live rig**, so shared UI state, not CPU, is the constraint. |
| `StoreAndForward.Tests` | `parallelizeTestCollections: true` | Pre-existing; that is the default, so this file is a no-op pin. Uncapped by choice — SQLite files are per-test. |
### Rules
- **Cap threads, do not serialize.** `maxParallelThreads` throttles how many collections run at once;
`parallelizeTestCollections: false` runs them one at a time forever. Reach for the second only when a
suite has a genuinely global resource it cannot partition (`IntegrationTests` and the Playwright suite
are the only two, and both predate this policy).
- **A cap is not a substitute for test isolation.** Rows 82/91/95 were all *correctness* bugs — leaked
ticking actors, absolute assertions on process-global counters — that a cap only makes rarer. Fix the
test; the cap is for resource pressure.
- **The json must be copied to the output directory.** The .NET SDK does not do this automatically:
each capped `.csproj` carries
`<Content Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" />`. Without it the file
is silently ignored and the suite runs uncapped.
- **Default for a new test project: no runner config.** Add one only when the project acquires a
per-class external resource (an MSSQL fixture, a real host/port, a browser). Record the reason in the
`.csproj` comment next to the `Content` item.
### Measured cost
The cap is close to free, because these suites are latency-bound on an external resource rather than
CPU-bound (CPU utilisation stayed at 14–25 % for the two slowest). Measured standalone on a 14-core dev
box, uncapped vs. `maxParallelThreads: 4`:
| Suite | Tests | Uncapped | Capped at 4 |
|-------|-------|----------|-------------|
| `AuditLog.Tests` | 383 | 1 m 52 s | 1 m 51 s |
| `Host.Tests` | 520 | 4 m 5 s | 4 m 8 s |
| `ConfigurationDatabase.Tests` | 396 | 10 s | 12 s |
| `SiteCallAudit.Tests` | 66 | 8 s | 8 s |
## Remote Test Infrastructure
In addition to the local Docker services, the following remote services are available for testing against real AVEVA System Platform hardware.
+4 -1
View File
@@ -43,7 +43,10 @@ Three behaviours are worth knowing when you look at this server during a test ru
solution-level `dotnet test` runs several MSSQL-backed test *processes* at once, so the throttle has
to live on the shared server. Without it, dozens of simultaneous fresh-database migrations saturated
the container and individual DDL statements blew past EF's default 30-second command timeout,
failing whole test classes at fixture construction (deferred-work register row 82).
failing whole test classes at fixture construction (deferred-work register row 82). The
*in-process* half of the same problem is bounded separately, by a `maxParallelThreads: 4` cap in
each MSSQL-backed suite's `xunit.runner.json` — deliberately the same number, so the two limits
agree. See [test_infra.md → Test Parallelism Policy](test_infra.md#test-parallelism-policy).
- **Migration commands use a 300s timeout**, and CREATE/DROP DATABASE use 120s. Under a bounded queue
the server is busy but progressing, so the correct response to a slow statement is to wait.
- **Strays are swept.** Each test process drops, once at start and best-effort, any