Joseph Doherty
d2d4db492b
maxParallelThreads was unset everywhere and only 3 of ~20 test projects carried an xunit.runner.json at all, so xunit ran Environment.ProcessorCount collections at once (14 on the dev box) in every suite. That is the general load multiplier behind the row-82/91/95 flake family. Policy: per-suite caps, NOT a global one. Pure unit suites stay uncapped for speed; a suite gets a config only when a test CLASS costs a real external or OS-level resource, so ProcessorCount of them at once is a resource question rather than a CPU question. Four suites qualify, all capped at maxParallelThreads: 4. - ConfigurationDatabase.Tests, AuditLog.Tests, SiteCallAudit.Tests — verified as the three that actually take MsSqlMigrationFixture (the latter two by project reference). Every MSSQL-backed test class means a CREATE DATABASE plus the ~100-migration chain against the one shared dev SQL Server. 4 is chosen to MATCH the default slot count of row 82's cross-process sp_getapplock migrate semaphore, so the in-process and cross-process limits agree rather than fighting. - Host.Tests — classes boot real ASP.NET Core hosts and real ActorSystems; Akka.Remote binds real sockets, Kestrel binds real ports, gRPC dials them. Threads are capped, collections are NOT serialized. IntegrationTests (parallelizeAssembly/parallelizeTestCollections false) and CentralUI.PlaywrightTests (maxParallelThreads 1) already carry stricter pre-existing settings and are left alone — the Playwright suite drives a browser against a live rig, where shared UI state, not CPU, is the constraint. Each capped csproj gains <Content Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" />; the SDK does not copy it automatically and without it the file is silently ignored. Verified by moving the copied file out of bin/ to take an uncapped baseline: ConfigurationDatabase.Tests ran at 216 % CPU uncapped vs 145 % capped, so the cap demonstrably takes effect. Cost is close to nil — these suites are latency-bound on an external resource, not CPU-bound (14-25 % utilisation for the two slowest). Standalone, uncapped -> capped: AuditLog 383 tests 1m52s -> 1m51s; Host 520 tests 4m05s -> 4m08s; ConfigurationDatabase 396 tests 10s -> 12s; SiteCallAudit 66 tests 8s -> 8s. All green. docs/test_infra/test_infra.md gains a Test Parallelism Policy section: the two axes (in-process collection threads = this cap; cross-process solution-level dotnet test = unbounded in general, bounded on the DB-migration path by the sp_getapplock semaphore), the per-suite table with reasons, the rules (cap threads rather than serialize; a cap is not a substitute for test isolation; the Content copy is required), and the default for new projects (no config until the project acquires a per-class external resource). test_infra_db.md cross-links it from the semaphore bullet.
191 lines
11 KiB
Markdown
191 lines
11 KiB
Markdown
# Test Infrastructure
|
||
|
||
This document describes the local Docker-based test infrastructure for ScadaBridge development. Seven services provide the external dependencies needed to run and test the system locally. The first seven run in `infra/docker-compose.yml`; Traefik runs alongside the cluster nodes in `docker/docker-compose.yml`.
|
||
|
||
## Services
|
||
|
||
| Service | Image | Port(s) | Config | Compose File |
|
||
|---------|-------|---------|--------|-------------|
|
||
| OPC UA Server | `mcr.microsoft.com/iotedge/opc-plc:latest` | 50000 (OPC UA), 8080 (web) | `infra/opcua/nodes.json` | `infra/` |
|
||
| OPC UA Server 2 | `mcr.microsoft.com/iotedge/opc-plc:latest` | 50010 (OPC UA), 8081 (web) | `infra/opcua/nodes.json` | `infra/` |
|
||
| LDAP Server | `glauth/glauth:latest` | 3893 | `scadaproj/infra/glauth/config.toml` | **Shared** — `zb-shared-glauth` on `10.100.0.35`; NOT started by `infra/` stack (retired 2026-06-04). See `scadaproj/infra/glauth/`. |
|
||
| MS SQL 2022 | `mcr.microsoft.com/mssql/server:2022-latest` | 1433 | `infra/mssql/setup.sql` | `infra/` |
|
||
| SMTP (Mailpit) | `axllent/mailpit:latest` | 1025 (SMTP), 8025 (web) | Environment vars | `infra/` |
|
||
| REST API (Flask) | Custom build (`infra/restapi/Dockerfile`) | 5200 | `infra/restapi/app.py` | `infra/` |
|
||
| Playwright | `mcr.microsoft.com/playwright:v1.58.2-noble` | 3000 (WebSocket) | Command args | `infra/` |
|
||
| Traefik LB | `traefik:v3.4` | 9000 (proxy), 8180 (dashboard) | `docker/traefik/` | `docker/` |
|
||
|
||
## Quick Start
|
||
|
||
```bash
|
||
cd infra
|
||
docker compose up -d
|
||
```
|
||
|
||
After the first startup, run the SQL setup and seed scripts:
|
||
|
||
```bash
|
||
docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
|
||
-S localhost -U sa -P 'ScadaBridge_Dev1#' -C \
|
||
-i /docker-entrypoint-initdb.d/setup.sql
|
||
|
||
docker exec -i scadabridge-mssql /opt/mssql-tools18/bin/sqlcmd \
|
||
-S localhost -U sa -P 'ScadaBridge_Dev1#' -C \
|
||
-i /docker-entrypoint-initdb.d/machinedata_seed.sql
|
||
```
|
||
|
||
## Per-Service Documentation
|
||
|
||
Each service has a dedicated document with configuration details, verification steps, and troubleshooting:
|
||
|
||
- [test_infra_opcua.md](test_infra_opcua.md) — OPC UA test server (Azure IoT OPC PLC)
|
||
- [test_infra_ldap.md](test_infra_ldap.md) — LDAP test server (GLAuth)
|
||
- [test_infra_db.md](test_infra_db.md) — MS SQL 2022 database
|
||
- [test_infra_smtp.md](test_infra_smtp.md) — SMTP test server (Mailpit)
|
||
- [test_infra_restapi.md](test_infra_restapi.md) — REST API test server (Flask)
|
||
- [test_infra_playwright.md](test_infra_playwright.md) — Playwright browser server (Central UI testing)
|
||
- Traefik LB — see `docker/README.md` and `docker/traefik/` (runs with the cluster, not in `infra/`)
|
||
|
||
## Test Parallelism Policy
|
||
|
||
There are **two independent parallelism axes**, and they are bounded by two different mechanisms.
|
||
Conflating them is what made the row-82/91/96 flake family hard to read: a fix on one axis leaves the
|
||
other untouched.
|
||
|
||
| Axis | What runs at once | Bounded by |
|
||
|------|-------------------|-----------|
|
||
| **In-process** — xunit collection threads | Test collections *within one test assembly*. Default is `Environment.ProcessorCount` (14 on the current dev box). | `xunit.runner.json` → `maxParallelThreads`, per suite. This section. |
|
||
| **Cross-process** — `dotnet test` on the solution | One test *process* per test project, several at a time. xunit cannot see across them. | For the MS SQL migration path only: the machine-wide `sp_getapplock` counting semaphore in `MsSqlMigrationFixture` (4 slots, `SCADABRIDGE_MSSQL_TEST_MIGRATE_SLOTS`) — see [test_infra_db.md](test_infra_db.md). Nothing bounds it in general. |
|
||
|
||
### Which suites carry a runner config, and why
|
||
|
||
The cap is **per suite, not global**: pure unit suites (Commons, ScriptAnalysis, TemplateEngine,
|
||
Transport, …) stay uncapped for speed. A suite gets a config only when a test *class* costs a real
|
||
external or OS-level resource, so that `ProcessorCount` of them at once is a resource question rather
|
||
than a CPU question.
|
||
|
||
| Suite | Setting | Why |
|
||
|-------|---------|-----|
|
||
| `ConfigurationDatabase.Tests` | `maxParallelThreads: 4` | Owns `MsSqlMigrationFixture`: every MSSQL-backed test **class** does a `CREATE DATABASE` plus the full ~100-migration chain against the one shared dev SQL Server. 4 matches the `sp_getapplock` slot count so the in-process and cross-process limits agree. |
|
||
| `AuditLog.Tests` | `maxParallelThreads: 4` | Same fixture (by project reference), plus most classes stand up an Akka TestKit `ActorSystem` with fast-ticking actors. |
|
||
| `SiteCallAudit.Tests` | `maxParallelThreads: 4` | Same fixture (by project reference) plus a TestKit `ActorSystem`. |
|
||
| `Host.Tests` | `maxParallelThreads: 4` | Classes boot real ASP.NET Core hosts and real Akka `ActorSystem`s — Akka.Remote binds real sockets, Kestrel binds real ports, gRPC channels dial them. |
|
||
| `IntegrationTests` | `parallelizeAssembly: false`, `parallelizeTestCollections: false` | Pre-existing. Fully serialized; a thread cap would be redundant. |
|
||
| `CentralUI.PlaywrightTests` | `maxParallelThreads: 1` | Pre-existing, and the strictest setting in the repo — the suite drives a browser against a **live rig**, so shared UI state, not CPU, is the constraint. |
|
||
| `StoreAndForward.Tests` | `parallelizeTestCollections: true` | Pre-existing; that is the default, so this file is a no-op pin. Uncapped by choice — SQLite files are per-test. |
|
||
|
||
### Rules
|
||
|
||
- **Cap threads, do not serialize.** `maxParallelThreads` throttles how many collections run at once;
|
||
`parallelizeTestCollections: false` runs them one at a time forever. Reach for the second only when a
|
||
suite has a genuinely global resource it cannot partition (`IntegrationTests` and the Playwright suite
|
||
are the only two, and both predate this policy).
|
||
- **A cap is not a substitute for test isolation.** Rows 82/91/95 were all *correctness* bugs — leaked
|
||
ticking actors, absolute assertions on process-global counters — that a cap only makes rarer. Fix the
|
||
test; the cap is for resource pressure.
|
||
- **The json must be copied to the output directory.** The .NET SDK does not do this automatically:
|
||
each capped `.csproj` carries
|
||
`<Content Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" />`. Without it the file
|
||
is silently ignored and the suite runs uncapped.
|
||
- **Default for a new test project: no runner config.** Add one only when the project acquires a
|
||
per-class external resource (an MSSQL fixture, a real host/port, a browser). Record the reason in the
|
||
`.csproj` comment next to the `Content` item.
|
||
|
||
### Measured cost
|
||
|
||
The cap is close to free, because these suites are latency-bound on an external resource rather than
|
||
CPU-bound (CPU utilisation stayed at 14–25 % for the two slowest). Measured standalone on a 14-core dev
|
||
box, uncapped vs. `maxParallelThreads: 4`:
|
||
|
||
| Suite | Tests | Uncapped | Capped at 4 |
|
||
|-------|-------|----------|-------------|
|
||
| `AuditLog.Tests` | 383 | 1 m 52 s | 1 m 51 s |
|
||
| `Host.Tests` | 520 | 4 m 5 s | 4 m 8 s |
|
||
| `ConfigurationDatabase.Tests` | 396 | 10 s | 12 s |
|
||
| `SiteCallAudit.Tests` | 66 | 8 s | 8 s |
|
||
|
||
## Remote Test Infrastructure
|
||
|
||
In addition to the local Docker services, the following remote services are available for testing against real AVEVA System Platform hardware.
|
||
|
||
**Primary/backup testing**: The dual OPC UA test servers (ports 50000 and 50010) in local Docker provide primary/backup endpoint pairs for testing Data Connection Layer failover. Use `docker compose stop opcua` to simulate primary failure and verify automatic failover to the backup.
|
||
|
||
**Alarms & Conditions (native alarms)**: The infra OPC PLC server **does** expose OPC UA Alarms & Conditions — a `ConditionRefresh` against its event notifier replays the active condition set and a `SnapshotComplete`, so the native alarm mirror (`IAlarmSubscribableConnection` → `NativeAlarmActor`) can be exercised live. The `OpcUaAlarmLiveSmokeTests.SubscribeAlarms_DeliversConditionRefreshSnapshot` `[SkippableFact]` (Trait `RequiresOpcUa`) round-trips against `opc.tcp://localhost:50000` and asserts the snapshot arrives; it reports **Skipped** (not failed) when the server is unreachable or — on a substitute server that lacks A&C — when no snapshot arrives within the window. The MxAccess Gateway alarm feed (`MxGateway` protocol) requires a live gateway and is verified via the `docker-env2` manual deploy check, not in CI.
|
||
|
||
## Connection Strings
|
||
|
||
For use in `appsettings.Development.json`:
|
||
|
||
```json
|
||
{
|
||
"ConnectionStrings": {
|
||
"ScadaBridgeConfig": "Server=localhost,1433;Database=ScadaBridgeConfig;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true",
|
||
"ScadaBridgeMachineData": "Server=localhost,1433;Database=ScadaBridgeMachineData;User Id=scadabridge_app;Password=ScadaBridge_Dev1#;TrustServerCertificate=true"
|
||
},
|
||
"Ldap": {
|
||
"Server": "10.100.0.35",
|
||
"Port": 3893,
|
||
"BaseDN": "dc=zb,dc=local",
|
||
"UseSsl": false
|
||
},
|
||
"OpcUa": {
|
||
"EndpointUrl": "opc.tcp://localhost:50000"
|
||
},
|
||
"OpcUa2": {
|
||
"EndpointUrl": "opc.tcp://localhost:50010"
|
||
},
|
||
"Smtp": {
|
||
"Server": "localhost",
|
||
"Port": 1025,
|
||
"AuthMode": "None",
|
||
"FromAddress": "scada-notifications@company.com",
|
||
"ConnectionTimeout": 30
|
||
},
|
||
"ExternalSystems": {
|
||
"TestApi": {
|
||
"BaseUrl": "http://localhost:5200",
|
||
"AuthMode": "ApiKey",
|
||
"ApiKey": "scadabridge-test-key-1"
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
## Stopping & Teardown
|
||
|
||
```bash
|
||
cd infra
|
||
docker compose down # stop containers, preserve SQL data volume
|
||
docker compose stop opcua # stop a single service (also: opcua2, mssql, smtp, restapi) # note: ldap is no longer a local service
|
||
```
|
||
|
||
**Full teardown** (removes volumes, optionally images and venv):
|
||
```bash
|
||
cd infra
|
||
./teardown.sh # stop containers + delete SQL data volume
|
||
./teardown.sh --images # also remove downloaded Docker images
|
||
./teardown.sh --all # also remove the Python venv
|
||
```
|
||
|
||
After a full teardown, the next `docker compose up -d` starts fresh — re-run the SQL setup script.
|
||
|
||
## Files
|
||
|
||
```
|
||
infra/
|
||
docker-compose.yml # All seven services
|
||
teardown.sh # Teardown script (volumes, images, venv)
|
||
glauth/config.toml # LDAP users and groups — HISTORICAL (retired 2026-06-04); live config is scadaproj/infra/glauth/config.toml
|
||
mssql/setup.sql # Database and user creation
|
||
mssql/machinedata_seed.sql # Machine Data tables, stored procedures, sample data
|
||
opcua/nodes.json # Custom OPC UA tag definitions
|
||
restapi/app.py # Flask REST API server
|
||
restapi/Dockerfile # REST API container build
|
||
tools/ # Python CLI tools (opcua, ldap, mssql, smtp, restapi)
|
||
README.md # Quick-start for the infra folder
|
||
|
||
docker/
|
||
traefik/traefik.yml # Traefik static config (entrypoints, file provider)
|
||
traefik/dynamic.yml # Traefik dynamic config (load balancer, health check routing)
|
||
```
|