Joseph Doherty
835fc08c3f
User-reported: the integration tests leave instances behind and take 18+ GB.
Measured before changing anything — a SINGLE Host.IntegrationTests process peaked
at 30,153 MB RSS across its ~200 tests.
The cause was not a leak and not parallelism. Referencing the Host drags in the
ASP.NET Core framework reference, which turns Server GC on by default: one heap
per core, tuned for throughput and deliberately reluctant to hand memory back.
Correct for a server, wrong for a test host. Workstation GC takes the same suite
from 30,153 MB to 6,717 MB — 78% less — with no change in runtime (45s).
Two hypotheses were measured and rejected on the way, recorded here so nobody
re-runs the experiment: capping xunit maxParallelThreads to 4 cost 29% wall-clock
and saved nothing (30,153 -> 31,115 MB, i.e. noise), and it turns out a single
process was doing all of it. Reverted.
The harness fix is real but was worth only 4% on its own: TwoNodeClusterHarness
disposed each node without stopping it first, and IHost.DisposeAsync only disposes
the service provider — it never invokes IHostedService.StopAsync, which is where
Akka.Hosting terminates the ActorSystem. So every test left two live ActorSystems,
remoting transports and dispatcher pools included, rooted for the process
lifetime. The class doc-comment asserted the opposite ("DisposeAsync, which runs
CoordinatedShutdown"); it was wrong, and StopNodeBAsync inherited the same bug on
a path failover tests depend on.
The payoff is larger than the memory number. Two failures I had classified as
known-flaky baseline noise now pass consistently:
RoslynVirtualTagEvaluatorTests.Evaluate_racing_ClearCompiledScripts_never_fails_with_disposed
and ContinuousHistorizationRecorderTests.Retry_after_writer_failure_eventually_acks.
They were never flaky — they were starved. Full solution: peak summed test-process
RSS 15.2 GB, zero new failures, two fewer.
Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
ZB.MOM.WW.OtOpcUa.Host.IntegrationTests
Two-node Akka cluster integration tests on top of TwoNodeClusterHarness.
Default mode (no infra required)
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests
Uses Microsoft.EntityFrameworkCore.InMemory for ConfigDb and a stub ILdapAuthService that
accepts any username when the password is valid-password. Each harness instance creates a
unique in-memory database scoped to its lifetime. This is the mode CI runs by default.
Real-infra mode (SQL Server + OpenLDAP)
When you need to exercise EF behaviors that diverge between providers (index uniqueness,
RowVersion concurrency, JSON columns, migration application) or a real LDAP bind, bring up
the bundled compose stack and set the env-var switches:
docker compose -f tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/docker-compose.yml up -d
export OTOPCUA_HARNESS_USE_SQL=1
export OTOPCUA_HARNESS_USE_LDAP=1
dotnet test tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests
docker compose -f tests/Server/ZB.MOM.WW.OtOpcUa.Host.IntegrationTests/docker-compose.yml down -v
SQL Server mode (OTOPCUA_HARNESS_USE_SQL=1)
- Container:
mcr.microsoft.com/mssql/server:2022-latestonlocalhost:14331 - Each
TwoNodeClusterHarness.StartAsync()creates a unique databaseOtOpcUa_Harness_{guid}viaDatabase.EnsureCreatedAsync()and drops it onDisposeAsync()(best-effort). - Port
14331chosen to avoid colliding with thedocker-dev/fleet (which uses14330).
LDAP mode (OTOPCUA_HARNESS_USE_LDAP=1)
- Container:
bitnami/openldap:2.6onlocalhost:3894 - Users
alice/alice123andbob/bob123, all underou=FleetAdmin. - Port
3894chosen to avoid colliding with thedocker-dev/fleet (which uses3893).
Local-dev caveat
This dev VM (DESKTOP-6JL3KKO) does not run Docker locally. Real-infra mode runs on the
shared Linux Docker host (10.100.0.35) per docs/v2/dev-environment.md, or in CI on Linux.