test(harness): serialize Host.IntegrationTests; drop the reconciler's unused singleton proxy
v2-ci / build (push) Successful in 3m51s
v2-ci / unit-tests (push) Failing after 13m29s

Closes the last branch-only failures. Measured across five full-suite runs on
this branch versus two on master in a clean worktree: master failed only AbCip
in this assembly (2/2), while the branch additionally failed one or two E2E
deploy tests, rotating between DeployHappyPath, DriverReconnect and
EquipmentNamespaceMaterialization (4/4). Each passed in isolation, and the
assembly on its own was 195/201 throughout — the failures only appeared under
full-suite CPU contention.

Cause is contention, not correctness. Every test here builds a real two-node
Akka cluster — two Kestrel hosts, two ActorSystems with remoting, six admin
singletons, an EF context, a deploy pipeline — and xUnit ran them concurrently
with each other and with every other assembly. That was survivable while the
coordinator sealed instantly on an empty expected-ack set; now that it waits for
a real ack from every configured node, these tests measure an end-to-end
round-trip and starve.

Serialising this one assembly makes them deterministic: 195/201 with only
AbCip_Green_AgainstSim, and the full-solution failure set is now IDENTICAL to
master's — 7 shared, zero branch-only. Cost is wall-clock for this assembly,
40s -> 3m35s; it is a handful of heavyweight E2E tests, not a broad unit suite,
and a widened timeout alone did not fix it (tried first, at 45s).

Also drops createProxyToo for the address reconciler. Nothing resolves
ClusterNodeAddressReconcilerKey from the registry — the actor only reads cluster
state and logs — so the proxy was a second actor per node hunting for a
singleton nobody addresses. Kept because it removes dead machinery, not because
it fixed the flakiness; measured on its own, it did not.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-22 09:51:10 -04:00
parent 0b7c53f64f
commit 7654f24dab
2 changed files with 25 additions and 2 deletions
@@ -26,7 +26,8 @@ public static class ServiceCollectionExtensions
public const string ClusterNodeAddressReconcilerSingletonName = "cluster-node-address-reconciler";
/// <summary>
/// Registers all five admin-role cluster singletons + their proxies on the AkkaConfigurationBuilder.
/// Registers the admin-role cluster singletons on the AkkaConfigurationBuilder — five with
/// registry proxies, plus the address reconciler, which is proxy-less because nothing addresses it.
/// Must be called against the same builder used by <c>AkkaHostedService</c> so the singletons
/// share the host's ActorSystem.
/// </summary>
@@ -94,6 +95,13 @@ public static class ServiceCollectionExtensions
// Per-cluster mesh Phase 1: ClusterNode.AkkaPort duplicates the node's own Cluster:Port and
// nothing makes them agree. Runs here rather than on each driver node because Phase 4 removes
// the driver nodes' ConfigDb connection entirely.
//
// createProxyToo: false — unlike the five singletons above, nothing ever resolves
// ClusterNodeAddressReconcilerKey from the registry. This actor only reads cluster state and
// logs; no one sends it messages. A proxy would be a second actor per node whose only job is
// to hunt for a singleton nobody addresses — visible as the "ClusterSingletonProxy failed to
// find an associated singleton after 30 seconds" startup warning, and as extra cluster
// chatter in every two-node test harness the integration suite builds.
builder.WithSingleton<ClusterNodeAddressReconcilerKey>(
ClusterNodeAddressReconcilerSingletonName,
(system, registry, resolver) =>
@@ -101,7 +109,8 @@ public static class ServiceCollectionExtensions
var dbFactory = resolver.GetService<IDbContextFactory<OtOpcUaConfigDbContext>>();
return ClusterNodeAddressReconcilerActor.Props(dbFactory);
},
singletonOptions);
singletonOptions,
createProxyToo: false);
return builder;
}
@@ -0,0 +1,14 @@
using Xunit;
// Every test in this assembly builds a real two-node Akka cluster: two in-process Kestrel hosts,
// two ActorSystems with remoting on ephemeral ports, six admin-role cluster singletons, an EF
// context and a deploy pipeline. Running several of those concurrently — on top of the other test
// assemblies dotnet test runs in parallel — starves the cluster-formation and ApplyAck timers these
// tests measure, so failures appear under full-suite load and vanish in isolation.
//
// That was tolerable while the deploy path sealed immediately on an empty expected-ack set. Since
// per-cluster mesh Phase 1 the coordinator waits for a real ack from every configured node, so these
// tests now measure an end-to-end round-trip and are correspondingly more sensitive to CPU
// starvation. Serialising this assembly trades wall-clock for determinism; the assembly is a handful
// of heavyweight E2E tests, not a broad unit suite.
[assembly: CollectionBehavior(DisableTestParallelization = true)]