dohertj2/lmxopcua
1
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(cluster): auto-down downing strategy — a two-node pair could not survive an oldest-node crash

Browse Source 
Ports the sister project's live-proven fix (ScadaBridge cf3bd52f). OtOpcUa ran
the identical configuration it indicts: keep-oldest with down-if-alone = on.

Akka.NET 1.5.62's KeepOldest.OldestDecision only lets the down-if-alone branch
rescue a side holding >= 2 members. A two-node cluster losing a peer is always a
1-vs-1 split, so the branch never fires, the lone survivor falls through to
DownReachable and downs ITSELF, and run-coordinated-shutdown-when-down then
terminates it. down-if-alone is a 3+-node feature and does not do what its name
suggests for a pair: a crash of the oldest node is a total outage, which is the
exact failure the redundancy pair exists to absorb.

Cluster:SplitBrainResolverStrategy now selects the provider, defaulting to
auto-down: Akka's AutoDowning with auto-down-unreachable-after = 15s, so the
leader among the reachable members downs the unreachable peer and a crash of
either node fails over in place. keep-oldest remains available for deployments
that would rather take an outage than ever run dual-active during a real
partition. An unrecognised value fails the host at startup rather than falling
through to the fatal default.

The tests assert the EFFECTIVE configuration — they start a real host through
WithOtOpcUaClusterBootstrap and read akka.cluster.downing-provider-class back off
the running ActorSystem. This is not incidental. The first draft inlined the
three calls the bootstrap makes instead of calling it, which pinned the test's
own wiring: sabotaging production's HOCON precedence left all 36 green. The
prior file had the same shape at a smaller scale, asserting only that
BuildClusterOptions returned a KeepOldestOption — true, and true of a
configuration that cannot fail over. Positive control: making BuildDowningHocon
emit nothing for auto-down turns exactly the two effective-config guards red.

Measured while verifying rather than assumed: HoconAddMode.Append also wins here,
purely because it is added last, so the mode name is not the guarantee. The
comment now says so instead of asserting a precedence rule that does not hold.

Not yet live-drilled on OtOpcUa. The docker-dev rig is a single six-node mesh
where the 1-vs-1 pathology cannot occur; the kill-the-oldest drill belongs with
the per-cluster mesh work that makes every mesh exactly two nodes (design doc
6.2 / Phase 0a). Recorded as an outstanding gate in docs/Redundancy.md.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-21 18:32:42 -04:00
parent c6fb8bb0ad
commit 2964361a6f
5 changed files with 427 additions and 97 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitBrainResolverActivationTests.cs
+195 -27
View File
@@ -1,47 +1,215 @@
using Akka.Actor;
using Akka.Cluster.Hosting;
using Akka.Cluster.Hosting.SBR;
using Akka.Configuration;
using Akka.Hosting;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
/// <summary>
/// Guards that the split-brain-resolver strategy stays set <b>explicitly in code</b> (arch-review 03/S1,
/// premise corrected — see #11). Akka.Cluster.Hosting already enables an SBR downing provider by default
/// (it applies <c>SplitBrainResolverOption.Default</c> when <see cref="ClusterOptions.SplitBrainResolver"/>
/// is null, which reads the akka.conf <c>keep-oldest</c> block), so the cluster is not NoDowning even
/// without the typed option. These tests assert
/// <see cref="ServiceCollectionExtensions.BuildClusterOptions"/> sets the typed
/// <c>KeepOldestOption { DownIfAlone = true }</c> so the strategy is explicit in code rather than relying on
/// the framework default — a future refactor cannot silently drop that explicitness. (Failover behaviour is
/// separately verified by <c>HardKillFailoverTests</c>, which passes regardless of activation path.)
/// Guards the downing strategy — the mechanism that decides whether a node that loses its peer
/// keeps running or shuts itself down.
/// </summary>
/// <remarks>
/// <para>
/// <b>Why these tests assert the EFFECTIVE config, not the typed option.</b> The previous
/// version of this file asserted only that
/// <see cref="ServiceCollectionExtensions.BuildClusterOptions"/> returned a
/// <see cref="KeepOldestOption"/>. That is a different question from "what does the running
/// ActorSystem actually do". Akka.Hosting merges several HOCON fragments — the akka.conf
/// resource, whatever <c>WithClustering</c> emits, and anything added explicitly — and which
/// one wins is not readable off the <see cref="HoconAddMode"/> names alone. A downing
/// setting can therefore be right in the resource file, right in the typed options object,
/// and still not be the one in force. These tests start a real host through the production
/// bootstrap and read <c>akka.cluster.downing-provider-class</c> back off
/// <see cref="ActorSystem.Settings"/>, which is the only assertion precedence cannot fool.
/// </para>
/// <para>
/// <b>Why the default is auto-down.</b> A two-node cluster running the SBR
/// <c>keep-oldest</c> resolver cannot survive a crash of the oldest node. Akka.NET 1.5.62's
/// <c>KeepOldest.OldestDecision</c> only lets <c>down-if-alone</c> rescue a side holding
/// &gt;= 2 members, so in a 1-vs-1 split the lone survivor takes <c>DownReachable</c> and
/// downs <i>itself</i> — then <c>run-coordinated-shutdown-when-down</c> terminates it. A
/// crash of one node becomes a total outage: exactly the failure the redundancy pair exists
/// to absorb. Proven live on the sister project's rig (see
/// <c>../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md</c>) and
/// adopted here for the same reason. The accepted trade is dual-active during a genuine
/// network partition; see <c>docs/Redundancy.md</c>.
/// </para>
/// </remarks>
public sealed class SplitBrainResolverActivationTests
{
private static AkkaClusterOptions SampleOptions() => new()
private static AkkaClusterOptions SampleOptions(string? strategy = null)
{
SeedNodes = new[] { "akka.tcp://otopcua@127.0.0.1:4053" },
Roles = new[] { "admin", "driver" },
};
var options = new AkkaClusterOptions
{
SeedNodes = new[] { "akka.tcp://otopcua@127.0.0.1:4053" },
Roles = new[] { "admin", "driver" },
};
if (strategy is not null)
{
options.SplitBrainResolverStrategy = strategy;
}
/// <summary>Verifies the cluster options carry an explicit split-brain resolver strategy.</summary>
[Fact]
public void BuildClusterOptions_sets_an_explicit_split_brain_resolver()
{
var options = ServiceCollectionExtensions.BuildClusterOptions(SampleOptions());
options.SplitBrainResolver.ShouldNotBeNull(
"SplitBrainResolver must be set explicitly in code — keeps the strategy independent of Akka.Cluster.Hosting's default provider rather than relying on it.");
return options;
}
/// <summary>Verifies the resolver is keep-oldest with down-if-alone (correct for a 2-node pair).</summary>
[Fact]
public void BuildClusterOptions_uses_keep_oldest_with_down_if_alone()
/// <summary>
/// Starts a real host through <see cref="ServiceCollectionExtensions.WithOtOpcUaClusterBootstrap"/>
/// — the production entry point, not a re-creation of it — and returns the ActorSystem's
/// effective configuration.
/// </summary>
/// <remarks>
/// Calling the production bootstrap rather than inlining the calls it makes is deliberate: an
/// earlier draft of this helper rebuilt the same three steps itself, which pinned the
/// <i>test's</i> wiring instead of the shipped one and would have stayed green through a change
/// to production's HOCON precedence.
/// </remarks>
private static async Task<Config> EffectiveConfigAsync(AkkaClusterOptions options)
{
var options = ServiceCollectionExtensions.BuildClusterOptions(SampleOptions());
// Port 0 so concurrently-running test hosts cannot collide on 4053, and no seed nodes so the
// system never tries to join a cluster.
options.Port = 0;
options.Hostname = "127.0.0.1";
options.PublicHostname = "127.0.0.1";
options.SeedNodes = Array.Empty<string>();
var keepOldest = options.SplitBrainResolver.ShouldBeOfType<KeepOldestOption>();
keepOldest.DownIfAlone.ShouldBe(true);
var builder = Host.CreateDefaultBuilder();
builder.ConfigureServices(services =>
{
services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
services.AddAkka("otopcua-sbr-test", (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp));
});
using var host = builder.Build();
await host.StartAsync();
try
{
return host.Services.GetRequiredService<ActorSystem>().Settings.Config;
}
finally
{
await host.StopAsync();
}
}
/// <summary>
/// The default strategy installs Akka's AutoDowning provider — the leader among the
/// <i>reachable</i> members downs the unreachable peer, so a crash of either node fails over.
/// </summary>
[Fact]
public async Task Default_strategy_installs_the_auto_downing_provider()
{
var config = await EffectiveConfigAsync(SampleOptions());
config.GetString("akka.cluster.downing-provider-class")
.ShouldStartWith(
"Akka.Cluster.AutoDowning, Akka.Cluster",
Case.Sensitive,
"the effective downing provider — not merely the configured one — must be AutoDowning, or a crash of the oldest node is a total outage");
}
/// <summary>
/// AutoDowning is inert without a downing window: <c>auto-down-unreachable-after</c> defaults to
/// <c>off</c>, under which the provider is installed but never downs anything. Selecting the
/// provider and forgetting the window would look correct and fail over never.
/// </summary>
[Fact]
public async Task Default_strategy_sets_the_auto_down_window()
{
var config = await EffectiveConfigAsync(SampleOptions());
config.GetTimeSpan("akka.cluster.auto-down-unreachable-after")
.ShouldBe(ServiceCollectionExtensions.DowningStableAfter);
}
/// <summary>
/// The opt-out is real: an operator who chooses partition-safety over availability gets the SBR
/// keep-oldest resolver back, in force rather than merely configured.
/// </summary>
[Fact]
public async Task Keep_oldest_strategy_installs_the_sbr_provider()
{
var config = await EffectiveConfigAsync(SampleOptions("keep-oldest"));
config.GetString("akka.cluster.downing-provider-class")
.ShouldStartWith("Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster");
config.GetString("akka.cluster.split-brain-resolver.active-strategy").ShouldBe("keep-oldest");
config.GetBoolean("akka.cluster.split-brain-resolver.keep-oldest.down-if-alone").ShouldBeTrue();
}
/// <summary>The default is auto-down, so an unconfigured deployment gets the survivable one.</summary>
[Fact]
public void Strategy_defaults_to_auto_down()
{
new AkkaClusterOptions().SplitBrainResolverStrategy.ShouldBe("auto-down");
}
/// <summary>
/// A typo in the strategy must fail the host at startup, not silently fall through to whichever
/// provider Akka.Hosting would have installed by default — which is the fatal one.
/// </summary>
[Fact]
public void Unknown_strategy_throws_rather_than_silently_falling_back()
{
var ex = Should.Throw<ArgumentOutOfRangeException>(
() => ServiceCollectionExtensions.BuildDowningHocon(SampleOptions("keep-majority")));
ex.Message.ShouldContain("keep-majority");
ex.Message.ShouldContain("auto-down");
}
/// <summary>Strategy comparison is case-insensitive — config casing is not a failure mode.</summary>
[Fact]
public void Strategy_matching_is_case_insensitive()
{
Should.NotThrow(() => ServiceCollectionExtensions.BuildDowningHocon(SampleOptions("Auto-Down")));
Should.NotThrow(() => ServiceCollectionExtensions.BuildDowningHocon(SampleOptions("Keep-Oldest")));
}
/// <summary>
/// Under keep-oldest the SBR resolver is still set as a typed option, so the strategy stays
/// explicit in code rather than depending on Akka.Cluster.Hosting's default.
/// </summary>
[Fact]
public void Keep_oldest_sets_the_typed_resolver_option()
{
var options = ServiceCollectionExtensions.BuildClusterOptions(SampleOptions("keep-oldest"));
options.SplitBrainResolver.ShouldBeOfType<KeepOldestOption>().DownIfAlone.ShouldBe(true);
}
/// <summary>
/// Pins the two stability windows together. <c>auto-down-unreachable-after</c> comes from
/// <see cref="ServiceCollectionExtensions.DowningStableAfter"/> while the SBR resolver reads
/// <c>stable-after</c> out of akka.conf; changing one and missing the other would leave the two
/// strategies failing over at different latencies for no stated reason.
/// </summary>
[Fact]
public void Downing_window_matches_the_akka_conf_stable_after()
{
var akkaConf = ConfigurationFactory.ParseString(HoconLoader.LoadBaseConfig());
akkaConf.GetTimeSpan("akka.cluster.split-brain-resolver.stable-after")
.ShouldBe(ServiceCollectionExtensions.DowningStableAfter);
}
/// <summary>
/// The stability window must stay above the failure detector's tolerance, or a node gets downed
/// while its peer is merely slow.
/// </summary>
[Fact]
public void Downing_window_exceeds_the_acceptable_heartbeat_pause()
{
var akkaConf = ConfigurationFactory.ParseString(HoconLoader.LoadBaseConfig());
ServiceCollectionExtensions.DowningStableAfter
.ShouldBeGreaterThan(akkaConf.GetTimeSpan("akka.cluster.failure-detector.acceptable-heartbeat-pause"));
}
/// <summary>Verifies seed nodes and roles are still threaded through unchanged.</summary>