dohertj2/lmxopcua
1
0
Fork 0
Code Issues 1 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(cluster): activate split-brain resolver (arch-review 03/S1)

Browse Source 
The split-brain-resolver HOCON block in akka.conf was inert — nothing
registered a downing provider, so the cluster ran Akka's default NoDowning:
a hard-crashed node was never downed, cluster singletons + the driver
role-leader never failed over, and a partition left both redundancy sides
at ServiceLevel 240 indefinitely.

Activate the resolver via the typed ClusterOptions.SplitBrainResolver
(KeepOldestOption { DownIfAlone = true }) in a new testable
BuildClusterOptions helper. keep-oldest + down-if-alone is correct for a
2-node warm-redundancy pair. HOCON keeps stable-after (typed option can't
express it) + a cross-reference comment. Deterministic unit guard added;
end-to-end verified the resolved ActorSystem config carries the provider
and stable-after survives the HOCON/Hosting merge. Redundancy.md corrected.

Hard-kill failover integration test tracked as a follow-up (effort-M harness).
This commit is contained in:
Joseph Doherty
2026-07-08 16:14:26 -04:00
parent 9cad9ed0fc
commit a81dea102c
4 changed files with 112 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Redundancy.md
+17 -1
View File
@@ -154,7 +154,23 @@ Node A lists Node B's `ApplicationUri` and vice-versa. Validated by `DualEndpoin
## Split-brain
`akka.conf` configures Akka's split-brain resolver with `active-strategy = keep-oldest`, `stable-after = 15s`, and `failure-detector.threshold = 10.0`. Under a clean partition: the oldest member stays up + the smaller (or younger) side downs itself within ~15 seconds. The `RedundancyStateActor` on the surviving partition re-computes from the post-partition `Cluster.State`.
The split-brain resolver is **activated in code** by the typed `ClusterOptions.SplitBrainResolver`
(`KeepOldestOption { DownIfAlone = true }`) set in
`ServiceCollectionExtensions.BuildClusterOptions` — this is what registers
`Akka.Cluster.SBR.SplitBrainResolverProvider`. **Without that typed registration the `split-brain-resolver`
HOCON block in `akka.conf` is inert and the cluster runs Akka's default `NoDowning`**, meaning a
hard-crashed node is never downed and neither the cluster singletons nor the `driver` role-leader ever fail
over (a partition would leave both redundancy sides at ServiceLevel 240 indefinitely). The HOCON block is the
tuning source only: `active-strategy = keep-oldest`, `stable-after = 15s`, `keep-oldest.down-if-alone = on`,
`failure-detector.threshold = 10.0` (its `active-strategy` + `down-if-alone` must stay consistent with the
typed option; `stable-after` lives only in HOCON because the typed option can't express it).
`keep-oldest` is the correct strategy for a 2-node warm-redundancy pair: under a clean partition the oldest
member (typically the long-running primary) stays up and the smaller (or younger) side downs itself within
~`stable-after` seconds; `down-if-alone` downs a node that loses its only peer. `keep-majority`/`static-quorum`
are wrong for two nodes (no majority in a 1-1 split). The `RedundancyStateActor` on the surviving partition
re-computes from the post-partition `Cluster.State`, and a hard-crashed (not gracefully stopped) node now
triggers the same failover.
There is no operator-driven role swap during a partition. Failover is what the cluster does automatically.
src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf
+7
View File
@@ -37,6 +37,13 @@ akka {
roles = []
min-nr-of-members = 1
# Split-brain resolver. This HOCON block is the TUNING source only — it does NOT
# activate the resolver. Activation happens in code via the typed
# ClusterOptions.SplitBrainResolver (KeepOldestOption { DownIfAlone = true }) set in
# ServiceCollectionExtensions.BuildClusterOptions; without that the cluster runs Akka's
# default NoDowning and hard-crashed nodes never fail over (arch-review 03/S1).
# active-strategy + keep-oldest.down-if-alone below MUST match the typed option.
# stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin.
split-brain-resolver {
active-strategy = "keep-oldest"
stable-after = 15s
src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs
+33 -5
View File
@@ -1,4 +1,5 @@
using Akka.Cluster.Hosting;
using Akka.Cluster.Hosting.SBR;
using Akka.Event;
using Akka.Hosting;
using Akka.Logger.Serilog;
@@ -77,12 +78,39 @@ public static class ServiceCollectionExtensions
PublicHostName = options.PublicHostname,
});
builder.WithClustering(new ClusterOptions
{
SeedNodes = options.SeedNodes,
Roles = options.Roles,
});
builder.WithClustering(BuildClusterOptions(options));
return builder;
}
/// <summary>
/// Builds the <see cref="ClusterOptions"/> for the fused-host cluster, including the
/// <b>activating</b> split-brain-resolver registration.
///
/// The <c>split-brain-resolver</c> HOCON block in <c>Resources/akka.conf</c> is inert on its own —
/// Akka.NET only runs the resolver when a downing provider is registered, otherwise the cluster
/// falls back to <b>NoDowning</b> and a hard-crashed node is never downed (singletons and the
/// <c>driver</c> role-leader never fail over, and a partition leaves both redundancy sides at
/// ServiceLevel 240 forever). Setting <see cref="ClusterOptions.SplitBrainResolver"/> is what
/// activates <c>Akka.Cluster.SBR.SplitBrainResolverProvider</c> under the hood — this is the piece
/// that was missing (arch-review 03/S1).
///
/// <see cref="KeepOldestOption"/> with <c>DownIfAlone=true</c> mirrors the HOCON intent and is the
/// correct strategy for a 2-node warm-redundancy pair: on an even split the oldest member (typically
/// the long-running primary) survives, and <c>down-if-alone</c> downs a singleton that loses its
/// peer. <c>keep-majority</c>/<c>static-quorum</c> are wrong for two nodes. The strategy + down-if-alone
/// here MUST stay consistent with the HOCON block; <c>stable-after</c> lives only in HOCON because the
/// typed option cannot express it (it must stay ≥ <c>failure-detector.acceptable-heartbeat-pause</c>).
/// </summary>
/// <param name="options">The bound cluster options carrying seed nodes and roles.</param>
/// <returns>The cluster options with seed nodes, roles, and the activated split-brain resolver.</returns>
public static ClusterOptions BuildClusterOptions(AkkaClusterOptions options)
{
return new ClusterOptions
{
SeedNodes = options.SeedNodes,
Roles = options.Roles,
SplitBrainResolver = new KeepOldestOption { DownIfAlone = true },
};
}
}
tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitBrainResolverActivationTests.cs
+55
View File
@@ -0,0 +1,55 @@
using Akka.Cluster.Hosting;
using Akka.Cluster.Hosting.SBR;
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
/// <summary>
/// Guards the split-brain-resolver ACTIVATION (arch-review 03/S1). The <c>split-brain-resolver</c>
/// HOCON block is inert on its own; the resolver only runs when the typed
/// <see cref="ClusterOptions.SplitBrainResolver"/> is set. These tests assert
/// <see cref="ServiceCollectionExtensions.BuildClusterOptions"/> registers it, so a future refactor
/// cannot silently drop back to Akka's default NoDowning (the exact "built-but-never-wired" failure
/// mode this finding described).
/// </summary>
public sealed class SplitBrainResolverActivationTests
{
private static AkkaClusterOptions SampleOptions() => new()
{
SeedNodes = new[] { "akka.tcp://otopcua@127.0.0.1:4053" },
Roles = new[] { "admin", "driver" },
};
/// <summary>Verifies the cluster options carry a split-brain resolver (provider gets activated).</summary>
[Fact]
public void BuildClusterOptions_activates_a_split_brain_resolver()
{
var options = ServiceCollectionExtensions.BuildClusterOptions(SampleOptions());
options.SplitBrainResolver.ShouldNotBeNull(
"SplitBrainResolver must be set — without it the cluster runs default NoDowning and never fails over on a hard crash.");
}
/// <summary>Verifies the resolver is keep-oldest with down-if-alone (correct for a 2-node pair).</summary>
[Fact]
public void BuildClusterOptions_uses_keep_oldest_with_down_if_alone()
{
var options = ServiceCollectionExtensions.BuildClusterOptions(SampleOptions());
var keepOldest = options.SplitBrainResolver.ShouldBeOfType<KeepOldestOption>();
keepOldest.DownIfAlone.ShouldBe(true);
}
/// <summary>Verifies seed nodes and roles are still threaded through unchanged.</summary>
[Fact]
public void BuildClusterOptions_preserves_seed_nodes_and_roles()
{
var input = SampleOptions();
var options = ServiceCollectionExtensions.BuildClusterOptions(input);
options.SeedNodes.ShouldBe(input.SeedNodes);
options.Roles.ShouldBe(input.Roles);
}
}