docs(archreview #11): correct Critical 1 premise — SBR was already active on master, not NoDowning
Akka.Cluster.Hosting's WithClustering enables an SBR downing provider by default (applies SplitBrainResolverOption.Default when ClusterOptions.SplitBrainResolver is null), which reads the pre-existing akka.conf keep-oldest block. So the cluster was NOT running NoDowning before Critical 1 and hard-crash failover already worked — the typed KeepOldestOption is reinforcing/explicit-in-code, not the sole activator. Corrects the inaccurate 'HOCON inert / NoDowning / never fails over' framing in: - ServiceCollectionExtensions.BuildClusterOptions XML comment - akka.conf split-brain-resolver comment - docs/Redundancy.md Split-brain section - SplitBrainResolverActivationTests summary + assertion message (+ method rename) No code revert (the typed option is correct belt-and-suspenders). Cluster.Tests 29/29. Surfaced by the #9 hard-kill failover negative control.
This commit is contained in:
Joseph Doherty
+17
-12
@@ -154,23 +154,28 @@ Node A lists Node B's `ApplicationUri` and vice-versa. Validated by `DualEndpoin
|
||||
|
||||
## Split-brain
|
||||
|
||||
The split-brain resolver is **activated in code** by the typed `ClusterOptions.SplitBrainResolver`
|
||||
(`KeepOldestOption { DownIfAlone = true }`) set in
|
||||
`ServiceCollectionExtensions.BuildClusterOptions` — this is what registers
|
||||
`Akka.Cluster.SBR.SplitBrainResolverProvider`. **Without that typed registration the `split-brain-resolver`
|
||||
HOCON block in `akka.conf` is inert and the cluster runs Akka's default `NoDowning`**, meaning a
|
||||
hard-crashed node is never downed and neither the cluster singletons nor the `driver` role-leader ever fail
|
||||
over (a partition would leave both redundancy sides at ServiceLevel 240 indefinitely). The HOCON block is the
|
||||
tuning source only: `active-strategy = keep-oldest`, `stable-after = 15s`, `keep-oldest.down-if-alone = on`,
|
||||
`failure-detector.threshold = 10.0` (its `active-strategy` + `down-if-alone` must stay consistent with the
|
||||
typed option; `stable-after` lives only in HOCON because the typed option can't express it).
|
||||
The split-brain resolver is **active by default**: Akka.Cluster.Hosting's `WithClustering` enables an SBR
|
||||
downing provider whenever `ClusterOptions.SplitBrainResolver` is null (it applies
|
||||
`SplitBrainResolverOption.Default`, which registers `Akka.Cluster.SBR.SplitBrainResolverProvider`), and that
|
||||
provider reads the `split-brain-resolver` HOCON block in `akka.conf`. On top of that,
|
||||
`ServiceCollectionExtensions.BuildClusterOptions` sets the typed `ClusterOptions.SplitBrainResolver`
|
||||
(`KeepOldestOption { DownIfAlone = true }`) to make the strategy **explicit in code** rather than relying on
|
||||
the framework default — it is reinforcing, not the sole activator, and yields the same effective behavior. So
|
||||
the cluster is **not** running `NoDowning`; hard-crashed nodes are downed and both the cluster singletons and
|
||||
the `driver` role-leader fail over. (Only an *explicit* `NoDowning`, e.g.
|
||||
`akka.cluster.downing-provider-class = ""`, would leave both redundancy sides at ServiceLevel 240
|
||||
indefinitely.) The HOCON block carries the tuning: `active-strategy = keep-oldest`, `stable-after = 15s`,
|
||||
`keep-oldest.down-if-alone = on`, `failure-detector.threshold = 10.0` (its `active-strategy` + `down-if-alone`
|
||||
must stay consistent with the typed option; `stable-after` lives only in HOCON because the typed option can't
|
||||
express it).
|
||||
|
||||
`keep-oldest` is the correct strategy for a 2-node warm-redundancy pair: under a clean partition the oldest
|
||||
member (typically the long-running primary) stays up and the smaller (or younger) side downs itself within
|
||||
~`stable-after` seconds; `down-if-alone` downs a node that loses its only peer. `keep-majority`/`static-quorum`
|
||||
are wrong for two nodes (no majority in a 1-1 split). The `RedundancyStateActor` on the surviving partition
|
||||
re-computes from the post-partition `Cluster.State`, and a hard-crashed (not gracefully stopped) node now
|
||||
triggers the same failover.
|
||||
re-computes from the post-partition `Cluster.State`, and a hard-crashed (not gracefully stopped) node
|
||||
triggers the same failover — verified by `HardKillFailoverTests` (its negative control confirms failover
|
||||
survives removing the typed option, and only breaks under an explicit `NoDowning`).
|
||||
|
||||
There is no operator-driven role swap during a partition. Failover is what the cluster does automatically.
|
||||
|
||||
|
||||
@@ -37,12 +37,13 @@ akka {
|
||||
roles = []
|
||||
min-nr-of-members = 1
|
||||
|
||||
# Split-brain resolver. This HOCON block is the TUNING source only — it does NOT
|
||||
# activate the resolver. Activation happens in code via the typed
|
||||
# ClusterOptions.SplitBrainResolver (KeepOldestOption { DownIfAlone = true }) set in
|
||||
# ServiceCollectionExtensions.BuildClusterOptions; without that the cluster runs Akka's
|
||||
# default NoDowning and hard-crashed nodes never fail over (arch-review 03/S1).
|
||||
# active-strategy + keep-oldest.down-if-alone below MUST match the typed option.
|
||||
# Split-brain resolver (arch-review 03/S1). This block IS active: Akka.Cluster.Hosting's
|
||||
# WithClustering enables an SBR downing provider by default (it applies
|
||||
# SplitBrainResolverOption.Default when ClusterOptions.SplitBrainResolver is null), and that
|
||||
# provider reads this block. ServiceCollectionExtensions.BuildClusterOptions additionally sets
|
||||
# the typed KeepOldestOption { DownIfAlone = true } to make the strategy EXPLICIT in code —
|
||||
# its active-strategy + keep-oldest.down-if-alone MUST match this block. (The cluster is NOT
|
||||
# NoDowning; only an explicit downing-provider-class = "" would disable failover.)
|
||||
# stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin.
|
||||
split-brain-resolver {
|
||||
active-strategy = "keep-oldest"
|
||||
|
||||
@@ -84,16 +84,19 @@ public static class ServiceCollectionExtensions
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Builds the <see cref="ClusterOptions"/> for the fused-host cluster, including the
|
||||
/// <b>activating</b> split-brain-resolver registration.
|
||||
/// Builds the <see cref="ClusterOptions"/> for the fused-host cluster, setting the split-brain-resolver
|
||||
/// strategy <b>explicitly in code</b>.
|
||||
///
|
||||
/// The <c>split-brain-resolver</c> HOCON block in <c>Resources/akka.conf</c> is inert on its own —
|
||||
/// Akka.NET only runs the resolver when a downing provider is registered, otherwise the cluster
|
||||
/// falls back to <b>NoDowning</b> and a hard-crashed node is never downed (singletons and the
|
||||
/// <c>driver</c> role-leader never fail over, and a partition leaves both redundancy sides at
|
||||
/// ServiceLevel 240 forever). Setting <see cref="ClusterOptions.SplitBrainResolver"/> is what
|
||||
/// activates <c>Akka.Cluster.SBR.SplitBrainResolverProvider</c> under the hood — this is the piece
|
||||
/// that was missing (arch-review 03/S1).
|
||||
/// <b>Activation note (arch-review 03/S1, corrected):</b> Akka.Cluster.Hosting's <c>WithClustering</c>
|
||||
/// already enables an SBR downing provider <b>by default</b> — when <see cref="ClusterOptions.SplitBrainResolver"/>
|
||||
/// is <c>null</c> it applies <c>SplitBrainResolverOption.Default</c>, which registers
|
||||
/// <c>Akka.Cluster.SBR.SplitBrainResolverProvider</c> and reads the <c>split-brain-resolver</c> HOCON block in
|
||||
/// <c>Resources/akka.conf</c>. So the cluster was <b>not</b> running <c>NoDowning</c> before this option was
|
||||
/// set: the pre-existing akka.conf <c>keep-oldest</c> block was already active, and hard-crashed nodes already
|
||||
/// failed over. Setting this typed option makes the strategy <b>explicit in code</b> (independent of the
|
||||
/// framework default) rather than being the sole activator — it is reinforcing/belt-and-suspenders and
|
||||
/// produces the same effective behavior. (Only an <i>explicit</i> <c>NoDowning</c>, e.g.
|
||||
/// <c>downing-provider-class = ""</c>, would disable failover.)
|
||||
///
|
||||
/// <see cref="KeepOldestOption"/> with <c>DownIfAlone=true</c> mirrors the HOCON intent and is the
|
||||
/// correct strategy for a 2-node warm-redundancy pair: on an even split the oldest member (typically
|
||||
@@ -103,7 +106,7 @@ public static class ServiceCollectionExtensions
|
||||
/// typed option cannot express it (it must stay ≥ <c>failure-detector.acceptable-heartbeat-pause</c>).
|
||||
/// </summary>
|
||||
/// <param name="options">The bound cluster options carrying seed nodes and roles.</param>
|
||||
/// <returns>The cluster options with seed nodes, roles, and the activated split-brain resolver.</returns>
|
||||
/// <returns>The cluster options with seed nodes, roles, and the explicit split-brain-resolver strategy.</returns>
|
||||
public static ClusterOptions BuildClusterOptions(AkkaClusterOptions options)
|
||||
{
|
||||
return new ClusterOptions
|
||||
|
||||
@@ -6,12 +6,15 @@ using Xunit;
|
||||
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Guards the split-brain-resolver ACTIVATION (arch-review 03/S1). The <c>split-brain-resolver</c>
|
||||
/// HOCON block is inert on its own; the resolver only runs when the typed
|
||||
/// <see cref="ClusterOptions.SplitBrainResolver"/> is set. These tests assert
|
||||
/// <see cref="ServiceCollectionExtensions.BuildClusterOptions"/> registers it, so a future refactor
|
||||
/// cannot silently drop back to Akka's default NoDowning (the exact "built-but-never-wired" failure
|
||||
/// mode this finding described).
|
||||
/// Guards that the split-brain-resolver strategy stays set <b>explicitly in code</b> (arch-review 03/S1,
|
||||
/// premise corrected — see #11). Akka.Cluster.Hosting already enables an SBR downing provider by default
|
||||
/// (it applies <c>SplitBrainResolverOption.Default</c> when <see cref="ClusterOptions.SplitBrainResolver"/>
|
||||
/// is null, which reads the akka.conf <c>keep-oldest</c> block), so the cluster is not NoDowning even
|
||||
/// without the typed option. These tests assert
|
||||
/// <see cref="ServiceCollectionExtensions.BuildClusterOptions"/> sets the typed
|
||||
/// <c>KeepOldestOption { DownIfAlone = true }</c> so the strategy is explicit in code rather than relying on
|
||||
/// the framework default — a future refactor cannot silently drop that explicitness. (Failover behaviour is
|
||||
/// separately verified by <c>HardKillFailoverTests</c>, which passes regardless of activation path.)
|
||||
/// </summary>
|
||||
public sealed class SplitBrainResolverActivationTests
|
||||
{
|
||||
@@ -21,14 +24,14 @@ public sealed class SplitBrainResolverActivationTests
|
||||
Roles = new[] { "admin", "driver" },
|
||||
};
|
||||
|
||||
/// <summary>Verifies the cluster options carry a split-brain resolver (provider gets activated).</summary>
|
||||
/// <summary>Verifies the cluster options carry an explicit split-brain resolver strategy.</summary>
|
||||
[Fact]
|
||||
public void BuildClusterOptions_activates_a_split_brain_resolver()
|
||||
public void BuildClusterOptions_sets_an_explicit_split_brain_resolver()
|
||||
{
|
||||
var options = ServiceCollectionExtensions.BuildClusterOptions(SampleOptions());
|
||||
|
||||
options.SplitBrainResolver.ShouldNotBeNull(
|
||||
"SplitBrainResolver must be set — without it the cluster runs default NoDowning and never fails over on a hard crash.");
|
||||
"SplitBrainResolver must be set explicitly in code — keeps the strategy independent of Akka.Cluster.Hosting's default provider rather than relying on it.");
|
||||
}
|
||||
|
||||
/// <summary>Verifies the resolver is keep-oldest with down-if-alone (correct for a 2-node pair).</summary>
|
||||
|
||||
Reference in New Issue
Block a user