feat(cluster): auto-down downing strategy — either-node crash now fails over (owner decision 2026-07-21: availability over partition-safety)
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.
SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.
Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
This commit is contained in:
@@ -12,9 +12,18 @@ namespace ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
|
||||
/// </summary>
|
||||
public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOptions>
|
||||
{
|
||||
/// <summary>Split-brain resolver strategies safe for ScadaBridge's two-node clusters.</summary>
|
||||
/// <summary>
|
||||
/// Downing strategies supported for ScadaBridge's two-node clusters.
|
||||
/// <c>auto-down</c> (default) survives a crash of either node at the accepted cost
|
||||
/// of dual-active during a real partition; <c>keep-oldest</c> is partition-safe but
|
||||
/// cannot survive a crash of the oldest node. Quorum strategies are rejected:
|
||||
/// <c>static-quorum</c> quorum-size 1 trips Akka's IsTooManyMembers guard (DownAll
|
||||
/// on any unreachability in a 2-node cluster) and <c>keep-majority</c> keys the
|
||||
/// fatal crash to the lowest-address node instead of the oldest.
|
||||
/// </summary>
|
||||
private static readonly HashSet<string> AllowedStrategies = new(StringComparer.OrdinalIgnoreCase)
|
||||
{
|
||||
"auto-down",
|
||||
"keep-oldest"
|
||||
};
|
||||
|
||||
@@ -37,8 +46,9 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
|
||||
builder.RequireThat(
|
||||
!string.IsNullOrWhiteSpace(options.SplitBrainResolverStrategy)
|
||||
&& AllowedStrategies.Contains(options.SplitBrainResolverStrategy),
|
||||
$"ClusterOptions.SplitBrainResolverStrategy must be 'keep-oldest' for a two-node cluster; " +
|
||||
$"'{options.SplitBrainResolverStrategy}' would risk a total cluster shutdown on a partition.");
|
||||
$"ClusterOptions.SplitBrainResolverStrategy must be 'auto-down' or 'keep-oldest' for a " +
|
||||
$"two-node cluster; '{options.SplitBrainResolverStrategy}' would risk a total cluster " +
|
||||
"shutdown on a partition or an unreachability event.");
|
||||
|
||||
builder.RequireThat(options.MinNrOfMembers == 1,
|
||||
$"ClusterOptions.MinNrOfMembers must be 1 (was {options.MinNrOfMembers}); " +
|
||||
@@ -58,7 +68,11 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
|
||||
$"FailureDetectionThreshold ({options.FailureDetectionThreshold}); otherwise nodes are " +
|
||||
"declared unreachable before a heartbeat can arrive.");
|
||||
|
||||
builder.RequireThat(options.DownIfAlone,
|
||||
// DownIfAlone is a keep-oldest knob; under auto-down each side downs the
|
||||
// unreachable peer regardless, so the flag is inert and any value is fine.
|
||||
var isKeepOldest = string.Equals(
|
||||
options.SplitBrainResolverStrategy, "keep-oldest", StringComparison.OrdinalIgnoreCase);
|
||||
builder.RequireThat(!isKeepOldest || options.DownIfAlone,
|
||||
"ClusterOptions.DownIfAlone must be true for the keep-oldest resolver "
|
||||
+ "(Component-ClusterInfrastructure.md → Split-Brain Resolution); with it false the "
|
||||
+ "oldest node can run as an isolated single-node cluster during a partition while the "
|
||||
|
||||
Reference in New Issue
Block a user