dohertj2/lmxopcua
1
0
Fork 0
Code Issues 8 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(cluster): auto-down downing strategy — a two-node pair could not survive an oldest-node crash

Browse Source 
Ports the sister project's live-proven fix (ScadaBridge cf3bd52f). OtOpcUa ran
the identical configuration it indicts: keep-oldest with down-if-alone = on.

Akka.NET 1.5.62's KeepOldest.OldestDecision only lets the down-if-alone branch
rescue a side holding >= 2 members. A two-node cluster losing a peer is always a
1-vs-1 split, so the branch never fires, the lone survivor falls through to
DownReachable and downs ITSELF, and run-coordinated-shutdown-when-down then
terminates it. down-if-alone is a 3+-node feature and does not do what its name
suggests for a pair: a crash of the oldest node is a total outage, which is the
exact failure the redundancy pair exists to absorb.

Cluster:SplitBrainResolverStrategy now selects the provider, defaulting to
auto-down: Akka's AutoDowning with auto-down-unreachable-after = 15s, so the
leader among the reachable members downs the unreachable peer and a crash of
either node fails over in place. keep-oldest remains available for deployments
that would rather take an outage than ever run dual-active during a real
partition. An unrecognised value fails the host at startup rather than falling
through to the fatal default.

The tests assert the EFFECTIVE configuration — they start a real host through
WithOtOpcUaClusterBootstrap and read akka.cluster.downing-provider-class back off
the running ActorSystem. This is not incidental. The first draft inlined the
three calls the bootstrap makes instead of calling it, which pinned the test's
own wiring: sabotaging production's HOCON precedence left all 36 green. The
prior file had the same shape at a smaller scale, asserting only that
BuildClusterOptions returned a KeepOldestOption — true, and true of a
configuration that cannot fail over. Positive control: making BuildDowningHocon
emit nothing for auto-down turns exactly the two effective-config guards red.

Measured while verifying rather than assumed: HoconAddMode.Append also wins here,
purely because it is added last, so the mode name is not the guarantee. The
comment now says so instead of asserting a precedence rule that does not hold.

Not yet live-drilled on OtOpcUa. The docker-dev rig is a single six-node mesh
where the 1-vs-1 pathology cannot occur; the kill-the-oldest drill belongs with
the per-cluster mesh work that makes every mesh exactly two nodes (design doc
6.2 / Phase 0a). Recorded as an outstanding gate in docs/Redundancy.md.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-21 18:32:42 -04:00
parent c6fb8bb0ad
commit 2964361a6f
5 changed files with 427 additions and 97 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs
+26
View File
@@ -29,4 +29,30 @@ public sealed class AkkaClusterOptions
/// <c>admin</c>, <c>driver</c>, <c>dev</c>.
/// </summary>
public string[] Roles { get; set; } = Array.Empty<string>();
/// <summary>
/// How the cluster decides to down a node it can no longer reach. One of <c>auto-down</c>
/// (default) or <c>keep-oldest</c>; any other value fails the host at startup.
/// </summary>
/// <remarks>
/// <para>
/// <b><c>auto-down</c> — availability.</b> The leader among the <i>reachable</i> members
/// downs the unreachable peer after
/// <see cref="ServiceCollectionExtensions.DowningStableAfter"/>. A hard crash of either
/// node — including the oldest — fails over to the survivor with no operator action.
/// The trade is that a genuine network partition (both nodes alive, link cut) leaves
/// both sides running active until an operator restarts one.
/// </para>
/// <para>
/// <b><c>keep-oldest</c> — partition-safety.</b> The SBR resolver sacrifices the younger
/// side of a split, so a partition can never run dual-active. <b>The cost is severe for a
/// two-node pair:</b> it cannot survive a crash of the oldest node at all. Akka.NET's
/// <c>KeepOldest.OldestDecision</c> only lets <c>down-if-alone</c> rescue a side holding
/// &gt;= 2 members, so the 1-vs-1 survivor downs <i>itself</i> and shuts down — the
/// redundancy pair turns a single-node crash into a total outage. Choose this only for
/// clusters of three or more nodes, or where dual-active is genuinely worse than an
/// outage.
/// </para>
/// </remarks>
public string SplitBrainResolverStrategy { get; set; } = "auto-down";
}
src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf
+20 -8
View File
@@ -37,14 +37,26 @@ akka {
roles = []
min-nr-of-members = 1
# Split-brain resolver (arch-review 03/S1). This block IS active: Akka.Cluster.Hosting's
# WithClustering enables an SBR downing provider by default (it applies
# SplitBrainResolverOption.Default when ClusterOptions.SplitBrainResolver is null), and that
# provider reads this block. ServiceCollectionExtensions.BuildClusterOptions additionally sets
# the typed KeepOldestOption { DownIfAlone = true } to make the strategy EXPLICIT in code —
# its active-strategy + keep-oldest.down-if-alone MUST match this block. (The cluster is NOT
# NoDowning; only an explicit downing-provider-class = "" would disable failover.)
# stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin.
# DOWNING. The provider is NOT selected here — this file is added with HoconAddMode.Append,
# which loses to what Akka.Hosting's WithClustering emits, so a downing-provider-class set
# here would be silently overridden. Selection lives in
# ServiceCollectionExtensions.BuildDowningHocon (Prepended, which outranks WithClustering),
# driven by Cluster:SplitBrainResolverStrategy.
#
# The DEFAULT strategy is "auto-down", under which this whole block is inert: Akka's
# AutoDowning provider is installed with auto-down-unreachable-after = 15s, and the leader
# among the reachable members downs the unreachable peer — so a hard crash of EITHER node,
# oldest included, fails over. The trade is dual-active during a real network partition.
#
# The block below applies only when an operator selects "keep-oldest". Note that keep-oldest
# is NOT safe for a two-node pair: Akka.NET's KeepOldest.OldestDecision only lets
# down-if-alone rescue a side holding >= 2 members, so a 1-vs-1 survivor downs ITSELF and
# (with run-coordinated-shutdown-when-down) exits — a crash of the oldest node becomes a
# total outage. See docs/Redundancy.md.
#
# stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin, and
# must equal ServiceCollectionExtensions.DowningStableAfter so both strategies fail over at
# the same latency. Both constraints are pinned by SplitBrainResolverActivationTests.
split-brain-resolver {
active-strategy = "keep-oldest"
stable-after = 15s
src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs
+108 -23
View File
@@ -80,40 +80,125 @@ public static class ServiceCollectionExtensions
builder.WithClustering(BuildClusterOptions(options));
// Must come AFTER WithClustering, which always emits a downing-provider-class of its own:
// Akka.Cluster.Hosting applies SplitBrainResolverOption.Default whenever the typed
// ClusterOptions.SplitBrainResolver is null, so there is always a competing value to beat.
// Prepend is the highest-precedence mode, which makes this block win irrespective of how
// many fragments are added or in what order.
//
// Do not take this comment's word for it. Which fragment actually wins is not obvious from
// the mode names — measured, Append also wins here, purely because it happens to be added
// last. SplitBrainResolverActivationTests reads the provider back off a *running*
// ActorSystem for exactly that reason: the effective value is the only one worth asserting.
builder.AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend);
return builder;
}
/// <summary>
/// Builds the <see cref="ClusterOptions"/> for the fused-host cluster, setting the split-brain-resolver
/// strategy <b>explicitly in code</b>.
///
/// <b>Activation note (arch-review 03/S1, corrected):</b> Akka.Cluster.Hosting's <c>WithClustering</c>
/// already enables an SBR downing provider <b>by default</b> — when <see cref="ClusterOptions.SplitBrainResolver"/>
/// is <c>null</c> it applies <c>SplitBrainResolverOption.Default</c>, which registers
/// <c>Akka.Cluster.SBR.SplitBrainResolverProvider</c> and reads the <c>split-brain-resolver</c> HOCON block in
/// <c>Resources/akka.conf</c>. So the cluster was <b>not</b> running <c>NoDowning</c> before this option was
/// set: the pre-existing akka.conf <c>keep-oldest</c> block was already active, and hard-crashed nodes already
/// failed over. Setting this typed option makes the strategy <b>explicit in code</b> (independent of the
/// framework default) rather than being the sole activator — it is reinforcing/belt-and-suspenders and
/// produces the same effective behavior. (Only an <i>explicit</i> <c>NoDowning</c>, e.g.
/// <c>downing-provider-class = ""</c>, would disable failover.)
///
/// <see cref="KeepOldestOption"/> with <c>DownIfAlone=true</c> mirrors the HOCON intent and is the
/// correct strategy for a 2-node warm-redundancy pair: on an even split the oldest member (typically
/// the long-running primary) survives, and <c>down-if-alone</c> downs a singleton that loses its
/// peer. <c>keep-majority</c>/<c>static-quorum</c> are wrong for two nodes. The strategy + down-if-alone
/// here MUST stay consistent with the HOCON block; <c>stable-after</c> lives only in HOCON because the
/// typed option cannot express it (it must stay ≥ <c>failure-detector.acceptable-heartbeat-pause</c>).
/// How long a member must stay unreachable before it is downed. Shared by both strategies
/// (<c>auto-down-unreachable-after</c> and the SBR resolver's <c>stable-after</c>) so the two
/// have the same failover latency. Must stay above
/// <c>akka.cluster.failure-detector.acceptable-heartbeat-pause</c> or a merely-slow node gets
/// downed; both constraints are pinned by tests.
/// </summary>
/// <param name="options">The bound cluster options carrying seed nodes and roles.</param>
/// <returns>The cluster options with seed nodes, roles, and the explicit split-brain-resolver strategy.</returns>
public static readonly TimeSpan DowningStableAfter = TimeSpan.FromSeconds(15);
/// <summary>
/// Builds the HOCON that selects the downing provider, per
/// <see cref="AkkaClusterOptions.SplitBrainResolverStrategy"/>.
/// </summary>
/// <param name="options">The bound cluster options carrying the strategy.</param>
/// <returns>
/// For <c>auto-down</c>, a block installing Akka's <c>AutoDowning</c> provider with a downing
/// window of <see cref="DowningStableAfter"/>. For <c>keep-oldest</c>, an empty string — the
/// typed <see cref="KeepOldestOption"/> from <see cref="BuildClusterOptions"/> already installs
/// the SBR provider, and the resolver's own settings live in <c>Resources/akka.conf</c>.
/// </returns>
/// <exception cref="ArgumentOutOfRangeException">The strategy is not a recognised value.</exception>
public static string BuildDowningHocon(AkkaClusterOptions options)
{
ArgumentNullException.ThrowIfNull(options);
if (IsAutoDown(options))
{
// auto-down-unreachable-after is load-bearing: it defaults to `off`, under which
// AutoDowning is installed but never downs anything — failover silently disabled.
return $$"""
akka.cluster {
downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"
auto-down-unreachable-after = {{(int)DowningStableAfter.TotalMilliseconds}}ms
}
""";
}
if (IsKeepOldest(options))
{
return string.Empty;
}
throw new ArgumentOutOfRangeException(
nameof(options),
options.SplitBrainResolverStrategy,
$"Unknown {AkkaClusterOptions.SectionName}:{nameof(AkkaClusterOptions.SplitBrainResolverStrategy)}. "
+ "Expected 'auto-down' (default; survives a crash of either node) or 'keep-oldest' "
+ "(partition-safe, but a two-node pair cannot survive a crash of the oldest node).");
}
private static bool IsAutoDown(AkkaClusterOptions options) =>
string.Equals(options.SplitBrainResolverStrategy, "auto-down", StringComparison.OrdinalIgnoreCase);
private static bool IsKeepOldest(AkkaClusterOptions options) =>
string.Equals(options.SplitBrainResolverStrategy, "keep-oldest", StringComparison.OrdinalIgnoreCase);
/// <summary>
/// Builds the <see cref="ClusterOptions"/> for the fused-host cluster.
/// </summary>
/// <remarks>
/// <para>
/// <b>Activation note (arch-review 03/S1, corrected twice).</b> Akka.Cluster.Hosting's
/// <c>WithClustering</c> always installs a downing provider: when
/// <see cref="ClusterOptions.SplitBrainResolver"/> is <c>null</c> it applies
/// <c>SplitBrainResolverOption.Default</c>, which registers
/// <c>Akka.Cluster.SBR.SplitBrainResolverProvider</c> and reads the
/// <c>split-brain-resolver</c> block in <c>Resources/akka.conf</c>. The cluster is
/// therefore never <c>NoDowning</c>. What this method chooses is only <i>which</i>
/// provider — and under the default <c>auto-down</c> strategy the choice is not made
/// here at all but by the Prepended HOCON in
/// <see cref="WithOtOpcUaClusterBootstrap"/>, which outranks whatever
/// <c>WithClustering</c> emits.
/// </para>
/// <para>
/// <b>Why keep-oldest is no longer the default (2026-07-21).</b> The previous
/// documentation here asserted that <see cref="KeepOldestOption"/> with
/// <c>DownIfAlone=true</c> was "the correct strategy for a 2-node warm-redundancy pair"
/// because <c>down-if-alone</c> would down a node that lost its peer. That is the
/// opposite of what Akka.NET does. In <c>KeepOldest.OldestDecision</c> the
/// <c>down-if-alone</c> branch requires the surviving side to hold &gt;= 2 members, so
/// in a 1-vs-1 split the survivor falls through to <c>DownReachable</c> and downs
/// <i>itself</i>; with <c>run-coordinated-shutdown-when-down = on</c> it then exits. A
/// two-node pair running keep-oldest converts a crash of the oldest node into a total
/// outage — precisely the failure redundancy exists to absorb. Live-proven on the sister
/// project's rig; see <c>docs/Redundancy.md</c> and
/// <see cref="AkkaClusterOptions.SplitBrainResolverStrategy"/> for the trade.
/// </para>
/// </remarks>
/// <param name="options">The bound cluster options carrying seed nodes, roles and the strategy.</param>
/// <returns>
/// The cluster options with seed nodes and roles. <see cref="ClusterOptions.SplitBrainResolver"/>
/// is set only under <c>keep-oldest</c>, where the SBR provider is the one wanted; under
/// <c>auto-down</c> it is left null because the Prepended HOCON replaces the provider anyway,
/// and naming a resolver that is not in force would be misleading.
/// </returns>
public static ClusterOptions BuildClusterOptions(AkkaClusterOptions options)
{
ArgumentNullException.ThrowIfNull(options);
return new ClusterOptions
{
SeedNodes = options.SeedNodes,
Roles = options.Roles,
SplitBrainResolver = new KeepOldestOption { DownIfAlone = true },
SplitBrainResolver = IsKeepOldest(options) ? new KeepOldestOption { DownIfAlone = true } : null,
};
}
}