fix(cluster): auto-down downing strategy — a two-node pair could not survive an oldest-node crash

Ports the sister project's live-proven fix (ScadaBridge cf3bd52f). OtOpcUa ran
the identical configuration it indicts: keep-oldest with down-if-alone = on.

Akka.NET 1.5.62's KeepOldest.OldestDecision only lets the down-if-alone branch
rescue a side holding >= 2 members. A two-node cluster losing a peer is always a
1-vs-1 split, so the branch never fires, the lone survivor falls through to
DownReachable and downs ITSELF, and run-coordinated-shutdown-when-down then
terminates it. down-if-alone is a 3+-node feature and does not do what its name
suggests for a pair: a crash of the oldest node is a total outage, which is the
exact failure the redundancy pair exists to absorb.

Cluster:SplitBrainResolverStrategy now selects the provider, defaulting to
auto-down: Akka's AutoDowning with auto-down-unreachable-after = 15s, so the
leader among the reachable members downs the unreachable peer and a crash of
either node fails over in place. keep-oldest remains available for deployments
that would rather take an outage than ever run dual-active during a real
partition. An unrecognised value fails the host at startup rather than falling
through to the fatal default.

The tests assert the EFFECTIVE configuration — they start a real host through
WithOtOpcUaClusterBootstrap and read akka.cluster.downing-provider-class back off
the running ActorSystem. This is not incidental. The first draft inlined the
three calls the bootstrap makes instead of calling it, which pinned the test's
own wiring: sabotaging production's HOCON precedence left all 36 green. The
prior file had the same shape at a smaller scale, asserting only that
BuildClusterOptions returned a KeepOldestOption — true, and true of a
configuration that cannot fail over. Positive control: making BuildDowningHocon
emit nothing for auto-down turns exactly the two effective-config guards red.

Measured while verifying rather than assumed: HoconAddMode.Append also wins here,
purely because it is added last, so the mode name is not the guarantee. The
comment now says so instead of asserting a precedence rule that does not hold.

Not yet live-drilled on OtOpcUa. The docker-dev rig is a single six-node mesh
where the 1-vs-1 pathology cannot occur; the kill-the-oldest drill belongs with
the per-cluster mesh work that makes every mesh exactly two nodes (design doc
6.2 / Phase 0a). Recorded as an outstanding gate in docs/Redundancy.md.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-21 18:32:42 -04:00
parent c6fb8bb0ad
commit 2964361a6f
5 changed files with 427 additions and 97 deletions
@@ -37,14 +37,26 @@ akka {
roles = []
min-nr-of-members = 1
# Split-brain resolver (arch-review 03/S1). This block IS active: Akka.Cluster.Hosting's
# WithClustering enables an SBR downing provider by default (it applies
# SplitBrainResolverOption.Default when ClusterOptions.SplitBrainResolver is null), and that
# provider reads this block. ServiceCollectionExtensions.BuildClusterOptions additionally sets
# the typed KeepOldestOption { DownIfAlone = true } to make the strategy EXPLICIT in code —
# its active-strategy + keep-oldest.down-if-alone MUST match this block. (The cluster is NOT
# NoDowning; only an explicit downing-provider-class = "" would disable failover.)
# stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin.
# DOWNING. The provider is NOT selected here — this file is added with HoconAddMode.Append,
# which loses to what Akka.Hosting's WithClustering emits, so a downing-provider-class set
# here would be silently overridden. Selection lives in
# ServiceCollectionExtensions.BuildDowningHocon (Prepended, which outranks WithClustering),
# driven by Cluster:SplitBrainResolverStrategy.
#
# The DEFAULT strategy is "auto-down", under which this whole block is inert: Akka's
# AutoDowning provider is installed with auto-down-unreachable-after = 15s, and the leader
# among the reachable members downs the unreachable peer — so a hard crash of EITHER node,
# oldest included, fails over. The trade is dual-active during a real network partition.
#
# The block below applies only when an operator selects "keep-oldest". Note that keep-oldest
# is NOT safe for a two-node pair: Akka.NET's KeepOldest.OldestDecision only lets
# down-if-alone rescue a side holding >= 2 members, so a 1-vs-1 survivor downs ITSELF and
# (with run-coordinated-shutdown-when-down) exits — a crash of the oldest node becomes a
# total outage. See docs/Redundancy.md.
#
# stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin, and
# must equal ServiceCollectionExtensions.DowningStableAfter so both strategies fail over at
# the same latency. Both constraints are pinned by SplitBrainResolverActivationTests.
split-brain-resolver {
active-strategy = "keep-oldest"
stable-after = 15s