4.1 KiB
Follow-up #11 — Correct Critical 1's premise: SBR was already active on master (not NoDowning)
Status: OPEN · Surfaced by: the #9 hard-kill failover test's negative control on
fix/archreview-crit1-split-brain-resolvera25c9ed0· Task: TaskCreate #11 · Severity: Low (docs/rationale accuracy; the code change is harmless) · Effort: S. SeeSTATUS.md.
The finding
Critical 1 (03/S1) claimed: the akka.conf split-brain-resolver block was inert, the cluster ran Akka's
default NoDowning, and setting the typed ClusterOptions.SplitBrainResolver = KeepOldestOption { DownIfAlone = true } is what activated the resolver so hard-crashed nodes fail over.
That premise is inaccurate. The cluster was never running NoDowning; hard-crash failover already worked on master before Critical 1.
Evidence (empirical + source)
-
The
#9failover test's negative control. With the typedSplitBrainResolveroption removed, the hard-kill failover test still passes — node B downs the alone crashed oldest node and takes over. Only forcing explicit NoDowning (akka.cluster.downing-provider-class = "") makes the test time out (node stuck Up-but-Unreachable). So removing Critical 1's change does not disable failover; only an explicit NoDowning does. -
Akka.Cluster.Hosting default (v1.5.51).
WithClustering(ClusterOptions)appliesSplitBrainResolverOption.DefaultwhenSplitBrainResolverisnull— i.e. it enables the SBR downing provider by default. That provider then reads theakka.cluster.split-brain-resolverHOCON. (Confirmed by IL inspection ofAkka.Cluster.Hosting.dll: the null branch loads…Defaultand calls.Apply(builder, …).) -
The akka.conf keep-oldest block pre-existed Critical 1.
git show master:…/Resources/akka.confalready containssplit-brain-resolver { active-strategy = "keep-oldest"; stable-after = 15s; keep-oldest { down-if-alone = on } }. Critical 1's diff to akka.conf added only comments (plus the typed option inServiceCollectionExtensions.BuildClusterOptions).
Net: the effective resolver on master was already keep-oldest + down-if-alone (active via
Akka.Cluster.Hosting's default provider reading the pre-existing akka.conf block). Critical 1's typed
option produces the same effective behavior — it is reinforcing/explicit, not the activator.
What is (and isn't) wrong
- ❌ Wrong: the claim that SBR was inert / the cluster ran NoDowning / hard-crash failover was broken
before Critical 1. This appears in the
BuildClusterOptionsXML comment, the akka.conf comment Critical 1 added, and theRedundancy.md/Architecture.mdwording Critical 1 touched. - ✅ Fine to keep: the typed
KeepOldestOption { DownIfAlone = true }itself. Making the strategy explicit in code (independent of the framework's default) is defensible belt-and-suspenders and matches the akka.conf block. Do not revert the code.
Proposed correction
- Rewrite the
BuildClusterOptionsXML comment to state accurately: Akka.Cluster.Hosting enables an SBR downing provider by default (appliesSplitBrainResolverOption.Defaultwhen the option is null), which reads the akka.confkeep-oldestblock; this typed option makes the strategy explicit in code rather than relying on the framework default — it is not the sole activator. - Correct the akka.conf comment (drop "inert" / "NoDowning" framing).
- Correct
docs/Redundancy.md+docs/v2/Architecture.mdwording Critical 1 added — do not describe Critical 1 as "fixing a NoDowning bug." - Keep the #9
HardKillFailoverTestsas the standing guard — it verifies the failover OUTCOME regardless of activation path, and its comments already reflect the corrected understanding.
Decision needed
Confirm the correction (docs + comments only, no code revert), or flag if there is production context where
the akka.conf block or the Hosting default would not apply (none found — production uses the same
WithClustering path and ships the same akka.conf). Bundle with the docs branch (low-risk).