- FOLLOWUP-11 → IMPLEMENTED with outcome (docs/comment-only, no code revert; the four corrected sites; Cluster.Tests 29/29). - STATUS.md: task #11 → completed; crit1 branch head → eaf78aad; Completed-table row; Findings #11 bullet flipped to CORRECTED; added task rows #12 (live gate) + #13 (ResilienceConfig-in-artifact).
5.7 KiB
Follow-up #11 — Correct Critical 1's premise: SBR was already active on master (not NoDowning)
Status: ✅ IMPLEMENTED (
fix/archreview-crit1-split-brain-resolvereaf78aad) · Surfaced by: the #9 hard-kill failover test's negative control ona25c9ed0· Task: TaskCreate #11 · Severity: Low (docs/rationale accuracy; the code change is harmless) · Effort: S. SeeSTATUS.md.
Outcome (implemented)
Docs/comment-only correction landed on the crit1 branch (eaf78aad) — no code reverted (the typed
KeepOldestOption { DownIfAlone = true } is correct belt-and-suspenders). Corrected the inaccurate
"HOCON inert / cluster runs NoDowning / never fails over" framing in all four places it appeared:
ServiceCollectionExtensions.BuildClusterOptionsXML comment → now states Akka.Cluster.Hosting enables an SBR provider by default (appliesSplitBrainResolverOption.Defaultwhen the option is null, reading the akka.confkeep-oldestblock); the typed option makes the strategy explicit in code, reinforcing not activating.akka.confsplit-brain-resolvercomment → the block IS active (via the Hosting default); the typed option makes it explicit; only an explicitdowning-provider-class = ""would disable failover.docs/Redundancy.mdSplit-brain section → rewrote the "activated in code / without it → NoDowning" paragraph; dropped the "now triggers failover" wording; cross-referencedHardKillFailoverTests+ its negative control.SplitBrainResolverActivationTestsclass summary + theShouldNotBeNullmessage; renamedBuildClusterOptions_activates_a_split_brain_resolver→..._sets_an_explicit_split_brain_resolver.
Verification: Cluster.Tests 29/29 green (the guards still assert the typed option is set — a refactor
can't silently drop the explicitness). Failover behaviour remains guarded by HardKillFailoverTests,
which passes regardless of activation path.
The finding
Critical 1 (03/S1) claimed: the akka.conf split-brain-resolver block was inert, the cluster ran Akka's
default NoDowning, and setting the typed ClusterOptions.SplitBrainResolver = KeepOldestOption { DownIfAlone = true } is what activated the resolver so hard-crashed nodes fail over.
That premise is inaccurate. The cluster was never running NoDowning; hard-crash failover already worked on master before Critical 1.
Evidence (empirical + source)
-
The
#9failover test's negative control. With the typedSplitBrainResolveroption removed, the hard-kill failover test still passes — node B downs the alone crashed oldest node and takes over. Only forcing explicit NoDowning (akka.cluster.downing-provider-class = "") makes the test time out (node stuck Up-but-Unreachable). So removing Critical 1's change does not disable failover; only an explicit NoDowning does. -
Akka.Cluster.Hosting default (v1.5.51).
WithClustering(ClusterOptions)appliesSplitBrainResolverOption.DefaultwhenSplitBrainResolverisnull— i.e. it enables the SBR downing provider by default. That provider then reads theakka.cluster.split-brain-resolverHOCON. (Confirmed by IL inspection ofAkka.Cluster.Hosting.dll: the null branch loads…Defaultand calls.Apply(builder, …).) -
The akka.conf keep-oldest block pre-existed Critical 1.
git show master:…/Resources/akka.confalready containssplit-brain-resolver { active-strategy = "keep-oldest"; stable-after = 15s; keep-oldest { down-if-alone = on } }. Critical 1's diff to akka.conf added only comments (plus the typed option inServiceCollectionExtensions.BuildClusterOptions).
Net: the effective resolver on master was already keep-oldest + down-if-alone (active via
Akka.Cluster.Hosting's default provider reading the pre-existing akka.conf block). Critical 1's typed
option produces the same effective behavior — it is reinforcing/explicit, not the activator.
What is (and isn't) wrong
- ❌ Wrong: the claim that SBR was inert / the cluster ran NoDowning / hard-crash failover was broken
before Critical 1. This appears in the
BuildClusterOptionsXML comment, the akka.conf comment Critical 1 added, and theRedundancy.md/Architecture.mdwording Critical 1 touched. - ✅ Fine to keep: the typed
KeepOldestOption { DownIfAlone = true }itself. Making the strategy explicit in code (independent of the framework's default) is defensible belt-and-suspenders and matches the akka.conf block. Do not revert the code.
Proposed correction
- Rewrite the
BuildClusterOptionsXML comment to state accurately: Akka.Cluster.Hosting enables an SBR downing provider by default (appliesSplitBrainResolverOption.Defaultwhen the option is null), which reads the akka.confkeep-oldestblock; this typed option makes the strategy explicit in code rather than relying on the framework default — it is not the sole activator. - Correct the akka.conf comment (drop "inert" / "NoDowning" framing).
- Correct
docs/Redundancy.md+docs/v2/Architecture.mdwording Critical 1 added — do not describe Critical 1 as "fixing a NoDowning bug." - Keep the #9
HardKillFailoverTestsas the standing guard — it verifies the failover OUTCOME regardless of activation path, and its comments already reflect the corrected understanding.
Decision needed
Confirm the correction (docs + comments only, no code revert), or flag if there is production context where
the akka.conf block or the Hosting default would not apply (none found — production uses the same
WithClustering path and ships the same akka.conf). Bundle with the docs branch (low-risk).