Files
lmxopcua/archreview/plans/FOLLOWUP-11-crit1-sbr-premise.md
T
Joseph Doherty e233db7456 docs(archreview): record #11 CORRECTED (Crit-1 premise) + track residual #12/#13
- FOLLOWUP-11 → IMPLEMENTED with outcome (docs/comment-only, no code revert; the four
  corrected sites; Cluster.Tests 29/29).
- STATUS.md: task #11 → completed; crit1 branch head → eaf78aad; Completed-table row;
  Findings #11 bullet flipped to CORRECTED; added task rows #12 (live gate) + #13
  (ResilienceConfig-in-artifact).
2026-07-08 22:36:10 -04:00

5.7 KiB

Follow-up #11 — Correct Critical 1's premise: SBR was already active on master (not NoDowning)

Status: IMPLEMENTED (fix/archreview-crit1-split-brain-resolver eaf78aad) · Surfaced by: the #9 hard-kill failover test's negative control on a25c9ed0 · Task: TaskCreate #11 · Severity: Low (docs/rationale accuracy; the code change is harmless) · Effort: S. See STATUS.md.

Outcome (implemented)

Docs/comment-only correction landed on the crit1 branch (eaf78aad) — no code reverted (the typed KeepOldestOption { DownIfAlone = true } is correct belt-and-suspenders). Corrected the inaccurate "HOCON inert / cluster runs NoDowning / never fails over" framing in all four places it appeared:

  1. ServiceCollectionExtensions.BuildClusterOptions XML comment → now states Akka.Cluster.Hosting enables an SBR provider by default (applies SplitBrainResolverOption.Default when the option is null, reading the akka.conf keep-oldest block); the typed option makes the strategy explicit in code, reinforcing not activating.
  2. akka.conf split-brain-resolver comment → the block IS active (via the Hosting default); the typed option makes it explicit; only an explicit downing-provider-class = "" would disable failover.
  3. docs/Redundancy.md Split-brain section → rewrote the "activated in code / without it → NoDowning" paragraph; dropped the "now triggers failover" wording; cross-referenced HardKillFailoverTests + its negative control.
  4. SplitBrainResolverActivationTests class summary + the ShouldNotBeNull message; renamed BuildClusterOptions_activates_a_split_brain_resolver..._sets_an_explicit_split_brain_resolver.

Verification: Cluster.Tests 29/29 green (the guards still assert the typed option is set — a refactor can't silently drop the explicitness). Failover behaviour remains guarded by HardKillFailoverTests, which passes regardless of activation path.

The finding

Critical 1 (03/S1) claimed: the akka.conf split-brain-resolver block was inert, the cluster ran Akka's default NoDowning, and setting the typed ClusterOptions.SplitBrainResolver = KeepOldestOption { DownIfAlone = true } is what activated the resolver so hard-crashed nodes fail over.

That premise is inaccurate. The cluster was never running NoDowning; hard-crash failover already worked on master before Critical 1.

Evidence (empirical + source)

  1. The #9 failover test's negative control. With the typed SplitBrainResolver option removed, the hard-kill failover test still passes — node B downs the alone crashed oldest node and takes over. Only forcing explicit NoDowning (akka.cluster.downing-provider-class = "") makes the test time out (node stuck Up-but-Unreachable). So removing Critical 1's change does not disable failover; only an explicit NoDowning does.

  2. Akka.Cluster.Hosting default (v1.5.51). WithClustering(ClusterOptions) applies SplitBrainResolverOption.Default when SplitBrainResolver is null — i.e. it enables the SBR downing provider by default. That provider then reads the akka.cluster.split-brain-resolver HOCON. (Confirmed by IL inspection of Akka.Cluster.Hosting.dll: the null branch loads …Default and calls .Apply(builder, …).)

  3. The akka.conf keep-oldest block pre-existed Critical 1. git show master:…/Resources/akka.conf already contains split-brain-resolver { active-strategy = "keep-oldest"; stable-after = 15s; keep-oldest { down-if-alone = on } }. Critical 1's diff to akka.conf added only comments (plus the typed option in ServiceCollectionExtensions.BuildClusterOptions).

Net: the effective resolver on master was already keep-oldest + down-if-alone (active via Akka.Cluster.Hosting's default provider reading the pre-existing akka.conf block). Critical 1's typed option produces the same effective behavior — it is reinforcing/explicit, not the activator.

What is (and isn't) wrong

  • Wrong: the claim that SBR was inert / the cluster ran NoDowning / hard-crash failover was broken before Critical 1. This appears in the BuildClusterOptions XML comment, the akka.conf comment Critical 1 added, and the Redundancy.md / Architecture.md wording Critical 1 touched.
  • Fine to keep: the typed KeepOldestOption { DownIfAlone = true } itself. Making the strategy explicit in code (independent of the framework's default) is defensible belt-and-suspenders and matches the akka.conf block. Do not revert the code.

Proposed correction

  1. Rewrite the BuildClusterOptions XML comment to state accurately: Akka.Cluster.Hosting enables an SBR downing provider by default (applies SplitBrainResolverOption.Default when the option is null), which reads the akka.conf keep-oldest block; this typed option makes the strategy explicit in code rather than relying on the framework default — it is not the sole activator.
  2. Correct the akka.conf comment (drop "inert" / "NoDowning" framing).
  3. Correct docs/Redundancy.md + docs/v2/Architecture.md wording Critical 1 added — do not describe Critical 1 as "fixing a NoDowning bug."
  4. Keep the #9 HardKillFailoverTests as the standing guard — it verifies the failover OUTCOME regardless of activation path, and its comments already reflect the corrected understanding.

Decision needed

Confirm the correction (docs + comments only, no code revert), or flag if there is production context where the akka.conf block or the Hosting default would not apply (none found — production uses the same WithClustering path and ships the same akka.conf). Bundle with the docs branch (low-risk).