e233db7456
- FOLLOWUP-11 → IMPLEMENTED with outcome (docs/comment-only, no code revert; the four corrected sites; Cluster.Tests 29/29). - STATUS.md: task #11 → completed; crit1 branch head → eaf78aad; Completed-table row; Findings #11 bullet flipped to CORRECTED; added task rows #12 (live gate) + #13 (ResilienceConfig-in-artifact).
87 lines
5.7 KiB
Markdown
87 lines
5.7 KiB
Markdown
# Follow-up #11 — Correct Critical 1's premise: SBR was already active on master (not NoDowning)
|
|
|
|
> **Status:** ✅ IMPLEMENTED (`fix/archreview-crit1-split-brain-resolver` `eaf78aad`) · **Surfaced by:** the #9
|
|
> hard-kill failover test's negative control on `a25c9ed0` · **Task:** TaskCreate #11 · **Severity:** Low
|
|
> (docs/rationale accuracy; the code change is harmless) · **Effort:** S. See [`STATUS.md`](STATUS.md).
|
|
|
|
## Outcome (implemented)
|
|
|
|
Docs/comment-only correction landed on the crit1 branch (`eaf78aad`) — **no code reverted** (the typed
|
|
`KeepOldestOption { DownIfAlone = true }` is correct belt-and-suspenders). Corrected the inaccurate
|
|
"HOCON inert / cluster runs NoDowning / never fails over" framing in all four places it appeared:
|
|
|
|
1. `ServiceCollectionExtensions.BuildClusterOptions` XML comment → now states Akka.Cluster.Hosting enables
|
|
an SBR provider by default (applies `SplitBrainResolverOption.Default` when the option is null, reading
|
|
the akka.conf `keep-oldest` block); the typed option makes the strategy **explicit in code**, reinforcing
|
|
not activating.
|
|
2. `akka.conf` `split-brain-resolver` comment → the block IS active (via the Hosting default); the typed
|
|
option makes it explicit; only an explicit `downing-provider-class = ""` would disable failover.
|
|
3. `docs/Redundancy.md` Split-brain section → rewrote the "activated in code / without it → NoDowning"
|
|
paragraph; dropped the "now triggers failover" wording; cross-referenced `HardKillFailoverTests` + its
|
|
negative control.
|
|
4. `SplitBrainResolverActivationTests` class summary + the `ShouldNotBeNull` message; renamed
|
|
`BuildClusterOptions_activates_a_split_brain_resolver` → `..._sets_an_explicit_split_brain_resolver`.
|
|
|
|
**Verification:** Cluster.Tests 29/29 green (the guards still assert the typed option is set — a refactor
|
|
can't silently drop the explicitness). Failover *behaviour* remains guarded by `HardKillFailoverTests`,
|
|
which passes regardless of activation path.
|
|
|
|
## The finding
|
|
|
|
Critical 1 (03/S1) claimed: *the akka.conf `split-brain-resolver` block was inert, the cluster ran Akka's
|
|
default **NoDowning**, and setting the typed `ClusterOptions.SplitBrainResolver = KeepOldestOption {
|
|
DownIfAlone = true }` is what activated the resolver so hard-crashed nodes fail over.*
|
|
|
|
**That premise is inaccurate.** The cluster was **never** running NoDowning; hard-crash failover already
|
|
worked on master before Critical 1.
|
|
|
|
### Evidence (empirical + source)
|
|
|
|
1. **The `#9` failover test's negative control.** With the typed `SplitBrainResolver` option **removed**,
|
|
the hard-kill failover test **still passes** — node B downs the alone crashed oldest node and takes
|
|
over. Only forcing **explicit** NoDowning (`akka.cluster.downing-provider-class = ""`) makes the test
|
|
time out (node stuck Up-but-Unreachable). So removing Critical 1's change does *not* disable failover;
|
|
only an explicit NoDowning does.
|
|
|
|
2. **Akka.Cluster.Hosting default (v1.5.51).** `WithClustering(ClusterOptions)` applies
|
|
`SplitBrainResolverOption.Default` when `SplitBrainResolver` is `null` — i.e. it **enables the SBR
|
|
downing provider by default**. That provider then reads the `akka.cluster.split-brain-resolver` HOCON.
|
|
(Confirmed by IL inspection of `Akka.Cluster.Hosting.dll`: the null branch loads `…Default` and calls
|
|
`.Apply(builder, …)`.)
|
|
|
|
3. **The akka.conf keep-oldest block pre-existed Critical 1.** `git show master:…/Resources/akka.conf`
|
|
already contains `split-brain-resolver { active-strategy = "keep-oldest"; stable-after = 15s;
|
|
keep-oldest { down-if-alone = on } }`. Critical 1's diff to akka.conf added **only comments** (plus the
|
|
typed option in `ServiceCollectionExtensions.BuildClusterOptions`).
|
|
|
|
**Net:** the effective resolver on master was already `keep-oldest` + `down-if-alone` (active via
|
|
Akka.Cluster.Hosting's default provider reading the pre-existing akka.conf block). Critical 1's typed
|
|
option produces the *same* effective behavior — it is **reinforcing/explicit**, not the activator.
|
|
|
|
## What is (and isn't) wrong
|
|
|
|
- ❌ **Wrong:** the claim that SBR was inert / the cluster ran NoDowning / hard-crash failover was broken
|
|
before Critical 1. This appears in the `BuildClusterOptions` XML comment, the akka.conf comment Critical 1
|
|
added, and the `Redundancy.md` / `Architecture.md` wording Critical 1 touched.
|
|
- ✅ **Fine to keep:** the typed `KeepOldestOption { DownIfAlone = true }` itself. Making the strategy
|
|
explicit in code (independent of the framework's default) is defensible belt-and-suspenders and matches
|
|
the akka.conf block. **Do not revert the code.**
|
|
|
|
## Proposed correction
|
|
|
|
1. Rewrite the `BuildClusterOptions` XML comment to state accurately: *Akka.Cluster.Hosting enables an SBR
|
|
downing provider by default (applies `SplitBrainResolverOption.Default` when the option is null), which
|
|
reads the akka.conf `keep-oldest` block; this typed option makes the strategy **explicit in code** rather
|
|
than relying on the framework default — it is not the sole activator.*
|
|
2. Correct the akka.conf comment (drop "inert" / "NoDowning" framing).
|
|
3. Correct `docs/Redundancy.md` + `docs/v2/Architecture.md` wording Critical 1 added — do **not** describe
|
|
Critical 1 as "fixing a NoDowning bug."
|
|
4. Keep the #9 `HardKillFailoverTests` as the standing guard — it verifies the failover OUTCOME regardless
|
|
of activation path, and its comments already reflect the corrected understanding.
|
|
|
|
## Decision needed
|
|
|
|
Confirm the correction (docs + comments only, no code revert), or flag if there is production context where
|
|
the akka.conf block or the Hosting default would *not* apply (none found — production uses the same
|
|
`WithClustering` path and ships the same akka.conf). Bundle with the docs branch (low-risk).
|