docs(archreview): file follow-up records for open decisions #10 (RESILIENCE-DISPATCH-GAP) + #11 (Crit-1 SBR premise)

2026-07-08 20:06:15 -04:00
parent 14f0cae7d9
commit 9cd1f8c4e7
3 changed files with 145 additions and 0 deletions
@@ -0,0 +1,64 @@
+# Follow-up #11 — Correct Critical 1's premise: SBR was already active on master (not NoDowning)
+
+> **Status:** OPEN · **Surfaced by:** the #9 hard-kill failover test's negative control on
+> `fix/archreview-crit1-split-brain-resolver` `a25c9ed0` · **Task:** TaskCreate #11 · **Severity:** Low
+> (docs/rationale accuracy; the code change is harmless) · **Effort:** S. See [`STATUS.md`](STATUS.md).
+
+## The finding
+
+Critical 1 (03/S1) claimed: *the akka.conf `split-brain-resolver` block was inert, the cluster ran Akka's
+default **NoDowning**, and setting the typed `ClusterOptions.SplitBrainResolver = KeepOldestOption {
+DownIfAlone = true }` is what activated the resolver so hard-crashed nodes fail over.*
+
+**That premise is inaccurate.** The cluster was **never** running NoDowning; hard-crash failover already
+worked on master before Critical 1.
+
+### Evidence (empirical + source)
+
+1. **The `#9` failover test's negative control.** With the typed `SplitBrainResolver` option **removed**,
+   the hard-kill failover test **still passes** — node B downs the alone crashed oldest node and takes
+   over. Only forcing **explicit** NoDowning (`akka.cluster.downing-provider-class = ""`) makes the test
+   time out (node stuck Up-but-Unreachable). So removing Critical 1's change does *not* disable failover;
+   only an explicit NoDowning does.
+
+2. **Akka.Cluster.Hosting default (v1.5.51).** `WithClustering(ClusterOptions)` applies
+   `SplitBrainResolverOption.Default` when `SplitBrainResolver` is `null` — i.e. it **enables the SBR
+   downing provider by default**. That provider then reads the `akka.cluster.split-brain-resolver` HOCON.
+   (Confirmed by IL inspection of `Akka.Cluster.Hosting.dll`: the null branch loads `…Default` and calls
+   `.Apply(builder, …)`.)
+
+3. **The akka.conf keep-oldest block pre-existed Critical 1.** `git show master:…/Resources/akka.conf`
+   already contains `split-brain-resolver { active-strategy = "keep-oldest"; stable-after = 15s;
+   keep-oldest { down-if-alone = on } }`. Critical 1's diff to akka.conf added **only comments** (plus the
+   typed option in `ServiceCollectionExtensions.BuildClusterOptions`).
+
+**Net:** the effective resolver on master was already `keep-oldest` + `down-if-alone` (active via
+Akka.Cluster.Hosting's default provider reading the pre-existing akka.conf block). Critical 1's typed
+option produces the *same* effective behavior — it is **reinforcing/explicit**, not the activator.
+
+## What is (and isn't) wrong
+
+- ❌ **Wrong:** the claim that SBR was inert / the cluster ran NoDowning / hard-crash failover was broken
+  before Critical 1. This appears in the `BuildClusterOptions` XML comment, the akka.conf comment Critical 1
+  added, and the `Redundancy.md` / `Architecture.md` wording Critical 1 touched.
+- ✅ **Fine to keep:** the typed `KeepOldestOption { DownIfAlone = true }` itself. Making the strategy
+  explicit in code (independent of the framework's default) is defensible belt-and-suspenders and matches
+  the akka.conf block. **Do not revert the code.**
+
+## Proposed correction
+
+1. Rewrite the `BuildClusterOptions` XML comment to state accurately: *Akka.Cluster.Hosting enables an SBR
+   downing provider by default (applies `SplitBrainResolverOption.Default` when the option is null), which
+   reads the akka.conf `keep-oldest` block; this typed option makes the strategy **explicit in code** rather
+   than relying on the framework default — it is not the sole activator.*
+2. Correct the akka.conf comment (drop "inert" / "NoDowning" framing).
+3. Correct `docs/Redundancy.md` + `docs/v2/Architecture.md` wording Critical 1 added — do **not** describe
+   Critical 1 as "fixing a NoDowning bug."
+4. Keep the #9 `HardKillFailoverTests` as the standing guard — it verifies the failover OUTCOME regardless
+   of activation path, and its comments already reflect the corrected understanding.
+
+## Decision needed
+
+Confirm the correction (docs + comments only, no code revert), or flag if there is production context where
+the akka.conf block or the Hosting default would *not* apply (none found — production uses the same
+`WithClustering` path and ships the same akka.conf). Bundle with the docs branch (low-risk).