fix(cluster): activate split-brain resolver (arch-review 03/S1)

The split-brain-resolver HOCON block in akka.conf was inert — nothing
registered a downing provider, so the cluster ran Akka's default NoDowning:
a hard-crashed node was never downed, cluster singletons + the driver
role-leader never failed over, and a partition left both redundancy sides
at ServiceLevel 240 indefinitely.

Activate the resolver via the typed ClusterOptions.SplitBrainResolver
(KeepOldestOption { DownIfAlone = true }) in a new testable
BuildClusterOptions helper. keep-oldest + down-if-alone is correct for a
2-node warm-redundancy pair. HOCON keeps stable-after (typed option can't
express it) + a cross-reference comment. Deterministic unit guard added;
end-to-end verified the resolved ActorSystem config carries the provider
and stable-after survives the HOCON/Hosting merge. Redundancy.md corrected.

Hard-kill failover integration test tracked as a follow-up (effort-M harness).
This commit is contained in:
Joseph Doherty
2026-07-08 16:14:26 -04:00
parent 9cad9ed0fc
commit a81dea102c
4 changed files with 112 additions and 6 deletions
@@ -37,6 +37,13 @@ akka {
roles = []
min-nr-of-members = 1
# Split-brain resolver. This HOCON block is the TUNING source only — it does NOT
# activate the resolver. Activation happens in code via the typed
# ClusterOptions.SplitBrainResolver (KeepOldestOption { DownIfAlone = true }) set in
# ServiceCollectionExtensions.BuildClusterOptions; without that the cluster runs Akka's
# default NoDowning and hard-crashed nodes never fail over (arch-review 03/S1).
# active-strategy + keep-oldest.down-if-alone below MUST match the typed option.
# stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin.
split-brain-resolver {
active-strategy = "keep-oldest"
stable-after = 15s