dohertj2/lmxopcua
1
0
Fork 0
Code Issues 1 Pull Requests 1 Actions 1 Packages Projects Releases Wiki Activity

fix(cluster): activate split-brain resolver (arch-review 03/S1)

Browse Source 
The split-brain-resolver HOCON block in akka.conf was inert — nothing
registered a downing provider, so the cluster ran Akka's default NoDowning:
a hard-crashed node was never downed, cluster singletons + the driver
role-leader never failed over, and a partition left both redundancy sides
at ServiceLevel 240 indefinitely.

Activate the resolver via the typed ClusterOptions.SplitBrainResolver
(KeepOldestOption { DownIfAlone = true }) in a new testable
BuildClusterOptions helper. keep-oldest + down-if-alone is correct for a
2-node warm-redundancy pair. HOCON keeps stable-after (typed option can't
express it) + a cross-reference comment. Deterministic unit guard added;
end-to-end verified the resolved ActorSystem config carries the provider
and stable-after survives the HOCON/Hosting merge. Redundancy.md corrected.

Hard-kill failover integration test tracked as a follow-up (effort-M harness).
This commit is contained in:
Joseph Doherty
2026-07-08 16:14:26 -04:00
parent 9cad9ed0fc
commit a81dea102c
4 changed files with 112 additions and 6 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Redundancy.md
+17 -1
View File
@@ -154,7 +154,23 @@ Node A lists Node B's `ApplicationUri` and vice-versa. Validated by `DualEndpoin
## Split-brain
`akka.conf` configures Akka's split-brain resolver with `active-strategy = keep-oldest`, `stable-after = 15s`, and `failure-detector.threshold = 10.0`. Under a clean partition: the oldest member stays up + the smaller (or younger) side downs itself within ~15 seconds. The `RedundancyStateActor` on the surviving partition re-computes from the post-partition `Cluster.State`.
The split-brain resolver is **activated in code** by the typed `ClusterOptions.SplitBrainResolver`
(`KeepOldestOption { DownIfAlone = true }`) set in
`ServiceCollectionExtensions.BuildClusterOptions` — this is what registers
`Akka.Cluster.SBR.SplitBrainResolverProvider`. **Without that typed registration the `split-brain-resolver`
HOCON block in `akka.conf` is inert and the cluster runs Akka's default `NoDowning`**, meaning a
hard-crashed node is never downed and neither the cluster singletons nor the `driver` role-leader ever fail
over (a partition would leave both redundancy sides at ServiceLevel 240 indefinitely). The HOCON block is the
tuning source only: `active-strategy = keep-oldest`, `stable-after = 15s`, `keep-oldest.down-if-alone = on`,
`failure-detector.threshold = 10.0` (its `active-strategy` + `down-if-alone` must stay consistent with the
typed option; `stable-after` lives only in HOCON because the typed option can't express it).
`keep-oldest` is the correct strategy for a 2-node warm-redundancy pair: under a clean partition the oldest
member (typically the long-running primary) stays up and the smaller (or younger) side downs itself within
~`stable-after` seconds; `down-if-alone` downs a node that loses its only peer. `keep-majority`/`static-quorum`
are wrong for two nodes (no majority in a 1-1 split). The `RedundancyStateActor` on the surviving partition
re-computes from the post-partition `Cluster.State`, and a hard-crashed (not gracefully stopped) node now
triggers the same failover.
There is no operator-driven role swap during a partition. Failover is what the cluster does automatically.