diff --git a/docs/Redundancy.md b/docs/Redundancy.md index 7d78c1f9..bd58859a 100644 --- a/docs/Redundancy.md +++ b/docs/Redundancy.md @@ -154,23 +154,28 @@ Node A lists Node B's `ApplicationUri` and vice-versa. Validated by `DualEndpoin ## Split-brain -The split-brain resolver is **activated in code** by the typed `ClusterOptions.SplitBrainResolver` -(`KeepOldestOption { DownIfAlone = true }`) set in -`ServiceCollectionExtensions.BuildClusterOptions` — this is what registers -`Akka.Cluster.SBR.SplitBrainResolverProvider`. **Without that typed registration the `split-brain-resolver` -HOCON block in `akka.conf` is inert and the cluster runs Akka's default `NoDowning`**, meaning a -hard-crashed node is never downed and neither the cluster singletons nor the `driver` role-leader ever fail -over (a partition would leave both redundancy sides at ServiceLevel 240 indefinitely). The HOCON block is the -tuning source only: `active-strategy = keep-oldest`, `stable-after = 15s`, `keep-oldest.down-if-alone = on`, -`failure-detector.threshold = 10.0` (its `active-strategy` + `down-if-alone` must stay consistent with the -typed option; `stable-after` lives only in HOCON because the typed option can't express it). +The split-brain resolver is **active by default**: Akka.Cluster.Hosting's `WithClustering` enables an SBR +downing provider whenever `ClusterOptions.SplitBrainResolver` is null (it applies +`SplitBrainResolverOption.Default`, which registers `Akka.Cluster.SBR.SplitBrainResolverProvider`), and that +provider reads the `split-brain-resolver` HOCON block in `akka.conf`. On top of that, +`ServiceCollectionExtensions.BuildClusterOptions` sets the typed `ClusterOptions.SplitBrainResolver` +(`KeepOldestOption { DownIfAlone = true }`) to make the strategy **explicit in code** rather than relying on +the framework default — it is reinforcing, not the sole activator, and yields the same effective behavior. So +the cluster is **not** running `NoDowning`; hard-crashed nodes are downed and both the cluster singletons and +the `driver` role-leader fail over. (Only an *explicit* `NoDowning`, e.g. +`akka.cluster.downing-provider-class = ""`, would leave both redundancy sides at ServiceLevel 240 +indefinitely.) The HOCON block carries the tuning: `active-strategy = keep-oldest`, `stable-after = 15s`, +`keep-oldest.down-if-alone = on`, `failure-detector.threshold = 10.0` (its `active-strategy` + `down-if-alone` +must stay consistent with the typed option; `stable-after` lives only in HOCON because the typed option can't +express it). `keep-oldest` is the correct strategy for a 2-node warm-redundancy pair: under a clean partition the oldest member (typically the long-running primary) stays up and the smaller (or younger) side downs itself within ~`stable-after` seconds; `down-if-alone` downs a node that loses its only peer. `keep-majority`/`static-quorum` are wrong for two nodes (no majority in a 1-1 split). The `RedundancyStateActor` on the surviving partition -re-computes from the post-partition `Cluster.State`, and a hard-crashed (not gracefully stopped) node now -triggers the same failover. +re-computes from the post-partition `Cluster.State`, and a hard-crashed (not gracefully stopped) node +triggers the same failover — verified by `HardKillFailoverTests` (its negative control confirms failover +survives removing the typed option, and only breaks under an explicit `NoDowning`). There is no operator-driven role swap during a partition. Failover is what the cluster does automatically. diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf b/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf index 4735fa89..806ac578 100644 --- a/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf @@ -37,12 +37,13 @@ akka { roles = [] min-nr-of-members = 1 - # Split-brain resolver. This HOCON block is the TUNING source only — it does NOT - # activate the resolver. Activation happens in code via the typed - # ClusterOptions.SplitBrainResolver (KeepOldestOption { DownIfAlone = true }) set in - # ServiceCollectionExtensions.BuildClusterOptions; without that the cluster runs Akka's - # default NoDowning and hard-crashed nodes never fail over (arch-review 03/S1). - # active-strategy + keep-oldest.down-if-alone below MUST match the typed option. + # Split-brain resolver (arch-review 03/S1). This block IS active: Akka.Cluster.Hosting's + # WithClustering enables an SBR downing provider by default (it applies + # SplitBrainResolverOption.Default when ClusterOptions.SplitBrainResolver is null), and that + # provider reads this block. ServiceCollectionExtensions.BuildClusterOptions additionally sets + # the typed KeepOldestOption { DownIfAlone = true } to make the strategy EXPLICIT in code — + # its active-strategy + keep-oldest.down-if-alone MUST match this block. (The cluster is NOT + # NoDowning; only an explicit downing-provider-class = "" would disable failover.) # stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin. split-brain-resolver { active-strategy = "keep-oldest" diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs index 8d4f1fab..37d108c7 100644 --- a/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs @@ -84,16 +84,19 @@ public static class ServiceCollectionExtensions } /// - /// Builds the for the fused-host cluster, including the - /// activating split-brain-resolver registration. + /// Builds the for the fused-host cluster, setting the split-brain-resolver + /// strategy explicitly in code. /// - /// The split-brain-resolver HOCON block in Resources/akka.conf is inert on its own — - /// Akka.NET only runs the resolver when a downing provider is registered, otherwise the cluster - /// falls back to NoDowning and a hard-crashed node is never downed (singletons and the - /// driver role-leader never fail over, and a partition leaves both redundancy sides at - /// ServiceLevel 240 forever). Setting is what - /// activates Akka.Cluster.SBR.SplitBrainResolverProvider under the hood — this is the piece - /// that was missing (arch-review 03/S1). + /// Activation note (arch-review 03/S1, corrected): Akka.Cluster.Hosting's WithClustering + /// already enables an SBR downing provider by default — when + /// is null it applies SplitBrainResolverOption.Default, which registers + /// Akka.Cluster.SBR.SplitBrainResolverProvider and reads the split-brain-resolver HOCON block in + /// Resources/akka.conf. So the cluster was not running NoDowning before this option was + /// set: the pre-existing akka.conf keep-oldest block was already active, and hard-crashed nodes already + /// failed over. Setting this typed option makes the strategy explicit in code (independent of the + /// framework default) rather than being the sole activator — it is reinforcing/belt-and-suspenders and + /// produces the same effective behavior. (Only an explicit NoDowning, e.g. + /// downing-provider-class = "", would disable failover.) /// /// with DownIfAlone=true mirrors the HOCON intent and is the /// correct strategy for a 2-node warm-redundancy pair: on an even split the oldest member (typically @@ -103,7 +106,7 @@ public static class ServiceCollectionExtensions /// typed option cannot express it (it must stay ≥ failure-detector.acceptable-heartbeat-pause). /// /// The bound cluster options carrying seed nodes and roles. - /// The cluster options with seed nodes, roles, and the activated split-brain resolver. + /// The cluster options with seed nodes, roles, and the explicit split-brain-resolver strategy. public static ClusterOptions BuildClusterOptions(AkkaClusterOptions options) { return new ClusterOptions diff --git a/tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitBrainResolverActivationTests.cs b/tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitBrainResolverActivationTests.cs index 9f404027..64c7e040 100644 --- a/tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitBrainResolverActivationTests.cs +++ b/tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitBrainResolverActivationTests.cs @@ -6,12 +6,15 @@ using Xunit; namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests; /// -/// Guards the split-brain-resolver ACTIVATION (arch-review 03/S1). The split-brain-resolver -/// HOCON block is inert on its own; the resolver only runs when the typed -/// is set. These tests assert -/// registers it, so a future refactor -/// cannot silently drop back to Akka's default NoDowning (the exact "built-but-never-wired" failure -/// mode this finding described). +/// Guards that the split-brain-resolver strategy stays set explicitly in code (arch-review 03/S1, +/// premise corrected — see #11). Akka.Cluster.Hosting already enables an SBR downing provider by default +/// (it applies SplitBrainResolverOption.Default when +/// is null, which reads the akka.conf keep-oldest block), so the cluster is not NoDowning even +/// without the typed option. These tests assert +/// sets the typed +/// KeepOldestOption { DownIfAlone = true } so the strategy is explicit in code rather than relying on +/// the framework default — a future refactor cannot silently drop that explicitness. (Failover behaviour is +/// separately verified by HardKillFailoverTests, which passes regardless of activation path.) /// public sealed class SplitBrainResolverActivationTests { @@ -21,14 +24,14 @@ public sealed class SplitBrainResolverActivationTests Roles = new[] { "admin", "driver" }, }; - /// Verifies the cluster options carry a split-brain resolver (provider gets activated). + /// Verifies the cluster options carry an explicit split-brain resolver strategy. [Fact] - public void BuildClusterOptions_activates_a_split_brain_resolver() + public void BuildClusterOptions_sets_an_explicit_split_brain_resolver() { var options = ServiceCollectionExtensions.BuildClusterOptions(SampleOptions()); options.SplitBrainResolver.ShouldNotBeNull( - "SplitBrainResolver must be set — without it the cluster runs default NoDowning and never fails over on a hard crash."); + "SplitBrainResolver must be set explicitly in code — keeps the strategy independent of Akka.Cluster.Hosting's default provider rather than relying on it."); } /// Verifies the resolver is keep-oldest with down-if-alone (correct for a 2-node pair).