diff --git a/docs/Redundancy.md b/docs/Redundancy.md index a0b5f3ad..6e30ef91 100644 --- a/docs/Redundancy.md +++ b/docs/Redundancy.md @@ -169,55 +169,94 @@ Each node advertises its partner via `OpcUaApplicationHostOptions.PeerApplicatio Node A lists Node B's `ApplicationUri` and vice-versa. Validated by `DualEndpointTests` in `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/` — boots two `OpcUaApplicationHost` instances on loopback, asserts a real OPCFoundation client `Session` reading `Server.ServerArray` from Node A sees both URIs. -## Split-brain +## Split-brain / downing -The split-brain resolver is **active by default**: Akka.Cluster.Hosting's `WithClustering` enables an SBR -downing provider whenever `ClusterOptions.SplitBrainResolver` is null (it applies -`SplitBrainResolverOption.Default`, which registers `Akka.Cluster.SBR.SplitBrainResolverProvider`), and that -provider reads the `split-brain-resolver` HOCON block in `akka.conf`. On top of that, -`ServiceCollectionExtensions.BuildClusterOptions` sets the typed `ClusterOptions.SplitBrainResolver` -(`KeepOldestOption { DownIfAlone = true }`) to make the strategy **explicit in code** rather than relying on -the framework default — it is reinforcing, not the sole activator, and yields the same effective behavior. So -the cluster is **not** running `NoDowning`; hard-crashed nodes are downed and the cluster recovers (how it -recovers depends on *which* node was lost — see the table below). (Only an *explicit* `NoDowning`, e.g. -`akka.cluster.downing-provider-class = ""`, would leave both redundancy sides at ServiceLevel 240 -indefinitely.) The HOCON block carries the tuning: `active-strategy = keep-oldest`, `stable-after = 15s`, -`keep-oldest.down-if-alone = on`, `failure-detector.threshold = 10.0` (its `active-strategy` + `down-if-alone` -must stay consistent with the typed option; `stable-after` lives only in HOCON because the typed option can't -express it). +**The default downing strategy is `auto-down` (changed 2026-07-21).** It is selected by +`Cluster:SplitBrainResolverStrategy`, and `ServiceCollectionExtensions.BuildDowningHocon` turns that +into the `akka.cluster.downing-provider-class` actually installed: -`keep-oldest` is the correct strategy for a 2-node warm-redundancy pair (`keep-majority`/`static-quorum` -are wrong for two nodes — no majority in a 1-1 split), but its two loss cases recover **differently**, and -this is inherent to any 2-node cluster: - -| Node lost | What keep-oldest does | Recovery | +| `Cluster:SplitBrainResolverStrategy` | Provider installed | Posture | |---|---|---| -| **Younger** node (crash or partition) | The oldest is on the surviving side → it stays up and downs the younger side (`DownUnreachable`). | **In-place, fast.** The oldest keeps its singletons + `driver` role-leadership; `RedundancyStateActor` re-computes from the post-loss `Cluster.State`. | -| **Oldest** node (crash or partition) | The lone survivor is "the side **without** the oldest" → keep-oldest downs the **survivor itself** (`DownReachable "including myself"`), and `run-coordinated-shutdown-when-down = on` terminates it. `down-if-alone` does **not** rescue a lone survivor (its rescue branch needs ≥ 2 surviving members, so it is a 3+-node feature). | **Exit-and-rejoin.** The self-downed survivor **and** the crashed oldest are both restarted by the service supervisor and re-form / rejoin. There is a brief restart-window outage; there is **no** in-place takeover. | +| `auto-down` (**default**) | `Akka.Cluster.AutoDowning` with `auto-down-unreachable-after = 15s` | **Availability.** The leader among the *reachable* members downs the unreachable peer, so a crash of **either** node — oldest included — fails over in place. | +| `keep-oldest` | `Akka.Cluster.SBR.SplitBrainResolverProvider` reading the `split-brain-resolver` block in `akka.conf` | **Partition-safety.** A partition can never run dual-active. **Not safe for a 2-node pair** — see below. | -**Recovery therefore depends on three things being in place** (the [ScadaBridge](../../ScadaBridge/CLAUDE.md) -sister project runs the same pattern): +Any other value **fails the host at startup** rather than falling through to a default. + +### Why keep-oldest is no longer the default + +The previous documentation in this section — and the code comments behind it — asserted that +`keep-oldest` with `down-if-alone = on` was "the correct strategy for a 2-node warm-redundancy pair." +**That was wrong, and the error was not conservative: it described a total-outage configuration as the +recommended one.** + +In Akka.NET 1.5.62's `KeepOldest.OldestDecision`, the `down-if-alone` rescue branch requires the +*surviving* side to hold **≥ 2 members**. In a two-node cluster a lost peer is always a 1-vs-1 split, so +the branch never fires and the lone survivor falls through to `DownReachable` — downing **itself** — and +`run-coordinated-shutdown-when-down = on` then terminates it. `down-if-alone` is a 3+-node feature; it +does not do what its name suggests here. Live-proven on the sister project's rig, whose survivor logged +`SBR took decision …DownReachable and is downing [self] including myself, [1] unreachable of [2] members`. +See [`../../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md`](../../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md) +for the upstream decision record, which OtOpcUa follows. + +The alternatives do not help a pair: `static-quorum` with quorum 1 hits Akka's `IsTooManyMembers` guard +and returns `DownAll`; with quorum 2 the survivor downs itself; `keep-majority` keeps the lowest-address +side, which just moves the fatal crash from "the oldest node" to "the other node"; `lease-majority` needs +a shared lease store both nodes can reach. + +### The accepted trade + +Under `auto-down`, a **genuine network partition** (both nodes alive, link cut) leaves each side downing +the other and continuing alone: **both run active**, both hold singletons, and both advertise the primary +`ServiceLevel`. Recovery is operator-driven — after the link heals, restart **one** side; it rejoins as a +fresh incarnation and settles as secondary. The two sides do not merge on their own, because the mutual +downing quarantines the association. + +This is a deliberate choice of availability over partition-safety, matching ScadaBridge: the pairs run one +node per VM with no external arbiter available, so a crash is a far more likely event than a partition, and +a crash under `keep-oldest` was unrecoverable without operator action anyway. + +Deployments that would rather take an outage than ever run dual-active can set +`Cluster:SplitBrainResolverStrategy: "keep-oldest"` — but should read the section above first, because for a +two-node pair that choice means *any* crash of the oldest node is a full outage. + +### Recovery still depends on supervision 1. **A service supervisor that restarts the process** on exit — production `Install-Services.ps1` sets `sc.exe failure OtOpcUaHost … actions= restart/5000/restart/30000/restart/60000`; the docker-dev rig sets - `restart: unless-stopped`. Without it a downed node stays down and an oldest-crash looks like a total outage. -2. **The recovery watchdog** `ActorSystemTerminationWatchdog` (registered in `Program.cs` after `AddAkka`) — it - watches `ActorSystem.WhenTerminated` and, on an unexpected self-down, calls - `IHostApplicationLifetime.StopApplication()` so the process exits (and the supervisor restarts it) instead of - idling forever with a dead actor system. -3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join the mesh via either peer. + `restart: unless-stopped`. +2. **The recovery watchdog** `ActorSystemTerminationWatchdog` (registered in `Program.cs` after `AddAkka`) — + it watches `ActorSystem.WhenTerminated` and, on an unexpected self-down, calls + `IHostApplicationLifetime.StopApplication()` so the process exits and the supervisor restarts it instead + of idling forever with a dead actor system. Under `auto-down` the *survivor* no longer needs this + (it is never the one downed); the **downed** node still does. +3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join via either peer. + Note the residual bootstrap constraint: only the **first** seed may form a cluster alone, so a node + cold-starting while its peer is dead still waits in `InitJoin`. Auto-down removes the crash outage, not + this one. > **On `HardKillFailoverTests`:** that in-process test hard-kills the oldest and asserts the survivor becomes -> sole `driver` role-leader, and it passes — but it simulates the crash with `provider.Transport.Shutdown()`, -> which leaves node A's `ActorSystem` **alive** (a transport partition with the oldest still running), not a -> real process death. It is therefore **not** representative of an oldest-process crash; a real 2-container -> `docker kill` of the oldest downs the survivor (verified 2026-07-15, see -> `archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`). Treat the recovery guarantee for -> the oldest as "exit-and-rejoin under supervision," not "in-place failover." +> sole `driver` role-leader. It passed even under `keep-oldest` — because it simulates the crash with +> `provider.Transport.Shutdown()`, which leaves node A's `ActorSystem` **alive** (a transport partition with +> the oldest still running), not a real process death. A real 2-container `docker kill` of the oldest downed +> the survivor (verified 2026-07-15, `archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`). +> **It is a standing example of a green test over a fatal defect**: keep the distinction between a transport +> partition and a process death in mind when reading it. -There is no operator-driven role swap during a partition. Failover / recovery is what the cluster + supervisor -do automatically. **Instant in-place takeover on *any* single-node loss requires 3+ members** (an odd cluster -or a lightweight witness) — a deliberate future option, not the current 2-node posture. +`SplitBrainResolverActivationTests` pins the strategy by starting a real host through the production +bootstrap and reading `akka.cluster.downing-provider-class` back off the running `ActorSystem` — asserting +the typed option or the akka.conf text is not sufficient, because several HOCON fragments compete and the +losing one still looks correct in the file. + +There is no operator-driven role swap during a partition; failover is what the cluster and supervisor do +automatically. + +> **Live gate outstanding.** The auto-down change is verified by unit tests against the effective +> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It has +> **not** yet been drilled on an OtOpcUa two-node rig — the docker-dev rig runs a single six-node mesh, where +> the 1-vs-1 pathology cannot occur. The drill (kill the oldest container of a genuine two-node cluster; +> assert the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) should run +> alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See +> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a. ## Primary data-plane gate (writes, acks, alerts emit) diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs index 9e46b0fa..c8c79436 100644 --- a/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs @@ -29,4 +29,30 @@ public sealed class AkkaClusterOptions /// admin, driver, dev. /// public string[] Roles { get; set; } = Array.Empty(); + + /// + /// How the cluster decides to down a node it can no longer reach. One of auto-down + /// (default) or keep-oldest; any other value fails the host at startup. + /// + /// + /// + /// auto-down — availability. The leader among the reachable members + /// downs the unreachable peer after + /// . A hard crash of either + /// node — including the oldest — fails over to the survivor with no operator action. + /// The trade is that a genuine network partition (both nodes alive, link cut) leaves + /// both sides running active until an operator restarts one. + /// + /// + /// keep-oldest — partition-safety. The SBR resolver sacrifices the younger + /// side of a split, so a partition can never run dual-active. The cost is severe for a + /// two-node pair: it cannot survive a crash of the oldest node at all. Akka.NET's + /// KeepOldest.OldestDecision only lets down-if-alone rescue a side holding + /// >= 2 members, so the 1-vs-1 survivor downs itself and shuts down — the + /// redundancy pair turns a single-node crash into a total outage. Choose this only for + /// clusters of three or more nodes, or where dual-active is genuinely worse than an + /// outage. + /// + /// + public string SplitBrainResolverStrategy { get; set; } = "auto-down"; } diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf b/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf index 806ac578..154de1e8 100644 --- a/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf @@ -37,14 +37,26 @@ akka { roles = [] min-nr-of-members = 1 - # Split-brain resolver (arch-review 03/S1). This block IS active: Akka.Cluster.Hosting's - # WithClustering enables an SBR downing provider by default (it applies - # SplitBrainResolverOption.Default when ClusterOptions.SplitBrainResolver is null), and that - # provider reads this block. ServiceCollectionExtensions.BuildClusterOptions additionally sets - # the typed KeepOldestOption { DownIfAlone = true } to make the strategy EXPLICIT in code — - # its active-strategy + keep-oldest.down-if-alone MUST match this block. (The cluster is NOT - # NoDowning; only an explicit downing-provider-class = "" would disable failover.) - # stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin. + # DOWNING. The provider is NOT selected here — this file is added with HoconAddMode.Append, + # which loses to what Akka.Hosting's WithClustering emits, so a downing-provider-class set + # here would be silently overridden. Selection lives in + # ServiceCollectionExtensions.BuildDowningHocon (Prepended, which outranks WithClustering), + # driven by Cluster:SplitBrainResolverStrategy. + # + # The DEFAULT strategy is "auto-down", under which this whole block is inert: Akka's + # AutoDowning provider is installed with auto-down-unreachable-after = 15s, and the leader + # among the reachable members downs the unreachable peer — so a hard crash of EITHER node, + # oldest included, fails over. The trade is dual-active during a real network partition. + # + # The block below applies only when an operator selects "keep-oldest". Note that keep-oldest + # is NOT safe for a two-node pair: Akka.NET's KeepOldest.OldestDecision only lets + # down-if-alone rescue a side holding >= 2 members, so a 1-vs-1 survivor downs ITSELF and + # (with run-coordinated-shutdown-when-down) exits — a crash of the oldest node becomes a + # total outage. See docs/Redundancy.md. + # + # stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin, and + # must equal ServiceCollectionExtensions.DowningStableAfter so both strategies fail over at + # the same latency. Both constraints are pinned by SplitBrainResolverActivationTests. split-brain-resolver { active-strategy = "keep-oldest" stable-after = 15s diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs index 37d108c7..e8dfdcd4 100644 --- a/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs @@ -80,40 +80,125 @@ public static class ServiceCollectionExtensions builder.WithClustering(BuildClusterOptions(options)); + // Must come AFTER WithClustering, which always emits a downing-provider-class of its own: + // Akka.Cluster.Hosting applies SplitBrainResolverOption.Default whenever the typed + // ClusterOptions.SplitBrainResolver is null, so there is always a competing value to beat. + // Prepend is the highest-precedence mode, which makes this block win irrespective of how + // many fragments are added or in what order. + // + // Do not take this comment's word for it. Which fragment actually wins is not obvious from + // the mode names — measured, Append also wins here, purely because it happens to be added + // last. SplitBrainResolverActivationTests reads the provider back off a *running* + // ActorSystem for exactly that reason: the effective value is the only one worth asserting. + builder.AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend); + return builder; } /// - /// Builds the for the fused-host cluster, setting the split-brain-resolver - /// strategy explicitly in code. - /// - /// Activation note (arch-review 03/S1, corrected): Akka.Cluster.Hosting's WithClustering - /// already enables an SBR downing provider by default — when - /// is null it applies SplitBrainResolverOption.Default, which registers - /// Akka.Cluster.SBR.SplitBrainResolverProvider and reads the split-brain-resolver HOCON block in - /// Resources/akka.conf. So the cluster was not running NoDowning before this option was - /// set: the pre-existing akka.conf keep-oldest block was already active, and hard-crashed nodes already - /// failed over. Setting this typed option makes the strategy explicit in code (independent of the - /// framework default) rather than being the sole activator — it is reinforcing/belt-and-suspenders and - /// produces the same effective behavior. (Only an explicit NoDowning, e.g. - /// downing-provider-class = "", would disable failover.) - /// - /// with DownIfAlone=true mirrors the HOCON intent and is the - /// correct strategy for a 2-node warm-redundancy pair: on an even split the oldest member (typically - /// the long-running primary) survives, and down-if-alone downs a singleton that loses its - /// peer. keep-majority/static-quorum are wrong for two nodes. The strategy + down-if-alone - /// here MUST stay consistent with the HOCON block; stable-after lives only in HOCON because the - /// typed option cannot express it (it must stay ≥ failure-detector.acceptable-heartbeat-pause). + /// How long a member must stay unreachable before it is downed. Shared by both strategies + /// (auto-down-unreachable-after and the SBR resolver's stable-after) so the two + /// have the same failover latency. Must stay above + /// akka.cluster.failure-detector.acceptable-heartbeat-pause or a merely-slow node gets + /// downed; both constraints are pinned by tests. /// - /// The bound cluster options carrying seed nodes and roles. - /// The cluster options with seed nodes, roles, and the explicit split-brain-resolver strategy. + public static readonly TimeSpan DowningStableAfter = TimeSpan.FromSeconds(15); + + /// + /// Builds the HOCON that selects the downing provider, per + /// . + /// + /// The bound cluster options carrying the strategy. + /// + /// For auto-down, a block installing Akka's AutoDowning provider with a downing + /// window of . For keep-oldest, an empty string — the + /// typed from already installs + /// the SBR provider, and the resolver's own settings live in Resources/akka.conf. + /// + /// The strategy is not a recognised value. + public static string BuildDowningHocon(AkkaClusterOptions options) + { + ArgumentNullException.ThrowIfNull(options); + + if (IsAutoDown(options)) + { + // auto-down-unreachable-after is load-bearing: it defaults to `off`, under which + // AutoDowning is installed but never downs anything — failover silently disabled. + return $$""" + akka.cluster { + downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster" + auto-down-unreachable-after = {{(int)DowningStableAfter.TotalMilliseconds}}ms + } + """; + } + + if (IsKeepOldest(options)) + { + return string.Empty; + } + + throw new ArgumentOutOfRangeException( + nameof(options), + options.SplitBrainResolverStrategy, + $"Unknown {AkkaClusterOptions.SectionName}:{nameof(AkkaClusterOptions.SplitBrainResolverStrategy)}. " + + "Expected 'auto-down' (default; survives a crash of either node) or 'keep-oldest' " + + "(partition-safe, but a two-node pair cannot survive a crash of the oldest node)."); + } + + private static bool IsAutoDown(AkkaClusterOptions options) => + string.Equals(options.SplitBrainResolverStrategy, "auto-down", StringComparison.OrdinalIgnoreCase); + + private static bool IsKeepOldest(AkkaClusterOptions options) => + string.Equals(options.SplitBrainResolverStrategy, "keep-oldest", StringComparison.OrdinalIgnoreCase); + + /// + /// Builds the for the fused-host cluster. + /// + /// + /// + /// Activation note (arch-review 03/S1, corrected twice). Akka.Cluster.Hosting's + /// WithClustering always installs a downing provider: when + /// is null it applies + /// SplitBrainResolverOption.Default, which registers + /// Akka.Cluster.SBR.SplitBrainResolverProvider and reads the + /// split-brain-resolver block in Resources/akka.conf. The cluster is + /// therefore never NoDowning. What this method chooses is only which + /// provider — and under the default auto-down strategy the choice is not made + /// here at all but by the Prepended HOCON in + /// , which outranks whatever + /// WithClustering emits. + /// + /// + /// Why keep-oldest is no longer the default (2026-07-21). The previous + /// documentation here asserted that with + /// DownIfAlone=true was "the correct strategy for a 2-node warm-redundancy pair" + /// because down-if-alone would down a node that lost its peer. That is the + /// opposite of what Akka.NET does. In KeepOldest.OldestDecision the + /// down-if-alone branch requires the surviving side to hold >= 2 members, so + /// in a 1-vs-1 split the survivor falls through to DownReachable and downs + /// itself; with run-coordinated-shutdown-when-down = on it then exits. A + /// two-node pair running keep-oldest converts a crash of the oldest node into a total + /// outage — precisely the failure redundancy exists to absorb. Live-proven on the sister + /// project's rig; see docs/Redundancy.md and + /// for the trade. + /// + /// + /// The bound cluster options carrying seed nodes, roles and the strategy. + /// + /// The cluster options with seed nodes and roles. + /// is set only under keep-oldest, where the SBR provider is the one wanted; under + /// auto-down it is left null because the Prepended HOCON replaces the provider anyway, + /// and naming a resolver that is not in force would be misleading. + /// public static ClusterOptions BuildClusterOptions(AkkaClusterOptions options) { + ArgumentNullException.ThrowIfNull(options); + return new ClusterOptions { SeedNodes = options.SeedNodes, Roles = options.Roles, - SplitBrainResolver = new KeepOldestOption { DownIfAlone = true }, + SplitBrainResolver = IsKeepOldest(options) ? new KeepOldestOption { DownIfAlone = true } : null, }; } } diff --git a/tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitBrainResolverActivationTests.cs b/tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitBrainResolverActivationTests.cs index 64c7e040..1de557e4 100644 --- a/tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitBrainResolverActivationTests.cs +++ b/tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitBrainResolverActivationTests.cs @@ -1,47 +1,215 @@ +using Akka.Actor; using Akka.Cluster.Hosting; using Akka.Cluster.Hosting.SBR; +using Akka.Configuration; +using Akka.Hosting; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Options; using Shouldly; using Xunit; namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests; /// -/// Guards that the split-brain-resolver strategy stays set explicitly in code (arch-review 03/S1, -/// premise corrected — see #11). Akka.Cluster.Hosting already enables an SBR downing provider by default -/// (it applies SplitBrainResolverOption.Default when -/// is null, which reads the akka.conf keep-oldest block), so the cluster is not NoDowning even -/// without the typed option. These tests assert -/// sets the typed -/// KeepOldestOption { DownIfAlone = true } so the strategy is explicit in code rather than relying on -/// the framework default — a future refactor cannot silently drop that explicitness. (Failover behaviour is -/// separately verified by HardKillFailoverTests, which passes regardless of activation path.) +/// Guards the downing strategy — the mechanism that decides whether a node that loses its peer +/// keeps running or shuts itself down. /// +/// +/// +/// Why these tests assert the EFFECTIVE config, not the typed option. The previous +/// version of this file asserted only that +/// returned a +/// . That is a different question from "what does the running +/// ActorSystem actually do". Akka.Hosting merges several HOCON fragments — the akka.conf +/// resource, whatever WithClustering emits, and anything added explicitly — and which +/// one wins is not readable off the names alone. A downing +/// setting can therefore be right in the resource file, right in the typed options object, +/// and still not be the one in force. These tests start a real host through the production +/// bootstrap and read akka.cluster.downing-provider-class back off +/// , which is the only assertion precedence cannot fool. +/// +/// +/// Why the default is auto-down. A two-node cluster running the SBR +/// keep-oldest resolver cannot survive a crash of the oldest node. Akka.NET 1.5.62's +/// KeepOldest.OldestDecision only lets down-if-alone rescue a side holding +/// >= 2 members, so in a 1-vs-1 split the lone survivor takes DownReachable and +/// downs itself — then run-coordinated-shutdown-when-down terminates it. A +/// crash of one node becomes a total outage: exactly the failure the redundancy pair exists +/// to absorb. Proven live on the sister project's rig (see +/// ../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md) and +/// adopted here for the same reason. The accepted trade is dual-active during a genuine +/// network partition; see docs/Redundancy.md. +/// +/// public sealed class SplitBrainResolverActivationTests { - private static AkkaClusterOptions SampleOptions() => new() + private static AkkaClusterOptions SampleOptions(string? strategy = null) { - SeedNodes = new[] { "akka.tcp://otopcua@127.0.0.1:4053" }, - Roles = new[] { "admin", "driver" }, - }; + var options = new AkkaClusterOptions + { + SeedNodes = new[] { "akka.tcp://otopcua@127.0.0.1:4053" }, + Roles = new[] { "admin", "driver" }, + }; + if (strategy is not null) + { + options.SplitBrainResolverStrategy = strategy; + } - /// Verifies the cluster options carry an explicit split-brain resolver strategy. - [Fact] - public void BuildClusterOptions_sets_an_explicit_split_brain_resolver() - { - var options = ServiceCollectionExtensions.BuildClusterOptions(SampleOptions()); - - options.SplitBrainResolver.ShouldNotBeNull( - "SplitBrainResolver must be set explicitly in code — keeps the strategy independent of Akka.Cluster.Hosting's default provider rather than relying on it."); + return options; } - /// Verifies the resolver is keep-oldest with down-if-alone (correct for a 2-node pair). - [Fact] - public void BuildClusterOptions_uses_keep_oldest_with_down_if_alone() + /// + /// Starts a real host through + /// — the production entry point, not a re-creation of it — and returns the ActorSystem's + /// effective configuration. + /// + /// + /// Calling the production bootstrap rather than inlining the calls it makes is deliberate: an + /// earlier draft of this helper rebuilt the same three steps itself, which pinned the + /// test's wiring instead of the shipped one and would have stayed green through a change + /// to production's HOCON precedence. + /// + private static async Task EffectiveConfigAsync(AkkaClusterOptions options) { - var options = ServiceCollectionExtensions.BuildClusterOptions(SampleOptions()); + // Port 0 so concurrently-running test hosts cannot collide on 4053, and no seed nodes so the + // system never tries to join a cluster. + options.Port = 0; + options.Hostname = "127.0.0.1"; + options.PublicHostname = "127.0.0.1"; + options.SeedNodes = Array.Empty(); - var keepOldest = options.SplitBrainResolver.ShouldBeOfType(); - keepOldest.DownIfAlone.ShouldBe(true); + var builder = Host.CreateDefaultBuilder(); + builder.ConfigureServices(services => + { + services.AddSingleton>(Options.Create(options)); + services.AddAkka("otopcua-sbr-test", (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp)); + }); + + using var host = builder.Build(); + await host.StartAsync(); + try + { + return host.Services.GetRequiredService().Settings.Config; + } + finally + { + await host.StopAsync(); + } + } + + /// + /// The default strategy installs Akka's AutoDowning provider — the leader among the + /// reachable members downs the unreachable peer, so a crash of either node fails over. + /// + [Fact] + public async Task Default_strategy_installs_the_auto_downing_provider() + { + var config = await EffectiveConfigAsync(SampleOptions()); + + config.GetString("akka.cluster.downing-provider-class") + .ShouldStartWith( + "Akka.Cluster.AutoDowning, Akka.Cluster", + Case.Sensitive, + "the effective downing provider — not merely the configured one — must be AutoDowning, or a crash of the oldest node is a total outage"); + } + + /// + /// AutoDowning is inert without a downing window: auto-down-unreachable-after defaults to + /// off, under which the provider is installed but never downs anything. Selecting the + /// provider and forgetting the window would look correct and fail over never. + /// + [Fact] + public async Task Default_strategy_sets_the_auto_down_window() + { + var config = await EffectiveConfigAsync(SampleOptions()); + + config.GetTimeSpan("akka.cluster.auto-down-unreachable-after") + .ShouldBe(ServiceCollectionExtensions.DowningStableAfter); + } + + /// + /// The opt-out is real: an operator who chooses partition-safety over availability gets the SBR + /// keep-oldest resolver back, in force rather than merely configured. + /// + [Fact] + public async Task Keep_oldest_strategy_installs_the_sbr_provider() + { + var config = await EffectiveConfigAsync(SampleOptions("keep-oldest")); + + config.GetString("akka.cluster.downing-provider-class") + .ShouldStartWith("Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"); + config.GetString("akka.cluster.split-brain-resolver.active-strategy").ShouldBe("keep-oldest"); + config.GetBoolean("akka.cluster.split-brain-resolver.keep-oldest.down-if-alone").ShouldBeTrue(); + } + + /// The default is auto-down, so an unconfigured deployment gets the survivable one. + [Fact] + public void Strategy_defaults_to_auto_down() + { + new AkkaClusterOptions().SplitBrainResolverStrategy.ShouldBe("auto-down"); + } + + /// + /// A typo in the strategy must fail the host at startup, not silently fall through to whichever + /// provider Akka.Hosting would have installed by default — which is the fatal one. + /// + [Fact] + public void Unknown_strategy_throws_rather_than_silently_falling_back() + { + var ex = Should.Throw( + () => ServiceCollectionExtensions.BuildDowningHocon(SampleOptions("keep-majority"))); + + ex.Message.ShouldContain("keep-majority"); + ex.Message.ShouldContain("auto-down"); + } + + /// Strategy comparison is case-insensitive — config casing is not a failure mode. + [Fact] + public void Strategy_matching_is_case_insensitive() + { + Should.NotThrow(() => ServiceCollectionExtensions.BuildDowningHocon(SampleOptions("Auto-Down"))); + Should.NotThrow(() => ServiceCollectionExtensions.BuildDowningHocon(SampleOptions("Keep-Oldest"))); + } + + /// + /// Under keep-oldest the SBR resolver is still set as a typed option, so the strategy stays + /// explicit in code rather than depending on Akka.Cluster.Hosting's default. + /// + [Fact] + public void Keep_oldest_sets_the_typed_resolver_option() + { + var options = ServiceCollectionExtensions.BuildClusterOptions(SampleOptions("keep-oldest")); + + options.SplitBrainResolver.ShouldBeOfType().DownIfAlone.ShouldBe(true); + } + + /// + /// Pins the two stability windows together. auto-down-unreachable-after comes from + /// while the SBR resolver reads + /// stable-after out of akka.conf; changing one and missing the other would leave the two + /// strategies failing over at different latencies for no stated reason. + /// + [Fact] + public void Downing_window_matches_the_akka_conf_stable_after() + { + var akkaConf = ConfigurationFactory.ParseString(HoconLoader.LoadBaseConfig()); + + akkaConf.GetTimeSpan("akka.cluster.split-brain-resolver.stable-after") + .ShouldBe(ServiceCollectionExtensions.DowningStableAfter); + } + + /// + /// The stability window must stay above the failure detector's tolerance, or a node gets downed + /// while its peer is merely slow. + /// + [Fact] + public void Downing_window_exceeds_the_acceptable_heartbeat_pause() + { + var akkaConf = ConfigurationFactory.ParseString(HoconLoader.LoadBaseConfig()); + + ServiceCollectionExtensions.DowningStableAfter + .ShouldBeGreaterThan(akkaConf.GetTimeSpan("akka.cluster.failure-detector.acceptable-heartbeat-pause")); } /// Verifies seed nodes and roles are still threaded through unchanged.