diff --git a/docs/Redundancy.md b/docs/Redundancy.md
index a0b5f3ad..6e30ef91 100644
--- a/docs/Redundancy.md
+++ b/docs/Redundancy.md
@@ -169,55 +169,94 @@ Each node advertises its partner via `OpcUaApplicationHostOptions.PeerApplicatio
Node A lists Node B's `ApplicationUri` and vice-versa. Validated by `DualEndpointTests` in `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/` — boots two `OpcUaApplicationHost` instances on loopback, asserts a real OPCFoundation client `Session` reading `Server.ServerArray` from Node A sees both URIs.
-## Split-brain
+## Split-brain / downing
-The split-brain resolver is **active by default**: Akka.Cluster.Hosting's `WithClustering` enables an SBR
-downing provider whenever `ClusterOptions.SplitBrainResolver` is null (it applies
-`SplitBrainResolverOption.Default`, which registers `Akka.Cluster.SBR.SplitBrainResolverProvider`), and that
-provider reads the `split-brain-resolver` HOCON block in `akka.conf`. On top of that,
-`ServiceCollectionExtensions.BuildClusterOptions` sets the typed `ClusterOptions.SplitBrainResolver`
-(`KeepOldestOption { DownIfAlone = true }`) to make the strategy **explicit in code** rather than relying on
-the framework default — it is reinforcing, not the sole activator, and yields the same effective behavior. So
-the cluster is **not** running `NoDowning`; hard-crashed nodes are downed and the cluster recovers (how it
-recovers depends on *which* node was lost — see the table below). (Only an *explicit* `NoDowning`, e.g.
-`akka.cluster.downing-provider-class = ""`, would leave both redundancy sides at ServiceLevel 240
-indefinitely.) The HOCON block carries the tuning: `active-strategy = keep-oldest`, `stable-after = 15s`,
-`keep-oldest.down-if-alone = on`, `failure-detector.threshold = 10.0` (its `active-strategy` + `down-if-alone`
-must stay consistent with the typed option; `stable-after` lives only in HOCON because the typed option can't
-express it).
+**The default downing strategy is `auto-down` (changed 2026-07-21).** It is selected by
+`Cluster:SplitBrainResolverStrategy`, and `ServiceCollectionExtensions.BuildDowningHocon` turns that
+into the `akka.cluster.downing-provider-class` actually installed:
-`keep-oldest` is the correct strategy for a 2-node warm-redundancy pair (`keep-majority`/`static-quorum`
-are wrong for two nodes — no majority in a 1-1 split), but its two loss cases recover **differently**, and
-this is inherent to any 2-node cluster:
-
-| Node lost | What keep-oldest does | Recovery |
+| `Cluster:SplitBrainResolverStrategy` | Provider installed | Posture |
|---|---|---|
-| **Younger** node (crash or partition) | The oldest is on the surviving side → it stays up and downs the younger side (`DownUnreachable`). | **In-place, fast.** The oldest keeps its singletons + `driver` role-leadership; `RedundancyStateActor` re-computes from the post-loss `Cluster.State`. |
-| **Oldest** node (crash or partition) | The lone survivor is "the side **without** the oldest" → keep-oldest downs the **survivor itself** (`DownReachable "including myself"`), and `run-coordinated-shutdown-when-down = on` terminates it. `down-if-alone` does **not** rescue a lone survivor (its rescue branch needs ≥ 2 surviving members, so it is a 3+-node feature). | **Exit-and-rejoin.** The self-downed survivor **and** the crashed oldest are both restarted by the service supervisor and re-form / rejoin. There is a brief restart-window outage; there is **no** in-place takeover. |
+| `auto-down` (**default**) | `Akka.Cluster.AutoDowning` with `auto-down-unreachable-after = 15s` | **Availability.** The leader among the *reachable* members downs the unreachable peer, so a crash of **either** node — oldest included — fails over in place. |
+| `keep-oldest` | `Akka.Cluster.SBR.SplitBrainResolverProvider` reading the `split-brain-resolver` block in `akka.conf` | **Partition-safety.** A partition can never run dual-active. **Not safe for a 2-node pair** — see below. |
-**Recovery therefore depends on three things being in place** (the [ScadaBridge](../../ScadaBridge/CLAUDE.md)
-sister project runs the same pattern):
+Any other value **fails the host at startup** rather than falling through to a default.
+
+### Why keep-oldest is no longer the default
+
+The previous documentation in this section — and the code comments behind it — asserted that
+`keep-oldest` with `down-if-alone = on` was "the correct strategy for a 2-node warm-redundancy pair."
+**That was wrong, and the error was not conservative: it described a total-outage configuration as the
+recommended one.**
+
+In Akka.NET 1.5.62's `KeepOldest.OldestDecision`, the `down-if-alone` rescue branch requires the
+*surviving* side to hold **≥ 2 members**. In a two-node cluster a lost peer is always a 1-vs-1 split, so
+the branch never fires and the lone survivor falls through to `DownReachable` — downing **itself** — and
+`run-coordinated-shutdown-when-down = on` then terminates it. `down-if-alone` is a 3+-node feature; it
+does not do what its name suggests here. Live-proven on the sister project's rig, whose survivor logged
+`SBR took decision …DownReachable and is downing [self] including myself, [1] unreachable of [2] members`.
+See [`../../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md`](../../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md)
+for the upstream decision record, which OtOpcUa follows.
+
+The alternatives do not help a pair: `static-quorum` with quorum 1 hits Akka's `IsTooManyMembers` guard
+and returns `DownAll`; with quorum 2 the survivor downs itself; `keep-majority` keeps the lowest-address
+side, which just moves the fatal crash from "the oldest node" to "the other node"; `lease-majority` needs
+a shared lease store both nodes can reach.
+
+### The accepted trade
+
+Under `auto-down`, a **genuine network partition** (both nodes alive, link cut) leaves each side downing
+the other and continuing alone: **both run active**, both hold singletons, and both advertise the primary
+`ServiceLevel`. Recovery is operator-driven — after the link heals, restart **one** side; it rejoins as a
+fresh incarnation and settles as secondary. The two sides do not merge on their own, because the mutual
+downing quarantines the association.
+
+This is a deliberate choice of availability over partition-safety, matching ScadaBridge: the pairs run one
+node per VM with no external arbiter available, so a crash is a far more likely event than a partition, and
+a crash under `keep-oldest` was unrecoverable without operator action anyway.
+
+Deployments that would rather take an outage than ever run dual-active can set
+`Cluster:SplitBrainResolverStrategy: "keep-oldest"` — but should read the section above first, because for a
+two-node pair that choice means *any* crash of the oldest node is a full outage.
+
+### Recovery still depends on supervision
1. **A service supervisor that restarts the process** on exit — production `Install-Services.ps1` sets
`sc.exe failure OtOpcUaHost … actions= restart/5000/restart/30000/restart/60000`; the docker-dev rig sets
- `restart: unless-stopped`. Without it a downed node stays down and an oldest-crash looks like a total outage.
-2. **The recovery watchdog** `ActorSystemTerminationWatchdog` (registered in `Program.cs` after `AddAkka`) — it
- watches `ActorSystem.WhenTerminated` and, on an unexpected self-down, calls
- `IHostApplicationLifetime.StopApplication()` so the process exits (and the supervisor restarts it) instead of
- idling forever with a dead actor system.
-3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join the mesh via either peer.
+ `restart: unless-stopped`.
+2. **The recovery watchdog** `ActorSystemTerminationWatchdog` (registered in `Program.cs` after `AddAkka`) —
+ it watches `ActorSystem.WhenTerminated` and, on an unexpected self-down, calls
+ `IHostApplicationLifetime.StopApplication()` so the process exits and the supervisor restarts it instead
+ of idling forever with a dead actor system. Under `auto-down` the *survivor* no longer needs this
+ (it is never the one downed); the **downed** node still does.
+3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join via either peer.
+ Note the residual bootstrap constraint: only the **first** seed may form a cluster alone, so a node
+ cold-starting while its peer is dead still waits in `InitJoin`. Auto-down removes the crash outage, not
+ this one.
> **On `HardKillFailoverTests`:** that in-process test hard-kills the oldest and asserts the survivor becomes
-> sole `driver` role-leader, and it passes — but it simulates the crash with `provider.Transport.Shutdown()`,
-> which leaves node A's `ActorSystem` **alive** (a transport partition with the oldest still running), not a
-> real process death. It is therefore **not** representative of an oldest-process crash; a real 2-container
-> `docker kill` of the oldest downs the survivor (verified 2026-07-15, see
-> `archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`). Treat the recovery guarantee for
-> the oldest as "exit-and-rejoin under supervision," not "in-place failover."
+> sole `driver` role-leader. It passed even under `keep-oldest` — because it simulates the crash with
+> `provider.Transport.Shutdown()`, which leaves node A's `ActorSystem` **alive** (a transport partition with
+> the oldest still running), not a real process death. A real 2-container `docker kill` of the oldest downed
+> the survivor (verified 2026-07-15, `archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`).
+> **It is a standing example of a green test over a fatal defect**: keep the distinction between a transport
+> partition and a process death in mind when reading it.
-There is no operator-driven role swap during a partition. Failover / recovery is what the cluster + supervisor
-do automatically. **Instant in-place takeover on *any* single-node loss requires 3+ members** (an odd cluster
-or a lightweight witness) — a deliberate future option, not the current 2-node posture.
+`SplitBrainResolverActivationTests` pins the strategy by starting a real host through the production
+bootstrap and reading `akka.cluster.downing-provider-class` back off the running `ActorSystem` — asserting
+the typed option or the akka.conf text is not sufficient, because several HOCON fragments compete and the
+losing one still looks correct in the file.
+
+There is no operator-driven role swap during a partition; failover is what the cluster and supervisor do
+automatically.
+
+> **Live gate outstanding.** The auto-down change is verified by unit tests against the effective
+> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It has
+> **not** yet been drilled on an OtOpcUa two-node rig — the docker-dev rig runs a single six-node mesh, where
+> the 1-vs-1 pathology cannot occur. The drill (kill the oldest container of a genuine two-node cluster;
+> assert the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) should run
+> alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See
+> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
## Primary data-plane gate (writes, acks, alerts emit)
diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs
index 9e46b0fa..c8c79436 100644
--- a/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs
+++ b/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs
@@ -29,4 +29,30 @@ public sealed class AkkaClusterOptions
/// admin, driver, dev.
///
public string[] Roles { get; set; } = Array.Empty();
+
+ ///
+ /// How the cluster decides to down a node it can no longer reach. One of auto-down
+ /// (default) or keep-oldest; any other value fails the host at startup.
+ ///
+ ///
+ ///
+ /// auto-down — availability. The leader among the reachable members
+ /// downs the unreachable peer after
+ /// . A hard crash of either
+ /// node — including the oldest — fails over to the survivor with no operator action.
+ /// The trade is that a genuine network partition (both nodes alive, link cut) leaves
+ /// both sides running active until an operator restarts one.
+ ///
+ ///
+ /// keep-oldest — partition-safety. The SBR resolver sacrifices the younger
+ /// side of a split, so a partition can never run dual-active. The cost is severe for a
+ /// two-node pair: it cannot survive a crash of the oldest node at all. Akka.NET's
+ /// KeepOldest.OldestDecision only lets down-if-alone rescue a side holding
+ /// >= 2 members, so the 1-vs-1 survivor downs itself and shuts down — the
+ /// redundancy pair turns a single-node crash into a total outage. Choose this only for
+ /// clusters of three or more nodes, or where dual-active is genuinely worse than an
+ /// outage.
+ ///
+ ///
+ public string SplitBrainResolverStrategy { get; set; } = "auto-down";
}
diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf b/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf
index 806ac578..154de1e8 100644
--- a/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf
+++ b/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/Resources/akka.conf
@@ -37,14 +37,26 @@ akka {
roles = []
min-nr-of-members = 1
- # Split-brain resolver (arch-review 03/S1). This block IS active: Akka.Cluster.Hosting's
- # WithClustering enables an SBR downing provider by default (it applies
- # SplitBrainResolverOption.Default when ClusterOptions.SplitBrainResolver is null), and that
- # provider reads this block. ServiceCollectionExtensions.BuildClusterOptions additionally sets
- # the typed KeepOldestOption { DownIfAlone = true } to make the strategy EXPLICIT in code —
- # its active-strategy + keep-oldest.down-if-alone MUST match this block. (The cluster is NOT
- # NoDowning; only an explicit downing-provider-class = "" would disable failover.)
- # stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin.
+ # DOWNING. The provider is NOT selected here — this file is added with HoconAddMode.Append,
+ # which loses to what Akka.Hosting's WithClustering emits, so a downing-provider-class set
+ # here would be silently overridden. Selection lives in
+ # ServiceCollectionExtensions.BuildDowningHocon (Prepended, which outranks WithClustering),
+ # driven by Cluster:SplitBrainResolverStrategy.
+ #
+ # The DEFAULT strategy is "auto-down", under which this whole block is inert: Akka's
+ # AutoDowning provider is installed with auto-down-unreachable-after = 15s, and the leader
+ # among the reachable members downs the unreachable peer — so a hard crash of EITHER node,
+ # oldest included, fails over. The trade is dual-active during a real network partition.
+ #
+ # The block below applies only when an operator selects "keep-oldest". Note that keep-oldest
+ # is NOT safe for a two-node pair: Akka.NET's KeepOldest.OldestDecision only lets
+ # down-if-alone rescue a side holding >= 2 members, so a 1-vs-1 survivor downs ITSELF and
+ # (with run-coordinated-shutdown-when-down) exits — a crash of the oldest node becomes a
+ # total outage. See docs/Redundancy.md.
+ #
+ # stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin, and
+ # must equal ServiceCollectionExtensions.DowningStableAfter so both strategies fail over at
+ # the same latency. Both constraints are pinned by SplitBrainResolverActivationTests.
split-brain-resolver {
active-strategy = "keep-oldest"
stable-after = 15s
diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs
index 37d108c7..e8dfdcd4 100644
--- a/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs
+++ b/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs
@@ -80,40 +80,125 @@ public static class ServiceCollectionExtensions
builder.WithClustering(BuildClusterOptions(options));
+ // Must come AFTER WithClustering, which always emits a downing-provider-class of its own:
+ // Akka.Cluster.Hosting applies SplitBrainResolverOption.Default whenever the typed
+ // ClusterOptions.SplitBrainResolver is null, so there is always a competing value to beat.
+ // Prepend is the highest-precedence mode, which makes this block win irrespective of how
+ // many fragments are added or in what order.
+ //
+ // Do not take this comment's word for it. Which fragment actually wins is not obvious from
+ // the mode names — measured, Append also wins here, purely because it happens to be added
+ // last. SplitBrainResolverActivationTests reads the provider back off a *running*
+ // ActorSystem for exactly that reason: the effective value is the only one worth asserting.
+ builder.AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend);
+
return builder;
}
///
- /// Builds the for the fused-host cluster, setting the split-brain-resolver
- /// strategy explicitly in code.
- ///
- /// Activation note (arch-review 03/S1, corrected): Akka.Cluster.Hosting's WithClustering
- /// already enables an SBR downing provider by default — when
- /// is null it applies SplitBrainResolverOption.Default, which registers
- /// Akka.Cluster.SBR.SplitBrainResolverProvider and reads the split-brain-resolver HOCON block in
- /// Resources/akka.conf. So the cluster was not running NoDowning before this option was
- /// set: the pre-existing akka.conf keep-oldest block was already active, and hard-crashed nodes already
- /// failed over. Setting this typed option makes the strategy explicit in code (independent of the
- /// framework default) rather than being the sole activator — it is reinforcing/belt-and-suspenders and
- /// produces the same effective behavior. (Only an explicit NoDowning, e.g.
- /// downing-provider-class = "", would disable failover.)
- ///
- /// with DownIfAlone=true mirrors the HOCON intent and is the
- /// correct strategy for a 2-node warm-redundancy pair: on an even split the oldest member (typically
- /// the long-running primary) survives, and down-if-alone downs a singleton that loses its
- /// peer. keep-majority/static-quorum are wrong for two nodes. The strategy + down-if-alone
- /// here MUST stay consistent with the HOCON block; stable-after lives only in HOCON because the
- /// typed option cannot express it (it must stay ≥ failure-detector.acceptable-heartbeat-pause).
+ /// How long a member must stay unreachable before it is downed. Shared by both strategies
+ /// (auto-down-unreachable-after and the SBR resolver's stable-after) so the two
+ /// have the same failover latency. Must stay above
+ /// akka.cluster.failure-detector.acceptable-heartbeat-pause or a merely-slow node gets
+ /// downed; both constraints are pinned by tests.
///
- /// The bound cluster options carrying seed nodes and roles.
- /// The cluster options with seed nodes, roles, and the explicit split-brain-resolver strategy.
+ public static readonly TimeSpan DowningStableAfter = TimeSpan.FromSeconds(15);
+
+ ///
+ /// Builds the HOCON that selects the downing provider, per
+ /// .
+ ///
+ /// The bound cluster options carrying the strategy.
+ ///
+ /// For auto-down, a block installing Akka's AutoDowning provider with a downing
+ /// window of . For keep-oldest, an empty string — the
+ /// typed from already installs
+ /// the SBR provider, and the resolver's own settings live in Resources/akka.conf.
+ ///
+ /// The strategy is not a recognised value.
+ public static string BuildDowningHocon(AkkaClusterOptions options)
+ {
+ ArgumentNullException.ThrowIfNull(options);
+
+ if (IsAutoDown(options))
+ {
+ // auto-down-unreachable-after is load-bearing: it defaults to `off`, under which
+ // AutoDowning is installed but never downs anything — failover silently disabled.
+ return $$"""
+ akka.cluster {
+ downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"
+ auto-down-unreachable-after = {{(int)DowningStableAfter.TotalMilliseconds}}ms
+ }
+ """;
+ }
+
+ if (IsKeepOldest(options))
+ {
+ return string.Empty;
+ }
+
+ throw new ArgumentOutOfRangeException(
+ nameof(options),
+ options.SplitBrainResolverStrategy,
+ $"Unknown {AkkaClusterOptions.SectionName}:{nameof(AkkaClusterOptions.SplitBrainResolverStrategy)}. "
+ + "Expected 'auto-down' (default; survives a crash of either node) or 'keep-oldest' "
+ + "(partition-safe, but a two-node pair cannot survive a crash of the oldest node).");
+ }
+
+ private static bool IsAutoDown(AkkaClusterOptions options) =>
+ string.Equals(options.SplitBrainResolverStrategy, "auto-down", StringComparison.OrdinalIgnoreCase);
+
+ private static bool IsKeepOldest(AkkaClusterOptions options) =>
+ string.Equals(options.SplitBrainResolverStrategy, "keep-oldest", StringComparison.OrdinalIgnoreCase);
+
+ ///
+ /// Builds the for the fused-host cluster.
+ ///
+ ///
+ ///
+ /// Activation note (arch-review 03/S1, corrected twice). Akka.Cluster.Hosting's
+ /// WithClustering always installs a downing provider: when
+ /// is null it applies
+ /// SplitBrainResolverOption.Default, which registers
+ /// Akka.Cluster.SBR.SplitBrainResolverProvider and reads the
+ /// split-brain-resolver block in Resources/akka.conf. The cluster is
+ /// therefore never NoDowning. What this method chooses is only which
+ /// provider — and under the default auto-down strategy the choice is not made
+ /// here at all but by the Prepended HOCON in
+ /// , which outranks whatever
+ /// WithClustering emits.
+ ///
+ ///
+ /// Why keep-oldest is no longer the default (2026-07-21). The previous
+ /// documentation here asserted that with
+ /// DownIfAlone=true was "the correct strategy for a 2-node warm-redundancy pair"
+ /// because down-if-alone would down a node that lost its peer. That is the
+ /// opposite of what Akka.NET does. In KeepOldest.OldestDecision the
+ /// down-if-alone branch requires the surviving side to hold >= 2 members, so
+ /// in a 1-vs-1 split the survivor falls through to DownReachable and downs
+ /// itself; with run-coordinated-shutdown-when-down = on it then exits. A
+ /// two-node pair running keep-oldest converts a crash of the oldest node into a total
+ /// outage — precisely the failure redundancy exists to absorb. Live-proven on the sister
+ /// project's rig; see docs/Redundancy.md and
+ /// for the trade.
+ ///
+ ///
+ /// The bound cluster options carrying seed nodes, roles and the strategy.
+ ///
+ /// The cluster options with seed nodes and roles.
+ /// is set only under keep-oldest, where the SBR provider is the one wanted; under
+ /// auto-down it is left null because the Prepended HOCON replaces the provider anyway,
+ /// and naming a resolver that is not in force would be misleading.
+ ///
public static ClusterOptions BuildClusterOptions(AkkaClusterOptions options)
{
+ ArgumentNullException.ThrowIfNull(options);
+
return new ClusterOptions
{
SeedNodes = options.SeedNodes,
Roles = options.Roles,
- SplitBrainResolver = new KeepOldestOption { DownIfAlone = true },
+ SplitBrainResolver = IsKeepOldest(options) ? new KeepOldestOption { DownIfAlone = true } : null,
};
}
}
diff --git a/tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitBrainResolverActivationTests.cs b/tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitBrainResolverActivationTests.cs
index 64c7e040..1de557e4 100644
--- a/tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitBrainResolverActivationTests.cs
+++ b/tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/SplitBrainResolverActivationTests.cs
@@ -1,47 +1,215 @@
+using Akka.Actor;
using Akka.Cluster.Hosting;
using Akka.Cluster.Hosting.SBR;
+using Akka.Configuration;
+using Akka.Hosting;
+using Microsoft.Extensions.DependencyInjection;
+using Microsoft.Extensions.Hosting;
+using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
///
-/// Guards that the split-brain-resolver strategy stays set explicitly in code (arch-review 03/S1,
-/// premise corrected — see #11). Akka.Cluster.Hosting already enables an SBR downing provider by default
-/// (it applies SplitBrainResolverOption.Default when
-/// is null, which reads the akka.conf keep-oldest block), so the cluster is not NoDowning even
-/// without the typed option. These tests assert
-/// sets the typed
-/// KeepOldestOption { DownIfAlone = true } so the strategy is explicit in code rather than relying on
-/// the framework default — a future refactor cannot silently drop that explicitness. (Failover behaviour is
-/// separately verified by HardKillFailoverTests, which passes regardless of activation path.)
+/// Guards the downing strategy — the mechanism that decides whether a node that loses its peer
+/// keeps running or shuts itself down.
///
+///
+///
+/// Why these tests assert the EFFECTIVE config, not the typed option. The previous
+/// version of this file asserted only that
+/// returned a
+/// . That is a different question from "what does the running
+/// ActorSystem actually do". Akka.Hosting merges several HOCON fragments — the akka.conf
+/// resource, whatever WithClustering emits, and anything added explicitly — and which
+/// one wins is not readable off the names alone. A downing
+/// setting can therefore be right in the resource file, right in the typed options object,
+/// and still not be the one in force. These tests start a real host through the production
+/// bootstrap and read akka.cluster.downing-provider-class back off
+/// , which is the only assertion precedence cannot fool.
+///
+///
+/// Why the default is auto-down. A two-node cluster running the SBR
+/// keep-oldest resolver cannot survive a crash of the oldest node. Akka.NET 1.5.62's
+/// KeepOldest.OldestDecision only lets down-if-alone rescue a side holding
+/// >= 2 members, so in a 1-vs-1 split the lone survivor takes DownReachable and
+/// downs itself — then run-coordinated-shutdown-when-down terminates it. A
+/// crash of one node becomes a total outage: exactly the failure the redundancy pair exists
+/// to absorb. Proven live on the sister project's rig (see
+/// ../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md) and
+/// adopted here for the same reason. The accepted trade is dual-active during a genuine
+/// network partition; see docs/Redundancy.md.
+///
+///
public sealed class SplitBrainResolverActivationTests
{
- private static AkkaClusterOptions SampleOptions() => new()
+ private static AkkaClusterOptions SampleOptions(string? strategy = null)
{
- SeedNodes = new[] { "akka.tcp://otopcua@127.0.0.1:4053" },
- Roles = new[] { "admin", "driver" },
- };
+ var options = new AkkaClusterOptions
+ {
+ SeedNodes = new[] { "akka.tcp://otopcua@127.0.0.1:4053" },
+ Roles = new[] { "admin", "driver" },
+ };
+ if (strategy is not null)
+ {
+ options.SplitBrainResolverStrategy = strategy;
+ }
- /// Verifies the cluster options carry an explicit split-brain resolver strategy.
- [Fact]
- public void BuildClusterOptions_sets_an_explicit_split_brain_resolver()
- {
- var options = ServiceCollectionExtensions.BuildClusterOptions(SampleOptions());
-
- options.SplitBrainResolver.ShouldNotBeNull(
- "SplitBrainResolver must be set explicitly in code — keeps the strategy independent of Akka.Cluster.Hosting's default provider rather than relying on it.");
+ return options;
}
- /// Verifies the resolver is keep-oldest with down-if-alone (correct for a 2-node pair).
- [Fact]
- public void BuildClusterOptions_uses_keep_oldest_with_down_if_alone()
+ ///
+ /// Starts a real host through
+ /// — the production entry point, not a re-creation of it — and returns the ActorSystem's
+ /// effective configuration.
+ ///
+ ///
+ /// Calling the production bootstrap rather than inlining the calls it makes is deliberate: an
+ /// earlier draft of this helper rebuilt the same three steps itself, which pinned the
+ /// test's wiring instead of the shipped one and would have stayed green through a change
+ /// to production's HOCON precedence.
+ ///
+ private static async Task EffectiveConfigAsync(AkkaClusterOptions options)
{
- var options = ServiceCollectionExtensions.BuildClusterOptions(SampleOptions());
+ // Port 0 so concurrently-running test hosts cannot collide on 4053, and no seed nodes so the
+ // system never tries to join a cluster.
+ options.Port = 0;
+ options.Hostname = "127.0.0.1";
+ options.PublicHostname = "127.0.0.1";
+ options.SeedNodes = Array.Empty();
- var keepOldest = options.SplitBrainResolver.ShouldBeOfType();
- keepOldest.DownIfAlone.ShouldBe(true);
+ var builder = Host.CreateDefaultBuilder();
+ builder.ConfigureServices(services =>
+ {
+ services.AddSingleton>(Options.Create(options));
+ services.AddAkka("otopcua-sbr-test", (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp));
+ });
+
+ using var host = builder.Build();
+ await host.StartAsync();
+ try
+ {
+ return host.Services.GetRequiredService().Settings.Config;
+ }
+ finally
+ {
+ await host.StopAsync();
+ }
+ }
+
+ ///
+ /// The default strategy installs Akka's AutoDowning provider — the leader among the
+ /// reachable members downs the unreachable peer, so a crash of either node fails over.
+ ///
+ [Fact]
+ public async Task Default_strategy_installs_the_auto_downing_provider()
+ {
+ var config = await EffectiveConfigAsync(SampleOptions());
+
+ config.GetString("akka.cluster.downing-provider-class")
+ .ShouldStartWith(
+ "Akka.Cluster.AutoDowning, Akka.Cluster",
+ Case.Sensitive,
+ "the effective downing provider — not merely the configured one — must be AutoDowning, or a crash of the oldest node is a total outage");
+ }
+
+ ///
+ /// AutoDowning is inert without a downing window: auto-down-unreachable-after defaults to
+ /// off, under which the provider is installed but never downs anything. Selecting the
+ /// provider and forgetting the window would look correct and fail over never.
+ ///
+ [Fact]
+ public async Task Default_strategy_sets_the_auto_down_window()
+ {
+ var config = await EffectiveConfigAsync(SampleOptions());
+
+ config.GetTimeSpan("akka.cluster.auto-down-unreachable-after")
+ .ShouldBe(ServiceCollectionExtensions.DowningStableAfter);
+ }
+
+ ///
+ /// The opt-out is real: an operator who chooses partition-safety over availability gets the SBR
+ /// keep-oldest resolver back, in force rather than merely configured.
+ ///
+ [Fact]
+ public async Task Keep_oldest_strategy_installs_the_sbr_provider()
+ {
+ var config = await EffectiveConfigAsync(SampleOptions("keep-oldest"));
+
+ config.GetString("akka.cluster.downing-provider-class")
+ .ShouldStartWith("Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster");
+ config.GetString("akka.cluster.split-brain-resolver.active-strategy").ShouldBe("keep-oldest");
+ config.GetBoolean("akka.cluster.split-brain-resolver.keep-oldest.down-if-alone").ShouldBeTrue();
+ }
+
+ /// The default is auto-down, so an unconfigured deployment gets the survivable one.
+ [Fact]
+ public void Strategy_defaults_to_auto_down()
+ {
+ new AkkaClusterOptions().SplitBrainResolverStrategy.ShouldBe("auto-down");
+ }
+
+ ///
+ /// A typo in the strategy must fail the host at startup, not silently fall through to whichever
+ /// provider Akka.Hosting would have installed by default — which is the fatal one.
+ ///
+ [Fact]
+ public void Unknown_strategy_throws_rather_than_silently_falling_back()
+ {
+ var ex = Should.Throw(
+ () => ServiceCollectionExtensions.BuildDowningHocon(SampleOptions("keep-majority")));
+
+ ex.Message.ShouldContain("keep-majority");
+ ex.Message.ShouldContain("auto-down");
+ }
+
+ /// Strategy comparison is case-insensitive — config casing is not a failure mode.
+ [Fact]
+ public void Strategy_matching_is_case_insensitive()
+ {
+ Should.NotThrow(() => ServiceCollectionExtensions.BuildDowningHocon(SampleOptions("Auto-Down")));
+ Should.NotThrow(() => ServiceCollectionExtensions.BuildDowningHocon(SampleOptions("Keep-Oldest")));
+ }
+
+ ///
+ /// Under keep-oldest the SBR resolver is still set as a typed option, so the strategy stays
+ /// explicit in code rather than depending on Akka.Cluster.Hosting's default.
+ ///
+ [Fact]
+ public void Keep_oldest_sets_the_typed_resolver_option()
+ {
+ var options = ServiceCollectionExtensions.BuildClusterOptions(SampleOptions("keep-oldest"));
+
+ options.SplitBrainResolver.ShouldBeOfType().DownIfAlone.ShouldBe(true);
+ }
+
+ ///
+ /// Pins the two stability windows together. auto-down-unreachable-after comes from
+ /// while the SBR resolver reads
+ /// stable-after out of akka.conf; changing one and missing the other would leave the two
+ /// strategies failing over at different latencies for no stated reason.
+ ///
+ [Fact]
+ public void Downing_window_matches_the_akka_conf_stable_after()
+ {
+ var akkaConf = ConfigurationFactory.ParseString(HoconLoader.LoadBaseConfig());
+
+ akkaConf.GetTimeSpan("akka.cluster.split-brain-resolver.stable-after")
+ .ShouldBe(ServiceCollectionExtensions.DowningStableAfter);
+ }
+
+ ///
+ /// The stability window must stay above the failure detector's tolerance, or a node gets downed
+ /// while its peer is merely slow.
+ ///
+ [Fact]
+ public void Downing_window_exceeds_the_acceptable_heartbeat_pause()
+ {
+ var akkaConf = ConfigurationFactory.ParseString(HoconLoader.LoadBaseConfig());
+
+ ServiceCollectionExtensions.DowningStableAfter
+ .ShouldBeGreaterThan(akkaConf.GetTimeSpan("akka.cluster.failure-detector.acceptable-heartbeat-pause"));
}
/// Verifies seed nodes and roles are still threaded through unchanged.