fix(cluster): auto-down downing strategy — a two-node pair could not survive an oldest-node crash
Ports the sister project's live-proven fix (ScadaBridge cf3bd52f). OtOpcUa ran the identical configuration it indicts: keep-oldest with down-if-alone = on. Akka.NET 1.5.62's KeepOldest.OldestDecision only lets the down-if-alone branch rescue a side holding >= 2 members. A two-node cluster losing a peer is always a 1-vs-1 split, so the branch never fires, the lone survivor falls through to DownReachable and downs ITSELF, and run-coordinated-shutdown-when-down then terminates it. down-if-alone is a 3+-node feature and does not do what its name suggests for a pair: a crash of the oldest node is a total outage, which is the exact failure the redundancy pair exists to absorb. Cluster:SplitBrainResolverStrategy now selects the provider, defaulting to auto-down: Akka's AutoDowning with auto-down-unreachable-after = 15s, so the leader among the reachable members downs the unreachable peer and a crash of either node fails over in place. keep-oldest remains available for deployments that would rather take an outage than ever run dual-active during a real partition. An unrecognised value fails the host at startup rather than falling through to the fatal default. The tests assert the EFFECTIVE configuration — they start a real host through WithOtOpcUaClusterBootstrap and read akka.cluster.downing-provider-class back off the running ActorSystem. This is not incidental. The first draft inlined the three calls the bootstrap makes instead of calling it, which pinned the test's own wiring: sabotaging production's HOCON precedence left all 36 green. The prior file had the same shape at a smaller scale, asserting only that BuildClusterOptions returned a KeepOldestOption — true, and true of a configuration that cannot fail over. Positive control: making BuildDowningHocon emit nothing for auto-down turns exactly the two effective-config guards red. Measured while verifying rather than assumed: HoconAddMode.Append also wins here, purely because it is added last, so the mode name is not the guarantee. The comment now says so instead of asserting a precedence rule that does not hold. Not yet live-drilled on OtOpcUa. The docker-dev rig is a single six-node mesh where the 1-vs-1 pathology cannot occur; the kill-the-oldest drill belongs with the per-cluster mesh work that makes every mesh exactly two nodes (design doc 6.2 / Phase 0a). Recorded as an outstanding gate in docs/Redundancy.md. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
+78
-39
@@ -169,55 +169,94 @@ Each node advertises its partner via `OpcUaApplicationHostOptions.PeerApplicatio
|
|||||||
|
|
||||||
Node A lists Node B's `ApplicationUri` and vice-versa. Validated by `DualEndpointTests` in `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/` — boots two `OpcUaApplicationHost` instances on loopback, asserts a real OPCFoundation client `Session` reading `Server.ServerArray` from Node A sees both URIs.
|
Node A lists Node B's `ApplicationUri` and vice-versa. Validated by `DualEndpointTests` in `tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/` — boots two `OpcUaApplicationHost` instances on loopback, asserts a real OPCFoundation client `Session` reading `Server.ServerArray` from Node A sees both URIs.
|
||||||
|
|
||||||
## Split-brain
|
## Split-brain / downing
|
||||||
|
|
||||||
The split-brain resolver is **active by default**: Akka.Cluster.Hosting's `WithClustering` enables an SBR
|
**The default downing strategy is `auto-down` (changed 2026-07-21).** It is selected by
|
||||||
downing provider whenever `ClusterOptions.SplitBrainResolver` is null (it applies
|
`Cluster:SplitBrainResolverStrategy`, and `ServiceCollectionExtensions.BuildDowningHocon` turns that
|
||||||
`SplitBrainResolverOption.Default`, which registers `Akka.Cluster.SBR.SplitBrainResolverProvider`), and that
|
into the `akka.cluster.downing-provider-class` actually installed:
|
||||||
provider reads the `split-brain-resolver` HOCON block in `akka.conf`. On top of that,
|
|
||||||
`ServiceCollectionExtensions.BuildClusterOptions` sets the typed `ClusterOptions.SplitBrainResolver`
|
|
||||||
(`KeepOldestOption { DownIfAlone = true }`) to make the strategy **explicit in code** rather than relying on
|
|
||||||
the framework default — it is reinforcing, not the sole activator, and yields the same effective behavior. So
|
|
||||||
the cluster is **not** running `NoDowning`; hard-crashed nodes are downed and the cluster recovers (how it
|
|
||||||
recovers depends on *which* node was lost — see the table below). (Only an *explicit* `NoDowning`, e.g.
|
|
||||||
`akka.cluster.downing-provider-class = ""`, would leave both redundancy sides at ServiceLevel 240
|
|
||||||
indefinitely.) The HOCON block carries the tuning: `active-strategy = keep-oldest`, `stable-after = 15s`,
|
|
||||||
`keep-oldest.down-if-alone = on`, `failure-detector.threshold = 10.0` (its `active-strategy` + `down-if-alone`
|
|
||||||
must stay consistent with the typed option; `stable-after` lives only in HOCON because the typed option can't
|
|
||||||
express it).
|
|
||||||
|
|
||||||
`keep-oldest` is the correct strategy for a 2-node warm-redundancy pair (`keep-majority`/`static-quorum`
|
| `Cluster:SplitBrainResolverStrategy` | Provider installed | Posture |
|
||||||
are wrong for two nodes — no majority in a 1-1 split), but its two loss cases recover **differently**, and
|
|
||||||
this is inherent to any 2-node cluster:
|
|
||||||
|
|
||||||
| Node lost | What keep-oldest does | Recovery |
|
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| **Younger** node (crash or partition) | The oldest is on the surviving side → it stays up and downs the younger side (`DownUnreachable`). | **In-place, fast.** The oldest keeps its singletons + `driver` role-leadership; `RedundancyStateActor` re-computes from the post-loss `Cluster.State`. |
|
| `auto-down` (**default**) | `Akka.Cluster.AutoDowning` with `auto-down-unreachable-after = 15s` | **Availability.** The leader among the *reachable* members downs the unreachable peer, so a crash of **either** node — oldest included — fails over in place. |
|
||||||
| **Oldest** node (crash or partition) | The lone survivor is "the side **without** the oldest" → keep-oldest downs the **survivor itself** (`DownReachable "including myself"`), and `run-coordinated-shutdown-when-down = on` terminates it. `down-if-alone` does **not** rescue a lone survivor (its rescue branch needs ≥ 2 surviving members, so it is a 3+-node feature). | **Exit-and-rejoin.** The self-downed survivor **and** the crashed oldest are both restarted by the service supervisor and re-form / rejoin. There is a brief restart-window outage; there is **no** in-place takeover. |
|
| `keep-oldest` | `Akka.Cluster.SBR.SplitBrainResolverProvider` reading the `split-brain-resolver` block in `akka.conf` | **Partition-safety.** A partition can never run dual-active. **Not safe for a 2-node pair** — see below. |
|
||||||
|
|
||||||
**Recovery therefore depends on three things being in place** (the [ScadaBridge](../../ScadaBridge/CLAUDE.md)
|
Any other value **fails the host at startup** rather than falling through to a default.
|
||||||
sister project runs the same pattern):
|
|
||||||
|
### Why keep-oldest is no longer the default
|
||||||
|
|
||||||
|
The previous documentation in this section — and the code comments behind it — asserted that
|
||||||
|
`keep-oldest` with `down-if-alone = on` was "the correct strategy for a 2-node warm-redundancy pair."
|
||||||
|
**That was wrong, and the error was not conservative: it described a total-outage configuration as the
|
||||||
|
recommended one.**
|
||||||
|
|
||||||
|
In Akka.NET 1.5.62's `KeepOldest.OldestDecision`, the `down-if-alone` rescue branch requires the
|
||||||
|
*surviving* side to hold **≥ 2 members**. In a two-node cluster a lost peer is always a 1-vs-1 split, so
|
||||||
|
the branch never fires and the lone survivor falls through to `DownReachable` — downing **itself** — and
|
||||||
|
`run-coordinated-shutdown-when-down = on` then terminates it. `down-if-alone` is a 3+-node feature; it
|
||||||
|
does not do what its name suggests here. Live-proven on the sister project's rig, whose survivor logged
|
||||||
|
`SBR took decision …DownReachable and is downing [self] including myself, [1] unreachable of [2] members`.
|
||||||
|
See [`../../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md`](../../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md)
|
||||||
|
for the upstream decision record, which OtOpcUa follows.
|
||||||
|
|
||||||
|
The alternatives do not help a pair: `static-quorum` with quorum 1 hits Akka's `IsTooManyMembers` guard
|
||||||
|
and returns `DownAll`; with quorum 2 the survivor downs itself; `keep-majority` keeps the lowest-address
|
||||||
|
side, which just moves the fatal crash from "the oldest node" to "the other node"; `lease-majority` needs
|
||||||
|
a shared lease store both nodes can reach.
|
||||||
|
|
||||||
|
### The accepted trade
|
||||||
|
|
||||||
|
Under `auto-down`, a **genuine network partition** (both nodes alive, link cut) leaves each side downing
|
||||||
|
the other and continuing alone: **both run active**, both hold singletons, and both advertise the primary
|
||||||
|
`ServiceLevel`. Recovery is operator-driven — after the link heals, restart **one** side; it rejoins as a
|
||||||
|
fresh incarnation and settles as secondary. The two sides do not merge on their own, because the mutual
|
||||||
|
downing quarantines the association.
|
||||||
|
|
||||||
|
This is a deliberate choice of availability over partition-safety, matching ScadaBridge: the pairs run one
|
||||||
|
node per VM with no external arbiter available, so a crash is a far more likely event than a partition, and
|
||||||
|
a crash under `keep-oldest` was unrecoverable without operator action anyway.
|
||||||
|
|
||||||
|
Deployments that would rather take an outage than ever run dual-active can set
|
||||||
|
`Cluster:SplitBrainResolverStrategy: "keep-oldest"` — but should read the section above first, because for a
|
||||||
|
two-node pair that choice means *any* crash of the oldest node is a full outage.
|
||||||
|
|
||||||
|
### Recovery still depends on supervision
|
||||||
|
|
||||||
1. **A service supervisor that restarts the process** on exit — production `Install-Services.ps1` sets
|
1. **A service supervisor that restarts the process** on exit — production `Install-Services.ps1` sets
|
||||||
`sc.exe failure OtOpcUaHost … actions= restart/5000/restart/30000/restart/60000`; the docker-dev rig sets
|
`sc.exe failure OtOpcUaHost … actions= restart/5000/restart/30000/restart/60000`; the docker-dev rig sets
|
||||||
`restart: unless-stopped`. Without it a downed node stays down and an oldest-crash looks like a total outage.
|
`restart: unless-stopped`.
|
||||||
2. **The recovery watchdog** `ActorSystemTerminationWatchdog` (registered in `Program.cs` after `AddAkka`) — it
|
2. **The recovery watchdog** `ActorSystemTerminationWatchdog` (registered in `Program.cs` after `AddAkka`) —
|
||||||
watches `ActorSystem.WhenTerminated` and, on an unexpected self-down, calls
|
it watches `ActorSystem.WhenTerminated` and, on an unexpected self-down, calls
|
||||||
`IHostApplicationLifetime.StopApplication()` so the process exits (and the supervisor restarts it) instead of
|
`IHostApplicationLifetime.StopApplication()` so the process exits and the supervisor restarts it instead
|
||||||
idling forever with a dead actor system.
|
of idling forever with a dead actor system. Under `auto-down` the *survivor* no longer needs this
|
||||||
3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join the mesh via either peer.
|
(it is never the one downed); the **downed** node still does.
|
||||||
|
3. **Both peers listed in `SeedNodes`** on every node, so a restarted node can re-join via either peer.
|
||||||
|
Note the residual bootstrap constraint: only the **first** seed may form a cluster alone, so a node
|
||||||
|
cold-starting while its peer is dead still waits in `InitJoin`. Auto-down removes the crash outage, not
|
||||||
|
this one.
|
||||||
|
|
||||||
> **On `HardKillFailoverTests`:** that in-process test hard-kills the oldest and asserts the survivor becomes
|
> **On `HardKillFailoverTests`:** that in-process test hard-kills the oldest and asserts the survivor becomes
|
||||||
> sole `driver` role-leader, and it passes — but it simulates the crash with `provider.Transport.Shutdown()`,
|
> sole `driver` role-leader. It passed even under `keep-oldest` — because it simulates the crash with
|
||||||
> which leaves node A's `ActorSystem` **alive** (a transport partition with the oldest still running), not a
|
> `provider.Transport.Shutdown()`, which leaves node A's `ActorSystem` **alive** (a transport partition with
|
||||||
> real process death. It is therefore **not** representative of an oldest-process crash; a real 2-container
|
> the oldest still running), not a real process death. A real 2-container `docker kill` of the oldest downed
|
||||||
> `docker kill` of the oldest downs the survivor (verified 2026-07-15, see
|
> the survivor (verified 2026-07-15, `archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`).
|
||||||
> `archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md`). Treat the recovery guarantee for
|
> **It is a standing example of a green test over a fatal defect**: keep the distinction between a transport
|
||||||
> the oldest as "exit-and-rejoin under supervision," not "in-place failover."
|
> partition and a process death in mind when reading it.
|
||||||
|
|
||||||
There is no operator-driven role swap during a partition. Failover / recovery is what the cluster + supervisor
|
`SplitBrainResolverActivationTests` pins the strategy by starting a real host through the production
|
||||||
do automatically. **Instant in-place takeover on *any* single-node loss requires 3+ members** (an odd cluster
|
bootstrap and reading `akka.cluster.downing-provider-class` back off the running `ActorSystem` — asserting
|
||||||
or a lightweight witness) — a deliberate future option, not the current 2-node posture.
|
the typed option or the akka.conf text is not sufficient, because several HOCON fragments compete and the
|
||||||
|
losing one still looks correct in the file.
|
||||||
|
|
||||||
|
There is no operator-driven role swap during a partition; failover is what the cluster and supervisor do
|
||||||
|
automatically.
|
||||||
|
|
||||||
|
> **Live gate outstanding.** The auto-down change is verified by unit tests against the effective
|
||||||
|
> configuration, and by the sister project's live drill on an equivalent Akka version and topology. It has
|
||||||
|
> **not** yet been drilled on an OtOpcUa two-node rig — the docker-dev rig runs a single six-node mesh, where
|
||||||
|
> the 1-vs-1 pathology cannot occur. The drill (kill the oldest container of a genuine two-node cluster;
|
||||||
|
> assert the survivor stays up, takes the singletons, and reports the primary `ServiceLevel`) should run
|
||||||
|
> alongside the per-cluster mesh work, which makes every mesh exactly two nodes. See
|
||||||
|
> `docs/plans/2026-07-21-per-cluster-mesh-design.md` §6.2 / Phase 0a.
|
||||||
|
|
||||||
## Primary data-plane gate (writes, acks, alerts emit)
|
## Primary data-plane gate (writes, acks, alerts emit)
|
||||||
|
|
||||||
|
|||||||
@@ -29,4 +29,30 @@ public sealed class AkkaClusterOptions
|
|||||||
/// <c>admin</c>, <c>driver</c>, <c>dev</c>.
|
/// <c>admin</c>, <c>driver</c>, <c>dev</c>.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
public string[] Roles { get; set; } = Array.Empty<string>();
|
public string[] Roles { get; set; } = Array.Empty<string>();
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// How the cluster decides to down a node it can no longer reach. One of <c>auto-down</c>
|
||||||
|
/// (default) or <c>keep-oldest</c>; any other value fails the host at startup.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// <para>
|
||||||
|
/// <b><c>auto-down</c> — availability.</b> The leader among the <i>reachable</i> members
|
||||||
|
/// downs the unreachable peer after
|
||||||
|
/// <see cref="ServiceCollectionExtensions.DowningStableAfter"/>. A hard crash of either
|
||||||
|
/// node — including the oldest — fails over to the survivor with no operator action.
|
||||||
|
/// The trade is that a genuine network partition (both nodes alive, link cut) leaves
|
||||||
|
/// both sides running active until an operator restarts one.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// <b><c>keep-oldest</c> — partition-safety.</b> The SBR resolver sacrifices the younger
|
||||||
|
/// side of a split, so a partition can never run dual-active. <b>The cost is severe for a
|
||||||
|
/// two-node pair:</b> it cannot survive a crash of the oldest node at all. Akka.NET's
|
||||||
|
/// <c>KeepOldest.OldestDecision</c> only lets <c>down-if-alone</c> rescue a side holding
|
||||||
|
/// >= 2 members, so the 1-vs-1 survivor downs <i>itself</i> and shuts down — the
|
||||||
|
/// redundancy pair turns a single-node crash into a total outage. Choose this only for
|
||||||
|
/// clusters of three or more nodes, or where dual-active is genuinely worse than an
|
||||||
|
/// outage.
|
||||||
|
/// </para>
|
||||||
|
/// </remarks>
|
||||||
|
public string SplitBrainResolverStrategy { get; set; } = "auto-down";
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -37,14 +37,26 @@ akka {
|
|||||||
roles = []
|
roles = []
|
||||||
min-nr-of-members = 1
|
min-nr-of-members = 1
|
||||||
|
|
||||||
# Split-brain resolver (arch-review 03/S1). This block IS active: Akka.Cluster.Hosting's
|
# DOWNING. The provider is NOT selected here — this file is added with HoconAddMode.Append,
|
||||||
# WithClustering enables an SBR downing provider by default (it applies
|
# which loses to what Akka.Hosting's WithClustering emits, so a downing-provider-class set
|
||||||
# SplitBrainResolverOption.Default when ClusterOptions.SplitBrainResolver is null), and that
|
# here would be silently overridden. Selection lives in
|
||||||
# provider reads this block. ServiceCollectionExtensions.BuildClusterOptions additionally sets
|
# ServiceCollectionExtensions.BuildDowningHocon (Prepended, which outranks WithClustering),
|
||||||
# the typed KeepOldestOption { DownIfAlone = true } to make the strategy EXPLICIT in code —
|
# driven by Cluster:SplitBrainResolverStrategy.
|
||||||
# its active-strategy + keep-oldest.down-if-alone MUST match this block. (The cluster is NOT
|
#
|
||||||
# NoDowning; only an explicit downing-provider-class = "" would disable failover.)
|
# The DEFAULT strategy is "auto-down", under which this whole block is inert: Akka's
|
||||||
# stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin.
|
# AutoDowning provider is installed with auto-down-unreachable-after = 15s, and the leader
|
||||||
|
# among the reachable members downs the unreachable peer — so a hard crash of EITHER node,
|
||||||
|
# oldest included, fails over. The trade is dual-active during a real network partition.
|
||||||
|
#
|
||||||
|
# The block below applies only when an operator selects "keep-oldest". Note that keep-oldest
|
||||||
|
# is NOT safe for a two-node pair: Akka.NET's KeepOldest.OldestDecision only lets
|
||||||
|
# down-if-alone rescue a side holding >= 2 members, so a 1-vs-1 survivor downs ITSELF and
|
||||||
|
# (with run-coordinated-shutdown-when-down) exits — a crash of the oldest node becomes a
|
||||||
|
# total outage. See docs/Redundancy.md.
|
||||||
|
#
|
||||||
|
# stable-after must stay >= failure-detector.acceptable-heartbeat-pause (10s) + margin, and
|
||||||
|
# must equal ServiceCollectionExtensions.DowningStableAfter so both strategies fail over at
|
||||||
|
# the same latency. Both constraints are pinned by SplitBrainResolverActivationTests.
|
||||||
split-brain-resolver {
|
split-brain-resolver {
|
||||||
active-strategy = "keep-oldest"
|
active-strategy = "keep-oldest"
|
||||||
stable-after = 15s
|
stable-after = 15s
|
||||||
|
|||||||
@@ -80,40 +80,125 @@ public static class ServiceCollectionExtensions
|
|||||||
|
|
||||||
builder.WithClustering(BuildClusterOptions(options));
|
builder.WithClustering(BuildClusterOptions(options));
|
||||||
|
|
||||||
|
// Must come AFTER WithClustering, which always emits a downing-provider-class of its own:
|
||||||
|
// Akka.Cluster.Hosting applies SplitBrainResolverOption.Default whenever the typed
|
||||||
|
// ClusterOptions.SplitBrainResolver is null, so there is always a competing value to beat.
|
||||||
|
// Prepend is the highest-precedence mode, which makes this block win irrespective of how
|
||||||
|
// many fragments are added or in what order.
|
||||||
|
//
|
||||||
|
// Do not take this comment's word for it. Which fragment actually wins is not obvious from
|
||||||
|
// the mode names — measured, Append also wins here, purely because it happens to be added
|
||||||
|
// last. SplitBrainResolverActivationTests reads the provider back off a *running*
|
||||||
|
// ActorSystem for exactly that reason: the effective value is the only one worth asserting.
|
||||||
|
builder.AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend);
|
||||||
|
|
||||||
return builder;
|
return builder;
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Builds the <see cref="ClusterOptions"/> for the fused-host cluster, setting the split-brain-resolver
|
/// How long a member must stay unreachable before it is downed. Shared by both strategies
|
||||||
/// strategy <b>explicitly in code</b>.
|
/// (<c>auto-down-unreachable-after</c> and the SBR resolver's <c>stable-after</c>) so the two
|
||||||
///
|
/// have the same failover latency. Must stay above
|
||||||
/// <b>Activation note (arch-review 03/S1, corrected):</b> Akka.Cluster.Hosting's <c>WithClustering</c>
|
/// <c>akka.cluster.failure-detector.acceptable-heartbeat-pause</c> or a merely-slow node gets
|
||||||
/// already enables an SBR downing provider <b>by default</b> — when <see cref="ClusterOptions.SplitBrainResolver"/>
|
/// downed; both constraints are pinned by tests.
|
||||||
/// is <c>null</c> it applies <c>SplitBrainResolverOption.Default</c>, which registers
|
|
||||||
/// <c>Akka.Cluster.SBR.SplitBrainResolverProvider</c> and reads the <c>split-brain-resolver</c> HOCON block in
|
|
||||||
/// <c>Resources/akka.conf</c>. So the cluster was <b>not</b> running <c>NoDowning</c> before this option was
|
|
||||||
/// set: the pre-existing akka.conf <c>keep-oldest</c> block was already active, and hard-crashed nodes already
|
|
||||||
/// failed over. Setting this typed option makes the strategy <b>explicit in code</b> (independent of the
|
|
||||||
/// framework default) rather than being the sole activator — it is reinforcing/belt-and-suspenders and
|
|
||||||
/// produces the same effective behavior. (Only an <i>explicit</i> <c>NoDowning</c>, e.g.
|
|
||||||
/// <c>downing-provider-class = ""</c>, would disable failover.)
|
|
||||||
///
|
|
||||||
/// <see cref="KeepOldestOption"/> with <c>DownIfAlone=true</c> mirrors the HOCON intent and is the
|
|
||||||
/// correct strategy for a 2-node warm-redundancy pair: on an even split the oldest member (typically
|
|
||||||
/// the long-running primary) survives, and <c>down-if-alone</c> downs a singleton that loses its
|
|
||||||
/// peer. <c>keep-majority</c>/<c>static-quorum</c> are wrong for two nodes. The strategy + down-if-alone
|
|
||||||
/// here MUST stay consistent with the HOCON block; <c>stable-after</c> lives only in HOCON because the
|
|
||||||
/// typed option cannot express it (it must stay ≥ <c>failure-detector.acceptable-heartbeat-pause</c>).
|
|
||||||
/// </summary>
|
/// </summary>
|
||||||
/// <param name="options">The bound cluster options carrying seed nodes and roles.</param>
|
public static readonly TimeSpan DowningStableAfter = TimeSpan.FromSeconds(15);
|
||||||
/// <returns>The cluster options with seed nodes, roles, and the explicit split-brain-resolver strategy.</returns>
|
|
||||||
|
/// <summary>
|
||||||
|
/// Builds the HOCON that selects the downing provider, per
|
||||||
|
/// <see cref="AkkaClusterOptions.SplitBrainResolverStrategy"/>.
|
||||||
|
/// </summary>
|
||||||
|
/// <param name="options">The bound cluster options carrying the strategy.</param>
|
||||||
|
/// <returns>
|
||||||
|
/// For <c>auto-down</c>, a block installing Akka's <c>AutoDowning</c> provider with a downing
|
||||||
|
/// window of <see cref="DowningStableAfter"/>. For <c>keep-oldest</c>, an empty string — the
|
||||||
|
/// typed <see cref="KeepOldestOption"/> from <see cref="BuildClusterOptions"/> already installs
|
||||||
|
/// the SBR provider, and the resolver's own settings live in <c>Resources/akka.conf</c>.
|
||||||
|
/// </returns>
|
||||||
|
/// <exception cref="ArgumentOutOfRangeException">The strategy is not a recognised value.</exception>
|
||||||
|
public static string BuildDowningHocon(AkkaClusterOptions options)
|
||||||
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(options);
|
||||||
|
|
||||||
|
if (IsAutoDown(options))
|
||||||
|
{
|
||||||
|
// auto-down-unreachable-after is load-bearing: it defaults to `off`, under which
|
||||||
|
// AutoDowning is installed but never downs anything — failover silently disabled.
|
||||||
|
return $$"""
|
||||||
|
akka.cluster {
|
||||||
|
downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"
|
||||||
|
auto-down-unreachable-after = {{(int)DowningStableAfter.TotalMilliseconds}}ms
|
||||||
|
}
|
||||||
|
""";
|
||||||
|
}
|
||||||
|
|
||||||
|
if (IsKeepOldest(options))
|
||||||
|
{
|
||||||
|
return string.Empty;
|
||||||
|
}
|
||||||
|
|
||||||
|
throw new ArgumentOutOfRangeException(
|
||||||
|
nameof(options),
|
||||||
|
options.SplitBrainResolverStrategy,
|
||||||
|
$"Unknown {AkkaClusterOptions.SectionName}:{nameof(AkkaClusterOptions.SplitBrainResolverStrategy)}. "
|
||||||
|
+ "Expected 'auto-down' (default; survives a crash of either node) or 'keep-oldest' "
|
||||||
|
+ "(partition-safe, but a two-node pair cannot survive a crash of the oldest node).");
|
||||||
|
}
|
||||||
|
|
||||||
|
private static bool IsAutoDown(AkkaClusterOptions options) =>
|
||||||
|
string.Equals(options.SplitBrainResolverStrategy, "auto-down", StringComparison.OrdinalIgnoreCase);
|
||||||
|
|
||||||
|
private static bool IsKeepOldest(AkkaClusterOptions options) =>
|
||||||
|
string.Equals(options.SplitBrainResolverStrategy, "keep-oldest", StringComparison.OrdinalIgnoreCase);
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Builds the <see cref="ClusterOptions"/> for the fused-host cluster.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// <para>
|
||||||
|
/// <b>Activation note (arch-review 03/S1, corrected twice).</b> Akka.Cluster.Hosting's
|
||||||
|
/// <c>WithClustering</c> always installs a downing provider: when
|
||||||
|
/// <see cref="ClusterOptions.SplitBrainResolver"/> is <c>null</c> it applies
|
||||||
|
/// <c>SplitBrainResolverOption.Default</c>, which registers
|
||||||
|
/// <c>Akka.Cluster.SBR.SplitBrainResolverProvider</c> and reads the
|
||||||
|
/// <c>split-brain-resolver</c> block in <c>Resources/akka.conf</c>. The cluster is
|
||||||
|
/// therefore never <c>NoDowning</c>. What this method chooses is only <i>which</i>
|
||||||
|
/// provider — and under the default <c>auto-down</c> strategy the choice is not made
|
||||||
|
/// here at all but by the Prepended HOCON in
|
||||||
|
/// <see cref="WithOtOpcUaClusterBootstrap"/>, which outranks whatever
|
||||||
|
/// <c>WithClustering</c> emits.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// <b>Why keep-oldest is no longer the default (2026-07-21).</b> The previous
|
||||||
|
/// documentation here asserted that <see cref="KeepOldestOption"/> with
|
||||||
|
/// <c>DownIfAlone=true</c> was "the correct strategy for a 2-node warm-redundancy pair"
|
||||||
|
/// because <c>down-if-alone</c> would down a node that lost its peer. That is the
|
||||||
|
/// opposite of what Akka.NET does. In <c>KeepOldest.OldestDecision</c> the
|
||||||
|
/// <c>down-if-alone</c> branch requires the surviving side to hold >= 2 members, so
|
||||||
|
/// in a 1-vs-1 split the survivor falls through to <c>DownReachable</c> and downs
|
||||||
|
/// <i>itself</i>; with <c>run-coordinated-shutdown-when-down = on</c> it then exits. A
|
||||||
|
/// two-node pair running keep-oldest converts a crash of the oldest node into a total
|
||||||
|
/// outage — precisely the failure redundancy exists to absorb. Live-proven on the sister
|
||||||
|
/// project's rig; see <c>docs/Redundancy.md</c> and
|
||||||
|
/// <see cref="AkkaClusterOptions.SplitBrainResolverStrategy"/> for the trade.
|
||||||
|
/// </para>
|
||||||
|
/// </remarks>
|
||||||
|
/// <param name="options">The bound cluster options carrying seed nodes, roles and the strategy.</param>
|
||||||
|
/// <returns>
|
||||||
|
/// The cluster options with seed nodes and roles. <see cref="ClusterOptions.SplitBrainResolver"/>
|
||||||
|
/// is set only under <c>keep-oldest</c>, where the SBR provider is the one wanted; under
|
||||||
|
/// <c>auto-down</c> it is left null because the Prepended HOCON replaces the provider anyway,
|
||||||
|
/// and naming a resolver that is not in force would be misleading.
|
||||||
|
/// </returns>
|
||||||
public static ClusterOptions BuildClusterOptions(AkkaClusterOptions options)
|
public static ClusterOptions BuildClusterOptions(AkkaClusterOptions options)
|
||||||
{
|
{
|
||||||
|
ArgumentNullException.ThrowIfNull(options);
|
||||||
|
|
||||||
return new ClusterOptions
|
return new ClusterOptions
|
||||||
{
|
{
|
||||||
SeedNodes = options.SeedNodes,
|
SeedNodes = options.SeedNodes,
|
||||||
Roles = options.Roles,
|
Roles = options.Roles,
|
||||||
SplitBrainResolver = new KeepOldestOption { DownIfAlone = true },
|
SplitBrainResolver = IsKeepOldest(options) ? new KeepOldestOption { DownIfAlone = true } : null,
|
||||||
};
|
};
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,47 +1,215 @@
|
|||||||
|
using Akka.Actor;
|
||||||
using Akka.Cluster.Hosting;
|
using Akka.Cluster.Hosting;
|
||||||
using Akka.Cluster.Hosting.SBR;
|
using Akka.Cluster.Hosting.SBR;
|
||||||
|
using Akka.Configuration;
|
||||||
|
using Akka.Hosting;
|
||||||
|
using Microsoft.Extensions.DependencyInjection;
|
||||||
|
using Microsoft.Extensions.Hosting;
|
||||||
|
using Microsoft.Extensions.Options;
|
||||||
using Shouldly;
|
using Shouldly;
|
||||||
using Xunit;
|
using Xunit;
|
||||||
|
|
||||||
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
|
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// Guards that the split-brain-resolver strategy stays set <b>explicitly in code</b> (arch-review 03/S1,
|
/// Guards the downing strategy — the mechanism that decides whether a node that loses its peer
|
||||||
/// premise corrected — see #11). Akka.Cluster.Hosting already enables an SBR downing provider by default
|
/// keeps running or shuts itself down.
|
||||||
/// (it applies <c>SplitBrainResolverOption.Default</c> when <see cref="ClusterOptions.SplitBrainResolver"/>
|
|
||||||
/// is null, which reads the akka.conf <c>keep-oldest</c> block), so the cluster is not NoDowning even
|
|
||||||
/// without the typed option. These tests assert
|
|
||||||
/// <see cref="ServiceCollectionExtensions.BuildClusterOptions"/> sets the typed
|
|
||||||
/// <c>KeepOldestOption { DownIfAlone = true }</c> so the strategy is explicit in code rather than relying on
|
|
||||||
/// the framework default — a future refactor cannot silently drop that explicitness. (Failover behaviour is
|
|
||||||
/// separately verified by <c>HardKillFailoverTests</c>, which passes regardless of activation path.)
|
|
||||||
/// </summary>
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// <para>
|
||||||
|
/// <b>Why these tests assert the EFFECTIVE config, not the typed option.</b> The previous
|
||||||
|
/// version of this file asserted only that
|
||||||
|
/// <see cref="ServiceCollectionExtensions.BuildClusterOptions"/> returned a
|
||||||
|
/// <see cref="KeepOldestOption"/>. That is a different question from "what does the running
|
||||||
|
/// ActorSystem actually do". Akka.Hosting merges several HOCON fragments — the akka.conf
|
||||||
|
/// resource, whatever <c>WithClustering</c> emits, and anything added explicitly — and which
|
||||||
|
/// one wins is not readable off the <see cref="HoconAddMode"/> names alone. A downing
|
||||||
|
/// setting can therefore be right in the resource file, right in the typed options object,
|
||||||
|
/// and still not be the one in force. These tests start a real host through the production
|
||||||
|
/// bootstrap and read <c>akka.cluster.downing-provider-class</c> back off
|
||||||
|
/// <see cref="ActorSystem.Settings"/>, which is the only assertion precedence cannot fool.
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// <b>Why the default is auto-down.</b> A two-node cluster running the SBR
|
||||||
|
/// <c>keep-oldest</c> resolver cannot survive a crash of the oldest node. Akka.NET 1.5.62's
|
||||||
|
/// <c>KeepOldest.OldestDecision</c> only lets <c>down-if-alone</c> rescue a side holding
|
||||||
|
/// >= 2 members, so in a 1-vs-1 split the lone survivor takes <c>DownReachable</c> and
|
||||||
|
/// downs <i>itself</i> — then <c>run-coordinated-shutdown-when-down</c> terminates it. A
|
||||||
|
/// crash of one node becomes a total outage: exactly the failure the redundancy pair exists
|
||||||
|
/// to absorb. Proven live on the sister project's rig (see
|
||||||
|
/// <c>../ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md</c>) and
|
||||||
|
/// adopted here for the same reason. The accepted trade is dual-active during a genuine
|
||||||
|
/// network partition; see <c>docs/Redundancy.md</c>.
|
||||||
|
/// </para>
|
||||||
|
/// </remarks>
|
||||||
public sealed class SplitBrainResolverActivationTests
|
public sealed class SplitBrainResolverActivationTests
|
||||||
{
|
{
|
||||||
private static AkkaClusterOptions SampleOptions() => new()
|
private static AkkaClusterOptions SampleOptions(string? strategy = null)
|
||||||
{
|
{
|
||||||
SeedNodes = new[] { "akka.tcp://otopcua@127.0.0.1:4053" },
|
var options = new AkkaClusterOptions
|
||||||
Roles = new[] { "admin", "driver" },
|
{
|
||||||
};
|
SeedNodes = new[] { "akka.tcp://otopcua@127.0.0.1:4053" },
|
||||||
|
Roles = new[] { "admin", "driver" },
|
||||||
|
};
|
||||||
|
if (strategy is not null)
|
||||||
|
{
|
||||||
|
options.SplitBrainResolverStrategy = strategy;
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>Verifies the cluster options carry an explicit split-brain resolver strategy.</summary>
|
return options;
|
||||||
[Fact]
|
|
||||||
public void BuildClusterOptions_sets_an_explicit_split_brain_resolver()
|
|
||||||
{
|
|
||||||
var options = ServiceCollectionExtensions.BuildClusterOptions(SampleOptions());
|
|
||||||
|
|
||||||
options.SplitBrainResolver.ShouldNotBeNull(
|
|
||||||
"SplitBrainResolver must be set explicitly in code — keeps the strategy independent of Akka.Cluster.Hosting's default provider rather than relying on it.");
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>Verifies the resolver is keep-oldest with down-if-alone (correct for a 2-node pair).</summary>
|
/// <summary>
|
||||||
[Fact]
|
/// Starts a real host through <see cref="ServiceCollectionExtensions.WithOtOpcUaClusterBootstrap"/>
|
||||||
public void BuildClusterOptions_uses_keep_oldest_with_down_if_alone()
|
/// — the production entry point, not a re-creation of it — and returns the ActorSystem's
|
||||||
|
/// effective configuration.
|
||||||
|
/// </summary>
|
||||||
|
/// <remarks>
|
||||||
|
/// Calling the production bootstrap rather than inlining the calls it makes is deliberate: an
|
||||||
|
/// earlier draft of this helper rebuilt the same three steps itself, which pinned the
|
||||||
|
/// <i>test's</i> wiring instead of the shipped one and would have stayed green through a change
|
||||||
|
/// to production's HOCON precedence.
|
||||||
|
/// </remarks>
|
||||||
|
private static async Task<Config> EffectiveConfigAsync(AkkaClusterOptions options)
|
||||||
{
|
{
|
||||||
var options = ServiceCollectionExtensions.BuildClusterOptions(SampleOptions());
|
// Port 0 so concurrently-running test hosts cannot collide on 4053, and no seed nodes so the
|
||||||
|
// system never tries to join a cluster.
|
||||||
|
options.Port = 0;
|
||||||
|
options.Hostname = "127.0.0.1";
|
||||||
|
options.PublicHostname = "127.0.0.1";
|
||||||
|
options.SeedNodes = Array.Empty<string>();
|
||||||
|
|
||||||
var keepOldest = options.SplitBrainResolver.ShouldBeOfType<KeepOldestOption>();
|
var builder = Host.CreateDefaultBuilder();
|
||||||
keepOldest.DownIfAlone.ShouldBe(true);
|
builder.ConfigureServices(services =>
|
||||||
|
{
|
||||||
|
services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
|
||||||
|
services.AddAkka("otopcua-sbr-test", (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp));
|
||||||
|
});
|
||||||
|
|
||||||
|
using var host = builder.Build();
|
||||||
|
await host.StartAsync();
|
||||||
|
try
|
||||||
|
{
|
||||||
|
return host.Services.GetRequiredService<ActorSystem>().Settings.Config;
|
||||||
|
}
|
||||||
|
finally
|
||||||
|
{
|
||||||
|
await host.StopAsync();
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The default strategy installs Akka's AutoDowning provider — the leader among the
|
||||||
|
/// <i>reachable</i> members downs the unreachable peer, so a crash of either node fails over.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public async Task Default_strategy_installs_the_auto_downing_provider()
|
||||||
|
{
|
||||||
|
var config = await EffectiveConfigAsync(SampleOptions());
|
||||||
|
|
||||||
|
config.GetString("akka.cluster.downing-provider-class")
|
||||||
|
.ShouldStartWith(
|
||||||
|
"Akka.Cluster.AutoDowning, Akka.Cluster",
|
||||||
|
Case.Sensitive,
|
||||||
|
"the effective downing provider — not merely the configured one — must be AutoDowning, or a crash of the oldest node is a total outage");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// AutoDowning is inert without a downing window: <c>auto-down-unreachable-after</c> defaults to
|
||||||
|
/// <c>off</c>, under which the provider is installed but never downs anything. Selecting the
|
||||||
|
/// provider and forgetting the window would look correct and fail over never.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public async Task Default_strategy_sets_the_auto_down_window()
|
||||||
|
{
|
||||||
|
var config = await EffectiveConfigAsync(SampleOptions());
|
||||||
|
|
||||||
|
config.GetTimeSpan("akka.cluster.auto-down-unreachable-after")
|
||||||
|
.ShouldBe(ServiceCollectionExtensions.DowningStableAfter);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The opt-out is real: an operator who chooses partition-safety over availability gets the SBR
|
||||||
|
/// keep-oldest resolver back, in force rather than merely configured.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public async Task Keep_oldest_strategy_installs_the_sbr_provider()
|
||||||
|
{
|
||||||
|
var config = await EffectiveConfigAsync(SampleOptions("keep-oldest"));
|
||||||
|
|
||||||
|
config.GetString("akka.cluster.downing-provider-class")
|
||||||
|
.ShouldStartWith("Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster");
|
||||||
|
config.GetString("akka.cluster.split-brain-resolver.active-strategy").ShouldBe("keep-oldest");
|
||||||
|
config.GetBoolean("akka.cluster.split-brain-resolver.keep-oldest.down-if-alone").ShouldBeTrue();
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>The default is auto-down, so an unconfigured deployment gets the survivable one.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void Strategy_defaults_to_auto_down()
|
||||||
|
{
|
||||||
|
new AkkaClusterOptions().SplitBrainResolverStrategy.ShouldBe("auto-down");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// A typo in the strategy must fail the host at startup, not silently fall through to whichever
|
||||||
|
/// provider Akka.Hosting would have installed by default — which is the fatal one.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Unknown_strategy_throws_rather_than_silently_falling_back()
|
||||||
|
{
|
||||||
|
var ex = Should.Throw<ArgumentOutOfRangeException>(
|
||||||
|
() => ServiceCollectionExtensions.BuildDowningHocon(SampleOptions("keep-majority")));
|
||||||
|
|
||||||
|
ex.Message.ShouldContain("keep-majority");
|
||||||
|
ex.Message.ShouldContain("auto-down");
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>Strategy comparison is case-insensitive — config casing is not a failure mode.</summary>
|
||||||
|
[Fact]
|
||||||
|
public void Strategy_matching_is_case_insensitive()
|
||||||
|
{
|
||||||
|
Should.NotThrow(() => ServiceCollectionExtensions.BuildDowningHocon(SampleOptions("Auto-Down")));
|
||||||
|
Should.NotThrow(() => ServiceCollectionExtensions.BuildDowningHocon(SampleOptions("Keep-Oldest")));
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Under keep-oldest the SBR resolver is still set as a typed option, so the strategy stays
|
||||||
|
/// explicit in code rather than depending on Akka.Cluster.Hosting's default.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Keep_oldest_sets_the_typed_resolver_option()
|
||||||
|
{
|
||||||
|
var options = ServiceCollectionExtensions.BuildClusterOptions(SampleOptions("keep-oldest"));
|
||||||
|
|
||||||
|
options.SplitBrainResolver.ShouldBeOfType<KeepOldestOption>().DownIfAlone.ShouldBe(true);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// Pins the two stability windows together. <c>auto-down-unreachable-after</c> comes from
|
||||||
|
/// <see cref="ServiceCollectionExtensions.DowningStableAfter"/> while the SBR resolver reads
|
||||||
|
/// <c>stable-after</c> out of akka.conf; changing one and missing the other would leave the two
|
||||||
|
/// strategies failing over at different latencies for no stated reason.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Downing_window_matches_the_akka_conf_stable_after()
|
||||||
|
{
|
||||||
|
var akkaConf = ConfigurationFactory.ParseString(HoconLoader.LoadBaseConfig());
|
||||||
|
|
||||||
|
akkaConf.GetTimeSpan("akka.cluster.split-brain-resolver.stable-after")
|
||||||
|
.ShouldBe(ServiceCollectionExtensions.DowningStableAfter);
|
||||||
|
}
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// The stability window must stay above the failure detector's tolerance, or a node gets downed
|
||||||
|
/// while its peer is merely slow.
|
||||||
|
/// </summary>
|
||||||
|
[Fact]
|
||||||
|
public void Downing_window_exceeds_the_acceptable_heartbeat_pause()
|
||||||
|
{
|
||||||
|
var akkaConf = ConfigurationFactory.ParseString(HoconLoader.LoadBaseConfig());
|
||||||
|
|
||||||
|
ServiceCollectionExtensions.DowningStableAfter
|
||||||
|
.ShouldBeGreaterThan(akkaConf.GetTimeSpan("akka.cluster.failure-detector.acceptable-heartbeat-pause"));
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>Verifies seed nodes and roles are still threaded through unchanged.</summary>
|
/// <summary>Verifies seed nodes and roles are still threaded through unchanged.</summary>
|
||||||
|
|||||||
Reference in New Issue
Block a user