2-node keep-oldest cannot survive an oldest-node crash (survivor downs ITSELF) — evaluate ScadaBridge's auto-down decision #492

New Issue

2026-07-21T14:30:30-04:00

dohertj2 commented

2026-07-21 14:30:30 -04:00

Summary

ScadaBridge proved today (2026-07-21, live on its docker rig + Akka.NET 1.5.62 source) that a 2-node keep-oldest cluster can never survive a hard crash of the oldest/active node — and OtOpcUa runs the identical posture: KeepOldestOption { DownIfAlone = true } on a 2-node cluster (ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs, Akka.Cluster.Hosting 1.5.62).

When the oldest crashes, the younger survivor does NOT take over. It takes DownReachable and downs itself → total cluster outage until the crashed node is manually restarted. Captured live on the ScadaBridge rig:

SBR took decision Akka.Cluster.SBR.DownReachable and is downing
[akka.tcp://scadabridge@scadabridge-central-b:8081] including myself,
[1] unreachable of [2] members

Root cause (upstream, by design)

Akka.Cluster/SBR/DowningStrategy.cs (1.5.62), KeepOldest.OldestDecision — when the oldest is on the unreachable side:

if (DownIfAlone && otherSide == 1 && thisSide >= 2)   // survivor side must be >= 2
    return DownUnreachable.Instance;
return DownReachable.Instance;                         // 1-vs-1 -> down MYSELF

down-if-alone only rescues a surviving side with >= 2 members; upstream comments say explicitly that with 2 nodes it prefers keeping the (dead) oldest. The XML doc in ServiceCollectionExtensions.cs ("the design-intended active node survives… down-if-alone downs a singleton that loses its peer") describes the partition case only — it does not hold for a crash of the oldest, and the ActorSystemTerminationWatchdog (#459) exit-and-rejoin path cannot save it either: the self-downed survivor's restart loops on InitJoin unless it is the first seed.

Known traps if you fix this (checked and rejected in ScadaBridge)

static-quorum quorum-size 1 looks right but is a trap: IsTooManyMembers (2 > 2*1-1) returns DownAll on ANY unreachability — strictly worse.
keep-majority: a 1-vs-1 split keeps the lowest-ADDRESS side — same hole, keyed to address instead of age.
lease-majority needs a shared lease store both nodes can reach.

What ScadaBridge did (decision 2026-07-21, owner-approved: availability over partition-safety)

Switched the downing provider to Akka's built-in AutoDowning:

akka.cluster.downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"
akka.cluster.auto-down-unreachable-after = 15s

The leader among the reachable members downs the unreachable peer, so a crash of EITHER node fails over to the survivor (~25s; drill-measured takeover in 28s with the victim still down, all singletons Younger -> Oldest). Accepted trade: a real network partition produces dual-active (each side downs the other and runs its own one-node cluster) until an operator restarts one side. Note Akka.Cluster.Hosting has no typed option for AutoDowning — ScadaBridge emits it in raw HOCON; OtOpcUa would set it in Resources/akka.conf (and drop the SplitBrainResolver option so Hosting doesn't also register the SBR provider).

Full decision record + evidence: ScadaBridge docs/plans/2026-07-21-auto-down-availability-decision.md (commit cf3bd52f).

Follow-up for OtOpcUa

Decide the same trade for OtOpcUa's 2-node fused-host cluster: keep keep-oldest (oldest-crash = total outage, partition-safe) or adopt auto-down (either-crash fails over, partition = dual-active). If OtOpcUa's nodes sit on the same VMs/LAN as ScadaBridge's site pairs, the same reasoning likely applies ("network partitions are less of a risk than if this stops working").
If adopting auto-down: fix the ServiceCollectionExtensions.cs XML doc claims, re-point the failover integration tests (the F22 deviation notes mention kill-mid-apply was non-deterministic under SBR — under auto-down the oldest-crash direction becomes a testable success path), and check the seed-node ordering note: only the FIRST seed can self-form, which under auto-down only matters for boot-alone (cold start with the peer dead), not for takeover.
Also mirror the dual-active operational note in the runbook: after a partition heals, restart one side; the sides never merge on their own.

## Summary ScadaBridge proved today (2026-07-21, live on its docker rig + Akka.NET 1.5.62 source) that a **2-node `keep-oldest` cluster can never survive a hard crash of the oldest/active node** — and OtOpcUa runs the identical posture: `KeepOldestOption { DownIfAlone = true }` on a 2-node cluster (`ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs`, Akka.Cluster.Hosting 1.5.62). When the oldest crashes, the younger survivor does NOT take over. It takes `DownReachable` and **downs itself** → total cluster outage until the crashed node is manually restarted. Captured live on the ScadaBridge rig: ``` SBR took decision Akka.Cluster.SBR.DownReachable and is downing [akka.tcp://scadabridge@scadabridge-central-b:8081] including myself, [1] unreachable of [2] members ``` ## Root cause (upstream, by design) `Akka.Cluster/SBR/DowningStrategy.cs` (1.5.62), `KeepOldest.OldestDecision` — when the oldest is on the unreachable side: ```csharp if (DownIfAlone && otherSide == 1 && thisSide >= 2) // survivor side must be >= 2 return DownUnreachable.Instance; return DownReachable.Instance; // 1-vs-1 -> down MYSELF ``` `down-if-alone` only rescues a surviving side with **>= 2 members**; upstream comments say explicitly that with 2 nodes it prefers keeping the (dead) oldest. The XML doc in `ServiceCollectionExtensions.cs` ("the design-intended active node survives… down-if-alone downs a singleton that loses its peer") describes the partition case only — it does not hold for a crash of the oldest, and the `ActorSystemTerminationWatchdog` (#459) exit-and-rejoin path cannot save it either: the self-downed survivor's restart loops on `InitJoin` unless it is the first seed. ## Known traps if you fix this (checked and rejected in ScadaBridge) - **`static-quorum` quorum-size 1** looks right but is a trap: `IsTooManyMembers` (`2 > 2*1-1`) returns **DownAll** on ANY unreachability — strictly worse. - **`keep-majority`**: a 1-vs-1 split keeps the lowest-ADDRESS side — same hole, keyed to address instead of age. - **`lease-majority`** needs a shared lease store both nodes can reach. ## What ScadaBridge did (decision 2026-07-21, owner-approved: availability over partition-safety) Switched the downing provider to Akka's built-in **`AutoDowning`**: ```hocon akka.cluster.downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster" akka.cluster.auto-down-unreachable-after = 15s ``` The leader among the *reachable* members downs the unreachable peer, so a crash of EITHER node fails over to the survivor (~25s; drill-measured takeover in 28s with the victim still down, all singletons `Younger -> Oldest`). **Accepted trade: a real network partition produces dual-active** (each side downs the other and runs its own one-node cluster) until an operator restarts one side. Note Akka.Cluster.Hosting has no typed option for AutoDowning — ScadaBridge emits it in raw HOCON; OtOpcUa would set it in `Resources/akka.conf` (and drop the `SplitBrainResolver` option so Hosting doesn't also register the SBR provider). Full decision record + evidence: ScadaBridge `docs/plans/2026-07-21-auto-down-availability-decision.md` (commit `cf3bd52f`). ## Follow-up for OtOpcUa 1. Decide the same trade for OtOpcUa's 2-node fused-host cluster: keep `keep-oldest` (oldest-crash = total outage, partition-safe) or adopt `auto-down` (either-crash fails over, partition = dual-active). If OtOpcUa's nodes sit on the same VMs/LAN as ScadaBridge's site pairs, the same reasoning likely applies ("network partitions are less of a risk than if this stops working"). 2. If adopting auto-down: fix the `ServiceCollectionExtensions.cs` XML doc claims, re-point the failover integration tests (the F22 deviation notes mention kill-mid-apply was non-deterministic under SBR — under auto-down the oldest-crash direction becomes a testable success path), and check the seed-node ordering note: only the FIRST seed can self-form, which under auto-down only matters for boot-alone (cold start with the peer dead), not for takeover. 3. Also mirror the dual-active operational note in the runbook: after a partition heals, restart one side; the sides never merge on their own.

dohertj2 commented

2026-07-26 10:07:32 -04:00

Already fixed and live-verified — closing.

Code. 2964361a (2026-07-21) switched the downing provider exactly as this issue proposes. AkkaClusterOptions.SplitBrainResolverStrategy now defaults to auto-down, BuildDowningHocon installs Akka's AutoDowning provider with a non-off auto-down-unreachable-after (the default off is load-bearing and called out in the code), and an unknown value fails host start rather than silently falling back. keep-oldest survives only as an explicit opt-in.

Live drill. The 1-vs-1 crash-the-oldest gate this issue is really about was closed by the Phase 7 failover drills (docs/plans/2026-07-24-mesh-phase7-failover-drills.md), 6/6 PASS:

D3 — killed site-b-1 (Primary and oldest); site-b-2 logged "Member removed [site-b-1]", went BecomingOldest → Oldest, and served ServiceLevel 250.
D4 — the explicit gate: every survivor across D2/D3/D5 stayed Up and did not self-down (e.g. site-a-1 "Up 49 minutes", site-b-2 "Up 56 minutes"). This is the drill that closes the gate deferred since Phase 0a.
D2 — verified in both directions (kill the Primary, then kill the new Primary).

So the DownReachable-downs-itself pathology this issue captured from ScadaBridge cannot occur on the current default.

The accepted trade is unchanged and documented: auto-down produces dual-active during a genuine network partition. That was the deliberate availability-over-partition-safety call, recorded in CLAUDE.md §Redundancy and in AkkaClusterOptions.SplitBrainResolverStrategy's XML doc.

One correction to the issue's closing note: it suggested setting this in Resources/akka.conf. It is not there — several HOCON fragments compete and the file is the wrong authority. It is emitted by BuildDowningHocon and Prepended, and the effective config must be asserted off a running ActorSystem (as SplitBrainResolverActivationTests does), never off the typed options or the akka.conf text.

Already fixed and live-verified — closing. **Code.** `2964361a` (2026-07-21) switched the downing provider exactly as this issue proposes. `AkkaClusterOptions.SplitBrainResolverStrategy` now **defaults to `auto-down`**, `BuildDowningHocon` installs Akka's `AutoDowning` provider with a non-`off` `auto-down-unreachable-after` (the default `off` is load-bearing and called out in the code), and an unknown value fails host start rather than silently falling back. `keep-oldest` survives only as an explicit opt-in. **Live drill.** The 1-vs-1 crash-the-oldest gate this issue is really about was closed by the Phase 7 failover drills (`docs/plans/2026-07-24-mesh-phase7-failover-drills.md`), 6/6 PASS: - **D3** — killed `site-b-1` (Primary *and* oldest); `site-b-2` logged "Member removed [site-b-1]", went `BecomingOldest → Oldest`, and served ServiceLevel 250. - **D4** — the explicit gate: every survivor across D2/D3/D5 stayed `Up` and **did not self-down** (e.g. `site-a-1` "Up 49 minutes", `site-b-2` "Up 56 minutes"). This is the drill that closes the gate deferred since Phase 0a. - **D2** — verified in *both* directions (kill the Primary, then kill the new Primary). So the `DownReachable`-downs-itself pathology this issue captured from ScadaBridge cannot occur on the current default. **The accepted trade is unchanged and documented:** `auto-down` produces dual-active during a genuine network partition. That was the deliberate availability-over-partition-safety call, recorded in `CLAUDE.md` §Redundancy and in `AkkaClusterOptions.SplitBrainResolverStrategy`'s XML doc. One correction to the issue's closing note: it suggested setting this in `Resources/akka.conf`. It is not there — several HOCON fragments compete and the file is the wrong authority. It is emitted by `BuildDowningHocon` and Prepended, and the effective config must be asserted off a running `ActorSystem` (as `SplitBrainResolverActivationTests` does), never off the typed options or the `akka.conf` text.

dohertj2 closed this issue

2026-07-26 10:07:32 -04:00

dohertj2 referenced this issue from a commit

2026-07-26 10:53:53 -04:00

merge: test-correctness, config-secret and AdminUI fixes (#503 #501 #493 #499 #488 #505)

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#492