Auto-Down Downing Strategy — Availability Over Partition-Safety (Decision, 2026-07-21)

Status: DECIDED and implemented (owner decision, 2026-07-21). Resolves the registered deferred "keep-oldest topology/strategy" question (master tracker 2026-07-08; docs/plans/2026-07-08-deferred-work-register.md → SBR row).

The decision

All two-node ScadaBridge clusters (central and every site pair) switch their downing strategy from the SBR keep-oldest resolver to Akka's AutoDowning provider (ClusterOptions.SplitBrainResolverStrategy: "auto-down", now the default):

downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"
auto-down-unreachable-after = ClusterOptions.StableAfter (15s production)

The leader among the reachable members downs the unreachable peer after the stability window. Consequence: a hard crash of either node — the active/oldest included — fails over to the survivor in ~25s (10s failure detection + 15s window), with no operator action and no victim restart required.

The owner's stated rationale, verbatim in effect: the pairs run one node per VM at each site with no Kubernetes and no SQL available site-side, and "network partitions are less of a risk than if this stops working." Availability wins.

The accepted trade (read this before debugging a dual-active)

In a real network partition (both nodes alive, link cut) each side downs the other and continues as a one-node cluster: both run active — two oldest-Up members, two sets of singletons, /health/active = 200 on both. The pre-decision keep-oldest resolver would instead have sacrificed the younger side. Recovery from dual-active is operator-driven: after the partition heals, restart ONE side; the restarted node rejoins its peer as a fresh incarnation and becomes standby. (The two sides do not merge on their own — the mutual downing quarantines the association.)

Why the crashed-oldest direction was unsurvivable before (evidence)

Live drill on the docker rig, 2026-07-21, keep-oldest + down-if-alone = on (config verified live): killing the active/oldest central-a produced, on central-b:

SBR took decision Akka.Cluster.SBR.DownReachable and is downing
[akka.tcp://scadabridge@scadabridge-central-b:8081] including myself,
[1] unreachable of [2] members

The survivor downed ITSELF, exited (run-coordinated-shutdown-when-down), and its restarted incarnation looped on InitJoin (non-first-seed cannot self-form) until the victim returned. Root cause in Akka.NET 1.5.62 KeepOldest.OldestDecision (src/core/Akka.Cluster/SBR/DowningStrategy.cs):

// oldest is on the OTHER (unreachable) side:
if (DownIfAlone && otherSide == 1 && thisSide >= 2)   // survivor side must be >= 2
    return DownUnreachable.Instance;
return DownReachable.Instance;                         // 1-vs-1 → down MYSELF

down-if-alone is designed for ≥3-node clusters; with 1-vs-1 it deliberately keeps the oldest side ("the node on the other side is no better" — upstream comment). So two-node keep-oldest can never survive an oldest crash. This corrected an earlier mis-explanation in the repo ("the alone-oldest is dead and cannot down itself").

Alternatives rejected

Option	Why not
keep-oldest (status quo)	Oldest crash = total outage (proven above). Remains a supported `SplitBrainResolverStrategy` value for deployments preferring partition-safety.
static-quorum, quorum 1	Akka's `IsTooManyMembers` guard (`2 > 21-1`) returns DownAll* on any unreachability — total shutdown, strictly worse.
static-quorum, quorum 2	Survivor (1 < 2) downs itself on any crash.
keep-majority	1-vs-1 tie keeps the lowest-address side — moves the fatal crash from "oldest" to "lowest address", same hole.
lease-majority	Needs a shared lease store (K8s API, SQL, …) reachable by both nodes — not available at sites.
third arbiter node	Would make `down-if-alone` work, but there is no third VM at sites.
custom downing provider	Would reimplement exactly what `AutoDowning` already does, tested upstream. If a future Akka.NET release removes `AutoDowning`, port it then.

What changed (implementation slice, same session)

ClusterOptions: SplitBrainResolverStrategy default → "auto-down"; docs rewritten. DownIfAlone kept (keep-oldest-only knob, validated only under keep-oldest).
ClusterOptionsValidator: allows auto-down | keep-oldest; DownIfAlone requirement scoped to keep-oldest.
AkkaHostedService.BuildHocon: downing block branches on the strategy (AutoDowning provider + auto-down-unreachable-after vs the SBR block).
All 16 appsettings (src Host ×2, docker/ ×8, docker-env2/ ×4, and the gitignored deploy/wonder-app-vd03/ ×2 on-disk overlay) flipped to auto-down. Owner action: sync the wonder-app-vd03 overlay to the host and restart both services together.
docker/failover-drill.sh: active mode now asserts the survivor TAKES OVER while the victim is down (previously it asserted the outage).
Tests: HOCON emission (HoconBuilderTests), validator/default tests, and two new real-cluster tests in SbrFailoverTests — AutoDown_HardCrashOfOldestNode_ YoungerSurvivorTakesOverSingleton (the direction keep-oldest could never pass) and AutoDown_HardCrashOfYoungerNode_OldestKeepsSingleton. TwoNodeClusterFixture gained a strategy parameter (default auto-down).
Docs: Component-ClusterInfrastructure.md (Downing Strategy section rewritten), docker/README.md (drill docs + results), CLAUDE.md, deferred-work register entry resolved.

Residual operational notes

~~Seed-node bootstrap constraint still applies to boot-alone~~ — SUPERSEDED (2026-07-22), no longer a residual. As written: only the first seed may self-form a cluster, so a node that must BOOT alone while its peer is dead (cold start of only the non-first-seed VM, or the survivor crashing while the peer is still down) waits in InitJoin for its peer. This was closed by the self-first seed-node ordering change (docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md): every node now lists ITSELF first in seed-nodes, so any node can cold-start alone and become operational unattended, and StartupValidator fails the boot if the ordering is broken — regression coverage in SelfFirstSeedBootstrapTests. The residual cost of self-first-on-both (a truly simultaneous cold start racing FirstSeedNodeProcess on both nodes) is in turn covered by the opt-in ScadaBridge:Cluster:BootstrapGuard (Gitea #33, default OFF).
Monitoring already surfaces dual-active if it ever happens: both nodes report IsActive in heartbeats / both /health/active = 200 — the Health dashboard shows two Primaries.

Postscript 2026-08-15 — the deprecation warning is benign (drill-verified), but pins us for Akka bumps

Every node logs at Cluster start: "The auto-down-unreachable-after feature has been deprecated as of Akka.NET v1.5.2 … The keep-majority split brain resolver will be used instead." Investigated and drill-verified on 1.5.62 (the same version this decision was made and drilled on): the warning fires purely on the presence of the key and the substitution does NOT happen when downing-provider-class = "Akka.Cluster.AutoDowning" is set explicitly, as BuildHocon does. Proof by discriminating drill (DRILL_MODE=active, 2026-08-15): killing central-a — oldest AND lowest-address — left central-b to take over (OldestChanged → all singletons moved, victim rejoined ready 2s after restart). Under keep-majority, a 2-node even split keeps the lowest-address side, so central-b would have downed ITSELF; it demonstrably did not. The decision stands exactly as written.

The real exposure is forward-looking: AutoDowning is deprecated and can be REMOVED by a future Akka.NET release. Any Akka version bump must re-run DRILL_MODE=active bash docker/failover-drill.sh and re-read the release notes before being accepted (deferred-work register row 70).

8.0 KiB Raw Blame History Unescape Escape