ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md

Auto-Down Downing Strategy — Availability Over Partition-Safety (Decision, 2026-07-21)

Status: DECIDED and implemented (owner decision, 2026-07-21). Resolves the registered deferred "keep-oldest topology/strategy" question (master tracker 2026-07-08; docs/plans/2026-07-08-deferred-work-register.md → SBR row).

The decision

All two-node ScadaBridge clusters (central and every site pair) switch their downing strategy from the SBR keep-oldest resolver to Akka's AutoDowning provider (ClusterOptions.SplitBrainResolverStrategy: "auto-down", now the default):

• downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"
• auto-down-unreachable-after = ClusterOptions.StableAfter (15s production)

The leader among the reachable members downs the unreachable peer after the stability window. Consequence: a hard crash of either node — the active/oldest included — fails over to the survivor in ~25s (10s failure detection + 15s window), with no operator action and no victim restart required.

The owner's stated rationale, verbatim in effect: the pairs run one node per VM at each site with no Kubernetes and no SQL available site-side, and "network partitions are less of a risk than if this stops working." Availability wins.

The accepted trade (read this before debugging a dual-active)

In a real network partition (both nodes alive, link cut) each side downs the other and continues as a one-node cluster: both run active — two oldest-Up members, two sets of singletons, /health/active = 200 on both. The pre-decision keep-oldest resolver would instead have sacrificed the younger side. Recovery from dual-active is operator-driven: after the partition heals, restart ONE side; the restarted node rejoins its peer as a fresh incarnation and becomes standby. (The two sides do not merge on their own — the mutual downing quarantines the association.)

Why the crashed-oldest direction was unsurvivable before (evidence)

Live drill on the docker rig, 2026-07-21, keep-oldest + down-if-alone = on (config verified live): killing the active/oldest central-a produced, on central-b:

SBR took decision Akka.Cluster.SBR.DownReachable and is downing
[akka.tcp://scadabridge@scadabridge-central-b:8081] including myself,
[1] unreachable of [2] members

The survivor downed ITSELF, exited (run-coordinated-shutdown-when-down), and its restarted incarnation looped on InitJoin (non-first-seed cannot self-form) until the victim returned. Root cause in Akka.NET 1.5.62 KeepOldest.OldestDecision (src/core/Akka.Cluster/SBR/DowningStrategy.cs):

// oldest is on the OTHER (unreachable) side:
if (DownIfAlone && otherSide == 1 && thisSide >= 2)   // survivor side must be >= 2
    return DownUnreachable.Instance;
return DownReachable.Instance;                         // 1-vs-1 → down MYSELF

down-if-alone is designed for ≥3-node clusters; with 1-vs-1 it deliberately keeps the oldest side ("the node on the other side is no better" — upstream comment). So two-node keep-oldest can never survive an oldest crash. This corrected an earlier mis-explanation in the repo ("the alone-oldest is dead and cannot down itself").

Alternatives rejected

Option	Why not
keep-oldest (status quo)	Oldest crash = total outage (proven above). Remains a supported SplitBrainResolverStrategy value for deployments preferring partition-safety.
static-quorum, quorum 1	Akka's IsTooManyMembers guard (2 > 2*1-1) returns DownAll on any unreachability — total shutdown, strictly worse.
static-quorum, quorum 2	Survivor (1 < 2) downs itself on any crash.
keep-majority	1-vs-1 tie keeps the lowest-address side — moves the fatal crash from "oldest" to "lowest address", same hole.
lease-majority	Needs a shared lease store (K8s API, SQL, …) reachable by both nodes — not available at sites.
third arbiter node	Would make down-if-alone work, but there is no third VM at sites.
custom downing provider	Would reimplement exactly what AutoDowning already does, tested upstream. If a future Akka.NET release removes AutoDowning, port it then.

What changed (implementation slice, same session)

• ClusterOptions: SplitBrainResolverStrategy default → "auto-down"; docs rewritten. DownIfAlone kept (keep-oldest-only knob, validated only under keep-oldest).
• ClusterOptionsValidator: allows auto-down | keep-oldest; DownIfAlone requirement scoped to keep-oldest.
• AkkaHostedService.BuildHocon: downing block branches on the strategy (AutoDowning provider + auto-down-unreachable-after vs the SBR block).
• All 16 appsettings (src Host ×2, docker/ ×8, docker-env2/ ×4, and the gitignored deploy/wonder-app-vd03/ ×2 on-disk overlay) flipped to auto-down. Owner action: sync the wonder-app-vd03 overlay to the host and restart both services together.
• docker/failover-drill.sh: active mode now asserts the survivor TAKES OVER while the victim is down (previously it asserted the outage).
• Tests: HOCON emission (HoconBuilderTests), validator/default tests, and two new real-cluster tests in SbrFailoverTests — AutoDown_HardCrashOfOldestNode_ YoungerSurvivorTakesOverSingleton (the direction keep-oldest could never pass) and AutoDown_HardCrashOfYoungerNode_OldestKeepsSingleton. TwoNodeClusterFixture gained a strategy parameter (default auto-down).
• Docs: Component-ClusterInfrastructure.md (Downing Strategy section rewritten), docker/README.md (drill docs + results), CLAUDE.md, deferred-work register entry resolved.

Residual operational notes

• Seed-node bootstrap constraint still applies to boot-alone: only the first seed may self-form a cluster. Auto-down removes the active-crash outage (the survivor never restarts), but a node that must BOOT alone while its peer is dead (cold start of only the non-first-seed VM, or the survivor crashing while the peer is still down) still waits in InitJoin for its peer. Operator recovery unchanged (restart first seed, or self-first seed override).
• Monitoring already surfaces dual-active if it ever happens: both nodes report IsActive in heartbeats / both /health/active = 200 — the Health dashboard shows two Primaries.