# Auto-Down Downing Strategy — Availability Over Partition-Safety (Decision, 2026-07-21)

**Status: DECIDED and implemented (owner decision, 2026-07-21).** Resolves the registered
deferred "keep-oldest topology/strategy" question (master tracker 2026-07-08;
`docs/plans/2026-07-08-deferred-work-register.md` → SBR row).

## The decision

All two-node ScadaBridge clusters (central and every site pair) switch their downing
strategy from the SBR **keep-oldest** resolver to Akka's **`AutoDowning`** provider
(`ClusterOptions.SplitBrainResolverStrategy: "auto-down"`, now the default):

- `downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"`
- `auto-down-unreachable-after` = `ClusterOptions.StableAfter` (15s production)

The leader among the **reachable** members downs the unreachable peer after the
stability window. Consequence: a hard crash of **either** node — the active/oldest
included — fails over to the survivor in ~25s (10s failure detection + 15s window),
with no operator action and no victim restart required.

**The owner's stated rationale, verbatim in effect:** the pairs run one node per VM at
each site with no Kubernetes and no SQL available site-side, and "network partitions are
less of a risk than if this stops working." Availability wins.

## The accepted trade (read this before debugging a dual-active)

In a **real network partition** (both nodes alive, link cut) each side downs the other
and continues as a one-node cluster: **both run active** — two oldest-Up members, two
sets of singletons, `/health/active` = 200 on both. The pre-decision keep-oldest
resolver would instead have sacrificed the younger side. Recovery from dual-active is
operator-driven: after the partition heals, restart ONE side; the restarted node rejoins
its peer as a fresh incarnation and becomes standby. (The two sides do not merge on
their own — the mutual downing quarantines the association.)

## Why the crashed-oldest direction was unsurvivable before (evidence)

Live drill on the docker rig, 2026-07-21, `keep-oldest` + `down-if-alone = on`
(config verified live): killing the active/oldest `central-a` produced, on `central-b`:

```
SBR took decision Akka.Cluster.SBR.DownReachable and is downing
[akka.tcp://scadabridge@scadabridge-central-b:8081] including myself,
[1] unreachable of [2] members
```

The survivor downed ITSELF, exited (`run-coordinated-shutdown-when-down`), and its
restarted incarnation looped on `InitJoin` (non-first-seed cannot self-form) until the
victim returned. Root cause in Akka.NET 1.5.62 `KeepOldest.OldestDecision`
(`src/core/Akka.Cluster/SBR/DowningStrategy.cs`):

```csharp
// oldest is on the OTHER (unreachable) side:
if (DownIfAlone && otherSide == 1 && thisSide >= 2)   // survivor side must be >= 2
    return DownUnreachable.Instance;
return DownReachable.Instance;                         // 1-vs-1 → down MYSELF
```

`down-if-alone` is designed for ≥3-node clusters; with 1-vs-1 it deliberately keeps the
oldest side ("the node on the other side is no better" — upstream comment). So two-node
keep-oldest can never survive an oldest crash. This corrected an earlier
mis-explanation in the repo ("the alone-oldest is dead and cannot down itself").

## Alternatives rejected

| Option | Why not |
|---|---|
| keep-oldest (status quo) | Oldest crash = total outage (proven above). Remains a supported `SplitBrainResolverStrategy` value for deployments preferring partition-safety. |
| static-quorum, quorum 1 | Akka's `IsTooManyMembers` guard (`2 > 2*1-1`) returns **DownAll** on any unreachability — total shutdown, strictly worse. |
| static-quorum, quorum 2 | Survivor (1 < 2) downs itself on any crash. |
| keep-majority | 1-vs-1 tie keeps the lowest-address side — moves the fatal crash from "oldest" to "lowest address", same hole. |
| lease-majority | Needs a shared lease store (K8s API, SQL, …) reachable by both nodes — not available at sites. |
| third arbiter node | Would make `down-if-alone` work, but there is no third VM at sites. |
| custom downing provider | Would reimplement exactly what `AutoDowning` already does, tested upstream. If a future Akka.NET release removes `AutoDowning`, port it then. |

## What changed (implementation slice, same session)

- `ClusterOptions`: `SplitBrainResolverStrategy` default → `"auto-down"`; docs rewritten.
  `DownIfAlone` kept (keep-oldest-only knob, validated only under keep-oldest).
- `ClusterOptionsValidator`: allows `auto-down` | `keep-oldest`; `DownIfAlone` requirement
  scoped to keep-oldest.
- `AkkaHostedService.BuildHocon`: downing block branches on the strategy (AutoDowning
  provider + `auto-down-unreachable-after` vs the SBR block).
- All 16 `appsettings` (src Host ×2, `docker/` ×8, `docker-env2/` ×4, and the gitignored
  `deploy/wonder-app-vd03/` ×2 on-disk overlay) flipped to `auto-down`.
  **Owner action: sync the wonder-app-vd03 overlay to the host and restart both services
  together.**
- `docker/failover-drill.sh`: `active` mode now asserts the survivor TAKES OVER while
  the victim is down (previously it asserted the outage).
- Tests: HOCON emission (`HoconBuilderTests`), validator/default tests, and two new
  real-cluster tests in `SbrFailoverTests` — `AutoDown_HardCrashOfOldestNode_
  YoungerSurvivorTakesOverSingleton` (the direction keep-oldest could never pass) and
  `AutoDown_HardCrashOfYoungerNode_OldestKeepsSingleton`. `TwoNodeClusterFixture` gained
  a `strategy` parameter (default `auto-down`).
- Docs: `Component-ClusterInfrastructure.md` (Downing Strategy section rewritten),
  `docker/README.md` (drill docs + results), `CLAUDE.md`, deferred-work register entry
  resolved.

## Residual operational notes

- **Seed-node bootstrap constraint still applies to boot-alone**: only the first seed
  may self-form a cluster. Auto-down removes the active-crash outage (the survivor never
  restarts), but a node that must BOOT alone while its peer is dead (cold start of only
  the non-first-seed VM, or the survivor crashing while the peer is still down) still
  waits in `InitJoin` for its peer. Operator recovery unchanged (restart first seed, or
  self-first seed override).
- Monitoring already surfaces dual-active if it ever happens: both nodes report
  `IsActive` in heartbeats / both `/health/active` = 200 — the Health dashboard shows
  two Primaries.