ScadaBridge/docs/plans/2026-07-21-auto-down-availability-decision.md

# Auto-Down Downing Strategy — Availability Over Partition-Safety (Decision, 2026-07-21)

**Status: DECIDED and implemented (owner decision, 2026-07-21).** Resolves the registered
deferred "keep-oldest topology/strategy" question (master tracker 2026-07-08;
`docs/plans/2026-07-08-deferred-work-register.md` → SBR row).

## The decision

All two-node ScadaBridge clusters (central and every site pair) switch their downing
strategy from the SBR **keep-oldest** resolver to Akka's **`AutoDowning`** provider
(`ClusterOptions.SplitBrainResolverStrategy: "auto-down"`, now the default):

- `downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"`
- `auto-down-unreachable-after` = `ClusterOptions.StableAfter` (15s production)

The leader among the **reachable** members downs the unreachable peer after the
stability window. Consequence: a hard crash of **either** node — the active/oldest
included — fails over to the survivor in ~25s (10s failure detection + 15s window),
with no operator action and no victim restart required.

**The owner's stated rationale, verbatim in effect:** the pairs run one node per VM at
each site with no Kubernetes and no SQL available site-side, and "network partitions are
less of a risk than if this stops working." Availability wins.

## The accepted trade (read this before debugging a dual-active)

In a **real network partition** (both nodes alive, link cut) each side downs the other
and continues as a one-node cluster: **both run active** — two oldest-Up members, two
sets of singletons, `/health/active` = 200 on both. The pre-decision keep-oldest
resolver would instead have sacrificed the younger side. Recovery from dual-active is
operator-driven: after the partition heals, restart ONE side; the restarted node rejoins
its peer as a fresh incarnation and becomes standby. (The two sides do not merge on
their own — the mutual downing quarantines the association.)

## Why the crashed-oldest direction was unsurvivable before (evidence)

Live drill on the docker rig, 2026-07-21, `keep-oldest` + `down-if-alone = on`
(config verified live): killing the active/oldest `central-a` produced, on `central-b`:

```
SBR took decision Akka.Cluster.SBR.DownReachable and is downing
[akka.tcp://scadabridge@scadabridge-central-b:8081] including myself,
[1] unreachable of [2] members
```

The survivor downed ITSELF, exited (`run-coordinated-shutdown-when-down`), and its
restarted incarnation looped on `InitJoin` (non-first-seed cannot self-form) until the
victim returned. Root cause in Akka.NET 1.5.62 `KeepOldest.OldestDecision`
(`src/core/Akka.Cluster/SBR/DowningStrategy.cs`):

```csharp
// oldest is on the OTHER (unreachable) side:
if (DownIfAlone && otherSide == 1 && thisSide >= 2)   // survivor side must be >= 2
    return DownUnreachable.Instance;
return DownReachable.Instance;                         // 1-vs-1 → down MYSELF
```

`down-if-alone` is designed for ≥3-node clusters; with 1-vs-1 it deliberately keeps the
oldest side ("the node on the other side is no better" — upstream comment). So two-node
keep-oldest can never survive an oldest crash. This corrected an earlier
mis-explanation in the repo ("the alone-oldest is dead and cannot down itself").

## Alternatives rejected

| Option | Why not |
|---|---|
| keep-oldest (status quo) | Oldest crash = total outage (proven above). Remains a supported `SplitBrainResolverStrategy` value for deployments preferring partition-safety. |
| static-quorum, quorum 1 | Akka's `IsTooManyMembers` guard (`2 > 2*1-1`) returns **DownAll** on any unreachability — total shutdown, strictly worse. |
| static-quorum, quorum 2 | Survivor (1 < 2) downs itself on any crash. |
| keep-majority | 1-vs-1 tie keeps the lowest-address side — moves the fatal crash from "oldest" to "lowest address", same hole. |
| lease-majority | Needs a shared lease store (K8s API, SQL, …) reachable by both nodes — not available at sites. |
| third arbiter node | Would make `down-if-alone` work, but there is no third VM at sites. |
| custom downing provider | Would reimplement exactly what `AutoDowning` already does, tested upstream. If a future Akka.NET release removes `AutoDowning`, port it then. |

## What changed (implementation slice, same session)

- `ClusterOptions`: `SplitBrainResolverStrategy` default → `"auto-down"`; docs rewritten.
  `DownIfAlone` kept (keep-oldest-only knob, validated only under keep-oldest).
- `ClusterOptionsValidator`: allows `auto-down` | `keep-oldest`; `DownIfAlone` requirement
  scoped to keep-oldest.
- `AkkaHostedService.BuildHocon`: downing block branches on the strategy (AutoDowning
  provider + `auto-down-unreachable-after` vs the SBR block).
- All 16 `appsettings` (src Host ×2, `docker/` ×8, `docker-env2/` ×4, and the gitignored
  `deploy/wonder-app-vd03/` ×2 on-disk overlay) flipped to `auto-down`.
  **Owner action: sync the wonder-app-vd03 overlay to the host and restart both services
  together.**
- `docker/failover-drill.sh`: `active` mode now asserts the survivor TAKES OVER while
  the victim is down (previously it asserted the outage).
- Tests: HOCON emission (`HoconBuilderTests`), validator/default tests, and two new
  real-cluster tests in `SbrFailoverTests` — `AutoDown_HardCrashOfOldestNode_
  YoungerSurvivorTakesOverSingleton` (the direction keep-oldest could never pass) and
  `AutoDown_HardCrashOfYoungerNode_OldestKeepsSingleton`. `TwoNodeClusterFixture` gained
  a `strategy` parameter (default `auto-down`).
- Docs: `Component-ClusterInfrastructure.md` (Downing Strategy section rewritten),
  `docker/README.md` (drill docs + results), `CLAUDE.md`, deferred-work register entry
  resolved.

## Residual operational notes

- ~~**Seed-node bootstrap constraint still applies to boot-alone**~~ — **SUPERSEDED
  (2026-07-22), no longer a residual.** As written: only the first seed may self-form a
  cluster, so a node that must BOOT alone while its peer is dead (cold start of only the
  non-first-seed VM, or the survivor crashing while the peer is still down) waits in
  `InitJoin` for its peer. This was **closed** by the self-first seed-node ordering change
  (`docs/plans/2026-07-22-selfform-fallback-and-manual-failover.md`): every node now lists
  ITSELF first in `seed-nodes`, so any node can cold-start alone and become operational
  unattended, and `StartupValidator` fails the boot if the ordering is broken — regression
  coverage in `SelfFirstSeedBootstrapTests`. The residual cost of self-first-on-both (a
  *truly simultaneous* cold start racing `FirstSeedNodeProcess` on both nodes) is in turn
  covered by the opt-in `ScadaBridge:Cluster:BootstrapGuard` (Gitea #33, default OFF).
- Monitoring already surfaces dual-active if it ever happens: both nodes report
  `IsActive` in heartbeats / both `/health/active` = 200 — the Health dashboard shows
  two Primaries.

## Postscript 2026-08-15 — the deprecation warning is benign (drill-verified), but pins us for Akka bumps

Every node logs at Cluster start: *"The `auto-down-unreachable-after` feature has been
deprecated as of Akka.NET v1.5.2 … The `keep-majority` split brain resolver will be used
instead."* Investigated and drill-verified on 1.5.62 (the same version this decision was
made and drilled on): the warning fires purely on the **presence of the key** and the
substitution does **NOT** happen when `downing-provider-class = "Akka.Cluster.AutoDowning"`
is set explicitly, as `BuildHocon` does. Proof by discriminating drill (`DRILL_MODE=active`,
2026-08-15): killing central-a — oldest AND lowest-address — left central-b to take over
(OldestChanged → all singletons moved, victim rejoined ready 2s after restart). Under
keep-majority, a 2-node even split keeps the lowest-address side, so central-b would have
downed ITSELF; it demonstrably did not. The decision stands exactly as written.

The real exposure is forward-looking: `AutoDowning` is deprecated and can be REMOVED by a
future Akka.NET release. **Any Akka version bump must re-run `DRILL_MODE=active
bash docker/failover-drill.sh` and re-read the release notes before being accepted**
(deferred-work register row 70).