Joseph Doherty
cf3bd52f93
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.
SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.
Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
109 lines
6.2 KiB
Markdown
109 lines
6.2 KiB
Markdown
# Auto-Down Downing Strategy — Availability Over Partition-Safety (Decision, 2026-07-21)
|
||
|
||
**Status: DECIDED and implemented (owner decision, 2026-07-21).** Resolves the registered
|
||
deferred "keep-oldest topology/strategy" question (master tracker 2026-07-08;
|
||
`docs/plans/2026-07-08-deferred-work-register.md` → SBR row).
|
||
|
||
## The decision
|
||
|
||
All two-node ScadaBridge clusters (central and every site pair) switch their downing
|
||
strategy from the SBR **keep-oldest** resolver to Akka's **`AutoDowning`** provider
|
||
(`ClusterOptions.SplitBrainResolverStrategy: "auto-down"`, now the default):
|
||
|
||
- `downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"`
|
||
- `auto-down-unreachable-after` = `ClusterOptions.StableAfter` (15s production)
|
||
|
||
The leader among the **reachable** members downs the unreachable peer after the
|
||
stability window. Consequence: a hard crash of **either** node — the active/oldest
|
||
included — fails over to the survivor in ~25s (10s failure detection + 15s window),
|
||
with no operator action and no victim restart required.
|
||
|
||
**The owner's stated rationale, verbatim in effect:** the pairs run one node per VM at
|
||
each site with no Kubernetes and no SQL available site-side, and "network partitions are
|
||
less of a risk than if this stops working." Availability wins.
|
||
|
||
## The accepted trade (read this before debugging a dual-active)
|
||
|
||
In a **real network partition** (both nodes alive, link cut) each side downs the other
|
||
and continues as a one-node cluster: **both run active** — two oldest-Up members, two
|
||
sets of singletons, `/health/active` = 200 on both. The pre-decision keep-oldest
|
||
resolver would instead have sacrificed the younger side. Recovery from dual-active is
|
||
operator-driven: after the partition heals, restart ONE side; the restarted node rejoins
|
||
its peer as a fresh incarnation and becomes standby. (The two sides do not merge on
|
||
their own — the mutual downing quarantines the association.)
|
||
|
||
## Why the crashed-oldest direction was unsurvivable before (evidence)
|
||
|
||
Live drill on the docker rig, 2026-07-21, `keep-oldest` + `down-if-alone = on`
|
||
(config verified live): killing the active/oldest `central-a` produced, on `central-b`:
|
||
|
||
```
|
||
SBR took decision Akka.Cluster.SBR.DownReachable and is downing
|
||
[akka.tcp://scadabridge@scadabridge-central-b:8081] including myself,
|
||
[1] unreachable of [2] members
|
||
```
|
||
|
||
The survivor downed ITSELF, exited (`run-coordinated-shutdown-when-down`), and its
|
||
restarted incarnation looped on `InitJoin` (non-first-seed cannot self-form) until the
|
||
victim returned. Root cause in Akka.NET 1.5.62 `KeepOldest.OldestDecision`
|
||
(`src/core/Akka.Cluster/SBR/DowningStrategy.cs`):
|
||
|
||
```csharp
|
||
// oldest is on the OTHER (unreachable) side:
|
||
if (DownIfAlone && otherSide == 1 && thisSide >= 2) // survivor side must be >= 2
|
||
return DownUnreachable.Instance;
|
||
return DownReachable.Instance; // 1-vs-1 → down MYSELF
|
||
```
|
||
|
||
`down-if-alone` is designed for ≥3-node clusters; with 1-vs-1 it deliberately keeps the
|
||
oldest side ("the node on the other side is no better" — upstream comment). So two-node
|
||
keep-oldest can never survive an oldest crash. This corrected an earlier
|
||
mis-explanation in the repo ("the alone-oldest is dead and cannot down itself").
|
||
|
||
## Alternatives rejected
|
||
|
||
| Option | Why not |
|
||
|---|---|
|
||
| keep-oldest (status quo) | Oldest crash = total outage (proven above). Remains a supported `SplitBrainResolverStrategy` value for deployments preferring partition-safety. |
|
||
| static-quorum, quorum 1 | Akka's `IsTooManyMembers` guard (`2 > 2*1-1`) returns **DownAll** on any unreachability — total shutdown, strictly worse. |
|
||
| static-quorum, quorum 2 | Survivor (1 < 2) downs itself on any crash. |
|
||
| keep-majority | 1-vs-1 tie keeps the lowest-address side — moves the fatal crash from "oldest" to "lowest address", same hole. |
|
||
| lease-majority | Needs a shared lease store (K8s API, SQL, …) reachable by both nodes — not available at sites. |
|
||
| third arbiter node | Would make `down-if-alone` work, but there is no third VM at sites. |
|
||
| custom downing provider | Would reimplement exactly what `AutoDowning` already does, tested upstream. If a future Akka.NET release removes `AutoDowning`, port it then. |
|
||
|
||
## What changed (implementation slice, same session)
|
||
|
||
- `ClusterOptions`: `SplitBrainResolverStrategy` default → `"auto-down"`; docs rewritten.
|
||
`DownIfAlone` kept (keep-oldest-only knob, validated only under keep-oldest).
|
||
- `ClusterOptionsValidator`: allows `auto-down` | `keep-oldest`; `DownIfAlone` requirement
|
||
scoped to keep-oldest.
|
||
- `AkkaHostedService.BuildHocon`: downing block branches on the strategy (AutoDowning
|
||
provider + `auto-down-unreachable-after` vs the SBR block).
|
||
- All 16 `appsettings` (src Host ×2, `docker/` ×8, `docker-env2/` ×4, and the gitignored
|
||
`deploy/wonder-app-vd03/` ×2 on-disk overlay) flipped to `auto-down`.
|
||
**Owner action: sync the wonder-app-vd03 overlay to the host and restart both services
|
||
together.**
|
||
- `docker/failover-drill.sh`: `active` mode now asserts the survivor TAKES OVER while
|
||
the victim is down (previously it asserted the outage).
|
||
- Tests: HOCON emission (`HoconBuilderTests`), validator/default tests, and two new
|
||
real-cluster tests in `SbrFailoverTests` — `AutoDown_HardCrashOfOldestNode_
|
||
YoungerSurvivorTakesOverSingleton` (the direction keep-oldest could never pass) and
|
||
`AutoDown_HardCrashOfYoungerNode_OldestKeepsSingleton`. `TwoNodeClusterFixture` gained
|
||
a `strategy` parameter (default `auto-down`).
|
||
- Docs: `Component-ClusterInfrastructure.md` (Downing Strategy section rewritten),
|
||
`docker/README.md` (drill docs + results), `CLAUDE.md`, deferred-work register entry
|
||
resolved.
|
||
|
||
## Residual operational notes
|
||
|
||
- **Seed-node bootstrap constraint still applies to boot-alone**: only the first seed
|
||
may self-form a cluster. Auto-down removes the active-crash outage (the survivor never
|
||
restarts), but a node that must BOOT alone while its peer is dead (cold start of only
|
||
the non-first-seed VM, or the survivor crashing while the peer is still down) still
|
||
waits in `InitJoin` for its peer. Operator recovery unchanged (restart first seed, or
|
||
self-first seed override).
|
||
- Monitoring already surfaces dual-active if it ever happens: both nodes report
|
||
`IsActive` in heartbeats / both `/health/active` = 200 — the Health dashboard shows
|
||
two Primaries.
|