feat(cluster): auto-down downing strategy — either-node crash now fails over (owner decision 2026-07-21: availability over partition-safety)
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.
SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.
Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
This commit is contained in:
Joseph Doherty
@@ -48,5 +48,5 @@ Two live items previously tracked ONLY in `archreview/plans/00-MASTER-TRACKER.md
|
||||
|
||||
| # | Item | Where noted | Rationale for deferral | Revisit trigger |
|
||||
|---|------|-------------|------------------------|-----------------|
|
||||
| SBR | **SBR oldest-crash total-outage gap** — 2-node `keep-oldest` downs the partition *without* the oldest, so a hard crash of the ACTIVE (oldest) central node makes the standby self-down (~10s) → total central outage until the crashed node restarts; only a younger-node crash fails over. | `archreview/plans/00-MASTER-TRACKER.md:194` + auto-memory `sbr-keep-oldest-2node-active-crash-gap` | Remedy is a production SBR topology/strategy decision (keep-majority + a 3rd/lighthouse seed node, static-quorum, or an accepted-risk note) — **owner: user decision**, not silently changeable. | Before the next production deployment that adds a central node, or the first real active-node crash. |
|
||||
| SBR | ~~**SBR oldest-crash total-outage gap**~~ **RESOLVED 2026-07-21 (owner decision — availability over partition-safety).** All clusters switched from `keep-oldest` to the `auto-down` downing strategy (Akka `AutoDowning`, `auto-down-unreachable-after` = 15s): a hard crash of EITHER node — active/oldest included — now fails over to the survivor in ~25s with no operator action. Accepted trade: a real network partition produces dual-active until an operator restarts one side. Decision record + evidence (live keep-oldest `DownReachable … including myself` log, Akka.NET 1.5.62 `KeepOldest.OldestDecision` source, rejected alternatives incl. the static-quorum-1 `DownAll` trap): `docs/plans/2026-07-21-auto-down-availability-decision.md`. | `archreview/plans/00-MASTER-TRACKER.md:194` + auto-memory `sbr-keep-oldest-2node-active-crash-gap` (both now historical) | — | Closed. Residual: seed-node boot-alone constraint (unchanged, documented in `Component-ClusterInfrastructure.md`); dual-active recovery is operator-driven. |
|
||||
| vd03 | **`deploy/wonder-app-vd03/` overlay edits unapplied** — `appsettings.Central.json` needs `AllowSingleNodeCluster: true` + phantom-seed removal + `NodeName: central-a`; `install.ps1` needs `sc.exe failure` recovery actions. The `deploy/wonder-app-vd03/` artifact directory is intentionally untracked (production config out of source control), so the repo cannot ship the fix. | `archreview/plans/00-MASTER-TRACKER.md:198` (PLAN-01 T16/T20/T23) | Needs on-host access; without `NodeName` that deployment's audit rows stamp NULL `SourceNode` — **partially mitigated once PLAN-R2-08 Task 7 lands: the host now FAILS AT BOOT with a key-naming error instead of silently NULLing, so applying the overlay becomes mandatory at the next upgrade.** Owner: whoever maintains the host (user). | Next wonder-app-vd03 deployment/upgrade — **the Task 7 validator makes this row unskippable then.** |
|
||||
|
||||
@@ -0,0 +1,108 @@
|
||||
# Auto-Down Downing Strategy — Availability Over Partition-Safety (Decision, 2026-07-21)
|
||||
|
||||
**Status: DECIDED and implemented (owner decision, 2026-07-21).** Resolves the registered
|
||||
deferred "keep-oldest topology/strategy" question (master tracker 2026-07-08;
|
||||
`docs/plans/2026-07-08-deferred-work-register.md` → SBR row).
|
||||
|
||||
## The decision
|
||||
|
||||
All two-node ScadaBridge clusters (central and every site pair) switch their downing
|
||||
strategy from the SBR **keep-oldest** resolver to Akka's **`AutoDowning`** provider
|
||||
(`ClusterOptions.SplitBrainResolverStrategy: "auto-down"`, now the default):
|
||||
|
||||
- `downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"`
|
||||
- `auto-down-unreachable-after` = `ClusterOptions.StableAfter` (15s production)
|
||||
|
||||
The leader among the **reachable** members downs the unreachable peer after the
|
||||
stability window. Consequence: a hard crash of **either** node — the active/oldest
|
||||
included — fails over to the survivor in ~25s (10s failure detection + 15s window),
|
||||
with no operator action and no victim restart required.
|
||||
|
||||
**The owner's stated rationale, verbatim in effect:** the pairs run one node per VM at
|
||||
each site with no Kubernetes and no SQL available site-side, and "network partitions are
|
||||
less of a risk than if this stops working." Availability wins.
|
||||
|
||||
## The accepted trade (read this before debugging a dual-active)
|
||||
|
||||
In a **real network partition** (both nodes alive, link cut) each side downs the other
|
||||
and continues as a one-node cluster: **both run active** — two oldest-Up members, two
|
||||
sets of singletons, `/health/active` = 200 on both. The pre-decision keep-oldest
|
||||
resolver would instead have sacrificed the younger side. Recovery from dual-active is
|
||||
operator-driven: after the partition heals, restart ONE side; the restarted node rejoins
|
||||
its peer as a fresh incarnation and becomes standby. (The two sides do not merge on
|
||||
their own — the mutual downing quarantines the association.)
|
||||
|
||||
## Why the crashed-oldest direction was unsurvivable before (evidence)
|
||||
|
||||
Live drill on the docker rig, 2026-07-21, `keep-oldest` + `down-if-alone = on`
|
||||
(config verified live): killing the active/oldest `central-a` produced, on `central-b`:
|
||||
|
||||
```
|
||||
SBR took decision Akka.Cluster.SBR.DownReachable and is downing
|
||||
[akka.tcp://scadabridge@scadabridge-central-b:8081] including myself,
|
||||
[1] unreachable of [2] members
|
||||
```
|
||||
|
||||
The survivor downed ITSELF, exited (`run-coordinated-shutdown-when-down`), and its
|
||||
restarted incarnation looped on `InitJoin` (non-first-seed cannot self-form) until the
|
||||
victim returned. Root cause in Akka.NET 1.5.62 `KeepOldest.OldestDecision`
|
||||
(`src/core/Akka.Cluster/SBR/DowningStrategy.cs`):
|
||||
|
||||
```csharp
|
||||
// oldest is on the OTHER (unreachable) side:
|
||||
if (DownIfAlone && otherSide == 1 && thisSide >= 2) // survivor side must be >= 2
|
||||
return DownUnreachable.Instance;
|
||||
return DownReachable.Instance; // 1-vs-1 → down MYSELF
|
||||
```
|
||||
|
||||
`down-if-alone` is designed for ≥3-node clusters; with 1-vs-1 it deliberately keeps the
|
||||
oldest side ("the node on the other side is no better" — upstream comment). So two-node
|
||||
keep-oldest can never survive an oldest crash. This corrected an earlier
|
||||
mis-explanation in the repo ("the alone-oldest is dead and cannot down itself").
|
||||
|
||||
## Alternatives rejected
|
||||
|
||||
| Option | Why not |
|
||||
|---|---|
|
||||
| keep-oldest (status quo) | Oldest crash = total outage (proven above). Remains a supported `SplitBrainResolverStrategy` value for deployments preferring partition-safety. |
|
||||
| static-quorum, quorum 1 | Akka's `IsTooManyMembers` guard (`2 > 2*1-1`) returns **DownAll** on any unreachability — total shutdown, strictly worse. |
|
||||
| static-quorum, quorum 2 | Survivor (1 < 2) downs itself on any crash. |
|
||||
| keep-majority | 1-vs-1 tie keeps the lowest-address side — moves the fatal crash from "oldest" to "lowest address", same hole. |
|
||||
| lease-majority | Needs a shared lease store (K8s API, SQL, …) reachable by both nodes — not available at sites. |
|
||||
| third arbiter node | Would make `down-if-alone` work, but there is no third VM at sites. |
|
||||
| custom downing provider | Would reimplement exactly what `AutoDowning` already does, tested upstream. If a future Akka.NET release removes `AutoDowning`, port it then. |
|
||||
|
||||
## What changed (implementation slice, same session)
|
||||
|
||||
- `ClusterOptions`: `SplitBrainResolverStrategy` default → `"auto-down"`; docs rewritten.
|
||||
`DownIfAlone` kept (keep-oldest-only knob, validated only under keep-oldest).
|
||||
- `ClusterOptionsValidator`: allows `auto-down` | `keep-oldest`; `DownIfAlone` requirement
|
||||
scoped to keep-oldest.
|
||||
- `AkkaHostedService.BuildHocon`: downing block branches on the strategy (AutoDowning
|
||||
provider + `auto-down-unreachable-after` vs the SBR block).
|
||||
- All 16 `appsettings` (src Host ×2, `docker/` ×8, `docker-env2/` ×4, and the gitignored
|
||||
`deploy/wonder-app-vd03/` ×2 on-disk overlay) flipped to `auto-down`.
|
||||
**Owner action: sync the wonder-app-vd03 overlay to the host and restart both services
|
||||
together.**
|
||||
- `docker/failover-drill.sh`: `active` mode now asserts the survivor TAKES OVER while
|
||||
the victim is down (previously it asserted the outage).
|
||||
- Tests: HOCON emission (`HoconBuilderTests`), validator/default tests, and two new
|
||||
real-cluster tests in `SbrFailoverTests` — `AutoDown_HardCrashOfOldestNode_
|
||||
YoungerSurvivorTakesOverSingleton` (the direction keep-oldest could never pass) and
|
||||
`AutoDown_HardCrashOfYoungerNode_OldestKeepsSingleton`. `TwoNodeClusterFixture` gained
|
||||
a `strategy` parameter (default `auto-down`).
|
||||
- Docs: `Component-ClusterInfrastructure.md` (Downing Strategy section rewritten),
|
||||
`docker/README.md` (drill docs + results), `CLAUDE.md`, deferred-work register entry
|
||||
resolved.
|
||||
|
||||
## Residual operational notes
|
||||
|
||||
- **Seed-node bootstrap constraint still applies to boot-alone**: only the first seed
|
||||
may self-form a cluster. Auto-down removes the active-crash outage (the survivor never
|
||||
restarts), but a node that must BOOT alone while its peer is dead (cold start of only
|
||||
the non-first-seed VM, or the survivor crashing while the peer is still down) still
|
||||
waits in `InitJoin` for its peer. Operator recovery unchanged (restart first seed, or
|
||||
self-first seed override).
|
||||
- Monitoring already surfaces dual-active if it ever happens: both nodes report
|
||||
`IsActive` in heartbeats / both `/health/active` = 200 — the Health dashboard shows
|
||||
two Primaries.
|
||||
Reference in New Issue
Block a user