Joseph Doherty
d7470c7960
2-container docker-dev rig: hard-killing the oldest node makes the younger survivor down ITSELF (SBR DownReachable including myself -> CoordinatedShutdown) -> no failover, total outage. Reproduced twice. Confirms #459 is REAL and disproves Option A (keep keep-oldest). Baseline ServiceLevel 250(leader)/240(follower) captured before the kill. Documents why (textbook 2-node keep-oldest limitation; down-if-alone doesn't rescue a crashed oldest) and why the in-process HardKillFailoverTests gave false confidence (Transport.Shutdown keeps the ActorSystem alive, not a real crash). Fix needs Option B (witness+keep-majority) or Option C (DB-lease arbiter).
2.8 KiB
#459 — LIVE finding: keep-oldest downs the survivor when the oldest crashes (2-container rig)
Date: 2026-07-15 · Rig: docker-dev, central-1 (seed/oldest, :4840) + central-2 (:4841), MAIN Warm/2-node.
Headline
On a real 2-container rig, hard-killing the oldest node makes the younger survivor down ITSELF
→ both processes exit → no failover, total outage. This CONFIRMS #459 (the premise is real, not stale)
and disproves Option A (keep keep-oldest is insufficient). Reproduced twice, deterministically.
Baseline (both nodes healthy, converged)
Server.ServiceLevel(NodeIdi=2267): central-1 (oldest) = 250 (healthy leader), central-2 = 240 (healthy follower). Matches the documented tiering.
The kill
docker kill otopcua-dev-central-1-1 (true process death, not a graceful Leave). central-2'"'"'s SBR:
[WRN] SBR took decision Akka.Cluster.SBR.DownReachable and is downing
[akka.tcp://otopcua@central-2:4053] including myself, [1] unreachable of [2] members
[INF] Cluster Node - Marking node [central-2:4053] as [Down]
[INF] Self downed, stopping ClusterSingletonManager
[INF] CoordinatedShutdown invoked due to [ClusterDowningReason]. Exiting with [ExitCode:3].
Result: central-2 Exited (255); opc.tcp://localhost:4841 → Connection refused. No survivor.
Why (this is textbook keep-oldest, not a misconfig)
keep-oldest keeps the partition CONTAINING the oldest and downs the other side. When the oldest CRASHES,
the survivor (younger) is "the side without the oldest" → it downs itself. down-if-alone = on does NOT
rescue it: that knob only makes the OLDEST down itself when the oldest is isolated-but-alive; a crashed
oldest can'"'"'t execute it, and the younger applies strict keep-oldest → DownReachable (self-down). This is
the documented 2-node keep-oldest limitation; Akka recommends 3+ members / a real quorum.
Why the in-process test gave false confidence
HardKillFailoverTests.Hard_kill_of_oldest_node_fails_over_to_survivor PASSES, but HardKillNodeAAsync
only calls provider.Transport.Shutdown() — node A'"'"'s ActorSystem stays alive (a transport partition
with the oldest still running), NOT a process crash. So the in-process test exercises a different scenario
than a real docker kill. It should be treated as non-representative for the oldest-CRASH case
(test-quality follow-up: make it kill the process, or reframe what it proves).
Consequence
Option A (keep keep-oldest) does not provide oldest-crash failover on a real 2-node deployment. The fix
requires Option B (3rd witness/lighthouse member + keep-majority — survives any single crash) or
Option C (DB-lease arbiter decoupling Primary from Akka oldest). A 2-node cluster fundamentally cannot
auto-survive an arbitrary single-node crash under any SBR strategy without a tiebreaker.