2026-07-15 10:03:51 -04:00
1 changed files with 44 additions and 0 deletions
@@ -0,0 +1,44 @@
+# #459 — LIVE finding: keep-oldest downs the survivor when the oldest crashes (2-container rig)
+
+**Date:** 2026-07-15 · **Rig:** docker-dev, central-1 (seed/oldest, :4840) + central-2 (:4841), MAIN Warm/2-node.
+
+## Headline
+On a **real 2-container rig**, hard-killing the **oldest** node makes the **younger survivor down ITSELF**
+→ both processes exit → **no failover, total outage**. This CONFIRMS #459 (the premise is real, not stale)
+and **disproves Option A** (keep `keep-oldest` is insufficient). Reproduced twice, deterministically.
+
+## Baseline (both nodes healthy, converged)
+- `Server.ServiceLevel` (NodeId `i=2267`): central-1 (oldest) = **250** (healthy leader), central-2 = **240** (healthy follower). Matches the documented tiering.
+
+## The kill
+`docker kill otopcua-dev-central-1-1` (true process death, not a graceful Leave). central-2'"'"'s SBR:
+
+```
+[WRN] SBR took decision Akka.Cluster.SBR.DownReachable and is downing
+      [akka.tcp://otopcua@central-2:4053] including myself, [1] unreachable of [2] members
+[INF] Cluster Node - Marking node [central-2:4053] as [Down]
+[INF] Self downed, stopping ClusterSingletonManager
+[INF] CoordinatedShutdown invoked due to [ClusterDowningReason]. Exiting with [ExitCode:3].
+```
+
+Result: central-2 Exited (255); `opc.tcp://localhost:4841` → Connection refused. No survivor.
+
+## Why (this is textbook keep-oldest, not a misconfig)
+`keep-oldest` keeps the partition CONTAINING the oldest and downs the other side. When the oldest CRASHES,
+the survivor (younger) is "the side without the oldest" → it downs itself. `down-if-alone = on` does NOT
+rescue it: that knob only makes the OLDEST down itself when the oldest is isolated-but-alive; a crashed
+oldest can'"'"'t execute it, and the younger applies strict keep-oldest → `DownReachable` (self-down). This is
+the documented 2-node keep-oldest limitation; Akka recommends 3+ members / a real quorum.
+
+## Why the in-process test gave false confidence
+`HardKillFailoverTests.Hard_kill_of_oldest_node_fails_over_to_survivor` PASSES, but `HardKillNodeAAsync`
+only calls `provider.Transport.Shutdown()` — node A'"'"'s **ActorSystem stays alive** (a transport partition
+with the oldest still running), NOT a process crash. So the in-process test exercises a different scenario
+than a real `docker kill`. It should be treated as non-representative for the oldest-CRASH case
+(test-quality follow-up: make it kill the process, or reframe what it proves).
+
+## Consequence
+Option A (keep `keep-oldest`) does not provide oldest-crash failover on a real 2-node deployment. The fix
+requires **Option B** (3rd witness/lighthouse member + `keep-majority` — survives any single crash) or
+**Option C** (DB-lease arbiter decoupling Primary from Akka oldest). A 2-node cluster fundamentally cannot
+auto-survive an arbitrary single-node crash under any SBR strategy without a tiebreaker.