diff --git a/archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md b/archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md new file mode 100644 index 00000000..585e76e4 --- /dev/null +++ b/archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md @@ -0,0 +1,44 @@ +# #459 — LIVE finding: keep-oldest downs the survivor when the oldest crashes (2-container rig) + +**Date:** 2026-07-15 · **Rig:** docker-dev, central-1 (seed/oldest, :4840) + central-2 (:4841), MAIN Warm/2-node. + +## Headline +On a **real 2-container rig**, hard-killing the **oldest** node makes the **younger survivor down ITSELF** +→ both processes exit → **no failover, total outage**. This CONFIRMS #459 (the premise is real, not stale) +and **disproves Option A** (keep `keep-oldest` is insufficient). Reproduced twice, deterministically. + +## Baseline (both nodes healthy, converged) +- `Server.ServiceLevel` (NodeId `i=2267`): central-1 (oldest) = **250** (healthy leader), central-2 = **240** (healthy follower). Matches the documented tiering. + +## The kill +`docker kill otopcua-dev-central-1-1` (true process death, not a graceful Leave). central-2'"'"'s SBR: + +``` +[WRN] SBR took decision Akka.Cluster.SBR.DownReachable and is downing + [akka.tcp://otopcua@central-2:4053] including myself, [1] unreachable of [2] members +[INF] Cluster Node - Marking node [central-2:4053] as [Down] +[INF] Self downed, stopping ClusterSingletonManager +[INF] CoordinatedShutdown invoked due to [ClusterDowningReason]. Exiting with [ExitCode:3]. +``` + +Result: central-2 Exited (255); `opc.tcp://localhost:4841` → Connection refused. No survivor. + +## Why (this is textbook keep-oldest, not a misconfig) +`keep-oldest` keeps the partition CONTAINING the oldest and downs the other side. When the oldest CRASHES, +the survivor (younger) is "the side without the oldest" → it downs itself. `down-if-alone = on` does NOT +rescue it: that knob only makes the OLDEST down itself when the oldest is isolated-but-alive; a crashed +oldest can'"'"'t execute it, and the younger applies strict keep-oldest → `DownReachable` (self-down). This is +the documented 2-node keep-oldest limitation; Akka recommends 3+ members / a real quorum. + +## Why the in-process test gave false confidence +`HardKillFailoverTests.Hard_kill_of_oldest_node_fails_over_to_survivor` PASSES, but `HardKillNodeAAsync` +only calls `provider.Transport.Shutdown()` — node A'"'"'s **ActorSystem stays alive** (a transport partition +with the oldest still running), NOT a process crash. So the in-process test exercises a different scenario +than a real `docker kill`. It should be treated as non-representative for the oldest-CRASH case +(test-quality follow-up: make it kill the process, or reframe what it proves). + +## Consequence +Option A (keep `keep-oldest`) does not provide oldest-crash failover on a real 2-node deployment. The fix +requires **Option B** (3rd witness/lighthouse member + `keep-majority` — survives any single crash) or +**Option C** (DB-lease arbiter decoupling Primary from Akka oldest). A 2-node cluster fundamentally cannot +auto-survive an arbitrary single-node crash under any SBR strategy without a tiebreaker.