From d7470c7960a4897a319fd279ac3775164faf8ac7 Mon Sep 17 00:00:00 2001
From: Joseph Doherty <dohejw01@gmail.com>
Date: Wed, 15 Jul 2026 10:03:31 -0400
Subject: [PATCH] =?UTF-8?q?docs(archreview):=20live=20finding=20=E2=80=94?=
 =?UTF-8?q?=20keep-oldest=20downs=20the=20survivor=20on=20oldest-crash=20(?=
 =?UTF-8?q?#459)?=
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

2-container docker-dev rig: hard-killing the oldest node makes the younger
survivor down ITSELF (SBR DownReachable including myself -> CoordinatedShutdown)
-> no failover, total outage. Reproduced twice. Confirms #459 is REAL and
disproves Option A (keep keep-oldest). Baseline ServiceLevel 250(leader)/240(follower)
captured before the kill. Documents why (textbook 2-node keep-oldest limitation;
down-if-alone doesn't rescue a crashed oldest) and why the in-process
HardKillFailoverTests gave false confidence (Transport.Shutdown keeps the
ActorSystem alive, not a real crash). Fix needs Option B (witness+keep-majority)
or Option C (DB-lease arbiter).
---
 ...59-oldest-crash-live-finding-2026-07-15.md | 44 +++++++++++++++++++
 1 file changed, 44 insertions(+)
 create mode 100644 archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md

diff --git a/archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md b/archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md
new file mode 100644
index 00000000..585e76e4
--- /dev/null
+++ b/archreview/plans/artifacts/459-oldest-crash-live-finding-2026-07-15.md
@@ -0,0 +1,44 @@
+# #459 — LIVE finding: keep-oldest downs the survivor when the oldest crashes (2-container rig)
+
+**Date:** 2026-07-15 · **Rig:** docker-dev, central-1 (seed/oldest, :4840) + central-2 (:4841), MAIN Warm/2-node.
+
+## Headline
+On a **real 2-container rig**, hard-killing the **oldest** node makes the **younger survivor down ITSELF**
+→ both processes exit → **no failover, total outage**. This CONFIRMS #459 (the premise is real, not stale)
+and **disproves Option A** (keep `keep-oldest` is insufficient). Reproduced twice, deterministically.
+
+## Baseline (both nodes healthy, converged)
+- `Server.ServiceLevel` (NodeId `i=2267`): central-1 (oldest) = **250** (healthy leader), central-2 = **240** (healthy follower). Matches the documented tiering.
+
+## The kill
+`docker kill otopcua-dev-central-1-1` (true process death, not a graceful Leave). central-2'"'"'s SBR:
+
+```
+[WRN] SBR took decision Akka.Cluster.SBR.DownReachable and is downing
+      [akka.tcp://otopcua@central-2:4053] including myself, [1] unreachable of [2] members
+[INF] Cluster Node - Marking node [central-2:4053] as [Down]
+[INF] Self downed, stopping ClusterSingletonManager
+[INF] CoordinatedShutdown invoked due to [ClusterDowningReason]. Exiting with [ExitCode:3].
+```
+
+Result: central-2 Exited (255); `opc.tcp://localhost:4841` → Connection refused. No survivor.
+
+## Why (this is textbook keep-oldest, not a misconfig)
+`keep-oldest` keeps the partition CONTAINING the oldest and downs the other side. When the oldest CRASHES,
+the survivor (younger) is "the side without the oldest" → it downs itself. `down-if-alone = on` does NOT
+rescue it: that knob only makes the OLDEST down itself when the oldest is isolated-but-alive; a crashed
+oldest can'"'"'t execute it, and the younger applies strict keep-oldest → `DownReachable` (self-down). This is
+the documented 2-node keep-oldest limitation; Akka recommends 3+ members / a real quorum.
+
+## Why the in-process test gave false confidence
+`HardKillFailoverTests.Hard_kill_of_oldest_node_fails_over_to_survivor` PASSES, but `HardKillNodeAAsync`
+only calls `provider.Transport.Shutdown()` — node A'"'"'s **ActorSystem stays alive** (a transport partition
+with the oldest still running), NOT a process crash. So the in-process test exercises a different scenario
+than a real `docker kill`. It should be treated as non-representative for the oldest-CRASH case
+(test-quality follow-up: make it kill the process, or reframe what it proves).
+
+## Consequence
+Option A (keep `keep-oldest`) does not provide oldest-crash failover on a real 2-node deployment. The fix
+requires **Option B** (3rd witness/lighthouse member + `keep-majority` — survives any single crash) or
+**Option C** (DB-lease arbiter decoupling Primary from Akka oldest). A 2-node cluster fundamentally cannot
+auto-survive an arbitrary single-node crash under any SBR strategy without a tiebreaker.