# #459 — LIVE finding: keep-oldest downs the survivor when the oldest crashes (2-container rig)

**Date:** 2026-07-15 · **Rig:** docker-dev, central-1 (seed/oldest, :4840) + central-2 (:4841), MAIN Warm/2-node.

## Headline
On a **real 2-container rig**, hard-killing the **oldest** node makes the **younger survivor down ITSELF**
→ both processes exit → **no failover, total outage**. This CONFIRMS #459 (the premise is real, not stale)
and **disproves Option A** (keep `keep-oldest` is insufficient). Reproduced twice, deterministically.

## Baseline (both nodes healthy, converged)
- `Server.ServiceLevel` (NodeId `i=2267`): central-1 (oldest) = **250** (healthy leader), central-2 = **240** (healthy follower). Matches the documented tiering.

## The kill
`docker kill otopcua-dev-central-1-1` (true process death, not a graceful Leave). central-2'"'"'s SBR:

```
[WRN] SBR took decision Akka.Cluster.SBR.DownReachable and is downing
      [akka.tcp://otopcua@central-2:4053] including myself, [1] unreachable of [2] members
[INF] Cluster Node - Marking node [central-2:4053] as [Down]
[INF] Self downed, stopping ClusterSingletonManager
[INF] CoordinatedShutdown invoked due to [ClusterDowningReason]. Exiting with [ExitCode:3].
```

Result: central-2 Exited (255); `opc.tcp://localhost:4841` → Connection refused. No survivor.

## Why (this is textbook keep-oldest, not a misconfig)
`keep-oldest` keeps the partition CONTAINING the oldest and downs the other side. When the oldest CRASHES,
the survivor (younger) is "the side without the oldest" → it downs itself. `down-if-alone = on` does NOT
rescue it: that knob only makes the OLDEST down itself when the oldest is isolated-but-alive; a crashed
oldest can'"'"'t execute it, and the younger applies strict keep-oldest → `DownReachable` (self-down). This is
the documented 2-node keep-oldest limitation; Akka recommends 3+ members / a real quorum.

## Why the in-process test gave false confidence
`HardKillFailoverTests.Hard_kill_of_oldest_node_fails_over_to_survivor` PASSES, but `HardKillNodeAAsync`
only calls `provider.Transport.Shutdown()` — node A'"'"'s **ActorSystem stays alive** (a transport partition
with the oldest still running), NOT a process crash. So the in-process test exercises a different scenario
than a real `docker kill`. It should be treated as non-representative for the oldest-CRASH case
(test-quality follow-up: make it kill the process, or reframe what it proves).

## Consequence
Option A (keep `keep-oldest`) does not provide oldest-crash failover on a real 2-node deployment. The fix
requires **Option B** (3rd witness/lighthouse member + `keep-majority` — survives any single crash) or
**Option C** (DB-lease arbiter decoupling Primary from Akka oldest). A 2-node cluster fundamentally cannot
auto-survive an arbitrary single-node crash under any SBR strategy without a tiebreaker.