env=env2 pair=site-x mode=active action=kill active=scadabridge-env2-site-x-a victim=scadabridge-env2-site-x-a survivor=scadabridge-env2-site-x-b hit at 2026-11-13T04:45:17Z Active kill: waiting for scadabridge-env2-site-x-b to take over as the active node (victim stays down; budget ~25s + hand-over)... survivor logged the Down at +29s PASS: scadabridge-env2-site-x-b took over as active in 29s with the victim still down (downed the dead oldest, assumed Oldest, re-hosted the singletons). Recovering scadabridge-env2-site-x-a (kill -> start)... Waiting for the victim to rejoin as a ready standby (scadabridge-env2-site-x-a /health/ready)... Recovered: scadabridge-env2-site-x-a is ready as standby (rejoined) 1s after recovery. Checking for a lingering dual-active (10 samples, 1s apart)... Exactly one active node in 10/10 samples. Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-site-x-b): [04:45:30 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 is leader: False) [04:45:30 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082); pending unreachable=[] [04:45:45 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082) is leader [04:45:45 INF] [Site/scadabridge-env2-site-x-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082] as [Down] [04:45:46 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082] [04:45:46 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082] [04:45:46 INF] [Site/scadabridge-env2-site-x-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData [04:45:46 INF] [Site/scadabridge-env2-site-x-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData [04:45:47 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082); pending unreachable=[] SUMMARY env=env2 pair=site-x mode=active action=kill down_logged_s=29 rejoin_s=1 Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.