env=env2 pair=central mode=active action=kill active=scadabridge-env2-central-a victim=scadabridge-env2-central-a survivor=scadabridge-env2-central-b hit at 2026-11-13T04:12:15Z Active kill: waiting for scadabridge-env2-central-b to take over as the active node (victim stays down; budget ~25s + hand-over)... survivor logged the Down at +27s PASS: scadabridge-env2-central-b took over as active in 27s with the victim still down (downed the dead oldest, assumed Oldest, re-hosted the singletons). Confirming Traefik routes to the new active node... Traefik routing recovered 29s after the hit. Recovering scadabridge-env2-central-a (kill -> start)... Waiting for the victim to rejoin as a ready standby (scadabridge-env2-central-a /health/ready)... Recovered: scadabridge-env2-central-a is ready as standby (rejoined) 2s after recovery. Checking for a lingering dual-active (10 samples, 1s apart)... Exactly one active node in 10/10 samples. Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-central-b): [04:12:41 INF] [Central/scadabridge-env2-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] - Leader is auto-downing unreachable node [akka.tcp://scadabridge@scadabridge-env2-central-a:8081] [04:12:41 INF] [Central/scadabridge-env2-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-central-a:8081] as [Down] [04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]] [04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData [04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]] [04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData [04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]] [04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]] [04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData [04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]] [04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData [04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData [04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]] [04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]] [04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData [04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData SUMMARY env=env2 pair=central mode=active action=kill down_logged_s=27 rejoin_s=2 Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.