pair=central mode=active action=kill active=scadabridge-central-a victim=scadabridge-central-a survivor=scadabridge-central-b hit at 2026-11-13T00:48:59Z Active kill: waiting for scadabridge-central-b to take over as the active node (victim stays down; budget ~25s + hand-over)... survivor logged the Down at +27s PASS: scadabridge-central-b took over as active in 27s with the victim still down (downed the dead oldest, assumed Oldest, re-hosted the singletons). Confirming Traefik routes to the new active node... Traefik routing recovered 30s after the hit. Recovering scadabridge-central-a (kill -> start)... Waiting for the victim to rejoin as a ready standby (scadabridge-central-a /health/ready)... Recovered: scadabridge-central-a is ready as standby (rejoined) 3s after recovery. Checking for a lingering dual-active (10 samples, 1s apart)... Exactly one active node in 10/10 samples. Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-central-b): [00:49:27 INF] [Central/scadabridge-central-b] Singleton manager started singleton actor [akka://scadabridge/user/site-call-audit-singleton/site-call-audit] [00:49:27 INF] [Central/scadabridge-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData [00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [audit-log-purge] in role [null] after 30 seconds. [00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [site-call-audit] in role [null] after 30 seconds. [00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [notification-outbox] in role [null] after 30 seconds. [00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [pending-deployment-purge] in role [null] after 30 seconds. [00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [site-audit-reconciliation] in role [null] after 30 seconds. [00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [audit-log-ingest] in role [null] after 30 seconds. [00:49:27 WRN] [Central/scadabridge-central-b] ClusterSingletonProxy failed to find an associated singleton named [kpi-history-recorder] in role [null] after 30 seconds. [00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/pending-deployment-purge-singleton/pending-deployment-purge] [00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/audit-log-ingest-singleton/audit-log-ingest] [00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/audit-log-purge-singleton/audit-log-purge] [00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/site-audit-reconciliation-singleton/site-audit-reconciliation] [00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/notification-outbox-singleton/notification-outbox] [00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/kpi-history-recorder-singleton/kpi-history-recorder] [00:49:27 INF] [Central/scadabridge-central-b] Singleton identified at [akka://scadabridge/user/site-call-audit-singleton/site-call-audit] SUMMARY pair=central mode=active action=kill down_logged_s=27 rejoin_s=3 Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.