644f564c30
Unreplicated, the site-x survivor came up empty (by design: no LocalDb peer); replicated, kill and freeze each lost only the 26-30 s takeover window, every buffered message was delivered once, and the 4 duplicate ticks came from the frozen victim's 6 s zombie window. Central kill: 0 gap. Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
23 lines
2.7 KiB
Plaintext
23 lines
2.7 KiB
Plaintext
env=env2 pair=site-x mode=active action=pause active=scadabridge-env2-site-x-b victim=scadabridge-env2-site-x-b survivor=scadabridge-env2-site-x-a
|
|
hit at 2026-11-13T04:48:14Z
|
|
Active pause: waiting for scadabridge-env2-site-x-a to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
|
survivor logged the Down at +27s
|
|
PASS: scadabridge-env2-site-x-a took over as active in 27s with the victim still down
|
|
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
|
Recovering scadabridge-env2-site-x-b (pause -> unpause)...
|
|
Waiting for the victim to rejoin as a ready standby (scadabridge-env2-site-x-b /health/ready)...
|
|
Recovered: scadabridge-env2-site-x-b is ready as standby (rejoined) 38s after recovery.
|
|
Checking for a lingering dual-active (10 samples, 1s apart)...
|
|
Exactly one active node in 10/10 samples.
|
|
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-site-x-a):
|
|
[04:48:26 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082 is leader: True)
|
|
[04:48:41 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082) is leader
|
|
[04:48:41 INF] [Site/scadabridge-env2-site-x-a] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] as [Down]
|
|
[04:48:42 INF] [Site/scadabridge-env2-site-x-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082]
|
|
[04:48:42 INF] [Site/scadabridge-env2-site-x-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082]
|
|
[04:48:42 INF] [Site/scadabridge-env2-site-x-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
|
[04:48:42 INF] [Site/scadabridge-env2-site-x-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
|
|
[04:48:47 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the tiebreak (a frozen node that was removed restarts on its own)
|
|
SUMMARY env=env2 pair=site-x mode=active action=pause down_logged_s=27 rejoin_s=38
|
|
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|