Files
ScadaBridge/docs/plans/2026-11-12-leader-auto-down-env2-drill/baseline-env2-site-x-active-kill.log
T

22 lines
2.4 KiB
Plaintext

env=env2 pair=site-x mode=active action=kill active=scadabridge-env2-site-x-a victim=scadabridge-env2-site-x-a survivor=scadabridge-env2-site-x-b
hit at 2026-11-13T04:13:06Z
Active kill: waiting for scadabridge-env2-site-x-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
survivor logged the Down at +28s
PASS: scadabridge-env2-site-x-b took over as active in 28s with the victim still down
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
Recovering scadabridge-env2-site-x-a (kill -> start)...
Waiting for the victim to rejoin as a ready standby (scadabridge-env2-site-x-a /health/ready)...
Recovered: scadabridge-env2-site-x-a is ready as standby (rejoined) 12s after recovery.
Checking for a lingering dual-active (10 samples, 1s apart)...
Exactly one active node in 10/10 samples.
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-site-x-b):
[04:13:34 INF] [Site/scadabridge-env2-site-x-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] - Leader is auto-downing unreachable node [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
[04:13:34 INF] [Site/scadabridge-env2-site-x-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082] as [Down]
[04:13:35 INF] [Site/scadabridge-env2-site-x-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] - New incarnation of existing member [UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082, 1261810842)] is trying to join. Existing will be removed from the cluster and then new member will be allowed to join.
[04:13:35 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
[04:13:35 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
[04:13:35 INF] [Site/scadabridge-env2-site-x-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
[04:13:35 INF] [Site/scadabridge-env2-site-x-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
SUMMARY env=env2 pair=site-x mode=active action=kill down_logged_s=28 rejoin_s=12
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.