Files
ScadaBridge/docs/plans/2026-11-12-leader-auto-down-rig-drill/sor-site-a-active-kill.log
T

21 lines
2.2 KiB
Plaintext

pair=site-a mode=active action=kill active=scadabridge-site-a-a victim=scadabridge-site-a-a survivor=scadabridge-site-a-b
hit at 2026-11-13T02:23:53Z
Active kill: waiting for scadabridge-site-a-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
survivor logged the Down at +28s
PASS: scadabridge-site-a-b took over as active in 28s with the victim still down
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
Recovering scadabridge-site-a-a (kill -> start)...
Waiting for the victim to rejoin as a ready standby (scadabridge-site-a-a /health/ready)...
Recovered: scadabridge-site-a-a is ready as standby (rejoined) 1s after recovery.
Checking for a lingering dual-active (10 samples, 1s apart)...
Exactly one active node in 10/10 samples.
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-site-a-b):
[02:24:05 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-site-a-b:8082 is leader: False)
[02:24:05 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-b:8082); pending unreachable=[]
[02:24:20 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-site-a-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-site-a-b:8082) is leader
[02:24:20 INF] [Site/scadabridge-site-a-b] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-site-a-a:8082] as [Down]
[02:24:22 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-a:8082); pending unreachable=[]
SUMMARY pair=site-a mode=active action=kill down_logged_s=28 rejoin_s=1
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
--- shut-out lines since hit (scadabridge-site-a-b, scadabridge-site-a-a)