482ac66548
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
32 lines
4.7 KiB
Plaintext
32 lines
4.7 KiB
Plaintext
pair=site-a mode=active action=pause active=scadabridge-site-a-a victim=scadabridge-site-a-a survivor=scadabridge-site-a-b
|
|
hit at 2026-11-13T02:16:24Z
|
|
Active pause: waiting for scadabridge-site-a-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
|
|
survivor logged the Down at +27s
|
|
PASS: scadabridge-site-a-b took over as active in 27s with the victim still down
|
|
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
|
|
Recovering scadabridge-site-a-a (pause -> unpause)...
|
|
Waiting for the victim to rejoin as a ready standby (scadabridge-site-a-a /health/ready)...
|
|
Recovered: scadabridge-site-a-a is ready as standby (rejoined) 6s after recovery.
|
|
Checking for a lingering dual-active (10 samples, 1s apart)...
|
|
Exactly one active node in 10/10 samples.
|
|
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-site-a-b):
|
|
[02:16:35 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-site-a-b:8082 is leader: False)
|
|
[02:16:35 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-b:8082); pending unreachable=[]
|
|
[02:16:50 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-site-a-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-site-a-b:8082) is leader
|
|
[02:16:50 INF] [Site/scadabridge-site-a-b] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-site-a-a:8082] as [Down]
|
|
[02:16:56 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-site-a-a:8082: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-site-a-b:8082) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the tiebreak (a frozen node that was removed restarts on its own)
|
|
[02:16:58 INF] [Site/scadabridge-site-a-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-a:8082); pending unreachable=[]
|
|
SUMMARY pair=site-a mode=active action=pause down_logged_s=27 rejoin_s=6
|
|
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|
|
--- victim scadabridge-site-a-a
|
|
[02:16:51 WRN] [Site/scadabridge-site-a-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-b:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-site-a-a:8082 is leader: True)
|
|
[02:16:56 WRN] [Site/scadabridge-site-a-a] [leader-auto-down] shut out by akka.tcp://scadabridge@scadabridge-site-a-b:8082: it has removed and quarantined this node (akka.tcp://scadabridge@scadabridge-site-a-a:8082) while this node still counts it as a member (Up), so only this side is shut out; tiebreak not needed — this node R
|
|
[02:16:56 FTL] [Site/scadabridge-site-a-a] ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down / leader-auto-down shut-out restart). Stopping the host process with exit code 75 so the service supervisor restarts this node as a fresh incarnation.
|
|
[02:16:56 WRN] [Site/scadabridge-site-a-a] ScadaBridge host stopped because its actor system terminated (self-down / shut-out restart); exiting with code 75 so the service supervisor restarts it
|
|
[02:16:57 INF] [Site/scadabridge-site-a-a] Starting ScadaBridge host as Site on scadabridge-site-a-a
|
|
[02:16:57 INF] [Site/scadabridge-site-a-a] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@scadabridge-site-a-a:8082: a node split from its pair peer restarts itself — at once if only it was removed, otherwise the higher address restarts after a 00:01:00 grace (ScadaBridge#132)
|
|
[02:16:57 INF] [Site/scadabridge-site-a-a] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-site-a-a:8082: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
|
|
[02:16:58 INF] [Site/scadabridge-site-a-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-site-a-a:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-site-a-a:8082); pending unreachable=[]
|
|
--- survivor scadabridge-site-a-b
|
|
[02:16:56 WRN] [Site/scadabridge-site-a-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-site-a-a:8082: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-site-a-b:8082) removed, so this node has shut it out. Giving it 00:01:00 to restart itself befo
|