Files
ScadaBridge/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-active-pause.log
T

35 lines
4.2 KiB
Plaintext

env=env2 pair=central mode=active action=pause active=scadabridge-env2-central-b victim=scadabridge-env2-central-b survivor=scadabridge-env2-central-a
hit at 2026-11-13T04:17:52Z
Active pause: waiting for scadabridge-env2-central-a to take over as the active node (victim stays down; budget ~25s + hand-over)...
survivor logged the Down at +27s
PASS: scadabridge-env2-central-a took over as active in 27s with the victim still down
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
Confirming Traefik routes to the new active node...
Traefik routing recovered 32s after the hit.
Recovering scadabridge-env2-central-b (pause -> unpause)...
Waiting for the victim to rejoin as a ready standby (scadabridge-env2-central-b /health/ready)...
Recovered: scadabridge-env2-central-b is ready as standby (rejoined) 3s after recovery.
Checking for a lingering dual-active (10 samples, 1s apart)...
Exactly one active node in 10/10 samples.
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-central-a):
[04:18:03 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-central-a:8081 is leader: True)
[04:18:18 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-central-b:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-central-a:8081) is leader
[04:18:18 INF] [Central/scadabridge-env2-central-a] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-central-a:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] as [Down]
[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
[04:18:24 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-central-b:8081: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-env2-central-a:8081) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the tiebreak (a frozen node that was removed restarts on its own)
SUMMARY env=env2 pair=central mode=active action=pause down_logged_s=27 rejoin_s=3
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.