cafdc0ce60
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
26 lines
3.1 KiB
Plaintext
26 lines
3.1 KiB
Plaintext
env=env2 pair=central mode=standby action=kill active=scadabridge-env2-central-b victim=scadabridge-env2-central-a survivor=scadabridge-env2-central-b
|
|
hit at 2026-11-13T04:15:38Z
|
|
Standby kill: waiting for scadabridge-env2-central-b to DOWN+REMOVE the victim (budget ~25s)...
|
|
PASS: survivor downed/removed the victim in 29s (budget ~25s: 10s detection + 15s down window).
|
|
Active-node /health/active blips during the drill: 0 (expected 0 — the active node was never touched).
|
|
Recovering scadabridge-env2-central-a (kill -> start)...
|
|
Waiting for the victim to rejoin as a ready standby (scadabridge-env2-central-a /health/ready)...
|
|
Recovered: scadabridge-env2-central-a is ready as standby (rejoined) 2s after recovery.
|
|
Checking for a lingering dual-active (10 samples, 1s apart)...
|
|
Exactly one active node in 10/10 samples.
|
|
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-central-b):
|
|
[04:15:51 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-central-b:8081 is leader: False)
|
|
[04:15:51 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-central-b:8081); pending unreachable=[]
|
|
[04:16:06 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-central-b:8081) is leader
|
|
[04:16:06 INF] [Central/scadabridge-env2-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-central-a:8081] as [Down]
|
|
[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
|
[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
|
[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
|
[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
|
[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
|
[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
|
[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
|
|
[04:16:09 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-central-a:8081); pending unreachable=[]
|
|
SUMMARY env=env2 pair=central mode=standby action=kill down_logged_s=29 rejoin_s=2
|
|
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
|