Files
ScadaBridge/docs/plans/2026-11-12-leader-auto-down-rig-drill/sor-central-active-pause.log
T

40 lines
6.8 KiB
Plaintext

pair=central mode=active action=pause active=scadabridge-central-a victim=scadabridge-central-a survivor=scadabridge-central-b
hit at 2026-11-13T02:14:36Z
Active pause: waiting for scadabridge-central-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
survivor logged the Down at +27s
PASS: scadabridge-central-b took over as active in 27s with the victim still down
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
Confirming Traefik routes to the new active node...
Traefik routing recovered 31s after the hit.
Recovering scadabridge-central-a (pause -> unpause)...
Waiting for the victim to rejoin as a ready standby (scadabridge-central-a /health/ready)...
Recovered: scadabridge-central-a is ready as standby (rejoined) 34s after recovery.
Checking for a lingering dual-active (10 samples, 1s apart)...
Exactly one active node in 10/10 samples.
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-central-b):
[02:14:47 WRN] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-b:8081 is leader: False)
[02:14:47 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-b:8081); pending unreachable=[]
[02:15:02 WRN] [Central/scadabridge-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-b:8081) is leader
[02:15:02 INF] [Central/scadabridge-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-central-a:8081] as [Down]
[02:15:08 WRN] [Central/scadabridge-central-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-central-a:8081: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-central-b:8081) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the tiebreak (a frozen node that was removed restarts on its own)
[02:15:41 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-central-a:8081); pending unreachable=[]
SUMMARY pair=central mode=active action=pause down_logged_s=27 rejoin_s=34
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
--- victim scadabridge-central-a
[02:15:07 WRN] [Central/scadabridge-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-a:8081 is leader: True)
[02:15:08 ERR] [Central/scadabridge-central-a] AssociationError [akka.tcp://scadabridge@scadabridge-central-a:8081] <- akka.tcp://scadabridge@scadabridge-central-b:8081: Error [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.] [ at Akka.Remote.EndpointReader.HandleDisas
[02:15:08 WRN] [Central/scadabridge-central-a] Tried to associate with unreachable remote address [akka.tcp://scadabridge@scadabridge-central-b:8081]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [The remote system has quarantined this system. No further associations to the remote system are possib
[02:15:08 WRN] [Central/scadabridge-central-a] [leader-auto-down] shut out by akka.tcp://scadabridge@scadabridge-central-b:8081: it has removed and quarantined this node (akka.tcp://scadabridge@scadabridge-central-a:8081) while this node still counts it as a member (Up), so only this side is shut out; tiebreak not needed — this node RESTARTS to rejoin as a f
[02:15:08 ERR] [Central/scadabridge-central-a] AssociationError [akka.tcp://scadabridge@scadabridge-central-a:8081] -> akka.tcp://scadabridge@scadabridge-central-b:8081: Error [The remote system has quarantined this system. No further associations to the remote system are possible until this system is restarted.] [ at Akka.Remote.EndpointReader.HandleDisas
[02:15:08 WRN] [Central/scadabridge-central-a] Tried to associate with unreachable remote address [akka.tcp://scadabridge@scadabridge-central-b:8081]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: [The remote system has quarantined this system. No further associations to the remote system are possib
[02:15:09 FTL] [Central/scadabridge-central-a] ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down / leader-auto-down shut-out restart). Stopping the host process with exit code 75 so the service supervisor restarts this node as a fresh incarnation.
[02:15:39 WRN] [Central/scadabridge-central-a] ScadaBridge host stopped because its actor system terminated (self-down / shut-out restart); exiting with code 75 so the service supervisor restarts it
[02:15:40 INF] [Central/scadabridge-central-a] Starting ScadaBridge host as Central on scadabridge-central-a
[02:15:41 INF] [Central/scadabridge-central-a] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@scadabridge-central-a:8081: a node split from its pair peer restarts itself — at once if only it was removed, otherwise the higher address restarts after a 00:01:00 grace (ScadaBridge#132)
[02:15:41 INF] [Central/scadabridge-central-a] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-central-a:8081: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
[02:15:42 INF] [Central/scadabridge-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-a:8081); pending unreachable=[]
--- survivor scadabridge-central-b
[02:15:08 WRN] [Central/scadabridge-central-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-central-a:8081: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-central-b:8081) removed, so this node has shut it out. Giving it 00:01:00 to restart itsel
[02:15:41 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-central-a:8081); pending unreachable=[]
[02:16:08 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 rejoined (Up) within the shut-out grace; this node (akka.tcp://scadabridge@scadabridge-central-b:8081) stays