Files
ScadaBridge/docs/plans/2026-11-12-leader-auto-down-rig-drill/baseline-site-a-active-kill.log
T
Joseph Doherty 139a43fb25 test(docker): leader-auto-down rig failover drill — #116 rig gate PASS on central + site-a
Ran every #116 gate on the local docker rig (main 2d2e9f35), with an auto-down baseline
on the same build:
- active kill: takeover 29s central / 26s site-a; standby kill: downed 29s / 26s, 0 blips;
- boot logs: no deprecation warning, [leader-auto-down] started/unreachable/DOWNING lines;
- partition past the window: accepted dual-active, restart of one side recovers;
- cold start of both pairs together: one cluster each; site-a S&F drains on the survivor.

Extra: docker pause of the active node leaves a permanent second active after unpause on
BOTH strategies (Akka.Cluster ignores ThisActorSystemQuarantinedEvent) — filed as #132.

- docker/docker-compose.leader-auto-down.yml: opt-in overlay (env override) for the central
  and site-a pairs; shipped default stays auto-down.
- docker/failover-drill.sh: DRILL_PAIR (central|site-a|site-b|site-c), DRILL_ACTION
  (kill|pause), sidecar probes for unpublished site health, strict rejoin + dual-active check.
- docs/plans/2026-11-12-leader-auto-down-rig-drill.md (+ raw logs), docker/README.md.

Refs #116, #106, #132

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 20:16:00 -05:00

21 lines
1.9 KiB
Plaintext

pair=site-a mode=active action=kill active=scadabridge-site-a-a victim=scadabridge-site-a-a survivor=scadabridge-site-a-b
hit at 2026-11-13T00:49:50Z
Active kill: waiting for scadabridge-site-a-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
survivor logged the Down at +27s
PASS: scadabridge-site-a-b took over as active in 27s with the victim still down
(downed the dead oldest, assumed Oldest, re-hosted the singletons).
Recovering scadabridge-site-a-a (kill -> start)...
Waiting for the victim to rejoin as a ready standby (scadabridge-site-a-a /health/ready)...
Recovered: scadabridge-site-a-a is ready as standby (rejoined) 1s after recovery.
Checking for a lingering dual-active (10 samples, 1s apart)...
Exactly one active node in 10/10 samples.
Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-site-a-b):
[00:50:16 INF] [Site/scadabridge-site-a-b] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] - Leader is auto-downing unreachable node [akka.tcp://scadabridge@scadabridge-site-a-a:8082]
[00:50:16 INF] [Site/scadabridge-site-a-b] Cluster Node [akka.tcp://scadabridge@scadabridge-site-a-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-site-a-a:8082] as [Down]
[00:50:17 INF] [Site/scadabridge-site-a-b] Member removed [akka.tcp://scadabridge@scadabridge-site-a-a:8082]
[00:50:17 INF] [Site/scadabridge-site-a-b] Member removed [akka.tcp://scadabridge@scadabridge-site-a-a:8082]
[00:50:17 INF] [Site/scadabridge-site-a-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
[00:50:17 INF] [Site/scadabridge-site-a-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
SUMMARY pair=site-a mode=active action=kill down_logged_s=27 rejoin_s=1
Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.