Files
ScadaBridge/docs/plans/2026-11-12-leader-auto-down-rig-drill/lad-central-partition.log
T
Joseph Doherty 139a43fb25 test(docker): leader-auto-down rig failover drill — #116 rig gate PASS on central + site-a
Ran every #116 gate on the local docker rig (main 2d2e9f35), with an auto-down baseline
on the same build:
- active kill: takeover 29s central / 26s site-a; standby kill: downed 29s / 26s, 0 blips;
- boot logs: no deprecation warning, [leader-auto-down] started/unreachable/DOWNING lines;
- partition past the window: accepted dual-active, restart of one side recovers;
- cold start of both pairs together: one cluster each; site-a S&F drains on the survivor.

Extra: docker pause of the active node leaves a permanent second active after unpause on
BOTH strategies (Akka.Cluster ignores ThisActorSystemQuarantinedEvent) — filed as #132.

- docker/docker-compose.leader-auto-down.yml: opt-in overlay (env override) for the central
  and site-a pairs; shipped default stays auto-down.
- docker/failover-drill.sh: DRILL_PAIR (central|site-a|site-b|site-c), DRILL_ACTION
  (kill|pause), sidecar probes for unpublished site health, strict rejoin + dual-active check.
- docs/plans/2026-11-12-leader-auto-down-rig-drill.md (+ raw logs), docker/README.md.

Refs #116, #106, #132

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
2026-11-12 20:16:00 -05:00

24 lines
2.6 KiB
Plaintext

01:07:22 before: scadabridge-central-a active=200 scadabridge-central-b active=503 (A ip 192.168.147.6)
01:07:23 partition ON (B drops all traffic to/from A)
01:07:51 DUAL-ACTIVE reached at +29s
01:08:12 end of hold (+50s): scadabridge-central-a active=200 scadabridge-central-b active=200
01:08:12 partition HEALED
01:08:22 +10s after heal: scadabridge-central-a active=200 scadabridge-central-b active=200
01:08:32 +20s after heal: scadabridge-central-a active=200 scadabridge-central-b active=200
01:08:43 +30s after heal: scadabridge-central-a active=200 scadabridge-central-b active=200
01:08:53 +40s after heal: scadabridge-central-a active=200 scadabridge-central-b active=200
01:09:03 +50s after heal: scadabridge-central-a active=200 scadabridge-central-b active=200
01:09:13 +60s after heal: scadabridge-central-a active=200 scadabridge-central-b active=200
--- scadabridge-central-a downing lines
[01:07:35 WRN] [Central/scadabridge-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-a:8081 is leader: True)
[01:07:50 WRN] [Central/scadabridge-central-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-b:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-a:8081) is leader
[01:07:50 INF] [Central/scadabridge-central-a] Cluster Node [akka.tcp://scadabridge@scadabridge-central-a:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-central-b:8081] as [Down]
--- scadabridge-central-b downing lines
[01:07:35 WRN] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-central-b:8081 is leader: False)
[01:07:35 INF] [Central/scadabridge-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-central-b:8081); pending unreachable=[]
[01:07:50 WRN] [Central/scadabridge-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-central-b:8081) is leader
[01:07:50 INF] [Central/scadabridge-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-central-a:8081] as [Down]
restart-one-side recovered in 12s
central-a ready=200 active=200
central-b ready=200 active=503