139a43fb25
Ran every #116 gate on the local docker rig (main 2d2e9f35), with an auto-down baseline
on the same build:
- active kill: takeover 29s central / 26s site-a; standby kill: downed 29s / 26s, 0 blips;
- boot logs: no deprecation warning, [leader-auto-down] started/unreachable/DOWNING lines;
- partition past the window: accepted dual-active, restart of one side recovers;
- cold start of both pairs together: one cluster each; site-a S&F drains on the survivor.
Extra: docker pause of the active node leaves a permanent second active after unpause on
BOTH strategies (Akka.Cluster ignores ThisActorSystemQuarantinedEvent) — filed as #132.
- docker/docker-compose.leader-auto-down.yml: opt-in overlay (env override) for the central
and site-a pairs; shipped default stays auto-down.
- docker/failover-drill.sh: DRILL_PAIR (central|site-a|site-b|site-c), DRILL_ACTION
(kill|pause), sidecar probes for unpublished site health, strict rejoin + dual-active check.
- docs/plans/2026-11-12-leader-auto-down-rig-drill.md (+ raw logs), docker/README.md.
Refs #116, #106, #132
Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
31 lines
2.3 KiB
Bash
31 lines
2.3 KiB
Bash
#!/usr/bin/env bash
|
|
# Evidence copy of the helper used for #116 gate 4 on 2026-11-12 (not a supported rig script).
|
|
# Prerequisites it assumed: a long-lived probe container on the rig network
|
|
# docker run -d --name sb-drill-probe --network scadabridge-net sb-drill-net:local sleep infinity
|
|
# and a local image with iptables + curl:
|
|
# printf 'FROM alpine:latest\nRUN apk add --no-cache iptables curl\n' | docker build -t sb-drill-net:local -
|
|
# partition.sh <pair> <hold_s> : cut Akka remoting between the two nodes of a pair (iptables in node B's netns),
|
|
# hold, heal, then watch. Leaves the dual-active in place for the caller to resolve.
|
|
set -euo pipefail
|
|
PAIR=$1; HOLD=$2
|
|
A=scadabridge-$PAIR-a; B=scadabridge-$PAIR-b
|
|
[ "$PAIR" = central ] && HP=5000 || HP=8084
|
|
AIP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $A)
|
|
c() { docker exec sb-drill-probe curl -s -m 2 -o /dev/null -w %{http_code} "http://$1:$HP/health/active" 2>/dev/null || echo ERR; }
|
|
echo "$(date -u +%T) before: $A active=$(c $A) $B active=$(c $B) (A ip $AIP)"
|
|
CUT_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ); T0=$(date +%s)
|
|
docker run --rm --net container:$B --cap-add NET_ADMIN sb-drill-net:local sh -c "iptables -I INPUT -s $AIP -j DROP && iptables -I OUTPUT -d $AIP -j DROP"
|
|
echo "$(date -u +%T) partition ON (B drops all traffic to/from A)"
|
|
DUAL_AT=""
|
|
while (( $(date +%s) - T0 < HOLD )); do
|
|
a=$(c $A); b=$(c $B)
|
|
if [ -z "$DUAL_AT" ] && [ "$a" = 200 ] && [ "$b" = 200 ]; then DUAL_AT=$(( $(date +%s) - T0 )); echo "$(date -u +%T) DUAL-ACTIVE reached at +${DUAL_AT}s"; fi
|
|
sleep 1
|
|
done
|
|
echo "$(date -u +%T) end of hold (+$(( $(date +%s)-T0 ))s): $A active=$(c $A) $B active=$(c $B)"
|
|
docker run --rm --net container:$B --cap-add NET_ADMIN sb-drill-net:local sh -c "iptables -D INPUT -s $AIP -j DROP && iptables -D OUTPUT -d $AIP -j DROP"
|
|
echo "$(date -u +%T) partition HEALED"
|
|
for i in $(seq 1 6); do sleep 10; echo "$(date -u +%T) +$((i*10))s after heal: $A active=$(c $A) $B active=$(c $B)"; done
|
|
echo "--- $A downing lines"; docker logs --since $CUT_AT $A 2>&1 | grep -E "leader-auto-down|auto-downing|Marking unreachable|quarantined this system" | head -6
|
|
echo "--- $B downing lines"; docker logs --since $CUT_AT $B 2>&1 | grep -E "leader-auto-down|auto-downing|Marking unreachable|quarantined this system" | head -6
|