dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(cluster): auto-down downing strategy — either-node crash now fails over (owner decision 2026-07-21: availability over partition-safety)

Browse Source 
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.

SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.

Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
This commit is contained in:
Joseph Doherty
2026-07-21 10:53:40 -04:00
parent dced0d2794
commit cf3bd52f93
29 changed files with 479 additions and 135 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docker/failover-drill.sh
+52 -39
View File
@@ -1,33 +1,33 @@
#!/usr/bin/env bash
# Failover drill against the running docker cluster (bash docker/deploy.sh first).
#
# ROUND-2 REWRITE (arch-review 01 round 2, N1). The original drill killed the
# ACTIVE central node — but under the unified oldest-member semantics the
# active node IS the oldest, i.e. the one crash two-node keep-oldest CANNOT
# survive (registered deferred user decision, master tracker 2026-07-08;
# SbrFailoverTests.cs XML doc). Two modes:
# AUTO-DOWN REWRITE (decision 2026-07-21). The cluster now runs the 'auto-down'
# downing strategy (availability-first): the leader among the REACHABLE members
# downs the unreachable peer after StableAfter, so a hard crash of EITHER
# central node — the active/oldest included — fails over to the survivor. The
# accepted trade (made explicitly by the owner) is dual-active during a real
# network partition. Both drill directions therefore expect RECOVERY:
#
# DRILL_MODE=standby (default) — kills the STANDBY (younger) central node.
# The survivable direction: SBR downs the crashed member, the active node
# keeps its singletons, and Traefik routing never goes dark. PASS = the
# survivor logs the member removal within TIMEOUT_S (budget ~25s+: 10s
# failure detection + 15s stable-after) while /health/active stays up.
# The active node is untouched: expect zero /health/active routing blips
# and member removal on the survivor within ~25s (10s failure detection +
# 15s auto-down-unreachable-after).
#
# DRILL_MODE=active — kills the ACTIVE (oldest) central node. THE EXPECTED
# OUTCOME IS A TOTAL CENTRAL OUTAGE: keep-oldest downs the partition
# without the oldest, so the younger survivor downs ITSELF (down-if-alone
# cannot help — the alone-oldest is dead and cannot down itself), and the
# self-downed survivor cannot re-form a cluster alone unless it is the
# FIRST seed (both nodes list central-a first; only the first seed may
# self-join). This mode measures the dark window and PASSes only when
# central recovers AFTER the victim container is restarted. It exists to
# make the registered gap observable — not to pretend it is covered.
# DRILL_MODE=active — kills the ACTIVE (oldest) central node. THE SURVIVOR
# MUST TAKE OVER: it downs the dead oldest, becomes oldest itself, hosts
# the singletons, and /health/active goes 200 on the survivor WHILE THE
# VICTIM IS STILL DOWN. Budget ~25s + singleton hand-over + health-probe
# margin. (Under the pre-2026-07-21 keep-oldest strategy this direction
# was a total outage — the younger survivor downed ITSELF, verified live;
# Akka's down-if-alone only rescues a side with >= 2 members.)
#
# Both modes finish by restarting the victim and confirming it rejoins as a
# fresh incarnation (standby).
set -euo pipefail
TRAEFIK_URL="${TRAEFIK_URL:-http://localhost:9000}"
TIMEOUT_S="${TIMEOUT_S:-90}"
DRILL_MODE="${DRILL_MODE:-standby}"
OUTAGE_CONFIRM_S="${OUTAGE_CONFIRM_S:-60}"
active_container() {
if curl -sf -o /dev/null "http://localhost:9001/health/active"; then echo scadabridge-central-a
@@ -35,6 +35,7 @@ active_container() {
else echo "ERROR: no active central node found" >&2; exit 1; fi
}
peer_of() { [ "$1" = scadabridge-central-a ] && echo scadabridge-central-b || echo scadabridge-central-a; }
port_of() { [ "$1" = scadabridge-central-a ] && echo 9001 || echo 9002; }
case "$DRILL_MODE" in
standby|active) ;;
@@ -47,6 +48,7 @@ if [ "$DRILL_MODE" = standby ]; then
else
VICTIM="$ACTIVE"; SURVIVOR=$(peer_of "$ACTIVE")
fi
SURVIVOR_PORT=$(port_of "$SURVIVOR")
echo "mode=${DRILL_MODE} active=${ACTIVE} victim=${VICTIM} survivor=${SURVIVOR}"
KILL_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ)
@@ -54,60 +56,71 @@ docker kill "${VICTIM}" > /dev/null
START=$(date +%s)
if [ "$DRILL_MODE" = standby ]; then
echo "Standby crash: waiting for ${SURVIVOR} to DOWN+REMOVE the dead member (SBR budget ~25s)..."
echo "Standby crash: waiting for ${SURVIVOR} to DOWN+REMOVE the dead member (budget ~25s)..."
BLIPS=0
while true; do
ELAPSED=$(( $(date +%s) - START ))
curl -sf -o /dev/null "${TRAEFIK_URL}/health/active" || BLIPS=$((BLIPS + 1))
if docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "marking.*node.*down|member removed|is removed"; then
echo "PASS: survivor removed the crashed member in ${ELAPSED}s (budget ~25s: 10s detection + 15s stable-after)."
if docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Eiq "auto-downing|marking.*node.*down|member removed|is removed"; then
echo "PASS: survivor downed/removed the crashed member in ${ELAPSED}s (budget ~25s: 10s detection + 15s auto-down)."
echo "Active-node routing blips during the drill: ${BLIPS} (expected 0 — the active node was never touched)."
break
fi
if (( ELAPSED > TIMEOUT_S )); then
echo "FAIL: no downing/removal evidence on ${SURVIVOR} after ${ELAPSED}s — SBR did not act" >&2
echo "FAIL: no downing/removal evidence on ${SURVIVOR} after ${ELAPSED}s — auto-down did not act" >&2
docker start "${VICTIM}" > /dev/null
exit 1
fi
sleep 1
done
else
echo "Active crash: EXPECTING a central outage (registered keep-oldest gap). Watching /health/active..."
DARK_STREAK=0
echo "Active crash: waiting for ${SURVIVOR} to take over as the active node (victim stays DOWN; budget ~25s + hand-over)..."
while true; do
ELAPSED=$(( $(date +%s) - START ))
if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then DARK_STREAK=0; else DARK_STREAK=$((DARK_STREAK + 1)); fi
if (( DARK_STREAK >= 10 )); then
echo "Outage confirmed at ${ELAPSED}s: no active central node — the younger survivor self-downed"
echo "(keep-oldest downs the partition WITHOUT the oldest; this is the registered deferred gap)."
if curl -sf -o /dev/null "http://localhost:${SURVIVOR_PORT}/health/active"; then
echo "PASS: ${SURVIVOR} took over as active in ${ELAPSED}s with the victim still down"
echo "(downed the dead oldest via auto-down, assumed Oldest, re-hosted the singletons)."
break
fi
if (( ELAPSED > OUTAGE_CONFIRM_S )); then
echo "NOTE: /health/active stayed reachable ${ELAPSED}s after killing the oldest — better than the"
echo "registered gap predicts. Do NOT celebrate: capture both nodes' logs and investigate before trusting it."
break
if (( ELAPSED > TIMEOUT_S )); then
echo "FAIL: ${SURVIVOR} never became active within ${ELAPSED}s of killing the oldest — takeover did not happen." >&2
docker logs --since "${KILL_AT}" "${SURVIVOR}" 2>&1 | grep -Ei "sbr|downing|oldest|shutting down|terminated" | tail -20 >&2 || true
docker start "${VICTIM}" > /dev/null
exit 1
fi
sleep 1
done
echo "Confirming Traefik routes to the new active node..."
TR_START=$(date +%s)
while ! curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; do
if (( $(date +%s) - TR_START > 60 )); then
echo "FAIL: survivor is active but not routable through Traefik after 60s" >&2
docker start "${VICTIM}" > /dev/null
exit 1
fi
sleep 1
done
echo "Traefik routing recovered $(( $(date +%s) - START ))s after the kill."
fi
echo "Restarting ${VICTIM}..."
docker start "${VICTIM}" > /dev/null
RESTART_AT=$(date +%s)
echo "Waiting for central to be routable again through Traefik (${TRAEFIK_URL}/health/active)..."
echo "Waiting for the restarted victim to rejoin as a ready standby (${VICTIM} /health/ready)..."
VICTIM_PORT=$(port_of "$VICTIM")
while true; do
ELAPSED=$(( $(date +%s) - RESTART_AT ))
if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then
echo "Recovered: an active central node is routable ${ELAPSED}s after the victim restart."
if curl -sf -o /dev/null "http://localhost:${VICTIM_PORT}/health/ready"; then
echo "Recovered: ${VICTIM} is ready (rejoined as a fresh incarnation) ${ELAPSED}s after restart."
break
fi
if (( ELAPSED > 120 )); then
echo "FAIL: central not routable 120s after restarting ${VICTIM}" >&2
echo "FAIL: ${VICTIM} not ready 120s after restart" >&2
exit 1
fi
sleep 1
done
echo "Survivor singleton/downing evidence (last 20 matching log lines from ${SURVIVOR}):"
docker logs "${SURVIVOR}" 2>&1 | grep -Ei "singleton|oldest|downing|removed" | tail -20 || true
echo "Survivor downing/singleton evidence (last 20 matching log lines from ${SURVIVOR}):"
docker logs "${SURVIVOR}" 2>&1 | grep -Ei "auto-downing|singleton|oldest|downing|removed" | tail -20 || true
echo "Drill complete (${DRILL_MODE}). Verify on the Health dashboard that both nodes show Up and exactly one is Primary."