diff --git a/deployments/docker-cluster.md b/deployments/docker-cluster.md index 705b6ca0..03d99e4d 100644 --- a/deployments/docker-cluster.md +++ b/deployments/docker-cluster.md @@ -127,3 +127,4 @@ All passwords are `password`. See `infra/glauth/config.toml` for the full list. - Reference deployment for the project — when a change ships, it gets validated here first. - Concurrent with [`docker-cluster-env2`](docker-cluster-env2.md) on the same host; the two stacks share the `scadabridge-net` network and `infra/` services but use disjoint host ports (`90XX` vs `91XX`) and databases. - Detailed setup, failover testing, and build-cache notes live in [`docker/README.md`](../docker/README.md). +- Automated failover drill: [`docker/failover-drill.sh`](../docker/failover-drill.sh) SIGKILLs the active central node and asserts Traefik recovers to the survivor (~25s + Traefik health interval); see the "Automated failover drill" subsection in `docker/README.md`. diff --git a/docker/README.md b/docker/README.md index f1de06b7..63530d93 100644 --- a/docker/README.md +++ b/docker/README.md @@ -270,6 +270,23 @@ All test passwords are `password`. See `infra/glauth/config.toml` for the full l ## Failover Testing +### Automated failover drill (`failover-drill.sh`) + +```bash +bash docker/failover-drill.sh +``` + +The scripted drill is the repeatable version of the manual steps below. It: + +1. Finds the **active** central node (via each node's `/health/active`). +2. **`docker kill`s it — SIGKILL, the hard-crash path.** This is the exact scenario arch-review 01 found had *no* automatic recovery before the SBR downing provider was enabled (`Akka.Cluster.SBR.SplitBrainResolverProvider` is now named explicitly in HOCON); a `docker stop` would take the graceful `CoordinatedShutdown` path instead and would not prove crash recovery. +3. Polls Traefik (`http://localhost:9000/health/active`, override with `TRAEFIK_URL`) until the survivor is routable, printing the observed failover time. Fails after `TIMEOUT_S` (default 90s). +4. Dumps the survivor's singleton/oldest/downing log evidence, then restarts the victim so it **rejoins as a fresh incarnation** (Task 20's recovery loop). + +Expected failover: **~25s** (2s heartbeat + 10s detection + 15s stable-after) **plus the Traefik ~5s health-check interval**. The drill exercises S1 (SBR downing on hard crash), S3 (single active node routed through Traefik), and the Task 20 restart/rejoin contract. Requires a running cluster (`bash docker/deploy.sh`) and `curl` + `docker` on the host. + +> **Observed failover time:** _run after next deploy and record here_ (the script was validated with `bash -n`; no live cluster was up when it was committed). + ### Central Failover ```bash diff --git a/docker/failover-drill.sh b/docker/failover-drill.sh new file mode 100755 index 00000000..d600eb72 --- /dev/null +++ b/docker/failover-drill.sh @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# Failover drill against the running docker cluster (bash docker/deploy.sh first). +# Kills the ACTIVE central node (docker kill = SIGKILL, the hard-crash path that +# review 01 found had NO automatic recovery before the SBR downing provider was +# enabled) and measures how long Traefik takes to route to the survivor. +set -euo pipefail + +TRAEFIK_URL="${TRAEFIK_URL:-http://localhost:9000}" +TIMEOUT_S="${TIMEOUT_S:-90}" + +active_container() { + if curl -sf -o /dev/null "http://localhost:9001/health/active"; then echo scadabridge-central-a + elif curl -sf -o /dev/null "http://localhost:9002/health/active"; then echo scadabridge-central-b + else echo "ERROR: no active central node found" >&2; exit 1; fi +} + +VICTIM=$(active_container) +echo "Active central node: ${VICTIM} — killing it (SIGKILL, crash path)" +docker kill "${VICTIM}" > /dev/null +START=$(date +%s) + +echo "Waiting for the survivor to become active through Traefik (${TRAEFIK_URL}/health/active)..." +while true; do + ELAPSED=$(( $(date +%s) - START )) + if curl -sf -o /dev/null "${TRAEFIK_URL}/health/active"; then + echo "PASS: failover complete in ${ELAPSED}s (design budget ~25s + Traefik health interval)" + break + fi + if (( ELAPSED > TIMEOUT_S )); then + echo "FAIL: no active central node after ${ELAPSED}s — SBR/singleton handover did not recover" >&2 + docker start "${VICTIM}" > /dev/null + exit 1 + fi + sleep 1 +done + +SURVIVOR=$([ "${VICTIM}" = scadabridge-central-a ] && echo scadabridge-central-b || echo scadabridge-central-a) +echo "Survivor singleton evidence (last 20 matching log lines from ${SURVIVOR}):" +docker logs "${SURVIVOR}" 2>&1 | grep -Ei "singleton|oldest|Downing|Removed" | tail -20 || true + +echo "Restarting ${VICTIM} and waiting for it to rejoin..." +docker start "${VICTIM}" > /dev/null +sleep 5 +echo "Drill complete. Verify on the Health dashboard that both nodes show Up and the survivor is Primary."