test(docker): failover drill script — SIGKILL the active central node, assert Traefik recovery

2026-07-08 17:05:31 -04:00
parent 1f1dbd916d
commit 5117fa97b9
3 changed files with 62 additions and 0 deletions
@@ -270,6 +270,23 @@ All test passwords are `password`. See `infra/glauth/config.toml` for the full l

 ## Failover Testing

+### Automated failover drill (`failover-drill.sh`)
+
+```bash
+bash docker/failover-drill.sh
+```
+
+The scripted drill is the repeatable version of the manual steps below. It:
+
+1. Finds the **active** central node (via each node's `/health/active`).
+2. **`docker kill`s it — SIGKILL, the hard-crash path.** This is the exact scenario arch-review 01 found had *no* automatic recovery before the SBR downing provider was enabled (`Akka.Cluster.SBR.SplitBrainResolverProvider` is now named explicitly in HOCON); a `docker stop` would take the graceful `CoordinatedShutdown` path instead and would not prove crash recovery.
+3. Polls Traefik (`http://localhost:9000/health/active`, override with `TRAEFIK_URL`) until the survivor is routable, printing the observed failover time. Fails after `TIMEOUT_S` (default 90s).
+4. Dumps the survivor's singleton/oldest/downing log evidence, then restarts the victim so it **rejoins as a fresh incarnation** (Task 20's recovery loop).
+
+Expected failover: **~25s** (2s heartbeat + 10s detection + 15s stable-after) **plus the Traefik ~5s health-check interval**. The drill exercises S1 (SBR downing on hard crash), S3 (single active node routed through Traefik), and the Task 20 restart/rejoin contract. Requires a running cluster (`bash docker/deploy.sh`) and `curl` + `docker` on the host.
+
+> **Observed failover time:** _run after next deploy and record here_ (the script was validated with `bash -n`; no live cluster was up when it was committed).
+
 ### Central Failover

 ```bash