test(docker): leader-auto-down rig failover drill — #116 rig gate PASS on central + site-a

Ran every #116 gate on the local docker rig (main 2d2e9f35), with an auto-down baseline
on the same build:
- active kill: takeover 29s central / 26s site-a; standby kill: downed 29s / 26s, 0 blips;
- boot logs: no deprecation warning, [leader-auto-down] started/unreachable/DOWNING lines;
- partition past the window: accepted dual-active, restart of one side recovers;
- cold start of both pairs together: one cluster each; site-a S&F drains on the survivor.

Extra: docker pause of the active node leaves a permanent second active after unpause on
BOTH strategies (Akka.Cluster ignores ThisActorSystemQuarantinedEvent) — filed as #132.

- docker/docker-compose.leader-auto-down.yml: opt-in overlay (env override) for the central
  and site-a pairs; shipped default stays auto-down.
- docker/failover-drill.sh: DRILL_PAIR (central|site-a|site-b|site-c), DRILL_ACTION
  (kill|pause), sidecar probes for unpublished site health, strict rejoin + dual-active check.
- docs/plans/2026-11-12-leader-auto-down-rig-drill.md (+ raw logs), docker/README.md.

Refs #116, #106, #132

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi

docker/README.md

@@ -385,6 +385,20 @@ All test passwords are `password`. See `infra/glauth/config.toml` for the full l
 ```bash
 DRILL_MODE=standby bash docker/failover-drill.sh   # default — younger-node crash, active untouched
 DRILL_MODE=active  bash docker/failover-drill.sh    # oldest-node crash — survivor must TAKE OVER
+DRILL_PAIR=site-a DRILL_MODE=active bash docker/failover-drill.sh   # same drill on a site pair (site-a|site-b|site-c)
+DRILL_ACTION=pause DRILL_MODE=active bash docker/failover-drill.sh  # freeze instead of kill (see #132 before expecting a clean rejoin)
+```
+
+`DRILL_PAIR` defaults to `central`. A site's `/health/*` is on container port 8084, which is not published, so the script probes every pair from a throwaway `alpine` sidecar on `scadabridge-net`. `DRILL_ACTION` defaults to `kill`. `pause` (`docker pause`, then `docker unpause`) models a frozen process or VM.
+
+The rejoin check is strict. The victim must report `/health/ready` 200 and `/health/active` 503, and the survivor must report `memberCount 2, unreachableCount 0`. After that, 10 one-second samples must each show exactly one active node.
+
+**Trying `leader-auto-down` (ScadaBridge#106/#116).** The opt-in overlay `docker/docker-compose.leader-auto-down.yml` switches the central and site-a pairs to the in-repo provider through an environment override. The shipped default stays `auto-down`. Switch both pairs together, never one node at a time:
+
+```bash
+docker compose -f docker/docker-compose.yml -f docker/docker-compose.leader-auto-down.yml \
+  up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b
+# back to the default: the same command without the overlay file
 ```
 
 The scripted drill (`docker kill` = SIGKILL, the hard-crash path — a `docker stop` would take the graceful `CoordinatedShutdown` path and would not prove crash recovery) has **two modes**, and since the **auto-down decision (2026-07-21)** both expect recovery — the cluster runs Akka's `AutoDowning` provider (`auto-down-unreachable-after` = 15s), under which the leader among the *reachable* members downs the unreachable peer, so a crash of either node fails over:
@@ -419,6 +433,17 @@ Self-first ordering is safe, and the three interesting cases are covered by `Sel
 > | `active` (oldest-node crash) | **PASS — TAKEOVER** — `central-b` auto-downed the dead oldest, went `Younger -> Oldest` on all 7 singletons, and served `/health/active` **while the victim was still down**; restarted victim rejoined as standby. | Survivor active + Traefik routing in **28s** (budget ~25s: 10s detection + 15s auto-down + hand-over); victim ready **2s** after restart. |
 > | `standby` (younger-node crash) | **PASS** — active node untouched; survivor downed+removed the crashed member; restarted victim rejoined as standby. | Member removed in **27s**; **0** `/health/active` routing blips; victim ready **2s** after restart. |
 >
+> **Run 2026-11-12, `leader-auto-down` (#116 rig gate).** Every #116 gate passed on central and site-a, within about 2 s of an `auto-down` baseline on the same build (`2d2e9f35`):
+>
+> - Active kill: takeover in 29 s (central) and 26 s (site-a).
+> - Standby kill: downed in 29 s and 26 s, with 0 blips.
+> - Boot logs: no deprecation warning.
+> - Partition: dual-active as accepted, recovered by restarting one side.
+> - Cold start of both pairs: one cluster each.
+> - Site-a S&F drained on the survivor.
+>
+> `docker pause` of the active node leaves a permanent second active after unpause on **both** strategies (#132). Evidence: `docs/plans/2026-11-12-leader-auto-down-rig-drill.md`.
+>
 > Historical baseline (keep-oldest, run 2026-07-13 on `99544985`): `standby` PASS with member removal in 27s / 0 routing blips; `active` was a **total outage** — `central-b` self-downed ~20s after the kill (live SBR log 2026-07-21: `SBR took decision Akka.Cluster.SBR.DownReachable … including myself`) and could not re-bootstrap until `central-a` returned. That result is what motivated the auto-down decision. In-process envelope (`FailoverTimingTests`) measured full failover at **33.7s**.
 
 ### Central Failover