From cafdc0ce60ec84e32447d63e88adc2801ed7a97d Mon Sep 17 00:00:00 2001
From: Joseph Doherty <dohejw01@gmail.com>
Date: Thu, 12 Nov 2026 23:28:34 -0500
Subject: [PATCH] docs(drill): leader-auto-down env2 rollout - all #116 gates
 pass on central and site-x; env2 stays on it as the soak (#116)

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
---
 docker-env2/README.md                         | 21 ++++++
 .../2026-11-12-leader-auto-down-env2-drill.md | 64 +++++++++++++++++++
 .../baseline-boot-deprecation.txt             |  4 ++
 .../baseline-env2-central-active-kill.log     | 32 ++++++++++
 .../baseline-env2-site-x-active-kill.log      | 21 ++++++
 .../final-status.txt                          |  8 +++
 .../lad-boot.txt                              | 14 ++++
 ...ad-env2-central-active-kill-regression.log | 35 ++++++++++
 .../lad-env2-central-active-kill.log          | 35 ++++++++++
 .../lad-env2-central-active-pause.log         | 34 ++++++++++
 .../lad-env2-central-partition-heal.log       | 24 +++++++
 .../lad-env2-central-standby-kill.log         | 25 ++++++++
 .../lad-env2-pause-victim-lines.txt           | 29 +++++++++
 ...lad-env2-site-x-active-kill-regression.log | 23 +++++++
 .../lad-env2-site-x-active-kill.log           | 23 +++++++
 .../lad-env2-site-x-active-pause.log          | 22 +++++++
 .../lad-env2-site-x-partition-heal.log        | 24 +++++++
 .../lad-env2-site-x-standby-kill.log          | 20 ++++++
 .../partition-heal.sh                         | 51 +++++++++++++++
 .../2026-11-12-leader-auto-down-rig-drill.md  |  4 ++
 20 files changed, 513 insertions(+)
 create mode 100644 docs/plans/2026-11-12-leader-auto-down-env2-drill.md
 create mode 100644 docs/plans/2026-11-12-leader-auto-down-env2-drill/baseline-boot-deprecation.txt
 create mode 100644 docs/plans/2026-11-12-leader-auto-down-env2-drill/baseline-env2-central-active-kill.log
 create mode 100644 docs/plans/2026-11-12-leader-auto-down-env2-drill/baseline-env2-site-x-active-kill.log
 create mode 100644 docs/plans/2026-11-12-leader-auto-down-env2-drill/final-status.txt
 create mode 100644 docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-boot.txt
 create mode 100644 docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-active-kill-regression.log
 create mode 100644 docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-active-kill.log
 create mode 100644 docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-active-pause.log
 create mode 100644 docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-partition-heal.log
 create mode 100644 docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-standby-kill.log
 create mode 100644 docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-pause-victim-lines.txt
 create mode 100644 docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-active-kill-regression.log
 create mode 100644 docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-active-kill.log
 create mode 100644 docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-active-pause.log
 create mode 100644 docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-partition-heal.log
 create mode 100644 docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-standby-kill.log
 create mode 100644 docs/plans/2026-11-12-leader-auto-down-env2-drill/partition-heal.sh

diff --git a/docker-env2/README.md b/docker-env2/README.md
index 4f37aa72..da5f4b76 100644
--- a/docker-env2/README.md
+++ b/docker-env2/README.md
@@ -108,6 +108,27 @@ Both UIs are now reachable:
 - Primary: http://localhost:9000
 - Env2:    http://localhost:9100
 
+### Own image tag, and the `leader-auto-down` soak (ScadaBridge#116)
+
+`SCADABRIDGE_IMAGE=<repo:tag>` makes `docker/build.sh` tag that image and this compose file
+run it, so an env2 build never retags the `scadabridge:latest` the primary rig runs. Unset
+keeps the old shared-image behaviour.
+
+Since 2026-11-12 env2 is the soak environment for the `leader-auto-down` rollout and runs
+`scadabridge:env2-116` with the opt-in overlay
+[`docker-compose.leader-auto-down.yml`](docker-compose.leader-auto-down.yml). Keep both
+when recreating nodes, and switch both pairs together:
+
+```bash
+SCADABRIDGE_IMAGE=scadabridge:env2-116 docker compose \
+  -f docker-env2/docker-compose.yml -f docker-env2/docker-compose.leader-auto-down.yml \
+  up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b
+```
+
+Plain `deploy.sh` goes back to `scadabridge:latest` on the default `auto-down`. Drill:
+`DRILL_ENV=env2 DRILL_PAIR=central|site-x bash docker/failover-drill.sh`. Record:
+[`docs/plans/2026-11-12-leader-auto-down-env2-drill.md`](../docs/plans/2026-11-12-leader-auto-down-env2-drill.md).
+
 ### Teardown
 
 ```bash
diff --git a/docs/plans/2026-11-12-leader-auto-down-env2-drill.md b/docs/plans/2026-11-12-leader-auto-down-env2-drill.md
new file mode 100644
index 00000000..7dc65141
--- /dev/null
+++ b/docs/plans/2026-11-12-leader-auto-down-env2-drill.md
@@ -0,0 +1,64 @@
+# leader-auto-down: docker-env2 rollout and failover drill (ScadaBridge#116)
+
+**Date:** 2026-11-12 (UTC 04:11–04:27 on 2026-11-13)
+**Scope:** `docker-env2` only (`scadabridge-env2-central-a/b`, `scadabridge-env2-site-x-a/b`, Traefik `:9100`). Approved by the owner as the next #116 rollout step. The primary rig (`docker/`), the shared infra containers, QA (vq03/vq04) and DEV (vd03) were not touched. The shipped default stays `auto-down`.
+**Build:** image **`scadabridge:env2-116`** (`sha256:42b177cd4dfd…`), built with `SCADABRIDGE_IMAGE=scadabridge:env2-116 bash docker/build.sh` from `b495f429` = `origin/main` `a2e9d718` plus the env2 tooling commit (no code change). `scadabridge:latest`, which the primary rig runs, was not rebuilt or retagged (`sha256:4389897ee144…` before and after).
+**Strategy under test:** `ScadaBridge:Cluster:SplitBrainResolverStrategy = leader-auto-down` on both env2 pairs, with the #132 shut-out restart (default on for that strategy, `ShutOutGrace` 60 s). `StableAfter` 15 s, heartbeat 2 s, failure detection 10 s (env2 values, the same as the rig).
+**Baseline:** `auto-down` on the same build, run first.
+**Raw output:** [`2026-11-12-leader-auto-down-env2-drill/`](2026-11-12-leader-auto-down-env2-drill/) (one file per run).
+
+## Result
+
+**Every #116 gate passes on both env2 pairs.** Timings match the primary-rig run (drill doc `2026-11-12-leader-auto-down-rig-drill.md` §3 and §7) to within a few seconds.
+
+| Gate | env2 central | env2 site-x |
+|---|---|---|
+| Baseline `auto-down`, active kill | Down +27 s, Traefik 29 s, rejoin 2 s, 10/10 | `Leader is auto-downing` +28 s, rejoin 12 s, 10/10 |
+| Baseline boot logs (positive control) | Deprecation warning present on both nodes | Present on both nodes |
+| Kill the active node | **PASS.** Unreachable +13 s, `DOWNING` +28 s, survivor active 28 s, Traefik 32 s, rejoin 2 s, 10/10 | **PASS.** `DOWNING` +27 s, survivor active 27 s, rejoin 1 s, 10/10 |
+| Kill the standby | **PASS.** `DOWNING` +29 s, **0** `/health/active` blips (node and Traefik), rejoin 2 s | **PASS.** `DOWNING` +28 s, **0** blips, rejoin 1 s |
+| Boot logs | **PASS.** 0 deprecation lines; `[leader-auto-down] started …` and `shut-out restart is ON …` on both nodes; drills show `is unreachable; it will be downed if still unreachable after 00:00:15`, then `DOWNING` | **PASS**, same |
+| Freeze (`DRILL_MODE=active DRILL_ACTION=pause`) | **PASS.** Takeover 27 s, Traefik 32 s. At unpause the victim (central-b) logged `shut out by … tiebreak not needed — this node RESTARTS`, then `exit code 75`, and rejoined as standby 3 s after unpause. 10/10 | **PASS.** Takeover 27 s. Victim (site-x-b) `… RESTARTS`, exit 75, rejoined 8 s after unpause. 10/10 |
+| Split past the window, then heal (iptables cut 50 s / 45 s) | **PASS.** Dual-active at +26 s (expected while cut). Healed +50 s. Tiebreak: central-a `STAY`, central-b `RESTART`; **only central-b restarted** (+74 s after heal). One active, 2 members at **+76 s** after heal, 10/10 stable | **PASS.** Dual-active at +26 s. Healed +46 s. site-x-a `STAY`, site-x-b `RESTART`; only site-x-b restarted (+79 s). One active, 2 members at **+80 s** after heal, 10/10 |
+| Kill regression (after the freeze/split runs) | **PASS, unchanged.** Down +28 s, Traefik 30 s, rejoin 2 s, 0 shut-out lines on either node | **PASS, unchanged.** Down +26 s, rejoin 1 s, 0 shut-out lines |
+
+Each rejoin check requires the victim at `/health/ready` 200 and `/health/active` 503, and the survivor at `memberCount 2, unreachableCount 0`; "10/10" is ten one-second samples with exactly one active node.
+
+The central freeze rejoin was 3 s here against 34 s on the primary rig. The difference is the victim's host shutdown: env2's central-b stopped in about 1 s after `exit code 75`, while the rig's central host took about 30 s. It is not a behaviour difference.
+
+## 1. Method
+
+- **Image isolation.** Two committed, default-preserving options (`b495f429`):
+  - `docker/build.sh` reads `SCADABRIDGE_IMAGE` (default `scadabridge:latest`) for the tag.
+  - `docker-env2/docker-compose.yml` runs `${SCADABRIDGE_IMAGE:-scadabridge:latest}` for the four nodes. Unset gives the old behaviour.
+- **Bind mounts.** The compose files came from the worktree, but every run used `--project-directory ~/Desktop/ScadaBridge/docker-env2 -p docker-env2`, so appsettings and logs stayed bind-mounted from the **main checkout**, not from the throwaway worktree. env2's appsettings are identical on `7e34d7c2` (main checkout) and `a2e9d718`.
+- **Strategy switch.** New opt-in overlay [`docker-env2/docker-compose.leader-auto-down.yml`](../../docker-env2/docker-compose.leader-auto-down.yml), mirroring the rig's. It is an environment override; the mounted appsettings still say `auto-down`. Both pairs were switched in one `up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b` at 04:14:11. Each pair formed one 2-member cluster with nobody downed.
+- **Drill script.** `docker/failover-drill.sh` gained `DRILL_ENV=primary|env2` (container prefix `scadabridge-env2-`, Traefik `:9100`, pair `site-x`). The primary default is unchanged.
+- **Split and heal.** The rig's `partition-heal.sh` with an optional `PREFIX` for the container names ([copy](2026-11-12-leader-auto-down-env2-drill/partition-heal.sh)). It needs the local `sb-drill-net:local` image (alpine + iptables + curl).
+
+## 2. Limits of this run
+
+- **env2 is not seeded.** `ScadaBridgeConfig2` has 0 sites and 0 instances, and site-x reports 0 data connections, so this run proves cluster mechanics only. The rig run's data-path and store-and-forward check (§3 of the rig doc) was not repeated here. `docker-env2/seed-sites.sh` would bind a data connection to the windev MxGateway (`10.100.0.48:5120`); it was not run, because that reaches outside env2.
+- **Transport smoke skipped.** `docker-env2/README.md` points only to `2026-05-24-second-environment-verification.md`, which is marked RETIRED (superseded by the Transport integration tests and the 2026-08-01 live run). The primary rig was also in use by another agent (a burst of about 300 `/management` calls at 04:15–04:17). No export or import was run.
+
+## 3. Final state: env2 stays on `leader-auto-down` (soak)
+
+Every gate passed, with no regressions against the baseline and no operator action needed in any scenario. So env2 is **left on `leader-auto-down`** as the soak environment for the rollout ([final status](2026-11-12-leader-auto-down-env2-drill/final-status.txt)):
+
+- All four nodes run `scadabridge:env2-116` with the overlay env set, all ready, each `memberCount 2, unreachableCount 0`.
+- The active nodes are **central-b** and **site-x-b**. Traefik `:9100` answers `/health/active` 200.
+- After the drills, central-b and site-x-b show RestartCount 2 (freeze plus split-heal); central-a and site-x-a show 0.
+
+**To keep the soak state** when touching env2, always pass both the image and the overlay:
+
+```bash
+SCADABRIDGE_IMAGE=scadabridge:env2-116 docker compose \
+  -f docker-env2/docker-compose.yml -f docker-env2/docker-compose.leader-auto-down.yml \
+  up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b
+```
+
+A plain `bash docker-env2/deploy.sh` rebuilds and runs `scadabridge:latest` on `auto-down`, which ends the soak. **Rollback** is the same command without the overlay file, applied to both pairs together.
+
+## 4. Next step
+
+QA pair vq03/vq04, which **needs owner approval**. On QA, check the exit-75 restart under the Windows service supervisor (#116 comment of 2026-11-12: with `failureflag 1`, recovery fires only on a non-zero exit, and the Host only started exiting 75 with #132). Then DEV vd03 and its sites, then the default flip.
diff --git a/docs/plans/2026-11-12-leader-auto-down-env2-drill/baseline-boot-deprecation.txt b/docs/plans/2026-11-12-leader-auto-down-env2-drill/baseline-boot-deprecation.txt
new file mode 100644
index 00000000..44193941
--- /dev/null
+++ b/docs/plans/2026-11-12-leader-auto-down-env2-drill/baseline-boot-deprecation.txt
@@ -0,0 +1,4 @@
+central-a deprecation=1
+central-b deprecation=1
+site-x-a deprecation=1
+site-x-b deprecation=1
diff --git a/docs/plans/2026-11-12-leader-auto-down-env2-drill/baseline-env2-central-active-kill.log b/docs/plans/2026-11-12-leader-auto-down-env2-drill/baseline-env2-central-active-kill.log
new file mode 100644
index 00000000..4851c5cc
--- /dev/null
+++ b/docs/plans/2026-11-12-leader-auto-down-env2-drill/baseline-env2-central-active-kill.log
@@ -0,0 +1,32 @@
+env=env2  pair=central  mode=active  action=kill  active=scadabridge-env2-central-a  victim=scadabridge-env2-central-a  survivor=scadabridge-env2-central-b
+hit at 2026-11-13T04:12:15Z
+Active kill: waiting for scadabridge-env2-central-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
+  survivor logged the Down at +27s
+PASS: scadabridge-env2-central-b took over as active in 27s with the victim still down
+(downed the dead oldest, assumed Oldest, re-hosted the singletons).
+Confirming Traefik routes to the new active node...
+Traefik routing recovered 29s after the hit.
+Recovering scadabridge-env2-central-a (kill -> start)...
+Waiting for the victim to rejoin as a ready standby (scadabridge-env2-central-a /health/ready)...
+Recovered: scadabridge-env2-central-a is ready as standby (rejoined) 2s after recovery.
+Checking for a lingering dual-active (10 samples, 1s apart)...
+Exactly one active node in 10/10 samples.
+Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-central-b):
+[04:12:41 INF] [Central/scadabridge-env2-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] - Leader is auto-downing unreachable node [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
+[04:12:41 INF] [Central/scadabridge-env2-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-central-a:8081] as [Down]
+[04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]]
+[04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
+[04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]]
+[04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
+[04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]]
+[04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]]
+[04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
+[04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]]
+[04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
+[04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
+[04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]]
+[04:12:42 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 1724624009)"]]
+[04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
+[04:12:42 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
+SUMMARY env=env2 pair=central mode=active action=kill down_logged_s=27 rejoin_s=2
+Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
diff --git a/docs/plans/2026-11-12-leader-auto-down-env2-drill/baseline-env2-site-x-active-kill.log b/docs/plans/2026-11-12-leader-auto-down-env2-drill/baseline-env2-site-x-active-kill.log
new file mode 100644
index 00000000..f64f102b
--- /dev/null
+++ b/docs/plans/2026-11-12-leader-auto-down-env2-drill/baseline-env2-site-x-active-kill.log
@@ -0,0 +1,21 @@
+env=env2  pair=site-x  mode=active  action=kill  active=scadabridge-env2-site-x-a  victim=scadabridge-env2-site-x-a  survivor=scadabridge-env2-site-x-b
+hit at 2026-11-13T04:13:06Z
+Active kill: waiting for scadabridge-env2-site-x-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
+  survivor logged the Down at +28s
+PASS: scadabridge-env2-site-x-b took over as active in 28s with the victim still down
+(downed the dead oldest, assumed Oldest, re-hosted the singletons).
+Recovering scadabridge-env2-site-x-a (kill -> start)...
+Waiting for the victim to rejoin as a ready standby (scadabridge-env2-site-x-a /health/ready)...
+Recovered: scadabridge-env2-site-x-a is ready as standby (rejoined) 12s after recovery.
+Checking for a lingering dual-active (10 samples, 1s apart)...
+Exactly one active node in 10/10 samples.
+Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-site-x-b):
+[04:13:34 INF] [Site/scadabridge-env2-site-x-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] - Leader is auto-downing unreachable node [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
+[04:13:34 INF] [Site/scadabridge-env2-site-x-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082] as [Down]
+[04:13:35 INF] [Site/scadabridge-env2-site-x-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] - New incarnation of existing member [UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082, 1261810842)] is trying to join. Existing will be removed from the cluster and then new member will be allowed to join.
+[04:13:35 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
+[04:13:35 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
+[04:13:35 INF] [Site/scadabridge-env2-site-x-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:13:35 INF] [Site/scadabridge-env2-site-x-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+SUMMARY env=env2 pair=site-x mode=active action=kill down_logged_s=28 rejoin_s=12
+Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
diff --git a/docs/plans/2026-11-12-leader-auto-down-env2-drill/final-status.txt b/docs/plans/2026-11-12-leader-auto-down-env2-drill/final-status.txt
new file mode 100644
index 00000000..4d6a003c
--- /dev/null
+++ b/docs/plans/2026-11-12-leader-auto-down-env2-drill/final-status.txt
@@ -0,0 +1,8 @@
+Fri Nov 13 04:27:28 UTC 2026
+image: sha256:42b177cd4dfd69e7a492d9dea003ba15d35b53cba11e6295025d1bd6b55484fd (scadabridge:env2-116, built from b495f429 = origin/main a2e9d718 + env2 tooling)
+scadabridge:latest untouched: sha256:4389897ee144a8c2cef98c5fede74b4a5bf5e4cd0cdc07788b23d58af4c6e5fc
+scadabridge-env2-central-a image=scadabridge:env2-116 state=running restarts=0 ready=200 active=503 "memberCount":2,"unreachableCount":0 ScadaBridge__Cluster__SplitBrainResolverStrategy=leader-auto-down
+scadabridge-env2-central-b image=scadabridge:env2-116 state=running restarts=2 ready=200 active=200 "memberCount":2,"unreachableCount":0 ScadaBridge__Cluster__SplitBrainResolverStrategy=leader-auto-down
+scadabridge-env2-site-x-a image=scadabridge:env2-116 state=running restarts=0 ready=200 active=503 "memberCount":2,"unreachableCount":0 ScadaBridge__Cluster__SplitBrainResolverStrategy=leader-auto-down
+scadabridge-env2-site-x-b image=scadabridge:env2-116 state=running restarts=2 ready=200 active=200 "memberCount":2,"unreachableCount":0 ScadaBridge__Cluster__SplitBrainResolverStrategy=leader-auto-down
+traefik :9100 /health/active 200
diff --git a/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-boot.txt b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-boot.txt
new file mode 100644
index 00000000..291a2daf
--- /dev/null
+++ b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-boot.txt
@@ -0,0 +1,14 @@
+central-a active=200 "memberCount":2,"unreachableCount":0 strategy_env=ScadaBridge__Cluster__SplitBrainResolverStrategy=leader-auto-down dep=0
+[04:14:24 INF] [Central/scadabridge-env2-central-a] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-env2-central-a:8081: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
+[04:14:24 INF] [Central/scadabridge-env2-central-a] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@scadabridge-env2-central-a:8081: a node split from its pair peer restarts itself — at once if only it was removed, otherwise the higher add
+[04:14:25 INF] [Central/scadabridge-env2-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-a:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-central-a:8081); pending unreachable=[]
+central-b active=503 "memberCount":2,"unreachableCount":0 strategy_env=ScadaBridge__Cluster__SplitBrainResolverStrategy=leader-auto-down dep=0
+[04:14:24 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@scadabridge-env2-central-b:8081: a node split from its pair peer restarts itself — at once if only it was removed, otherwise the higher add
+[04:14:24 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-env2-central-b:8081: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
+site-x-a active=200 "memberCount":2,"unreachableCount":0 strategy_env=ScadaBridge__Cluster__SplitBrainResolverStrategy=leader-auto-down dep=0
+[04:14:23 INF] [Site/scadabridge-env2-site-x-a] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
+[04:14:23 INF] [Site/scadabridge-env2-site-x-a] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: a node split from its pair peer restarts itself — at once if only it was removed, otherwise the higher address 
+[04:14:24 INF] [Site/scadabridge-env2-site-x-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082); pending unreachable=[]
+site-x-b active=503 "memberCount":2,"unreachableCount":0 strategy_env=ScadaBridge__Cluster__SplitBrainResolverStrategy=leader-auto-down dep=0
+[04:14:23 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: a node split from its pair peer restarts itself — at once if only it was removed, otherwise the higher address 
+[04:14:23 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
diff --git a/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-active-kill-regression.log b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-active-kill-regression.log
new file mode 100644
index 00000000..14d92e57
--- /dev/null
+++ b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-active-kill-regression.log
@@ -0,0 +1,35 @@
+env=env2  pair=central  mode=active  action=kill  active=scadabridge-env2-central-a  victim=scadabridge-env2-central-a  survivor=scadabridge-env2-central-b
+hit at 2026-11-13T04:25:04Z
+Active kill: waiting for scadabridge-env2-central-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
+  survivor logged the Down at +28s
+PASS: scadabridge-env2-central-b took over as active in 28s with the victim still down
+(downed the dead oldest, assumed Oldest, re-hosted the singletons).
+Confirming Traefik routes to the new active node...
+Traefik routing recovered 30s after the hit.
+Recovering scadabridge-env2-central-a (kill -> start)...
+Waiting for the victim to rejoin as a ready standby (scadabridge-env2-central-a /health/ready)...
+Recovered: scadabridge-env2-central-a is ready as standby (rejoined) 2s after recovery.
+Checking for a lingering dual-active (10 samples, 1s apart)...
+Exactly one active node in 10/10 samples.
+Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-central-b):
+[04:25:16 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-central-b:8081 is leader: False)
+[04:25:16 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-central-b:8081); pending unreachable=[]
+[04:25:31 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-central-b:8081) is leader
+[04:25:31 INF] [Central/scadabridge-env2-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-central-a:8081] as [Down]
+[04:25:32 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
+[04:25:32 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
+[04:25:32 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
+[04:25:32 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
+[04:25:32 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
+[04:25:32 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
+[04:25:32 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
+[04:25:32 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:25:32 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:25:32 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:25:32 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:25:32 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:25:32 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:25:32 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:25:35 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-central-a:8081); pending unreachable=[]
+SUMMARY env=env2 pair=central mode=active action=kill down_logged_s=28 rejoin_s=2
+Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
diff --git a/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-active-kill.log b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-active-kill.log
new file mode 100644
index 00000000..2ecab74b
--- /dev/null
+++ b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-active-kill.log
@@ -0,0 +1,35 @@
+env=env2  pair=central  mode=active  action=kill  active=scadabridge-env2-central-a  victim=scadabridge-env2-central-a  survivor=scadabridge-env2-central-b
+hit at 2026-11-13T04:14:52Z
+Active kill: waiting for scadabridge-env2-central-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
+  survivor logged the Down at +28s
+PASS: scadabridge-env2-central-b took over as active in 28s with the victim still down
+(downed the dead oldest, assumed Oldest, re-hosted the singletons).
+Confirming Traefik routes to the new active node...
+Traefik routing recovered 32s after the hit.
+Recovering scadabridge-env2-central-a (kill -> start)...
+Waiting for the victim to rejoin as a ready standby (scadabridge-env2-central-a /health/ready)...
+Recovered: scadabridge-env2-central-a is ready as standby (rejoined) 2s after recovery.
+Checking for a lingering dual-active (10 samples, 1s apart)...
+Exactly one active node in 10/10 samples.
+Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-central-b):
+[04:15:05 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-central-b:8081 is leader: False)
+[04:15:05 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-central-b:8081); pending unreachable=[]
+[04:15:20 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-central-b:8081) is leader
+[04:15:20 INF] [Central/scadabridge-env2-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-central-a:8081] as [Down]
+[04:15:21 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
+[04:15:21 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:15:21 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 686527974)"]]
+[04:15:21 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
+[04:15:21 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 686527974)"]]
+[04:15:21 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
+[04:15:21 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 686527974)"]]
+[04:15:21 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
+[04:15:21 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 686527974)"]]
+[04:15:21 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
+[04:15:21 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 686527974)"]]
+[04:15:21 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081], previous oldest [["UniqueAddress: (akka.tcp://scadabridge@scadabridge-env2-central-a:8081, 686527974)"]]
+[04:15:21 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
+[04:15:21 INF] [Central/scadabridge-env2-central-b] ClusterSingletonManager state change [BecomingOldest -> Oldest] Akka.Cluster.Tools.Singleton.BecomingOldestData
+[04:15:25 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-central-a:8081); pending unreachable=[]
+SUMMARY env=env2 pair=central mode=active action=kill down_logged_s=28 rejoin_s=2
+Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
diff --git a/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-active-pause.log b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-active-pause.log
new file mode 100644
index 00000000..1a0ec1ba
--- /dev/null
+++ b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-active-pause.log
@@ -0,0 +1,34 @@
+env=env2  pair=central  mode=active  action=pause  active=scadabridge-env2-central-b  victim=scadabridge-env2-central-b  survivor=scadabridge-env2-central-a
+hit at 2026-11-13T04:17:52Z
+Active pause: waiting for scadabridge-env2-central-a to take over as the active node (victim stays down; budget ~25s + hand-over)...
+  survivor logged the Down at +27s
+PASS: scadabridge-env2-central-a took over as active in 27s with the victim still down
+(downed the dead oldest, assumed Oldest, re-hosted the singletons).
+Confirming Traefik routes to the new active node...
+Traefik routing recovered 32s after the hit.
+Recovering scadabridge-env2-central-b (pause -> unpause)...
+Waiting for the victim to rejoin as a ready standby (scadabridge-env2-central-b /health/ready)...
+Recovered: scadabridge-env2-central-b is ready as standby (rejoined) 3s after recovery.
+Checking for a lingering dual-active (10 samples, 1s apart)...
+Exactly one active node in 10/10 samples.
+Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-central-a):
+[04:18:03 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-central-a:8081 is leader: True)
+[04:18:18 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-central-b:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-central-a:8081) is leader
+[04:18:18 INF] [Central/scadabridge-env2-central-a] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-central-a:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] as [Down]
+[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
+[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
+[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
+[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
+[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
+[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
+[04:18:19 INF] [Central/scadabridge-env2-central-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-b:8081]
+[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:18:19 INF] [Central/scadabridge-env2-central-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:18:24 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-central-b:8081: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-env2-central-a:8081) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the tiebreak (a frozen node that was removed restarts on its own)
+SUMMARY env=env2 pair=central mode=active action=pause down_logged_s=27 rejoin_s=3
+Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
diff --git a/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-partition-heal.log b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-partition-heal.log
new file mode 100644
index 00000000..76979afc
--- /dev/null
+++ b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-partition-heal.log
@@ -0,0 +1,24 @@
+04:19:53 before: scadabridge-env2-central-a active=200 scadabridge-env2-central-b active=503  restarts A=0/2026-11-13T04:16:07.324214092Z B=1/2026-11-13T04:18:25.981120308Z  (A ip 192.168.147.15)
+04:19:53 partition ON (B drops all traffic to/from A)
+04:20:19 DUAL-ACTIVE reached at +26s
+04:20:43 end of hold (+50s): scadabridge-env2-central-a active=200 scadabridge-env2-central-b active=200
+04:20:43 partition HEALED (+50s)
+04:21:57 +74s after heal: A stayed, B RESTARTED
+04:21:59 one active (scadabridge-env2-central-a=200 scadabridge-env2-central-b=503), "memberCount":2 "memberCount":2 — converged +76s after heal
+04:22:20 final: scadabridge-env2-central-a active=200 ready=200 restarts=0/2026-11-13T04:16:07.324214092Z | scadabridge-env2-central-b active=503 ready=200 restarts=2/2026-11-13T04:21:56.583595766Z
+result: restart=[+74s after heal: A stayed, B RESTARTED] converged=[+76s after heal] stable-samples=10
+--- scadabridge-env2-central-a [leader-auto-down] + quarantine lines
+[04:20:05 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-central-a:8081 is leader: True)
+[04:20:20 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-central-b:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-central-a:8081) is leader
+[04:20:56 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-central-b:8081: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-env2-central-a:8081) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the ti
+[04:21:56 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-central-b:8081: its old incarnation is still running 00:01:00 later and it has not restarted itself; both sides are shut out. Tiebreak says STAY: this node (akka.tcp://scadabridge@scadabridge-env2-central-a:8
+--- scadabridge-env2-central-b [leader-auto-down] + quarantine lines
+[04:20:04 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-central-b:8081 is leader: False)
+[04:20:04 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-central-b:8081); pending unreachable=[]
+[04:20:19 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-central-b:8081) is leader
+[04:20:55 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-central-a:8081: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-env2-central-b:8081) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the ti
+[04:21:55 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-central-a:8081: its old incarnation is still running 00:01:00 later and it has not restarted itself; both sides are shut out. Tiebreak says RESTART: this node (akka.tcp://scadabridge@scadabridge-env2-central-
+[04:21:55 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 lost leadership (leader is now none); pending unreachable=[]
+[04:21:56 FTL] [Central/scadabridge-env2-central-b] ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down / leader-auto-down shut-out restart). Stopping the host process with exit code 75 so the service supervisor restarts this node as a fresh incarnation.
+[04:21:57 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@scadabridge-env2-central-b:8081: a node split from its pair peer restarts itself — at once if only it was removed, otherwise the higher address restarts after a 00:01:00 grace (ScadaBridge#132)
+[04:21:57 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-env2-central-b:8081: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
diff --git a/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-standby-kill.log b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-standby-kill.log
new file mode 100644
index 00000000..bcdf09b5
--- /dev/null
+++ b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-central-standby-kill.log
@@ -0,0 +1,25 @@
+env=env2  pair=central  mode=standby  action=kill  active=scadabridge-env2-central-b  victim=scadabridge-env2-central-a  survivor=scadabridge-env2-central-b
+hit at 2026-11-13T04:15:38Z
+Standby kill: waiting for scadabridge-env2-central-b to DOWN+REMOVE the victim (budget ~25s)...
+PASS: survivor downed/removed the victim in 29s (budget ~25s: 10s detection + 15s down window).
+Active-node /health/active blips during the drill: 0 (expected 0 — the active node was never touched).
+Recovering scadabridge-env2-central-a (kill -> start)...
+Waiting for the victim to rejoin as a ready standby (scadabridge-env2-central-a /health/ready)...
+Recovered: scadabridge-env2-central-a is ready as standby (rejoined) 2s after recovery.
+Checking for a lingering dual-active (10 samples, 1s apart)...
+Exactly one active node in 10/10 samples.
+Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-central-b):
+[04:15:51 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-a:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-central-b:8081 is leader: False)
+[04:15:51 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-central-b:8081); pending unreachable=[]
+[04:16:06 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-central-a:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-central-b:8081) is leader
+[04:16:06 INF] [Central/scadabridge-env2-central-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-central-a:8081] as [Down]
+[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
+[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
+[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
+[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
+[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
+[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
+[04:16:07 INF] [Central/scadabridge-env2-central-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-central-a:8081]
+[04:16:09 INF] [Central/scadabridge-env2-central-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-central-a:8081); pending unreachable=[]
+SUMMARY env=env2 pair=central mode=standby action=kill down_logged_s=29 rejoin_s=2
+Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
diff --git a/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-pause-victim-lines.txt b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-pause-victim-lines.txt
new file mode 100644
index 00000000..8291995f
--- /dev/null
+++ b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-pause-victim-lines.txt
@@ -0,0 +1,29 @@
+== central-b
+[04:18:24 ERR] [Central/scadabridge-env2-central-b] AssociationError [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] <- akka.tcp://scadabridge@scadabridge-env2-central-a:8081: Error [The remote system has quarantined this system. No further associations to the remote system are possible unt
+[04:18:24 WRN] [Central/scadabridge-env2-central-b] [leader-auto-down] shut out by akka.tcp://scadabridge@scadabridge-env2-central-a:8081: it has removed and quarantined this node (akka.tcp://scadabridge@scadabridge-env2-central-b:8081) while this node still counts it as a member (Up), so only this 
+[04:18:24 WRN] [Central/scadabridge-env2-central-b] Dead letter: HandleListenerRegistered from [akka://scadabridge/deadLetters] to [akka://scadabridge/system/transports/akkaprotocolmanager.tcp.0/akkaProtocol-tcp%3A%2F%2Fscadabridge%40scadabridge-env2-central-a%3A8081-59#67067550]
+[04:18:24 ERR] [Central/scadabridge-env2-central-b] AssociationError [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] -> akka.tcp://scadabridge@scadabridge-env2-central-a:8081: Error [The remote system has quarantined this system. No further associations to the remote system are possible unt
+[04:18:25 ERR] [Central/scadabridge-env2-central-b] Health check akka-cluster with status Unhealthy completed after 0.0475ms with message 'Akka cluster member status: Down'
+[04:18:25 INF] [Central/scadabridge-env2-central-b] CoordinatedShutdown invoked due to [Reason:Akka.Actor.CoordinatedShutdown+ClusterDowningReason]. Exiting with [ExitCode:3].
+[04:18:25 ERR] [Central/scadabridge-env2-central-b] AssociationError [akka.tcp://scadabridge@scadabridge-env2-central-b:8081] -> akka.tcp://scadabridge@scadabridge-env2-central-a:8081: Error [The remote system has quarantined this system. No further associations to the remote system are possible unt
+[04:18:25 FTL] [Central/scadabridge-env2-central-b] ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down / leader-auto-down shut-out restart). Stopping the host process with exit code 75 so the service supervisor restarts this node as a fresh incarnation.
+== site-x-b
+[04:17:41 INF] [Site/scadabridge-env2-site-x-b] Request finished HTTP/1.1 GET http://scadabridge-env2-site-x-b:8084/health/active - 200 null application/json; charset=utf-8 0.2759ms
+[04:19:11 ERR] [Site/scadabridge-env2-site-x-b] AssociationError [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] -> akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: Error [The remote system has quarantined this system. No further associations to the remote system are possible until thi
+[04:19:11 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] shut out by akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: it has removed and quarantined this node (akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082) while this node still counts it as a member (Up), so only this side i
+[04:19:11 ERR] [Site/scadabridge-env2-site-x-b] AssociationError [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] -> akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: Error [The remote system has quarantined this system. No further associations to the remote system are possible until thi
+[04:19:12 ERR] [Site/scadabridge-env2-site-x-b] AssociationError [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] <- akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: Error [The remote system has quarantined this system. No further associations to the remote system are possible until thi
+[04:19:12 ERR] [Site/scadabridge-env2-site-x-b] AssociationError [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] -> akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: Error [The remote system has quarantined this system. No further associations to the remote system are possible until thi
+[04:19:12 INF] [Site/scadabridge-env2-site-x-b] CoordinatedShutdown invoked due to [Reason:Akka.Actor.CoordinatedShutdown+ClusterDowningReason]. Exiting with [ExitCode:3].
+[04:19:12 ERR] [Site/scadabridge-env2-site-x-b] AssociationError [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] -> akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: Error [The remote system has quarantined this system. No further associations to the remote system are possible until thi
+== survivor central-a
+[04:18:03 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-central-a:808
+[04:18:18 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-central-b:8081: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-central-a:8081) is leader
+[04:18:24 WRN] [Central/scadabridge-env2-central-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-central-b:8081: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-env2-central-a:8081) removed, so this no
+[04:19:24 INF] [Central/scadabridge-env2-central-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-central-b:8081 rejoined (Up) within the shut-out grace; this node (akka.tcp://scadabridge@scadabridge-env2-central-a:8081) stays
+== survivor site-x-a
+[04:18:51 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082 is l
+[04:19:06 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082) is leader
+[04:19:12 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082) removed, so this node has
+central-b: RESTARTS=1 exit75=1
+site-x-b: RESTARTS=1 exit75=1
diff --git a/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-active-kill-regression.log b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-active-kill-regression.log
new file mode 100644
index 00000000..fe81c35c
--- /dev/null
+++ b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-active-kill-regression.log
@@ -0,0 +1,23 @@
+env=env2  pair=site-x  mode=active  action=kill  active=scadabridge-env2-site-x-a  victim=scadabridge-env2-site-x-a  survivor=scadabridge-env2-site-x-b
+hit at 2026-11-13T04:25:49Z
+Active kill: waiting for scadabridge-env2-site-x-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
+  survivor logged the Down at +26s
+PASS: scadabridge-env2-site-x-b took over as active in 26s with the victim still down
+(downed the dead oldest, assumed Oldest, re-hosted the singletons).
+Recovering scadabridge-env2-site-x-a (kill -> start)...
+Waiting for the victim to rejoin as a ready standby (scadabridge-env2-site-x-a /health/ready)...
+Recovered: scadabridge-env2-site-x-a is ready as standby (rejoined) 1s after recovery.
+Checking for a lingering dual-active (10 samples, 1s apart)...
+Exactly one active node in 10/10 samples.
+Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-site-x-b):
+[04:26:00 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 is leader: False)
+[04:26:00 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082); pending unreachable=[]
+[04:26:15 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082) is leader
+[04:26:15 INF] [Site/scadabridge-env2-site-x-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082] as [Down]
+[04:26:16 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
+[04:26:16 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
+[04:26:16 INF] [Site/scadabridge-env2-site-x-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:26:16 INF] [Site/scadabridge-env2-site-x-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:26:17 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082); pending unreachable=[]
+SUMMARY env=env2 pair=site-x mode=active action=kill down_logged_s=26 rejoin_s=1
+Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
diff --git a/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-active-kill.log b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-active-kill.log
new file mode 100644
index 00000000..d9c384c7
--- /dev/null
+++ b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-active-kill.log
@@ -0,0 +1,23 @@
+env=env2  pair=site-x  mode=active  action=kill  active=scadabridge-env2-site-x-a  victim=scadabridge-env2-site-x-a  survivor=scadabridge-env2-site-x-b
+hit at 2026-11-13T04:16:22Z
+Active kill: waiting for scadabridge-env2-site-x-b to take over as the active node (victim stays down; budget ~25s + hand-over)...
+  survivor logged the Down at +27s
+PASS: scadabridge-env2-site-x-b took over as active in 27s with the victim still down
+(downed the dead oldest, assumed Oldest, re-hosted the singletons).
+Recovering scadabridge-env2-site-x-a (kill -> start)...
+Waiting for the victim to rejoin as a ready standby (scadabridge-env2-site-x-a /health/ready)...
+Recovered: scadabridge-env2-site-x-a is ready as standby (rejoined) 1s after recovery.
+Checking for a lingering dual-active (10 samples, 1s apart)...
+Exactly one active node in 10/10 samples.
+Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-site-x-b):
+[04:16:33 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 is leader: False)
+[04:16:33 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082); pending unreachable=[]
+[04:16:48 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082) is leader
+[04:16:48 INF] [Site/scadabridge-env2-site-x-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082] as [Down]
+[04:16:49 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
+[04:16:49 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
+[04:16:49 INF] [Site/scadabridge-env2-site-x-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:16:49 INF] [Site/scadabridge-env2-site-x-b] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:16:50 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082); pending unreachable=[]
+SUMMARY env=env2 pair=site-x mode=active action=kill down_logged_s=27 rejoin_s=1
+Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
diff --git a/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-active-pause.log b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-active-pause.log
new file mode 100644
index 00000000..a2fdab9d
--- /dev/null
+++ b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-active-pause.log
@@ -0,0 +1,22 @@
+env=env2  pair=site-x  mode=active  action=pause  active=scadabridge-env2-site-x-b  victim=scadabridge-env2-site-x-b  survivor=scadabridge-env2-site-x-a
+hit at 2026-11-13T04:18:39Z
+Active pause: waiting for scadabridge-env2-site-x-a to take over as the active node (victim stays down; budget ~25s + hand-over)...
+  survivor logged the Down at +27s
+PASS: scadabridge-env2-site-x-a took over as active in 27s with the victim still down
+(downed the dead oldest, assumed Oldest, re-hosted the singletons).
+Recovering scadabridge-env2-site-x-b (pause -> unpause)...
+Waiting for the victim to rejoin as a ready standby (scadabridge-env2-site-x-b /health/ready)...
+Recovered: scadabridge-env2-site-x-b is ready as standby (rejoined) 8s after recovery.
+Checking for a lingering dual-active (10 samples, 1s apart)...
+Exactly one active node in 10/10 samples.
+Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-site-x-a):
+[04:18:51 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082 is leader: True)
+[04:19:06 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082) is leader
+[04:19:06 INF] [Site/scadabridge-env2-site-x-a] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] as [Down]
+[04:19:07 INF] [Site/scadabridge-env2-site-x-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082]
+[04:19:07 INF] [Site/scadabridge-env2-site-x-a] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082]
+[04:19:07 INF] [Site/scadabridge-env2-site-x-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:19:07 INF] [Site/scadabridge-env2-site-x-a] ClusterSingletonManager state change [Younger -> Oldest] Akka.Cluster.Tools.Singleton.YoungerData
+[04:19:12 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the tiebreak (a frozen node that was removed restarts on its own)
+SUMMARY env=env2 pair=site-x mode=active action=pause down_logged_s=27 rejoin_s=8
+Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
diff --git a/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-partition-heal.log b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-partition-heal.log
new file mode 100644
index 00000000..60d90ff2
--- /dev/null
+++ b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-partition-heal.log
@@ -0,0 +1,24 @@
+04:22:29 before: scadabridge-env2-site-x-a active=200 scadabridge-env2-site-x-b active=503  restarts A=0/2026-11-13T04:17:31.539611497Z B=1/2026-11-13T04:19:13.144396278Z  (A ip 192.168.147.18)
+04:22:29 partition ON (B drops all traffic to/from A)
+04:22:55 DUAL-ACTIVE reached at +26s
+04:23:14 end of hold (+45s): scadabridge-env2-site-x-a active=200 scadabridge-env2-site-x-b active=200
+04:23:15 partition HEALED (+46s)
+04:24:34 +79s after heal: A stayed, B RESTARTED
+04:24:35 one active (scadabridge-env2-site-x-a=200 scadabridge-env2-site-x-b=503), "memberCount":2 "memberCount":2 — converged +80s after heal
+04:24:55 final: scadabridge-env2-site-x-a active=200 ready=200 restarts=0/2026-11-13T04:17:31.539611497Z | scadabridge-env2-site-x-b active=503 ready=200 restarts=2/2026-11-13T04:24:33.135347935Z
+result: restart=[+79s after heal: A stayed, B RESTARTED] converged=[+80s after heal] stable-samples=10
+--- scadabridge-env2-site-x-a [leader-auto-down] + quarantine lines
+[04:22:41 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082 is leader: True)
+[04:22:56 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082) is leader
+[04:23:32 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the tiebreak
+[04:24:32 WRN] [Site/scadabridge-env2-site-x-a] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: its old incarnation is still running 00:01:00 later and it has not restarted itself; both sides are shut out. Tiebreak says STAY: this node (akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082) h
+--- scadabridge-env2-site-x-b [leader-auto-down] + quarantine lines
+[04:22:40 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 is leader: False)
+[04:22:40 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082); pending unreachable=[]
+[04:22:55 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082) is leader
+[04:23:31 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: it is still running the incarnation this node (akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082) removed, so this node has shut it out. Giving it 00:01:00 to restart itself before the tiebreak
+[04:24:31 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] split from akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: its old incarnation is still running 00:01:00 later and it has not restarted itself; both sides are shut out. Tiebreak says RESTART: this node (akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082
+[04:24:31 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 lost leadership (leader is now none); pending unreachable=[]
+[04:24:32 FTL] [Site/scadabridge-env2-site-x-b] ActorSystem terminated outside host shutdown (SBR down / run-coordinated-shutdown-when-down / leader-auto-down shut-out restart). Stopping the host process with exit code 75 so the service supervisor restarts this node as a fresh incarnation.
+[04:24:33 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] shut-out restart is ON for akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: a node split from its pair peer restarts itself — at once if only it was removed, otherwise the higher address restarts after a 00:01:00 grace (ScadaBridge#132)
+[04:24:33 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] started on akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082: leader=none (this node is leader: False), unreachable-after=00:00:15, already unreachable=[]
diff --git a/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-standby-kill.log b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-standby-kill.log
new file mode 100644
index 00000000..b0e240a2
--- /dev/null
+++ b/docs/plans/2026-11-12-leader-auto-down-env2-drill/lad-env2-site-x-standby-kill.log
@@ -0,0 +1,20 @@
+env=env2  pair=site-x  mode=standby  action=kill  active=scadabridge-env2-site-x-b  victim=scadabridge-env2-site-x-a  survivor=scadabridge-env2-site-x-b
+hit at 2026-11-13T04:17:02Z
+Standby kill: waiting for scadabridge-env2-site-x-b to DOWN+REMOVE the victim (budget ~25s)...
+PASS: survivor downed/removed the victim in 28s (budget ~25s: 10s detection + 15s down window).
+Active-node /health/active blips during the drill: 0 (expected 0 — the active node was never touched).
+Recovering scadabridge-env2-site-x-a (kill -> start)...
+Waiting for the victim to rejoin as a ready standby (scadabridge-env2-site-x-a /health/ready)...
+Recovered: scadabridge-env2-site-x-a is ready as standby (rejoined) 1s after recovery.
+Checking for a lingering dual-active (10 samples, 1s apart)...
+Exactly one active node in 10/10 samples.
+Survivor downing/singleton evidence (last 20 matching log lines from scadabridge-env2-site-x-b):
+[04:17:15 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082 is unreachable; it will be downed if still unreachable after 00:00:15 (this node akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 is leader: False)
+[04:17:15 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 gained leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082); pending unreachable=[]
+[04:17:30 WRN] [Site/scadabridge-env2-site-x-b] [leader-auto-down] DOWNING akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082: unreachable for 00:00:15 and this node (akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082) is leader
+[04:17:30 INF] [Site/scadabridge-env2-site-x-b] Cluster Node [akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082] - Marking unreachable node [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082] as [Down]
+[04:17:31 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
+[04:17:31 INF] [Site/scadabridge-env2-site-x-b] Member removed [akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082]
+[04:17:32 INF] [Site/scadabridge-env2-site-x-b] [leader-auto-down] akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082 lost leadership (leader is now akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082); pending unreachable=[]
+SUMMARY env=env2 pair=site-x mode=standby action=kill down_logged_s=28 rejoin_s=1
+Drill complete. Verify on the Health dashboard / Overview that both nodes show Up and exactly one is Primary.
diff --git a/docs/plans/2026-11-12-leader-auto-down-env2-drill/partition-heal.sh b/docs/plans/2026-11-12-leader-auto-down-env2-drill/partition-heal.sh
new file mode 100644
index 00000000..860a5542
--- /dev/null
+++ b/docs/plans/2026-11-12-leader-auto-down-env2-drill/partition-heal.sh
@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+# ScadaBridge#132 rig gate: partition a pair past the leader-auto-down window, heal it, and
+# watch the shut-out restart converge the pair WITHOUT operator action.
+# Usage: partition-heal.sh <pair> <hold_s> [watch_s]
+# Needs: sb-drill-net:local (alpine + iptables + curl) and a probe container on scadabridge-net.
+set -uo pipefail
+PAIR=$1; HOLD=$2; WATCH=${3:-300}
+A=${PREFIX:-scadabridge-}$PAIR-a; B=${PREFIX:-scadabridge-}$PAIR-b
+[ "$PAIR" = central ] && HP=5000 || HP=8084
+PROBE=sb116env2-probe
+docker rm -f $PROBE >/dev/null 2>&1
+docker run -d --name $PROBE --network scadabridge-net sb-drill-net:local sleep infinity >/dev/null
+AIP=$(docker inspect -f '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}' $A)
+c() { docker exec $PROBE curl -s -m 2 -o /dev/null -w %{http_code} "http://$1:$HP/health/$2" 2>/dev/null || echo ERR; }
+mem() { docker exec $PROBE curl -s -m 2 "http://$1:$HP/health/ready" 2>/dev/null | grep -o '"memberCount":[0-9]*' | head -1; }
+rc() { docker inspect -f '{{.RestartCount}}/{{.State.StartedAt}}' $1; }
+RA0=$(rc $A); RB0=$(rc $B)
+echo "$(date -u +%T) before: $A active=$(c $A active) $B active=$(c $B active)  restarts A=$RA0 B=$RB0  (A ip $AIP)"
+CUT_AT=$(date -u +%Y-%m-%dT%H:%M:%SZ); T0=$(date +%s)
+docker run --rm --net container:$B --cap-add NET_ADMIN sb-drill-net:local sh -c "iptables -I INPUT -s $AIP -j DROP && iptables -I OUTPUT -d $AIP -j DROP"
+echo "$(date -u +%T) partition ON (B drops all traffic to/from A)"
+DUAL_AT=""
+while (( $(date +%s) - T0 < HOLD )); do
+  a=$(c $A active); b=$(c $B active)
+  if [ -z "$DUAL_AT" ] && [ "$a" = 200 ] && [ "$b" = 200 ]; then DUAL_AT=$(( $(date +%s) - T0 )); echo "$(date -u +%T) DUAL-ACTIVE reached at +${DUAL_AT}s"; fi
+  sleep 1
+done
+echo "$(date -u +%T) end of hold (+$(( $(date +%s)-T0 ))s): $A active=$(c $A active) $B active=$(c $B active)"
+docker run --rm --net container:$B --cap-add NET_ADMIN sb-drill-net:local sh -c "iptables -D INPUT -s $AIP -j DROP && iptables -D OUTPUT -d $AIP -j DROP"
+H0=$(date +%s); echo "$(date -u +%T) partition HEALED (+$(( H0-T0 ))s)"
+RESTARTED=""; CONVERGED=""; STABLE=0
+while (( $(date +%s) - H0 < WATCH )); do
+  sleep 2
+  ra=$(rc $A); rb=$(rc $B); a=$(c $A active); b=$(c $B active)
+  if [ -z "$RESTARTED" ] && { [ "$ra" != "$RA0" ] || [ "$rb" != "$RB0" ]; }; then
+    RESTARTED="+$(( $(date +%s)-H0 ))s after heal: A $([ "$ra" != "$RA0" ] && echo RESTARTED || echo stayed), B $([ "$rb" != "$RB0" ] && echo RESTARTED || echo stayed)"
+    echo "$(date -u +%T) $RESTARTED"
+  fi
+  if [ -n "$RESTARTED" ] && [ "$(c $A ready)" = 200 ] && [ "$(c $B ready)" = 200 ] && { [ "$a$b" = 200503 ] || [ "$a$b" = 503200 ]; }; then
+    STABLE=$((STABLE+1))
+    if [ -z "$CONVERGED" ]; then CONVERGED="+$(( $(date +%s)-H0 ))s after heal"; echo "$(date -u +%T) one active ($A=$a $B=$b), $(mem $A) $(mem $B) — converged $CONVERGED"; fi
+    (( STABLE >= 10 )) && break
+  else
+    [ -n "$CONVERGED" ] && { echo "$(date -u +%T) lost convergence: $A=$a $B=$b"; CONVERGED=""; STABLE=0; }
+  fi
+done
+echo "$(date -u +%T) final: $A active=$(c $A active) ready=$(c $A ready) restarts=$(rc $A) | $B active=$(c $B active) ready=$(c $B ready) restarts=$(rc $B)"
+echo "result: restart=[$RESTARTED] converged=[$CONVERGED] stable-samples=$STABLE"
+echo "--- $A [leader-auto-down] + quarantine lines"; docker logs --since $CUT_AT $A 2>&1 | grep -E "leader-auto-down\]|quarantined this system|terminated outside host" | cut -c1-330
+echo "--- $B [leader-auto-down] + quarantine lines"; docker logs --since $CUT_AT $B 2>&1 | grep -E "leader-auto-down\]|quarantined this system|terminated outside host" | cut -c1-330
+docker rm -f $PROBE >/dev/null 2>&1
diff --git a/docs/plans/2026-11-12-leader-auto-down-rig-drill.md b/docs/plans/2026-11-12-leader-auto-down-rig-drill.md
index 200c7f51..cd30e6ee 100644
--- a/docs/plans/2026-11-12-leader-auto-down-rig-drill.md
+++ b/docs/plans/2026-11-12-leader-auto-down-rig-drill.md
@@ -168,3 +168,7 @@ For comparison, §3 measured a kill takeover of 29 s / 26 s, and §5 recorded a
 **Rig state afterwards.** All 8 nodes were recreated on the **default `auto-down`** with no overlay, running image `db8d9ffe`. All 8 are ready. Each pair has one active node and `memberCount 2, unreachableCount 0`: central-a, site-a-a, site-b-a and site-c-a are active ([final status](2026-11-12-leader-auto-down-rig-drill/final-status-132.txt)). The deprecation warning is back on the default nodes, and no `shut-out restart` line appears, because the actor exists only under `leader-auto-down`.
 
 Helper for the split gate: [`partition-heal.sh`](2026-11-12-leader-auto-down-rig-drill/partition-heal.sh). It is an evidence copy, not a supported rig script.
+
+## 8. docker-env2 (2026-11-12)
+
+The owner approved the next rollout step. All gates, including the freeze and split-then-heal gates, passed on env2's central and site-x pairs, and env2 stays on `leader-auto-down` as the soak environment. Record: [`2026-11-12-leader-auto-down-env2-drill.md`](2026-11-12-leader-auto-down-env2-drill.md).