feat(cluster)!: leader-auto-down is the shipped default downing strategy (#116)

SplitBrainResolverStrategy now defaults to leader-auto-down (the in-repo
LeaderAutoDowningProvider, #106, with the #132 shut-out restart on). auto-down and
keep-oldest stay supported. Drilled on the docker rig, docker-env2 and the QA pair
(central + site, kill and stop-and-hold) before the flip.

- ClusterOptions default + validator/HOCON comments updated.
- Shipped Host appsettings.Central/Site.json and every docker/ and docker-env2/
  node appsettings now say leader-auto-down. The leader-auto-down overlays are
  kept as no-ops (drill records reference them); new docker-compose.auto-down.yml
  overlays keep Akka AutoDowning available for comparison drills.
- Tests: default asserts leader-auto-down; new HOCON test for an unset strategy;
  TwoNodeClusterFixture defaults to the production posture.
- Docs: CLAUDE.md, Component-ClusterInfrastructure, components/Host + Cluster,
  README, failover procedures, Traefik, docker + env2 READMEs; register row 70 closed.

No deploy needed: DEV, QA and both rigs set the key explicitly.

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
This commit is contained in:
Joseph Doherty
2026-11-15 19:49:31 -05:00
parent 1213a74353
commit efade2eab0
36 changed files with 153 additions and 83 deletions
+4 -2
View File
@@ -400,14 +400,16 @@ DRILL_ACTION=pause DRILL_MODE=active bash docker/failover-drill.sh # freeze ins
The rejoin check is strict. The victim must report `/health/ready` 200 and `/health/active` 503, and the survivor must report `memberCount 2, unreachableCount 0`. After that, 10 one-second samples must each show exactly one active node.
**Trying `leader-auto-down` (ScadaBridge#106/#116).** The opt-in overlay `docker/docker-compose.leader-auto-down.yml` switches the central and site-a pairs to the in-repo provider through an environment override. The shipped default stays `auto-down`. Switch both pairs together, never one node at a time:
**Downing provider (ScadaBridge#106/#116).** Since 2026-11-15 the shipped default is `leader-auto-down`, the in-repo `LeaderAutoDowningProvider`, and every node's mounted appsettings say so. It was drilled on this rig, on docker-env2 and on the QA pair before the flip. To compare against Akka's deprecated `AutoDowning` provider (the default from 2026-07-21 until 2026-11-15), the opt-in overlay `docker/docker-compose.auto-down.yml` switches the central and site-a pairs back through an environment override. Switch both pairs together, never one node at a time:
```bash
docker compose -f docker/docker-compose.yml -f docker/docker-compose.leader-auto-down.yml \
docker compose -f docker/docker-compose.yml -f docker/docker-compose.auto-down.yml \
up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b
# back to the default: the same command without the overlay file
```
`docker/docker-compose.leader-auto-down.yml` is now a no-op, kept because the #116 drill records reference it.
The scripted drill (`docker kill` = SIGKILL, the hard-crash path — a `docker stop` would take the graceful `CoordinatedShutdown` path and would not prove crash recovery) has **two modes**, and since the **auto-down decision (2026-07-21)** both expect recovery — the cluster runs Akka's `AutoDowning` provider (`auto-down-unreachable-after` = 15s), under which the leader among the *reachable* members downs the unreachable peer, so a crash of either node fails over:
- **`DRILL_MODE=standby` (default) — kills the STANDBY (younger) central node.** The active node is untouched: expected result is **no routing outage at all** (`/health/active` blips = 0) and member removal on the survivor within **~25s** (10s failure-detection threshold + 15s auto-down window; the 2s heartbeat interval is not additive). PASS = the survivor logs the downing/removal within `TIMEOUT_S` (default 90s) while routing stays up.
@@ -13,7 +13,7 @@
"akka.tcp://scadabridge@scadabridge-central-a:8081",
"akka.tcp://scadabridge@scadabridge-central-b:8081"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -13,7 +13,7 @@
"akka.tcp://scadabridge@scadabridge-central-b:8081",
"akka.tcp://scadabridge@scadabridge-central-a:8081"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
+25
View File
@@ -0,0 +1,25 @@
# Opt-in overlay: run the central pair and the site-a pair on Akka's deprecated
# `AutoDowning` provider (`auto-down`), the shipped default from 2026-07-21 until
# 2026-11-15, instead of the default `leader-auto-down` (ScadaBridge#116). For comparison
# drills only. Env wins over the mounted appsettings.
#
# A downing-provider change must reach both nodes of a pair at the same time - never
# roll it one node at a time. Apply (from the main checkout, both pairs together):
#
# docker compose -f docker/docker-compose.yml -f docker/docker-compose.auto-down.yml \
# up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b
#
# Back to the default by re-running the same command without the overlay file.
services:
central-a:
environment:
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
central-b:
environment:
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
site-a-a:
environment:
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
site-a-b:
environment:
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
+4 -11
View File
@@ -1,17 +1,10 @@
# Opt-in overlay: run the central pair and the site-a pair on the in-repo
# `leader-auto-down` downing provider (LeaderAutoDowningProvider, ScadaBridge#106)
# instead of the shipped default `auto-down`. Used by the #116 rig failover drill.
#
# The shipped default is NOT changed by this file: the mounted appsettings still say
# `auto-down`; this overlay layers an environment override on top (env wins over JSON).
#
# A downing-provider change must reach both nodes of a pair at the same time - never
# roll it one node at a time. Apply (from the main checkout, both pairs together):
# REDUNDANT since 2026-11-15 (ScadaBridge#116): `leader-auto-down` is now the shipped
# default and the mounted appsettings say so, so this overlay changes nothing. It is kept
# because the #116 drill records reference it. To run the old Akka `AutoDowning` provider
# for a comparison, use docker/docker-compose.auto-down.yml instead.
#
# docker compose -f docker/docker-compose.yml -f docker/docker-compose.leader-auto-down.yml \
# up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b
#
# Revert to the default by re-running the same command without the overlay file.
services:
central-a:
environment:
+2 -2
View File
@@ -2,8 +2,8 @@
# Failover drill against the running docker cluster (bash docker/deploy.sh first).
#
# AUTO-DOWN REWRITE (decision 2026-07-21). The cluster runs an availability-first
# downing strategy — `auto-down` (default) or the in-repo `leader-auto-down`
# (ScadaBridge#106; opt-in overlay docker/docker-compose.leader-auto-down.yml) —
# downing strategy — the in-repo `leader-auto-down` (default since 2026-11-15,
# ScadaBridge#106/#116) or Akka's `auto-down` (opt-in overlay docker/docker-compose.auto-down.yml) —
# under which the leader among the REACHABLE members downs the unreachable peer
# after StableAfter, so a hard crash of EITHER node of a pair — the active/oldest
# included — fails over to the survivor. The accepted trade (made explicitly by
+1 -1
View File
@@ -22,7 +22,7 @@
"akka.tcp://scadabridge@scadabridge-site-a-a:8082",
"akka.tcp://scadabridge@scadabridge-site-a-b:8082"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
+1 -1
View File
@@ -22,7 +22,7 @@
"akka.tcp://scadabridge@scadabridge-site-a-b:8082",
"akka.tcp://scadabridge@scadabridge-site-a-a:8082"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
+1 -1
View File
@@ -14,7 +14,7 @@
"akka.tcp://scadabridge@scadabridge-site-b-a:8082",
"akka.tcp://scadabridge@scadabridge-site-b-b:8082"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
+1 -1
View File
@@ -14,7 +14,7 @@
"akka.tcp://scadabridge@scadabridge-site-b-b:8082",
"akka.tcp://scadabridge@scadabridge-site-b-a:8082"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
+1 -1
View File
@@ -14,7 +14,7 @@
"akka.tcp://scadabridge@scadabridge-site-c-a:8082",
"akka.tcp://scadabridge@scadabridge-site-c-b:8082"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
+1 -1
View File
@@ -14,7 +14,7 @@
"akka.tcp://scadabridge@scadabridge-site-c-b:8082",
"akka.tcp://scadabridge@scadabridge-site-c-a:8082"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",