feat(cluster)!: leader-auto-down is the shipped default downing strategy (#116)
SplitBrainResolverStrategy now defaults to leader-auto-down (the in-repo LeaderAutoDowningProvider, #106, with the #132 shut-out restart on). auto-down and keep-oldest stay supported. Drilled on the docker rig, docker-env2 and the QA pair (central + site, kill and stop-and-hold) before the flip. - ClusterOptions default + validator/HOCON comments updated. - Shipped Host appsettings.Central/Site.json and every docker/ and docker-env2/ node appsettings now say leader-auto-down. The leader-auto-down overlays are kept as no-ops (drill records reference them); new docker-compose.auto-down.yml overlays keep Akka AutoDowning available for comparison drills. - Tests: default asserts leader-auto-down; new HOCON test for an unset strategy; TwoNodeClusterFixture defaults to the production posture. - Docs: CLAUDE.md, Component-ClusterInfrastructure, components/Host + Cluster, README, failover procedures, Traefik, docker + env2 READMEs; register row 70 closed. No deploy needed: DEV, QA and both rigs set the key explicitly. Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
This commit is contained in:
+4
-2
@@ -400,14 +400,16 @@ DRILL_ACTION=pause DRILL_MODE=active bash docker/failover-drill.sh # freeze ins
|
||||
|
||||
The rejoin check is strict. The victim must report `/health/ready` 200 and `/health/active` 503, and the survivor must report `memberCount 2, unreachableCount 0`. After that, 10 one-second samples must each show exactly one active node.
|
||||
|
||||
**Trying `leader-auto-down` (ScadaBridge#106/#116).** The opt-in overlay `docker/docker-compose.leader-auto-down.yml` switches the central and site-a pairs to the in-repo provider through an environment override. The shipped default stays `auto-down`. Switch both pairs together, never one node at a time:
|
||||
**Downing provider (ScadaBridge#106/#116).** Since 2026-11-15 the shipped default is `leader-auto-down`, the in-repo `LeaderAutoDowningProvider`, and every node's mounted appsettings say so. It was drilled on this rig, on docker-env2 and on the QA pair before the flip. To compare against Akka's deprecated `AutoDowning` provider (the default from 2026-07-21 until 2026-11-15), the opt-in overlay `docker/docker-compose.auto-down.yml` switches the central and site-a pairs back through an environment override. Switch both pairs together, never one node at a time:
|
||||
|
||||
```bash
|
||||
docker compose -f docker/docker-compose.yml -f docker/docker-compose.leader-auto-down.yml \
|
||||
docker compose -f docker/docker-compose.yml -f docker/docker-compose.auto-down.yml \
|
||||
up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b
|
||||
# back to the default: the same command without the overlay file
|
||||
```
|
||||
|
||||
`docker/docker-compose.leader-auto-down.yml` is now a no-op, kept because the #116 drill records reference it.
|
||||
|
||||
The scripted drill (`docker kill` = SIGKILL, the hard-crash path — a `docker stop` would take the graceful `CoordinatedShutdown` path and would not prove crash recovery) has **two modes**, and since the **auto-down decision (2026-07-21)** both expect recovery — the cluster runs Akka's `AutoDowning` provider (`auto-down-unreachable-after` = 15s), under which the leader among the *reachable* members downs the unreachable peer, so a crash of either node fails over:
|
||||
|
||||
- **`DRILL_MODE=standby` (default) — kills the STANDBY (younger) central node.** The active node is untouched: expected result is **no routing outage at all** (`/health/active` blips = 0) and member removal on the survivor within **~25s** (10s failure-detection threshold + 15s auto-down window; the 2s heartbeat interval is not additive). PASS = the survivor logs the downing/removal within `TIMEOUT_S` (default 90s) while routing stays up.
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-central-a:8081",
|
||||
"akka.tcp://scadabridge@scadabridge-central-b:8081"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-central-b:8081",
|
||||
"akka.tcp://scadabridge@scadabridge-central-a:8081"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
# Opt-in overlay: run the central pair and the site-a pair on Akka's deprecated
|
||||
# `AutoDowning` provider (`auto-down`), the shipped default from 2026-07-21 until
|
||||
# 2026-11-15, instead of the default `leader-auto-down` (ScadaBridge#116). For comparison
|
||||
# drills only. Env wins over the mounted appsettings.
|
||||
#
|
||||
# A downing-provider change must reach both nodes of a pair at the same time - never
|
||||
# roll it one node at a time. Apply (from the main checkout, both pairs together):
|
||||
#
|
||||
# docker compose -f docker/docker-compose.yml -f docker/docker-compose.auto-down.yml \
|
||||
# up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b
|
||||
#
|
||||
# Back to the default by re-running the same command without the overlay file.
|
||||
services:
|
||||
central-a:
|
||||
environment:
|
||||
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
|
||||
central-b:
|
||||
environment:
|
||||
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
|
||||
site-a-a:
|
||||
environment:
|
||||
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
|
||||
site-a-b:
|
||||
environment:
|
||||
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
|
||||
@@ -1,17 +1,10 @@
|
||||
# Opt-in overlay: run the central pair and the site-a pair on the in-repo
|
||||
# `leader-auto-down` downing provider (LeaderAutoDowningProvider, ScadaBridge#106)
|
||||
# instead of the shipped default `auto-down`. Used by the #116 rig failover drill.
|
||||
#
|
||||
# The shipped default is NOT changed by this file: the mounted appsettings still say
|
||||
# `auto-down`; this overlay layers an environment override on top (env wins over JSON).
|
||||
#
|
||||
# A downing-provider change must reach both nodes of a pair at the same time - never
|
||||
# roll it one node at a time. Apply (from the main checkout, both pairs together):
|
||||
# REDUNDANT since 2026-11-15 (ScadaBridge#116): `leader-auto-down` is now the shipped
|
||||
# default and the mounted appsettings say so, so this overlay changes nothing. It is kept
|
||||
# because the #116 drill records reference it. To run the old Akka `AutoDowning` provider
|
||||
# for a comparison, use docker/docker-compose.auto-down.yml instead.
|
||||
#
|
||||
# docker compose -f docker/docker-compose.yml -f docker/docker-compose.leader-auto-down.yml \
|
||||
# up -d --force-recreate --no-deps central-a central-b site-a-a site-a-b
|
||||
#
|
||||
# Revert to the default by re-running the same command without the overlay file.
|
||||
services:
|
||||
central-a:
|
||||
environment:
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
# Failover drill against the running docker cluster (bash docker/deploy.sh first).
|
||||
#
|
||||
# AUTO-DOWN REWRITE (decision 2026-07-21). The cluster runs an availability-first
|
||||
# downing strategy — `auto-down` (default) or the in-repo `leader-auto-down`
|
||||
# (ScadaBridge#106; opt-in overlay docker/docker-compose.leader-auto-down.yml) —
|
||||
# downing strategy — the in-repo `leader-auto-down` (default since 2026-11-15,
|
||||
# ScadaBridge#106/#116) or Akka's `auto-down` (opt-in overlay docker/docker-compose.auto-down.yml) —
|
||||
# under which the leader among the REACHABLE members downs the unreachable peer
|
||||
# after StableAfter, so a hard crash of EITHER node of a pair — the active/oldest
|
||||
# included — fails over to the survivor. The accepted trade (made explicitly by
|
||||
|
||||
@@ -22,7 +22,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-site-a-a:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-a-b:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -22,7 +22,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-site-a-b:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-a-a:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-site-b-a:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-b-b:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-site-b-b:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-b-a:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-site-c-a:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-c-b:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
"akka.tcp://scadabridge@scadabridge-site-c-b:8082",
|
||||
"akka.tcp://scadabridge@scadabridge-site-c-a:8082"
|
||||
],
|
||||
"SplitBrainResolverStrategy": "auto-down",
|
||||
"SplitBrainResolverStrategy": "leader-auto-down",
|
||||
"StableAfter": "00:00:15",
|
||||
"HeartbeatInterval": "00:00:02",
|
||||
"FailureDetectionThreshold": "00:00:10",
|
||||
|
||||
Reference in New Issue
Block a user