feat(cluster)!: leader-auto-down is the shipped default downing strategy (#116)

SplitBrainResolverStrategy now defaults to leader-auto-down (the in-repo
LeaderAutoDowningProvider, #106, with the #132 shut-out restart on). auto-down and
keep-oldest stay supported. Drilled on the docker rig, docker-env2 and the QA pair
(central + site, kill and stop-and-hold) before the flip.

- ClusterOptions default + validator/HOCON comments updated.
- Shipped Host appsettings.Central/Site.json and every docker/ and docker-env2/
  node appsettings now say leader-auto-down. The leader-auto-down overlays are
  kept as no-ops (drill records reference them); new docker-compose.auto-down.yml
  overlays keep Akka AutoDowning available for comparison drills.
- Tests: default asserts leader-auto-down; new HOCON test for an unset strategy;
  TwoNodeClusterFixture defaults to the production posture.
- Docs: CLAUDE.md, Component-ClusterInfrastructure, components/Host + Cluster,
  README, failover procedures, Traefik, docker + env2 READMEs; register row 70 closed.

No deploy needed: DEV, QA and both rigs set the key explicitly.

Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
This commit is contained in:
Joseph Doherty
2026-11-15 19:49:31 -05:00
parent 1213a74353
commit efade2eab0
36 changed files with 153 additions and 83 deletions
+8 -6
View File
@@ -129,18 +129,20 @@ Both UIs are now reachable:
run it, so an env2 build never retags the `scadabridge:latest` the primary rig runs. Unset
keeps the old shared-image behaviour.
Since 2026-11-12 env2 is the soak environment for the `leader-auto-down` rollout and runs
`scadabridge:env2-116` with the opt-in overlay
[`docker-compose.leader-auto-down.yml`](docker-compose.leader-auto-down.yml). Keep both
when recreating nodes, and switch both pairs together:
env2 was the soak environment for the `leader-auto-down` rollout from 2026-11-12. Since
2026-11-15 `leader-auto-down` is the shipped default, and env2's mounted appsettings say so, so
plain `deploy.sh` now runs it. The old overlay
[`docker-compose.leader-auto-down.yml`](docker-compose.leader-auto-down.yml) is a no-op. For a
comparison run on Akka's `AutoDowning`, use
[`docker-compose.auto-down.yml`](docker-compose.auto-down.yml), switching both pairs together:
```bash
SCADABRIDGE_IMAGE=scadabridge:env2-116 docker compose \
-f docker-env2/docker-compose.yml -f docker-env2/docker-compose.leader-auto-down.yml \
-f docker-env2/docker-compose.yml -f docker-env2/docker-compose.auto-down.yml \
up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b
```
Plain `deploy.sh` goes back to `scadabridge:latest` on the default `auto-down`. Drill:
Drill:
`DRILL_ENV=env2 DRILL_PAIR=central|site-x bash docker/failover-drill.sh`. Record:
[`docs/plans/2026-11-12-leader-auto-down-env2-drill.md`](../docs/plans/2026-11-12-leader-auto-down-env2-drill.md).
@@ -13,7 +13,7 @@
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -13,7 +13,7 @@
"akka.tcp://scadabridge@scadabridge-env2-central-b:8081",
"akka.tcp://scadabridge@scadabridge-env2-central-a:8081"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
+25
View File
@@ -0,0 +1,25 @@
# Opt-in overlay: run the central pair and the site-x pair on Akka's deprecated
# `AutoDowning` provider (`auto-down`), the shipped default from 2026-07-21 until
# 2026-11-15, instead of the default `leader-auto-down` (ScadaBridge#116). For comparison
# drills only. Env wins over the mounted appsettings.
#
# A downing-provider change must reach both nodes of a pair at the same time - never
# roll it one node at a time. Apply (from the main checkout, both pairs together):
#
# docker compose -f docker-env2/docker-compose.yml -f docker-env2/docker-compose.auto-down.yml \
# up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b
#
# Back to the default by re-running the same command without the overlay file.
services:
central-a:
environment:
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
central-b:
environment:
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
site-x-a:
environment:
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
site-x-b:
environment:
ScadaBridge__Cluster__SplitBrainResolverStrategy: "auto-down"
@@ -1,18 +1,10 @@
# Opt-in overlay: run env2's central pair and its site-x pair on the in-repo
# `leader-auto-down` downing provider (LeaderAutoDowningProvider, ScadaBridge#106)
# instead of the shipped default `auto-down`. env2 is the soak environment for the
# #116 rollout; mirrors docker/docker-compose.leader-auto-down.yml.
#
# The shipped default is NOT changed by this file: the mounted appsettings still say
# `auto-down`; this overlay layers an environment override on top (env wins over JSON).
#
# A downing-provider change must reach both nodes of a pair at the same time - never
# roll it one node at a time. Apply (from the main checkout, both pairs together):
# REDUNDANT since 2026-11-15 (ScadaBridge#116): `leader-auto-down` is now the shipped
# default and the mounted appsettings say so, so this overlay changes nothing. It is kept
# because the #116 drill records reference it. To run the old Akka `AutoDowning` provider
# for a comparison, use docker-env2/docker-compose.auto-down.yml instead.
#
# docker compose -f docker-env2/docker-compose.yml -f docker-env2/docker-compose.leader-auto-down.yml \
# up -d --force-recreate --no-deps central-a central-b site-x-a site-x-b
#
# Revert to the default by re-running the same command without the overlay file.
services:
central-a:
environment:
@@ -13,7 +13,7 @@
"akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082",
"akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",
@@ -13,7 +13,7 @@
"akka.tcp://scadabridge@scadabridge-env2-site-x-b:8082",
"akka.tcp://scadabridge@scadabridge-env2-site-x-a:8082"
],
"SplitBrainResolverStrategy": "auto-down",
"SplitBrainResolverStrategy": "leader-auto-down",
"StableAfter": "00:00:15",
"HeartbeatInterval": "00:00:02",
"FailureDetectionThreshold": "00:00:10",