feat(cluster)!: leader-auto-down is the shipped default downing strategy (#116)
SplitBrainResolverStrategy now defaults to leader-auto-down (the in-repo LeaderAutoDowningProvider, #106, with the #132 shut-out restart on). auto-down and keep-oldest stay supported. Drilled on the docker rig, docker-env2 and the QA pair (central + site, kill and stop-and-hold) before the flip. - ClusterOptions default + validator/HOCON comments updated. - Shipped Host appsettings.Central/Site.json and every docker/ and docker-env2/ node appsettings now say leader-auto-down. The leader-auto-down overlays are kept as no-ops (drill records reference them); new docker-compose.auto-down.yml overlays keep Akka AutoDowning available for comparison drills. - Tests: default asserts leader-auto-down; new HOCON test for an unset strategy; TwoNodeClusterFixture defaults to the production posture. - Docs: CLAUDE.md, Component-ClusterInfrastructure, components/Host + Cluster, README, failover procedures, Traefik, docker + env2 READMEs; register row 70 closed. No deploy needed: DEV, QA and both rigs set the key explicitly. Claude-Session: https://claude.ai/code/session_01GT67AFKC45MBz1wPZnVbX8
This commit is contained in:
@@ -101,7 +101,7 @@ Nothing in the emitted document enables remoting TLS or an Akka secure cookie
|
||||
|
||||
### Downing strategy (auto-down — availability-first)
|
||||
|
||||
**Decision 2026-07-21** (`docs/plans/2026-07-21-auto-down-availability-decision.md`): the default strategy is **`auto-down`** — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15 s). The leader among the *reachable* members downs the unreachable peer once the stability window elapses.
|
||||
**Since 2026-11-15 (ScadaBridge#116) the default strategy is `leader-auto-down`**, the in-repo `LeaderAutoDowningProvider`. It has the same semantics as the `auto-down` decision below without Akka's deprecated provider, and adds the #132 shut-out restart; see `docs/requirements/Component-ClusterInfrastructure.md`. **Decision 2026-07-21** (`docs/plans/2026-07-21-auto-down-availability-decision.md`): the default strategy was **`auto-down`** — Akka's `AutoDowning` provider with `auto-down-unreachable-after` = `StableAfter` (15 s). The leader among the *reachable* members downs the unreachable peer once the stability window elapses.
|
||||
|
||||
- **Either-node crash is survivable.** If the standby crashes, the active node downs it and continues. If the **active/oldest** node crashes, the younger survivor downs the dead oldest, becomes the oldest itself, re-hosts every cluster singleton, and `/health/active` flips to it — no operator action and no victim restart.
|
||||
- **The accepted trade is dual-active during a real network partition.** With both nodes alive but the link cut, each side downs the other and continues as a one-node cluster; both claim active until an operator restarts one side after the partition heals. This was chosen deliberately — pairs run one node per VM with no shared lease store (no Kubernetes, no site-side SQL) to arbitrate, and a stalled system is a bigger operational risk than a rare LAN partition.
|
||||
|
||||
@@ -184,7 +184,7 @@ Options are bound via the .NET Options pattern (`IOptions<T>`). Each component o
|
||||
| Key | Default | Description |
|
||||
|-----|---------|-------------|
|
||||
| `SeedNodes` | — | List of Akka seed-node URIs (`akka.tcp://scadabridge@host:port`). At least 2 required. Must reference remoting ports, not gRPC ports. |
|
||||
| `SplitBrainResolverStrategy` | `auto-down` | Downing strategy: `"auto-down"` (default) or `"keep-oldest"`. |
|
||||
| `SplitBrainResolverStrategy` | `leader-auto-down` | Downing strategy: `"leader-auto-down"` (default since 2026-11-15, ScadaBridge#116 — the in-repo `LeaderAutoDowningProvider`), `"auto-down"` (Akka's deprecated `AutoDowning`; the default until 2026-11-15) or `"keep-oldest"`. |
|
||||
| `StableAfter` | `"00:00:15"` | Sustained unreachability before downing (`auto-down-unreachable-after` under `auto-down`). |
|
||||
| `HeartbeatInterval` | `"00:00:02"` | Akka failure-detector heartbeat cadence. |
|
||||
| `FailureDetectionThreshold` | `"00:00:10"` | Acceptable heartbeat pause before a node is considered unreachable. |
|
||||
|
||||
@@ -38,7 +38,7 @@ The identical oldest-member check backs `ActiveNodeGate` — the `IActiveNodeGat
|
||||
|
||||
### Automatic failover
|
||||
|
||||
When the active central node goes down, the Akka cluster's `auto-down` downing strategy (the leader among the reachable members downs the dead node) leaves the surviving node as the oldest Up member and the singletons migrate to it (roughly 25 seconds: 10-second heartbeat threshold plus a 15-second `auto-down-unreachable-after` window). Once the surviving node's `OldestNodeActiveHealthCheck` starts returning 200, Traefik's next poll cycle — within the 5-second interval — removes the failed backend from the pool and routes all subsequent requests to the new active node. No config change or restart is required on the Traefik side.
|
||||
When the active central node goes down, the Akka cluster's `leader-auto-down` downing strategy (formerly `auto-down`) (the leader among the reachable members downs the dead node) leaves the surviving node as the oldest Up member and the singletons migrate to it (roughly 25 seconds: 10-second heartbeat threshold plus a 15-second `auto-down-unreachable-after` window). Once the surviving node's `OldestNodeActiveHealthCheck` starts returning 200, Traefik's next poll cycle — within the 5-second interval — removes the failed backend from the pool and routes all subsequent requests to the new active node. No config change or restart is required on the Traefik side.
|
||||
|
||||
## Architecture
|
||||
|
||||
|
||||
Reference in New Issue
Block a user