Joseph Doherty
ab561faa43
Adds LeaderAutoDowningProvider (Host/Actors), a line-for-line port of Akka.NET 1.5.62 AutoDowning/AutoDown/AutoDownBase read from the decompiled assembly: the leader among the reachable members downs a peer unreachable for StableAfter; a non-leader records it as pending and downs it on becoming leader; reachable-again or removed members are forgotten; Down/Exiting members are skipped. Its window is carried under scadabridge.cluster.leader-auto-down.unreachable-after, so no deprecated auto-down-unreachable-after key is emitted and Akka's boot-time "keep-majority will be used instead" warning goes away. DownRemovalMargin reads akka.cluster.down-removal-margin exactly as AutoDowning does (default off). New SplitBrainResolverStrategy value "leader-auto-down", wired through BuildHocon and ClusterOptionsValidator. The default stays "auto-down"; auto-down and keep-oldest HOCON is byte-identical to before. Placed in Host, not ClusterInfrastructure, per the Code Placement rule (ClusterInfrastructure stays Akka-free). Tests: HOCON + validator units; LeaderAutoDownTests ports Akka's AutoDownSpec cases (leader-only, pending-on-leader-change, losing leadership, reachable again, removed, Down/Exiting skipped, single Down per member); LeaderAutoDownFailoverTests clones SbrFailoverTests' oldest-crash and younger-crash cases on a real two-node cluster plus a test that Akka actually loaded the provider and its downing actor. Mutation-checked: a provider that never calls Cluster.Down fails both failover tests; never-downs, leader-check-dropped, reachable-ignored and pending-not-flushed mutations each fail the unit tests. Remaining gate before switching the default: the docker failover drill (DRILL_MODE=active and standby mode) on leader-auto-down. Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
97 lines
7.9 KiB
Markdown
97 lines
7.9 KiB
Markdown
# Replacing Akka `AutoDowning` before an Akka.NET upgrade removes it (plan, 2026-11-09)
|
|
|
|
**Status: PLAN ONLY (ScadaBridge#51, register row 70).** No config or code changes are made by this document.
|
|
|
|
**Update 2026-11-09 (ScadaBridge#106):** option B is implemented as the opt-in strategy `leader-auto-down`
|
|
(`LeaderAutoDowningProvider`, placed in **Host** rather than ClusterInfrastructure: the Code Placement rule in
|
|
`Component-ClusterInfrastructure.md` keeps ClusterInfrastructure Akka-free, and the provider is runtime wiring).
|
|
`DownRemovalMargin` matches `AutoDowning` exactly (`akka.cluster.down-removal-margin`, default off), not a `StableAfter`-derived value.
|
|
Unit, HOCON, validator and two-node real-cluster tests are green. Remaining gate before switching the default: the
|
|
docker failover drill (section 6) on the new value, then the per-environment rollout in section 5.
|
|
Related decision: [`2026-07-21-auto-down-availability-decision.md`](2026-07-21-auto-down-availability-decision.md).
|
|
|
|
## 1. Current state (read from the code)
|
|
|
|
- Akka.NET pinned at **1.5.62** (`Directory.Packages.props`: Akka, Akka.Cluster, Akka.Cluster.Tools, Akka.Remote).
|
|
- `AkkaHostedService.BuildHocon` always installs a downing provider explicitly (Akka defaults to `NoDowning`).
|
|
`ClusterOptions.SplitBrainResolverStrategy` is `"auto-down"` (default) or `"keep-oldest"`:
|
|
|
|
```hocon
|
|
# auto-down (default, every 2-node pair)
|
|
downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"
|
|
auto-down-unreachable-after = <ClusterOptions.StableAfter> # 15s production
|
|
|
|
# keep-oldest (opt-in)
|
|
downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"
|
|
split-brain-resolver { active-strategy = keep-oldest; stable-after = <StableAfter>;
|
|
keep-oldest { down-if-alone = on|off } }
|
|
```
|
|
- All 16 appsettings (src Host x2, `docker/` x8, `docker-env2/` x4, wonder overlay x2) use `auto-down`.
|
|
- `ClusterOptionsValidator` accepts only `auto-down` | `keep-oldest`.
|
|
- Every node logs on boot that `auto-down-unreachable-after` is deprecated and "keep-majority will be used instead".
|
|
Drill-verified (2026-08-15, 1.5.62) that the substitution does NOT happen while `downing-provider-class` is explicit.
|
|
|
|
## 2. Which Akka.NET version deprecates / removes it
|
|
|
|
- **Deprecated: v1.5.2** (Akka.NET v1.5 upgrade advisory). Behavioural change from 1.5.2: the advisory states that `auto-down-unreachable-after`
|
|
is deprecated and keep-majority SBR is used instead when the key is present and no explicit provider is set.
|
|
- **Removed: no version announced.** The advisory says only "will be removed in a future version". The 1.5.62 `Akka.Cluster.xml` still documents
|
|
`Akka.Cluster.AutoDowning` and carries no removal notice (checked in `~/.nuget/packages/akka.cluster/1.5.62`). It has survived 60 patch releases
|
|
since deprecation, so the realistic removal window is a 1.6 / 2.0 line, but this must be re-checked per bump.
|
|
- Sources: [Akka.NET v1.5 Upgrade Advisories](https://akkadotnet.github.io/community/whats-new/akkadotnet-v1.5-upgrade-advisories.html),
|
|
[Petabridge SBR post](https://petabridge.com/blog/akkadotnet-cluster-split-brain-resolver/).
|
|
|
|
## 3. Candidate replacements (2-node pairs, availability over partition-safety)
|
|
|
|
Evidence for rejecting every built-in SBR strategy is in the 2026-07-21 decision (live drill + Akka source): keep-oldest (+down-if-alone) downs the
|
|
survivor when the oldest crashes in a 1-vs-1; keep-majority keeps the lowest-address side (same hole); static-quorum 1 = DownAll, 2 = survivor
|
|
self-downs; lease-majority needs a shared lease store the sites do not have. None of them gives "either node crashes -> survivor takes over" on two nodes.
|
|
|
|
| Option | Availability (either node crashes) | Partition behaviour | Effort / risk |
|
|
|---|---|---|---|
|
|
| A. Keep `AutoDowning` until removed, pin Akka | Yes | dual-active (accepted) | none now; blocks Akka bumps once removed |
|
|
| **B. In-repo custom `DowningProvider` reproducing AutoDowning** | **Yes (identical)** | **dual-active (accepted)** | small: ~100 lines, public `DowningProvider` API |
|
|
| C. SBR keep-oldest, `down-if-alone=on` | NO - oldest crash = outage | partition-safe | rejected (proven) |
|
|
| D. SBR keep-majority | NO - lowest-address crash = outage | partition-safe | rejected |
|
|
| E. Third arbiter node / lease | Yes | partition-safe | needs infra sites do not have |
|
|
|
|
## 4. Recommendation
|
|
|
|
**Option B**, consistent with the family preference (availability over partition safety for 2-node pairs, accepting dual-active on a real partition).
|
|
The 2026-07-21 decision already said "if a future Akka.NET release removes AutoDowning, port it then"; this plan makes that concrete:
|
|
|
|
1. Implement `ScadaBridgeDowningProvider : Akka.Cluster.DowningProvider` in `ClusterInfrastructure` (`DownRemovalMargin` = `StableAfter`-derived,
|
|
`DowningActorProps` = a small actor that, on the leader among reachable members, downs members unreachable for `stable-after`, using
|
|
`Cluster.Down(address)`; skip when this node is not leader). Behaviour must match `AutoDowning` exactly so no operational contract changes.
|
|
2. Add `ClusterOptions.SplitBrainResolverStrategy` value `"leader-auto-down"` (name TBD), wire it in `BuildHocon` with
|
|
`downing-provider-class = "<our type>, ZB.MOM.WW.ScadaBridge.ClusterInfrastructure"` and no `auto-down-unreachable-after` key, which also removes the
|
|
boot deprecation warning. Keep `"auto-down"` working while Akka still ships it; switch the default only after the drills below pass.
|
|
3. If the provider API proves to be removed/unstable in the target Akka version, fallback is a leader-side actor using `Cluster.Down` driven by
|
|
`ReachabilityEvent`/`UnreachableMember` with a scheduled timer - same semantics.
|
|
|
|
Do NOT adopt SBR keep-oldest/keep-majority for pairs: it re-introduces the proven total outage on oldest/lowest-address crash.
|
|
|
|
## 5. Migration plan
|
|
|
|
1. **Trigger:** any Akka.NET bump. Read release notes for `AutoDowning` / `auto-down-unreachable-after` / `DowningProvider` changes before accepting it.
|
|
2. Build option B on a branch while still on 1.5.62 (so the new provider is proven against the current runtime before any bump).
|
|
3. Roll out strategy-by-strategy: docker rig, then `docker-env2`, then QA pair vq03/vq04, then DEV vd03 and sites. Pairs stop/start together
|
|
(no rolling restart across a downing-provider change). Flip appsettings value only; rollback = flip back to `auto-down`.
|
|
4. Update `Component-ClusterInfrastructure.md`, `docker/README.md`, CLAUDE.md, and close register row 70 / this issue once the default is switched.
|
|
|
|
## 6. Test plan
|
|
|
|
- **Unit:** `HoconBuilderTests` - new strategy emits our provider class and no `auto-down-unreachable-after` key; validator accepts it;
|
|
existing `auto-down`/`keep-oldest` emission unchanged.
|
|
- **2-node TestKit / real-cluster (IntegrationTests/Cluster, `TwoNodeClusterFixture` with its `strategy` parameter):** clone the two existing
|
|
`SbrFailoverTests` cases for the new strategy:
|
|
`..._HardCrashOfOldestNode_YoungerSurvivorTakesOverSingleton` (the direction keep-oldest cannot pass) and
|
|
`..._HardCrashOfYoungerNode_OldestKeepsSingleton`; add a "leader is not the unreachable node, not leader -> no Down call" case and a
|
|
"reachable again before stable-after -> no Down" case. Positive control: run the oldest-crash test against `keep-oldest` and confirm it fails.
|
|
- **Docker rig drill:** `DRILL_MODE=active bash docker/failover-drill.sh` (kill active/oldest `central-a`; survivor must take over in ~25s; victim
|
|
rejoins as standby) on central and one site pair, with the new strategy; repeat with the default drill mode for the standby-crash direction.
|
|
Run the same drill once after every Akka bump regardless of option.
|
|
- **Partition check:** confirm the accepted dual-active still occurs and recovers by restarting one side (documented operator path), and that
|
|
both nodes then show `/health/active` 200 until recovery (monitoring contract unchanged).
|
|
- **Gate for the bump PR:** new-strategy integration tests green, drill green, no deprecation warning in boot logs.
|