# Replacing Akka `AutoDowning` before an Akka.NET upgrade removes it (plan, 2026-11-09) **Status: PLAN ONLY (ScadaBridge#51, register row 70).** No config or code changes are made by this document. **Update 2026-11-09 (ScadaBridge#106):** option B is implemented as the opt-in strategy `leader-auto-down` (`LeaderAutoDowningProvider`, placed in **Host** rather than ClusterInfrastructure: the Code Placement rule in `Component-ClusterInfrastructure.md` keeps ClusterInfrastructure Akka-free, and the provider is runtime wiring). `DownRemovalMargin` matches `AutoDowning` exactly (`akka.cluster.down-removal-margin`, default off), not a `StableAfter`-derived value. Unit, HOCON, validator and two-node real-cluster tests are green. Remaining gate before switching the default: the docker failover drill (section 6) on the new value, then the per-environment rollout in section 5. **Update 2026-11-10 (ScadaBridge#106 complete):** every downing decision is now logged with a `[leader-auto-down]` prefix (start-up snapshot, unreachable observed + window at Warning, skipped Down/Exiting, cancelled on reachable/removed, timed out on a non-leader → pending, leadership gained/lost, and `DOWNING` at Warning before `Cluster.Down`). New real-cluster scenarios in `IntegrationTests/Cluster/LeaderAutoDownScenarioTests.cs`: joint cold start, lone cold start with the peer dead then the peer joining, a flapping peer (Akka `trttl` throttler blackhole, unreachable shorter than the window, repeated past it — never downed, with a positive control that the cuts produced unreachability), a partition held past the window (each side downs the other, both stay Up — the accepted dual-active), and the switch off (`auto-down` → `AutoDowning`, `keep-oldest` → `SplitBrainResolverProvider`). The docker drill and rollout are tracked as the deploy gate **ScadaBridge#116**. Related decision: [`2026-07-21-auto-down-availability-decision.md`](2026-07-21-auto-down-availability-decision.md). ## 1. Current state (read from the code) - Akka.NET pinned at **1.5.62** (`Directory.Packages.props`: Akka, Akka.Cluster, Akka.Cluster.Tools, Akka.Remote). - `AkkaHostedService.BuildHocon` always installs a downing provider explicitly (Akka defaults to `NoDowning`). `ClusterOptions.SplitBrainResolverStrategy` is `"auto-down"` (default) or `"keep-oldest"`: ```hocon # auto-down (default, every 2-node pair) downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster" auto-down-unreachable-after = # 15s production # keep-oldest (opt-in) downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster" split-brain-resolver { active-strategy = keep-oldest; stable-after = ; keep-oldest { down-if-alone = on|off } } ``` - All 16 appsettings (src Host x2, `docker/` x8, `docker-env2/` x4, wonder overlay x2) use `auto-down`. - `ClusterOptionsValidator` accepts only `auto-down` | `keep-oldest`. - Every node logs on boot that `auto-down-unreachable-after` is deprecated and "keep-majority will be used instead". Drill-verified (2026-08-15, 1.5.62) that the substitution does NOT happen while `downing-provider-class` is explicit. ## 2. Which Akka.NET version deprecates / removes it - **Deprecated: v1.5.2** (Akka.NET v1.5 upgrade advisory). Behavioural change from 1.5.2: the advisory states that `auto-down-unreachable-after` is deprecated and keep-majority SBR is used instead when the key is present and no explicit provider is set. - **Removed: no version announced.** The advisory says only "will be removed in a future version". The 1.5.62 `Akka.Cluster.xml` still documents `Akka.Cluster.AutoDowning` and carries no removal notice (checked in `~/.nuget/packages/akka.cluster/1.5.62`). It has survived 60 patch releases since deprecation, so the realistic removal window is a 1.6 / 2.0 line, but this must be re-checked per bump. - Sources: [Akka.NET v1.5 Upgrade Advisories](https://akkadotnet.github.io/community/whats-new/akkadotnet-v1.5-upgrade-advisories.html), [Petabridge SBR post](https://petabridge.com/blog/akkadotnet-cluster-split-brain-resolver/). ## 3. Candidate replacements (2-node pairs, availability over partition-safety) Evidence for rejecting every built-in SBR strategy is in the 2026-07-21 decision (live drill + Akka source): keep-oldest (+down-if-alone) downs the survivor when the oldest crashes in a 1-vs-1; keep-majority keeps the lowest-address side (same hole); static-quorum 1 = DownAll, 2 = survivor self-downs; lease-majority needs a shared lease store the sites do not have. None of them gives "either node crashes -> survivor takes over" on two nodes. | Option | Availability (either node crashes) | Partition behaviour | Effort / risk | |---|---|---|---| | A. Keep `AutoDowning` until removed, pin Akka | Yes | dual-active (accepted) | none now; blocks Akka bumps once removed | | **B. In-repo custom `DowningProvider` reproducing AutoDowning** | **Yes (identical)** | **dual-active (accepted)** | small: ~100 lines, public `DowningProvider` API | | C. SBR keep-oldest, `down-if-alone=on` | NO - oldest crash = outage | partition-safe | rejected (proven) | | D. SBR keep-majority | NO - lowest-address crash = outage | partition-safe | rejected | | E. Third arbiter node / lease | Yes | partition-safe | needs infra sites do not have | ## 4. Recommendation **Option B**, consistent with the family preference (availability over partition safety for 2-node pairs, accepting dual-active on a real partition). The 2026-07-21 decision already said "if a future Akka.NET release removes AutoDowning, port it then"; this plan makes that concrete: 1. Implement `ScadaBridgeDowningProvider : Akka.Cluster.DowningProvider` in `ClusterInfrastructure` (`DownRemovalMargin` = `StableAfter`-derived, `DowningActorProps` = a small actor that, on the leader among reachable members, downs members unreachable for `stable-after`, using `Cluster.Down(address)`; skip when this node is not leader). Behaviour must match `AutoDowning` exactly so no operational contract changes. 2. Add `ClusterOptions.SplitBrainResolverStrategy` value `"leader-auto-down"` (name TBD), wire it in `BuildHocon` with `downing-provider-class = ", ZB.MOM.WW.ScadaBridge.ClusterInfrastructure"` and no `auto-down-unreachable-after` key, which also removes the boot deprecation warning. Keep `"auto-down"` working while Akka still ships it; switch the default only after the drills below pass. 3. If the provider API proves to be removed/unstable in the target Akka version, fallback is a leader-side actor using `Cluster.Down` driven by `ReachabilityEvent`/`UnreachableMember` with a scheduled timer - same semantics. Do NOT adopt SBR keep-oldest/keep-majority for pairs: it re-introduces the proven total outage on oldest/lowest-address crash. ## 5. Migration plan 1. **Trigger:** any Akka.NET bump. Read release notes for `AutoDowning` / `auto-down-unreachable-after` / `DowningProvider` changes before accepting it. 2. Build option B on a branch while still on 1.5.62 (so the new provider is proven against the current runtime before any bump). 3. Roll out strategy-by-strategy: docker rig, then `docker-env2`, then QA pair vq03/vq04, then DEV vd03 and sites. Pairs stop/start together (no rolling restart across a downing-provider change). Flip appsettings value only; rollback = flip back to `auto-down`. 4. Update `Component-ClusterInfrastructure.md`, `docker/README.md`, CLAUDE.md, and close register row 70 / this issue once the default is switched. ## 6. Test plan - **Unit:** `HoconBuilderTests` - new strategy emits our provider class and no `auto-down-unreachable-after` key; validator accepts it; existing `auto-down`/`keep-oldest` emission unchanged. - **2-node TestKit / real-cluster (IntegrationTests/Cluster, `TwoNodeClusterFixture` with its `strategy` parameter):** clone the two existing `SbrFailoverTests` cases for the new strategy: `..._HardCrashOfOldestNode_YoungerSurvivorTakesOverSingleton` (the direction keep-oldest cannot pass) and `..._HardCrashOfYoungerNode_OldestKeepsSingleton`; add a "leader is not the unreachable node, not leader -> no Down call" case and a "reachable again before stable-after -> no Down" case. Positive control: run the oldest-crash test against `keep-oldest` and confirm it fails. - **Docker rig drill:** `DRILL_MODE=active bash docker/failover-drill.sh` (kill active/oldest `central-a`; survivor must take over in ~25s; victim rejoins as standby) on central and one site pair, with the new strategy; repeat with the default drill mode for the standby-crash direction. Run the same drill once after every Akka bump regardless of option. - **Partition check:** confirm the accepted dual-active still occurs and recovers by restarting one side (documented operator path), and that both nodes then show `/health/active` 200 until recovery (monitoring contract unchanged). - **Gate for the bump PR:** new-strategy integration tests green, drill green, no deprecation warning in boot logs. ## 7. Addendum 2026-11-12: shut-out restart (ScadaBridge#132) The #116 rig drill found a gap that `AutoDowning` and its port share: a node frozen past failure detection plus the window comes back as a **permanent second active node**, and the same happens to both sides of a partition that heals after each side downed the other. Root cause (Akka.NET 1.5.62): the peer's cluster quarantines the removed node's UID, Akka.Remote publishes `ThisActorSystemQuarantinedEvent` on the stale node, and Akka.Cluster never subscribes to it. **Owner decision (#132):** under `leader-auto-down`, a node that is shut out restarts itself; when both sides are shut out, exactly one restarts, by a deterministic tiebreak. Implemented as `ShutOutRestart` (Host, a child of the downing actor, so `LeaderAutoDownBase` stays a line-for-line port), behind `ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut` (default `true` for this strategy only) and `ShutOutGrace` (default 60 s). The shipped default strategy stays `auto-down`. - **Two signals, confirmed in-process.** S1 `ThisActorSystemQuarantinedEvent` (the frozen node). S2 an `AssociationErrorEvent` whose cause says "has a UID that has been quarantined" (the peer runs the incarnation this node removed). A healed partition produces **only S2, on both sides** — each side aborts the other's handshake with its own refused UID before a quarantine disassociate can arrive — so S1 alone would not have fixed the partition case. A removed peer is probed with `Identify` (5 s → 60 s back-off) because after mutual removal nothing else would ever contact it. - **Decision.** S1 while the peer is still a live member here: restart now (only this side can be shut out). S1 after this node removed the peer: tiebreak. S2: wait `ShutOutGrace` (a frozen peer restarts itself meanwhile), then tiebreak if still split. Tiebreak: **higher address restarts**, the lower stays and restarts only if the peer is still split from it one more grace later. - **Why address, not age:** see `Component-ClusterInfrastructure.md` → *Shut-out restart*. In short, both nodes hold both addresses always; the peer's up-number is history that can differ after the split. - **Exit code.** The restart is the existing self-down path, and it exposed a latent gap: the watchdog's `StopApplication()` exited 0, and Windows service recovery with `failureflag 1` acts only on a non-zero exit. The Host now exits 75 after an unplanned ActorSystem termination (also set on the Windows service before it reports stopped). Docker restarts on any code. - **Contract change for #116 gate 4:** the partition still produces dual-active *while it lasts*; after it heals, one side (the higher address) restarts and the pair converges with no operator action. `LeaderAutoDownScenarioTests.PartitionLongerThanWindow_EachSideDownsTheOther_BothStayUp` still holds (it never heals). - **Tests:** `ShutOutRestartTests` (unit), `ShutOutRestartScenarioTests` (two real nodes: freeze shape both address orders, healed partition, switch off), `UnexpectedTerminationTests`, `HoconBuilderTests`, `ClusterOptionsValidatorTests`. Rig evidence: `2026-11-12-leader-auto-down-rig-drill.md` §7.