ScadaBridge/docs/plans/2026-11-09-autodowning-replacement-plan.md

Replacing Akka AutoDowning before an Akka.NET upgrade removes it (plan, 2026-11-09)

Status: PLAN ONLY (ScadaBridge#51, register row 70). No config or code changes are made by this document.

Update 2026-11-09 (ScadaBridge#106): option B is implemented as the opt-in strategy leader-auto-down (LeaderAutoDowningProvider, placed in Host rather than ClusterInfrastructure: the Code Placement rule in Component-ClusterInfrastructure.md keeps ClusterInfrastructure Akka-free, and the provider is runtime wiring). DownRemovalMargin matches AutoDowning exactly (akka.cluster.down-removal-margin, default off), not a StableAfter-derived value. Unit, HOCON, validator and two-node real-cluster tests are green. Remaining gate before switching the default: the docker failover drill (section 6) on the new value, then the per-environment rollout in section 5.

Update 2026-11-10 (ScadaBridge#106 complete): every downing decision is now logged with a [leader-auto-down] prefix (start-up snapshot, unreachable observed + window at Warning, skipped Down/Exiting, cancelled on reachable/removed, timed out on a non-leader → pending, leadership gained/lost, and DOWNING at Warning before Cluster.Down). New real-cluster scenarios in IntegrationTests/Cluster/LeaderAutoDownScenarioTests.cs: joint cold start, lone cold start with the peer dead then the peer joining, a flapping peer (Akka trttl throttler blackhole, unreachable shorter than the window, repeated past it — never downed, with a positive control that the cuts produced unreachability), a partition held past the window (each side downs the other, both stay Up — the accepted dual-active), and the switch off (auto-down → AutoDowning, keep-oldest → SplitBrainResolverProvider). The docker drill and rollout are tracked as the deploy gate ScadaBridge#116. Related decision: 2026-07-21-auto-down-availability-decision.md.

1. Current state (read from the code)

• Akka.NET pinned at 1.5.62 (Directory.Packages.props: Akka, Akka.Cluster, Akka.Cluster.Tools, Akka.Remote).

• AkkaHostedService.BuildHocon always installs a downing provider explicitly (Akka defaults to NoDowning). ClusterOptions.SplitBrainResolverStrategy is "auto-down" (default) or "keep-oldest":

  # auto-down (default, every 2-node pair)
  downing-provider-class = "Akka.Cluster.AutoDowning, Akka.Cluster"
  auto-down-unreachable-after = <ClusterOptions.StableAfter>   # 15s production
  
  # keep-oldest (opt-in)
  downing-provider-class = "Akka.Cluster.SBR.SplitBrainResolverProvider, Akka.Cluster"
  split-brain-resolver { active-strategy = keep-oldest; stable-after = <StableAfter>;
                         keep-oldest { down-if-alone = on|off } }
  

• All 16 appsettings (src Host x2, docker/ x8, docker-env2/ x4, wonder overlay x2) use auto-down.

• ClusterOptionsValidator accepts only auto-down | keep-oldest.

• Every node logs on boot that auto-down-unreachable-after is deprecated and "keep-majority will be used instead". Drill-verified (2026-08-15, 1.5.62) that the substitution does NOT happen while downing-provider-class is explicit.

2. Which Akka.NET version deprecates / removes it

• Deprecated: v1.5.2 (Akka.NET v1.5 upgrade advisory). Behavioural change from 1.5.2: the advisory states that auto-down-unreachable-after is deprecated and keep-majority SBR is used instead when the key is present and no explicit provider is set.
• Removed: no version announced. The advisory says only "will be removed in a future version". The 1.5.62 Akka.Cluster.xml still documents Akka.Cluster.AutoDowning and carries no removal notice (checked in ~/.nuget/packages/akka.cluster/1.5.62). It has survived 60 patch releases since deprecation, so the realistic removal window is a 1.6 / 2.0 line, but this must be re-checked per bump.
• Sources: Akka.NET v1.5 Upgrade Advisories, Petabridge SBR post.

3. Candidate replacements (2-node pairs, availability over partition-safety)

Evidence for rejecting every built-in SBR strategy is in the 2026-07-21 decision (live drill + Akka source): keep-oldest (+down-if-alone) downs the survivor when the oldest crashes in a 1-vs-1; keep-majority keeps the lowest-address side (same hole); static-quorum 1 = DownAll, 2 = survivor self-downs; lease-majority needs a shared lease store the sites do not have. None of them gives "either node crashes -> survivor takes over" on two nodes.

Option	Availability (either node crashes)	Partition behaviour	Effort / risk
A. Keep AutoDowning until removed, pin Akka	Yes	dual-active (accepted)	none now; blocks Akka bumps once removed
B. In-repo custom DowningProvider reproducing AutoDowning	Yes (identical)	dual-active (accepted)	small: ~100 lines, public DowningProvider API
C. SBR keep-oldest, down-if-alone=on	NO - oldest crash = outage	partition-safe	rejected (proven)
D. SBR keep-majority	NO - lowest-address crash = outage	partition-safe	rejected
E. Third arbiter node / lease	Yes	partition-safe	needs infra sites do not have

4. Recommendation

Option B, consistent with the family preference (availability over partition safety for 2-node pairs, accepting dual-active on a real partition). The 2026-07-21 decision already said "if a future Akka.NET release removes AutoDowning, port it then"; this plan makes that concrete:

1. Implement ScadaBridgeDowningProvider : Akka.Cluster.DowningProvider in ClusterInfrastructure (DownRemovalMargin = StableAfter-derived, DowningActorProps = a small actor that, on the leader among reachable members, downs members unreachable for stable-after, using Cluster.Down(address); skip when this node is not leader). Behaviour must match AutoDowning exactly so no operational contract changes.
2. Add ClusterOptions.SplitBrainResolverStrategy value "leader-auto-down" (name TBD), wire it in BuildHocon with downing-provider-class = "<our type>, ZB.MOM.WW.ScadaBridge.ClusterInfrastructure" and no auto-down-unreachable-after key, which also removes the boot deprecation warning. Keep "auto-down" working while Akka still ships it; switch the default only after the drills below pass.
3. If the provider API proves to be removed/unstable in the target Akka version, fallback is a leader-side actor using Cluster.Down driven by ReachabilityEvent/UnreachableMember with a scheduled timer - same semantics.

Do NOT adopt SBR keep-oldest/keep-majority for pairs: it re-introduces the proven total outage on oldest/lowest-address crash.

5. Migration plan

1. Trigger: any Akka.NET bump. Read release notes for AutoDowning / auto-down-unreachable-after / DowningProvider changes before accepting it.
2. Build option B on a branch while still on 1.5.62 (so the new provider is proven against the current runtime before any bump).
3. Roll out strategy-by-strategy: docker rig, then docker-env2, then QA pair vq03/vq04, then DEV vd03 and sites. Pairs stop/start together (no rolling restart across a downing-provider change). Flip appsettings value only; rollback = flip back to auto-down.
4. Update Component-ClusterInfrastructure.md, docker/README.md, CLAUDE.md, and close register row 70 / this issue once the default is switched.

6. Test plan

• Unit: HoconBuilderTests - new strategy emits our provider class and no auto-down-unreachable-after key; validator accepts it; existing auto-down/keep-oldest emission unchanged.
• 2-node TestKit / real-cluster (IntegrationTests/Cluster, TwoNodeClusterFixture with its strategy parameter): clone the two existing SbrFailoverTests cases for the new strategy: ..._HardCrashOfOldestNode_YoungerSurvivorTakesOverSingleton (the direction keep-oldest cannot pass) and ..._HardCrashOfYoungerNode_OldestKeepsSingleton; add a "leader is not the unreachable node, not leader -> no Down call" case and a "reachable again before stable-after -> no Down" case. Positive control: run the oldest-crash test against keep-oldest and confirm it fails.
• Docker rig drill: DRILL_MODE=active bash docker/failover-drill.sh (kill active/oldest central-a; survivor must take over in ~25s; victim rejoins as standby) on central and one site pair, with the new strategy; repeat with the default drill mode for the standby-crash direction. Run the same drill once after every Akka bump regardless of option.
• Partition check: confirm the accepted dual-active still occurs and recovers by restarting one side (documented operator path), and that both nodes then show /health/active 200 until recovery (monitoring contract unchanged).
• Gate for the bump PR: new-strategy integration tests green, drill green, no deprecation warning in boot logs.

7. Addendum 2026-11-12: shut-out restart (ScadaBridge#132)

The #116 rig drill found a gap that AutoDowning and its port share: a node frozen past failure detection plus the window comes back as a permanent second active node, and the same happens to both sides of a partition that heals after each side downed the other. Root cause (Akka.NET 1.5.62): the peer's cluster quarantines the removed node's UID, Akka.Remote publishes ThisActorSystemQuarantinedEvent on the stale node, and Akka.Cluster never subscribes to it.

Owner decision (#132): under leader-auto-down, a node that is shut out restarts itself; when both sides are shut out, exactly one restarts, by a deterministic tiebreak. Implemented as ShutOutRestart (Host, a child of the downing actor, so LeaderAutoDownBase stays a line-for-line port), behind ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut (default true for this strategy only) and ShutOutGrace (default 60 s). The shipped default strategy stays auto-down.

• Two signals, confirmed in-process. S1 ThisActorSystemQuarantinedEvent (the frozen node). S2 an AssociationErrorEvent whose cause says "has a UID that has been quarantined" (the peer runs the incarnation this node removed). A healed partition produces only S2, on both sides — each side aborts the other's handshake with its own refused UID before a quarantine disassociate can arrive — so S1 alone would not have fixed the partition case. A removed peer is probed with Identify (5 s → 60 s back-off) because after mutual removal nothing else would ever contact it.
• Decision. S1 while the peer is still a live member here: restart now (only this side can be shut out). S1 after this node removed the peer: tiebreak. S2: wait ShutOutGrace (a frozen peer restarts itself meanwhile), then tiebreak if still split. Tiebreak: higher address restarts, the lower stays and restarts only if the peer is still split from it one more grace later.
• Why address, not age: see Component-ClusterInfrastructure.md → Shut-out restart. In short, both nodes hold both addresses always; the peer's up-number is history that can differ after the split.
• Exit code. The restart is the existing self-down path, and it exposed a latent gap: the watchdog's StopApplication() exited 0, and Windows service recovery with failureflag 1 acts only on a non-zero exit. The Host now exits 75 after an unplanned ActorSystem termination (also set on the Windows service before it reports stopped). Docker restarts on any code.
• Contract change for #116 gate 4: the partition still produces dual-active while it lasts; after it heals, one side (the higher address) restarts and the pair converges with no operator action. LeaderAutoDownScenarioTests.PartitionLongerThanWindow_EachSideDownsTheOther_BothStayUp still holds (it never heals).
• Tests: ShutOutRestartTests (unit), ShutOutRestartScenarioTests (two real nodes: freeze shape both address orders, healed partition, switch off), UnexpectedTerminationTests, HoconBuilderTests, ClusterOptionsValidatorTests. Rig evidence: 2026-11-12-leader-auto-down-rig-drill.md §7.