dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 5 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(cluster): leader-auto-down restarts a node its peer has shut out; higher address restarts when both are (#132)

Browse Source 
A node frozen past failure detection + StableAfter, or both sides of a partition
that heals after each downed the other, used to stay a second active node until an
operator restarted one: the peer quarantines the removed UID and Akka.Cluster
1.5.62 never acts on ThisActorSystemQuarantinedEvent.

ShutOutRestart (child of the LeaderAutoDown downing actor; opt-out via
ScadaBridge:Cluster:LeaderAutoDown:RestartWhenShutOut, default on for
leader-auto-down only) watches two signals:
- S1 ThisActorSystemQuarantinedEvent: the peer quarantined this node. If the peer
  is still a live member here, only this side can be shut out -> restart now.
- S2 AssociationErrorEvent 'has a UID that has been quarantined': the peer still
  runs the incarnation this node removed. A healed partition yields only S2, on
  both sides. Wait ShutOutGrace (60s) for a frozen peer to restart itself, then
  tiebreak.
Tiebreak: the higher address (Akka Member.AddressOrdering) restarts, the lower
stays and restarts only if still split one grace later. Removed peers are probed
(Identify, 5s -> 60s back-off) since nothing else contacts them after mutual
removal. Restart = Cluster.Down(self) -> existing self-down path.

The Host now exits 75 after an unplanned ActorSystem termination (and sets it on
the Windows service before it reports stopped): SCM recovery with failureflag 1
only restarts a service that stops with a non-zero code, and the watchdog used
to exit 0. Shipped default strategy stays auto-down.

Claude-Session: https://claude.ai/code/session_01GT67ACmpMR7b9LGRPLijMi
This commit is contained in:
Joseph Doherty
2026-11-12 20:57:42 -05:00
parent a633005f4f
commit 952db02aab
14 changed files with 1646 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure.Tests/ClusterOptionsValidatorTests.cs
+28
View File
@@ -327,4 +327,32 @@ public class ClusterOptionsValidatorTests
Assert.True(result.Failed);
Assert.Contains("'leader-auto-down'", result.FailureMessage);
}
/// <summary>
/// ScadaBridge#132: a non-positive shut-out grace is refused under leader-auto-down with the
/// restart on (it would make both sides of a healed partition restart together), and ignored
/// when the restart is off or another strategy is selected.
/// </summary>
/// <param name="strategy">The strategy.</param>
/// <param name="restart">The shut-out restart switch.</param>
/// <param name="expectValid">Whether validation must pass.</param>
[Theory]
[InlineData("leader-auto-down", true, false)]
[InlineData("leader-auto-down", false, true)]
[InlineData("auto-down", true, true)]
public void LeaderAutoDown_ZeroShutOutGrace(string strategy, bool restart, bool expectValid)
{
var options = ValidOptions();
options.SplitBrainResolverStrategy = strategy;
options.LeaderAutoDown = new LeaderAutoDownOptions { RestartWhenShutOut = restart, ShutOutGrace = TimeSpan.Zero };
var result = new ClusterOptionsValidator().Validate(null, options);
Assert.Equal(expectValid, result.Succeeded);
if (!expectValid)
{
Assert.Contains("ShutOutGrace", result.FailureMessage);
}
}
}