feat(cluster): auto-down downing strategy — either-node crash now fails over (owner decision 2026-07-21: availability over partition-safety)
Two-node keep-oldest could NEVER survive a crash of the oldest/active node:
Akka.NET 1.5.62 KeepOldest.OldestDecision only lets down-if-alone rescue a
side with >= 2 members, so the 1-vs-1 survivor takes DownReachable and downs
ITSELF — proven live on the rig ('SBR took decision ... including myself')
before this change. static-quorum(1) is worse (IsTooManyMembers -> DownAll);
keep-majority just re-keys the fatal crash to the lowest address.
SplitBrainResolverStrategy gains 'auto-down' (new default): BuildHocon emits
Akka's AutoDowning provider with auto-down-unreachable-after = StableAfter.
The leader among the REACHABLE members downs the unreachable peer, so the
survivor takes over singletons and /health/active in ~25s regardless of which
node died. Accepted trade (explicit owner decision): a real network partition
runs dual-active until an operator restarts one side. keep-oldest remains
supported; DownIfAlone validation is now scoped to it.
Live drill on the rebuilt rig: active-crash TAKEOVER in 28s (victim still
down; all 7 singletons Younger->Oldest), standby-crash removal 27s with 0
routing blips; victims rejoin as standby in 2s. New real-cluster tests pin
both directions (SbrFailoverTests.AutoDown_*); TwoNodeClusterFixture gains a
strategy knob. All 16 appsettings flipped (src, docker, docker-env2, and the
gitignored wonder-app-vd03 overlay on disk — owner must sync to the host).
Docs: decision record docs/plans/2026-07-21-auto-down-availability-decision.md,
Component-ClusterInfrastructure downing section rewritten, drill + README
reworked (active mode now asserts takeover), deferred-work SBR row resolved.
This commit is contained in:
@@ -12,13 +12,14 @@ namespace ZB.MOM.WW.ScadaBridge.PerformanceTests.Failover;
|
||||
/// 2s heartbeat / 10s failure-detection threshold / 15s SBR stable-after —
|
||||
/// the CLAUDE.md "total failover ~25s" design envelope.
|
||||
///
|
||||
/// Measures the SURVIVABLE direction only: hard-crash of the YOUNGER node,
|
||||
/// timed to the survivor's member REMOVAL (detection + stable-after + gossip)
|
||||
/// with singleton continuity asserted on the oldest. The oldest/active-node
|
||||
/// crash is NOT a recovery to time — two-node keep-oldest makes the younger
|
||||
/// survivor down itself (total outage; registered deferred user decision,
|
||||
/// see SbrFailoverTests XML doc + master tracker 2026-07-08). Covers overall
|
||||
/// review P2-10 / report-08 NF2 and report-01 round-2 N1's measurement ask.
|
||||
/// Runs under the production default downing strategy (auto-down, decision
|
||||
/// 2026-07-21 — either-direction crash fails over; see SbrFailoverTests XML
|
||||
/// doc). Measures a hard-crash of the YOUNGER node, timed to the survivor's
|
||||
/// member REMOVAL (detection + stability window + gossip) with singleton
|
||||
/// continuity asserted on the oldest; the oldest-crash direction is covered
|
||||
/// behaviorally by SbrFailoverTests.AutoDown_HardCrashOfOldestNode_* and by
|
||||
/// the docker failover drill. Covers overall review P2-10 / report-08 NF2
|
||||
/// and report-01 round-2 N1's measurement ask.
|
||||
/// </summary>
|
||||
public class FailoverTimingTests(ITestOutputHelper output)
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user