feat(cluster): simultaneous-cold-start split-brain guard (opt-in)
Closes the residual Phase-7 gap: when BOTH nodes of a 2-node pair cold-start at the same instant, each self-first seed runs FirstSeedNodeProcess, times out waiting for the other, and forms its own 1-node cluster — two Primaries in one pair (the Phase 6 live gate reproduced this reliably on docker). The prior mitigation was operational (staggered start / compose depends_on), which does not exist on production hardware. The guard (dark switch Cluster:BootstrapGuard:Enabled, default OFF) prevents the split without giving up cold-start-alone: - The lower-address node is the preferred founder: self-first, forms immediately. - The higher node probes its partner's Akka port (TCP connect) up to PartnerProbeSeconds: reachable => peer-first (join the founder, never race it); unreachable => self-first (partner is dead, form alone). The order is decided BEFORE the single JoinSeedNodes, from an explicit reachability signal — never re-formed mid-handshake (the retired SelfFormAfter failure mode). When on, Akka gets no config seeds (BuildClusterOptions) and ClusterBootstrapCoordinator drives the join. Review-driven hardening: case-insensitive tie-break (a hostname-casing mismatch would reopen the split); fail-fast validation of the timing knobs; the residual "founder dies in the probe->join window" hang is documented and made operator-visible (warning + a restart recovers). Real-ActorSystem coordinator tests cover the load-bearing higher-node cold-start-alone case; 147/147 Cluster tests pass. Live-gated on docker-dev (site-a = enablement demo, serialization removed, guard on; site-b keeps depends_on-serialization + guard off as the A/B control): simultaneous start -> site-a-1 founds, site-a-2 probes-reachable-joins -> 250/240, NO split; higher-node-alone -> probes dead founder, self-forms -> 250; founder rejoin -> 240. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
@@ -79,4 +79,46 @@ public sealed class AkkaClusterOptions
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public string SplitBrainResolverStrategy { get; set; } = "auto-down";
|
||||
|
||||
/// <summary>
|
||||
/// The simultaneous-cold-start split-brain guard (<c>Cluster:BootstrapGuard</c>). Default OFF — a
|
||||
/// dark switch, so existing deployments and tests keep Akka's config-driven self-first auto-join
|
||||
/// unchanged. See <see cref="ClusterBootstrapGuard"/> for the decision logic and
|
||||
/// <c>ClusterBootstrapCoordinator</c> for the runtime.
|
||||
/// </summary>
|
||||
public ClusterBootstrapGuardOptions BootstrapGuard { get; set; } = new();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Configuration for the simultaneous-cold-start split-brain guard. When
|
||||
/// <see cref="Enabled"/>, the node does NOT auto-join from its config seeds; a coordinator picks the
|
||||
/// join order (founder self-first / joiner peer-first) after a reachability probe. See
|
||||
/// <see cref="ClusterBootstrapGuard"/>.
|
||||
/// </summary>
|
||||
public sealed class ClusterBootstrapGuardOptions
|
||||
{
|
||||
/// <summary>
|
||||
/// Gets or sets whether the guard is active. Default <see langword="false"/> — Akka auto-joins from
|
||||
/// the config seed list exactly as before. Only meaningful on a node that is one of its own two
|
||||
/// pair seeds; inert everywhere else.
|
||||
/// </summary>
|
||||
public bool Enabled { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Gets or sets how long the higher-address node probes its partner's Akka endpoint before
|
||||
/// concluding the partner is dead and forming alone. Must comfortably exceed the partner's
|
||||
/// worst-case process-start-to-Akka-bind time so a slow-but-alive partner is never mistaken for a
|
||||
/// dead one (which would re-open the split). Default 25 s.
|
||||
/// </summary>
|
||||
public int PartnerProbeSeconds { get; set; } = 25;
|
||||
|
||||
/// <summary>
|
||||
/// Gets or sets the interval between partner reachability probes, in milliseconds. Default 500 ms.
|
||||
/// </summary>
|
||||
public int PartnerProbeIntervalMs { get; set; } = 500;
|
||||
|
||||
/// <summary>
|
||||
/// Gets or sets the per-probe TCP connect timeout, in milliseconds. Default 1000 ms.
|
||||
/// </summary>
|
||||
public int ProbeConnectTimeoutMs { get; set; } = 1000;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user