feat(cluster): simultaneous-cold-start split-brain guard (opt-in)
Closes the residual Phase-7 gap: when BOTH nodes of a 2-node pair cold-start at the same instant, each self-first seed runs FirstSeedNodeProcess, times out waiting for the other, and forms its own 1-node cluster — two Primaries in one pair (the Phase 6 live gate reproduced this reliably on docker). The prior mitigation was operational (staggered start / compose depends_on), which does not exist on production hardware. The guard (dark switch Cluster:BootstrapGuard:Enabled, default OFF) prevents the split without giving up cold-start-alone: - The lower-address node is the preferred founder: self-first, forms immediately. - The higher node probes its partner's Akka port (TCP connect) up to PartnerProbeSeconds: reachable => peer-first (join the founder, never race it); unreachable => self-first (partner is dead, form alone). The order is decided BEFORE the single JoinSeedNodes, from an explicit reachability signal — never re-formed mid-handshake (the retired SelfFormAfter failure mode). When on, Akka gets no config seeds (BuildClusterOptions) and ClusterBootstrapCoordinator drives the join. Review-driven hardening: case-insensitive tie-break (a hostname-casing mismatch would reopen the split); fail-fast validation of the timing knobs; the residual "founder dies in the probe->join window" hang is documented and made operator-visible (warning + a restart recovers). Real-ActorSystem coordinator tests cover the load-bearing higher-node cold-start-alone case; 147/147 Cluster tests pass. Live-gated on docker-dev (site-a = enablement demo, serialization removed, guard on; site-b keeps depends_on-serialization + guard off as the A/B control): simultaneous start -> site-a-1 founds, site-a-2 probes-reachable-joins -> 250/240, NO split; higher-node-alone -> probes dead founder, self-forms -> 250; founder rejoin -> 240. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
@@ -41,6 +41,8 @@ public sealed class AkkaClusterOptionsValidator : OptionsValidatorBase<AkkaClust
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(options);
|
||||
|
||||
ValidateBootstrapGuard(builder, options.BootstrapGuard);
|
||||
|
||||
var seeds = options.SeedNodes ?? Array.Empty<string>();
|
||||
if (seeds.Length == 0)
|
||||
{
|
||||
@@ -68,6 +70,25 @@ public sealed class AkkaClusterOptionsValidator : OptionsValidatorBase<AkkaClust
|
||||
+ "the entries — see docs/Redundancy.md → 'Bootstrap: self-first seed ordering'.");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// When the bootstrap guard is enabled, its timing knobs must be positive. A zero/negative
|
||||
/// <c>PartnerProbeSeconds</c> in particular silently degrades the guard to "never wait, always
|
||||
/// conclude the partner is dead" — it would form alone immediately and re-open the very split it
|
||||
/// exists to close. Fail-fast at boot rather than producing silence (the same rationale every
|
||||
/// sibling options validator in this project cites).
|
||||
/// </summary>
|
||||
private static void ValidateBootstrapGuard(ValidationBuilder builder, ClusterBootstrapGuardOptions guard)
|
||||
{
|
||||
if (guard is null || !guard.Enabled) return;
|
||||
|
||||
if (guard.PartnerProbeSeconds <= 0)
|
||||
builder.Add($"Cluster:BootstrapGuard:PartnerProbeSeconds must be > 0 when the guard is enabled (was {guard.PartnerProbeSeconds}).");
|
||||
if (guard.PartnerProbeIntervalMs <= 0)
|
||||
builder.Add($"Cluster:BootstrapGuard:PartnerProbeIntervalMs must be > 0 when the guard is enabled (was {guard.PartnerProbeIntervalMs}).");
|
||||
if (guard.ProbeConnectTimeoutMs <= 0)
|
||||
builder.Add($"Cluster:BootstrapGuard:ProbeConnectTimeoutMs must be > 0 when the guard is enabled (was {guard.ProbeConnectTimeoutMs}).");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// True when <paramref name="seed"/> addresses this node itself — host AND port.
|
||||
/// </summary>
|
||||
|
||||
Reference in New Issue
Block a user