feat(cluster): simultaneous-cold-start split-brain guard (#33)
Port OtOpcUa's bootstrap guard (lmxopcua d1dac87f) to ScadaBridge's BuildHocon bootstrap. Both pair nodes are self-first seeds, so a true simultaneous cold start (shared site power event) races FirstSeedNodeProcess on both and forms two 1-node clusters that never merge. Opt-in dark switch ScadaBridge:Cluster:BootstrapGuard:Enabled (default off, guard-off behavior byte-identical). When on: BuildHocon emits an empty seed list so Akka does not auto-join, and ClusterBootstrapCoordinator (IHostedService, registered in both the Central and Site composition roots) picks the join order from ClusterBootstrapGuard's pure decision core — the lower canonical host:port is the founder (self-first, forms immediately); the higher node TCP-probes the founder up to PartnerProbeSeconds and joins peer-first if reachable, else self-first (cold-start-alone preserved). Decides before a single JoinSeedNodes, never re-forms mid-handshake. Review notes carried over: case-insensitive founder tie-break; fail-fast validation of probe timings when enabled; higher-node-cold-start-alone covered by a real-ActorSystem test. 21 unit + 5 real-cluster tests (incl. the headline both-cold-start-together-form-one-cluster).
This commit is contained in:
@@ -114,4 +114,50 @@ public class ClusterOptions
|
||||
/// dials forever — log noise and a defeated validation intent (review 01).
|
||||
/// </summary>
|
||||
public bool AllowSingleNodeCluster { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// The simultaneous-cold-start split-brain guard (<c>ScadaBridge:Cluster:BootstrapGuard</c>).
|
||||
/// Default OFF — a dark switch, so existing deployments and tests keep Akka's config-driven
|
||||
/// self-first auto-join byte-identical. See <see cref="ClusterBootstrapGuard"/> for the
|
||||
/// decision logic and <c>ClusterBootstrapCoordinator</c> for the runtime (Gitea #33).
|
||||
/// </summary>
|
||||
public ClusterBootstrapGuardOptions BootstrapGuard { get; set; } = new();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Configuration for the simultaneous-cold-start split-brain guard. Both nodes of a pair are
|
||||
/// self-first seeds so <b>either</b> can cold-start alone when its partner is dead — but the cost
|
||||
/// is that when BOTH cold-start at the same instant each runs Akka's <c>FirstSeedNodeProcess</c>,
|
||||
/// times out waiting for the other, and forms its own single-node cluster (a split brain that does
|
||||
/// not auto-merge). When <see cref="Enabled"/>, the node does NOT auto-join from its config seeds
|
||||
/// (<c>AkkaHostedService.BuildHocon</c> emits an empty seed list); a coordinator picks the join
|
||||
/// order — founder self-first / joiner peer-first — after a reachability probe. See
|
||||
/// <see cref="ClusterBootstrapGuard"/>.
|
||||
/// </summary>
|
||||
public sealed class ClusterBootstrapGuardOptions
|
||||
{
|
||||
/// <summary>
|
||||
/// Whether the guard is active. Default <see langword="false"/> — Akka auto-joins from the
|
||||
/// config seed list exactly as before. Only meaningful on a node that is one of its own two
|
||||
/// pair seeds; inert everywhere else (single-seed site node, self absent, 3+ seeds).
|
||||
/// </summary>
|
||||
public bool Enabled { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// How long the higher-address node probes its partner's Akka endpoint before concluding the
|
||||
/// partner is dead and forming alone. Must comfortably exceed the partner's worst-case
|
||||
/// process-start-to-Akka-bind time so a slow-but-alive partner is never mistaken for a dead one
|
||||
/// (which would re-open the split). Default 25s. Validated <c>> 0</c> at boot when the guard is on.
|
||||
/// </summary>
|
||||
public int PartnerProbeSeconds { get; set; } = 25;
|
||||
|
||||
/// <summary>
|
||||
/// The interval between partner reachability probes, in milliseconds. Default 500ms.
|
||||
/// </summary>
|
||||
public int PartnerProbeIntervalMs { get; set; } = 500;
|
||||
|
||||
/// <summary>
|
||||
/// The per-probe TCP connect timeout, in milliseconds. Default 1000ms.
|
||||
/// </summary>
|
||||
public int ProbeConnectTimeoutMs { get; set; } = 1000;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user