feat(cluster): simultaneous-cold-start split-brain guard (#33)
Port OtOpcUa's bootstrap guard (lmxopcua d1dac87f) to ScadaBridge's BuildHocon bootstrap. Both pair nodes are self-first seeds, so a true simultaneous cold start (shared site power event) races FirstSeedNodeProcess on both and forms two 1-node clusters that never merge. Opt-in dark switch ScadaBridge:Cluster:BootstrapGuard:Enabled (default off, guard-off behavior byte-identical). When on: BuildHocon emits an empty seed list so Akka does not auto-join, and ClusterBootstrapCoordinator (IHostedService, registered in both the Central and Site composition roots) picks the join order from ClusterBootstrapGuard's pure decision core — the lower canonical host:port is the founder (self-first, forms immediately); the higher node TCP-probes the founder up to PartnerProbeSeconds and joins peer-first if reachable, else self-first (cold-start-alone preserved). Decides before a single JoinSeedNodes, never re-forms mid-handshake. Review notes carried over: case-insensitive founder tie-break; fail-fast validation of probe timings when enabled; higher-node-cold-start-alone covered by a real-ActorSystem test. 21 unit + 5 real-cluster tests (incl. the headline both-cold-start-together-form-one-cluster).
This commit is contained in:
@@ -30,6 +30,8 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
|
||||
/// <inheritdoc />
|
||||
protected override void Validate(ValidationBuilder builder, ClusterOptions options)
|
||||
{
|
||||
ValidateBootstrapGuard(builder, options.BootstrapGuard);
|
||||
|
||||
// The design doc states "both nodes are seed nodes — each node lists
|
||||
// both itself and its partner" so a properly-configured deployment lists
|
||||
// two. Accepting a single-seed configuration silently defeats the
|
||||
@@ -78,4 +80,33 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
|
||||
+ "oldest node can run as an isolated single-node cluster during a partition while the "
|
||||
+ "younger node forms its own, producing two live clusters.");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// When the bootstrap guard is enabled (Gitea #33), its timing knobs must be positive. A
|
||||
/// zero/negative <see cref="ClusterBootstrapGuardOptions.PartnerProbeSeconds"/> in particular
|
||||
/// silently degrades the guard to "never wait, always conclude the partner is dead" — the
|
||||
/// higher node would form alone immediately and re-open the very split the guard exists to
|
||||
/// close. Fail fast at boot rather than producing that silent degradation. Nothing is checked
|
||||
/// when the guard is off (the knobs are inert), so a disabled guard never blocks a boot.
|
||||
/// </summary>
|
||||
private static void ValidateBootstrapGuard(ValidationBuilder builder, ClusterBootstrapGuardOptions? guard)
|
||||
{
|
||||
if (guard is null || !guard.Enabled)
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
builder.RequireThat(guard.PartnerProbeSeconds > 0,
|
||||
$"ClusterOptions.BootstrapGuard.PartnerProbeSeconds must be > 0 when the guard is enabled "
|
||||
+ $"(was {guard.PartnerProbeSeconds}); a non-positive value makes the higher node conclude its "
|
||||
+ "partner is dead without waiting and form alone, re-opening the split the guard prevents.");
|
||||
|
||||
builder.RequireThat(guard.PartnerProbeIntervalMs > 0,
|
||||
$"ClusterOptions.BootstrapGuard.PartnerProbeIntervalMs must be > 0 when the guard is enabled "
|
||||
+ $"(was {guard.PartnerProbeIntervalMs}).");
|
||||
|
||||
builder.RequireThat(guard.ProbeConnectTimeoutMs > 0,
|
||||
$"ClusterOptions.BootstrapGuard.ProbeConnectTimeoutMs must be > 0 when the guard is enabled "
|
||||
+ $"(was {guard.ProbeConnectTimeoutMs}).");
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user