feat(cluster): simultaneous-cold-start split-brain guard (#33)

Port OtOpcUa's bootstrap guard (lmxopcua d1dac87f) to ScadaBridge's
BuildHocon bootstrap. Both pair nodes are self-first seeds, so a true
simultaneous cold start (shared site power event) races FirstSeedNodeProcess
on both and forms two 1-node clusters that never merge.

Opt-in dark switch ScadaBridge:Cluster:BootstrapGuard:Enabled (default off,
guard-off behavior byte-identical). When on: BuildHocon emits an empty seed
list so Akka does not auto-join, and ClusterBootstrapCoordinator (IHostedService,
registered in both the Central and Site composition roots) picks the join order
from ClusterBootstrapGuard's pure decision core — the lower canonical host:port
is the founder (self-first, forms immediately); the higher node TCP-probes the
founder up to PartnerProbeSeconds and joins peer-first if reachable, else
self-first (cold-start-alone preserved). Decides before a single JoinSeedNodes,
never re-forms mid-handshake.

Review notes carried over: case-insensitive founder tie-break; fail-fast
validation of probe timings when enabled; higher-node-cold-start-alone covered
by a real-ActorSystem test. 21 unit + 5 real-cluster tests (incl. the headline
both-cold-start-together-form-one-cluster).
This commit is contained in:
Joseph Doherty
2026-07-24 08:55:48 -04:00
parent bb138d5254
commit 248676ed16
10 changed files with 895 additions and 2 deletions
@@ -30,6 +30,8 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
/// <inheritdoc />
protected override void Validate(ValidationBuilder builder, ClusterOptions options)
{
ValidateBootstrapGuard(builder, options.BootstrapGuard);
// The design doc states "both nodes are seed nodes — each node lists
// both itself and its partner" so a properly-configured deployment lists
// two. Accepting a single-seed configuration silently defeats the
@@ -78,4 +80,33 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
+ "oldest node can run as an isolated single-node cluster during a partition while the "
+ "younger node forms its own, producing two live clusters.");
}
/// <summary>
/// When the bootstrap guard is enabled (Gitea #33), its timing knobs must be positive. A
/// zero/negative <see cref="ClusterBootstrapGuardOptions.PartnerProbeSeconds"/> in particular
/// silently degrades the guard to "never wait, always conclude the partner is dead" — the
/// higher node would form alone immediately and re-open the very split the guard exists to
/// close. Fail fast at boot rather than producing that silent degradation. Nothing is checked
/// when the guard is off (the knobs are inert), so a disabled guard never blocks a boot.
/// </summary>
private static void ValidateBootstrapGuard(ValidationBuilder builder, ClusterBootstrapGuardOptions? guard)
{
if (guard is null || !guard.Enabled)
{
return;
}
builder.RequireThat(guard.PartnerProbeSeconds > 0,
$"ClusterOptions.BootstrapGuard.PartnerProbeSeconds must be > 0 when the guard is enabled "
+ $"(was {guard.PartnerProbeSeconds}); a non-positive value makes the higher node conclude its "
+ "partner is dead without waiting and form alone, re-opening the split the guard prevents.");
builder.RequireThat(guard.PartnerProbeIntervalMs > 0,
$"ClusterOptions.BootstrapGuard.PartnerProbeIntervalMs must be > 0 when the guard is enabled "
+ $"(was {guard.PartnerProbeIntervalMs}).");
builder.RequireThat(guard.ProbeConnectTimeoutMs > 0,
$"ClusterOptions.BootstrapGuard.ProbeConnectTimeoutMs must be > 0 when the guard is enabled "
+ $"(was {guard.ProbeConnectTimeoutMs}).");
}
}