feat(cluster): simultaneous-cold-start split-brain guard (#33)

Port OtOpcUa's bootstrap guard (lmxopcua d1dac87f) to ScadaBridge's
BuildHocon bootstrap. Both pair nodes are self-first seeds, so a true
simultaneous cold start (shared site power event) races FirstSeedNodeProcess
on both and forms two 1-node clusters that never merge.

Opt-in dark switch ScadaBridge:Cluster:BootstrapGuard:Enabled (default off,
guard-off behavior byte-identical). When on: BuildHocon emits an empty seed
list so Akka does not auto-join, and ClusterBootstrapCoordinator (IHostedService,
registered in both the Central and Site composition roots) picks the join order
from ClusterBootstrapGuard's pure decision core — the lower canonical host:port
is the founder (self-first, forms immediately); the higher node TCP-probes the
founder up to PartnerProbeSeconds and joins peer-first if reachable, else
self-first (cold-start-alone preserved). Decides before a single JoinSeedNodes,
never re-forms mid-handshake.

Review notes carried over: case-insensitive founder tie-break; fail-fast
validation of probe timings when enabled; higher-node-cold-start-alone covered
by a real-ActorSystem test. 21 unit + 5 real-cluster tests (incl. the headline
both-cold-start-together-form-one-cluster).
This commit is contained in:
Joseph Doherty
2026-07-24 08:55:48 -04:00
parent bb138d5254
commit 248676ed16
10 changed files with 895 additions and 2 deletions
@@ -114,4 +114,50 @@ public class ClusterOptions
/// dials forever — log noise and a defeated validation intent (review 01).
/// </summary>
public bool AllowSingleNodeCluster { get; set; }
/// <summary>
/// The simultaneous-cold-start split-brain guard (<c>ScadaBridge:Cluster:BootstrapGuard</c>).
/// Default OFF — a dark switch, so existing deployments and tests keep Akka's config-driven
/// self-first auto-join byte-identical. See <see cref="ClusterBootstrapGuard"/> for the
/// decision logic and <c>ClusterBootstrapCoordinator</c> for the runtime (Gitea #33).
/// </summary>
public ClusterBootstrapGuardOptions BootstrapGuard { get; set; } = new();
}
/// <summary>
/// Configuration for the simultaneous-cold-start split-brain guard. Both nodes of a pair are
/// self-first seeds so <b>either</b> can cold-start alone when its partner is dead — but the cost
/// is that when BOTH cold-start at the same instant each runs Akka's <c>FirstSeedNodeProcess</c>,
/// times out waiting for the other, and forms its own single-node cluster (a split brain that does
/// not auto-merge). When <see cref="Enabled"/>, the node does NOT auto-join from its config seeds
/// (<c>AkkaHostedService.BuildHocon</c> emits an empty seed list); a coordinator picks the join
/// order — founder self-first / joiner peer-first — after a reachability probe. See
/// <see cref="ClusterBootstrapGuard"/>.
/// </summary>
public sealed class ClusterBootstrapGuardOptions
{
/// <summary>
/// Whether the guard is active. Default <see langword="false"/> — Akka auto-joins from the
/// config seed list exactly as before. Only meaningful on a node that is one of its own two
/// pair seeds; inert everywhere else (single-seed site node, self absent, 3+ seeds).
/// </summary>
public bool Enabled { get; set; }
/// <summary>
/// How long the higher-address node probes its partner's Akka endpoint before concluding the
/// partner is dead and forming alone. Must comfortably exceed the partner's worst-case
/// process-start-to-Akka-bind time so a slow-but-alive partner is never mistaken for a dead one
/// (which would re-open the split). Default 25s. Validated <c>&gt; 0</c> at boot when the guard is on.
/// </summary>
public int PartnerProbeSeconds { get; set; } = 25;
/// <summary>
/// The interval between partner reachability probes, in milliseconds. Default 500ms.
/// </summary>
public int PartnerProbeIntervalMs { get; set; } = 500;
/// <summary>
/// The per-probe TCP connect timeout, in milliseconds. Default 1000ms.
/// </summary>
public int ProbeConnectTimeoutMs { get; set; } = 1000;
}