feat(cluster): simultaneous-cold-start split-brain guard (#33)

Port OtOpcUa's bootstrap guard (lmxopcua d1dac87f) to ScadaBridge's
BuildHocon bootstrap. Both pair nodes are self-first seeds, so a true
simultaneous cold start (shared site power event) races FirstSeedNodeProcess
on both and forms two 1-node clusters that never merge.

Opt-in dark switch ScadaBridge:Cluster:BootstrapGuard:Enabled (default off,
guard-off behavior byte-identical). When on: BuildHocon emits an empty seed
list so Akka does not auto-join, and ClusterBootstrapCoordinator (IHostedService,
registered in both the Central and Site composition roots) picks the join order
from ClusterBootstrapGuard's pure decision core — the lower canonical host:port
is the founder (self-first, forms immediately); the higher node TCP-probes the
founder up to PartnerProbeSeconds and joins peer-first if reachable, else
self-first (cold-start-alone preserved). Decides before a single JoinSeedNodes,
never re-forms mid-handshake.

Review notes carried over: case-insensitive founder tie-break; fail-fast
validation of probe timings when enabled; higher-node-cold-start-alone covered
by a real-ActorSystem test. 21 unit + 5 real-cluster tests (incl. the headline
both-cold-start-together-form-one-cluster).
This commit is contained in:
Joseph Doherty
2026-07-24 08:55:48 -04:00
parent bb138d5254
commit 248676ed16
10 changed files with 895 additions and 2 deletions
@@ -168,6 +168,19 @@ public static class SiteServiceRegistration
services.AddSingleton<Akka.Actor.ActorSystem>(sp =>
sp.GetRequiredService<AkkaHostedService>().GetOrCreateActorSystem());
// Simultaneous-cold-start split-brain guard (Gitea #33, dark switch
// ScadaBridge:Cluster:BootstrapGuard:Enabled, default off). This is the case the guard was
// built for — a shared power event that powers up both site VMs together. Registered AFTER
// the AkkaHostedService/ActorSystem bridge (mirrors the Central composition root in
// Program.cs); it no-ops when the guard is off, so registering it unconditionally is safe.
// When on, the node started unjoined (empty HOCON seed list) and this coordinator issues the
// single reachability-gated JoinSeedNodes, resolving the ActorSystem lazily.
services.AddHostedService(sp => new ClusterBootstrapCoordinator(
() => sp.GetRequiredService<AkkaHostedService>().GetOrCreateActorSystem(),
sp.GetRequiredService<IOptions<ClusterOptions>>(),
sp.GetRequiredService<IOptions<NodeOptions>>(),
sp.GetRequiredService<ILogger<ClusterBootstrapCoordinator>>()));
// Cluster node status provider for health reports
services.AddSingleton<IClusterNodeProvider>(sp =>
{