feat(cluster): simultaneous-cold-start split-brain guard (#33)

Port OtOpcUa's bootstrap guard (lmxopcua d1dac87f) to ScadaBridge's
BuildHocon bootstrap. Both pair nodes are self-first seeds, so a true
simultaneous cold start (shared site power event) races FirstSeedNodeProcess
on both and forms two 1-node clusters that never merge.

Opt-in dark switch ScadaBridge:Cluster:BootstrapGuard:Enabled (default off,
guard-off behavior byte-identical). When on: BuildHocon emits an empty seed
list so Akka does not auto-join, and ClusterBootstrapCoordinator (IHostedService,
registered in both the Central and Site composition roots) picks the join order
from ClusterBootstrapGuard's pure decision core — the lower canonical host:port
is the founder (self-first, forms immediately); the higher node TCP-probes the
founder up to PartnerProbeSeconds and joins peer-first if reachable, else
self-first (cold-start-alone preserved). Decides before a single JoinSeedNodes,
never re-forms mid-handshake.

Review notes carried over: case-insensitive founder tie-break; fail-fast
validation of probe timings when enabled; higher-node-cold-start-alone covered
by a real-ActorSystem test. 21 unit + 5 real-cluster tests (incl. the headline
both-cold-start-together-form-one-cluster).
This commit is contained in:
Joseph Doherty
2026-07-24 08:55:48 -04:00
parent bb138d5254
commit 248676ed16
10 changed files with 895 additions and 2 deletions
+14
View File
@@ -394,6 +394,20 @@ try
builder.Services.AddSingleton<Akka.Actor.ActorSystem>(sp =>
sp.GetRequiredService<AkkaHostedService>().GetOrCreateActorSystem());
// Simultaneous-cold-start split-brain guard (Gitea #33, dark switch
// ScadaBridge:Cluster:BootstrapGuard:Enabled, default off). Registered AFTER the
// AkkaHostedService/ActorSystem bridge so its StartAsync resolves the system the main path
// built; when the guard is off it no-ops (Akka auto-joined from config seeds), so registering
// it unconditionally is safe. When on, the node started unjoined (empty HOCON seed list, see
// AkkaHostedService.BuildHocon) and this coordinator issues the single reachability-gated
// JoinSeedNodes. It resolves the ActorSystem lazily via GetOrCreateActorSystem so it never
// races startup or caches a null.
builder.Services.AddHostedService(sp => new ClusterBootstrapCoordinator(
() => sp.GetRequiredService<AkkaHostedService>().GetOrCreateActorSystem(),
sp.GetRequiredService<IOptions<ClusterOptions>>(),
sp.GetRequiredService<IOptions<NodeOptions>>(),
sp.GetRequiredService<ILogger<ClusterBootstrapCoordinator>>()));
// Register the production IActiveNodeGate implementation so
// standby-node gating is actually enforced (the InboundApiEndpointFilter
// consults IActiveNodeGate and defaults to "allow" when none is registered,