feat(cluster): simultaneous-cold-start split-brain guard (#33)

Port OtOpcUa's bootstrap guard (lmxopcua d1dac87f) to ScadaBridge's
BuildHocon bootstrap. Both pair nodes are self-first seeds, so a true
simultaneous cold start (shared site power event) races FirstSeedNodeProcess
on both and forms two 1-node clusters that never merge.

Opt-in dark switch ScadaBridge:Cluster:BootstrapGuard:Enabled (default off,
guard-off behavior byte-identical). When on: BuildHocon emits an empty seed
list so Akka does not auto-join, and ClusterBootstrapCoordinator (IHostedService,
registered in both the Central and Site composition roots) picks the join order
from ClusterBootstrapGuard's pure decision core — the lower canonical host:port
is the founder (self-first, forms immediately); the higher node TCP-probes the
founder up to PartnerProbeSeconds and joins peer-first if reachable, else
self-first (cold-start-alone preserved). Decides before a single JoinSeedNodes,
never re-forms mid-handshake.

Review notes carried over: case-insensitive founder tie-break; fail-fast
validation of probe timings when enabled; higher-node-cold-start-alone covered
by a real-ActorSystem test. 21 unit + 5 real-cluster tests (incl. the headline
both-cold-start-together-form-one-cluster).
This commit is contained in:
Joseph Doherty
2026-07-24 08:55:48 -04:00
parent bb138d5254
commit 248676ed16
10 changed files with 895 additions and 2 deletions
@@ -261,8 +261,16 @@ public class AkkaHostedService : IHostedService
TimeSpan transportHeartbeat,
TimeSpan transportFailure)
{
var seedNodesStr = string.Join(",",
clusterOptions.SeedNodes.Select(QuoteHocon));
// Simultaneous-cold-start split-brain guard (Gitea #33): when enabled the node must NOT
// auto-join from its config seeds — Akka's FirstSeedNodeProcess is exactly what races the
// peer and forms two 1-node clusters. Emit an EMPTY seed list so the ActorSystem comes up
// unjoined, and ClusterBootstrapCoordinator issues the single reachability-gated
// JoinSeedNodes (founder self-first / joiner peer-first). Default OFF, so guard-off configs
// render byte-identical to before.
var effectiveSeeds = clusterOptions.BootstrapGuard.Enabled
? Enumerable.Empty<string>()
: clusterOptions.SeedNodes;
var seedNodesStr = string.Join(",", effectiveSeeds.Select(QuoteHocon));
var rolesStr = string.Join(",", roles.Select(QuoteHocon));
// auto-down (default): AutoDowning provider — the leader among the reachable