feat(cluster): simultaneous-cold-start split-brain guard (#33)

Port OtOpcUa's bootstrap guard (lmxopcua d1dac87f) to ScadaBridge's
BuildHocon bootstrap. Both pair nodes are self-first seeds, so a true
simultaneous cold start (shared site power event) races FirstSeedNodeProcess
on both and forms two 1-node clusters that never merge.

Opt-in dark switch ScadaBridge:Cluster:BootstrapGuard:Enabled (default off,
guard-off behavior byte-identical). When on: BuildHocon emits an empty seed
list so Akka does not auto-join, and ClusterBootstrapCoordinator (IHostedService,
registered in both the Central and Site composition roots) picks the join order
from ClusterBootstrapGuard's pure decision core — the lower canonical host:port
is the founder (self-first, forms immediately); the higher node TCP-probes the
founder up to PartnerProbeSeconds and joins peer-first if reachable, else
self-first (cold-start-alone preserved). Decides before a single JoinSeedNodes,
never re-forms mid-handshake.

Review notes carried over: case-insensitive founder tie-break; fail-fast
validation of probe timings when enabled; higher-node-cold-start-alone covered
by a real-ActorSystem test. 21 unit + 5 real-cluster tests (incl. the headline
both-cold-start-together-form-one-cluster).

src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterBootstrapGuard.cs

@@ -0,0 +1,155 @@
+using System.Text.RegularExpressions;
+
+namespace ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
+
+/// <summary>
+/// Pure decision logic for the simultaneous-cold-start split-brain guard (Gitea #33).
+/// </summary>
+/// <remarks>
+///     <para>
+///         Both nodes of a 2-node pair are self-first seeds so <b>either</b> can cold-start alone when
+///         its partner is dead (see <see cref="ClusterOptions.SeedNodes"/>). The cost is that when
+///         BOTH cold-start at the same instant, each runs Akka's <c>FirstSeedNodeProcess</c>, times out
+///         waiting for the other, and forms its OWN single-node cluster — a split brain (two oldest
+///         nodes, two singletons, dual-active). Two independent clusters do not auto-merge, so the
+///         split persists until an operator restarts one side. It bites on any event that powers up
+///         both site VMs together (site power restoration, hypervisor host reboot).
+///     </para>
+///     <para>
+///         This guard breaks the symmetry deterministically without giving up cold-start-alone. The
+///         node with the lexicographically <b>lower</b> canonical address is the <i>preferred
+///         founder</i>: it always uses self-first order and forms immediately. The <b>higher</b> node
+///         first probes whether its partner's Akka endpoint is reachable (see
+///         <see cref="ClusterBootstrapCoordinator"/>): reachable ⇒ peer-first order so it JOINS the
+///         founder rather than racing it; unreachable after the probe window ⇒ the partner is genuinely
+///         down, so it falls back to self-first and forms alone. Exactly one node founds when both are
+///         present; the lower node always founds when it starts alone.
+///     </para>
+///     <para>
+///         <b>Residual trade-off (accepted).</b> Once the higher node observes the partner reachable it
+///         commits to peer-first — Akka's <c>JoinSeedNodeProcess</c> retries <c>InitJoin</c> forever and
+///         never self-forms. If the founder dies in the small window between the probe succeeding and
+///         the join completing, the higher node hangs unjoined until it is restarted (a restart re-runs
+///         the guard, finds the founder unreachable, and self-forms). We deliberately do NOT re-decide
+///         mid-handshake — that is exactly the retired <c>SelfFormAfter</c> failure mode. Instead
+///         <see cref="ClusterBootstrapCoordinator"/> makes the hang operator-visible with a warning if
+///         the node has not come Up within a bounded time after committing peer-first.
+///     </para>
+///     <para>
+///         This decides the join order BEFORE issuing a single <c>JoinSeedNodes</c>, from an explicit
+///         reachability signal — unlike a timer-based watchdog, which fires a <c>Join(self)</c>
+///         mid-handshake on a bare timeout and could not tell "no seed answered" from "a join is in
+///         flight", islanding a node a failover had just bounced (the rejected self-form-watchdog
+///         design; see <c>SelfFirstSeedBootstrapTests</c>).
+///     </para>
+/// </remarks>
+public static class ClusterBootstrapGuard
+{
+    private static readonly Regex SeedPattern =
+        new(@"^akka(?:\.[a-z0-9]+)?://[^@/]+@(?<host>[^:/]+):(?<port>\d+)", RegexOptions.IgnoreCase | RegexOptions.Compiled);
+
+    /// <summary>The analyzed bootstrap role of this node within its seed list.</summary>
+    /// <param name="Applies">True only when the guard engages: exactly two seeds, one of them this node.</param>
+    /// <param name="IsFounder">True when this node is the preferred founder (lower address) — form immediately, no probe.</param>
+    /// <param name="PartnerHost">The partner's advertised host (to probe when this node is the higher one); null when not applicable.</param>
+    /// <param name="PartnerPort">The partner's Akka port.</param>
+    /// <param name="SelfFirstOrder"><c>[self, partner]</c> — self-first; forms a new cluster when no peer answers.</param>
+    /// <param name="PeerFirstOrder"><c>[partner, self]</c> — peer-first; joins the partner's cluster, never self-forms while it is up.</param>
+    public sealed record BootstrapRole(
+        bool Applies,
+        bool IsFounder,
+        string? PartnerHost,
+        int PartnerPort,
+        string[] SelfFirstOrder,
+        string[] PeerFirstOrder);
+
+    /// <summary>
+    /// Analyzes the configured seed list to decide this node's bootstrap role. The guard engages only
+    /// for the pair shape (exactly two seeds, one of them this node); any other shape
+    /// (single-seed / single-node install, three+ seeds, self absent) returns
+    /// <see cref="BootstrapRole.Applies"/> = false and the caller joins the configured seeds unchanged.
+    /// </summary>
+    /// <param name="seeds">The configured seed URIs (self-first by convention, but order is not relied on here).</param>
+    /// <param name="selfHost">This node's advertised host (<c>NodeOptions.NodeHostname</c>).</param>
+    /// <param name="selfPort">This node's Akka remoting port (<c>NodeOptions.RemotingPort</c>).</param>
+    /// <returns>The analyzed role; never null.</returns>
+    public static BootstrapRole Analyze(IReadOnlyList<string> seeds, string selfHost, int selfPort)
+    {
+        ArgumentNullException.ThrowIfNull(seeds);
+        ArgumentException.ThrowIfNullOrWhiteSpace(selfHost);
+
+        var na = new BootstrapRole(false, false, null, 0, Array.Empty<string>(), Array.Empty<string>());
+        if (seeds.Count != 2) return na;
+
+        var selfKey = Key(selfHost, selfPort);
+        string? self = null, partner = null;
+        string? partnerHost = null;
+        var partnerPort = 0;
+
+        foreach (var seed in seeds)
+        {
+            if (!TryParse(seed, out var host, out var port)) return na;
+            if (string.Equals(Key(host, port), selfKey, StringComparison.OrdinalIgnoreCase))
+                self = seed;
+            else
+            {
+                partner = seed;
+                partnerHost = host;
+                partnerPort = port;
+            }
+        }
+
+        // Self must be exactly one of the two, and the other must be a distinct partner.
+        if (self is null || partner is null || partnerHost is null) return na;
+
+        var selfFirst = new[] { self, partner };
+        var peerFirst = new[] { partner, self };
+
+        // Deterministic tie-break: the lower canonical address is the preferred founder. Both nodes
+        // compute the same comparison (same two seeds), so exactly one is the founder. Case-INSENSITIVE
+        // to match how self/partner are classified above (and StartupValidator.SeedNodeIsSelf):
+        // container/DNS hostnames are conventionally case-insensitive, and a casing difference between
+        // the two sides' seed config must NOT make both think they are the founder (which would reopen
+        // the very split this guard closes).
+        var isFounder = string.Compare(
+            Key(selfHost, selfPort),
+            Key(partnerHost, partnerPort),
+            StringComparison.OrdinalIgnoreCase) < 0;
+
+        return new BootstrapRole(true, isFounder, partnerHost, partnerPort, selfFirst, peerFirst);
+    }
+
+    /// <summary>
+    /// The order the higher (non-founder) node uses once it has learned whether its partner is
+    /// reachable: peer-first when the founder is up (join it), self-first when the founder is absent
+    /// (form alone — cold-start-alone preserved).
+    /// </summary>
+    /// <param name="role">The analyzed role (must be applicable and NOT the founder).</param>
+    /// <param name="partnerReachable">Whether the partner's Akka endpoint became reachable within the probe window.</param>
+    /// <returns>The seed order to pass to <c>Cluster.JoinSeedNodes</c>.</returns>
+    public static string[] HigherNodeOrder(BootstrapRole role, bool partnerReachable)
+    {
+        ArgumentNullException.ThrowIfNull(role);
+        return partnerReachable ? role.PeerFirstOrder : role.SelfFirstOrder;
+    }
+
+    /// <summary>Parses <c>akka.tcp://system@host:port[/...]</c> into host + port.</summary>
+    /// <param name="seed">The seed URI.</param>
+    /// <param name="host">The parsed advertised host.</param>
+    /// <param name="port">The parsed Akka port.</param>
+    /// <returns>True when the seed matched the expected shape.</returns>
+    public static bool TryParse(string? seed, out string host, out int port)
+    {
+        host = string.Empty;
+        port = 0;
+        if (string.IsNullOrWhiteSpace(seed)) return false;
+
+        var m = SeedPattern.Match(seed.Trim());
+        if (!m.Success) return false;
+        if (!int.TryParse(m.Groups["port"].Value, out port)) return false;
+        host = m.Groups["host"].Value;
+        return true;
+    }
+
+    private static string Key(string host, int port) => $"{host}:{port}";
+}

src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs

@@ -114,4 +114,50 @@ public class ClusterOptions
     /// dials forever — log noise and a defeated validation intent (review 01).
     /// </summary>
     public bool AllowSingleNodeCluster { get; set; }
+
+    /// <summary>
+    /// The simultaneous-cold-start split-brain guard (<c>ScadaBridge:Cluster:BootstrapGuard</c>).
+    /// Default OFF — a dark switch, so existing deployments and tests keep Akka's config-driven
+    /// self-first auto-join byte-identical. See <see cref="ClusterBootstrapGuard"/> for the
+    /// decision logic and <c>ClusterBootstrapCoordinator</c> for the runtime (Gitea #33).
+    /// </summary>
+    public ClusterBootstrapGuardOptions BootstrapGuard { get; set; } = new();
+}
+
+/// <summary>
+/// Configuration for the simultaneous-cold-start split-brain guard. Both nodes of a pair are
+/// self-first seeds so <b>either</b> can cold-start alone when its partner is dead — but the cost
+/// is that when BOTH cold-start at the same instant each runs Akka's <c>FirstSeedNodeProcess</c>,
+/// times out waiting for the other, and forms its own single-node cluster (a split brain that does
+/// not auto-merge). When <see cref="Enabled"/>, the node does NOT auto-join from its config seeds
+/// (<c>AkkaHostedService.BuildHocon</c> emits an empty seed list); a coordinator picks the join
+/// order — founder self-first / joiner peer-first — after a reachability probe. See
+/// <see cref="ClusterBootstrapGuard"/>.
+/// </summary>
+public sealed class ClusterBootstrapGuardOptions
+{
+    /// <summary>
+    /// Whether the guard is active. Default <see langword="false"/> — Akka auto-joins from the
+    /// config seed list exactly as before. Only meaningful on a node that is one of its own two
+    /// pair seeds; inert everywhere else (single-seed site node, self absent, 3+ seeds).
+    /// </summary>
+    public bool Enabled { get; set; }
+
+    /// <summary>
+    /// How long the higher-address node probes its partner's Akka endpoint before concluding the
+    /// partner is dead and forming alone. Must comfortably exceed the partner's worst-case
+    /// process-start-to-Akka-bind time so a slow-but-alive partner is never mistaken for a dead one
+    /// (which would re-open the split). Default 25s. Validated <c>&gt; 0</c> at boot when the guard is on.
+    /// </summary>
+    public int PartnerProbeSeconds { get; set; } = 25;
+
+    /// <summary>
+    /// The interval between partner reachability probes, in milliseconds. Default 500ms.
+    /// </summary>
+    public int PartnerProbeIntervalMs { get; set; } = 500;
+
+    /// <summary>
+    /// The per-probe TCP connect timeout, in milliseconds. Default 1000ms.
+    /// </summary>
+    public int ProbeConnectTimeoutMs { get; set; } = 1000;
 }

src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptionsValidator.cs

@@ -30,6 +30,8 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
     /// <inheritdoc />
     protected override void Validate(ValidationBuilder builder, ClusterOptions options)
     {
+        ValidateBootstrapGuard(builder, options.BootstrapGuard);
+
         // The design doc states "both nodes are seed nodes — each node lists
         // both itself and its partner" so a properly-configured deployment lists
         // two. Accepting a single-seed configuration silently defeats the
@@ -78,4 +80,33 @@ public sealed class ClusterOptionsValidator : OptionsValidatorBase<ClusterOption
             + "oldest node can run as an isolated single-node cluster during a partition while the "
             + "younger node forms its own, producing two live clusters.");
     }
+
+    /// <summary>
+    /// When the bootstrap guard is enabled (Gitea #33), its timing knobs must be positive. A
+    /// zero/negative <see cref="ClusterBootstrapGuardOptions.PartnerProbeSeconds"/> in particular
+    /// silently degrades the guard to "never wait, always conclude the partner is dead" — the
+    /// higher node would form alone immediately and re-open the very split the guard exists to
+    /// close. Fail fast at boot rather than producing that silent degradation. Nothing is checked
+    /// when the guard is off (the knobs are inert), so a disabled guard never blocks a boot.
+    /// </summary>
+    private static void ValidateBootstrapGuard(ValidationBuilder builder, ClusterBootstrapGuardOptions? guard)
+    {
+        if (guard is null || !guard.Enabled)
+        {
+            return;
+        }
+
+        builder.RequireThat(guard.PartnerProbeSeconds > 0,
+            $"ClusterOptions.BootstrapGuard.PartnerProbeSeconds must be > 0 when the guard is enabled "
+            + $"(was {guard.PartnerProbeSeconds}); a non-positive value makes the higher node conclude its "
+            + "partner is dead without waiting and form alone, re-opening the split the guard prevents.");
+
+        builder.RequireThat(guard.PartnerProbeIntervalMs > 0,
+            $"ClusterOptions.BootstrapGuard.PartnerProbeIntervalMs must be > 0 when the guard is enabled "
+            + $"(was {guard.PartnerProbeIntervalMs}).");
+
+        builder.RequireThat(guard.ProbeConnectTimeoutMs > 0,
+            $"ClusterOptions.BootstrapGuard.ProbeConnectTimeoutMs must be > 0 when the guard is enabled "
+            + $"(was {guard.ProbeConnectTimeoutMs}).");
+    }
 }