Files
ScadaBridge/src/ZB.MOM.WW.ScadaBridge.ClusterInfrastructure/ClusterOptions.cs
T
Joseph Doherty 248676ed16 feat(cluster): simultaneous-cold-start split-brain guard (#33)
Port OtOpcUa's bootstrap guard (lmxopcua d1dac87f) to ScadaBridge's
BuildHocon bootstrap. Both pair nodes are self-first seeds, so a true
simultaneous cold start (shared site power event) races FirstSeedNodeProcess
on both and forms two 1-node clusters that never merge.

Opt-in dark switch ScadaBridge:Cluster:BootstrapGuard:Enabled (default off,
guard-off behavior byte-identical). When on: BuildHocon emits an empty seed
list so Akka does not auto-join, and ClusterBootstrapCoordinator (IHostedService,
registered in both the Central and Site composition roots) picks the join order
from ClusterBootstrapGuard's pure decision core — the lower canonical host:port
is the founder (self-first, forms immediately); the higher node TCP-probes the
founder up to PartnerProbeSeconds and joins peer-first if reachable, else
self-first (cold-start-alone preserved). Decides before a single JoinSeedNodes,
never re-forms mid-handshake.

Review notes carried over: case-insensitive founder tie-break; fail-fast
validation of probe timings when enabled; higher-node-cold-start-alone covered
by a real-ActorSystem test. 21 unit + 5 real-cluster tests (incl. the headline
both-cold-start-together-form-one-cluster).
2026-07-24 08:55:48 -04:00

164 lines
9.0 KiB
C#

namespace ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
/// <summary>
/// Cluster configuration model, bound from the <c>ScadaBridge:Cluster</c> section
/// of <c>appsettings.json</c> via the Options pattern.
/// <para>
/// This project owns the cluster <em>configuration contract</em>. The actual
/// Akka.NET bootstrap — building the HOCON from these values, starting the
/// <c>ActorSystem</c>, configuring the split-brain resolver and wiring
/// <c>CoordinatedShutdown</c> — lives in <c>ZB.MOM.WW.ScadaBridge.Host</c>
/// (see <c>Component-ClusterInfrastructure.md</c> → "Implementation Note — Code Placement").
/// </para>
/// <para>
/// Node-identity settings (remoting hostname/port, cluster role, site identifier,
/// gRPC port) are deliberately <em>not</em> here — they are owned by
/// <c>ZB.MOM.WW.ScadaBridge.Host.NodeOptions</c> (<c>ScadaBridge:Node</c> section). Local SQLite
/// storage paths are owned by the database / store-and-forward options. This class
/// holds only the cluster-formation and failure-detection settings shared by every node.
/// </para>
/// </summary>
public class ClusterOptions
{
// The previous `public const string SectionName = "ScadaBridge:Cluster";`
// was documented as "single source of truth so binding sites do not hard-code the
// magic string" but no caller ever read it — the Host's SiteServiceRegistration and
// StartupValidator both hard-code the literal directly. Wiring those binding sites
// to reference the constant lives in the Host's edit scope (a separate code-review
// task); rather than carry a public constant whose guarantee the code does not
// deliver, the constant is removed and the literal stays in the Host until the
// Host-side wiring is done. If a future Host change wants the constant back, add it
// when the binding sites can be updated in the same commit.
/// <summary>
/// Akka.NET cluster seed nodes. Both nodes are seed nodes — each node lists itself and its
/// partner.
/// <para>
/// <b>ORDER IS LOAD-BEARING (decision 2026-07-22): every node must list ITSELF first.</b>
/// Akka runs <c>FirstSeedNodeProcess</c> — the only bootstrap path that can form a NEW
/// cluster when no peer answers <c>InitJoin</c> — exclusively when <c>seed-nodes[0]</c> is
/// this node's own address; any other node runs <c>JoinSeedNodeProcess</c> and retries
/// <c>InitJoin</c> forever. So merely listing both nodes does NOT mean either can start
/// first: a node that lists its partner first can never cold-start while that partner is
/// down (the "registered outage gap", <c>docker/README.md</c>). Self-first ordering closes
/// it using Akka's own protocol, which — unlike an external self-form timer — is part of
/// the join handshake and so cannot mistake an in-flight join for an absent peer.
/// Enforced at boot by <c>StartupValidator</c>.
/// </para>
/// Must contain at least one entry.
/// </summary>
public List<string> SeedNodes { get; set; } = new();
/// <summary>
/// Downing strategy for unreachable members. Two supported values:
/// <list type="bullet">
/// <item><c>auto-down</c> (default, decision 2026-07-21) — availability-first: each
/// side downs the unreachable peer after <see cref="StableAfter"/>, so a hard crash
/// of EITHER node (oldest included) fails over to the survivor. The accepted trade:
/// a true network partition produces two live one-node clusters (dual-active) until
/// an operator restarts one side. Chosen because ScadaBridge pairs run one node per
/// VM with no shared lease infrastructure, and a stalled system is a bigger risk
/// than a rare partition.</item>
/// <item><c>keep-oldest</c> — partition-safe SBR: downs the side without the oldest
/// member. In a TWO-node cluster this makes a crash of the oldest/active node a
/// total outage: Akka's <c>down-if-alone</c> only rescues the survivor when its own
/// side has ≥2 members (verified against Akka.NET 1.5.62 <c>KeepOldest.Decide</c>
/// and live on the docker rig, 2026-07-21).</item>
/// </list>
/// Other SBR strategies are rejected: <c>static-quorum</c> with quorum 1 hits Akka's
/// <c>IsTooManyMembers</c> guard (2 &gt; 2*1-1) and downs ALL on any unreachability;
/// <c>keep-majority</c> just moves the fatal crash from the oldest to the
/// lowest-address node.
/// </summary>
public string SplitBrainResolverStrategy { get; set; } = "auto-down";
/// <summary>
/// Time the cluster membership must remain stable before the split-brain
/// resolver acts to down unreachable nodes. Must be positive. Default 15s.
/// </summary>
public TimeSpan StableAfter { get; set; } = TimeSpan.FromSeconds(15);
/// <summary>
/// Frequency of cluster failure-detector heartbeat messages between nodes.
/// Must be well below <see cref="FailureDetectionThreshold"/>. Default 2s.
/// </summary>
public TimeSpan HeartbeatInterval { get; set; } = TimeSpan.FromSeconds(2);
/// <summary>
/// Time without a heartbeat before a node is considered unreachable
/// (Akka's <c>acceptable-heartbeat-pause</c>). Default 10s.
/// </summary>
public TimeSpan FailureDetectionThreshold { get; set; } = TimeSpan.FromSeconds(10);
/// <summary>
/// Akka's <c>min-nr-of-members</c>. Must be <c>1</c>: after failover only one
/// node runs, and a value of <c>2</c> blocks the cluster singleton (Site Runtime
/// Deployment Manager) — and therefore all data collection — indefinitely.
/// </summary>
public int MinNrOfMembers { get; set; } = 1;
/// <summary>
/// The keep-oldest resolver's <c>down-if-alone</c> flag; only consulted when
/// <see cref="SplitBrainResolverStrategy"/> is <c>keep-oldest</c>. When <c>true</c>,
/// the oldest node downs itself if it finds it has no other reachable members,
/// rather than running as an isolated single-node cluster. Note that in a two-node
/// cluster this does NOT let the younger survivor take over from a crashed oldest —
/// Akka's alone-check requires the surviving side to have ≥2 members.
/// </summary>
public bool DownIfAlone { get; set; } = true;
/// <summary>
/// Acknowledges a deliberate single-node deployment: permits exactly one seed
/// node (normally ≥2 are required so either node can start first). Without this
/// flag a single-node install must list a phantom second seed, which the node
/// dials forever — log noise and a defeated validation intent (review 01).
/// </summary>
public bool AllowSingleNodeCluster { get; set; }
/// <summary>
/// The simultaneous-cold-start split-brain guard (<c>ScadaBridge:Cluster:BootstrapGuard</c>).
/// Default OFF — a dark switch, so existing deployments and tests keep Akka's config-driven
/// self-first auto-join byte-identical. See <see cref="ClusterBootstrapGuard"/> for the
/// decision logic and <c>ClusterBootstrapCoordinator</c> for the runtime (Gitea #33).
/// </summary>
public ClusterBootstrapGuardOptions BootstrapGuard { get; set; } = new();
}
/// <summary>
/// Configuration for the simultaneous-cold-start split-brain guard. Both nodes of a pair are
/// self-first seeds so <b>either</b> can cold-start alone when its partner is dead — but the cost
/// is that when BOTH cold-start at the same instant each runs Akka's <c>FirstSeedNodeProcess</c>,
/// times out waiting for the other, and forms its own single-node cluster (a split brain that does
/// not auto-merge). When <see cref="Enabled"/>, the node does NOT auto-join from its config seeds
/// (<c>AkkaHostedService.BuildHocon</c> emits an empty seed list); a coordinator picks the join
/// order — founder self-first / joiner peer-first — after a reachability probe. See
/// <see cref="ClusterBootstrapGuard"/>.
/// </summary>
public sealed class ClusterBootstrapGuardOptions
{
/// <summary>
/// Whether the guard is active. Default <see langword="false"/> — Akka auto-joins from the
/// config seed list exactly as before. Only meaningful on a node that is one of its own two
/// pair seeds; inert everywhere else (single-seed site node, self absent, 3+ seeds).
/// </summary>
public bool Enabled { get; set; }
/// <summary>
/// How long the higher-address node probes its partner's Akka endpoint before concluding the
/// partner is dead and forming alone. Must comfortably exceed the partner's worst-case
/// process-start-to-Akka-bind time so a slow-but-alive partner is never mistaken for a dead one
/// (which would re-open the split). Default 25s. Validated <c>&gt; 0</c> at boot when the guard is on.
/// </summary>
public int PartnerProbeSeconds { get; set; } = 25;
/// <summary>
/// The interval between partner reachability probes, in milliseconds. Default 500ms.
/// </summary>
public int PartnerProbeIntervalMs { get; set; } = 500;
/// <summary>
/// The per-probe TCP connect timeout, in milliseconds. Default 1000ms.
/// </summary>
public int ProbeConnectTimeoutMs { get; set; } = 1000;
}