248676ed16
Port OtOpcUa's bootstrap guard (lmxopcua d1dac87f) to ScadaBridge's BuildHocon bootstrap. Both pair nodes are self-first seeds, so a true simultaneous cold start (shared site power event) races FirstSeedNodeProcess on both and forms two 1-node clusters that never merge. Opt-in dark switch ScadaBridge:Cluster:BootstrapGuard:Enabled (default off, guard-off behavior byte-identical). When on: BuildHocon emits an empty seed list so Akka does not auto-join, and ClusterBootstrapCoordinator (IHostedService, registered in both the Central and Site composition roots) picks the join order from ClusterBootstrapGuard's pure decision core — the lower canonical host:port is the founder (self-first, forms immediately); the higher node TCP-probes the founder up to PartnerProbeSeconds and joins peer-first if reachable, else self-first (cold-start-alone preserved). Decides before a single JoinSeedNodes, never re-forms mid-handshake. Review notes carried over: case-insensitive founder tie-break; fail-fast validation of probe timings when enabled; higher-node-cold-start-alone covered by a real-ActorSystem test. 21 unit + 5 real-cluster tests (incl. the headline both-cold-start-together-form-one-cluster).
164 lines
9.0 KiB
C#
164 lines
9.0 KiB
C#
namespace ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
|
|
|
|
/// <summary>
|
|
/// Cluster configuration model, bound from the <c>ScadaBridge:Cluster</c> section
|
|
/// of <c>appsettings.json</c> via the Options pattern.
|
|
/// <para>
|
|
/// This project owns the cluster <em>configuration contract</em>. The actual
|
|
/// Akka.NET bootstrap — building the HOCON from these values, starting the
|
|
/// <c>ActorSystem</c>, configuring the split-brain resolver and wiring
|
|
/// <c>CoordinatedShutdown</c> — lives in <c>ZB.MOM.WW.ScadaBridge.Host</c>
|
|
/// (see <c>Component-ClusterInfrastructure.md</c> → "Implementation Note — Code Placement").
|
|
/// </para>
|
|
/// <para>
|
|
/// Node-identity settings (remoting hostname/port, cluster role, site identifier,
|
|
/// gRPC port) are deliberately <em>not</em> here — they are owned by
|
|
/// <c>ZB.MOM.WW.ScadaBridge.Host.NodeOptions</c> (<c>ScadaBridge:Node</c> section). Local SQLite
|
|
/// storage paths are owned by the database / store-and-forward options. This class
|
|
/// holds only the cluster-formation and failure-detection settings shared by every node.
|
|
/// </para>
|
|
/// </summary>
|
|
public class ClusterOptions
|
|
{
|
|
// The previous `public const string SectionName = "ScadaBridge:Cluster";`
|
|
// was documented as "single source of truth so binding sites do not hard-code the
|
|
// magic string" but no caller ever read it — the Host's SiteServiceRegistration and
|
|
// StartupValidator both hard-code the literal directly. Wiring those binding sites
|
|
// to reference the constant lives in the Host's edit scope (a separate code-review
|
|
// task); rather than carry a public constant whose guarantee the code does not
|
|
// deliver, the constant is removed and the literal stays in the Host until the
|
|
// Host-side wiring is done. If a future Host change wants the constant back, add it
|
|
// when the binding sites can be updated in the same commit.
|
|
|
|
/// <summary>
|
|
/// Akka.NET cluster seed nodes. Both nodes are seed nodes — each node lists itself and its
|
|
/// partner.
|
|
/// <para>
|
|
/// <b>ORDER IS LOAD-BEARING (decision 2026-07-22): every node must list ITSELF first.</b>
|
|
/// Akka runs <c>FirstSeedNodeProcess</c> — the only bootstrap path that can form a NEW
|
|
/// cluster when no peer answers <c>InitJoin</c> — exclusively when <c>seed-nodes[0]</c> is
|
|
/// this node's own address; any other node runs <c>JoinSeedNodeProcess</c> and retries
|
|
/// <c>InitJoin</c> forever. So merely listing both nodes does NOT mean either can start
|
|
/// first: a node that lists its partner first can never cold-start while that partner is
|
|
/// down (the "registered outage gap", <c>docker/README.md</c>). Self-first ordering closes
|
|
/// it using Akka's own protocol, which — unlike an external self-form timer — is part of
|
|
/// the join handshake and so cannot mistake an in-flight join for an absent peer.
|
|
/// Enforced at boot by <c>StartupValidator</c>.
|
|
/// </para>
|
|
/// Must contain at least one entry.
|
|
/// </summary>
|
|
public List<string> SeedNodes { get; set; } = new();
|
|
|
|
/// <summary>
|
|
/// Downing strategy for unreachable members. Two supported values:
|
|
/// <list type="bullet">
|
|
/// <item><c>auto-down</c> (default, decision 2026-07-21) — availability-first: each
|
|
/// side downs the unreachable peer after <see cref="StableAfter"/>, so a hard crash
|
|
/// of EITHER node (oldest included) fails over to the survivor. The accepted trade:
|
|
/// a true network partition produces two live one-node clusters (dual-active) until
|
|
/// an operator restarts one side. Chosen because ScadaBridge pairs run one node per
|
|
/// VM with no shared lease infrastructure, and a stalled system is a bigger risk
|
|
/// than a rare partition.</item>
|
|
/// <item><c>keep-oldest</c> — partition-safe SBR: downs the side without the oldest
|
|
/// member. In a TWO-node cluster this makes a crash of the oldest/active node a
|
|
/// total outage: Akka's <c>down-if-alone</c> only rescues the survivor when its own
|
|
/// side has ≥2 members (verified against Akka.NET 1.5.62 <c>KeepOldest.Decide</c>
|
|
/// and live on the docker rig, 2026-07-21).</item>
|
|
/// </list>
|
|
/// Other SBR strategies are rejected: <c>static-quorum</c> with quorum 1 hits Akka's
|
|
/// <c>IsTooManyMembers</c> guard (2 > 2*1-1) and downs ALL on any unreachability;
|
|
/// <c>keep-majority</c> just moves the fatal crash from the oldest to the
|
|
/// lowest-address node.
|
|
/// </summary>
|
|
public string SplitBrainResolverStrategy { get; set; } = "auto-down";
|
|
|
|
/// <summary>
|
|
/// Time the cluster membership must remain stable before the split-brain
|
|
/// resolver acts to down unreachable nodes. Must be positive. Default 15s.
|
|
/// </summary>
|
|
public TimeSpan StableAfter { get; set; } = TimeSpan.FromSeconds(15);
|
|
|
|
/// <summary>
|
|
/// Frequency of cluster failure-detector heartbeat messages between nodes.
|
|
/// Must be well below <see cref="FailureDetectionThreshold"/>. Default 2s.
|
|
/// </summary>
|
|
public TimeSpan HeartbeatInterval { get; set; } = TimeSpan.FromSeconds(2);
|
|
|
|
/// <summary>
|
|
/// Time without a heartbeat before a node is considered unreachable
|
|
/// (Akka's <c>acceptable-heartbeat-pause</c>). Default 10s.
|
|
/// </summary>
|
|
public TimeSpan FailureDetectionThreshold { get; set; } = TimeSpan.FromSeconds(10);
|
|
|
|
/// <summary>
|
|
/// Akka's <c>min-nr-of-members</c>. Must be <c>1</c>: after failover only one
|
|
/// node runs, and a value of <c>2</c> blocks the cluster singleton (Site Runtime
|
|
/// Deployment Manager) — and therefore all data collection — indefinitely.
|
|
/// </summary>
|
|
public int MinNrOfMembers { get; set; } = 1;
|
|
|
|
/// <summary>
|
|
/// The keep-oldest resolver's <c>down-if-alone</c> flag; only consulted when
|
|
/// <see cref="SplitBrainResolverStrategy"/> is <c>keep-oldest</c>. When <c>true</c>,
|
|
/// the oldest node downs itself if it finds it has no other reachable members,
|
|
/// rather than running as an isolated single-node cluster. Note that in a two-node
|
|
/// cluster this does NOT let the younger survivor take over from a crashed oldest —
|
|
/// Akka's alone-check requires the surviving side to have ≥2 members.
|
|
/// </summary>
|
|
public bool DownIfAlone { get; set; } = true;
|
|
|
|
/// <summary>
|
|
/// Acknowledges a deliberate single-node deployment: permits exactly one seed
|
|
/// node (normally ≥2 are required so either node can start first). Without this
|
|
/// flag a single-node install must list a phantom second seed, which the node
|
|
/// dials forever — log noise and a defeated validation intent (review 01).
|
|
/// </summary>
|
|
public bool AllowSingleNodeCluster { get; set; }
|
|
|
|
/// <summary>
|
|
/// The simultaneous-cold-start split-brain guard (<c>ScadaBridge:Cluster:BootstrapGuard</c>).
|
|
/// Default OFF — a dark switch, so existing deployments and tests keep Akka's config-driven
|
|
/// self-first auto-join byte-identical. See <see cref="ClusterBootstrapGuard"/> for the
|
|
/// decision logic and <c>ClusterBootstrapCoordinator</c> for the runtime (Gitea #33).
|
|
/// </summary>
|
|
public ClusterBootstrapGuardOptions BootstrapGuard { get; set; } = new();
|
|
}
|
|
|
|
/// <summary>
|
|
/// Configuration for the simultaneous-cold-start split-brain guard. Both nodes of a pair are
|
|
/// self-first seeds so <b>either</b> can cold-start alone when its partner is dead — but the cost
|
|
/// is that when BOTH cold-start at the same instant each runs Akka's <c>FirstSeedNodeProcess</c>,
|
|
/// times out waiting for the other, and forms its own single-node cluster (a split brain that does
|
|
/// not auto-merge). When <see cref="Enabled"/>, the node does NOT auto-join from its config seeds
|
|
/// (<c>AkkaHostedService.BuildHocon</c> emits an empty seed list); a coordinator picks the join
|
|
/// order — founder self-first / joiner peer-first — after a reachability probe. See
|
|
/// <see cref="ClusterBootstrapGuard"/>.
|
|
/// </summary>
|
|
public sealed class ClusterBootstrapGuardOptions
|
|
{
|
|
/// <summary>
|
|
/// Whether the guard is active. Default <see langword="false"/> — Akka auto-joins from the
|
|
/// config seed list exactly as before. Only meaningful on a node that is one of its own two
|
|
/// pair seeds; inert everywhere else (single-seed site node, self absent, 3+ seeds).
|
|
/// </summary>
|
|
public bool Enabled { get; set; }
|
|
|
|
/// <summary>
|
|
/// How long the higher-address node probes its partner's Akka endpoint before concluding the
|
|
/// partner is dead and forming alone. Must comfortably exceed the partner's worst-case
|
|
/// process-start-to-Akka-bind time so a slow-but-alive partner is never mistaken for a dead one
|
|
/// (which would re-open the split). Default 25s. Validated <c>> 0</c> at boot when the guard is on.
|
|
/// </summary>
|
|
public int PartnerProbeSeconds { get; set; } = 25;
|
|
|
|
/// <summary>
|
|
/// The interval between partner reachability probes, in milliseconds. Default 500ms.
|
|
/// </summary>
|
|
public int PartnerProbeIntervalMs { get; set; } = 500;
|
|
|
|
/// <summary>
|
|
/// The per-probe TCP connect timeout, in milliseconds. Default 1000ms.
|
|
/// </summary>
|
|
public int ProbeConnectTimeoutMs { get; set; } = 1000;
|
|
}
|