Files
lmxopcua/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs
T
Joseph Doherty 279d1d0fb1
v2-ci / build (push) Successful in 4m46s
v2-ci / unit-tests (push) Failing after 16m9s
feat(cluster): simultaneous-cold-start split-brain guard (opt-in)
Closes the residual Phase-7 gap: when BOTH nodes of a 2-node pair cold-start at the
same instant, each self-first seed runs FirstSeedNodeProcess, times out waiting for
the other, and forms its own 1-node cluster — two Primaries in one pair (the Phase 6
live gate reproduced this reliably on docker). The prior mitigation was operational
(staggered start / compose depends_on), which does not exist on production hardware.

The guard (dark switch Cluster:BootstrapGuard:Enabled, default OFF) prevents the split
without giving up cold-start-alone:
- The lower-address node is the preferred founder: self-first, forms immediately.
- The higher node probes its partner's Akka port (TCP connect) up to PartnerProbeSeconds:
  reachable => peer-first (join the founder, never race it); unreachable => self-first
  (partner is dead, form alone). The order is decided BEFORE the single JoinSeedNodes,
  from an explicit reachability signal — never re-formed mid-handshake (the retired
  SelfFormAfter failure mode). When on, Akka gets no config seeds (BuildClusterOptions)
  and ClusterBootstrapCoordinator drives the join.

Review-driven hardening: case-insensitive tie-break (a hostname-casing mismatch would
reopen the split); fail-fast validation of the timing knobs; the residual "founder dies
in the probe->join window" hang is documented and made operator-visible (warning + a
restart recovers). Real-ActorSystem coordinator tests cover the load-bearing higher-node
cold-start-alone case; 147/147 Cluster tests pass.

Live-gated on docker-dev (site-a = enablement demo, serialization removed, guard on;
site-b keeps depends_on-serialization + guard off as the A/B control): simultaneous
start -> site-a-1 founds, site-a-2 probes-reachable-joins -> 250/240, NO split;
higher-node-alone -> probes dead founder, self-forms -> 250; founder rejoin -> 240.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-24 08:34:29 -04:00

125 lines
6.4 KiB
C#

namespace ZB.MOM.WW.OtOpcUa.Cluster;
public sealed class AkkaClusterOptions
{
public const string SectionName = "Cluster";
/// <summary>Gets or sets the Akka system name.</summary>
public string SystemName { get; set; } = "otopcua";
/// <summary>Gets or sets the hostname to bind to (default 0.0.0.0).</summary>
public string Hostname { get; set; } = "0.0.0.0";
/// <summary>Gets or sets the port to listen on (default 4053).</summary>
public int Port { get; set; } = 4053;
/// <summary>
/// Hostname advertised in cluster gossip. Must be reachable by other nodes.
/// In docker-compose this is the container DNS name; in bare metal it's the
/// host's stable LAN address.
/// </summary>
public string PublicHostname { get; set; } = "127.0.0.1";
/// <summary>
/// Gets or sets the seed nodes for cluster bootstrapping.
/// </summary>
/// <remarks>
/// <para>
/// <b>ORDER IS LOAD-BEARING (decision 2026-07-22): a node that is one of its own seeds
/// must list ITSELF first.</b> Akka runs <c>FirstSeedNodeProcess</c> — the only bootstrap
/// path that can form a NEW cluster when no peer answers <c>InitJoin</c> — exclusively
/// when <c>seed-nodes[0]</c> is this node's own address; any other node runs
/// <c>JoinSeedNodeProcess</c>, which retries <c>InitJoin</c> forever and can never form a
/// cluster. So listing both peers does NOT mean either can cold-start alone: a node that
/// lists its partner first never comes Up while that partner is down. Self-first ordering
/// closes that gap inside Akka's own handshake — unlike the retired
/// <c>Cluster:SelfFormAfter</c> watchdog, which sat outside it and could not tell "no
/// seed answered" from "a seed answered and the join is in flight".
/// </para>
/// <para>
/// The rule is <b>conditional</b>: it binds only when this node's own address appears in
/// this list. A node seeded exclusively by someone else — today's docker-dev site nodes,
/// which list only <c>central-1</c> — is legitimately not a seed and is exempt. Enforced
/// at boot by <see cref="AkkaClusterOptionsValidator"/>; identity is
/// <see cref="PublicHostname"/> + <see cref="Port"/> (what Akka puts in
/// <c>SelfAddress</c>), never the <see cref="Hostname"/> bind address.
/// </para>
/// </remarks>
public string[] SeedNodes { get; set; } = Array.Empty<string>();
/// <summary>
/// Cluster roles for this node. When empty the role list comes from
/// <c>OTOPCUA_ROLES</c> via <see cref="RoleParser"/>. Allowed values:
/// <c>admin</c>, <c>driver</c>, <c>dev</c>.
/// </summary>
public string[] Roles { get; set; } = Array.Empty<string>();
/// <summary>
/// How the cluster decides to down a node it can no longer reach. One of <c>auto-down</c>
/// (default) or <c>keep-oldest</c>; any other value fails the host at startup.
/// </summary>
/// <remarks>
/// <para>
/// <b><c>auto-down</c> — availability.</b> The leader among the <i>reachable</i> members
/// downs the unreachable peer after
/// <see cref="ServiceCollectionExtensions.DowningStableAfter"/>. A hard crash of either
/// node — including the oldest — fails over to the survivor with no operator action.
/// The trade is that a genuine network partition (both nodes alive, link cut) leaves
/// both sides running active until an operator restarts one.
/// </para>
/// <para>
/// <b><c>keep-oldest</c> — partition-safety.</b> The SBR resolver sacrifices the younger
/// side of a split, so a partition can never run dual-active. <b>The cost is severe for a
/// two-node pair:</b> it cannot survive a crash of the oldest node at all. Akka.NET's
/// <c>KeepOldest.OldestDecision</c> only lets <c>down-if-alone</c> rescue a side holding
/// &gt;= 2 members, so the 1-vs-1 survivor downs <i>itself</i> and shuts down — the
/// redundancy pair turns a single-node crash into a total outage. Choose this only for
/// clusters of three or more nodes, or where dual-active is genuinely worse than an
/// outage.
/// </para>
/// </remarks>
public string SplitBrainResolverStrategy { get; set; } = "auto-down";
/// <summary>
/// The simultaneous-cold-start split-brain guard (<c>Cluster:BootstrapGuard</c>). Default OFF — a
/// dark switch, so existing deployments and tests keep Akka's config-driven self-first auto-join
/// unchanged. See <see cref="ClusterBootstrapGuard"/> for the decision logic and
/// <c>ClusterBootstrapCoordinator</c> for the runtime.
/// </summary>
public ClusterBootstrapGuardOptions BootstrapGuard { get; set; } = new();
}
/// <summary>
/// Configuration for the simultaneous-cold-start split-brain guard. When
/// <see cref="Enabled"/>, the node does NOT auto-join from its config seeds; a coordinator picks the
/// join order (founder self-first / joiner peer-first) after a reachability probe. See
/// <see cref="ClusterBootstrapGuard"/>.
/// </summary>
public sealed class ClusterBootstrapGuardOptions
{
/// <summary>
/// Gets or sets whether the guard is active. Default <see langword="false"/> — Akka auto-joins from
/// the config seed list exactly as before. Only meaningful on a node that is one of its own two
/// pair seeds; inert everywhere else.
/// </summary>
public bool Enabled { get; set; }
/// <summary>
/// Gets or sets how long the higher-address node probes its partner's Akka endpoint before
/// concluding the partner is dead and forming alone. Must comfortably exceed the partner's
/// worst-case process-start-to-Akka-bind time so a slow-but-alive partner is never mistaken for a
/// dead one (which would re-open the split). Default 25 s.
/// </summary>
public int PartnerProbeSeconds { get; set; } = 25;
/// <summary>
/// Gets or sets the interval between partner reachability probes, in milliseconds. Default 500 ms.
/// </summary>
public int PartnerProbeIntervalMs { get; set; } = 500;
/// <summary>
/// Gets or sets the per-probe TCP connect timeout, in milliseconds. Default 1000 ms.
/// </summary>
public int ProbeConnectTimeoutMs { get; set; } = 1000;
}