Files
lmxopcua/src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs
T
Joseph Doherty 3f24d4d6bf feat(cluster)!: self-first seed ordering replaces the InitJoin self-form watchdog
Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW
cluster when no peer answers InitJoin -- exclusively when seed-nodes[0] is the
node's own address. Every other node runs JoinSeedNodeProcess and retries
InitJoin forever. That is why "both peers are seeds" never meant "either can
cold-start alone", and it is fixed here the way Akka itself intends: each seed
node lists ITSELF first.

- docker-dev central-2 now lists itself as SeedNodes__0 (central-1 was already
  self-first). The site nodes are untouched -- they seed off central-1 only and
  are deliberately not seeds.
- AkkaClusterOptionsValidator enforces the invariant at boot via the shared
  ZB.MOM.WW.Configuration AddValidatedOptions/ValidateOnStart seam. The rule is
  CONDITIONAL -- self must be seed[0] only IF self is in the list at all -- or it
  would refuse to boot every driver-only site node. Identity is compared on
  PublicHostname (falling back to Hostname when blank) AND port, i.e. the address
  Akka puts in SelfAddress: matching the 0.0.0.0 bind address would find no seed
  anywhere and leave the rule silently inert on the whole docker-dev rig.
- ClusterBootstrapFallback + Cluster:SelfFormAfter are DELETED. The watchdog sat
  outside Akka's join handshake, so it could not distinguish "no seed answered"
  from "a seed answered and the join is in flight" -- and Cluster.Join(SelfAddress)
  is not ignored mid-handshake, it wins. The live gate (ea45ace1) caught it
  islanding a node that a manual failover had bounced: InitJoinAck received, no
  Welcome inside the window because the peer's ring still held the old
  incarnation, watchdog fired, second cluster. The TCP reachability guard patched
  that one shape; the race stayed. It was also inert for every non-seed node, so
  after this change it can never usefully fire.
- SelfFormBootstrapTests -> SelfFirstSeedBootstrapTests: real in-process clusters
  through the production bootstrap at production failure-detection timings, incl.
  a falsifiability control proving the OLD peer-first ordering never forms (that
  test failed at 11s against the watchdog, which is what proved the retirement),
  the restart-into-a-live-peer case that killed the watchdog, and a simultaneous
  cold start converging on ONE cluster.

Mirrors ScadaBridge 4a6341d8; anticipates per-cluster-mesh-program Phase 6, which
had this retirement queued.
2026-07-22 07:42:06 -04:00

83 lines
4.5 KiB
C#

namespace ZB.MOM.WW.OtOpcUa.Cluster;
public sealed class AkkaClusterOptions
{
public const string SectionName = "Cluster";
/// <summary>Gets or sets the Akka system name.</summary>
public string SystemName { get; set; } = "otopcua";
/// <summary>Gets or sets the hostname to bind to (default 0.0.0.0).</summary>
public string Hostname { get; set; } = "0.0.0.0";
/// <summary>Gets or sets the port to listen on (default 4053).</summary>
public int Port { get; set; } = 4053;
/// <summary>
/// Hostname advertised in cluster gossip. Must be reachable by other nodes.
/// In docker-compose this is the container DNS name; in bare metal it's the
/// host's stable LAN address.
/// </summary>
public string PublicHostname { get; set; } = "127.0.0.1";
/// <summary>
/// Gets or sets the seed nodes for cluster bootstrapping.
/// </summary>
/// <remarks>
/// <para>
/// <b>ORDER IS LOAD-BEARING (decision 2026-07-22): a node that is one of its own seeds
/// must list ITSELF first.</b> Akka runs <c>FirstSeedNodeProcess</c> — the only bootstrap
/// path that can form a NEW cluster when no peer answers <c>InitJoin</c> — exclusively
/// when <c>seed-nodes[0]</c> is this node's own address; any other node runs
/// <c>JoinSeedNodeProcess</c>, which retries <c>InitJoin</c> forever and can never form a
/// cluster. So listing both peers does NOT mean either can cold-start alone: a node that
/// lists its partner first never comes Up while that partner is down. Self-first ordering
/// closes that gap inside Akka's own handshake — unlike the retired
/// <c>Cluster:SelfFormAfter</c> watchdog, which sat outside it and could not tell "no
/// seed answered" from "a seed answered and the join is in flight".
/// </para>
/// <para>
/// The rule is <b>conditional</b>: it binds only when this node's own address appears in
/// this list. A node seeded exclusively by someone else — today's docker-dev site nodes,
/// which list only <c>central-1</c> — is legitimately not a seed and is exempt. Enforced
/// at boot by <see cref="AkkaClusterOptionsValidator"/>; identity is
/// <see cref="PublicHostname"/> + <see cref="Port"/> (what Akka puts in
/// <c>SelfAddress</c>), never the <see cref="Hostname"/> bind address.
/// </para>
/// </remarks>
public string[] SeedNodes { get; set; } = Array.Empty<string>();
/// <summary>
/// Cluster roles for this node. When empty the role list comes from
/// <c>OTOPCUA_ROLES</c> via <see cref="RoleParser"/>. Allowed values:
/// <c>admin</c>, <c>driver</c>, <c>dev</c>.
/// </summary>
public string[] Roles { get; set; } = Array.Empty<string>();
/// <summary>
/// How the cluster decides to down a node it can no longer reach. One of <c>auto-down</c>
/// (default) or <c>keep-oldest</c>; any other value fails the host at startup.
/// </summary>
/// <remarks>
/// <para>
/// <b><c>auto-down</c> — availability.</b> The leader among the <i>reachable</i> members
/// downs the unreachable peer after
/// <see cref="ServiceCollectionExtensions.DowningStableAfter"/>. A hard crash of either
/// node — including the oldest — fails over to the survivor with no operator action.
/// The trade is that a genuine network partition (both nodes alive, link cut) leaves
/// both sides running active until an operator restarts one.
/// </para>
/// <para>
/// <b><c>keep-oldest</c> — partition-safety.</b> The SBR resolver sacrifices the younger
/// side of a split, so a partition can never run dual-active. <b>The cost is severe for a
/// two-node pair:</b> it cannot survive a crash of the oldest node at all. Akka.NET's
/// <c>KeepOldest.OldestDecision</c> only lets <c>down-if-alone</c> rescue a side holding
/// &gt;= 2 members, so the 1-vs-1 survivor downs <i>itself</i> and shuts down — the
/// redundancy pair turns a single-node crash into a total outage. Choose this only for
/// clusters of three or more nodes, or where dual-active is genuinely worse than an
/// outage.
/// </para>
/// </remarks>
public string SplitBrainResolverStrategy { get; set; } = "auto-down";
}