feat(cluster)!: self-first seed ordering replaces the InitJoin self-form watchdog
Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW
cluster when no peer answers InitJoin -- exclusively when seed-nodes[0] is the
node's own address. Every other node runs JoinSeedNodeProcess and retries
InitJoin forever. That is why "both peers are seeds" never meant "either can
cold-start alone", and it is fixed here the way Akka itself intends: each seed
node lists ITSELF first.
- docker-dev central-2 now lists itself as SeedNodes__0 (central-1 was already
self-first). The site nodes are untouched -- they seed off central-1 only and
are deliberately not seeds.
- AkkaClusterOptionsValidator enforces the invariant at boot via the shared
ZB.MOM.WW.Configuration AddValidatedOptions/ValidateOnStart seam. The rule is
CONDITIONAL -- self must be seed[0] only IF self is in the list at all -- or it
would refuse to boot every driver-only site node. Identity is compared on
PublicHostname (falling back to Hostname when blank) AND port, i.e. the address
Akka puts in SelfAddress: matching the 0.0.0.0 bind address would find no seed
anywhere and leave the rule silently inert on the whole docker-dev rig.
- ClusterBootstrapFallback + Cluster:SelfFormAfter are DELETED. The watchdog sat
outside Akka's join handshake, so it could not distinguish "no seed answered"
from "a seed answered and the join is in flight" -- and Cluster.Join(SelfAddress)
is not ignored mid-handshake, it wins. The live gate (ea45ace1) caught it
islanding a node that a manual failover had bounced: InitJoinAck received, no
Welcome inside the window because the peer's ring still held the old
incarnation, watchdog fired, second cluster. The TCP reachability guard patched
that one shape; the race stayed. It was also inert for every non-seed node, so
after this change it can never usefully fire.
- SelfFormBootstrapTests -> SelfFirstSeedBootstrapTests: real in-process clusters
through the production bootstrap at production failure-detection timings, incl.
a falsifiability control proving the OLD peer-first ordering never forms (that
test failed at 11s against the watchdog, which is what proved the retirement),
the restart-into-a-live-peer case that killed the watchdog, and a simultaneous
cold start converging on ONE cluster.
Mirrors ScadaBridge 4a6341d8; anticipates per-cluster-mesh-program Phase 6, which
had this retirement queued.
This commit is contained in:
@@ -20,7 +20,31 @@ public sealed class AkkaClusterOptions
|
||||
/// </summary>
|
||||
public string PublicHostname { get; set; } = "127.0.0.1";
|
||||
|
||||
/// <summary>Gets or sets the seed nodes for cluster bootstrapping.</summary>
|
||||
/// <summary>
|
||||
/// Gets or sets the seed nodes for cluster bootstrapping.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>ORDER IS LOAD-BEARING (decision 2026-07-22): a node that is one of its own seeds
|
||||
/// must list ITSELF first.</b> Akka runs <c>FirstSeedNodeProcess</c> — the only bootstrap
|
||||
/// path that can form a NEW cluster when no peer answers <c>InitJoin</c> — exclusively
|
||||
/// when <c>seed-nodes[0]</c> is this node's own address; any other node runs
|
||||
/// <c>JoinSeedNodeProcess</c>, which retries <c>InitJoin</c> forever and can never form a
|
||||
/// cluster. So listing both peers does NOT mean either can cold-start alone: a node that
|
||||
/// lists its partner first never comes Up while that partner is down. Self-first ordering
|
||||
/// closes that gap inside Akka's own handshake — unlike the retired
|
||||
/// <c>Cluster:SelfFormAfter</c> watchdog, which sat outside it and could not tell "no
|
||||
/// seed answered" from "a seed answered and the join is in flight".
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The rule is <b>conditional</b>: it binds only when this node's own address appears in
|
||||
/// this list. A node seeded exclusively by someone else — today's docker-dev site nodes,
|
||||
/// which list only <c>central-1</c> — is legitimately not a seed and is exempt. Enforced
|
||||
/// at boot by <see cref="AkkaClusterOptionsValidator"/>; identity is
|
||||
/// <see cref="PublicHostname"/> + <see cref="Port"/> (what Akka puts in
|
||||
/// <c>SelfAddress</c>), never the <see cref="Hostname"/> bind address.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public string[] SeedNodes { get; set; } = Array.Empty<string>();
|
||||
|
||||
/// <summary>
|
||||
@@ -55,19 +79,4 @@ public sealed class AkkaClusterOptions
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public string SplitBrainResolverStrategy { get; set; } = "auto-down";
|
||||
|
||||
/// <summary>
|
||||
/// Bootstrap self-form fallback window (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
|
||||
/// Akka only lets the FIRST listed seed form a new cluster; a non-first seed cold-starting
|
||||
/// while its peer is down loops on InitJoin forever ("auto-down removes the crash outage,
|
||||
/// not this one" — docs/Redundancy.md). When this node has waited longer than this window
|
||||
/// without cluster membership it joins itself — but ONLY if its own address appears in
|
||||
/// <see cref="SeedNodes"/>; a non-seed node (e.g. a site node whose only seed is central-1)
|
||||
/// stays waiting, because self-forming there creates a permanent island. Default 10s
|
||||
/// (same-datacenter pair; a live peer answers InitJoin in milliseconds). <c>null</c> or a
|
||||
/// non-positive value disables the fallback. Accepted trade: both pair nodes cold-starting
|
||||
/// inside the window while mutually unreachable form two clusters — the same dual-active
|
||||
/// class the auto-down strategy already accepts, same recovery (restart one side).
|
||||
/// </summary>
|
||||
public TimeSpan? SelfFormAfter { get; set; } = TimeSpan.FromSeconds(10);
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user