feat(cluster)!: self-first seed ordering replaces the InitJoin self-form watchdog

Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW
cluster when no peer answers InitJoin -- exclusively when seed-nodes[0] is the
node's own address. Every other node runs JoinSeedNodeProcess and retries
InitJoin forever. That is why "both peers are seeds" never meant "either can
cold-start alone", and it is fixed here the way Akka itself intends: each seed
node lists ITSELF first.

- docker-dev central-2 now lists itself as SeedNodes__0 (central-1 was already
  self-first). The site nodes are untouched -- they seed off central-1 only and
  are deliberately not seeds.
- AkkaClusterOptionsValidator enforces the invariant at boot via the shared
  ZB.MOM.WW.Configuration AddValidatedOptions/ValidateOnStart seam. The rule is
  CONDITIONAL -- self must be seed[0] only IF self is in the list at all -- or it
  would refuse to boot every driver-only site node. Identity is compared on
  PublicHostname (falling back to Hostname when blank) AND port, i.e. the address
  Akka puts in SelfAddress: matching the 0.0.0.0 bind address would find no seed
  anywhere and leave the rule silently inert on the whole docker-dev rig.
- ClusterBootstrapFallback + Cluster:SelfFormAfter are DELETED. The watchdog sat
  outside Akka's join handshake, so it could not distinguish "no seed answered"
  from "a seed answered and the join is in flight" -- and Cluster.Join(SelfAddress)
  is not ignored mid-handshake, it wins. The live gate (ea45ace1) caught it
  islanding a node that a manual failover had bounced: InitJoinAck received, no
  Welcome inside the window because the peer's ring still held the old
  incarnation, watchdog fired, second cluster. The TCP reachability guard patched
  that one shape; the race stayed. It was also inert for every non-seed node, so
  after this change it can never usefully fire.
- SelfFormBootstrapTests -> SelfFirstSeedBootstrapTests: real in-process clusters
  through the production bootstrap at production failure-detection timings, incl.
  a falsifiability control proving the OLD peer-first ordering never forms (that
  test failed at 11s against the watchdog, which is what proved the retirement),
  the restart-into-a-live-peer case that killed the watchdog, and a simultaneous
  cold start converging on ONE cluster.

Mirrors ScadaBridge 4a6341d8; anticipates per-cluster-mesh-program Phase 6, which
had this retirement queued.
This commit is contained in:
Joseph Doherty
2026-07-22 07:42:06 -04:00
parent a78425ea8f
commit 3f24d4d6bf
9 changed files with 625 additions and 506 deletions
@@ -0,0 +1,96 @@
using Akka.Actor;
using ZB.MOM.WW.Configuration;
namespace ZB.MOM.WW.OtOpcUa.Cluster;
/// <summary>
/// Fail-fast startup validator for <see cref="AkkaClusterOptions"/>, built on the shared
/// <c>ZB.MOM.WW.Configuration</c> <see cref="OptionsValidatorBase{TOptions}"/> and wired through
/// <c>AddValidatedOptions</c> in <see cref="ServiceCollectionExtensions.AddOtOpcUaCluster"/>.
/// </summary>
/// <remarks>
/// <para>
/// <b>Self-first seed ordering (decision 2026-07-22).</b> Akka runs
/// <c>FirstSeedNodeProcess</c> — the only bootstrap path that can form a NEW cluster when no
/// peer answers <c>InitJoin</c> — exclusively when <c>seed-nodes[0]</c> is this node's own
/// address. Every other node runs <c>JoinSeedNodeProcess</c> and retries <c>InitJoin</c>
/// forever. A node that lists its partner first therefore cannot cold-start while that
/// partner is down, and the failure is <i>silent</i>: the process runs, the port listens, the
/// node simply never becomes a member. Enforced here so it is loud, at boot, on the node
/// that has the problem.
/// </para>
/// <para>
/// <b>The rule is conditional and must stay that way.</b> It binds only when this node's own
/// address appears in its own seed list. A driver-only site node is seeded solely by
/// <c>central-1</c> and is legitimately not a seed of anything; an unconditional "self must
/// be first" rule would refuse to boot every one of them.
/// </para>
/// <para>
/// <b>Identity is the advertised address, never the bind address.</b> In docker-dev
/// <c>Cluster:Hostname</c> is <c>0.0.0.0</c> and <c>Cluster:PublicHostname</c> is the
/// container DNS name — the value Akka puts in <c>SelfAddress</c> and the only one a seed URI
/// can match. Comparing against <c>Hostname</c> would match nothing anywhere, silently exempt
/// every node, and leave this rule inert. <c>PublicHostname</c> falling back to
/// <c>Hostname</c> when blank mirrors Akka's own <c>public-hostname</c> semantics.
/// </para>
/// </remarks>
public sealed class AkkaClusterOptionsValidator : OptionsValidatorBase<AkkaClusterOptions>
{
/// <inheritdoc />
protected override void Validate(ValidationBuilder builder, AkkaClusterOptions options)
{
ArgumentNullException.ThrowIfNull(options);
var seeds = options.SeedNodes ?? Array.Empty<string>();
if (seeds.Length == 0)
{
// Single-node / dev default: nothing is claimed about ordering, so there is nothing to
// enforce. (Akka bootstraps from an empty seed list by waiting for an explicit join.)
return;
}
var selfHost = string.IsNullOrWhiteSpace(options.PublicHostname)
? options.Hostname
: options.PublicHostname;
var selfIndex = Array.FindIndex(seeds, s => IsSelf(s, selfHost, options.Port));
if (selfIndex <= 0)
{
// -1 = this node is not one of its own seeds (exempt); 0 = correctly ordered.
return;
}
builder.Add(
$"Cluster:SeedNodes must list this node itself first: entry {selfIndex} is this node "
+ $"('{seeds[selfIndex]}') but entry 0 is '{seeds[0]}'. Akka only lets seed-nodes[0] form "
+ "a new cluster (FirstSeedNodeProcess); every other node retries InitJoin forever, so "
+ "with the partner listed first this node can never start while its peer is down. Swap "
+ "the entries — see docs/Redundancy.md → 'Bootstrap: self-first seed ordering'.");
}
/// <summary>
/// True when <paramref name="seed"/> addresses this node itself — host AND port.
/// </summary>
/// <remarks>
/// Host comparison is case-insensitive because DNS names are, but is otherwise exact: Akka does
/// no DNS canonicalisation either, so <c>central-2</c> and <c>central-2.example.com</c> are
/// genuinely different seed identities to the cluster. A seed entry Akka itself cannot parse is
/// treated as "not this node" — reporting it is Akka's job, with Akka's wording, and this rule
/// must not turn a parse problem into a misleading ordering complaint.
/// </remarks>
private static bool IsSelf(string seed, string selfHost, int selfPort)
{
Address address;
try
{
address = Address.Parse(seed);
}
catch (Exception)
{
return false;
}
return address.Port == selfPort
&& string.Equals(address.Host, selfHost, StringComparison.OrdinalIgnoreCase);
}
}