Files
lmxopcua/tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/AkkaClusterOptionsValidatorTests.cs
T
Joseph Doherty 3f24d4d6bf feat(cluster)!: self-first seed ordering replaces the InitJoin self-form watchdog
Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW
cluster when no peer answers InitJoin -- exclusively when seed-nodes[0] is the
node's own address. Every other node runs JoinSeedNodeProcess and retries
InitJoin forever. That is why "both peers are seeds" never meant "either can
cold-start alone", and it is fixed here the way Akka itself intends: each seed
node lists ITSELF first.

- docker-dev central-2 now lists itself as SeedNodes__0 (central-1 was already
  self-first). The site nodes are untouched -- they seed off central-1 only and
  are deliberately not seeds.
- AkkaClusterOptionsValidator enforces the invariant at boot via the shared
  ZB.MOM.WW.Configuration AddValidatedOptions/ValidateOnStart seam. The rule is
  CONDITIONAL -- self must be seed[0] only IF self is in the list at all -- or it
  would refuse to boot every driver-only site node. Identity is compared on
  PublicHostname (falling back to Hostname when blank) AND port, i.e. the address
  Akka puts in SelfAddress: matching the 0.0.0.0 bind address would find no seed
  anywhere and leave the rule silently inert on the whole docker-dev rig.
- ClusterBootstrapFallback + Cluster:SelfFormAfter are DELETED. The watchdog sat
  outside Akka's join handshake, so it could not distinguish "no seed answered"
  from "a seed answered and the join is in flight" -- and Cluster.Join(SelfAddress)
  is not ignored mid-handshake, it wins. The live gate (ea45ace1) caught it
  islanding a node that a manual failover had bounced: InitJoinAck received, no
  Welcome inside the window because the peer's ring still held the old
  incarnation, watchdog fired, second cluster. The TCP reachability guard patched
  that one shape; the race stayed. It was also inert for every non-seed node, so
  after this change it can never usefully fire.
- SelfFormBootstrapTests -> SelfFirstSeedBootstrapTests: real in-process clusters
  through the production bootstrap at production failure-detection timings, incl.
  a falsifiability control proving the OLD peer-first ordering never forms (that
  test failed at 11s against the watchdog, which is what proved the retirement),
  the restart-into-a-live-peer case that killed the watchdog, and a simultaneous
  cold start converging on ONE cluster.

Mirrors ScadaBridge 4a6341d8; anticipates per-cluster-mesh-program Phase 6, which
had this retirement queued.
2026-07-22 07:42:06 -04:00

178 lines
7.3 KiB
C#

using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
/// <summary>
/// Guards the self-first seed-ordering rule at the only moment it can still be fixed cheaply: boot.
/// </summary>
/// <remarks>
/// <para>
/// The invariant (decision 2026-07-22) is <b>conditional</b>: <i>if</i> a node's own address
/// appears in its own <see cref="AkkaClusterOptions.SeedNodes"/>, it must be entry 0. Akka
/// runs <c>FirstSeedNodeProcess</c> — the only path that can form a NEW cluster when no peer
/// answers <c>InitJoin</c> — exclusively for <c>seed-nodes[0]</c>; every other node retries
/// <c>InitJoin</c> forever. A node listed second therefore cannot cold-start while its peer
/// is down, and it fails <i>silently</i>: the process is healthy, the port is open, the node
/// simply never becomes a cluster member. That is why the rule is enforced loudly here.
/// </para>
/// <para>
/// The conditional form is required, not incidental: a flat "self must be first" rule would
/// reject every driver-only site node, which is seeded solely by <c>central-1</c> and is
/// legitimately not a seed of anything —
/// see <see cref="Node_absent_from_its_own_seed_list_is_exempt"/>.
/// </para>
/// </remarks>
public sealed class AkkaClusterOptionsValidatorTests
{
private static AkkaClusterOptions CentralNode(string publicHostname, params string[] seeds) =>
new()
{
// The docker-dev shape: bind to every interface, advertise the container DNS name.
Hostname = "0.0.0.0",
PublicHostname = publicHostname,
Port = 4053,
Roles = new[] { "admin", "driver" },
SeedNodes = seeds,
};
private static string Seed(string host, int port = 4053) => $"akka.tcp://otopcua@{host}:{port}";
/// <summary>The shipped central-1 / central-2 shape: own address first, partner second.</summary>
[Fact]
public void Self_first_seed_order_passes()
{
var options = CentralNode("central-2", Seed("central-2"), Seed("central-1"));
var result = new AkkaClusterOptionsValidator().Validate(null, options);
result.Failed.ShouldBeFalse(result.FailureMessage);
}
/// <summary>
/// The pre-2026-07-22 ordering — partner first — is the boot-alone outage gap, and must not
/// start. This is the shape <c>docker-dev</c>'s <c>central-2</c> carried.
/// </summary>
[Fact]
public void Peer_first_seed_order_fails()
{
var options = CentralNode("central-2", Seed("central-1"), Seed("central-2"));
var result = new AkkaClusterOptionsValidator().Validate(null, options);
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldNotBeNull().ShouldContain("SeedNodes");
result.FailureMessage.ShouldContain("must list this node itself first");
}
/// <summary>
/// THE conditional exemption. A driver-only site node lists only <c>central-1</c> — it is not a
/// seed at all, is never legitimately first, and must pass. An unconditional "self must be
/// first" rule would refuse to boot every site node in the fleet.
/// </summary>
[Fact]
public void Node_absent_from_its_own_seed_list_is_exempt()
{
var options = CentralNode("site-a-1", Seed("central-1"));
var result = new AkkaClusterOptionsValidator().Validate(null, options);
result.Failed.ShouldBeFalse(result.FailureMessage);
}
/// <summary>
/// The bind address is NOT the node's identity. In docker-dev <c>Cluster:Hostname</c> is
/// <c>0.0.0.0</c> and <c>Cluster:PublicHostname</c> is the container name — the value Akka puts
/// in <c>SelfAddress</c> and therefore the only one a seed entry can match. A validator that
/// compared against <c>Hostname</c> would find no match anywhere, silently exempt every node,
/// and pass this misordered config.
/// </summary>
[Fact]
public void Identity_is_the_public_hostname_not_the_bind_address()
{
var options = CentralNode("central-2", Seed("central-1"), Seed("central-2"));
options.Hostname.ShouldBe("0.0.0.0", "the trap only exists when the bind address is a wildcard");
var result = new AkkaClusterOptionsValidator().Validate(null, options);
result.Failed.ShouldBeTrue("matching on the 0.0.0.0 bind address would make this rule vacuous");
}
/// <summary>
/// With <c>PublicHostname</c> blank Akka advertises the bind hostname, so that is the identity to
/// match — otherwise a loopback/bare-metal pair configured that way would be silently exempt.
/// </summary>
[Fact]
public void Identity_falls_back_to_the_bind_hostname_when_no_public_hostname_is_set()
{
var options = CentralNode("central-2", Seed("node-a"), Seed("node-b"));
options.Hostname = "node-b";
options.PublicHostname = string.Empty;
var result = new AkkaClusterOptionsValidator().Validate(null, options);
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldNotBeNull().ShouldContain("must list this node itself first");
}
/// <summary>
/// Host AND port: a two-node install sharing one hostname and differing only by port (a
/// loopback/dev pair) is a real topology, and matching on host alone would clear the misordered
/// node.
/// </summary>
[Fact]
public void Self_match_compares_host_and_port()
{
var options = CentralNode("127.0.0.1", Seed("127.0.0.1", 4053), Seed("127.0.0.1", 4054));
options.Port = 4054;
var result = new AkkaClusterOptionsValidator().Validate(null, options);
result.Failed.ShouldBeTrue("this node is 127.0.0.1:4054, which is the SECOND seed");
}
/// <summary>
/// Positive control for the pair above: the same host on the same port really is a match, so the
/// port comparison cannot be passing merely by never matching anything.
/// </summary>
[Fact]
public void Self_match_on_the_same_host_and_port_passes()
{
var options = CentralNode("127.0.0.1", Seed("127.0.0.1", 4054), Seed("127.0.0.1", 4053));
options.Port = 4054;
var result = new AkkaClusterOptionsValidator().Validate(null, options);
result.Failed.ShouldBeFalse(result.FailureMessage);
}
/// <summary>
/// An empty seed list is the single-node/dev default and says nothing about ordering — the rule
/// has nothing to bind to and must not invent a failure.
/// </summary>
[Fact]
public void Empty_seed_list_passes()
{
var options = CentralNode("central-1");
var result = new AkkaClusterOptionsValidator().Validate(null, options);
result.Failed.ShouldBeFalse(result.FailureMessage);
}
/// <summary>
/// A malformed seed URI is Akka's error to report, with Akka's wording. This rule must not
/// crash on it, and must not turn a parse problem into a misleading "ordering" complaint —
/// here the ordering is correct and only a later entry is unparseable.
/// </summary>
[Fact]
public void Malformed_seed_entry_does_not_throw()
{
var options = CentralNode("central-1", Seed("central-1"), "not-an-akka-address");
var result = new AkkaClusterOptionsValidator().Validate(null, options);
result.Failed.ShouldBeFalse(result.FailureMessage);
}
}