feat(cluster)!: self-first seed ordering replaces the InitJoin self-form watchdog
Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW
cluster when no peer answers InitJoin -- exclusively when seed-nodes[0] is the
node's own address. Every other node runs JoinSeedNodeProcess and retries
InitJoin forever. That is why "both peers are seeds" never meant "either can
cold-start alone", and it is fixed here the way Akka itself intends: each seed
node lists ITSELF first.
- docker-dev central-2 now lists itself as SeedNodes__0 (central-1 was already
self-first). The site nodes are untouched -- they seed off central-1 only and
are deliberately not seeds.
- AkkaClusterOptionsValidator enforces the invariant at boot via the shared
ZB.MOM.WW.Configuration AddValidatedOptions/ValidateOnStart seam. The rule is
CONDITIONAL -- self must be seed[0] only IF self is in the list at all -- or it
would refuse to boot every driver-only site node. Identity is compared on
PublicHostname (falling back to Hostname when blank) AND port, i.e. the address
Akka puts in SelfAddress: matching the 0.0.0.0 bind address would find no seed
anywhere and leave the rule silently inert on the whole docker-dev rig.
- ClusterBootstrapFallback + Cluster:SelfFormAfter are DELETED. The watchdog sat
outside Akka's join handshake, so it could not distinguish "no seed answered"
from "a seed answered and the join is in flight" -- and Cluster.Join(SelfAddress)
is not ignored mid-handshake, it wins. The live gate (ea45ace1) caught it
islanding a node that a manual failover had bounced: InitJoinAck received, no
Welcome inside the window because the peer's ring still held the old
incarnation, watchdog fired, second cluster. The TCP reachability guard patched
that one shape; the race stayed. It was also inert for every non-seed node, so
after this change it can never usefully fire.
- SelfFormBootstrapTests -> SelfFirstSeedBootstrapTests: real in-process clusters
through the production bootstrap at production failure-detection timings, incl.
a falsifiability control proving the OLD peer-first ordering never forms (that
test failed at 11s against the watchdog, which is what proved the retirement),
the restart-into-a-live-peer case that killed the watchdog, and a simultaneous
cold start converging on ONE cluster.
Mirrors ScadaBridge 4a6341d8; anticipates per-cluster-mesh-program Phase 6, which
had this retirement queued.
This commit is contained in:
Joseph Doherty
@@ -0,0 +1,177 @@
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Guards the self-first seed-ordering rule at the only moment it can still be fixed cheaply: boot.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// The invariant (decision 2026-07-22) is <b>conditional</b>: <i>if</i> a node's own address
|
||||
/// appears in its own <see cref="AkkaClusterOptions.SeedNodes"/>, it must be entry 0. Akka
|
||||
/// runs <c>FirstSeedNodeProcess</c> — the only path that can form a NEW cluster when no peer
|
||||
/// answers <c>InitJoin</c> — exclusively for <c>seed-nodes[0]</c>; every other node retries
|
||||
/// <c>InitJoin</c> forever. A node listed second therefore cannot cold-start while its peer
|
||||
/// is down, and it fails <i>silently</i>: the process is healthy, the port is open, the node
|
||||
/// simply never becomes a cluster member. That is why the rule is enforced loudly here.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The conditional form is required, not incidental: a flat "self must be first" rule would
|
||||
/// reject every driver-only site node, which is seeded solely by <c>central-1</c> and is
|
||||
/// legitimately not a seed of anything —
|
||||
/// see <see cref="Node_absent_from_its_own_seed_list_is_exempt"/>.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class AkkaClusterOptionsValidatorTests
|
||||
{
|
||||
private static AkkaClusterOptions CentralNode(string publicHostname, params string[] seeds) =>
|
||||
new()
|
||||
{
|
||||
// The docker-dev shape: bind to every interface, advertise the container DNS name.
|
||||
Hostname = "0.0.0.0",
|
||||
PublicHostname = publicHostname,
|
||||
Port = 4053,
|
||||
Roles = new[] { "admin", "driver" },
|
||||
SeedNodes = seeds,
|
||||
};
|
||||
|
||||
private static string Seed(string host, int port = 4053) => $"akka.tcp://otopcua@{host}:{port}";
|
||||
|
||||
/// <summary>The shipped central-1 / central-2 shape: own address first, partner second.</summary>
|
||||
[Fact]
|
||||
public void Self_first_seed_order_passes()
|
||||
{
|
||||
var options = CentralNode("central-2", Seed("central-2"), Seed("central-1"));
|
||||
|
||||
var result = new AkkaClusterOptionsValidator().Validate(null, options);
|
||||
|
||||
result.Failed.ShouldBeFalse(result.FailureMessage);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The pre-2026-07-22 ordering — partner first — is the boot-alone outage gap, and must not
|
||||
/// start. This is the shape <c>docker-dev</c>'s <c>central-2</c> carried.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Peer_first_seed_order_fails()
|
||||
{
|
||||
var options = CentralNode("central-2", Seed("central-1"), Seed("central-2"));
|
||||
|
||||
var result = new AkkaClusterOptionsValidator().Validate(null, options);
|
||||
|
||||
result.Failed.ShouldBeTrue();
|
||||
result.FailureMessage.ShouldNotBeNull().ShouldContain("SeedNodes");
|
||||
result.FailureMessage.ShouldContain("must list this node itself first");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// THE conditional exemption. A driver-only site node lists only <c>central-1</c> — it is not a
|
||||
/// seed at all, is never legitimately first, and must pass. An unconditional "self must be
|
||||
/// first" rule would refuse to boot every site node in the fleet.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Node_absent_from_its_own_seed_list_is_exempt()
|
||||
{
|
||||
var options = CentralNode("site-a-1", Seed("central-1"));
|
||||
|
||||
var result = new AkkaClusterOptionsValidator().Validate(null, options);
|
||||
|
||||
result.Failed.ShouldBeFalse(result.FailureMessage);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The bind address is NOT the node's identity. In docker-dev <c>Cluster:Hostname</c> is
|
||||
/// <c>0.0.0.0</c> and <c>Cluster:PublicHostname</c> is the container name — the value Akka puts
|
||||
/// in <c>SelfAddress</c> and therefore the only one a seed entry can match. A validator that
|
||||
/// compared against <c>Hostname</c> would find no match anywhere, silently exempt every node,
|
||||
/// and pass this misordered config.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Identity_is_the_public_hostname_not_the_bind_address()
|
||||
{
|
||||
var options = CentralNode("central-2", Seed("central-1"), Seed("central-2"));
|
||||
options.Hostname.ShouldBe("0.0.0.0", "the trap only exists when the bind address is a wildcard");
|
||||
|
||||
var result = new AkkaClusterOptionsValidator().Validate(null, options);
|
||||
|
||||
result.Failed.ShouldBeTrue("matching on the 0.0.0.0 bind address would make this rule vacuous");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// With <c>PublicHostname</c> blank Akka advertises the bind hostname, so that is the identity to
|
||||
/// match — otherwise a loopback/bare-metal pair configured that way would be silently exempt.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Identity_falls_back_to_the_bind_hostname_when_no_public_hostname_is_set()
|
||||
{
|
||||
var options = CentralNode("central-2", Seed("node-a"), Seed("node-b"));
|
||||
options.Hostname = "node-b";
|
||||
options.PublicHostname = string.Empty;
|
||||
|
||||
var result = new AkkaClusterOptionsValidator().Validate(null, options);
|
||||
|
||||
result.Failed.ShouldBeTrue();
|
||||
result.FailureMessage.ShouldNotBeNull().ShouldContain("must list this node itself first");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Host AND port: a two-node install sharing one hostname and differing only by port (a
|
||||
/// loopback/dev pair) is a real topology, and matching on host alone would clear the misordered
|
||||
/// node.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Self_match_compares_host_and_port()
|
||||
{
|
||||
var options = CentralNode("127.0.0.1", Seed("127.0.0.1", 4053), Seed("127.0.0.1", 4054));
|
||||
options.Port = 4054;
|
||||
|
||||
var result = new AkkaClusterOptionsValidator().Validate(null, options);
|
||||
|
||||
result.Failed.ShouldBeTrue("this node is 127.0.0.1:4054, which is the SECOND seed");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Positive control for the pair above: the same host on the same port really is a match, so the
|
||||
/// port comparison cannot be passing merely by never matching anything.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Self_match_on_the_same_host_and_port_passes()
|
||||
{
|
||||
var options = CentralNode("127.0.0.1", Seed("127.0.0.1", 4054), Seed("127.0.0.1", 4053));
|
||||
options.Port = 4054;
|
||||
|
||||
var result = new AkkaClusterOptionsValidator().Validate(null, options);
|
||||
|
||||
result.Failed.ShouldBeFalse(result.FailureMessage);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// An empty seed list is the single-node/dev default and says nothing about ordering — the rule
|
||||
/// has nothing to bind to and must not invent a failure.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Empty_seed_list_passes()
|
||||
{
|
||||
var options = CentralNode("central-1");
|
||||
|
||||
var result = new AkkaClusterOptionsValidator().Validate(null, options);
|
||||
|
||||
result.Failed.ShouldBeFalse(result.FailureMessage);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A malformed seed URI is Akka's error to report, with Akka's wording. This rule must not
|
||||
/// crash on it, and must not turn a parse problem into a misleading "ordering" complaint —
|
||||
/// here the ordering is correct and only a later entry is unparseable.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void Malformed_seed_entry_does_not_throw()
|
||||
{
|
||||
var options = CentralNode("central-1", Seed("central-1"), "not-an-akka-address");
|
||||
|
||||
var result = new AkkaClusterOptionsValidator().Validate(null, options);
|
||||
|
||||
result.Failed.ShouldBeFalse(result.FailureMessage);
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,287 @@
|
||||
using System.Net;
|
||||
using System.Net.Sockets;
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.Hosting;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Options;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Guards the self-first seed-ordering invariant (decision 2026-07-22): every node that is a seed
|
||||
/// of its own mesh lists ITSELF as <c>SeedNodes[0]</c> and its partner second.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Why the ordering is the mechanism.</b> Akka runs a different bootstrap process
|
||||
/// depending on whether <c>seed-nodes[0]</c> is this node's own address. When it is,
|
||||
/// <c>FirstSeedNodeProcess</c> runs: it <c>InitJoin</c>s the other seeds and self-joins once
|
||||
/// <c>seed-node-timeout</c> (5 s) passes with nobody answering. When it is not,
|
||||
/// <c>JoinSeedNodeProcess</c> runs, and that process can never form a new cluster — it
|
||||
/// retries <c>InitJoin</c> forever. That is why "both peers are listed in SeedNodes" never
|
||||
/// meant "either can cold-start alone", and why a node listed second stayed out of the
|
||||
/// cluster indefinitely while its peer was down.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Why not the self-form watchdog this replaces.</b> <c>ClusterBootstrapFallback</c>
|
||||
/// waited <c>Cluster:SelfFormAfter</c> for membership and then called
|
||||
/// <c>Cluster.Join(SelfAddress)</c>. A timer outside Akka's join handshake cannot
|
||||
/// distinguish "no seed answered" from "a seed answered and the join is in flight", and
|
||||
/// <c>Join(SelfAddress)</c> is <b>not</b> ignored mid-handshake — it wins. The live gate on
|
||||
/// the docker-dev rig (2026-07-22) caught exactly that: a node bounced by a manual failover
|
||||
/// restarted, got <c>InitJoinAck</c> from its live peer, but no Welcome inside the window
|
||||
/// because the peer's ring still held its previous incarnation
|
||||
/// (<c>Exiting</c> → <c>Down</c> → <c>Removed</c>); the watchdog fired and formed a SECOND
|
||||
/// cluster. A TCP reachability guard patched that one shape; the race stayed.
|
||||
/// <see cref="Restarting_node_rejoins_its_live_peer_instead_of_islanding"/> is that scenario,
|
||||
/// and self-first ordering has no such race because the decision is made inside the
|
||||
/// handshake by Akka itself.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Why these start real hosts through the production bootstrap.</b> Same reason as
|
||||
/// <see cref="SplitBrainResolverActivationTests"/>: the question is what a running node does.
|
||||
/// Every node here is built with
|
||||
/// <see cref="ServiceCollectionExtensions.WithOtOpcUaClusterBootstrap"/> — the shipped entry
|
||||
/// point — so these run at the production failure-detection envelope (heartbeat 2 s,
|
||||
/// acceptable pause 10 s, auto-down 15 s, down-removal-margin 15 s) from
|
||||
/// <c>Resources/akka.conf</c> rather than at timings invented by the test.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class SelfFirstSeedBootstrapTests
|
||||
{
|
||||
private static int FreePort()
|
||||
{
|
||||
var listener = new TcpListener(IPAddress.Loopback, 0);
|
||||
listener.Start();
|
||||
var port = ((IPEndPoint)listener.LocalEndpoint).Port;
|
||||
listener.Stop();
|
||||
return port;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Starts a node the way every shipped node config is written: its own address first, the
|
||||
/// partner second.
|
||||
/// </summary>
|
||||
/// <param name="systemName">Actor-system name; also the system part of every seed address.</param>
|
||||
/// <param name="selfPort">This node's remoting port.</param>
|
||||
/// <param name="peerPort">The partner's remoting port (may have nothing listening on it).</param>
|
||||
/// <param name="selfFirst">
|
||||
/// <see langword="false"/> reproduces the pre-2026-07-22 ordering (partner first), which is what
|
||||
/// makes <see cref="Peer_first_ordering_is_the_outage_gap_and_never_forms"/> a falsifiability
|
||||
/// control rather than a restatement of the fix.
|
||||
/// </param>
|
||||
private static async Task<IHost> StartNodeAsync(
|
||||
string systemName,
|
||||
int selfPort,
|
||||
int peerPort,
|
||||
bool selfFirst = true)
|
||||
{
|
||||
var self = $"akka.tcp://{systemName}@127.0.0.1:{selfPort}";
|
||||
var peer = $"akka.tcp://{systemName}@127.0.0.1:{peerPort}";
|
||||
var options = new AkkaClusterOptions
|
||||
{
|
||||
SystemName = systemName,
|
||||
Hostname = "127.0.0.1",
|
||||
PublicHostname = "127.0.0.1",
|
||||
Port = selfPort,
|
||||
Roles = new[] { "admin", "driver" },
|
||||
SeedNodes = selfFirst ? new[] { self, peer } : new[] { peer, self },
|
||||
};
|
||||
|
||||
var builder = Host.CreateDefaultBuilder();
|
||||
builder.ConfigureServices(services =>
|
||||
{
|
||||
services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
|
||||
services.AddAkka(systemName, (ab, sp) =>
|
||||
{
|
||||
ab.WithOtOpcUaClusterBootstrap(sp);
|
||||
|
||||
// Crash simulation, test-only: with this off, ActorSystem.Terminate() skips
|
||||
// CoordinatedShutdown and so gossips no Leave — a process death, not a graceful
|
||||
// drain. Without it "restart the standby" would degenerate into the easy path where
|
||||
// the peer had already removed the old member before the new one dialled in.
|
||||
// Prepend is Akka.Hosting's highest-precedence merge mode (see BuildDowningHocon).
|
||||
ab.AddHocon(
|
||||
"akka.coordinated-shutdown.run-by-actor-system-terminate = off",
|
||||
HoconAddMode.Prepend);
|
||||
});
|
||||
});
|
||||
|
||||
var host = builder.Build();
|
||||
await host.StartAsync();
|
||||
return host;
|
||||
}
|
||||
|
||||
/// <summary>Polls until the node sees <paramref name="expected"/> Up members, or the deadline passes.</summary>
|
||||
private static async Task<bool> WaitForUpMembersAsync(IHost host, int expected, TimeSpan timeout)
|
||||
{
|
||||
var cluster = Akka.Cluster.Cluster.Get(host.Services.GetRequiredService<ActorSystem>());
|
||||
var deadline = DateTime.UtcNow + timeout;
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
if (cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expected)
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
await Task.Delay(200);
|
||||
}
|
||||
|
||||
return cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expected;
|
||||
}
|
||||
|
||||
private static async Task StopAsync(IHost host)
|
||||
{
|
||||
try
|
||||
{
|
||||
await host.StopAsync();
|
||||
}
|
||||
catch (Exception)
|
||||
{
|
||||
// Teardown only. A host whose ActorSystem was terminated by a crash-simulation cannot
|
||||
// run CoordinatedShutdown again; that must not fail the test that already asserted.
|
||||
}
|
||||
|
||||
host.Dispose();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The unattended cold-start-alone guarantee: a node whose partner is dead forms the cluster by
|
||||
/// itself, through Akka's own <c>FirstSeedNodeProcess</c> — no watchdog, no timer, no operator.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Lone_cold_start_forms_a_cluster_when_the_peer_is_dead()
|
||||
{
|
||||
var selfPort = FreePort();
|
||||
var deadPeerPort = FreePort(); // nothing is listening there
|
||||
var host = await StartNodeAsync("otopcua-selffirst-1", selfPort, deadPeerPort);
|
||||
try
|
||||
{
|
||||
// seed-node-timeout is Akka's 5 s default; 30 s is slack for a loaded CI box.
|
||||
(await WaitForUpMembersAsync(host, 1, TimeSpan.FromSeconds(30)))
|
||||
.ShouldBeTrue("a self-first seed must form a cluster alone when no peer answers InitJoin");
|
||||
}
|
||||
finally
|
||||
{
|
||||
await StopAsync(host);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// FALSIFIABILITY CONTROL for the test above. With the OLD ordering — partner first — the
|
||||
/// identical scenario never comes Up, because <c>JoinSeedNodeProcess</c> has no self-join path.
|
||||
/// If this test ever starts passing quickly, the seed ordering has stopped being the thing doing
|
||||
/// the work and the suite above has gone vacuous.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// This is also the test that fails while the retired <c>ClusterBootstrapFallback</c> watchdog
|
||||
/// is still armed: that timer would bring this node Up at <c>SelfFormAfter</c> (10 s) despite the
|
||||
/// ordering, which is precisely the "something other than Akka's handshake is deciding" that the
|
||||
/// retirement removes.
|
||||
/// </remarks>
|
||||
[Fact]
|
||||
public async Task Peer_first_ordering_is_the_outage_gap_and_never_forms()
|
||||
{
|
||||
var selfPort = FreePort();
|
||||
var deadPeerPort = FreePort();
|
||||
var host = await StartNodeAsync("otopcua-selffirst-2", selfPort, deadPeerPort, selfFirst: false);
|
||||
try
|
||||
{
|
||||
// 3x the 5 s seed-node-timeout, and longer than the retired watchdog's 10 s window.
|
||||
(await WaitForUpMembersAsync(host, 1, TimeSpan.FromSeconds(15)))
|
||||
.ShouldBeFalse("a node listed behind its peer cannot form a cluster — it InitJoin-loops forever");
|
||||
|
||||
var cluster = Akka.Cluster.Cluster.Get(host.Services.GetRequiredService<ActorSystem>());
|
||||
cluster.State.Members.ShouldBeEmpty();
|
||||
|
||||
// Positive control: the node was formable all along; only the ordering blocked it.
|
||||
cluster.Join(cluster.SelfAddress);
|
||||
(await WaitForUpMembersAsync(host, 1, TimeSpan.FromSeconds(30)))
|
||||
.ShouldBeTrue("positive control — an explicit self-join must bring this node Up");
|
||||
}
|
||||
finally
|
||||
{
|
||||
await StopAsync(host);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The case that killed the self-form watchdog (live gate, docker-dev, 2026-07-22): a routine
|
||||
/// standby restart while the peer is alive must REJOIN, never island. Under self-first ordering
|
||||
/// the restarted node's <c>FirstSeedNodeProcess</c> only self-joins when no seed answers, so the
|
||||
/// live peer's <c>InitJoinAck</c> keeps it on the join path however long the peer takes to
|
||||
/// retire its previous incarnation.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Restarting_node_rejoins_its_live_peer_instead_of_islanding()
|
||||
{
|
||||
const string systemName = "otopcua-selffirst-3";
|
||||
var portA = FreePort();
|
||||
var portB = FreePort();
|
||||
|
||||
var nodeA = await StartNodeAsync(systemName, portA, portB);
|
||||
IHost? nodeB = null;
|
||||
IHost? nodeB2 = null;
|
||||
try
|
||||
{
|
||||
(await WaitForUpMembersAsync(nodeA, 1, TimeSpan.FromSeconds(30))).ShouldBeTrue();
|
||||
|
||||
nodeB = await StartNodeAsync(systemName, portB, portA);
|
||||
(await WaitForUpMembersAsync(nodeA, 2, TimeSpan.FromSeconds(60)))
|
||||
.ShouldBeTrue("the pair must converge before the restart is meaningful");
|
||||
|
||||
// Hard-crash B and immediately restart it at the SAME address, as a service restart or
|
||||
// container recreate does. No Leave gossip: A still holds B's previous incarnation.
|
||||
await nodeB.Services.GetRequiredService<ActorSystem>().Terminate()
|
||||
.WaitAsync(TimeSpan.FromSeconds(30), TestContext.Current.CancellationToken);
|
||||
nodeB2 = await StartNodeAsync(systemName, portB, portA);
|
||||
|
||||
// ONE cluster of two, never two clusters of one. The budget covers auto-down (15 s) plus
|
||||
// down-removal-margin (15 s) before the new incarnation is admitted.
|
||||
(await WaitForUpMembersAsync(nodeB2, 2, TimeSpan.FromSeconds(120)))
|
||||
.ShouldBeTrue("the restarted node must rejoin its live peer, not form a second cluster");
|
||||
(await WaitForUpMembersAsync(nodeA, 2, TimeSpan.FromSeconds(120)))
|
||||
.ShouldBeTrue("the surviving node must see the restarted peer as a member of ITS cluster");
|
||||
}
|
||||
finally
|
||||
{
|
||||
if (nodeB2 is not null) await StopAsync(nodeB2);
|
||||
if (nodeB is not null) await StopAsync(nodeB);
|
||||
await StopAsync(nodeA);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The obvious objection to self-first on BOTH nodes: does a simultaneous cold start produce two
|
||||
/// clusters? While the two are mutually reachable it does not — each runs
|
||||
/// <c>FirstSeedNodeProcess</c>, and the <c>InitJoin</c> handshake resolves which one forms before
|
||||
/// either self-join deadline expires. (A genuine boot-time PARTITION would still split, the same
|
||||
/// dual-active class the <c>auto-down</c> strategy already accepts, with the same recovery.)
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Both_nodes_cold_starting_together_converge_on_one_cluster()
|
||||
{
|
||||
const string systemName = "otopcua-selffirst-4";
|
||||
var portA = FreePort();
|
||||
var portB = FreePort();
|
||||
|
||||
var nodeA = await StartNodeAsync(systemName, portA, portB);
|
||||
var nodeB = await StartNodeAsync(systemName, portB, portA);
|
||||
try
|
||||
{
|
||||
(await WaitForUpMembersAsync(nodeA, 2, TimeSpan.FromSeconds(60)))
|
||||
.ShouldBeTrue("both nodes cold-starting self-first must converge, not split");
|
||||
(await WaitForUpMembersAsync(nodeB, 2, TimeSpan.FromSeconds(60)))
|
||||
.ShouldBeTrue("both nodes cold-starting self-first must converge, not split");
|
||||
}
|
||||
finally
|
||||
{
|
||||
await StopAsync(nodeB);
|
||||
await StopAsync(nodeA);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1,259 +0,0 @@
|
||||
using System.Net;
|
||||
using System.Net.Sockets;
|
||||
using Akka.Actor;
|
||||
using Akka.Cluster;
|
||||
using Akka.Hosting;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Hosting;
|
||||
using Microsoft.Extensions.Options;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Guards the InitJoin self-form fallback — whether a node whose peer is down at boot ever becomes
|
||||
/// operational.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Why these tests start real hosts through the production bootstrap.</b> Like
|
||||
/// <see cref="SplitBrainResolverActivationTests"/>, the question is what a running node
|
||||
/// actually does, not what an options object holds. Every host here is built with
|
||||
/// <see cref="ServiceCollectionExtensions.WithOtOpcUaClusterBootstrap"/> — the shipped entry
|
||||
/// point — so a change that stops arming the fallback (or arms it from a place production
|
||||
/// does not reach) turns these red. A test that called
|
||||
/// <see cref="ClusterBootstrapFallback.Arm"/> itself would pin the test's wiring instead.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Seed ordering is load-bearing.</b> The node under test lists itself SECOND in its own
|
||||
/// seed list, behind a dead peer port. Akka only lets the FIRST listed seed self-join, so
|
||||
/// nothing but the fallback can bring these nodes Up — which is what makes the negative
|
||||
/// scenarios (disabled window, non-seed node) meaningful rather than vacuous. Each of those
|
||||
/// carries a positive control (an explicit <c>Cluster.Join(SelfAddress)</c>) proving the node
|
||||
/// was formable all along and only the fallback was absent.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
public sealed class SelfFormBootstrapTests
|
||||
{
|
||||
private static int FreePort()
|
||||
{
|
||||
var listener = new TcpListener(IPAddress.Loopback, 0);
|
||||
listener.Start();
|
||||
var port = ((IPEndPoint)listener.LocalEndpoint).Port;
|
||||
listener.Stop();
|
||||
return port;
|
||||
}
|
||||
|
||||
private static async Task<IHost> StartNodeAsync(
|
||||
string systemName,
|
||||
int selfPort,
|
||||
int peerPort,
|
||||
TimeSpan? selfFormAfter,
|
||||
bool selfInSeeds = true)
|
||||
{
|
||||
var options = new AkkaClusterOptions
|
||||
{
|
||||
SystemName = systemName,
|
||||
Hostname = "127.0.0.1",
|
||||
PublicHostname = "127.0.0.1",
|
||||
Port = selfPort,
|
||||
Roles = new[] { "admin", "driver" },
|
||||
SelfFormAfter = selfFormAfter,
|
||||
|
||||
// Self listed SECOND, behind a port nothing is listening on: Akka lets only the first
|
||||
// seed self-join, so this node can never form a cluster on its own except via the
|
||||
// fallback.
|
||||
SeedNodes = selfInSeeds
|
||||
? new[]
|
||||
{
|
||||
$"akka.tcp://{systemName}@127.0.0.1:{peerPort}",
|
||||
$"akka.tcp://{systemName}@127.0.0.1:{selfPort}",
|
||||
}
|
||||
: new[] { $"akka.tcp://{systemName}@127.0.0.1:{peerPort}" },
|
||||
};
|
||||
|
||||
var builder = Host.CreateDefaultBuilder();
|
||||
builder.ConfigureServices(services =>
|
||||
{
|
||||
services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
|
||||
services.AddAkka(systemName, (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp));
|
||||
});
|
||||
|
||||
var host = builder.Build();
|
||||
await host.StartAsync();
|
||||
return host;
|
||||
}
|
||||
|
||||
/// <summary>Polls cluster state until it holds at least one Up member, or the deadline passes.</summary>
|
||||
private static async Task<bool> WaitForUpMemberAsync(ActorSystem system, TimeSpan timeout)
|
||||
{
|
||||
var cluster = Akka.Cluster.Cluster.Get(system);
|
||||
var deadline = DateTime.UtcNow + timeout;
|
||||
while (DateTime.UtcNow < deadline)
|
||||
{
|
||||
if (cluster.State.Members.Any(m => m.Status == MemberStatus.Up))
|
||||
{
|
||||
return true;
|
||||
}
|
||||
|
||||
await Task.Delay(200);
|
||||
}
|
||||
|
||||
return cluster.State.Members.Any(m => m.Status == MemberStatus.Up);
|
||||
}
|
||||
|
||||
private static async Task StopAsync(IHost host)
|
||||
{
|
||||
await host.StopAsync();
|
||||
host.Dispose();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// An unconfigured deployment gets the fallback: a pair node that cold-starts while its peer is
|
||||
/// down must come up on its own, not wait for an operator.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void SelfFormAfter_defaults_to_ten_seconds()
|
||||
{
|
||||
new AkkaClusterOptions().SelfFormAfter.ShouldBe(TimeSpan.FromSeconds(10));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The core scenario: a node listed as a non-first seed, booting with its peer dead, becomes Up
|
||||
/// on its own after the window. Without the fallback it loops on InitJoin forever.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Lone_non_first_seed_self_forms_after_the_window()
|
||||
{
|
||||
var selfPort = FreePort();
|
||||
var peerPort = FreePort();
|
||||
var host = await StartNodeAsync(
|
||||
"otopcua-selfform-1", selfPort, peerPort, TimeSpan.FromSeconds(2));
|
||||
try
|
||||
{
|
||||
var system = host.Services.GetRequiredService<ActorSystem>();
|
||||
|
||||
(await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(20)))
|
||||
.ShouldBeTrue("a lone non-first seed must self-form and come Up unattended");
|
||||
}
|
||||
finally
|
||||
{
|
||||
await StopAsync(host);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The opt-out is real: with the window unset the node keeps waiting on InitJoin. The positive
|
||||
/// control proves the node was formable and only the fallback was missing — otherwise this test
|
||||
/// would pass just as happily against a node that could never join anything.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Disabled_fallback_keeps_waiting()
|
||||
{
|
||||
var selfPort = FreePort();
|
||||
var peerPort = FreePort();
|
||||
var host = await StartNodeAsync("otopcua-selfform-2", selfPort, peerPort, selfFormAfter: null);
|
||||
try
|
||||
{
|
||||
var system = host.Services.GetRequiredService<ActorSystem>();
|
||||
|
||||
(await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(6)))
|
||||
.ShouldBeFalse("with SelfFormAfter unset the node must keep waiting on InitJoin");
|
||||
|
||||
// Positive control: the node WAS formable all along.
|
||||
var cluster = Akka.Cluster.Cluster.Get(system);
|
||||
cluster.Join(cluster.SelfAddress);
|
||||
(await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(20)))
|
||||
.ShouldBeTrue("positive control — an explicit self-join must bring this node Up");
|
||||
}
|
||||
finally
|
||||
{
|
||||
await StopAsync(host);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Regression for the live-gate islanding defect (2026-07-22). A peer that is <b>reachable</b>
|
||||
/// but not yet admitting the join must NOT be treated as "peer down at boot".
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// On the docker-dev rig, a node bounced by a manual failover restarted, received
|
||||
/// <c>InitJoinAck</c> from its live peer — the join was in flight and healthy — but did
|
||||
/// not get the Welcome inside the window, because the peer's ring still held the node's
|
||||
/// previous incarnation. The fallback fired on the timer and the node formed a
|
||||
/// <i>second</i> cluster, islanded until an operator restarted it. The original design
|
||||
/// assumed <c>Join(SelfAddress)</c> would be ignored mid-handshake; it is not.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The peer here is a bare <see cref="TcpListener"/> that accepts connections and speaks
|
||||
/// no Akka at all — reachable at the transport level, never answering a join. That is the
|
||||
/// defect's shape with nothing faked: under the pre-fix code this node self-formed on the
|
||||
/// timer; under the guard it keeps waiting. The positive control then proves the guard is
|
||||
/// a <i>guard</i> and not a disablement: drop the listener and the same node self-forms.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
[Fact]
|
||||
public async Task Reachable_peer_suppresses_self_forming_until_it_goes_away()
|
||||
{
|
||||
var selfPort = FreePort();
|
||||
var peerPort = FreePort();
|
||||
|
||||
// A peer that accepts TCP and does nothing else — reachable, but no join will ever complete.
|
||||
var peer = new TcpListener(IPAddress.Loopback, peerPort);
|
||||
peer.Start();
|
||||
|
||||
var host = await StartNodeAsync(
|
||||
"otopcua-selfform-4", selfPort, peerPort, TimeSpan.FromSeconds(2));
|
||||
try
|
||||
{
|
||||
var system = host.Services.GetRequiredService<ActorSystem>();
|
||||
|
||||
// Several windows pass. Pre-fix this self-formed after the first one.
|
||||
(await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(10)))
|
||||
.ShouldBeFalse("a reachable peer means a join may be in flight — self-forming here islands this node");
|
||||
|
||||
// Positive control: the peer really goes away, and now the fallback does its job.
|
||||
peer.Stop();
|
||||
|
||||
(await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(30)))
|
||||
.ShouldBeTrue("once the peer is genuinely gone the fallback must still self-form");
|
||||
}
|
||||
finally
|
||||
{
|
||||
try { peer.Stop(); } catch (SocketException) { /* already stopped by the test body */ }
|
||||
await StopAsync(host);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// THE island guard. A node that is not in its own seed list — the current docker-dev site-node
|
||||
/// topology, where site-a/site-b list only central-1 — must never self-form: it would island
|
||||
/// itself from the real seeds permanently. Positive control as above.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Site_node_seeded_only_by_central_never_self_forms()
|
||||
{
|
||||
var selfPort = FreePort();
|
||||
var peerPort = FreePort();
|
||||
var host = await StartNodeAsync(
|
||||
"otopcua-selfform-3", selfPort, peerPort, TimeSpan.FromSeconds(1), selfInSeeds: false);
|
||||
try
|
||||
{
|
||||
var system = host.Services.GetRequiredService<ActorSystem>();
|
||||
|
||||
(await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(5)))
|
||||
.ShouldBeFalse("a node absent from its own seed list must wait, not island itself");
|
||||
|
||||
var cluster = Akka.Cluster.Cluster.Get(system);
|
||||
cluster.Join(cluster.SelfAddress);
|
||||
(await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(20)))
|
||||
.ShouldBeTrue("positive control — an explicit self-join must bring this node Up");
|
||||
}
|
||||
finally
|
||||
{
|
||||
await StopAsync(host);
|
||||
}
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user