dohertj2/ScadaBridge
1
0
Fork 0
Code Issues 2 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

feat(cluster): self-first seed ordering closes the boot-alone outage gap

Browse Source 
Every node now lists ITSELF as seed-nodes[0] and its partner second. Akka runs
FirstSeedNodeProcess -- the only bootstrap path that can form a NEW cluster when
no peer answers InitJoin -- exclusively for seed-nodes[0]; every other node runs
JoinSeedNodeProcess and retries InitJoin forever. That is why a lone cold-starting
central-b never came Up (the "registered outage gap"), and self-first ordering
closes it using Akka's own protocol.

- 6 node appsettings swapped (the *-node-b configs; the -a nodes were already
  self-first). All 14 shipped node configs now satisfy the invariant.
- StartupValidator enforces it at boot, comparing host AND port -- the invariant
  fails silently when broken, so it is enforced loudly. NOTE: the gitignored
  deploy/wonder-app-vd03/ overlay must be reordered before its next deploy or
  that node will refuse to boot.
- SelfFirstSeedBootstrapTests: real in-process clusters at production
  failure-detection timings, incl. a falsifiability control proving the OLD
  peer-first ordering never forms.

Rejected alternative (implemented, measured, discarded): an external self-form
timer calling Cluster.Join(SelfAddress) after a window. It sits outside Akka's
join handshake and so cannot tell "no seed answered" from "a seed answered and
the join is in flight". On a routine standby restart the peer is alive but the
join stalls behind removal of the node's own stale incarnation; a Join(self)
during TryingToJoin abandons the in-flight join and forms a second cluster at
the same address -- still split after 90s. Docs that claimed self-first ordering
was unsafe for simultaneous cold start are corrected: while mutually reachable
the InitJoin handshake converges them to one cluster (measured).
This commit is contained in:
Joseph Doherty
2026-07-22 06:32:00 -04:00
parent 69b3ccfc37
commit 4a6341d871
15 changed files with 316 additions and 24 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/ZB.MOM.WW.ScadaBridge.Host.Tests/StartupValidatorTests.cs
+50
View File
@@ -124,6 +124,10 @@ public class StartupValidatorTests
{
var values = ValidCentralConfig();
values["ScadaBridge:Node:RemotingPort"] = port;
// The self-first seed rule (2026-07-22) compares host AND port, so this node's own
// seed entry moves with its remoting port — otherwise this port-range test would be
// asserting against a config that is inconsistent for an unrelated reason.
values["ScadaBridge:Cluster:SeedNodes:0"] = $"akka.tcp://scadabridge@central-node1:{port}";
var config = BuildConfig(values);
var ex = Record.Exception(() => StartupValidator.Validate(config));
@@ -273,6 +277,52 @@ public class StartupValidatorTests
Assert.Contains("SeedNodes must have at least 2 entries", ex.Message);
}
[Fact]
public void PeerFirstSeedOrder_FailsValidation()
{
// Decision 2026-07-22: every node must list ITSELF as seed-nodes[0]. Akka only runs
// FirstSeedNodeProcess (the process that can form a new cluster when no peer answers)
// when seed-nodes[0] is this node's own address; with the peer first the node can
// never cold-start alone — the "registered outage gap".
var values = ValidCentralConfig();
values["ScadaBridge:Cluster:SeedNodes:0"] = "akka.tcp://scadabridge@central-node2:8081";
values["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@central-node1:8081";
var config = BuildConfig(values);
var ex = Assert.Throws<InvalidOperationException>(() => StartupValidator.Validate(config));
Assert.Contains("SeedNodes", ex.Message);
Assert.Contains("must list this node itself first", ex.Message);
}
[Fact]
public void SelfFirstSeedOrder_OnASiteNode_PassesValidation()
{
// Positive control for the rule above: the shipped ordering must validate on a Site
// node too (the rule is unconditional, not Central-only).
var config = BuildConfig(ValidSiteConfig());
var ex = Record.Exception(() => StartupValidator.Validate(config));
Assert.Null(ex);
}
[Fact]
public void SelfFirstSeed_MatchedOnHostAndPort_NotJustHost()
{
// Both nodes of a pair can share a hostname when they differ by port (a two-node
// dev/loopback install). The rule must compare host AND port, or such a node passes
// while actually being the non-first seed.
var values = ValidCentralConfig();
values["ScadaBridge:Node:NodeHostname"] = "localhost";
values["ScadaBridge:Node:RemotingPort"] = "8082";
values["ScadaBridge:Cluster:SeedNodes:0"] = "akka.tcp://scadabridge@localhost:8081";
values["ScadaBridge:Cluster:SeedNodes:1"] = "akka.tcp://scadabridge@localhost:8082";
var config = BuildConfig(values);
var ex = Assert.Throws<InvalidOperationException>(() => StartupValidator.Validate(config));
Assert.Contains("must list this node itself first", ex.Message);
}
[Theory]
[InlineData("0")]
[InlineData("-1")]
tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Cluster/SelfFirstSeedBootstrapTests.cs
+152
View File
@@ -0,0 +1,152 @@
using Akka.Actor;
using Akka.Cluster;
using Akka.Configuration;
using ZB.MOM.WW.ScadaBridge.ClusterInfrastructure;
using ZB.MOM.WW.ScadaBridge.Host;
using ZB.MOM.WW.ScadaBridge.Host.Actors;
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Cluster;
/// <summary>
/// Guards the self-first seed-ordering invariant (decision 2026-07-22): every node lists
/// ITSELF as <c>seed-nodes[0]</c> and its partner second.
///
/// <para><b>Why the ordering is the mechanism.</b> Akka runs a different bootstrap process
/// depending on whether <c>seed-nodes[0]</c> is this node's own address. When it is,
/// <c>FirstSeedNodeProcess</c> runs: it InitJoins the OTHER seeds and self-joins only after
/// <c>seed-node-timeout</c> passes with nobody answering. When it is not,
/// <c>JoinSeedNodeProcess</c> runs, which can never form a new cluster — it retries InitJoin
/// forever. That is the "registered outage gap" (docker/README.md): a lone cold-starting
/// non-first seed never came Up. Listing self first closes it using Akka's own protocol.</para>
///
/// <para><b>Why not an external self-form watchdog.</b> A timer that waits N seconds for
/// membership and then calls <c>Cluster.Join(SelfAddress)</c> cannot see Akka's join
/// handshake, so it cannot distinguish "no seed answered" from "a seed answered and the join
/// is in flight". <see cref="Restarting_node_rejoins_its_live_peer_instead_of_self_forming"/>
/// is the case that killed that design: on a routine standby restart the peer is alive but
/// the join is stalled behind removal of this node's own stale incarnation, and a Join(self)
/// issued during <c>TryingToJoin</c> abandons the in-flight join and forms a SECOND cluster at
/// the same address — a permanent split (measured: still split after 90s). Akka's own
/// first-seed process has no such race because it is part of the handshake.</para>
///
/// These build REAL clusters from the production <c>BuildHocon</c> output, like
/// <see cref="TwoNodeClusterFixture"/>, at production failure-detection timings.
/// </summary>
public sealed class SelfFirstSeedBootstrapTests : IAsyncLifetime
{
private readonly List<ActorSystem> _systems = new();
/// <summary>Starts a node configured the way every shipped node appsettings is: its own
/// address first, the partner second. <paramref name="selfFirst"/> = false reproduces the
/// pre-fix ordering, which is what makes the outage-gap assertions falsifiable.</summary>
private ActorSystem StartNode(int selfPort, int peerPort, bool selfFirst = true)
{
var nodeOptions = new NodeOptions { Role = "Central", NodeHostname = "127.0.0.1", RemotingPort = selfPort };
var self = $"akka.tcp://scadabridge@127.0.0.1:{selfPort}";
var peer = $"akka.tcp://scadabridge@127.0.0.1:{peerPort}";
var clusterOptions = new ClusterOptions
{
SeedNodes = selfFirst ? new List<string> { self, peer } : new List<string> { peer, self },
// Production failure-detection envelope: this is what sets how long a restarting
// node's join stays stalled behind its own stale incarnation.
StableAfter = TimeSpan.FromSeconds(15),
HeartbeatInterval = TimeSpan.FromSeconds(2),
FailureDetectionThreshold = TimeSpan.FromSeconds(10),
MinNrOfMembers = 1,
};
var hocon = AkkaHostedService.BuildHocon(
nodeOptions, clusterOptions, new[] { "Central" },
TimeSpan.FromSeconds(1), TimeSpan.FromSeconds(3));
// Crash simulation: Terminate() must NOT run CoordinatedShutdown (no Leave gossip),
// matching TwoNodeClusterFixture — otherwise a "restart after crash" degenerates to
// the graceful path where the peer has already removed the old member.
var config = ConfigurationFactory
.ParseString("akka.coordinated-shutdown.run-by-actor-system-terminate = off")
.WithFallback(ConfigurationFactory.ParseString(hocon));
var system = ActorSystem.Create("scadabridge", config);
_systems.Add(system);
return system;
}
[Fact]
public async Task Lone_cold_start_forms_a_cluster_when_the_peer_is_dead()
{
var selfPort = TwoNodeClusterFixture.GetFreeTcpPort();
var deadPeerPort = TwoNodeClusterFixture.GetFreeTcpPort(); // nothing listening
var node = StartNode(selfPort, deadPeerPort);
// Akka's FirstSeedNodeProcess self-joins after seed-node-timeout (~5s) once the dead
// peer fails to answer InitJoin. This is the unattended cold-start-alone guarantee.
await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(30));
Assert.Equal(MemberStatus.Up, Akka.Cluster.Cluster.Get(node).SelfMember.Status);
}
[Fact]
public async Task Peer_first_ordering_is_the_outage_gap_and_never_forms()
{
// FALSIFIABILITY CONTROL for the test above: with the OLD ordering (peer first) the
// identical scenario never comes Up. If this test ever starts passing quickly, the
// self-first ordering has stopped being the thing doing the work.
var selfPort = TwoNodeClusterFixture.GetFreeTcpPort();
var deadPeerPort = TwoNodeClusterFixture.GetFreeTcpPort();
var node = StartNode(selfPort, deadPeerPort, selfFirst: false);
await Task.Delay(TimeSpan.FromSeconds(15)); // 3x the seed-node-timeout
var cluster = Akka.Cluster.Cluster.Get(node);
Assert.Empty(cluster.State.Members); // still InitJoin-looping — the registered outage gap
// Positive control: the node was formable all along; only the ordering blocked it.
cluster.Join(cluster.SelfAddress);
await TwoNodeClusterFixture.WaitForMembersUp(node, 1, TimeSpan.FromSeconds(30));
}
[Fact]
public async Task Restarting_node_rejoins_its_live_peer_instead_of_self_forming()
{
// The case that rejected the self-form-watchdog design (code review 2026-07-22, C1):
// a routine standby restart while the peer is alive must rejoin, never island.
var portA = TwoNodeClusterFixture.GetFreeTcpPort();
var portB = TwoNodeClusterFixture.GetFreeTcpPort();
var nodeA = StartNode(portA, portB);
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 1, TimeSpan.FromSeconds(30));
var nodeB = StartNode(portB, portA);
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(30));
// Hard-crash B and immediately restart it at the SAME address, as a container
// recreate / service restart does.
await nodeB.Terminate().WaitAsync(TimeSpan.FromSeconds(10));
var nodeB2 = StartNode(portB, portA);
// Both must converge on ONE 2-member cluster — never two 1-member clusters.
await TwoNodeClusterFixture.WaitForMembersUp(nodeB2, 2, TimeSpan.FromSeconds(90));
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(90));
}
[Fact]
public async Task Both_nodes_cold_starting_together_converge_on_one_cluster()
{
// Self-first on BOTH nodes raises the obvious question: does a simultaneous cold
// start produce two clusters? While they are mutually reachable it does not — the
// InitJoin handshake resolves it before either self-joins. (A genuine boot-time
// PARTITION would still split, which is the same class auto-down already accepts.)
var portA = TwoNodeClusterFixture.GetFreeTcpPort();
var portB = TwoNodeClusterFixture.GetFreeTcpPort();
var nodeA = StartNode(portA, portB);
var nodeB = StartNode(portB, portA);
await TwoNodeClusterFixture.WaitForMembersUp(nodeA, 2, TimeSpan.FromSeconds(60));
await TwoNodeClusterFixture.WaitForMembersUp(nodeB, 2, TimeSpan.FromSeconds(60));
}
public Task InitializeAsync() => Task.CompletedTask;
public async Task DisposeAsync()
{
foreach (var s in _systems)
{
try { await s.Terminate().WaitAsync(TimeSpan.FromSeconds(10)); } catch { /* teardown */ }
}
}
}