feat(cluster): self-first seed ordering closes the boot-alone outage gap
Every node now lists ITSELF as seed-nodes[0] and its partner second. Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW cluster when no peer answers InitJoin -- exclusively for seed-nodes[0]; every other node runs JoinSeedNodeProcess and retries InitJoin forever. That is why a lone cold-starting central-b never came Up (the "registered outage gap"), and self-first ordering closes it using Akka's own protocol. - 6 node appsettings swapped (the *-node-b configs; the -a nodes were already self-first). All 14 shipped node configs now satisfy the invariant. - StartupValidator enforces it at boot, comparing host AND port -- the invariant fails silently when broken, so it is enforced loudly. NOTE: the gitignored deploy/wonder-app-vd03/ overlay must be reordered before its next deploy or that node will refuse to boot. - SelfFirstSeedBootstrapTests: real in-process clusters at production failure-detection timings, incl. a falsifiability control proving the OLD peer-first ordering never forms. Rejected alternative (implemented, measured, discarded): an external self-form timer calling Cluster.Join(SelfAddress) after a window. It sits outside Akka's join handshake and so cannot tell "no seed answered" from "a seed answered and the join is in flight". On a routine standby restart the peer is alive but the join stalls behind removal of the node's own stale incarnation; a Join(self) during TryingToJoin abandons the in-flight join and forms a second cluster at the same address -- still split after 90s. Docs that claimed self-first ordering was unsafe for simultaneous cold start are corrected: while mutually reachable the InitJoin handshake converges them to one cluster (measured).
This commit is contained in:
@@ -93,6 +93,22 @@ public static class StartupValidator
|
||||
.Require("ScadaBridge:Cluster:SeedNodes",
|
||||
_ => seedNodes != null && seedNodes.Count >= 2,
|
||||
"must have at least 2 entries")
|
||||
// Self-first seed ordering (decision 2026-07-22). Akka runs FirstSeedNodeProcess —
|
||||
// the ONLY bootstrap path that can form a new cluster when no peer answers InitJoin —
|
||||
// exclusively when seed-nodes[0] is this node's own address. Every other node runs
|
||||
// JoinSeedNodeProcess and retries InitJoin forever, so a node listing its partner
|
||||
// first cannot cold-start alone: that is the "registered outage gap"
|
||||
// (docker/README.md), and it is a silent failure at boot rather than a loud one.
|
||||
// Enforced here rather than in ClusterOptionsValidator because only this validator
|
||||
// sees both the node identity and the seed list.
|
||||
.Require("ScadaBridge:Cluster:SeedNodes",
|
||||
_ => seedNodes is not { Count: >= 1 }
|
||||
|| SeedNodeIsSelf(seedNodes[0], nodeSection["NodeHostname"], port),
|
||||
"must list this node itself first: seed-nodes[0] has to be this node's own "
|
||||
+ $"'akka.tcp://scadabridge@{nodeSection["NodeHostname"]}:{port}'. Akka only lets "
|
||||
+ "seed-nodes[0] form a new cluster, so with the partner listed first this node "
|
||||
+ "can never start while its peer is down (Component-ClusterInfrastructure.md → "
|
||||
+ "Seed Node Ordering)")
|
||||
// The big Site-only block: GrpcPort/MetricsPort validity + cross-field
|
||||
// collisions + seed-node-port loop, in the original order.
|
||||
.When(role == "Site", p =>
|
||||
@@ -155,4 +171,33 @@ public static class StartupValidator
|
||||
|
||||
return int.TryParse(seedNode[(lastColon + 1)..], out var port) ? port : -1;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Extracts the host from an Akka seed-node address of the form
|
||||
/// <c>akka.tcp://system@host:port</c>. Returns an empty string when no host can be parsed.
|
||||
/// </summary>
|
||||
private static string SeedNodeHost(string seedNode)
|
||||
{
|
||||
if (string.IsNullOrWhiteSpace(seedNode))
|
||||
return string.Empty;
|
||||
|
||||
var at = seedNode.LastIndexOf('@');
|
||||
var lastColon = seedNode.LastIndexOf(':');
|
||||
if (at < 0 || lastColon <= at)
|
||||
return string.Empty;
|
||||
|
||||
return seedNode[(at + 1)..lastColon];
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// True when <paramref name="seedNode"/> addresses this node itself (host AND port).
|
||||
/// Host comparison is case-insensitive because DNS names are; it is otherwise exact —
|
||||
/// Akka does no DNS canonicalisation either, so <c>node-a</c> and
|
||||
/// <c>node-a.example.com</c> are genuinely different seed identities to the cluster.
|
||||
/// </summary>
|
||||
private static bool SeedNodeIsSelf(string seedNode, string? nodeHostname, int remotingPort)
|
||||
{
|
||||
return SeedNodePort(seedNode) == remotingPort
|
||||
&& string.Equals(SeedNodeHost(seedNode), nodeHostname, StringComparison.OrdinalIgnoreCase);
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user