dohertj2/lmxopcua
1
0
Fork 0
Code Issues 9 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(cluster)!: self-first seed ordering replaces the InitJoin self-form watchdog

Browse Source 
Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW
cluster when no peer answers InitJoin -- exclusively when seed-nodes[0] is the
node's own address. Every other node runs JoinSeedNodeProcess and retries
InitJoin forever. That is why "both peers are seeds" never meant "either can
cold-start alone", and it is fixed here the way Akka itself intends: each seed
node lists ITSELF first.

- docker-dev central-2 now lists itself as SeedNodes__0 (central-1 was already
  self-first). The site nodes are untouched -- they seed off central-1 only and
  are deliberately not seeds.
- AkkaClusterOptionsValidator enforces the invariant at boot via the shared
  ZB.MOM.WW.Configuration AddValidatedOptions/ValidateOnStart seam. The rule is
  CONDITIONAL -- self must be seed[0] only IF self is in the list at all -- or it
  would refuse to boot every driver-only site node. Identity is compared on
  PublicHostname (falling back to Hostname when blank) AND port, i.e. the address
  Akka puts in SelfAddress: matching the 0.0.0.0 bind address would find no seed
  anywhere and leave the rule silently inert on the whole docker-dev rig.
- ClusterBootstrapFallback + Cluster:SelfFormAfter are DELETED. The watchdog sat
  outside Akka's join handshake, so it could not distinguish "no seed answered"
  from "a seed answered and the join is in flight" -- and Cluster.Join(SelfAddress)
  is not ignored mid-handshake, it wins. The live gate (ea45ace1) caught it
  islanding a node that a manual failover had bounced: InitJoinAck received, no
  Welcome inside the window because the peer's ring still held the old
  incarnation, watchdog fired, second cluster. The TCP reachability guard patched
  that one shape; the race stayed. It was also inert for every non-seed node, so
  after this change it can never usefully fire.
- SelfFormBootstrapTests -> SelfFirstSeedBootstrapTests: real in-process clusters
  through the production bootstrap at production failure-detection timings, incl.
  a falsifiability control proving the OLD peer-first ordering never forms (that
  test failed at 11s against the watchdog, which is what proved the retirement),
  the restart-into-a-live-peer case that killed the watchdog, and a simultaneous
  cold start converging on ONE cluster.

Mirrors ScadaBridge 4a6341d8; anticipates per-cluster-mesh-program Phase 6, which
had this retirement queued.
This commit is contained in:
Joseph Doherty
2026-07-22 07:42:06 -04:00
parent a78425ea8f
commit 3f24d4d6bf
9 changed files with 625 additions and 506 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptions.cs
+25 -16
View File
@@ -20,7 +20,31 @@ public sealed class AkkaClusterOptions
/// </summary>
public string PublicHostname { get; set; } = "127.0.0.1";
/// <summary>Gets or sets the seed nodes for cluster bootstrapping.</summary>
/// <summary>
/// Gets or sets the seed nodes for cluster bootstrapping.
/// </summary>
/// <remarks>
/// <para>
/// <b>ORDER IS LOAD-BEARING (decision 2026-07-22): a node that is one of its own seeds
/// must list ITSELF first.</b> Akka runs <c>FirstSeedNodeProcess</c> — the only bootstrap
/// path that can form a NEW cluster when no peer answers <c>InitJoin</c> — exclusively
/// when <c>seed-nodes[0]</c> is this node's own address; any other node runs
/// <c>JoinSeedNodeProcess</c>, which retries <c>InitJoin</c> forever and can never form a
/// cluster. So listing both peers does NOT mean either can cold-start alone: a node that
/// lists its partner first never comes Up while that partner is down. Self-first ordering
/// closes that gap inside Akka's own handshake — unlike the retired
/// <c>Cluster:SelfFormAfter</c> watchdog, which sat outside it and could not tell "no
/// seed answered" from "a seed answered and the join is in flight".
/// </para>
/// <para>
/// The rule is <b>conditional</b>: it binds only when this node's own address appears in
/// this list. A node seeded exclusively by someone else — today's docker-dev site nodes,
/// which list only <c>central-1</c> — is legitimately not a seed and is exempt. Enforced
/// at boot by <see cref="AkkaClusterOptionsValidator"/>; identity is
/// <see cref="PublicHostname"/> + <see cref="Port"/> (what Akka puts in
/// <c>SelfAddress</c>), never the <see cref="Hostname"/> bind address.
/// </para>
/// </remarks>
public string[] SeedNodes { get; set; } = Array.Empty<string>();
/// <summary>
@@ -55,19 +79,4 @@ public sealed class AkkaClusterOptions
/// </para>
/// </remarks>
public string SplitBrainResolverStrategy { get; set; } = "auto-down";
/// <summary>
/// Bootstrap self-form fallback window (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
/// Akka only lets the FIRST listed seed form a new cluster; a non-first seed cold-starting
/// while its peer is down loops on InitJoin forever ("auto-down removes the crash outage,
/// not this one" — docs/Redundancy.md). When this node has waited longer than this window
/// without cluster membership it joins itself — but ONLY if its own address appears in
/// <see cref="SeedNodes"/>; a non-seed node (e.g. a site node whose only seed is central-1)
/// stays waiting, because self-forming there creates a permanent island. Default 10s
/// (same-datacenter pair; a live peer answers InitJoin in milliseconds). <c>null</c> or a
/// non-positive value disables the fallback. Accepted trade: both pair nodes cold-starting
/// inside the window while mutually unreachable form two clusters — the same dual-active
/// class the auto-down strategy already accepts, same recovery (restart one side).
/// </summary>
public TimeSpan? SelfFormAfter { get; set; } = TimeSpan.FromSeconds(10);
}
src/Core/ZB.MOM.WW.OtOpcUa.Cluster/AkkaClusterOptionsValidator.cs
+96
View File
@@ -0,0 +1,96 @@
using Akka.Actor;
using ZB.MOM.WW.Configuration;
namespace ZB.MOM.WW.OtOpcUa.Cluster;
/// <summary>
/// Fail-fast startup validator for <see cref="AkkaClusterOptions"/>, built on the shared
/// <c>ZB.MOM.WW.Configuration</c> <see cref="OptionsValidatorBase{TOptions}"/> and wired through
/// <c>AddValidatedOptions</c> in <see cref="ServiceCollectionExtensions.AddOtOpcUaCluster"/>.
/// </summary>
/// <remarks>
/// <para>
/// <b>Self-first seed ordering (decision 2026-07-22).</b> Akka runs
/// <c>FirstSeedNodeProcess</c> — the only bootstrap path that can form a NEW cluster when no
/// peer answers <c>InitJoin</c> — exclusively when <c>seed-nodes[0]</c> is this node's own
/// address. Every other node runs <c>JoinSeedNodeProcess</c> and retries <c>InitJoin</c>
/// forever. A node that lists its partner first therefore cannot cold-start while that
/// partner is down, and the failure is <i>silent</i>: the process runs, the port listens, the
/// node simply never becomes a member. Enforced here so it is loud, at boot, on the node
/// that has the problem.
/// </para>
/// <para>
/// <b>The rule is conditional and must stay that way.</b> It binds only when this node's own
/// address appears in its own seed list. A driver-only site node is seeded solely by
/// <c>central-1</c> and is legitimately not a seed of anything; an unconditional "self must
/// be first" rule would refuse to boot every one of them.
/// </para>
/// <para>
/// <b>Identity is the advertised address, never the bind address.</b> In docker-dev
/// <c>Cluster:Hostname</c> is <c>0.0.0.0</c> and <c>Cluster:PublicHostname</c> is the
/// container DNS name — the value Akka puts in <c>SelfAddress</c> and the only one a seed URI
/// can match. Comparing against <c>Hostname</c> would match nothing anywhere, silently exempt
/// every node, and leave this rule inert. <c>PublicHostname</c> falling back to
/// <c>Hostname</c> when blank mirrors Akka's own <c>public-hostname</c> semantics.
/// </para>
/// </remarks>
public sealed class AkkaClusterOptionsValidator : OptionsValidatorBase<AkkaClusterOptions>
{
/// <inheritdoc />
protected override void Validate(ValidationBuilder builder, AkkaClusterOptions options)
{
ArgumentNullException.ThrowIfNull(options);
var seeds = options.SeedNodes ?? Array.Empty<string>();
if (seeds.Length == 0)
{
// Single-node / dev default: nothing is claimed about ordering, so there is nothing to
// enforce. (Akka bootstraps from an empty seed list by waiting for an explicit join.)
return;
}
var selfHost = string.IsNullOrWhiteSpace(options.PublicHostname)
? options.Hostname
: options.PublicHostname;
var selfIndex = Array.FindIndex(seeds, s => IsSelf(s, selfHost, options.Port));
if (selfIndex <= 0)
{
// -1 = this node is not one of its own seeds (exempt); 0 = correctly ordered.
return;
}
builder.Add(
$"Cluster:SeedNodes must list this node itself first: entry {selfIndex} is this node "
+ $"('{seeds[selfIndex]}') but entry 0 is '{seeds[0]}'. Akka only lets seed-nodes[0] form "
+ "a new cluster (FirstSeedNodeProcess); every other node retries InitJoin forever, so "
+ "with the partner listed first this node can never start while its peer is down. Swap "
+ "the entries — see docs/Redundancy.md → 'Bootstrap: self-first seed ordering'.");
}
/// <summary>
/// True when <paramref name="seed"/> addresses this node itself — host AND port.
/// </summary>
/// <remarks>
/// Host comparison is case-insensitive because DNS names are, but is otherwise exact: Akka does
/// no DNS canonicalisation either, so <c>central-2</c> and <c>central-2.example.com</c> are
/// genuinely different seed identities to the cluster. A seed entry Akka itself cannot parse is
/// treated as "not this node" — reporting it is Akka's job, with Akka's wording, and this rule
/// must not turn a parse problem into a misleading ordering complaint.
/// </remarks>
private static bool IsSelf(string seed, string selfHost, int selfPort)
{
Address address;
try
{
address = Address.Parse(seed);
}
catch (Exception)
{
return false;
}
return address.Port == selfPort
&& string.Equals(address.Host, selfHost, StringComparison.OrdinalIgnoreCase);
}
}
src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ClusterBootstrapFallback.cs
-206
View File
@@ -1,206 +0,0 @@
using Akka.Actor;
using Microsoft.Extensions.Logging;
namespace ZB.MOM.WW.OtOpcUa.Cluster;
/// <summary>
/// InitJoin self-form fallback (decision 2026-07-22, scadaproj/akka_failover.md §6.1).
/// Akka only lets the FIRST listed seed form a NEW cluster; every other node retries InitJoin
/// forever. So "both nodes are seed nodes" (<see cref="AkkaClusterOptions.SeedNodes"/>) does NOT
/// mean either can cold-start alone — a non-first seed booting while its peer is down waits
/// indefinitely ("auto-down removes the crash outage, not this one" — <c>docs/Redundancy.md</c>).
/// This watchdog waits <see cref="AkkaClusterOptions.SelfFormAfter"/> for membership; on expiry it
/// forms a cluster on itself.
///
/// <para><b>Island safety.</b> Fires ONLY when this node's own address is in its own seed list.
/// A node that is not a seed (never legitimately first) must keep waiting: if it self-formed, a
/// later-booting real seed would form a second cluster and the two can never merge. That is the
/// current docker-dev site-node topology — site-a/site-b nodes list only central-1. For nodes that
/// ARE seeds, sequential recovery is island-free — Akka's join protocol prefers an existing cluster
/// (a booting node only self-forms when NO seed answers InitJoin), so a peer booting after this
/// node self-formed simply joins it.</para>
///
/// <para><b>Reachability guard (added 2026-07-22 after a live-gate failure).</b> An expired window
/// is NOT sufficient evidence that the peer is down, and joining on the timer alone islands nodes.
/// Drilled on the docker-dev rig: a node bounced by a manual failover restarted, received
/// <c>InitJoinAck</c> from its live peer — a join in flight and healthy — but did not get the
/// Welcome inside the window, because the peer's ring still held the node's previous incarnation
/// (Exiting → Down → Removed). The fallback fired anyway and the node formed a SECOND cluster,
/// islanded until an operator restarted it. So <c>Join(SelfAddress)</c> is <b>not</b> ignored
/// mid-handshake — it wins, which is the opposite of what the original design assumed. Before
/// self-forming, this therefore TCP-probes the other seed addresses: if any accepts a connection
/// the peer is alive, a join is likely already in flight, and the fallback waits another window
/// instead. That is also exactly what the fallback claims to detect — "no seed answered InitJoin
/// (peer down at boot)".</para>
///
/// <para><b>Residual risk.</b> Both pair nodes cold-starting inside the window while mutually
/// unreachable (a boot-time partition): both self-form, the same dual-active class the auto-down
/// downing strategy already accepts, with the same recovery (restart one side).</para>
///
/// <para>Deliberately duplicated from ScadaBridge's equivalent (like the termination watchdogs) —
/// the two repos share the behavior, not a package.</para>
/// </summary>
public static class ClusterBootstrapFallback
{
/// <summary>
/// Arms the fallback on a freshly created actor system. Safe to call unconditionally: it no-ops
/// (with an explanatory log) when the fallback is disabled or when this node is not one of its
/// own seeds.
/// </summary>
/// <param name="system">The actor system whose cluster membership is being watched.</param>
/// <param name="options">The bound cluster options carrying the window and the seed list.</param>
/// <param name="logger">Logger for the armed / inert / self-forming decisions.</param>
/// <param name="peerProbe">
/// Overrides the reachability probe (host, port) → reachable. Production passes
/// <see langword="null"/> for the real TCP connect; tests substitute it to drive the guard
/// deterministically.
/// </param>
public static void Arm(
ActorSystem system,
AkkaClusterOptions options,
ILogger logger,
Func<string, int, Task<bool>>? peerProbe = null)
{
ArgumentNullException.ThrowIfNull(system);
ArgumentNullException.ThrowIfNull(options);
ArgumentNullException.ThrowIfNull(logger);
if (options.SelfFormAfter is not { } window || window <= TimeSpan.Zero)
{
logger.LogInformation(
"Cluster self-form fallback disabled (SelfFormAfter not set) — a node cold-starting "
+ "while its peer is down will wait on InitJoin indefinitely.");
return;
}
var cluster = Akka.Cluster.Cluster.Get(system);
var self = cluster.SelfAddress;
var isSeed = options.SeedNodes.Any(s => TryParseAddress(s, out var a) && a.Equals(self));
if (!isSeed)
{
logger.LogInformation(
"Cluster self-form fallback inactive: this node ({Self}) is not in its own seed list "
+ "[{Seeds}] — self-forming here would island it from the real seeds.",
self,
string.Join(", ", options.SeedNodes));
return;
}
// Every seed that is not this node. These are the addresses whose reachability decides
// whether an expired window means "peer is down" or "peer is alive and we are mid-join".
var peerSeeds = options.SeedNodes
.Select(s => TryParseAddress(s, out var a) ? a : null)
.Where(a => a is not null && !a.Equals(self))
.Select(a => (Host: a!.Host ?? string.Empty, Port: a!.Port ?? 0))
.Where(p => !string.IsNullOrWhiteSpace(p.Host) && p.Port > 0)
.ToList();
var probe = peerProbe ?? TryConnectAsync;
var joined = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously);
cluster.RegisterOnMemberUp(() => joined.TrySetResult());
_ = Task.Run(async () =>
{
while (true)
{
var winner = await Task.WhenAny(joined.Task, Task.Delay(window));
if (winner == joined.Task || system.WhenTerminated.IsCompleted)
{
return;
}
var reachable = await AnyReachableAsync(peerSeeds, probe);
if (reachable is not null)
{
// Do NOT self-form. A reachable peer means either a join already in flight (the
// failure this guard exists for) or a peer about to answer — self-forming here
// creates a second cluster that can never merge.
logger.LogInformation(
"Self-form window ({Window}) expired, but seed peer {Peer} is reachable — a join is "
+ "most likely already in flight (a node re-joining after a restart is not admitted "
+ "until its previous incarnation is removed). Continuing to wait rather than "
+ "forming a second cluster.",
window,
reachable);
continue;
}
logger.LogWarning(
"No cluster membership after {Window} and no seed peer is reachable — the peer is down "
+ "at boot. Self-forming a cluster at {Self} so this node becomes operational; if the "
+ "peer was merely partitioned (not dead), the pair is now dual-active — restart one "
+ "side after the partition heals (accepted availability-first trade, decision "
+ "2026-07-22).",
window,
self);
cluster.Join(self);
return;
}
});
}
/// <summary>How long a single peer-reachability connect attempt may take.</summary>
public static readonly TimeSpan ProbeTimeout = TimeSpan.FromSeconds(2);
/// <summary>
/// Returns the first reachable peer as <c>host:port</c>, or <see langword="null"/> when none
/// answered — including when there are no peer seeds at all (a lone-seed config, where an
/// expired window really does mean this node is on its own).
/// </summary>
private static async Task<string?> AnyReachableAsync(
IReadOnlyList<(string Host, int Port)> peers,
Func<string, int, Task<bool>> probe)
{
foreach (var (host, port) in peers)
{
bool ok;
try
{
ok = await probe(host, port);
}
catch (Exception)
{
// A probe that cannot even be attempted (DNS gone, socket exhaustion) is treated as
// unreachable — it must not throw out of the watchdog and disarm the fallback.
ok = false;
}
if (ok) return $"{host}:{port}";
}
return null;
}
private static async Task<bool> TryConnectAsync(string host, int port)
{
using var client = new System.Net.Sockets.TcpClient();
using var cts = new CancellationTokenSource(ProbeTimeout);
try
{
await client.ConnectAsync(host, port, cts.Token);
return client.Connected;
}
catch (Exception)
{
// Refused, unresolvable, or timed out — all mean "not reachable right now".
return false;
}
}
private static bool TryParseAddress(string seed, out Address address)
{
try
{
address = Address.Parse(seed);
return true;
}
catch (Exception)
{
// A malformed seed entry must not disarm the fallback; bad seed URIs surface from
// Akka's own bootstrap.
address = Address.AllSystems;
return false;
}
}
}
src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ServiceCollectionExtensions.cs
+16 -18
View File
@@ -7,6 +7,7 @@ using Akka.Remote.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.Configuration;
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
namespace ZB.MOM.WW.OtOpcUa.Cluster;
@@ -19,13 +20,19 @@ public static class ServiceCollectionExtensions
/// configurator via <see cref="WithOtOpcUaClusterBootstrap"/> — keeping the entire Akka graph
/// under Akka.Hosting's management so cluster singletons land on the same ActorSystem.
/// </summary>
/// <remarks>
/// The binding is validated at startup (<c>ValidateOnStart</c>) by
/// <see cref="AkkaClusterOptionsValidator"/>, so a seed list that cannot bootstrap this node —
/// self listed behind its partner — fails the host loudly instead of leaving it running but
/// permanently outside the cluster.
/// </remarks>
/// <param name="services">The service collection to configure.</param>
/// <param name="configuration">The application configuration containing cluster options.</param>
/// <returns>The same service collection, for chaining.</returns>
public static IServiceCollection AddOtOpcUaCluster(this IServiceCollection services, IConfiguration configuration)
{
services.AddOptions<AkkaClusterOptions>()
.Bind(configuration.GetSection(AkkaClusterOptions.SectionName));
services.AddValidatedOptions<AkkaClusterOptions, AkkaClusterOptionsValidator>(
configuration, AkkaClusterOptions.SectionName);
services.AddSingleton<IClusterRoleInfo, ClusterRoleInfo>();
@@ -92,22 +99,13 @@ public static class ServiceCollectionExtensions
// ActorSystem for exactly that reason: the effective value is the only one worth asserting.
builder.AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend);
// InitJoin self-form fallback (decision 2026-07-22): registered as an Akka.Hosting startup
// task so every host built through this bootstrap — production AND the test hosts in
// Cluster.Tests — arms it identically. Guarded inside Arm: disabled when SelfFormAfter is
// unset; inert when this node is not in its own seed list (site nodes seeded only by
// central-1 must wait, not island).
// ILoggerFactory is fully qualified: `using Microsoft.Extensions.Logging` would make the
// LogLevel in ConfigureLoggers above ambiguous with Akka.Event.LogLevel.
var fallbackLogger =
(serviceProvider.GetService<Microsoft.Extensions.Logging.ILoggerFactory>()
?? Microsoft.Extensions.Logging.Abstractions.NullLoggerFactory.Instance)
.CreateLogger("ZB.MOM.WW.OtOpcUa.Cluster.ClusterBootstrapFallback");
builder.AddStartup((system, _) =>
{
ClusterBootstrapFallback.Arm(system, options, fallbackLogger);
return Task.CompletedTask;
});
// NOTE (2026-07-22): no bootstrap watchdog is armed here any more. The cold-start-alone gap
// it existed for — Akka lets only seed-nodes[0] form a NEW cluster — is now closed by
// self-first seed ordering (AkkaClusterOptions.SeedNodes), enforced at boot by
// AkkaClusterOptionsValidator. The retired ClusterBootstrapFallback timer sat OUTSIDE Akka's
// join handshake, so it could not distinguish "no seed answered" from "a seed answered and
// the join is in flight", and Cluster.Join(SelfAddress) is not ignored mid-handshake — it
// wins. See docs/Redundancy.md → "Bootstrap: self-first seed ordering".
return builder;
}
src/Core/ZB.MOM.WW.OtOpcUa.Cluster/ZB.MOM.WW.OtOpcUa.Cluster.csproj
+4
View File
@@ -14,6 +14,10 @@
<PackageReference Include="Akka.Remote.Hosting"/>
<PackageReference Include="Microsoft.Extensions.Hosting"/>
<PackageReference Include="Microsoft.Extensions.Options.ConfigurationExtensions"/>
<!-- Shared options-validation primitives (OptionsValidatorBase / AddValidatedOptions), the
same seam the Host's Ldap/OpcUa/Historian validators use. Pulled in here because the
self-first seed-ordering rule belongs with the options it validates. -->
<PackageReference Include="ZB.MOM.WW.Configuration"/>
</ItemGroup>
<ItemGroup>