feat(cluster)!: self-first seed ordering replaces the InitJoin self-form watchdog
Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW
cluster when no peer answers InitJoin -- exclusively when seed-nodes[0] is the
node's own address. Every other node runs JoinSeedNodeProcess and retries
InitJoin forever. That is why "both peers are seeds" never meant "either can
cold-start alone", and it is fixed here the way Akka itself intends: each seed
node lists ITSELF first.
- docker-dev central-2 now lists itself as SeedNodes__0 (central-1 was already
self-first). The site nodes are untouched -- they seed off central-1 only and
are deliberately not seeds.
- AkkaClusterOptionsValidator enforces the invariant at boot via the shared
ZB.MOM.WW.Configuration AddValidatedOptions/ValidateOnStart seam. The rule is
CONDITIONAL -- self must be seed[0] only IF self is in the list at all -- or it
would refuse to boot every driver-only site node. Identity is compared on
PublicHostname (falling back to Hostname when blank) AND port, i.e. the address
Akka puts in SelfAddress: matching the 0.0.0.0 bind address would find no seed
anywhere and leave the rule silently inert on the whole docker-dev rig.
- ClusterBootstrapFallback + Cluster:SelfFormAfter are DELETED. The watchdog sat
outside Akka's join handshake, so it could not distinguish "no seed answered"
from "a seed answered and the join is in flight" -- and Cluster.Join(SelfAddress)
is not ignored mid-handshake, it wins. The live gate (ea45ace1) caught it
islanding a node that a manual failover had bounced: InitJoinAck received, no
Welcome inside the window because the peer's ring still held the old
incarnation, watchdog fired, second cluster. The TCP reachability guard patched
that one shape; the race stayed. It was also inert for every non-seed node, so
after this change it can never usefully fire.
- SelfFormBootstrapTests -> SelfFirstSeedBootstrapTests: real in-process clusters
through the production bootstrap at production failure-detection timings, incl.
a falsifiability control proving the OLD peer-first ordering never forms (that
test failed at 11s against the watchdog, which is what proved the retirement),
the restart-into-a-live-peer case that killed the watchdog, and a simultaneous
cold start converging on ONE cluster.
Mirrors ScadaBridge 4a6341d8; anticipates per-cluster-mesh-program Phase 6, which
had this retirement queued.
This commit is contained in:
@@ -7,6 +7,7 @@ using Akka.Remote.Hosting;
|
||||
using Microsoft.Extensions.Configuration;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.Configuration;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Cluster;
|
||||
@@ -19,13 +20,19 @@ public static class ServiceCollectionExtensions
|
||||
/// configurator via <see cref="WithOtOpcUaClusterBootstrap"/> — keeping the entire Akka graph
|
||||
/// under Akka.Hosting's management so cluster singletons land on the same ActorSystem.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The binding is validated at startup (<c>ValidateOnStart</c>) by
|
||||
/// <see cref="AkkaClusterOptionsValidator"/>, so a seed list that cannot bootstrap this node —
|
||||
/// self listed behind its partner — fails the host loudly instead of leaving it running but
|
||||
/// permanently outside the cluster.
|
||||
/// </remarks>
|
||||
/// <param name="services">The service collection to configure.</param>
|
||||
/// <param name="configuration">The application configuration containing cluster options.</param>
|
||||
/// <returns>The same service collection, for chaining.</returns>
|
||||
public static IServiceCollection AddOtOpcUaCluster(this IServiceCollection services, IConfiguration configuration)
|
||||
{
|
||||
services.AddOptions<AkkaClusterOptions>()
|
||||
.Bind(configuration.GetSection(AkkaClusterOptions.SectionName));
|
||||
services.AddValidatedOptions<AkkaClusterOptions, AkkaClusterOptionsValidator>(
|
||||
configuration, AkkaClusterOptions.SectionName);
|
||||
|
||||
services.AddSingleton<IClusterRoleInfo, ClusterRoleInfo>();
|
||||
|
||||
@@ -92,22 +99,13 @@ public static class ServiceCollectionExtensions
|
||||
// ActorSystem for exactly that reason: the effective value is the only one worth asserting.
|
||||
builder.AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend);
|
||||
|
||||
// InitJoin self-form fallback (decision 2026-07-22): registered as an Akka.Hosting startup
|
||||
// task so every host built through this bootstrap — production AND the test hosts in
|
||||
// Cluster.Tests — arms it identically. Guarded inside Arm: disabled when SelfFormAfter is
|
||||
// unset; inert when this node is not in its own seed list (site nodes seeded only by
|
||||
// central-1 must wait, not island).
|
||||
// ILoggerFactory is fully qualified: `using Microsoft.Extensions.Logging` would make the
|
||||
// LogLevel in ConfigureLoggers above ambiguous with Akka.Event.LogLevel.
|
||||
var fallbackLogger =
|
||||
(serviceProvider.GetService<Microsoft.Extensions.Logging.ILoggerFactory>()
|
||||
?? Microsoft.Extensions.Logging.Abstractions.NullLoggerFactory.Instance)
|
||||
.CreateLogger("ZB.MOM.WW.OtOpcUa.Cluster.ClusterBootstrapFallback");
|
||||
builder.AddStartup((system, _) =>
|
||||
{
|
||||
ClusterBootstrapFallback.Arm(system, options, fallbackLogger);
|
||||
return Task.CompletedTask;
|
||||
});
|
||||
// NOTE (2026-07-22): no bootstrap watchdog is armed here any more. The cold-start-alone gap
|
||||
// it existed for — Akka lets only seed-nodes[0] form a NEW cluster — is now closed by
|
||||
// self-first seed ordering (AkkaClusterOptions.SeedNodes), enforced at boot by
|
||||
// AkkaClusterOptionsValidator. The retired ClusterBootstrapFallback timer sat OUTSIDE Akka's
|
||||
// join handshake, so it could not distinguish "no seed answered" from "a seed answered and
|
||||
// the join is in flight", and Cluster.Join(SelfAddress) is not ignored mid-handshake — it
|
||||
// wins. See docs/Redundancy.md → "Bootstrap: self-first seed ordering".
|
||||
|
||||
return builder;
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user