feat(cluster)!: self-first seed ordering replaces the InitJoin self-form watchdog

Akka runs FirstSeedNodeProcess -- the only bootstrap path that can form a NEW
cluster when no peer answers InitJoin -- exclusively when seed-nodes[0] is the
node's own address. Every other node runs JoinSeedNodeProcess and retries
InitJoin forever. That is why "both peers are seeds" never meant "either can
cold-start alone", and it is fixed here the way Akka itself intends: each seed
node lists ITSELF first.

- docker-dev central-2 now lists itself as SeedNodes__0 (central-1 was already
  self-first). The site nodes are untouched -- they seed off central-1 only and
  are deliberately not seeds.
- AkkaClusterOptionsValidator enforces the invariant at boot via the shared
  ZB.MOM.WW.Configuration AddValidatedOptions/ValidateOnStart seam. The rule is
  CONDITIONAL -- self must be seed[0] only IF self is in the list at all -- or it
  would refuse to boot every driver-only site node. Identity is compared on
  PublicHostname (falling back to Hostname when blank) AND port, i.e. the address
  Akka puts in SelfAddress: matching the 0.0.0.0 bind address would find no seed
  anywhere and leave the rule silently inert on the whole docker-dev rig.
- ClusterBootstrapFallback + Cluster:SelfFormAfter are DELETED. The watchdog sat
  outside Akka's join handshake, so it could not distinguish "no seed answered"
  from "a seed answered and the join is in flight" -- and Cluster.Join(SelfAddress)
  is not ignored mid-handshake, it wins. The live gate (ea45ace1) caught it
  islanding a node that a manual failover had bounced: InitJoinAck received, no
  Welcome inside the window because the peer's ring still held the old
  incarnation, watchdog fired, second cluster. The TCP reachability guard patched
  that one shape; the race stayed. It was also inert for every non-seed node, so
  after this change it can never usefully fire.
- SelfFormBootstrapTests -> SelfFirstSeedBootstrapTests: real in-process clusters
  through the production bootstrap at production failure-detection timings, incl.
  a falsifiability control proving the OLD peer-first ordering never forms (that
  test failed at 11s against the watchdog, which is what proved the retirement),
  the restart-into-a-live-peer case that killed the watchdog, and a simultaneous
  cold start converging on ONE cluster.

Mirrors ScadaBridge 4a6341d8; anticipates per-cluster-mesh-program Phase 6, which
had this retirement queued.
This commit is contained in:
Joseph Doherty
2026-07-22 07:42:06 -04:00
parent a78425ea8f
commit 3f24d4d6bf
9 changed files with 625 additions and 506 deletions
@@ -7,6 +7,7 @@ using Akka.Remote.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Options;
using ZB.MOM.WW.Configuration;
using ZB.MOM.WW.OtOpcUa.Commons.Interfaces;
namespace ZB.MOM.WW.OtOpcUa.Cluster;
@@ -19,13 +20,19 @@ public static class ServiceCollectionExtensions
/// configurator via <see cref="WithOtOpcUaClusterBootstrap"/> — keeping the entire Akka graph
/// under Akka.Hosting's management so cluster singletons land on the same ActorSystem.
/// </summary>
/// <remarks>
/// The binding is validated at startup (<c>ValidateOnStart</c>) by
/// <see cref="AkkaClusterOptionsValidator"/>, so a seed list that cannot bootstrap this node —
/// self listed behind its partner — fails the host loudly instead of leaving it running but
/// permanently outside the cluster.
/// </remarks>
/// <param name="services">The service collection to configure.</param>
/// <param name="configuration">The application configuration containing cluster options.</param>
/// <returns>The same service collection, for chaining.</returns>
public static IServiceCollection AddOtOpcUaCluster(this IServiceCollection services, IConfiguration configuration)
{
services.AddOptions<AkkaClusterOptions>()
.Bind(configuration.GetSection(AkkaClusterOptions.SectionName));
services.AddValidatedOptions<AkkaClusterOptions, AkkaClusterOptionsValidator>(
configuration, AkkaClusterOptions.SectionName);
services.AddSingleton<IClusterRoleInfo, ClusterRoleInfo>();
@@ -92,22 +99,13 @@ public static class ServiceCollectionExtensions
// ActorSystem for exactly that reason: the effective value is the only one worth asserting.
builder.AddHocon(BuildDowningHocon(options), HoconAddMode.Prepend);
// InitJoin self-form fallback (decision 2026-07-22): registered as an Akka.Hosting startup
// task so every host built through this bootstrap — production AND the test hosts in
// Cluster.Tests — arms it identically. Guarded inside Arm: disabled when SelfFormAfter is
// unset; inert when this node is not in its own seed list (site nodes seeded only by
// central-1 must wait, not island).
// ILoggerFactory is fully qualified: `using Microsoft.Extensions.Logging` would make the
// LogLevel in ConfigureLoggers above ambiguous with Akka.Event.LogLevel.
var fallbackLogger =
(serviceProvider.GetService<Microsoft.Extensions.Logging.ILoggerFactory>()
?? Microsoft.Extensions.Logging.Abstractions.NullLoggerFactory.Instance)
.CreateLogger("ZB.MOM.WW.OtOpcUa.Cluster.ClusterBootstrapFallback");
builder.AddStartup((system, _) =>
{
ClusterBootstrapFallback.Arm(system, options, fallbackLogger);
return Task.CompletedTask;
});
// NOTE (2026-07-22): no bootstrap watchdog is armed here any more. The cold-start-alone gap
// it existed for — Akka lets only seed-nodes[0] form a NEW cluster — is now closed by
// self-first seed ordering (AkkaClusterOptions.SeedNodes), enforced at boot by
// AkkaClusterOptionsValidator. The retired ClusterBootstrapFallback timer sat OUTSIDE Akka's
// join handshake, so it could not distinguish "no seed answered" from "a seed answered and
// the join is in flight", and Cluster.Join(SelfAddress) is not ignored mid-handshake — it
// wins. See docs/Redundancy.md → "Bootstrap: self-first seed ordering".
return builder;
}