feat(cluster): simultaneous-cold-start split-brain guard (opt-in)
v2-ci / build (push) Successful in 4m46s
v2-ci / unit-tests (push) Failing after 16m9s

Closes the residual Phase-7 gap: when BOTH nodes of a 2-node pair cold-start at the
same instant, each self-first seed runs FirstSeedNodeProcess, times out waiting for
the other, and forms its own 1-node cluster — two Primaries in one pair (the Phase 6
live gate reproduced this reliably on docker). The prior mitigation was operational
(staggered start / compose depends_on), which does not exist on production hardware.

The guard (dark switch Cluster:BootstrapGuard:Enabled, default OFF) prevents the split
without giving up cold-start-alone:
- The lower-address node is the preferred founder: self-first, forms immediately.
- The higher node probes its partner's Akka port (TCP connect) up to PartnerProbeSeconds:
  reachable => peer-first (join the founder, never race it); unreachable => self-first
  (partner is dead, form alone). The order is decided BEFORE the single JoinSeedNodes,
  from an explicit reachability signal — never re-formed mid-handshake (the retired
  SelfFormAfter failure mode). When on, Akka gets no config seeds (BuildClusterOptions)
  and ClusterBootstrapCoordinator drives the join.

Review-driven hardening: case-insensitive tie-break (a hostname-casing mismatch would
reopen the split); fail-fast validation of the timing knobs; the residual "founder dies
in the probe->join window" hang is documented and made operator-visible (warning + a
restart recovers). Real-ActorSystem coordinator tests cover the load-bearing higher-node
cold-start-alone case; 147/147 Cluster tests pass.

Live-gated on docker-dev (site-a = enablement demo, serialization removed, guard on;
site-b keeps depends_on-serialization + guard off as the A/B control): simultaneous
start -> site-a-1 founds, site-a-2 probes-reachable-joins -> 250/240, NO split;
higher-node-alone -> probes dead founder, self-forms -> 250; founder rejoin -> 240.

Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
This commit is contained in:
Joseph Doherty
2026-07-24 08:34:29 -04:00
parent c50ebcf7dc
commit 279d1d0fb1
11 changed files with 884 additions and 13 deletions
+14 -7
View File
@@ -516,6 +516,12 @@ services:
# FetchAndCache, never gossip; central and site-a share no seed list. # FetchAndCache, never gossip; central and site-a share no seed list.
Cluster__SeedNodes__0: "akka.tcp://otopcua@site-a-1:4053" Cluster__SeedNodes__0: "akka.tcp://otopcua@site-a-1:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@site-a-2:4053" Cluster__SeedNodes__1: "akka.tcp://otopcua@site-a-2:4053"
# Simultaneous-cold-start split-brain guard (SITE-A enablement demo). With the pair no longer
# startup-serialized, this is what stops both self-first seeds forming their own 1-node cluster:
# the lower-address node (site-a-1) founds immediately; the higher (site-a-2) probes site-a-1 and
# joins it when reachable, or forms alone only if site-a-1 is truly down. Akka gets NO config
# seeds when this is on (BuildClusterOptions); ClusterBootstrapCoordinator issues JoinSeedNodes.
Cluster__BootstrapGuard__Enabled: "true"
Cluster__Roles__0: "driver" Cluster__Roles__0: "driver"
Cluster__Roles__1: "cluster-SITE-A" Cluster__Roles__1: "cluster-SITE-A"
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see # Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see
@@ -588,13 +594,11 @@ services:
sql: { condition: service_healthy } sql: { condition: service_healthy }
central-1: { condition: service_started } central-1: { condition: service_started }
migrator: { condition: service_completed_successfully } migrator: { condition: service_completed_successfully }
# Serialize the SITE-A pair's startup so site-a-1 forms the mesh first and site-a-2 JOINS it — # SITE-A is the BOOTSTRAP-GUARD enablement demo: NO startup serialization on the pair — both
# exactly as central-2 waits on central-1. Both nodes are self-first seeds (seed[0]=self), so BOTH # nodes start simultaneously (the exact race that split-brained in the Phase 6 gate). The
# run Akka's FirstSeedNodeProcess and, if they start simultaneously, EACH forms its own single-node # Cluster:BootstrapGuard (enabled in both site-a env blocks) is the ONLY thing preventing the
# cluster (split brain — observed in the Phase 6 live gate: both logged "JOINING itself ... forming # split: it makes the lower-address node the founder and the higher-address node probe-then-join.
# a new cluster", both elected themselves Primary → 250/250 instead of 250/240). Waiting on site-a-1 # site-b keeps the depends_on:service_healthy serialization + guard OFF, as the A/B control.
# gives it the head start to be Up before site-a-2's InitJoin, so site-a-2 joins the existing mesh.
site-a-1: { condition: service_healthy }
environment: environment:
OTOPCUA_ROLES: "driver,cluster-SITE-A" OTOPCUA_ROLES: "driver,cluster-SITE-A"
# Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1). # Per-cluster mesh Phase 4: no ConfigDb string on driver-only site nodes (see site-a-1).
@@ -617,6 +621,9 @@ services:
# second) — it no longer seeds off central-1. # second) — it no longer seeds off central-1.
Cluster__SeedNodes__0: "akka.tcp://otopcua@site-a-2:4053" Cluster__SeedNodes__0: "akka.tcp://otopcua@site-a-2:4053"
Cluster__SeedNodes__1: "akka.tcp://otopcua@site-a-1:4053" Cluster__SeedNodes__1: "akka.tcp://otopcua@site-a-1:4053"
# Bootstrap-guard enablement demo — see site-a-1. site-a-2 is the HIGHER address, so the guard
# makes it probe site-a-1 and join it (peer-first) rather than race-form its own cluster.
Cluster__BootstrapGuard__Enabled: "true"
Cluster__Roles__0: "driver" Cluster__Roles__0: "driver"
Cluster__Roles__1: "cluster-SITE-A" Cluster__Roles__1: "cluster-SITE-A"
# Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see # Mesh command transport (per-cluster mesh Phase 2, made mandatory by Phase 6) — see
+55 -5
View File
@@ -401,11 +401,61 @@ Comparing against `Cluster:Hostname` would be worse than wrong: in docker-dev it
no seed URI anywhere, so the rule would silently exempt the entire rig. no seed URI anywhere, so the rule would silently exempt the entire rig.
Sequential recovery is island-free by construction: a `FirstSeedNodeProcess` node self-joins only when **no** Sequential recovery is island-free by construction: a `FirstSeedNodeProcess` node self-joins only when **no**
seed answered, so a peer booting after the survivor formed simply joins it as the youngest member. Both seed answered, so a peer booting after the survivor formed simply joins it as the youngest member.
nodes cold-starting *simultaneously* converge on one cluster while they are mutually reachable — the
`InitJoin` handshake resolves it before either self-join deadline expires. The residual risk is a boot-time **Simultaneous cold-start CAN split — this is the residual gap the bootstrap guard below closes.** When
**partition** (both cold-starting, mutually unreachable): both form, which is the same dual-active class the BOTH nodes start from nothing at the same instant, each runs `FirstSeedNodeProcess`, and if neither has
`auto-down` strategy already accepts, with the same recovery (restart one side). formed before the other's `seed-node-timeout` expires, EACH self-joins and forms its own single-node
cluster — two Primaries in one pair. On in-process loopback the `InitJoin` handshake usually resolves fast
enough to converge, but the Phase 6 live gate reproduced the split reliably on the docker-dev rig (real
container start timing + DNS): both nodes logged `JOINING itself … forming a new cluster` and both advertised
ServiceLevel 250. Two independent clusters do **not** auto-merge, so the split persists until an operator
restarts one side.
### Bootstrap guard (`Cluster:BootstrapGuard`, opt-in)
The guard eliminates the simultaneous-start split without giving up cold-start-alone. It is a **dark switch,
default off** — a node with `Cluster:BootstrapGuard:Enabled=false` (the default) keeps Akka's config-driven
self-first auto-join exactly as above.
When enabled, the node is given **no config seed nodes** (so Akka does not auto-join), and
`ClusterBootstrapCoordinator` picks the join order from a deterministic address tie-break plus a reachability
probe:
- The node with the lexicographically **lower** canonical `host:port` is the **preferred founder**: it joins
self-first and forms immediately if no peer answers — no probe, no delay.
- The **higher** node probes its partner's Akka port (a plain TCP connect; the partner binds its port well
before it forms) for up to `PartnerProbeSeconds` (default 25 s): **reachable ⇒ peer-first** (it joins the
founder, never races it); **unreachable after the window ⇒ self-first** (the partner is genuinely down, so
it forms alone — cold-start-alone preserved for the higher node too).
The order is decided **before** issuing a single `Cluster.JoinSeedNodes`, from an explicit reachability
signal — it never re-decides mid-handshake, the failure mode that retired the `SelfFormAfter` watchdog.
**Residual trade-off (accepted, operator-visible):** once the higher node commits peer-first it cannot
self-form (Akka's `JoinSeedNodeProcess` retries forever). If the founder dies in the small probe→join window,
the higher node hangs unjoined; the coordinator logs a clear WARNING after a bounded grace, and a **restart
recovers it** (the guard re-runs, finds the founder down, and forms alone). Config keys:
| Key | Default | Notes |
|---|---|---|
| `Cluster:BootstrapGuard:Enabled` | `false` | Dark switch. Only meaningful on a node that is one of its own two pair seeds; inert elsewhere. |
| `Cluster:BootstrapGuard:PartnerProbeSeconds` | `25` | Higher node's probe window. Must comfortably exceed the founder's process-start-to-Akka-bind time, or a slow-but-alive founder is mistaken for dead and the split re-opens. Validated `> 0` at boot. |
| `Cluster:BootstrapGuard:PartnerProbeIntervalMs` | `500` | Interval between probes. |
| `Cluster:BootstrapGuard:ProbeConnectTimeoutMs` | `1000` | Per-probe TCP connect timeout. |
On the docker-dev rig the **site-a pair is the enablement demo** (guard on, startup serialization removed so
both nodes cold-start simultaneously); **site-b keeps the `depends_on: service_healthy` startup serialization
with the guard off**, as the A/B control. Either mechanism prevents the split; the guard is the one that also
works on production hardware, where compose `depends_on` does not exist.
The alternative to the guard is **operational**: stagger the two VMs' service-manager start, or bring the
designated founder VM up first — see the Phase 7 co-located operator runbook
(`docs/plans/2026-07-24-mesh-phase7-failover-drills.md`).
The residual risk that remains even with the guard is a boot-time **partition** (both cold-starting, mutually
unreachable from the start): both form, which is the same dual-active class the `auto-down` strategy already
accepts, with the same recovery (restart one side).
Pinned by `SelfFirstSeedBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real Pinned by `SelfFirstSeedBootstrapTests` (`tests/Core/ZB.MOM.WW.OtOpcUa.Cluster.Tests/`), which starts real
in-process clusters through the production bootstrap at production failure-detection timings: a lone in-process clusters through the production bootstrap at production failure-detection timings: a lone
@@ -79,4 +79,46 @@ public sealed class AkkaClusterOptions
/// </para> /// </para>
/// </remarks> /// </remarks>
public string SplitBrainResolverStrategy { get; set; } = "auto-down"; public string SplitBrainResolverStrategy { get; set; } = "auto-down";
/// <summary>
/// The simultaneous-cold-start split-brain guard (<c>Cluster:BootstrapGuard</c>). Default OFF — a
/// dark switch, so existing deployments and tests keep Akka's config-driven self-first auto-join
/// unchanged. See <see cref="ClusterBootstrapGuard"/> for the decision logic and
/// <c>ClusterBootstrapCoordinator</c> for the runtime.
/// </summary>
public ClusterBootstrapGuardOptions BootstrapGuard { get; set; } = new();
}
/// <summary>
/// Configuration for the simultaneous-cold-start split-brain guard. When
/// <see cref="Enabled"/>, the node does NOT auto-join from its config seeds; a coordinator picks the
/// join order (founder self-first / joiner peer-first) after a reachability probe. See
/// <see cref="ClusterBootstrapGuard"/>.
/// </summary>
public sealed class ClusterBootstrapGuardOptions
{
/// <summary>
/// Gets or sets whether the guard is active. Default <see langword="false"/> — Akka auto-joins from
/// the config seed list exactly as before. Only meaningful on a node that is one of its own two
/// pair seeds; inert everywhere else.
/// </summary>
public bool Enabled { get; set; }
/// <summary>
/// Gets or sets how long the higher-address node probes its partner's Akka endpoint before
/// concluding the partner is dead and forming alone. Must comfortably exceed the partner's
/// worst-case process-start-to-Akka-bind time so a slow-but-alive partner is never mistaken for a
/// dead one (which would re-open the split). Default 25 s.
/// </summary>
public int PartnerProbeSeconds { get; set; } = 25;
/// <summary>
/// Gets or sets the interval between partner reachability probes, in milliseconds. Default 500 ms.
/// </summary>
public int PartnerProbeIntervalMs { get; set; } = 500;
/// <summary>
/// Gets or sets the per-probe TCP connect timeout, in milliseconds. Default 1000 ms.
/// </summary>
public int ProbeConnectTimeoutMs { get; set; } = 1000;
} }
@@ -41,6 +41,8 @@ public sealed class AkkaClusterOptionsValidator : OptionsValidatorBase<AkkaClust
{ {
ArgumentNullException.ThrowIfNull(options); ArgumentNullException.ThrowIfNull(options);
ValidateBootstrapGuard(builder, options.BootstrapGuard);
var seeds = options.SeedNodes ?? Array.Empty<string>(); var seeds = options.SeedNodes ?? Array.Empty<string>();
if (seeds.Length == 0) if (seeds.Length == 0)
{ {
@@ -68,6 +70,25 @@ public sealed class AkkaClusterOptionsValidator : OptionsValidatorBase<AkkaClust
+ "the entries — see docs/Redundancy.md → 'Bootstrap: self-first seed ordering'."); + "the entries — see docs/Redundancy.md → 'Bootstrap: self-first seed ordering'.");
} }
/// <summary>
/// When the bootstrap guard is enabled, its timing knobs must be positive. A zero/negative
/// <c>PartnerProbeSeconds</c> in particular silently degrades the guard to "never wait, always
/// conclude the partner is dead" — it would form alone immediately and re-open the very split it
/// exists to close. Fail-fast at boot rather than producing silence (the same rationale every
/// sibling options validator in this project cites).
/// </summary>
private static void ValidateBootstrapGuard(ValidationBuilder builder, ClusterBootstrapGuardOptions guard)
{
if (guard is null || !guard.Enabled) return;
if (guard.PartnerProbeSeconds <= 0)
builder.Add($"Cluster:BootstrapGuard:PartnerProbeSeconds must be > 0 when the guard is enabled (was {guard.PartnerProbeSeconds}).");
if (guard.PartnerProbeIntervalMs <= 0)
builder.Add($"Cluster:BootstrapGuard:PartnerProbeIntervalMs must be > 0 when the guard is enabled (was {guard.PartnerProbeIntervalMs}).");
if (guard.ProbeConnectTimeoutMs <= 0)
builder.Add($"Cluster:BootstrapGuard:ProbeConnectTimeoutMs must be > 0 when the guard is enabled (was {guard.ProbeConnectTimeoutMs}).");
}
/// <summary> /// <summary>
/// True when <paramref name="seed"/> addresses this node itself — host AND port. /// True when <paramref name="seed"/> addresses this node itself — host AND port.
/// </summary> /// </summary>
@@ -0,0 +1,201 @@
using System.Diagnostics;
using System.Net.Sockets;
using Akka.Actor;
using Akka.Cluster;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
namespace ZB.MOM.WW.OtOpcUa.Cluster;
/// <summary>
/// Runs the simultaneous-cold-start split-brain guard: when <c>Cluster:BootstrapGuard:Enabled</c>, the
/// node starts with NO config seed nodes (see <c>BuildClusterOptions</c>), and this coordinator picks
/// the join order and issues the single <see cref="Akka.Cluster.Cluster.JoinSeedNodes"/> once the
/// ActorSystem is up. See <see cref="ClusterBootstrapGuard"/> for the decision logic and the
/// split-brain rationale.
/// </summary>
/// <remarks>
/// <para>
/// The founder (lower canonical address) joins its self-first order immediately. The higher
/// node probes its partner's Akka endpoint (a plain TCP connect — the partner has bound its
/// port well before it forms) up to <see cref="ClusterBootstrapGuardOptions.PartnerProbeSeconds"/>:
/// reachable ⇒ peer-first (join the founder); unreachable ⇒ self-first (partner is dead, form
/// alone). The join runs on a background task so it never blocks host startup, and it is issued
/// exactly once — the coordinator never re-forms a node mid-handshake, the failure mode of the
/// retired <c>SelfFormAfter</c> watchdog.
/// </para>
/// </remarks>
public sealed class ClusterBootstrapCoordinator : IHostedService
{
private readonly Func<ActorSystem> _system;
private readonly AkkaClusterOptions _options;
private readonly ILogger<ClusterBootstrapCoordinator> _log;
private readonly CancellationTokenSource _cts = new();
private Task? _joinTask;
/// <summary>Creates the coordinator.</summary>
/// <param name="system">Lazy accessor for the node's ActorSystem (resolved after Akka's hosted service starts it).</param>
/// <param name="options">The bound cluster options (seed list + guard settings).</param>
/// <param name="log">Logger for the bootstrap decision.</param>
public ClusterBootstrapCoordinator(
Func<ActorSystem> system, IOptions<AkkaClusterOptions> options, ILogger<ClusterBootstrapCoordinator> log)
{
_system = system ?? throw new ArgumentNullException(nameof(system));
_options = options?.Value ?? throw new ArgumentNullException(nameof(options));
_log = log ?? throw new ArgumentNullException(nameof(log));
}
/// <inheritdoc/>
public Task StartAsync(CancellationToken cancellationToken)
{
if (!_options.BootstrapGuard.Enabled)
return Task.CompletedTask; // dark switch off — Akka auto-joined from config seeds already.
// Fire-and-forget so a higher node with a dead partner (which waits the full probe window)
// never blocks host startup. Exceptions are logged; a failed join leaves the node unjoined,
// which is visible and recoverable, never a silent split.
_joinTask = Task.Run(() => RunAsync(_cts.Token), _cts.Token);
return Task.CompletedTask;
}
/// <inheritdoc/>
public async Task StopAsync(CancellationToken cancellationToken)
{
await _cts.CancelAsync().ConfigureAwait(false);
if (_joinTask is not null)
{
try { await _joinTask.ConfigureAwait(false); }
catch (OperationCanceledException) { /* expected on shutdown */ }
}
}
private async Task RunAsync(CancellationToken ct)
{
try
{
var seeds = _options.SeedNodes ?? Array.Empty<string>();
var role = ClusterBootstrapGuard.Analyze(seeds, _options.PublicHostname, _options.Port);
string[] order;
var committedPeerFirst = false;
if (!role.Applies)
{
// Not a pair seed (single-seed legacy site node, self absent, malformed, or 3+ seeds):
// join the configured seeds unchanged — the guard arbitrates only the 2-node pair race.
order = seeds;
_log.LogInformation(
"Bootstrap guard: not a pair seed ({SeedCount} seed(s)); joining configured seeds unchanged.",
seeds.Length);
}
else if (role.IsFounder)
{
order = role.SelfFirstOrder;
_log.LogInformation(
"Bootstrap guard: this node is the preferred founder (lower address); joining self-first, forming immediately if no peer answers.");
}
else
{
var reachable = await ProbePartnerAsync(role.PartnerHost!, role.PartnerPort, ct).ConfigureAwait(false);
order = ClusterBootstrapGuard.HigherNodeOrder(role, reachable);
committedPeerFirst = reachable;
_log.LogInformation(
"Bootstrap guard: this node is the higher address; partner {PartnerHost}:{PartnerPort} was {Reachability} within the probe window; joining {Order}.",
role.PartnerHost, role.PartnerPort, reachable ? "REACHABLE" : "UNREACHABLE",
reachable ? "peer-first (join the founder)" : "self-first (partner down, forming alone)");
}
if (order.Length == 0)
{
_log.LogWarning("Bootstrap guard: no seed nodes to join; node stays unjoined until a seed is configured.");
return;
}
var cluster = Akka.Cluster.Cluster.Get(_system());
var addresses = order.Select(Address.Parse).ToArray();
cluster.JoinSeedNodes(addresses);
// Peer-first is a one-way commitment: JoinSeedNodeProcess never self-forms. If the founder
// died in the probe→join window, this node hangs unjoined forever. We do NOT re-form here
// (the retired SelfFormAfter mid-handshake failure mode) — we make the hang operator-visible
// so a restart, which re-runs the guard against the now-dead founder and self-forms, recovers
// it. Nothing is done on the founder / self-first paths, which always self-form on their own.
if (committedPeerFirst)
await WarnIfNotUpAsync(cluster, ct).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
// Host shutting down before the join completed — nothing to do.
}
catch (Exception ex)
{
_log.LogError(ex, "Bootstrap guard: failed to issue JoinSeedNodes; node remains unjoined.");
}
}
/// <summary>
/// After committing peer-first, waits a bounded time for this node to reach <c>Up</c> and logs a
/// clear warning if it does not — the founder must have died in the probe→join window, leaving this
/// node hung in <c>JoinSeedNodeProcess</c>. Read-only: it never re-forms or re-joins (that is the
/// retired mid-handshake failure mode); it only surfaces the condition so an operator restarts the node.
/// </summary>
private async Task WarnIfNotUpAsync(Akka.Cluster.Cluster cluster, CancellationToken ct)
{
// Give the join a generous grace: the founder's own self-form (seed-node-timeout) plus this
// node's join round-trip. Two probe windows is comfortably beyond both.
var grace = TimeSpan.FromSeconds(Math.Max(10, _options.BootstrapGuard.PartnerProbeSeconds * 2));
var sw = Stopwatch.StartNew();
while (!ct.IsCancellationRequested && sw.Elapsed < grace)
{
if (cluster.SelfMember.Status == MemberStatus.Up) return; // joined — all good
try { await Task.Delay(TimeSpan.FromSeconds(1), ct).ConfigureAwait(false); }
catch (OperationCanceledException) { return; }
}
if (!ct.IsCancellationRequested && cluster.SelfMember.Status != MemberStatus.Up)
_log.LogWarning(
"Bootstrap guard: committed peer-first but this node is still {Status} after {Grace}s — its founder likely died in the probe→join window. This node will NOT self-form on its own (peer-first never does). RESTART it to recover: the guard will re-run, find the founder down, and form alone.",
cluster.SelfMember.Status, (int)grace.TotalSeconds);
}
/// <summary>
/// Polls the partner's Akka endpoint with a plain TCP connect until it is reachable or the probe
/// window elapses. Reachable-at-TCP is a sound proxy for "the partner is coming up": a node binds
/// its Akka port near the start of startup, well before it forms or joins a cluster.
/// </summary>
private async Task<bool> ProbePartnerAsync(string host, int port, CancellationToken ct)
{
var window = TimeSpan.FromSeconds(Math.Max(0, _options.BootstrapGuard.PartnerProbeSeconds));
var interval = TimeSpan.FromMilliseconds(Math.Max(50, _options.BootstrapGuard.PartnerProbeIntervalMs));
var sw = Stopwatch.StartNew();
while (!ct.IsCancellationRequested)
{
if (await TryConnectAsync(host, port, ct).ConfigureAwait(false)) return true;
if (sw.Elapsed >= window) return false;
try { await Task.Delay(interval, ct).ConfigureAwait(false); }
catch (OperationCanceledException) { return false; }
}
return false;
}
private async Task<bool> TryConnectAsync(string host, int port, CancellationToken ct)
{
try
{
using var client = new TcpClient();
using var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
cts.CancelAfter(Math.Max(100, _options.BootstrapGuard.ProbeConnectTimeoutMs));
await client.ConnectAsync(host, port, cts.Token).ConfigureAwait(false);
return client.Connected;
}
catch (OperationCanceledException) when (ct.IsCancellationRequested)
{
throw; // host shutdown — propagate
}
catch
{
return false; // connect refused / timed out / DNS not resolvable yet — partner not up
}
}
}
@@ -0,0 +1,153 @@
using System.Text.RegularExpressions;
namespace ZB.MOM.WW.OtOpcUa.Cluster;
/// <summary>
/// Pure decision logic for the simultaneous-cold-start split-brain guard.
/// </summary>
/// <remarks>
/// <para>
/// Both nodes of a 2-node pair are self-first seeds so <b>either</b> can cold-start alone when
/// its partner is dead (see <see cref="AkkaClusterOptions.SeedNodes"/>). The cost is that when
/// BOTH cold-start at the same instant, each runs Akka's <c>FirstSeedNodeProcess</c>, times out
/// waiting for the other, and forms its OWN single-node cluster — a split brain (two Primaries
/// in one pair). Two independent clusters do not auto-merge, so the split persists until an
/// operator intervenes.
/// </para>
/// <para>
/// This guard breaks the symmetry deterministically without giving up cold-start-alone. The
/// node with the lexicographically <b>lower</b> canonical address is the <i>preferred
/// founder</i>: it always uses self-first order and forms immediately. The <b>higher</b> node
/// first probes whether its partner's Akka endpoint is reachable (see
/// <see cref="ClusterBootstrapCoordinator"/>): reachable ⇒ peer-first order so it JOINS the
/// founder rather than racing it; unreachable after the probe window ⇒ the partner is genuinely
/// down, so it falls back to self-first and forms alone. Exactly one node founds when both are
/// present; the lower node always founds when it starts alone.
/// </para>
/// <para>
/// <b>Residual trade-off (accepted).</b> Once the higher node observes the partner reachable it
/// commits to peer-first — Akka's <c>JoinSeedNodeProcess</c> retries <c>InitJoin</c> forever and
/// never self-forms. If the founder dies in the small window between the probe succeeding and
/// the join completing, the higher node hangs unjoined until it is restarted (a restart re-runs
/// the guard, finds the founder unreachable, and self-forms). We deliberately do NOT re-decide
/// mid-handshake — that is exactly the retired <c>SelfFormAfter</c> failure mode. Instead
/// <see cref="ClusterBootstrapCoordinator"/> makes the hang operator-visible with a warning if
/// the node has not come Up within a bounded time after committing peer-first.
/// </para>
/// <para>
/// This decides the join order BEFORE issuing a single <c>JoinSeedNodes</c>, from an explicit
/// reachability signal — unlike the retired <c>SelfFormAfter</c> watchdog, which fired a
/// <c>Join(self)</c> mid-handshake on a bare timeout and could not tell "no seed answered" from
/// "a join is in flight", islanding a node a failover had just bounced.
/// </para>
/// </remarks>
public static class ClusterBootstrapGuard
{
private static readonly Regex SeedPattern =
new(@"^akka(?:\.[a-z0-9]+)?://[^@/]+@(?<host>[^:/]+):(?<port>\d+)", RegexOptions.IgnoreCase | RegexOptions.Compiled);
/// <summary>The analyzed bootstrap role of this node within its seed list.</summary>
/// <param name="Applies">True only when the guard engages: exactly two seeds, one of them this node.</param>
/// <param name="IsFounder">True when this node is the preferred founder (lower address) — form immediately, no probe.</param>
/// <param name="PartnerHost">The partner's advertised host (to probe when this node is the higher one); null when not applicable.</param>
/// <param name="PartnerPort">The partner's Akka port.</param>
/// <param name="SelfFirstOrder"><c>[self, partner]</c> — self-first; forms a new cluster when no peer answers.</param>
/// <param name="PeerFirstOrder"><c>[partner, self]</c> — peer-first; joins the partner's cluster, never self-forms while it is up.</param>
public sealed record BootstrapRole(
bool Applies,
bool IsFounder,
string? PartnerHost,
int PartnerPort,
string[] SelfFirstOrder,
string[] PeerFirstOrder);
/// <summary>
/// Analyzes the configured seed list to decide this node's bootstrap role. The guard engages only
/// for the Phase-6 pair shape (exactly two seeds, one of them this node); any other shape
/// (single-seed legacy site node, three+ seeds, self absent) returns <see cref="BootstrapRole.Applies"/>
/// = false and the caller joins the configured seeds unchanged.
/// </summary>
/// <param name="seeds">The configured seed URIs (self-first by convention, but order is not relied on here).</param>
/// <param name="selfHost">This node's advertised host (<see cref="AkkaClusterOptions.PublicHostname"/>).</param>
/// <param name="selfPort">This node's Akka port (<see cref="AkkaClusterOptions.Port"/>).</param>
/// <returns>The analyzed role; never null.</returns>
public static BootstrapRole Analyze(IReadOnlyList<string> seeds, string selfHost, int selfPort)
{
ArgumentNullException.ThrowIfNull(seeds);
ArgumentException.ThrowIfNullOrWhiteSpace(selfHost);
var na = new BootstrapRole(false, false, null, 0, Array.Empty<string>(), Array.Empty<string>());
if (seeds.Count != 2) return na;
var selfKey = Key(selfHost, selfPort);
string? self = null, partner = null;
string? partnerHost = null;
var partnerPort = 0;
foreach (var seed in seeds)
{
if (!TryParse(seed, out var host, out var port)) return na;
if (string.Equals(Key(host, port), selfKey, StringComparison.OrdinalIgnoreCase))
self = seed;
else
{
partner = seed;
partnerHost = host;
partnerPort = port;
}
}
// Self must be exactly one of the two, and the other must be a distinct partner.
if (self is null || partner is null || partnerHost is null) return na;
var selfFirst = new[] { self, partner };
var peerFirst = new[] { partner, self };
// Deterministic tie-break: the lower canonical address is the preferred founder. Both nodes
// compute the same comparison (same two seeds), so exactly one is the founder. Case-INSENSITIVE
// to match how self/partner are classified above (and AkkaClusterOptionsValidator.IsSelf):
// container/DNS hostnames are conventionally case-insensitive, and a casing difference between
// the two sides' seed config must NOT make both think they are the founder (which would reopen
// the very split this guard closes).
var isFounder = string.Compare(
Key(selfHost, selfPort),
Key(partnerHost, partnerPort),
StringComparison.OrdinalIgnoreCase) < 0;
return new BootstrapRole(true, isFounder, partnerHost, partnerPort, selfFirst, peerFirst);
}
/// <summary>
/// The order the higher (non-founder) node uses once it has learned whether its partner is
/// reachable: peer-first when the founder is up (join it), self-first when the founder is absent
/// (form alone — cold-start-alone preserved).
/// </summary>
/// <param name="role">The analyzed role (must be applicable and NOT the founder).</param>
/// <param name="partnerReachable">Whether the partner's Akka endpoint became reachable within the probe window.</param>
/// <returns>The seed order to pass to <c>Cluster.JoinSeedNodes</c>.</returns>
public static string[] HigherNodeOrder(BootstrapRole role, bool partnerReachable)
{
ArgumentNullException.ThrowIfNull(role);
return partnerReachable ? role.PeerFirstOrder : role.SelfFirstOrder;
}
/// <summary>Parses <c>akka.tcp://system@host:port[/...]</c> into host + port.</summary>
/// <param name="seed">The seed URI.</param>
/// <param name="host">The parsed advertised host.</param>
/// <param name="port">The parsed Akka port.</param>
/// <returns>True when the seed matched the expected shape.</returns>
public static bool TryParse(string? seed, out string host, out int port)
{
host = string.Empty;
port = 0;
if (string.IsNullOrWhiteSpace(seed)) return false;
var m = SeedPattern.Match(seed.Trim());
if (!m.Success) return false;
if (!int.TryParse(m.Groups["port"].Value, out port)) return false;
host = m.Groups["host"].Value;
return true;
}
private static string Key(string host, int port) => $"{host}:{port}";
}
@@ -241,7 +241,13 @@ public static class ServiceCollectionExtensions
return new ClusterOptions return new ClusterOptions
{ {
SeedNodes = options.SeedNodes, // When the bootstrap guard is on, DO NOT hand Akka the seed list: Akka.Cluster.Hosting
// auto-joins from ClusterOptions.SeedNodes the instant the ActorSystem starts, which is
// exactly the self-first race the guard exists to arbitrate. Left empty, the node starts
// unjoined and ClusterBootstrapCoordinator picks the order (founder self-first / joiner
// peer-first, after a reachability probe) and issues the single Cluster.JoinSeedNodes.
// The config seed list is still read — by the coordinator, off AkkaClusterOptions.SeedNodes.
SeedNodes = options.BootstrapGuard.Enabled ? Array.Empty<string>() : options.SeedNodes,
Roles = options.Roles, Roles = options.Roles,
SplitBrainResolver = IsKeepOldest(options) ? new KeepOldestOption { DownIfAlone = true } : null, SplitBrainResolver = IsKeepOldest(options) ? new KeepOldestOption { DownIfAlone = true } : null,
}; };
@@ -390,6 +390,16 @@ builder.Services.AddAkka("otopcua", (ab, sp) =>
} }
}); });
// Simultaneous-cold-start split-brain guard (dark switch Cluster:BootstrapGuard:Enabled, default off).
// Registered AFTER AddAkka so its StartAsync runs once Akka's hosted service has built (and, when the
// guard is off, already auto-joined) the ActorSystem. When on, the node started with no config seeds
// (BuildClusterOptions) and this coordinator issues the single reachability-gated JoinSeedNodes. It
// no-ops when the guard is off, so registering it unconditionally is safe.
builder.Services.AddHostedService(sp => new ClusterBootstrapCoordinator(
() => sp.GetRequiredService<ActorSystem>(),
sp.GetRequiredService<IOptions<AkkaClusterOptions>>(),
sp.GetRequiredService<ILogger<ClusterBootstrapCoordinator>>()));
// Down-if-alone recovery watchdog (#459). Registered AFTER AddAkka so it starts after Akka's own // Down-if-alone recovery watchdog (#459). Registered AFTER AddAkka so it starts after Akka's own
// hosted service has built the ActorSystem; it resolves the system lazily (never at construction) so // hosted service has built the ActorSystem; it resolves the system lazily (never at construction) so
// it can't race startup. On an unexpected SBR self-down it stops the host so the service supervisor // it can't race startup. On an unexpected SBR self-down it stops the host so the service supervisor
@@ -174,4 +174,44 @@ public sealed class AkkaClusterOptionsValidatorTests
result.Failed.ShouldBeFalse(result.FailureMessage); result.Failed.ShouldBeFalse(result.FailureMessage);
} }
/// <summary>The bootstrap-guard timing knobs default to positive values and pass when enabled.</summary>
[Fact]
public void Bootstrap_guard_defaults_pass()
{
var options = CentralNode("site-a-1", Seed("site-a-1"), Seed("site-a-2"));
options.BootstrapGuard = new ClusterBootstrapGuardOptions { Enabled = true };
var result = new AkkaClusterOptionsValidator().Validate(null, options);
result.Failed.ShouldBeFalse(result.FailureMessage);
}
/// <summary>
/// A zero <c>PartnerProbeSeconds</c> silently degrades the guard to "never wait, form alone
/// immediately" — re-opening the split. It must fail the host at boot, not run degraded.
/// </summary>
[Fact]
public void Bootstrap_guard_with_non_positive_probe_seconds_fails()
{
var options = CentralNode("site-a-1", Seed("site-a-1"), Seed("site-a-2"));
options.BootstrapGuard = new ClusterBootstrapGuardOptions { Enabled = true, PartnerProbeSeconds = 0 };
var result = new AkkaClusterOptionsValidator().Validate(null, options);
result.Failed.ShouldBeTrue();
result.FailureMessage.ShouldContain("PartnerProbeSeconds");
}
/// <summary>The timing knobs are NOT validated when the guard is off — a disabled guard is inert.</summary>
[Fact]
public void Bootstrap_guard_disabled_does_not_validate_timings()
{
var options = CentralNode("site-a-1", Seed("site-a-1"), Seed("site-a-2"));
options.BootstrapGuard = new ClusterBootstrapGuardOptions { Enabled = false, PartnerProbeSeconds = 0 };
var result = new AkkaClusterOptionsValidator().Validate(null, options);
result.Failed.ShouldBeFalse(result.FailureMessage);
}
} }
@@ -0,0 +1,204 @@
using System.Net;
using System.Net.Sockets;
using Akka.Actor;
using Akka.Cluster;
using Akka.Hosting;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
/// <summary>
/// Real-ActorSystem tests for the simultaneous-cold-start split-brain guard
/// (<see cref="ClusterBootstrapCoordinator"/> + <see cref="ClusterBootstrapGuard"/>), started through
/// the SAME wiring as production (<see cref="ServiceCollectionExtensions.WithOtOpcUaClusterBootstrap"/>
/// with <c>Cluster:BootstrapGuard:Enabled</c> + the coordinator hosted service). This subsystem has a
/// history of bugs invisible to pure unit tests, so the load-bearing correctness properties are
/// asserted against running nodes, mirroring <see cref="SelfFirstSeedBootstrapTests"/>.
/// </summary>
/// <remarks>
/// Ephemeral loopback ports are 5-digit, so numeric order equals the guard's ordinal "host:port"
/// string order — <c>Min(port)</c> is the founder, <c>Max(port)</c> is the higher (probing) node.
/// </remarks>
public sealed class ClusterBootstrapCoordinatorTests
{
private static int FreePort()
{
var listener = new TcpListener(IPAddress.Loopback, 0);
listener.Start();
var port = ((IPEndPoint)listener.LocalEndpoint).Port;
listener.Stop();
return port;
}
/// <summary>
/// Starts a node with the bootstrap guard ENABLED, wired exactly as Program.cs does: empty Akka
/// seed list (via <c>BuildClusterOptions</c>) plus the coordinator hosted service that issues the
/// reachability-gated <c>JoinSeedNodes</c>. Seeds are listed self-first (as every shipped config
/// is); the guard, not the order, decides founder vs. joiner.
/// </summary>
private static async Task<IHost> StartGuardedNodeAsync(
string systemName, int selfPort, int peerPort, int probeSeconds = 4)
{
var self = $"akka.tcp://{systemName}@127.0.0.1:{selfPort}";
var peer = $"akka.tcp://{systemName}@127.0.0.1:{peerPort}";
var options = new AkkaClusterOptions
{
SystemName = systemName,
Hostname = "127.0.0.1",
PublicHostname = "127.0.0.1",
Port = selfPort,
Roles = new[] { "driver" },
SeedNodes = new[] { self, peer },
BootstrapGuard = new ClusterBootstrapGuardOptions
{
Enabled = true,
PartnerProbeSeconds = probeSeconds,
PartnerProbeIntervalMs = 250,
ProbeConnectTimeoutMs = 500,
},
};
var builder = Host.CreateDefaultBuilder();
builder.ConfigureServices(services =>
{
services.AddSingleton<IOptions<AkkaClusterOptions>>(Options.Create(options));
services.AddAkka(systemName, (ab, sp) => ab.WithOtOpcUaClusterBootstrap(sp));
// Mirror Program.cs — the coordinator is what drives the guarded join.
services.AddHostedService(sp => new ClusterBootstrapCoordinator(
() => sp.GetRequiredService<ActorSystem>(),
sp.GetRequiredService<IOptions<AkkaClusterOptions>>(),
sp.GetRequiredService<ILogger<ClusterBootstrapCoordinator>>()));
});
var host = builder.Build();
await host.StartAsync();
return host;
}
private static async Task<bool> WaitForUpMembersAsync(IHost host, int expected, TimeSpan timeout)
{
var cluster = Akka.Cluster.Cluster.Get(host.Services.GetRequiredService<ActorSystem>());
var deadline = DateTime.UtcNow + timeout;
while (DateTime.UtcNow < deadline)
{
if (cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expected) return true;
await Task.Delay(200);
}
return cluster.State.Members.Count(m => m.Status == MemberStatus.Up) >= expected;
}
private static async Task StopAsync(IHost host)
{
try { await host.StopAsync(); }
catch (Exception) { /* teardown only */ }
host.Dispose();
}
/// <summary>
/// The FOUNDER (lower address) with the guard on forms a cluster alone when its partner is dead —
/// exactly as un-guarded self-first does. The guard must not regress the preferred founder.
/// </summary>
[Fact]
public async Task Founder_forms_alone_when_partner_is_dead()
{
var low = FreePort();
var high = FreePort();
var founderPort = Math.Min(low, high);
var deadPeerPort = Math.Max(low, high); // nothing listening
var host = await StartGuardedNodeAsync("otopcua-guard-1", founderPort, deadPeerPort);
try
{
(await WaitForUpMembersAsync(host, 1, TimeSpan.FromSeconds(30)))
.ShouldBeTrue("the founder (lower address) must self-form immediately when its partner is down");
}
finally { await StopAsync(host); }
}
/// <summary>
/// THE LOAD-BEARING CASE. The HIGHER node with the guard on must still cold-start ALONE when its
/// partner is genuinely dead: it probes the (dead) founder, times out, and falls back to self-first.
/// If the guard's peer-first logic were wrong, this node would hang in JoinSeedNodeProcess forever —
/// the very regression a naive "always let the lower node found" rule would introduce.
/// </summary>
[Fact]
public async Task Higher_node_forms_alone_when_partner_is_dead_after_probing()
{
var low = FreePort();
var high = FreePort();
var higherPort = Math.Max(low, high);
var deadFounderPort = Math.Min(low, high); // the founder is dead
var host = await StartGuardedNodeAsync("otopcua-guard-2", higherPort, deadFounderPort, probeSeconds: 3);
try
{
// Must come Up AFTER the probe window (~3 s) expires and it falls back to self-first.
(await WaitForUpMembersAsync(host, 1, TimeSpan.FromSeconds(30)))
.ShouldBeTrue("the higher node must fall back to self-first and form alone once the dead partner never answers the probe");
}
finally { await StopAsync(host); }
}
/// <summary>
/// FALSIFIABILITY CONTROL for the test above: the higher node does NOT form alone during the probe
/// window — it is genuinely waiting/probing, not self-forming immediately (which would mean the
/// guard is inert). If this ever starts seeing a member before the window, the probe is not gating.
/// </summary>
[Fact]
public async Task Higher_node_does_not_form_before_the_probe_window_expires()
{
var low = FreePort();
var high = FreePort();
var higherPort = Math.Max(low, high);
var deadFounderPort = Math.Min(low, high);
var host = await StartGuardedNodeAsync("otopcua-guard-3", higherPort, deadFounderPort, probeSeconds: 10);
try
{
// Well within the 10 s probe window: still no cluster, because it is probing the dead founder.
(await WaitForUpMembersAsync(host, 1, TimeSpan.FromSeconds(3)))
.ShouldBeFalse("the higher node must probe for the founder, not self-form immediately");
}
finally { await StopAsync(host); }
}
/// <summary>
/// The higher node JOINS a live founder rather than forming a second cluster: start the founder,
/// then the higher node — its probe finds the founder reachable, it commits peer-first, and the
/// pair converges to ONE cluster of two.
/// </summary>
[Fact]
public async Task Higher_node_joins_a_live_founder()
{
const string systemName = "otopcua-guard-4";
var low = FreePort();
var high = FreePort();
var founderPort = Math.Min(low, high);
var higherPort = Math.Max(low, high);
var founder = await StartGuardedNodeAsync(systemName, founderPort, higherPort);
IHost? higher = null;
try
{
(await WaitForUpMembersAsync(founder, 1, TimeSpan.FromSeconds(30)))
.ShouldBeTrue("the founder must be Up before the higher node probes it");
higher = await StartGuardedNodeAsync(systemName, higherPort, founderPort);
(await WaitForUpMembersAsync(higher, 2, TimeSpan.FromSeconds(60)))
.ShouldBeTrue("the higher node must join the founder, forming one cluster of two");
(await WaitForUpMembersAsync(founder, 2, TimeSpan.FromSeconds(60)))
.ShouldBeTrue("the founder must see the higher node as a member of ITS cluster");
}
finally
{
if (higher is not null) await StopAsync(higher);
await StopAsync(founder);
}
}
}
@@ -0,0 +1,137 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Cluster;
namespace ZB.MOM.WW.OtOpcUa.Cluster.Tests;
/// <summary>
/// Unit tests for <see cref="ClusterBootstrapGuard"/> — the pure decision core of the
/// simultaneous-cold-start split-brain guard. The runtime probe + JoinSeedNodes wiring lives in
/// <c>ClusterBootstrapCoordinator</c> and is covered by the live gate.
/// </summary>
public sealed class ClusterBootstrapGuardTests
{
private const string A = "akka.tcp://otopcua@site-a-1:4053";
private const string B = "akka.tcp://otopcua@site-a-2:4053";
[Fact]
public void Lower_address_node_is_the_founder_and_needs_no_probe()
{
// site-a-1 < site-a-2, so on site-a-1 the guard makes it the founder.
var role = ClusterBootstrapGuard.Analyze(new[] { A, B }, "site-a-1", 4053);
role.Applies.ShouldBeTrue();
role.IsFounder.ShouldBeTrue();
role.SelfFirstOrder.ShouldBe(new[] { A, B });
}
[Fact]
public void Higher_address_node_is_not_the_founder_and_targets_the_partner_for_probing()
{
// On site-a-2, the partner is site-a-1 (the lower/founder).
var role = ClusterBootstrapGuard.Analyze(new[] { B, A }, "site-a-2", 4053);
role.Applies.ShouldBeTrue();
role.IsFounder.ShouldBeFalse();
role.PartnerHost.ShouldBe("site-a-1");
role.PartnerPort.ShouldBe(4053);
}
[Fact]
public void Tie_break_is_symmetric_regardless_of_seed_order()
{
// Both nodes must agree on who founds no matter how each lists its seeds.
var onLower = ClusterBootstrapGuard.Analyze(new[] { A, B }, "site-a-1", 4053);
var onHigher = ClusterBootstrapGuard.Analyze(new[] { B, A }, "site-a-2", 4053);
onLower.IsFounder.ShouldBeTrue();
onHigher.IsFounder.ShouldBeFalse();
// Exactly one founder.
(onLower.IsFounder ^ onHigher.IsFounder).ShouldBeTrue();
}
[Fact]
public void Higher_node_joins_the_founder_when_partner_is_reachable()
{
var role = ClusterBootstrapGuard.Analyze(new[] { B, A }, "site-a-2", 4053);
// Reachable ⇒ peer-first ⇒ JoinSeedNodeProcess ⇒ joins the founder, never self-forms.
ClusterBootstrapGuard.HigherNodeOrder(role, partnerReachable: true)
.ShouldBe(new[] { A, B }); // partner (site-a-1) first
}
[Fact]
public void Higher_node_forms_alone_when_partner_is_unreachable()
{
var role = ClusterBootstrapGuard.Analyze(new[] { B, A }, "site-a-2", 4053);
// Unreachable after the window ⇒ partner is dead ⇒ self-first ⇒ cold-start-alone preserved.
ClusterBootstrapGuard.HigherNodeOrder(role, partnerReachable: false)
.ShouldBe(new[] { B, A }); // self (site-a-2) first
}
[Fact]
public void Guard_does_not_apply_to_a_single_seed_node()
{
// A driver-only site node seeded solely by central-1 is not a pair seed — guard is inert.
var role = ClusterBootstrapGuard.Analyze(new[] { "akka.tcp://otopcua@central-1:4053" }, "site-x", 4053);
role.Applies.ShouldBeFalse();
}
[Fact]
public void Guard_does_not_apply_when_self_is_absent_from_the_two_seeds()
{
var role = ClusterBootstrapGuard.Analyze(new[] { A, B }, "central-1", 4053);
role.Applies.ShouldBeFalse();
}
[Fact]
public void Guard_does_not_apply_to_three_or_more_seeds()
{
var role = ClusterBootstrapGuard.Analyze(
new[] { A, B, "akka.tcp://otopcua@site-a-3:4053" }, "site-a-1", 4053);
role.Applies.ShouldBeFalse();
}
[Fact]
public void Guard_does_not_apply_when_a_seed_is_malformed()
{
var role = ClusterBootstrapGuard.Analyze(new[] { A, "not-a-seed" }, "site-a-1", 4053);
role.Applies.ShouldBeFalse();
}
[Theory]
[InlineData("akka.tcp://otopcua@site-a-1:4053", "site-a-1", 4053)]
[InlineData("akka.tcp://otopcua@10.0.0.5:4053/", "10.0.0.5", 4053)]
[InlineData(" akka.tcp://otopcua@host.lan:1234 ", "host.lan", 1234)]
public void TryParse_extracts_host_and_port(string seed, string expectedHost, int expectedPort)
{
ClusterBootstrapGuard.TryParse(seed, out var host, out var port).ShouldBeTrue();
host.ShouldBe(expectedHost);
port.ShouldBe(expectedPort);
}
[Theory]
[InlineData("")]
[InlineData("garbage")]
[InlineData("akka.tcp://otopcua@host")] // no port
public void TryParse_rejects_malformed_seeds(string seed)
{
ClusterBootstrapGuard.TryParse(seed, out _, out _).ShouldBeFalse();
}
[Fact]
public void Port_participates_in_the_tie_break_when_hosts_are_equal()
{
// Shared-host loopback pair (test rig): same host, different ports — port breaks the tie.
const string low = "akka.tcp://otopcua@127.0.0.1:4053";
const string high = "akka.tcp://otopcua@127.0.0.1:4054";
ClusterBootstrapGuard.Analyze(new[] { low, high }, "127.0.0.1", 4053).IsFounder.ShouldBeTrue();
ClusterBootstrapGuard.Analyze(new[] { high, low }, "127.0.0.1", 4054).IsFounder.ShouldBeFalse();
}
}