fix(cluster): don't self-form while a seed peer is reachable — the fallback islanded a restarting node

The live gate for the manual-failover control caught the self-form fallback forming a SECOND cluster. A node bounced by a failover restarted, received InitJoinAck from its live peer — the join was in flight and healthy — but did not get the Welcome inside the 10s window, because the peer's ring still held the node's previous incarnation (Exiting -> Down -> Removed). The fallback fired on the timer and the node islanded itself until an operator restarted it. The original design assumed Cluster.Join(SelfAddress) would be ignored mid-handshake. It is not — it wins. And since manual failover deliberately produces that restart, every failover could island the node it bounced. Before self-forming, TCP-probe the other seed addresses; a reachable peer means wait another window instead. That is also what the fallback claims to detect: 'no seed answered InitJoin (peer down at boot)'. Claude-Session: https://claude.ai/code/session_01GASWkNEi68FSCtvr6rLoEW
2026-07-22 07:03:07 -04:00
parent d8a85c3d89
commit ea45ace1c3
3 changed files with 189 additions and 20 deletions
@@ -173,6 +173,60 @@ public sealed class SelfFormBootstrapTests
        }
    }

+    /// <summary>
+    /// Regression for the live-gate islanding defect (2026-07-22). A peer that is <b>reachable</b>
+    /// but not yet admitting the join must NOT be treated as "peer down at boot".
+    /// </summary>
+    /// <remarks>
+    ///     <para>
+    ///         On the docker-dev rig, a node bounced by a manual failover restarted, received
+    ///         <c>InitJoinAck</c> from its live peer — the join was in flight and healthy — but did
+    ///         not get the Welcome inside the window, because the peer's ring still held the node's
+    ///         previous incarnation. The fallback fired on the timer and the node formed a
+    ///         <i>second</i> cluster, islanded until an operator restarted it. The original design
+    ///         assumed <c>Join(SelfAddress)</c> would be ignored mid-handshake; it is not.
+    ///     </para>
+    ///     <para>
+    ///         The peer here is a bare <see cref="TcpListener"/> that accepts connections and speaks
+    ///         no Akka at all — reachable at the transport level, never answering a join. That is the
+    ///         defect's shape with nothing faked: under the pre-fix code this node self-formed on the
+    ///         timer; under the guard it keeps waiting. The positive control then proves the guard is
+    ///         a <i>guard</i> and not a disablement: drop the listener and the same node self-forms.
+    ///     </para>
+    /// </remarks>
+    [Fact]
+    public async Task Reachable_peer_suppresses_self_forming_until_it_goes_away()
+    {
+        var selfPort = FreePort();
+        var peerPort = FreePort();
+
+        // A peer that accepts TCP and does nothing else — reachable, but no join will ever complete.
+        var peer = new TcpListener(IPAddress.Loopback, peerPort);
+        peer.Start();
+
+        var host = await StartNodeAsync(
+            "otopcua-selfform-4", selfPort, peerPort, TimeSpan.FromSeconds(2));
+        try
+        {
+            var system = host.Services.GetRequiredService<ActorSystem>();
+
+            // Several windows pass. Pre-fix this self-formed after the first one.
+            (await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(10)))
+                .ShouldBeFalse("a reachable peer means a join may be in flight — self-forming here islands this node");
+
+            // Positive control: the peer really goes away, and now the fallback does its job.
+            peer.Stop();
+
+            (await WaitForUpMemberAsync(system, TimeSpan.FromSeconds(30)))
+                .ShouldBeTrue("once the peer is genuinely gone the fallback must still self-form");
+        }
+        finally
+        {
+            try { peer.Stop(); } catch (SocketException) { /* already stopped by the test body */ }
+            await StopAsync(host);
+        }
+    }
+
    /// <summary>
    /// THE island guard. A node that is not in its own seed list — the current docker-dev site-node
    /// topology, where site-a/site-b list only central-1 — must never self-form: it would island