test(cluster): hard-kill failover integration test on the 2-node harness (arch-review 03/S1)

The load-bearing integration half of Critical 1. Adds TwoNodeClusterHarness.HardKillNodeAAsync (canonical in-process crash sim: shut down node A's remoting transport via IRemoteActorRefProvider.Transport.Shutdown() — associations drop, heartbeats/gossip stop, NO graceful Cluster.Leave, so node B's failure detector marks A Unreachable) + a WaitForNodeBSoleDriverLeaderAsync helper (waits for a STABLE takeover: B sole Up member + driver role-leader). HardKillFailoverTests crashes the oldest node and asserts the survivor takes over — the exact scenario the graceful StopNodeBAsync path (Cluster.Leave, no downing decision) cannot exercise. Live-verified in-process (~35s: acceptable-heartbeat-pause 10s + stable-after 15s + convergence). Negative control confirmed the test BITES: forcing explicit NoDowning (akka.cluster.downing-provider-class = "") makes it time out with the node stuck Up-but-Unreachable. FINDING surfaced by the negative control: removing Critical 1's typed ClusterOptions.SplitBrainResolver did NOT break failover, because Akka.Cluster.Hosting's WithClustering applies SplitBrainResolverOption.Default when the option is null — enabling the SBR downing provider that reads the akka.conf keep-oldest block (present on master BEFORE Critical 1). So the cluster was NOT running NoDowning before Critical 1; the typed option reinforces the akka.conf strategy rather than being the sole activator, and Critical 1's 'HOCON inert / NoDowning' premise is inaccurate. The test guards the failover OUTCOME regardless of activation path. Comments corrected to reflect this; flagged for review.
2026-07-08 18:24:49 -04:00
parent a81dea102c
commit a25c9ed097
2 changed files with 133 additions and 0 deletions
@@ -175,6 +175,74 @@ public sealed class TwoNodeClusterHarness : IAsyncDisposable
            formationTimeout ?? TimeSpan.FromSeconds(20));
    }

+    /// <summary>
+    /// Abruptly crashes node A (the seed / oldest member) by shutting down its remoting transport — the
+    /// canonical in-process crash simulation. All associations drop and heartbeats/gossip stop instantly
+    /// with NO graceful <c>Cluster.Leave</c> (the transport is dead, so A cannot gossip anything). Node B's
+    /// failure detector therefore marks A <b>Unreachable</b> (not Removed), which is the only state the
+    /// split-brain resolver acts on: with an active resolver (keep-oldest + down-if-alone, active here via
+    /// both the akka.conf block and Akka.Cluster.Hosting's default downing provider) B downs the alone
+    /// oldest node and takes over; with downing disabled (explicit NoDowning) B keeps A Unreachable forever
+    /// and never fails over. This is the crash the graceful <see cref="StopNodeBAsync"/> path (a real
+    /// <c>Cluster.Leave</c>, which removes a member with no downing decision) can NOT simulate.
+    /// The ActorSystem itself stays alive with a dead transport until harness teardown disposes it.
+    /// </summary>
+    public async Task HardKillNodeAAsync()
+    {
+        if (NodeA is null) return;
+        var provider = (Akka.Remote.IRemoteActorRefProvider)((ExtendedActorSystem)NodeASystem).Provider;
+        await provider.Transport.Shutdown();
+    }
+
+    /// <summary>
+    /// Waits until node B's own cluster view has the crashed node A removed (B is the sole Up member) AND
+    /// B has become the <c>driver</c>-role leader — i.e. the split-brain resolver downed the alone oldest
+    /// node and B took over. Queries <see cref="NodeBSystem"/> (node A is dead after a hard kill, so its
+    /// view can't be read). Returns the elapsed time to takeover; throws <see cref="TimeoutException"/> if
+    /// B never becomes the sole driver-leader within <paramref name="timeout"/> (the failure signature of
+    /// an INACTIVE resolver — A stays Unreachable and B never fails over).
+    /// </summary>
+    /// <param name="timeout">Upper bound; must exceed acceptable-heartbeat-pause (10s) + stable-after (15s).</param>
+    public async Task<TimeSpan> WaitForNodeBSoleDriverLeaderAsync(TimeSpan timeout)
+    {
+        var start = DateTime.UtcNow;
+        var deadline = start + timeout;
+        // Require the outcome to hold across several consecutive polls so a transient during A's
+        // Up -> Down convergence (the members list can briefly flap while gossip settles) can't be
+        // mistaken for a settled takeover.
+        const int requiredStableStreak = 4;
+        var streak = 0;
+        while (DateTime.UtcNow < deadline)
+        {
+            var clusterB = Akka.Cluster.Cluster.Get(NodeBSystem);
+            var state = clusterB.State;
+            var upCount = state.Members.Count(m => m.Status == MemberStatus.Up);
+            var driverLeader = state.RoleLeader("driver");
+            // SBR proof: the crashed oldest node has been DOWNED (no longer Up, so B is the sole Up
+            // member) and B has become the driver-role leader. With the resolver INACTIVE (NoDowning) A
+            // would instead stay Up-but-Unreachable — B would still see 2 Up members and never gain a
+            // stable role leader. (A lingers as a Down member in the Unreachable set until reaped; that
+            // reaping is not part of the failover contract, so it is NOT required here.)
+            if (upCount == 1 && driverLeader == clusterB.SelfAddress)
+            {
+                if (++streak >= requiredStableStreak)
+                    return DateTime.UtcNow - start;
+            }
+            else
+            {
+                streak = 0;
+            }
+            await Task.Delay(250);
+        }
+        var finalState = Akka.Cluster.Cluster.Get(NodeBSystem).State;
+        throw new TimeoutException(
+            $"Node B did not become the sole driver-role leader within {timeout} after node A was hard-killed. " +
+            $"Up={finalState.Members.Count(m => m.Status == MemberStatus.Up)}, " +
+            $"Unreachable={finalState.Unreachable.Count}, " +
+            $"driverLeader={finalState.RoleLeader("driver")}, selfB={Akka.Cluster.Cluster.Get(NodeBSystem).SelfAddress}. " +
+            "A stuck-Unreachable A with no takeover is the signature of an INACTIVE split-brain resolver.");
+    }
+
    /// <summary>
    /// Waits for node A's cluster view to reach <paramref name="expectedUpMembers"/> members in
    /// <see cref="MemberStatus.Up"/>. Used for asserting shrink-after-stop or grow-after-restart.