feat(dcl): failover on repeated unstable connections (connect-then-stale pattern)

Previously, failover only triggered when ConnectAsync failed consecutively.
If a connection succeeded but went stale quickly (e.g., heartbeat timeout),
the failure counter reset on each successful connect and failover never
triggered.

Added a separate _consecutiveUnstableDisconnects counter that increments
when a connection lasts less than StableConnectionThreshold (60s) before
disconnecting. When this counter reaches failoverRetryCount, the actor
fails over to the backup endpoint. Stable connections (lasting >60s)
reset this counter.

The original connection-failure failover path is unchanged.
This commit is contained in:
Joseph Doherty
2026-03-24 15:23:54 -04:00
parent ff2784b862
commit 5fdeaf613f
2 changed files with 64 additions and 3 deletions

View File

@@ -347,7 +347,7 @@ public class DataConnectionActorTests : TestKit
var count = Interlocked.Increment(ref connectCount);
// count 1: initial connect → success
// count 2,3: reconnect failures
// count 4: reconnect success (resets counter)
// count 4: reconnect success
// count 5,6: reconnect failures again
// count 7: reconnect success again
return count switch
@@ -366,7 +366,7 @@ public class DataConnectionActorTests : TestKit
AwaitCondition(() => connectCount >= 1, TimeSpan.FromSeconds(2));
await Task.Delay(200);
// Disconnect: triggers 2 failures then success (count 2,3,4)
// Disconnect: triggers 1 unstable disconnect + 2 failures then success (count 2,3,4)
RaiseDisconnected(primaryAdapter);
// Wait for successful reconnect (count 4)
@@ -380,7 +380,8 @@ public class DataConnectionActorTests : TestKit
AwaitCondition(() => connectCount >= 7, TimeSpan.FromSeconds(5));
await Task.Delay(200);
// Factory should never be called — counter reset each time before reaching 3
// Factory should never be called — connection failures counter resets on each
// successful reconnect, and unstable disconnect counter is separate
_mockFactory.DidNotReceive().Create(Arg.Any<string>(), Arg.Any<IDictionary<string, string>>());
}