fix(communication): P3-4 open-attempt deadline + P3-5 duplicate-subscribe cap order
P3-4 — SiteAlarmAggregatorActor pinned _openInFlight forever on a stream open that produced none of its three peer-driven outcomes. SiteStreamGrpcClient gives up on response headers after ConnectedHeaderTimeout (10s) and then treats the first received EVENT as its fallback connected signal, so an alarm-only site that stays quiet supplies neither — leaving _streamDown=true AND _openInFlight= true, which makes OnReconcileTick's reopen guard (_streamDown && !_openInFlight) skip every tick until the site's 4h max stream lifetime ends the call. Fix: bound one open attempt with a generation-fenced single-shot deadline (GrpcAlarmStreamOpenTimedOut, ctor-injected, production default 30s — above the 10s header timeout, below the 60s reconcile interval). Firing it releases the flag and nothing else: no retry-budget spend, no node flip, no liveness republish, and the abandoned subscription is left running so a merely-slow connect still comes up normally. Self-healing rather than reachability analysis. The deadline is cancelled on connect / error / graceful completion. P3-5 — SiteStreamGrpcServer cancelled the incumbent stream on a duplicate correlation id and only THEN checked the concurrency cap. Checking Count >= max after the removal is right at exactly the cap but wrong above it, and above it is reachable because the cap check is documented check-then-act: two subscribes that both clear it before either registers overshoot by one. In that window a routine reconnect killed a healthy stream AND was refused with ResourceExhausted, leaving the correlation with ZERO streams until its next retry. Fix: claim the removal, consult the cap only for a genuinely NEW correlation id, then cancel. A replacement is net-neutral on the count so it never consults the cap; a refused new subscriber cancelled nothing (TryRemove found nothing), so the incumbent survives. Newest-wins is unchanged — a fresh authenticated subscribe still displaces a dead/wedged incumbent. Tests (both shown failing against the unfixed code): - OpenAttemptThatNeverResolves_ReleasesTheInFlightGuard_SoTheTickCanReopen — fails (reopen never happens) with the deadline arming disabled. - OpenDeadline_DoesNotFire_OnAStreamThatConnects_NorFlipNodeOrBurnBudget — inertness lock on every healthy path. - AtCapOvershoot_DuplicateReconnect_IsNotRefused_AndLeavesExactlyOneLiveStream — fails with ResourceExhausted against the old ordering; the overshoot is driven deterministically by parking both openers on the ServerCallContext token read, which sits between the cap check and the _activeStreams insert. - AtCap_AGenuinelyNewSubscriber_IsStillRefused_AndTheIncumbentSurvives. The P3-4 test's doc comment carries the honesty note: the aggregator-side starvation reproduces exactly, but how often the >10s header delay that gets you there occurs in production is NOT demonstrated — the site flushes headers explicitly as soon as its relay is attached, so the budget has to be consumed by connect + handshake + server scheduling instead.
This commit is contained in:
Joseph Doherty
+99
-2
@@ -166,7 +166,8 @@ public class SiteAlarmAggregatorActorTests : TestKit
|
||||
}
|
||||
|
||||
private (IActorRef Actor, SeedStub Seed, PublishSink Sink, MockSiteAlarmStreamClientFactory Factory) CreateActor(
|
||||
TimeSpan? reconcileInterval = null, TimeSpan? publishCoalesce = null, bool autoConnect = true)
|
||||
TimeSpan? reconcileInterval = null, TimeSpan? publishCoalesce = null, bool autoConnect = true,
|
||||
TimeSpan? openAttemptTimeout = null)
|
||||
{
|
||||
var seed = new SeedStub();
|
||||
var sink = new PublishSink();
|
||||
@@ -178,7 +179,11 @@ public class SiteAlarmAggregatorActorTests : TestKit
|
||||
publishCoalesce ?? TimeSpan.Zero,
|
||||
TestReconnectDelay,
|
||||
TestStabilityWindow,
|
||||
0.0)); // no reconcile jitter in tests — determinism
|
||||
0.0, // no reconcile jitter in tests — determinism
|
||||
// Production default (30 s) unless a test overrides it: every other test in this
|
||||
// class finishes far inside it, so the open-attempt deadline stays dormant and
|
||||
// their behaviour is unchanged.
|
||||
openAttemptTimeout ?? SiteAlarmAggregatorActor.DefaultOpenAttemptTimeout));
|
||||
|
||||
var actor = Sys.ActorOf(props);
|
||||
return (actor, seed, sink, factory);
|
||||
@@ -849,6 +854,98 @@ public class SiteAlarmAggregatorActorTests : TestKit
|
||||
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3));
|
||||
}
|
||||
|
||||
// ── P3-4: the stream open attempt is deadline-bounded ───────────────────────
|
||||
|
||||
/// <summary>
|
||||
/// Proof of the P3-4 starvation and of its fix.
|
||||
/// <para>
|
||||
/// An open attempt is cleared by exactly three peer-produced outcomes: connected, error,
|
||||
/// or graceful completion. A stream that produces NONE of them pins <c>_openInFlight</c>.
|
||||
/// That is reachable: <c>SiteStreamGrpcClient</c> gives up on response headers after
|
||||
/// <c>ConnectedHeaderTimeout</c> (10 s) and then treats the FIRST RECEIVED EVENT as its
|
||||
/// fallback connected signal — which an alarm-only site that happens to be quiet never
|
||||
/// sends (proven independently by
|
||||
/// <c>SiteStreamGrpcClientTests.ConsumeStream_HeaderTimeoutOnADeadSite_NeverReportsConnected</c>).
|
||||
/// The aggregator then sits with <c>_streamDown == true</c> AND <c>_openInFlight == true</c>,
|
||||
/// and <see cref="SiteAlarmAggregatorActor.OnReconcileTick"/>'s reopen guard
|
||||
/// (<c>_streamDown && !_openInFlight</c>) is skipped every tick until the site's
|
||||
/// 4 h max stream lifetime finally ends the call.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Honesty note.</b> The mock reproduces the aggregator-side state machine exactly and
|
||||
/// deterministically (that is what <c>autoConnect: false</c> models). What it does NOT
|
||||
/// prove is how often the >10 s header delay that gets you there actually happens in
|
||||
/// production: the site flushes response headers explicitly as soon as its relay is
|
||||
/// attached (<c>SiteStreamGrpcServer.RunSubscriptionStreamAsync</c>), so the 10 s budget
|
||||
/// has to be consumed by channel connect + HTTP/2 handshake + server scheduling instead —
|
||||
/// plausible under a WAN stall or a GC/thread-pool-starved site node, but not something
|
||||
/// this suite can demonstrate. The deadline is therefore a self-healing guard rather than
|
||||
/// a fix for a measured production incident.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public void OpenAttemptThatNeverResolves_ReleasesTheInFlightGuard_SoTheTickCanReopen()
|
||||
{
|
||||
// 250 ms deadline; the reconcile timer is parked 10 minutes away so the ONLY tick is
|
||||
// the one this test issues by hand.
|
||||
var (actor, seed, sink, factory) = CreateActor(
|
||||
reconcileInterval: TimeSpan.FromMinutes(10),
|
||||
autoConnect: false,
|
||||
openAttemptTimeout: TimeSpan.FromMilliseconds(250));
|
||||
|
||||
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
||||
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
||||
seed.CompleteNext();
|
||||
|
||||
// The stream never reports connected, never faults, never completes — the quiet-site
|
||||
// header-timeout shape. Liveness is (correctly) false the whole time.
|
||||
AssertHoldsFor(TimeSpan.FromMilliseconds(300),
|
||||
(() => !sink.StreamLive, "a stream that never connected must never read as live"));
|
||||
|
||||
// Pre-fix this tick found _openInFlight still true and skipped the reopen, leaving
|
||||
// node A on one subscription forever. Post-fix the deadline has released the flag.
|
||||
actor.Tell(new RunReconcile());
|
||||
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 2, TimeSpan.FromSeconds(5));
|
||||
|
||||
// And the reopen is a real one: its connect brings the feed up and earns a re-seed.
|
||||
factory.ClientFor(GrpcNodeA).Subs[1].OnConnected!();
|
||||
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3));
|
||||
seed.CompleteNext();
|
||||
AwaitCondition(() => sink.StreamLive, TimeSpan.FromSeconds(3));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void OpenDeadline_DoesNotFire_OnAStreamThatConnects_NorFlipNodeOrBurnBudget()
|
||||
{
|
||||
// The deadline must be inert on every healthy path: a connect cancels it, so no
|
||||
// spurious reopen, no node flip, and no retry-budget spend can be attributed to it.
|
||||
// Timing-independent in the sense that matters — the assertions below would fail for
|
||||
// ANY firing of the timer, and the window is 8x the deadline.
|
||||
var (actor, seed, sink, factory) = CreateActor(
|
||||
reconcileInterval: TimeSpan.FromMinutes(10),
|
||||
autoConnect: true,
|
||||
openAttemptTimeout: TimeSpan.FromMilliseconds(100));
|
||||
|
||||
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
||||
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
||||
seed.CompleteNext();
|
||||
AwaitCondition(() => sink.StreamLive, TimeSpan.FromSeconds(3));
|
||||
|
||||
AssertHoldsFor(TimeSpan.FromMilliseconds(800),
|
||||
(() => sink.StreamLive, "a connected stream must stay live across the open deadline"),
|
||||
(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, "no second open on node A"),
|
||||
(() => factory.ClientFor(GrpcNodeB).Subs.Count == 0, "no node flip"),
|
||||
(() => seed.CallCount == 1, "no extra fan-out"));
|
||||
|
||||
// A reconcile tick on a live stream still must not reopen. (First tick is consumed by
|
||||
// the initial seed's skip flag; the second is the real fan-out.)
|
||||
actor.Tell(new RunReconcile());
|
||||
actor.Tell(new RunReconcile());
|
||||
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3)); // tick fan-out only
|
||||
seed.CompleteNext();
|
||||
Assert.Single(factory.ClientFor(GrpcNodeA).Subs);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void FailedSeed_IsRetried_WithBackoff_BeforeTheNextReconcile()
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user