fix(communication): P3-4 open-attempt deadline + P3-5 duplicate-subscribe cap order

P3-4 — SiteAlarmAggregatorActor pinned _openInFlight forever on a stream open
that produced none of its three peer-driven outcomes. SiteStreamGrpcClient gives
up on response headers after ConnectedHeaderTimeout (10s) and then treats the
first received EVENT as its fallback connected signal, so an alarm-only site
that stays quiet supplies neither — leaving _streamDown=true AND _openInFlight=
true, which makes OnReconcileTick's reopen guard (_streamDown && !_openInFlight)
skip every tick until the site's 4h max stream lifetime ends the call.

Fix: bound one open attempt with a generation-fenced single-shot deadline
(GrpcAlarmStreamOpenTimedOut, ctor-injected, production default 30s — above the
10s header timeout, below the 60s reconcile interval). Firing it releases the
flag and nothing else: no retry-budget spend, no node flip, no liveness
republish, and the abandoned subscription is left running so a merely-slow
connect still comes up normally. Self-healing rather than reachability analysis.
The deadline is cancelled on connect / error / graceful completion.

P3-5 — SiteStreamGrpcServer cancelled the incumbent stream on a duplicate
correlation id and only THEN checked the concurrency cap. Checking Count >= max
after the removal is right at exactly the cap but wrong above it, and above it
is reachable because the cap check is documented check-then-act: two subscribes
that both clear it before either registers overshoot by one. In that window a
routine reconnect killed a healthy stream AND was refused with
ResourceExhausted, leaving the correlation with ZERO streams until its next
retry.

Fix: claim the removal, consult the cap only for a genuinely NEW correlation id,
then cancel. A replacement is net-neutral on the count so it never consults the
cap; a refused new subscriber cancelled nothing (TryRemove found nothing), so
the incumbent survives. Newest-wins is unchanged — a fresh authenticated
subscribe still displaces a dead/wedged incumbent.

Tests (both shown failing against the unfixed code):
- OpenAttemptThatNeverResolves_ReleasesTheInFlightGuard_SoTheTickCanReopen —
  fails (reopen never happens) with the deadline arming disabled.
- OpenDeadline_DoesNotFire_OnAStreamThatConnects_NorFlipNodeOrBurnBudget —
  inertness lock on every healthy path.
- AtCapOvershoot_DuplicateReconnect_IsNotRefused_AndLeavesExactlyOneLiveStream —
  fails with ResourceExhausted against the old ordering; the overshoot is driven
  deterministically by parking both openers on the ServerCallContext token read,
  which sits between the cap check and the _activeStreams insert.
- AtCap_AGenuinelyNewSubscriber_IsStillRefused_AndTheIncumbentSurvives.

The P3-4 test's doc comment carries the honesty note: the aggregator-side
starvation reproduces exactly, but how often the >10s header delay that gets you
there occurs in production is NOT demonstrated — the site flushes headers
explicitly as soon as its relay is attached, so the budget has to be consumed by
connect + handshake + server scheduling instead.
This commit is contained in:
Joseph Doherty
2026-08-19 16:18:53 -04:00
parent 6823e7e3fa
commit 99da32d635
6 changed files with 368 additions and 16 deletions
@@ -224,7 +224,10 @@ public class SiteAlarmStreamEndToEndTests : TestKit
var aggregator = Sys.ActorOf(Props.Create(() => new SiteAlarmAggregatorActor(
"site-alpha", "e2e-parity", _ => Task.FromResult<IReadOnlyList<AlarmStateChanged>>(new[] { seedRow }),
sink.Publish, factory, "http://a:5100", "http://b:5100", TimeSpan.FromMinutes(10),
TimeSpan.Zero, TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(60))));
TimeSpan.Zero, TimeSpan.FromSeconds(5), TimeSpan.FromSeconds(60),
// Props.Create is an expression tree, which may not omit optional arguments
// (CS0854) — jitter and the P3-4 open-attempt deadline are passed explicitly.
0.0, SiteAlarmAggregatorActor.DefaultOpenAttemptTimeout)));
await WaitForConditionAsync(() => sink.Latest is { Count: 1 }); // seed applied
aggregator.Tell(liveRow);