fix(communication): P3-4 open-attempt deadline + P3-5 duplicate-subscribe cap order

P3-4 — SiteAlarmAggregatorActor pinned _openInFlight forever on a stream open
that produced none of its three peer-driven outcomes. SiteStreamGrpcClient gives
up on response headers after ConnectedHeaderTimeout (10s) and then treats the
first received EVENT as its fallback connected signal, so an alarm-only site
that stays quiet supplies neither — leaving _streamDown=true AND _openInFlight=
true, which makes OnReconcileTick's reopen guard (_streamDown && !_openInFlight)
skip every tick until the site's 4h max stream lifetime ends the call.

Fix: bound one open attempt with a generation-fenced single-shot deadline
(GrpcAlarmStreamOpenTimedOut, ctor-injected, production default 30s — above the
10s header timeout, below the 60s reconcile interval). Firing it releases the
flag and nothing else: no retry-budget spend, no node flip, no liveness
republish, and the abandoned subscription is left running so a merely-slow
connect still comes up normally. Self-healing rather than reachability analysis.
The deadline is cancelled on connect / error / graceful completion.

P3-5 — SiteStreamGrpcServer cancelled the incumbent stream on a duplicate
correlation id and only THEN checked the concurrency cap. Checking Count >= max
after the removal is right at exactly the cap but wrong above it, and above it
is reachable because the cap check is documented check-then-act: two subscribes
that both clear it before either registers overshoot by one. In that window a
routine reconnect killed a healthy stream AND was refused with
ResourceExhausted, leaving the correlation with ZERO streams until its next
retry.

Fix: claim the removal, consult the cap only for a genuinely NEW correlation id,
then cancel. A replacement is net-neutral on the count so it never consults the
cap; a refused new subscriber cancelled nothing (TryRemove found nothing), so
the incumbent survives. Newest-wins is unchanged — a fresh authenticated
subscribe still displaces a dead/wedged incumbent.

Tests (both shown failing against the unfixed code):
- OpenAttemptThatNeverResolves_ReleasesTheInFlightGuard_SoTheTickCanReopen —
  fails (reopen never happens) with the deadline arming disabled.
- OpenDeadline_DoesNotFire_OnAStreamThatConnects_NorFlipNodeOrBurnBudget —
  inertness lock on every healthy path.
- AtCapOvershoot_DuplicateReconnect_IsNotRefused_AndLeavesExactlyOneLiveStream —
  fails with ResourceExhausted against the old ordering; the overshoot is driven
  deterministically by parking both openers on the ServerCallContext token read,
  which sits between the cap check and the _activeStreams insert.
- AtCap_AGenuinelyNewSubscriber_IsStillRefused_AndTheIncumbentSurvives.

The P3-4 test's doc comment carries the honesty note: the aggregator-side
starvation reproduces exactly, but how often the >10s header delay that gets you
there occurs in production is NOT demonstrated — the site flushes headers
explicitly as soon as its relay is attached, so the budget has to be consumed by
connect + handshake + server scheduling instead.
This commit is contained in:
Joseph Doherty
2026-08-19 16:18:53 -04:00
parent 6823e7e3fa
commit 99da32d635
6 changed files with 368 additions and 16 deletions
@@ -353,8 +353,35 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
StatusCode.InvalidArgument, "correlation_id is missing or not a valid identifier"));
}
// Duplicate prevention -- cancel existing stream for this correlationId.
// Duplicate prevention -- REPLACE the existing stream for this correlationId.
//
// The removal is claimed FIRST, but nothing is cancelled until the concurrency cap
// has had its say, and a REPLACEMENT never consults the cap at all (arch-review
// 2026-08-19 P3-5). Both halves matter:
//
// * A replacement is net-neutral on the stream count (one entry out, one back in),
// so refusing it can only ever destroy capacity, never protect it. The old code
// checked `Count >= max` after the removal, which is correct at exactly the cap
// but NOT above it — and above it is reachable, because the check below is
// deliberately check-then-act (see the note there). In that window a routine
// reconnect cancelled a healthy incumbent and was then refused with
// ResourceExhausted, leaving the correlation with ZERO streams until its next
// retry. Excluding the replacement from the cap removes that outcome entirely.
// * A genuinely NEW subscriber that the cap refuses must leave the existing streams
// untouched, which is automatic here: TryRemove found nothing, so nothing was
// cancelled before the throw.
//
// Newest-wins is preserved exactly: a fresh subscribe on a live correlation id
// always displaces the incumbent, whether that incumbent is healthy or wedged.
var replacingExistingStream = _activeStreams.TryRemove(correlationId, out var existingEntry);
// Max concurrent streams, applied only to a genuinely new correlation id.
// Deliberately check-then-act: two concurrent subscribes at the boundary can
// both pass, overshooting the cap by at most (concurrent subscribers - 1).
// Bounded and benign — not worth a lock on this path.
if (!replacingExistingStream && _activeStreams.Count >= _maxConcurrentStreams)
throw new RpcException(new GrpcStatus(StatusCode.ResourceExhausted, "Max concurrent streams reached"));
// CANCEL ONLY, never Dispose. The replaced stream's CTS belongs to its own
// handler's `using var streamCts`, which is still running and still reads
// `streamCts.Token` (at the pump call below, and previously at the
@@ -365,11 +392,11 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
// pre-existing hazard (the same Dispose + the same first-token-read relationship
// exist unchanged before R2). Cancellation alone is what replacement needs; the
// owning handler's `using` still disposes it exactly once on every exit path.
if (_activeStreams.TryRemove(correlationId, out var existingEntry))
if (replacingExistingStream)
{
try
{
existingEntry.Cts.Cancel();
existingEntry!.Cts.Cancel();
}
catch (ObjectDisposedException)
{
@@ -378,13 +405,6 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
}
}
// Check max concurrent streams after duplicate removal.
// Deliberately check-then-act: two concurrent subscribes at the boundary can
// both pass, overshooting the cap by at most (concurrent subscribers - 1).
// Bounded and benign — not worth a lock on this path.
if (_activeStreams.Count >= _maxConcurrentStreams)
throw new RpcException(new GrpcStatus(StatusCode.ResourceExhausted, "Max concurrent streams reached"));
using var streamCts = CancellationTokenSource.CreateLinkedTokenSource(context.CancellationToken);
// Session timeout (design doc "gRPC Connection Keepalive": 4-hour third layer
// of dead-client detection) — forces a long-lived zombie stream to terminate