fix(communication): P3-4 open-attempt deadline + P3-5 duplicate-subscribe cap order
P3-4 — SiteAlarmAggregatorActor pinned _openInFlight forever on a stream open that produced none of its three peer-driven outcomes. SiteStreamGrpcClient gives up on response headers after ConnectedHeaderTimeout (10s) and then treats the first received EVENT as its fallback connected signal, so an alarm-only site that stays quiet supplies neither — leaving _streamDown=true AND _openInFlight= true, which makes OnReconcileTick's reopen guard (_streamDown && !_openInFlight) skip every tick until the site's 4h max stream lifetime ends the call. Fix: bound one open attempt with a generation-fenced single-shot deadline (GrpcAlarmStreamOpenTimedOut, ctor-injected, production default 30s — above the 10s header timeout, below the 60s reconcile interval). Firing it releases the flag and nothing else: no retry-budget spend, no node flip, no liveness republish, and the abandoned subscription is left running so a merely-slow connect still comes up normally. Self-healing rather than reachability analysis. The deadline is cancelled on connect / error / graceful completion. P3-5 — SiteStreamGrpcServer cancelled the incumbent stream on a duplicate correlation id and only THEN checked the concurrency cap. Checking Count >= max after the removal is right at exactly the cap but wrong above it, and above it is reachable because the cap check is documented check-then-act: two subscribes that both clear it before either registers overshoot by one. In that window a routine reconnect killed a healthy stream AND was refused with ResourceExhausted, leaving the correlation with ZERO streams until its next retry. Fix: claim the removal, consult the cap only for a genuinely NEW correlation id, then cancel. A replacement is net-neutral on the count so it never consults the cap; a refused new subscriber cancelled nothing (TryRemove found nothing), so the incumbent survives. Newest-wins is unchanged — a fresh authenticated subscribe still displaces a dead/wedged incumbent. Tests (both shown failing against the unfixed code): - OpenAttemptThatNeverResolves_ReleasesTheInFlightGuard_SoTheTickCanReopen — fails (reopen never happens) with the deadline arming disabled. - OpenDeadline_DoesNotFire_OnAStreamThatConnects_NorFlipNodeOrBurnBudget — inertness lock on every healthy path. - AtCapOvershoot_DuplicateReconnect_IsNotRefused_AndLeavesExactlyOneLiveStream — fails with ResourceExhausted against the old ordering; the overshoot is driven deterministically by parking both openers on the ServerCallContext token read, which sits between the cap check and the _activeStreams insert. - AtCap_AGenuinelyNewSubscriber_IsStillRefused_AndTheIncumbentSurvives. The P3-4 test's doc comment carries the honesty note: the aggregator-side starvation reproduces exactly, but how often the >10s header delay that gets you there occurs in production is NOT demonstrated — the site flushes headers explicitly as soon as its relay is attached, so the budget has to be consumed by connect + handshake + server scheduling instead.
This commit is contained in:
@@ -353,8 +353,35 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
StatusCode.InvalidArgument, "correlation_id is missing or not a valid identifier"));
|
||||
}
|
||||
|
||||
// Duplicate prevention -- cancel existing stream for this correlationId.
|
||||
// Duplicate prevention -- REPLACE the existing stream for this correlationId.
|
||||
//
|
||||
// The removal is claimed FIRST, but nothing is cancelled until the concurrency cap
|
||||
// has had its say, and a REPLACEMENT never consults the cap at all (arch-review
|
||||
// 2026-08-19 P3-5). Both halves matter:
|
||||
//
|
||||
// * A replacement is net-neutral on the stream count (one entry out, one back in),
|
||||
// so refusing it can only ever destroy capacity, never protect it. The old code
|
||||
// checked `Count >= max` after the removal, which is correct at exactly the cap
|
||||
// but NOT above it — and above it is reachable, because the check below is
|
||||
// deliberately check-then-act (see the note there). In that window a routine
|
||||
// reconnect cancelled a healthy incumbent and was then refused with
|
||||
// ResourceExhausted, leaving the correlation with ZERO streams until its next
|
||||
// retry. Excluding the replacement from the cap removes that outcome entirely.
|
||||
// * A genuinely NEW subscriber that the cap refuses must leave the existing streams
|
||||
// untouched, which is automatic here: TryRemove found nothing, so nothing was
|
||||
// cancelled before the throw.
|
||||
//
|
||||
// Newest-wins is preserved exactly: a fresh subscribe on a live correlation id
|
||||
// always displaces the incumbent, whether that incumbent is healthy or wedged.
|
||||
var replacingExistingStream = _activeStreams.TryRemove(correlationId, out var existingEntry);
|
||||
|
||||
// Max concurrent streams, applied only to a genuinely new correlation id.
|
||||
// Deliberately check-then-act: two concurrent subscribes at the boundary can
|
||||
// both pass, overshooting the cap by at most (concurrent subscribers - 1).
|
||||
// Bounded and benign — not worth a lock on this path.
|
||||
if (!replacingExistingStream && _activeStreams.Count >= _maxConcurrentStreams)
|
||||
throw new RpcException(new GrpcStatus(StatusCode.ResourceExhausted, "Max concurrent streams reached"));
|
||||
|
||||
// CANCEL ONLY, never Dispose. The replaced stream's CTS belongs to its own
|
||||
// handler's `using var streamCts`, which is still running and still reads
|
||||
// `streamCts.Token` (at the pump call below, and previously at the
|
||||
@@ -365,11 +392,11 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
// pre-existing hazard (the same Dispose + the same first-token-read relationship
|
||||
// exist unchanged before R2). Cancellation alone is what replacement needs; the
|
||||
// owning handler's `using` still disposes it exactly once on every exit path.
|
||||
if (_activeStreams.TryRemove(correlationId, out var existingEntry))
|
||||
if (replacingExistingStream)
|
||||
{
|
||||
try
|
||||
{
|
||||
existingEntry.Cts.Cancel();
|
||||
existingEntry!.Cts.Cancel();
|
||||
}
|
||||
catch (ObjectDisposedException)
|
||||
{
|
||||
@@ -378,13 +405,6 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
}
|
||||
}
|
||||
|
||||
// Check max concurrent streams after duplicate removal.
|
||||
// Deliberately check-then-act: two concurrent subscribes at the boundary can
|
||||
// both pass, overshooting the cap by at most (concurrent subscribers - 1).
|
||||
// Bounded and benign — not worth a lock on this path.
|
||||
if (_activeStreams.Count >= _maxConcurrentStreams)
|
||||
throw new RpcException(new GrpcStatus(StatusCode.ResourceExhausted, "Max concurrent streams reached"));
|
||||
|
||||
using var streamCts = CancellationTokenSource.CreateLinkedTokenSource(context.CancellationToken);
|
||||
// Session timeout (design doc "gRPC Connection Keepalive": 4-hour third layer
|
||||
// of dead-client detection) — forces a long-lived zombie stream to terminate
|
||||
|
||||
Reference in New Issue
Block a user