fix(communication): P3-4 open-attempt deadline + P3-5 duplicate-subscribe cap order

P3-4 — SiteAlarmAggregatorActor pinned _openInFlight forever on a stream open
that produced none of its three peer-driven outcomes. SiteStreamGrpcClient gives
up on response headers after ConnectedHeaderTimeout (10s) and then treats the
first received EVENT as its fallback connected signal, so an alarm-only site
that stays quiet supplies neither — leaving _streamDown=true AND _openInFlight=
true, which makes OnReconcileTick's reopen guard (_streamDown && !_openInFlight)
skip every tick until the site's 4h max stream lifetime ends the call.

Fix: bound one open attempt with a generation-fenced single-shot deadline
(GrpcAlarmStreamOpenTimedOut, ctor-injected, production default 30s — above the
10s header timeout, below the 60s reconcile interval). Firing it releases the
flag and nothing else: no retry-budget spend, no node flip, no liveness
republish, and the abandoned subscription is left running so a merely-slow
connect still comes up normally. Self-healing rather than reachability analysis.
The deadline is cancelled on connect / error / graceful completion.

P3-5 — SiteStreamGrpcServer cancelled the incumbent stream on a duplicate
correlation id and only THEN checked the concurrency cap. Checking Count >= max
after the removal is right at exactly the cap but wrong above it, and above it
is reachable because the cap check is documented check-then-act: two subscribes
that both clear it before either registers overshoot by one. In that window a
routine reconnect killed a healthy stream AND was refused with
ResourceExhausted, leaving the correlation with ZERO streams until its next
retry.

Fix: claim the removal, consult the cap only for a genuinely NEW correlation id,
then cancel. A replacement is net-neutral on the count so it never consults the
cap; a refused new subscriber cancelled nothing (TryRemove found nothing), so
the incumbent survives. Newest-wins is unchanged — a fresh authenticated
subscribe still displaces a dead/wedged incumbent.

Tests (both shown failing against the unfixed code):
- OpenAttemptThatNeverResolves_ReleasesTheInFlightGuard_SoTheTickCanReopen —
  fails (reopen never happens) with the deadline arming disabled.
- OpenDeadline_DoesNotFire_OnAStreamThatConnects_NorFlipNodeOrBurnBudget —
  inertness lock on every healthy path.
- AtCapOvershoot_DuplicateReconnect_IsNotRefused_AndLeavesExactlyOneLiveStream —
  fails with ResourceExhausted against the old ordering; the overshoot is driven
  deterministically by parking both openers on the ServerCallContext token read,
  which sits between the cap check and the _activeStreams insert.
- AtCap_AGenuinelyNewSubscriber_IsStillRefused_AndTheIncumbentSurvives.

The P3-4 test's doc comment carries the honesty note: the aggregator-side
starvation reproduces exactly, but how often the >10s header delay that gets you
there occurs in production is NOT demonstrated — the site flushes headers
explicitly as soon as its relay is attached, so the budget has to be consumed by
connect + handshake + server scheduling instead.

src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs

@@ -65,6 +65,15 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
     private const string ReconcileTimerKey = "alarm-reconcile";
     private const string PublishTimerKey = "alarm-publish-coalesce";
     private const string SeedRetryTimerKey = "alarm-seed-retry";
+    private const string OpenDeadlineTimerKey = "alarm-grpc-open-deadline";
+
+    /// <summary>
+    /// Production default for the stream open-attempt deadline (arch-review 2026-08-19 P3-4).
+    /// Comfortably above <c>SiteStreamGrpcClient.ConnectedHeaderTimeout</c> (10 s) so a
+    /// normally-slow connect is never pre-empted, and comfortably below the 60 s reconcile
+    /// interval so the tick that follows always finds the flag released.
+    /// </summary>
+    internal static readonly TimeSpan DefaultOpenAttemptTimeout = TimeSpan.FromSeconds(30);
 
     /// <summary>True while a coalesced publish is armed (dirty deltas awaiting one tick). Actor-thread only.</summary>
     private bool _publishPending;
@@ -179,10 +188,38 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
     /// <summary>
     /// True between issuing a stream open and its first outcome (connected / error /
     /// completed), so a reconcile tick never stacks a second open on an in-flight one.
-    /// Actor-thread only.
+    /// Bounded by <see cref="_openAttemptTimeout"/> — see there for why an unbounded flag
+    /// was a starvation hazard. Actor-thread only.
     /// </summary>
     private bool _openInFlight;
 
+    /// <summary>
+    /// Deadline on ONE stream open attempt, after which <see cref="_openInFlight"/> is
+    /// released so <see cref="OnReconcileTick"/> can reopen (arch-review 2026-08-19 P3-4).
+    /// <para>
+    /// The three outcomes that clear the flag — connected, error, graceful completion — are
+    /// all events the peer has to produce. A stream that produces NONE of them pins the flag:
+    /// the client gives up on response headers after
+    /// <c>SiteStreamGrpcClient.ConnectedHeaderTimeout</c> (10 s) and then falls back to the
+    /// first received event as its connected signal, so an alarm-only site that is quiet
+    /// supplies neither. In that state <c>_streamDown</c> is still true (nothing ever
+    /// reported connected) but <c>_openInFlight</c> is also true, and the tick's reopen guard
+    /// — <c>_streamDown &amp;&amp; !_openInFlight</c> — is skipped until the site's 4 h max
+    /// stream lifetime finally ends the call. The deadline makes the flag self-healing
+    /// instead of relying on that analysis staying true.
+    /// </para>
+    /// <para>
+    /// Firing it is deliberately CHEAP: it releases the flag and nothing else. The retry
+    /// budget is untouched (this is not a fault), the node is not flipped, liveness is not
+    /// republished (<c>_streamDown</c> was never cleared), and the abandoned subscription is
+    /// left running — if it was merely slow and connects late, <c>GrpcAlarmStreamConnected</c>
+    /// still matches the current generation and brings it up normally; if a reconcile tick
+    /// reopens first, <see cref="OpenGrpcStream"/> cancels its token source and fences its
+    /// generation out.
+    /// </para>
+    /// </summary>
+    private readonly TimeSpan _openAttemptTimeout;
+
     /// <summary>Consecutive whole-fan-out failures, driving the seed-retry backoff. Actor-thread only.</summary>
     private int _consecutiveSeedFailures;
 
@@ -246,6 +283,11 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
     /// N per-site aggregators on one central node do not fan out in lockstep (production 0.2 =
     /// up to +20%). Zero disables jitter, which is what tests want for determinism.
     /// </param>
+    /// <param name="openAttemptTimeout">
+    /// Deadline on one gRPC stream open attempt (P3-4); <c>null</c> takes
+    /// <see cref="DefaultOpenAttemptTimeout"/> (30 s) and <see cref="TimeSpan.Zero"/> or less
+    /// disables the deadline. See <see cref="_openAttemptTimeout"/>.
+    /// </param>
     public SiteAlarmAggregatorActor(
         string siteIdentifier,
         string correlationId,
@@ -258,7 +300,8 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
         TimeSpan publishCoalesce,
         TimeSpan reconnectDelay,
         TimeSpan stabilityWindow,
-        double reconcileJitterFraction = 0.0)
+        double reconcileJitterFraction = 0.0,
+        TimeSpan? openAttemptTimeout = null)
     {
         _siteIdentifier = siteIdentifier;
         _correlationId = correlationId;
@@ -272,6 +315,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
         _reconnectDelay = reconnectDelay;
         _stabilityWindow = stabilityWindow;
         _reconcileJitterFraction = Math.Clamp(reconcileJitterFraction, 0.0, 1.0);
+        _openAttemptTimeout = openAttemptTimeout ?? DefaultOpenAttemptTimeout;
 
         // Live delta from the site-wide alarm stream (marshalled in via Self.Tell).
         // A received delta must NOT reset the retry budget (a flapping stream that
@@ -306,6 +350,23 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
             OnStreamConnected();
         });
 
+        // The open attempt produced none of its three outcomes inside the deadline (P3-4):
+        // release the in-flight flag so the reconcile tick's reopen guard can run again.
+        // Generation-fenced like every other stream callback, and a no-op once the attempt
+        // has already resolved (a late timer racing its own stream's connect).
+        Receive<GrpcAlarmStreamOpenTimedOut>(msg =>
+        {
+            if (_stopped) return;
+            if (msg.Generation != _streamGeneration) return;
+            if (!_openInFlight) return;
+
+            _openInFlight = false;
+            _log.Warning(
+                "Site-alarm gRPC stream open for {0} did not report connected, error or completion within {1}; " +
+                "releasing the in-flight guard so the next reconcile tick can reopen",
+                _siteIdentifier, _openAttemptTimeout);
+        });
+
         // Coalesced-publish tick: one publish for a batch of dirtying deltas (N6).
         Receive<PublishCoalesced>(_ =>
         {
@@ -799,6 +860,15 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
 
         _openInFlight = true;
         var generation = ++_streamGeneration;
+
+        // Bound this attempt (P3-4). Single-shot on a shared key, so a re-open replaces the
+        // previous attempt's deadline rather than stacking one.
+        if (_openAttemptTimeout > TimeSpan.Zero)
+        {
+            Timers.StartSingleTimer(
+                OpenDeadlineTimerKey, new GrpcAlarmStreamOpenTimedOut(generation), _openAttemptTimeout);
+        }
+
         var client = _grpcFactory.GetOrCreate(_siteIdentifier, endpoint);
         var self = Self;
         var ct = _grpcCts.Token;
@@ -834,6 +904,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
     /// </summary>
     private void OnStreamConnected()
     {
+        Timers.Cancel(OpenDeadlineTimerKey);
         _openInFlight = false;
 
         var wasDown = _streamDown;
@@ -858,6 +929,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
 
         // Stream failed before the stability window — retry budget NOT recovered.
         Timers.Cancel(StabilityTimerKey);
+        Timers.Cancel(OpenDeadlineTimerKey);
 
         _openInFlight = false;
         var wasLive = !_streamDown;
@@ -920,6 +992,7 @@ public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
         // The stream is gone, so its armed stability timer must not later "recover" a
         // budget that its successor has since spent.
         Timers.Cancel(StabilityTimerKey);
+        Timers.Cancel(OpenDeadlineTimerKey);
 
         _log.Info("Site-alarm gRPC stream for {0} completed gracefully (server end of stream); " +
                   "reopening on the next reconcile tick", _siteIdentifier);
@@ -1005,3 +1078,8 @@ internal sealed record ReconnectAlarmStream;
 
 /// <summary>Internal: the current site-alarm gRPC stream has stayed up long enough to recover the retry budget.</summary>
 internal sealed record GrpcAlarmStreamStable;
+
+/// <summary>Internal: a stream OPEN attempt produced none of connected/error/completed inside its
+/// deadline (P3-4), stamped with the generation it was armed for so a late timer from a superseded
+/// attempt is ignored. Releases the in-flight guard only — it is not a fault.</summary>
+internal sealed record GrpcAlarmStreamOpenTimedOut(int Generation);

src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs

@@ -353,8 +353,35 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
                 StatusCode.InvalidArgument, "correlation_id is missing or not a valid identifier"));
         }
 
-        // Duplicate prevention -- cancel existing stream for this correlationId.
+        // Duplicate prevention -- REPLACE the existing stream for this correlationId.
         //
+        // The removal is claimed FIRST, but nothing is cancelled until the concurrency cap
+        // has had its say, and a REPLACEMENT never consults the cap at all (arch-review
+        // 2026-08-19 P3-5). Both halves matter:
+        //
+        //  * A replacement is net-neutral on the stream count (one entry out, one back in),
+        //    so refusing it can only ever destroy capacity, never protect it. The old code
+        //    checked `Count >= max` after the removal, which is correct at exactly the cap
+        //    but NOT above it — and above it is reachable, because the check below is
+        //    deliberately check-then-act (see the note there). In that window a routine
+        //    reconnect cancelled a healthy incumbent and was then refused with
+        //    ResourceExhausted, leaving the correlation with ZERO streams until its next
+        //    retry. Excluding the replacement from the cap removes that outcome entirely.
+        //  * A genuinely NEW subscriber that the cap refuses must leave the existing streams
+        //    untouched, which is automatic here: TryRemove found nothing, so nothing was
+        //    cancelled before the throw.
+        //
+        // Newest-wins is preserved exactly: a fresh subscribe on a live correlation id
+        // always displaces the incumbent, whether that incumbent is healthy or wedged.
+        var replacingExistingStream = _activeStreams.TryRemove(correlationId, out var existingEntry);
+
+        // Max concurrent streams, applied only to a genuinely new correlation id.
+        // Deliberately check-then-act: two concurrent subscribes at the boundary can
+        // both pass, overshooting the cap by at most (concurrent subscribers - 1).
+        // Bounded and benign — not worth a lock on this path.
+        if (!replacingExistingStream && _activeStreams.Count >= _maxConcurrentStreams)
+            throw new RpcException(new GrpcStatus(StatusCode.ResourceExhausted, "Max concurrent streams reached"));
+
         // CANCEL ONLY, never Dispose. The replaced stream's CTS belongs to its own
         // handler's `using var streamCts`, which is still running and still reads
         // `streamCts.Token` (at the pump call below, and previously at the
@@ -365,11 +392,11 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
         // pre-existing hazard (the same Dispose + the same first-token-read relationship
         // exist unchanged before R2). Cancellation alone is what replacement needs; the
         // owning handler's `using` still disposes it exactly once on every exit path.
-        if (_activeStreams.TryRemove(correlationId, out var existingEntry))
+        if (replacingExistingStream)
         {
             try
             {
-                existingEntry.Cts.Cancel();
+                existingEntry!.Cts.Cancel();
             }
             catch (ObjectDisposedException)
             {
@@ -378,13 +405,6 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
             }
         }
 
-        // Check max concurrent streams after duplicate removal.
-        // Deliberately check-then-act: two concurrent subscribes at the boundary can
-        // both pass, overshooting the cap by at most (concurrent subscribers - 1).
-        // Bounded and benign — not worth a lock on this path.
-        if (_activeStreams.Count >= _maxConcurrentStreams)
-            throw new RpcException(new GrpcStatus(StatusCode.ResourceExhausted, "Max concurrent streams reached"));
-
         using var streamCts = CancellationTokenSource.CreateLinkedTokenSource(context.CancellationToken);
         // Session timeout (design doc "gRPC Connection Keepalive": 4-hour third layer
         // of dead-client detection) — forces a long-lived zombie stream to terminate

src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs

@@ -245,7 +245,11 @@ public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache
                     _options.LiveAlarmCachePublishCoalesce,
                     TimeSpan.FromSeconds(5),    // reconnect delay — former ReconnectDelay static default
                     TimeSpan.FromSeconds(60),   // stability window — former StabilityWindow static default
-                    _options.LiveAlarmCacheReconcileJitterFraction));
+                    _options.LiveAlarmCacheReconcileJitterFraction,
+                    // Open-attempt deadline (P3-4). Passed explicitly rather than defaulted:
+                    // Props.Create takes an expression tree, which may not omit an optional
+                    // argument (CS0854).
+                    SiteAlarmAggregatorActor.DefaultOpenAttemptTimeout));
 
                 entry.Actor = system.ActorOf(props, $"site-alarm-aggregator-{entry.SiteId}-{Guid.NewGuid():N}");