fix(GWC-24): bound the worker event staging channel and unify the depth gauge

The GWC-04 remediation decoupled the read loop from event backpressure by
staging events into an unbounded channel, so its TryWrite always succeeded and
the only overflow fault was a single timed WriteAsync exceeding
EventChannelFullModeTimeout. A consumer draining slower than the worker
produces — each individual write still completing inside the window — therefore
grew gateway memory without bound, without a fault, and without a metric: the
queue-depth gauge counted only the bounded consumer channel, so staged events
were invisible.

Bound _eventStaging at 2 x EventChannelCapacity (Wait, single reader/writer, no
synchronous continuations). A rejected staging TryWrite is the sustained
slow-drain signal and faults the client ProtocolViolation with
QueueOverflow("worker-event-staging"), guarded by IsTerminalState() so a
completed channel during shutdown stays a silent drop. SetFaulted is
non-blocking, so the read loop still never awaits behind events. The timed-write
fault is unchanged and still catches the full-stall case earlier.

Move the queue-depth increment from EnqueueWorkerEventAsync to StageWorkerEvent
so the single counter reports total undelivered events (staged + queued); the
decrement at consumer read was already correct. No new configuration key: the
bound is derived, and gateway-side buffering per session is now at most
3 x MxGateway:Events:QueueCapacity. Coordination with still-open GWC-21
(EventChannelFullModeTimeout configurability) remains open and was not blocked
on.

Tests: StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout (5-min
full-mode timeout so only the staging bound can fire; asserts an interleaved
command reply still completes) and WorkerEventQueueDepthGaugeCountsStagedEvents.
Docs updated in the same change: GatewayProcessDesign, MxAccessWorkerInstanceDesign,
GatewayConfiguration, Metrics. GWC-24 flipped to Done in both trackers.

src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerClient.cs

@@ -27,10 +27,16 @@ public sealed class WorkerClient : IWorkerClient
 
     // Staging hand-off between the read loop and the dedicated event writer. The read loop writes
     // here with a non-blocking TryWrite so a full consumer channel (_events) can never stall the read
-    // loop behind an event — replies and heartbeats keep flowing. Unbounded, but only fills
-    // during the bounded EventChannelFullModeTimeout window before EventWriteLoopAsync faults on a
-    // sustained backlog, after which the read loop stops.
+    // loop behind an event — replies and heartbeats keep flowing. Bounded at 2 × EventChannelCapacity
+    // (GWC-24): an unbounded staging channel let a consumer that drains slower than the worker
+    // produces grow gateway memory without limit and without a fault, because each individual timed
+    // write into _events still completed inside EventChannelFullModeTimeout. A rejected TryWrite here
+    // is the sustained-slow-drain signal and faults the client immediately (ProtocolViolation), which
+    // is the fail-fast backpressure policy in docs/DesignDecisions.md. Total gateway-side buffering
+    // per session is therefore 3 × MxGateway:Events:QueueCapacity, and the single _eventQueueDepth
+    // gauge covers staged + queued events so the whole backlog is observable.
     private readonly Channel<WorkerEvent> _eventStaging;
+    private readonly int _eventStagingCapacity;
     private readonly ConcurrentDictionary<string, PendingCommand> _pendingCommands = new(StringComparer.Ordinal);
     private readonly SemaphoreSlim _pendingCommandSlots;
     private readonly CancellationTokenSource _stopCts = new();
@@ -90,12 +96,17 @@ public sealed class WorkerClient : IWorkerClient
                 FullMode = BoundedChannelFullMode.Wait,
                 AllowSynchronousContinuations = false,
             });
-        _eventStaging = Channel.CreateUnbounded<WorkerEvent>(
-            new UnboundedChannelOptions
+        _eventStagingCapacity = checked(2 * _options.EventChannelCapacity);
+        _eventStaging = Channel.CreateBounded<WorkerEvent>(
+            new BoundedChannelOptions(_eventStagingCapacity)
             {
                 // The read loop is the only writer; EventWriteLoopAsync is the only reader.
                 SingleReader = true,
                 SingleWriter = true,
+                // Wait (not Drop*) so the read loop's non-blocking TryWrite returns false exactly
+                // when the bound is reached — the same Wait+TryWrite overflow-detection idiom the
+                // session event distributor uses. The read loop never awaits this channel.
+                FullMode = BoundedChannelFullMode.Wait,
                 AllowSynchronousContinuations = false,
             });
         _lastHeartbeatAt = _timeProvider.GetUtcNow();
@@ -555,11 +566,16 @@ public sealed class WorkerClient : IWorkerClient
 
     /// <summary>
     ///     Hands a received worker event to the dedicated event writer without blocking the read loop.
-    ///     The staging channel is unbounded and this is the only writer, so <c>TryWrite</c> always
-    ///     succeeds unless the channel has been completed during shutdown — in which case the event is
-    ///     safely dropped because the client is closing. Backpressure and the sustained-overflow fault
-    ///     are applied by <see cref="EventWriteLoopAsync"/> against the bounded consumer channel,
-    ///     off the read loop.
+    ///     <c>TryWrite</c> is non-blocking, so the read loop keeps dispatching replies, heartbeats and
+    ///     faults regardless of how backed up the event path is. It returns <c>false</c> in two cases:
+    ///     the staging channel has been completed during shutdown (the event is safely dropped because
+    ///     the client is already terminal), or the channel is full at its
+    ///     2 × <see cref="WorkerClientOptions.EventChannelCapacity"/> bound. The latter means the
+    ///     consumer has been draining slower than the worker produces for the whole time it took the
+    ///     worker to emit that many further events, so the client is faulted immediately (GWC-24) —
+    ///     <see cref="SetFaulted"/> is non-blocking, so the read loop still never awaits here. The
+    ///     complementary full-stall case (a consumer that stops entirely) is caught earlier by the
+    ///     timed write in <see cref="EnqueueWorkerEventAsync"/>.
     /// </summary>
     /// <param name="workerEvent">The event received from the worker.</param>
     private void StageWorkerEvent(WorkerEvent workerEvent)
@@ -569,7 +585,30 @@ public sealed class WorkerClient : IWorkerClient
             _metrics?.EventReceived(SessionId, workerEvent.Event.Family.ToString());
         }
 
-        _eventStaging.Writer.TryWrite(workerEvent);
+        if (_eventStaging.Writer.TryWrite(workerEvent))
+        {
+            // Counted here rather than at the _events write so the single gauge reports total
+            // undelivered events (staged + queued). ReadEventsCoreAsync decrements on consumer read.
+            int queueDepth = Interlocked.Increment(ref _eventQueueDepth);
+            _metrics?.SetWorkerEventQueueDepth(queueDepth);
+            return;
+        }
+
+        if (IsTerminalState())
+        {
+            // Shutdown completed the staging channel; dropping the event is the documented behavior.
+            return;
+        }
+
+        _metrics?.QueueOverflow("worker-event-staging");
+        int depthAtOverflow = Volatile.Read(ref _eventQueueDepth);
+        SetFaulted(
+            WorkerClientErrorCode.ProtocolViolation,
+            $"Worker event staging channel is full at its {_eventStagingCapacity}-event bound "
+                + $"(2 x EventChannelCapacity {_options.EventChannelCapacity}); undelivered depth is "
+                + $"{depthAtOverflow}. The event consumer is draining slower than the worker produces. "
+                + $"Attach or unblock the StreamEvents consumer or raise MxGateway:Events:QueueCapacity.",
+            null);
     }
 
     /// <summary>
@@ -603,7 +642,11 @@ public sealed class WorkerClient : IWorkerClient
     ///     missed slot even though the wait-mode channel would have absorbed
     ///     the burst. The diagnostic now names the capacity, current depth, and
     ///     the actionable fix (attach <c>StreamEvents</c> or raise
-    ///     <c>MxGateway:Events:QueueCapacity</c>).
+    ///     <c>MxGateway:Events:QueueCapacity</c>). This is the full-stall half of
+    ///     the backpressure policy; a consumer that merely drains too slowly is
+    ///     caught by the staging bound in <see cref="StageWorkerEvent"/> (GWC-24).
+    ///     Queue depth is not counted here — the event was already counted when it
+    ///     was staged, so moving it between the two channels changes nothing.
     /// </summary>
     /// <param name="workerEvent">The event to enqueue.</param>
     /// <param name="cancellationToken">Cancellation token.</param>
@@ -613,8 +656,6 @@ public sealed class WorkerClient : IWorkerClient
     {
         if (_events.Writer.TryWrite(workerEvent))
         {
-            int queueDepth = Interlocked.Increment(ref _eventQueueDepth);
-            _metrics?.SetWorkerEventQueueDepth(queueDepth);
             return;
         }
 
@@ -624,8 +665,6 @@ public sealed class WorkerClient : IWorkerClient
         try
         {
             await _events.Writer.WriteAsync(workerEvent, fullModeCts.Token).ConfigureAwait(false);
-            int queueDepth = Interlocked.Increment(ref _eventQueueDepth);
-            _metrics?.SetWorkerEventQueueDepth(queueDepth);
             return;
         }
         catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested)
@@ -641,7 +680,8 @@ public sealed class WorkerClient : IWorkerClient
             WorkerClientErrorCode.ProtocolViolation,
             $"Worker event channel rejected an event after waiting "
                 + $"{_options.EventChannelFullModeTimeout.TotalMilliseconds:F0} ms; "
-                + $"channel depth is {depthAtOverflow} of {_options.EventChannelCapacity} capacity. "
+                + $"undelivered depth is {depthAtOverflow} against a consumer channel "
+                + $"of {_options.EventChannelCapacity} capacity. "
                 + $"Attach a StreamEvents consumer or raise MxGateway:Events:QueueCapacity.",
             null);
     }

src/ZB.MOM.WW.MxGateway.Tests/Gateway/Workers/WorkerClientTests.cs

@@ -650,6 +650,147 @@ public sealed class WorkerClientTests
         Assert.Contains("MxGateway:Events:QueueCapacity", fault.Message);
     }
 
+    /// <summary>
+    ///     GWC-24: the staging channel between the read loop and the event writer is bounded at
+    ///     2 × <see cref="WorkerClientOptions.EventChannelCapacity"/>, so a consumer that drains
+    ///     slower than the worker produces faults the client at the bound instead of growing
+    ///     gateway memory silently. The full-mode timeout here is far longer than the test could
+    ///     ever wait, which proves the fault came from the staging bound and not from the timed
+    ///     write in <c>EnqueueWorkerEventAsync</c>. A command reply interleaved before the fault
+    ///     must still complete: the read loop never blocks behind events (the GWC-04 guarantee).
+    /// </summary>
+    /// <returns>A task that represents the asynchronous operation.</returns>
+    [Fact]
+    public async Task StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout()
+    {
+        const int capacity = 4;
+        const int stagingBound = 2 * capacity;
+        using GatewayMetrics metrics = new();
+        await using PipePair pipePair = await PipePair.CreateAsync();
+        await using WorkerClient client = CreateClient(
+            pipePair,
+            new WorkerClientOptions
+            {
+                EventChannelCapacity = capacity,
+                // Five minutes: the timed-write fault physically cannot be the trigger.
+                EventChannelFullModeTimeout = TimeSpan.FromMinutes(5),
+                HeartbeatGrace = TimeSpan.FromSeconds(30),
+                HeartbeatCheckInterval = TimeSpan.FromSeconds(30),
+            },
+            metrics: metrics);
+        await CompleteHandshakeAsync(client, pipePair);
+
+        // The staging channel alone holds 2 × capacity, and the event writer only ever removes
+        // from it, so this batch cannot fault however the writer happens to be scheduled. Waiting
+        // on the gauge (rather than a delay) proves the read loop consumed every one of them.
+        ulong sequence = 1;
+        for (; sequence <= (ulong)stagingBound; sequence++)
+        {
+            await pipePair.WorkerWriter.WriteAsync(
+                CreateEventEnvelope(sequence, MxEventFamily.OnDataChange));
+        }
+
+        await WaitUntilAsync(
+            () => metrics.GetSnapshot().WorkerEventQueueDepth == stagingBound,
+            TestTimeout);
+        Assert.Equal(WorkerClientState.Ready, client.State);
+
+        // The event path is backed up with no consumer attached, yet a command still round-trips.
+        Task<WorkerCommandReply> invokeTask = client.InvokeAsync(
+            CreateCommand(MxCommandKind.Ping),
+            TestTimeout,
+            CancellationToken.None);
+        WorkerEnvelope commandEnvelope = await pipePair.WorkerReader.ReadAsync().AsTask().WaitAsync(TestTimeout);
+        await pipePair.WorkerWriter.WriteAsync(
+            CreateCommandReplyEnvelope(commandEnvelope.CorrelationId, MxCommandKind.Ping));
+        WorkerCommandReply reply = await invokeTask.WaitAsync(TestTimeout);
+        Assert.Equal(MxCommandKind.Ping, reply.Reply.Kind);
+        Assert.Equal(WorkerClientState.Ready, client.State);
+
+        // The absolute ceiling is capacity (consumer channel) + 1 (in flight in the blocked event
+        // writer) + 2 × capacity (staging). Push comfortably past it. The events are tiny, so the
+        // ones the stopped read loop never drains stay in the OS pipe buffer instead of blocking.
+        for (int extra = 0; extra < 3 * capacity; extra++, sequence++)
+        {
+            await pipePair.WorkerWriter.WriteAsync(
+                CreateEventEnvelope(sequence, MxEventFamily.OnDataChange));
+        }
+
+        await WaitUntilAsync(() => client.State == WorkerClientState.Faulted, TestTimeout);
+        Assert.Equal(WorkerClientState.Faulted, client.State);
+
+        using CancellationTokenSource drainTimeout = new(TestTimeout);
+        WorkerClientException fault = await Assert.ThrowsAsync<WorkerClientException>(async () =>
+        {
+            await foreach (WorkerEvent _ in client.ReadEventsAsync(drainTimeout.Token))
+            {
+            }
+        });
+
+        Assert.Equal(WorkerClientErrorCode.ProtocolViolation, fault.ErrorCode);
+        Assert.Contains("staging", fault.Message, StringComparison.OrdinalIgnoreCase);
+        Assert.Contains($"{2 * capacity}", fault.Message, StringComparison.Ordinal);
+        Assert.Contains("StreamEvents", fault.Message, StringComparison.Ordinal);
+        Assert.Contains("MxGateway:Events:QueueCapacity", fault.Message, StringComparison.Ordinal);
+
+        // The timed-write diagnostic must not be what fired.
+        Assert.DoesNotContain("Worker event channel rejected", fault.Message, StringComparison.Ordinal);
+    }
+
+    /// <summary>
+    ///     GWC-24: the worker event queue-depth gauge counts staged *and* queued events, so a
+    ///     backlog held in the staging channel is visible rather than invisible. Depth is
+    ///     incremented at staging and decremented when the consumer reads, so the single counter
+    ///     reports total undelivered events and returns to zero once drained.
+    /// </summary>
+    /// <returns>A task that represents the asynchronous operation.</returns>
+    [Fact]
+    public async Task WorkerEventQueueDepthGaugeCountsStagedEvents()
+    {
+        const int capacity = 4;
+        const int eventCount = 8;
+        using GatewayMetrics metrics = new();
+        await using PipePair pipePair = await PipePair.CreateAsync();
+        await using WorkerClient client = CreateClient(
+            pipePair,
+            new WorkerClientOptions
+            {
+                EventChannelCapacity = capacity,
+                EventChannelFullModeTimeout = TimeSpan.FromMinutes(5),
+                HeartbeatGrace = TimeSpan.FromSeconds(30),
+                HeartbeatCheckInterval = TimeSpan.FromSeconds(30),
+            },
+            metrics: metrics);
+        await CompleteHandshakeAsync(client, pipePair);
+
+        // Above EventChannelCapacity but below the 2× staging bound: no consumer, no fault.
+        for (ulong sequence = 1; sequence <= eventCount; sequence++)
+        {
+            await pipePair.WorkerWriter.WriteAsync(
+                CreateEventEnvelope(sequence, MxEventFamily.OnDataChange));
+        }
+
+        await WaitUntilAsync(
+            () => metrics.GetSnapshot().WorkerEventQueueDepth == eventCount,
+            TestTimeout);
+        Assert.Equal(eventCount, metrics.GetSnapshot().WorkerEventQueueDepth);
+        Assert.Equal(WorkerClientState.Ready, client.State);
+
+        using CancellationTokenSource drainTimeout = new(TestTimeout);
+        await using IAsyncEnumerator<WorkerEvent> events = client
+            .ReadEventsAsync(drainTimeout.Token)
+            .GetAsyncEnumerator(drainTimeout.Token);
+        for (int read = 0; read < eventCount; read++)
+        {
+            Assert.True(await events.MoveNextAsync());
+        }
+
+        await WaitUntilAsync(
+            () => metrics.GetSnapshot().WorkerEventQueueDepth == 0,
+            TestTimeout);
+        Assert.Equal(0, metrics.GetSnapshot().WorkerEventQueueDepth);
+    }
+
     private static WorkerClient CreateClient(
         PipePair pipePair,
         WorkerClientOptions? options = null,