dohertj2/mxaccessgw
1
0
Fork 0
Code Issues 14 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

fix(GWC-24): bound the worker event staging channel and unify the depth gauge

Browse Source 
The GWC-04 remediation decoupled the read loop from event backpressure by
staging events into an unbounded channel, so its TryWrite always succeeded and
the only overflow fault was a single timed WriteAsync exceeding
EventChannelFullModeTimeout. A consumer draining slower than the worker
produces — each individual write still completing inside the window — therefore
grew gateway memory without bound, without a fault, and without a metric: the
queue-depth gauge counted only the bounded consumer channel, so staged events
were invisible.

Bound _eventStaging at 2 x EventChannelCapacity (Wait, single reader/writer, no
synchronous continuations). A rejected staging TryWrite is the sustained
slow-drain signal and faults the client ProtocolViolation with
QueueOverflow("worker-event-staging"), guarded by IsTerminalState() so a
completed channel during shutdown stays a silent drop. SetFaulted is
non-blocking, so the read loop still never awaits behind events. The timed-write
fault is unchanged and still catches the full-stall case earlier.

Move the queue-depth increment from EnqueueWorkerEventAsync to StageWorkerEvent
so the single counter reports total undelivered events (staged + queued); the
decrement at consumer read was already correct. No new configuration key: the
bound is derived, and gateway-side buffering per session is now at most
3 x MxGateway:Events:QueueCapacity. Coordination with still-open GWC-21
(EventChannelFullModeTimeout configurability) remains open and was not blocked
on.

Tests: StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout (5-min
full-mode timeout so only the staging bound can fire; asserts an interleaved
command reply still completes) and WorkerEventQueueDepthGaugeCountsStagedEvents.
Docs updated in the same change: GatewayProcessDesign, MxAccessWorkerInstanceDesign,
GatewayConfiguration, Metrics. GWC-24 flipped to Done in both trackers.
This commit is contained in:
Joseph Doherty
2026-08-07 05:35:07 -04:00
parent ead921cace
commit d4154e340c
8 changed files with 254 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerClient.cs
+57 -17
View File
@@ -27,10 +27,16 @@ public sealed class WorkerClient : IWorkerClient
// Staging hand-off between the read loop and the dedicated event writer. The read loop writes
// here with a non-blocking TryWrite so a full consumer channel (_events) can never stall the read
// loop behind an event — replies and heartbeats keep flowing. Unbounded, but only fills
// during the bounded EventChannelFullModeTimeout window before EventWriteLoopAsync faults on a
// sustained backlog, after which the read loop stops.
// loop behind an event — replies and heartbeats keep flowing. Bounded at 2 × EventChannelCapacity
// (GWC-24): an unbounded staging channel let a consumer that drains slower than the worker
// produces grow gateway memory without limit and without a fault, because each individual timed
// write into _events still completed inside EventChannelFullModeTimeout. A rejected TryWrite here
// is the sustained-slow-drain signal and faults the client immediately (ProtocolViolation), which
// is the fail-fast backpressure policy in docs/DesignDecisions.md. Total gateway-side buffering
// per session is therefore 3 × MxGateway:Events:QueueCapacity, and the single _eventQueueDepth
// gauge covers staged + queued events so the whole backlog is observable.
private readonly Channel<WorkerEvent> _eventStaging;
private readonly int _eventStagingCapacity;
private readonly ConcurrentDictionary<string, PendingCommand> _pendingCommands = new(StringComparer.Ordinal);
private readonly SemaphoreSlim _pendingCommandSlots;
private readonly CancellationTokenSource _stopCts = new();
@@ -90,12 +96,17 @@ public sealed class WorkerClient : IWorkerClient
FullMode = BoundedChannelFullMode.Wait,
AllowSynchronousContinuations = false,
});
_eventStaging = Channel.CreateUnbounded<WorkerEvent>(
new UnboundedChannelOptions
_eventStagingCapacity = checked(2 * _options.EventChannelCapacity);
_eventStaging = Channel.CreateBounded<WorkerEvent>(
new BoundedChannelOptions(_eventStagingCapacity)
{
// The read loop is the only writer; EventWriteLoopAsync is the only reader.
SingleReader = true,
SingleWriter = true,
// Wait (not Drop*) so the read loop's non-blocking TryWrite returns false exactly
// when the bound is reached — the same Wait+TryWrite overflow-detection idiom the
// session event distributor uses. The read loop never awaits this channel.
FullMode = BoundedChannelFullMode.Wait,
AllowSynchronousContinuations = false,
});
_lastHeartbeatAt = _timeProvider.GetUtcNow();
@@ -555,11 +566,16 @@ public sealed class WorkerClient : IWorkerClient
/// <summary>
/// Hands a received worker event to the dedicated event writer without blocking the read loop.
/// The staging channel is unbounded and this is the only writer, so <c>TryWrite</c> always
/// succeeds unless the channel has been completed during shutdown — in which case the event is
/// safely dropped because the client is closing. Backpressure and the sustained-overflow fault
/// are applied by <see cref="EventWriteLoopAsync"/> against the bounded consumer channel,
/// off the read loop.
/// <c>TryWrite</c> is non-blocking, so the read loop keeps dispatching replies, heartbeats and
/// faults regardless of how backed up the event path is. It returns <c>false</c> in two cases:
/// the staging channel has been completed during shutdown (the event is safely dropped because
/// the client is already terminal), or the channel is full at its
/// 2 × <see cref="WorkerClientOptions.EventChannelCapacity"/> bound. The latter means the
/// consumer has been draining slower than the worker produces for the whole time it took the
/// worker to emit that many further events, so the client is faulted immediately (GWC-24) —
/// <see cref="SetFaulted"/> is non-blocking, so the read loop still never awaits here. The
/// complementary full-stall case (a consumer that stops entirely) is caught earlier by the
/// timed write in <see cref="EnqueueWorkerEventAsync"/>.
/// </summary>
/// <param name="workerEvent">The event received from the worker.</param>
private void StageWorkerEvent(WorkerEvent workerEvent)
@@ -569,7 +585,30 @@ public sealed class WorkerClient : IWorkerClient
_metrics?.EventReceived(SessionId, workerEvent.Event.Family.ToString());
}
_eventStaging.Writer.TryWrite(workerEvent);
if (_eventStaging.Writer.TryWrite(workerEvent))
{
// Counted here rather than at the _events write so the single gauge reports total
// undelivered events (staged + queued). ReadEventsCoreAsync decrements on consumer read.
int queueDepth = Interlocked.Increment(ref _eventQueueDepth);
_metrics?.SetWorkerEventQueueDepth(queueDepth);
return;
}
if (IsTerminalState())
{
// Shutdown completed the staging channel; dropping the event is the documented behavior.
return;
}
_metrics?.QueueOverflow("worker-event-staging");
int depthAtOverflow = Volatile.Read(ref _eventQueueDepth);
SetFaulted(
WorkerClientErrorCode.ProtocolViolation,
$"Worker event staging channel is full at its {_eventStagingCapacity}-event bound "
+ $"(2 x EventChannelCapacity {_options.EventChannelCapacity}); undelivered depth is "
+ $"{depthAtOverflow}. The event consumer is draining slower than the worker produces. "
+ $"Attach or unblock the StreamEvents consumer or raise MxGateway:Events:QueueCapacity.",
null);
}
/// <summary>
@@ -603,7 +642,11 @@ public sealed class WorkerClient : IWorkerClient
/// missed slot even though the wait-mode channel would have absorbed
/// the burst. The diagnostic now names the capacity, current depth, and
/// the actionable fix (attach <c>StreamEvents</c> or raise
/// <c>MxGateway:Events:QueueCapacity</c>).
/// <c>MxGateway:Events:QueueCapacity</c>). This is the full-stall half of
/// the backpressure policy; a consumer that merely drains too slowly is
/// caught by the staging bound in <see cref="StageWorkerEvent"/> (GWC-24).
/// Queue depth is not counted here — the event was already counted when it
/// was staged, so moving it between the two channels changes nothing.
/// </summary>
/// <param name="workerEvent">The event to enqueue.</param>
/// <param name="cancellationToken">Cancellation token.</param>
@@ -613,8 +656,6 @@ public sealed class WorkerClient : IWorkerClient
{
if (_events.Writer.TryWrite(workerEvent))
{
int queueDepth = Interlocked.Increment(ref _eventQueueDepth);
_metrics?.SetWorkerEventQueueDepth(queueDepth);
return;
}
@@ -624,8 +665,6 @@ public sealed class WorkerClient : IWorkerClient
try
{
await _events.Writer.WriteAsync(workerEvent, fullModeCts.Token).ConfigureAwait(false);
int queueDepth = Interlocked.Increment(ref _eventQueueDepth);
_metrics?.SetWorkerEventQueueDepth(queueDepth);
return;
}
catch (OperationCanceledException) when (!cancellationToken.IsCancellationRequested)
@@ -641,7 +680,8 @@ public sealed class WorkerClient : IWorkerClient
WorkerClientErrorCode.ProtocolViolation,
$"Worker event channel rejected an event after waiting "
+ $"{_options.EventChannelFullModeTimeout.TotalMilliseconds:F0} ms; "
+ $"channel depth is {depthAtOverflow} of {_options.EventChannelCapacity} capacity. "
+ $"undelivered depth is {depthAtOverflow} against a consumer channel "
+ $"of {_options.EventChannelCapacity} capacity. "
+ $"Attach a StreamEvents consumer or raise MxGateway:Events:QueueCapacity.",
null);
}