fix(GWC-24): bound the worker event staging channel and unify the depth gauge

The GWC-04 remediation decoupled the read loop from event backpressure by
staging events into an unbounded channel, so its TryWrite always succeeded and
the only overflow fault was a single timed WriteAsync exceeding
EventChannelFullModeTimeout. A consumer draining slower than the worker
produces — each individual write still completing inside the window — therefore
grew gateway memory without bound, without a fault, and without a metric: the
queue-depth gauge counted only the bounded consumer channel, so staged events
were invisible.

Bound _eventStaging at 2 x EventChannelCapacity (Wait, single reader/writer, no
synchronous continuations). A rejected staging TryWrite is the sustained
slow-drain signal and faults the client ProtocolViolation with
QueueOverflow("worker-event-staging"), guarded by IsTerminalState() so a
completed channel during shutdown stays a silent drop. SetFaulted is
non-blocking, so the read loop still never awaits behind events. The timed-write
fault is unchanged and still catches the full-stall case earlier.

Move the queue-depth increment from EnqueueWorkerEventAsync to StageWorkerEvent
so the single counter reports total undelivered events (staged + queued); the
decrement at consumer read was already correct. No new configuration key: the
bound is derived, and gateway-side buffering per session is now at most
3 x MxGateway:Events:QueueCapacity. Coordination with still-open GWC-21
(EventChannelFullModeTimeout configurability) remains open and was not blocked
on.

Tests: StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout (5-min
full-mode timeout so only the staging bound can fire; asserts an interleaved
command reply still completes) and WorkerEventQueueDepthGaugeCountsStagedEvents.
Docs updated in the same change: GatewayProcessDesign, MxAccessWorkerInstanceDesign,
GatewayConfiguration, Metrics. GWC-24 flipped to Done in both trackers.
This commit is contained in:
Joseph Doherty
2026-08-07 05:35:07 -04:00
parent ead921cace
commit d4154e340c
8 changed files with 254 additions and 37 deletions
+29 -9
View File
@@ -483,7 +483,7 @@ Internally it owns:
- write loop,
- event write loop,
- outbound command/control channel serialized by the write loop,
- unbounded event staging channel drained by the event write loop,
- bounded event staging channel drained by the event write loop,
- bounded inbound event channel,
- pending command dictionary keyed by correlation id,
- heartbeat monitor,
@@ -513,16 +513,36 @@ If the pipe closes while the session is not closing, fault the session.
The read loop never awaits event enqueue. Events are staged to the event write
loop with a non-blocking write, so a full inbound event channel (a slow or
absent `StreamEvents` consumer) cannot stall the read loop behind an event and
delay a command reply or heartbeat (GWC-04). The bounded-channel backpressure
window (`EventChannelFullModeTimeout`) and the sustained-overflow fault are
applied by the event write loop, not the read loop.
delay a command reply or heartbeat (GWC-04). The timed backpressure window
(`EventChannelFullModeTimeout`) is applied by the event write loop, not the read
loop; the read loop's only event-path fault is the staging-bound rejection
described below, which uses the non-blocking `SetFaulted`.
### Event write loop
### Event write loop and the two overflow faults
The event write loop drains the staging channel and performs the timed write
into the bounded inbound event channel. When the inbound channel stays full past
`EventChannelFullModeTimeout` it faults the session (`ProtocolViolation`) — the
same overflow contract as before, moved off the read loop.
The staging channel is bounded at `2 ×` the inbound event channel capacity, so
gateway-side buffering per session is at most `3 × MxGateway:Events:QueueCapacity`
— the fault fires as soon as *staging* is full, which is anywhere between `2 ×`
and `3 ×` depending on how much the event writer has already drained.
An unbounded staging channel would let a consumer that drains slower than the
worker produces grow gateway memory without limit and without any fault or
metric, because each individual timed write still completed inside the window
(GWC-24). Two distinct faults now bound the event path, both `ProtocolViolation`
and both killing the worker:
- **Full stall** — the event write loop's timed write into the bounded inbound
channel stays blocked past `EventChannelFullModeTimeout`. Recorded as
`QueueOverflow("worker-events")`. Catches a consumer that stopped entirely,
earlier than the staging bound would.
- **Sustained slow drain** — the read loop's staging `TryWrite` is rejected
because staging is full at its `2 ×` bound, meaning the writer has been
saturated for as long as the worker took to emit that many further events.
Recorded as `QueueOverflow("worker-event-staging")`. A rejected `TryWrite`
during shutdown (the staging channel is completed) stays a silent drop.
The worker event queue-depth gauge (`mxgateway.events.worker_queue.depth`) is
incremented at staging and decremented at consumer read, so it reports total
undelivered events across both channels rather than only the inbound channel.
### Write loop