fix(GWC-24): bound the worker event staging channel and unify the depth gauge
The GWC-04 remediation decoupled the read loop from event backpressure by
staging events into an unbounded channel, so its TryWrite always succeeded and
the only overflow fault was a single timed WriteAsync exceeding
EventChannelFullModeTimeout. A consumer draining slower than the worker
produces — each individual write still completing inside the window — therefore
grew gateway memory without bound, without a fault, and without a metric: the
queue-depth gauge counted only the bounded consumer channel, so staged events
were invisible.
Bound _eventStaging at 2 x EventChannelCapacity (Wait, single reader/writer, no
synchronous continuations). A rejected staging TryWrite is the sustained
slow-drain signal and faults the client ProtocolViolation with
QueueOverflow("worker-event-staging"), guarded by IsTerminalState() so a
completed channel during shutdown stays a silent drop. SetFaulted is
non-blocking, so the read loop still never awaits behind events. The timed-write
fault is unchanged and still catches the full-stall case earlier.
Move the queue-depth increment from EnqueueWorkerEventAsync to StageWorkerEvent
so the single counter reports total undelivered events (staged + queued); the
decrement at consumer read was already correct. No new configuration key: the
bound is derived, and gateway-side buffering per session is now at most
3 x MxGateway:Events:QueueCapacity. Coordination with still-open GWC-21
(EventChannelFullModeTimeout configurability) remains open and was not blocked
on.
Tests: StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout (5-min
full-mode timeout so only the staging bound can fire; asserts an interleaved
command reply still completes) and WorkerEventQueueDepthGaugeCountsStagedEvents.
Docs updated in the same change: GatewayProcessDesign, MxAccessWorkerInstanceDesign,
GatewayConfiguration, Metrics. GWC-24 flipped to Done in both trackers.
This commit is contained in:
@@ -483,7 +483,7 @@ Internally it owns:
|
||||
- write loop,
|
||||
- event write loop,
|
||||
- outbound command/control channel serialized by the write loop,
|
||||
- unbounded event staging channel drained by the event write loop,
|
||||
- bounded event staging channel drained by the event write loop,
|
||||
- bounded inbound event channel,
|
||||
- pending command dictionary keyed by correlation id,
|
||||
- heartbeat monitor,
|
||||
@@ -513,16 +513,36 @@ If the pipe closes while the session is not closing, fault the session.
|
||||
The read loop never awaits event enqueue. Events are staged to the event write
|
||||
loop with a non-blocking write, so a full inbound event channel (a slow or
|
||||
absent `StreamEvents` consumer) cannot stall the read loop behind an event and
|
||||
delay a command reply or heartbeat (GWC-04). The bounded-channel backpressure
|
||||
window (`EventChannelFullModeTimeout`) and the sustained-overflow fault are
|
||||
applied by the event write loop, not the read loop.
|
||||
delay a command reply or heartbeat (GWC-04). The timed backpressure window
|
||||
(`EventChannelFullModeTimeout`) is applied by the event write loop, not the read
|
||||
loop; the read loop's only event-path fault is the staging-bound rejection
|
||||
described below, which uses the non-blocking `SetFaulted`.
|
||||
|
||||
### Event write loop
|
||||
### Event write loop and the two overflow faults
|
||||
|
||||
The event write loop drains the staging channel and performs the timed write
|
||||
into the bounded inbound event channel. When the inbound channel stays full past
|
||||
`EventChannelFullModeTimeout` it faults the session (`ProtocolViolation`) — the
|
||||
same overflow contract as before, moved off the read loop.
|
||||
The staging channel is bounded at `2 ×` the inbound event channel capacity, so
|
||||
gateway-side buffering per session is at most `3 × MxGateway:Events:QueueCapacity`
|
||||
— the fault fires as soon as *staging* is full, which is anywhere between `2 ×`
|
||||
and `3 ×` depending on how much the event writer has already drained.
|
||||
An unbounded staging channel would let a consumer that drains slower than the
|
||||
worker produces grow gateway memory without limit and without any fault or
|
||||
metric, because each individual timed write still completed inside the window
|
||||
(GWC-24). Two distinct faults now bound the event path, both `ProtocolViolation`
|
||||
and both killing the worker:
|
||||
|
||||
- **Full stall** — the event write loop's timed write into the bounded inbound
|
||||
channel stays blocked past `EventChannelFullModeTimeout`. Recorded as
|
||||
`QueueOverflow("worker-events")`. Catches a consumer that stopped entirely,
|
||||
earlier than the staging bound would.
|
||||
- **Sustained slow drain** — the read loop's staging `TryWrite` is rejected
|
||||
because staging is full at its `2 ×` bound, meaning the writer has been
|
||||
saturated for as long as the worker took to emit that many further events.
|
||||
Recorded as `QueueOverflow("worker-event-staging")`. A rejected `TryWrite`
|
||||
during shutdown (the staging channel is completed) stays a silent drop.
|
||||
|
||||
The worker event queue-depth gauge (`mxgateway.events.worker_queue.depth`) is
|
||||
incremented at staging and decremented at consumer read, so it reports total
|
||||
undelivered events across both channels rather than only the inbound channel.
|
||||
|
||||
### Write loop
|
||||
|
||||
|
||||
Reference in New Issue
Block a user