fix(GWC-24): bound the worker event staging channel and unify the depth gauge
The GWC-04 remediation decoupled the read loop from event backpressure by
staging events into an unbounded channel, so its TryWrite always succeeded and
the only overflow fault was a single timed WriteAsync exceeding
EventChannelFullModeTimeout. A consumer draining slower than the worker
produces — each individual write still completing inside the window — therefore
grew gateway memory without bound, without a fault, and without a metric: the
queue-depth gauge counted only the bounded consumer channel, so staged events
were invisible.
Bound _eventStaging at 2 x EventChannelCapacity (Wait, single reader/writer, no
synchronous continuations). A rejected staging TryWrite is the sustained
slow-drain signal and faults the client ProtocolViolation with
QueueOverflow("worker-event-staging"), guarded by IsTerminalState() so a
completed channel during shutdown stays a silent drop. SetFaulted is
non-blocking, so the read loop still never awaits behind events. The timed-write
fault is unchanged and still catches the full-stall case earlier.
Move the queue-depth increment from EnqueueWorkerEventAsync to StageWorkerEvent
so the single counter reports total undelivered events (staged + queued); the
decrement at consumer read was already correct. No new configuration key: the
bound is derived, and gateway-side buffering per session is now at most
3 x MxGateway:Events:QueueCapacity. Coordination with still-open GWC-21
(EventChannelFullModeTimeout configurability) remains open and was not blocked
on.
Tests: StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout (5-min
full-mode timeout so only the staging bound can fire; asserts an interleaved
command reply still completes) and WorkerEventQueueDepthGaugeCountsStagedEvents.
Docs updated in the same change: GatewayProcessDesign, MxAccessWorkerInstanceDesign,
GatewayConfiguration, Metrics. GWC-24 flipped to Done in both trackers.
This commit is contained in:
@@ -149,17 +149,19 @@ All numeric session options must be greater than zero.
|
||||
|
||||
| Option | Default | Description |
|
||||
|--------|---------|-------------|
|
||||
| `MxGateway:Events:QueueCapacity` | `10000` | Capacity for bounded per-session event queues used by the gateway worker event channel and the public gRPC event stream queue. |
|
||||
| `MxGateway:Events:QueueCapacity` | `10000` | Capacity for bounded per-session event queues used by the gateway worker event channel and the public gRPC event stream queue. Gateway-side buffering per session is at most `3 ×` this value: the bounded worker event channel plus the read loop's staging channel, which is bounded at `2 ×` it. Overflow of either bound faults the session with `ProtocolViolation` and kills its worker. |
|
||||
| `MxGateway:Events:BackpressurePolicy` | `FailFast` | Per-subscriber event backpressure behavior when a subscriber's bounded event channel overflows. Overflow is isolated to the offending subscriber: it is always disconnected with an `EventQueueOverflow` fault while the session pump and other subscribers keep running. `FailFast` additionally faults the whole session only in the legacy single-subscriber case (the current default mode); with multiple subscribers it degrades to a per-subscriber disconnect so one slow consumer never faults a shared session. `DisconnectSubscriber` disconnects only the slow subscriber in all cases. |
|
||||
| `MxGateway:Events:ReplayBufferCapacity` | `1024` | Maximum number of events retained per session in the replay ring buffer, used to re-deliver events a returning subscriber missed (reconnect/reattach). The oldest retained event is evicted once this count is exceeded. `0` disables replay retention. |
|
||||
| `MxGateway:Events:ReplayRetentionSeconds` | `300` | Maximum age, in seconds, of an event retained in the replay ring buffer. Entries older than this are evicted regardless of capacity. `0` disables age-based eviction. |
|
||||
| `MxGateway:Events:MaxSparseArrayLength` | `1000000` | Maximum `total_length` a sparse-array write (`MxSparseArray`) may declare. A write above this cap is rejected with `InvalidArgument` before the full array is materialized, guarding against a single write forcing a multi-GB allocation. Must be between `1` and `Array.MaxLength`. |
|
||||
|
||||
`QueueCapacity` must be greater than zero; it bounds each per-subscriber event
|
||||
channel fed by the session's single event pump. A slow subscriber overflows only
|
||||
its own channel and is always disconnected with an `EventQueueOverflow` fault
|
||||
rather than silently dropping MXAccess events — the pump, the session, and other
|
||||
subscribers are unaffected. With `FailFast` in the single-subscriber case (the
|
||||
channel fed by the session's single event pump, and — at `2 ×` — the worker
|
||||
read loop's event staging channel, so a consumer that drains slower than its
|
||||
worker produces faults the session at a fixed ceiling instead of growing gateway
|
||||
memory (GWC-24). A slow subscriber overflows only its own channel and is always
|
||||
disconnected with an `EventQueueOverflow` fault rather than silently dropping
|
||||
MXAccess events — the pump, the session, and other subscribers are unaffected. With `FailFast` in the single-subscriber case (the
|
||||
default mode), that overflow additionally faults the whole session; with multiple
|
||||
subscribers `FailFast` degrades to a per-subscriber disconnect, matching
|
||||
`DisconnectSubscriber`, so one slow consumer cannot fault a session shared by
|
||||
|
||||
@@ -483,7 +483,7 @@ Internally it owns:
|
||||
- write loop,
|
||||
- event write loop,
|
||||
- outbound command/control channel serialized by the write loop,
|
||||
- unbounded event staging channel drained by the event write loop,
|
||||
- bounded event staging channel drained by the event write loop,
|
||||
- bounded inbound event channel,
|
||||
- pending command dictionary keyed by correlation id,
|
||||
- heartbeat monitor,
|
||||
@@ -513,16 +513,36 @@ If the pipe closes while the session is not closing, fault the session.
|
||||
The read loop never awaits event enqueue. Events are staged to the event write
|
||||
loop with a non-blocking write, so a full inbound event channel (a slow or
|
||||
absent `StreamEvents` consumer) cannot stall the read loop behind an event and
|
||||
delay a command reply or heartbeat (GWC-04). The bounded-channel backpressure
|
||||
window (`EventChannelFullModeTimeout`) and the sustained-overflow fault are
|
||||
applied by the event write loop, not the read loop.
|
||||
delay a command reply or heartbeat (GWC-04). The timed backpressure window
|
||||
(`EventChannelFullModeTimeout`) is applied by the event write loop, not the read
|
||||
loop; the read loop's only event-path fault is the staging-bound rejection
|
||||
described below, which uses the non-blocking `SetFaulted`.
|
||||
|
||||
### Event write loop
|
||||
### Event write loop and the two overflow faults
|
||||
|
||||
The event write loop drains the staging channel and performs the timed write
|
||||
into the bounded inbound event channel. When the inbound channel stays full past
|
||||
`EventChannelFullModeTimeout` it faults the session (`ProtocolViolation`) — the
|
||||
same overflow contract as before, moved off the read loop.
|
||||
The staging channel is bounded at `2 ×` the inbound event channel capacity, so
|
||||
gateway-side buffering per session is at most `3 × MxGateway:Events:QueueCapacity`
|
||||
— the fault fires as soon as *staging* is full, which is anywhere between `2 ×`
|
||||
and `3 ×` depending on how much the event writer has already drained.
|
||||
An unbounded staging channel would let a consumer that drains slower than the
|
||||
worker produces grow gateway memory without limit and without any fault or
|
||||
metric, because each individual timed write still completed inside the window
|
||||
(GWC-24). Two distinct faults now bound the event path, both `ProtocolViolation`
|
||||
and both killing the worker:
|
||||
|
||||
- **Full stall** — the event write loop's timed write into the bounded inbound
|
||||
channel stays blocked past `EventChannelFullModeTimeout`. Recorded as
|
||||
`QueueOverflow("worker-events")`. Catches a consumer that stopped entirely,
|
||||
earlier than the staging bound would.
|
||||
- **Sustained slow drain** — the read loop's staging `TryWrite` is rejected
|
||||
because staging is full at its `2 ×` bound, meaning the writer has been
|
||||
saturated for as long as the worker took to emit that many further events.
|
||||
Recorded as `QueueOverflow("worker-event-staging")`. A rejected `TryWrite`
|
||||
during shutdown (the staging channel is completed) stays a silent drop.
|
||||
|
||||
The worker event queue-depth gauge (`mxgateway.events.worker_queue.depth`) is
|
||||
incremented at staging and decremented at consumer read, so it reports total
|
||||
undelivered events across both channels rather than only the inbound channel.
|
||||
|
||||
### Write loop
|
||||
|
||||
|
||||
+3
-2
@@ -72,7 +72,7 @@ Observable gauges are pull-based; the `Meter` invokes the supplied callback when
|
||||
|------------|--------------|-------------|
|
||||
| `mxgateway.sessions.open` | `_openSessions` | Currently open sessions tracked by `SessionManager`. |
|
||||
| `mxgateway.workers.running` | `_workersRunning` | Worker clients in a running state. |
|
||||
| `mxgateway.events.worker_queue.depth` | `_workerEventQueueDepth` | Last reported depth of the worker-side event queue. |
|
||||
| `mxgateway.events.worker_queue.depth` | `_workerEventQueueDepth` | Undelivered worker events held by `WorkerClient` — staged *and* queued (GWC-24). Incremented when the read loop stages an event, decremented when the consumer reads it, so a backlog stuck in the staging channel is visible rather than invisible. |
|
||||
| `mxgateway.events.grpc_stream_queue.depth` | `_eventStreamBacklogSources` (summed on demand) | Live backlog buffered across every active `EventStreamService` subscriber, summed from the subscribers' channel `Count` at collection time. |
|
||||
|
||||
## Snapshot Shape
|
||||
@@ -146,8 +146,9 @@ _metrics.RemoveSessionEvents(session.SessionId);
|
||||
- `RecordWorkerStoppedOnce` calls `WorkerStopped(reason)` exactly once per worker, guarding against double-counting on simultaneous fault and exit signals.
|
||||
- `WorkerKilled(reason)` when the client forcibly terminates the worker.
|
||||
- `HeartbeatFailed(SessionId)` per missed heartbeat.
|
||||
- `SetWorkerEventQueueDepth(queueDepth)` after each event ingest.
|
||||
- `SetWorkerEventQueueDepth(queueDepth)` when the read loop stages an event and when the consumer reads one, so the gauge tracks staged + queued events.
|
||||
- `EventReceived(SessionId, workerEvent.Event.Family.ToString())` for each worker event.
|
||||
- `QueueOverflow("worker-events")` when the timed write into the bounded consumer channel exceeds `EventChannelFullModeTimeout`, and `QueueOverflow("worker-event-staging")` when the staging channel is full at its `2 × EventChannelCapacity` bound. The two labels distinguish a stalled consumer from one that merely drains too slowly; both fault the session with `ProtocolViolation`.
|
||||
|
||||
### Worker process launcher
|
||||
|
||||
|
||||
@@ -623,6 +623,18 @@ queue fills:
|
||||
Production coalescing may be added later, but it must be explicit and tested.
|
||||
Do not drop or coalesce events in v1.
|
||||
|
||||
The gateway side of the event path is bounded to match. `WorkerClient` buffers
|
||||
inbound events in a bounded consumer channel plus a staging channel bounded at
|
||||
`2 ×` that capacity, so a session holds at most three times
|
||||
`MxGateway:Events:QueueCapacity` undelivered events before it faults with
|
||||
`ProtocolViolation` and kills this worker. Two faults cover the two failure
|
||||
shapes — a consumer that stops entirely (the timed write past
|
||||
`EventChannelFullModeTimeout`, metric `QueueOverflow("worker-events")`) and a
|
||||
consumer that merely drains slower than this worker produces (the staging bound,
|
||||
metric `QueueOverflow("worker-event-staging")`). See
|
||||
`docs/GatewayProcessDesign.md`. A worker that outruns its consumer therefore
|
||||
dies loudly rather than growing gateway memory silently.
|
||||
|
||||
## Heartbeat And Watchdog
|
||||
|
||||
`WorkerPipeSession` starts the heartbeat loop after the gateway validates
|
||||
|
||||
Reference in New Issue
Block a user