fix(GWC-24): bound the worker event staging channel and unify the depth gauge

The GWC-04 remediation decoupled the read loop from event backpressure by
staging events into an unbounded channel, so its TryWrite always succeeded and
the only overflow fault was a single timed WriteAsync exceeding
EventChannelFullModeTimeout. A consumer draining slower than the worker
produces — each individual write still completing inside the window — therefore
grew gateway memory without bound, without a fault, and without a metric: the
queue-depth gauge counted only the bounded consumer channel, so staged events
were invisible.

Bound _eventStaging at 2 x EventChannelCapacity (Wait, single reader/writer, no
synchronous continuations). A rejected staging TryWrite is the sustained
slow-drain signal and faults the client ProtocolViolation with
QueueOverflow("worker-event-staging"), guarded by IsTerminalState() so a
completed channel during shutdown stays a silent drop. SetFaulted is
non-blocking, so the read loop still never awaits behind events. The timed-write
fault is unchanged and still catches the full-stall case earlier.

Move the queue-depth increment from EnqueueWorkerEventAsync to StageWorkerEvent
so the single counter reports total undelivered events (staged + queued); the
decrement at consumer read was already correct. No new configuration key: the
bound is derived, and gateway-side buffering per session is now at most
3 x MxGateway:Events:QueueCapacity. Coordination with still-open GWC-21
(EventChannelFullModeTimeout configurability) remains open and was not blocked
on.

Tests: StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout (5-min
full-mode timeout so only the staging bound can fire; asserts an interleaved
command reply still completes) and WorkerEventQueueDepthGaugeCountsStagedEvents.
Docs updated in the same change: GatewayProcessDesign, MxAccessWorkerInstanceDesign,
GatewayConfiguration, Metrics. GWC-24 flipped to Done in both trackers.
This commit is contained in:
Joseph Doherty
2026-08-07 05:35:07 -04:00
parent ead921cace
commit d4154e340c
8 changed files with 254 additions and 37 deletions
+7 -5
View File
@@ -149,17 +149,19 @@ All numeric session options must be greater than zero.
| Option | Default | Description |
|--------|---------|-------------|
| `MxGateway:Events:QueueCapacity` | `10000` | Capacity for bounded per-session event queues used by the gateway worker event channel and the public gRPC event stream queue. |
| `MxGateway:Events:QueueCapacity` | `10000` | Capacity for bounded per-session event queues used by the gateway worker event channel and the public gRPC event stream queue. Gateway-side buffering per session is at most `3 ×` this value: the bounded worker event channel plus the read loop's staging channel, which is bounded at `2 ×` it. Overflow of either bound faults the session with `ProtocolViolation` and kills its worker. |
| `MxGateway:Events:BackpressurePolicy` | `FailFast` | Per-subscriber event backpressure behavior when a subscriber's bounded event channel overflows. Overflow is isolated to the offending subscriber: it is always disconnected with an `EventQueueOverflow` fault while the session pump and other subscribers keep running. `FailFast` additionally faults the whole session only in the legacy single-subscriber case (the current default mode); with multiple subscribers it degrades to a per-subscriber disconnect so one slow consumer never faults a shared session. `DisconnectSubscriber` disconnects only the slow subscriber in all cases. |
| `MxGateway:Events:ReplayBufferCapacity` | `1024` | Maximum number of events retained per session in the replay ring buffer, used to re-deliver events a returning subscriber missed (reconnect/reattach). The oldest retained event is evicted once this count is exceeded. `0` disables replay retention. |
| `MxGateway:Events:ReplayRetentionSeconds` | `300` | Maximum age, in seconds, of an event retained in the replay ring buffer. Entries older than this are evicted regardless of capacity. `0` disables age-based eviction. |
| `MxGateway:Events:MaxSparseArrayLength` | `1000000` | Maximum `total_length` a sparse-array write (`MxSparseArray`) may declare. A write above this cap is rejected with `InvalidArgument` before the full array is materialized, guarding against a single write forcing a multi-GB allocation. Must be between `1` and `Array.MaxLength`. |
`QueueCapacity` must be greater than zero; it bounds each per-subscriber event
channel fed by the session's single event pump. A slow subscriber overflows only
its own channel and is always disconnected with an `EventQueueOverflow` fault
rather than silently dropping MXAccess events — the pump, the session, and other
subscribers are unaffected. With `FailFast` in the single-subscriber case (the
channel fed by the session's single event pump, and — at `2 ×` — the worker
read loop's event staging channel, so a consumer that drains slower than its
worker produces faults the session at a fixed ceiling instead of growing gateway
memory (GWC-24). A slow subscriber overflows only its own channel and is always
disconnected with an `EventQueueOverflow` fault rather than silently dropping
MXAccess events — the pump, the session, and other subscribers are unaffected. With `FailFast` in the single-subscriber case (the
default mode), that overflow additionally faults the whole session; with multiple
subscribers `FailFast` degrades to a per-subscriber disconnect, matching
`DisconnectSubscriber`, so one slow consumer cannot fault a session shared by
+29 -9
View File
@@ -483,7 +483,7 @@ Internally it owns:
- write loop,
- event write loop,
- outbound command/control channel serialized by the write loop,
- unbounded event staging channel drained by the event write loop,
- bounded event staging channel drained by the event write loop,
- bounded inbound event channel,
- pending command dictionary keyed by correlation id,
- heartbeat monitor,
@@ -513,16 +513,36 @@ If the pipe closes while the session is not closing, fault the session.
The read loop never awaits event enqueue. Events are staged to the event write
loop with a non-blocking write, so a full inbound event channel (a slow or
absent `StreamEvents` consumer) cannot stall the read loop behind an event and
delay a command reply or heartbeat (GWC-04). The bounded-channel backpressure
window (`EventChannelFullModeTimeout`) and the sustained-overflow fault are
applied by the event write loop, not the read loop.
delay a command reply or heartbeat (GWC-04). The timed backpressure window
(`EventChannelFullModeTimeout`) is applied by the event write loop, not the read
loop; the read loop's only event-path fault is the staging-bound rejection
described below, which uses the non-blocking `SetFaulted`.
### Event write loop
### Event write loop and the two overflow faults
The event write loop drains the staging channel and performs the timed write
into the bounded inbound event channel. When the inbound channel stays full past
`EventChannelFullModeTimeout` it faults the session (`ProtocolViolation`) — the
same overflow contract as before, moved off the read loop.
The staging channel is bounded at `2 ×` the inbound event channel capacity, so
gateway-side buffering per session is at most `3 × MxGateway:Events:QueueCapacity`
— the fault fires as soon as *staging* is full, which is anywhere between `2 ×`
and `3 ×` depending on how much the event writer has already drained.
An unbounded staging channel would let a consumer that drains slower than the
worker produces grow gateway memory without limit and without any fault or
metric, because each individual timed write still completed inside the window
(GWC-24). Two distinct faults now bound the event path, both `ProtocolViolation`
and both killing the worker:
- **Full stall** — the event write loop's timed write into the bounded inbound
channel stays blocked past `EventChannelFullModeTimeout`. Recorded as
`QueueOverflow("worker-events")`. Catches a consumer that stopped entirely,
earlier than the staging bound would.
- **Sustained slow drain** — the read loop's staging `TryWrite` is rejected
because staging is full at its `2 ×` bound, meaning the writer has been
saturated for as long as the worker took to emit that many further events.
Recorded as `QueueOverflow("worker-event-staging")`. A rejected `TryWrite`
during shutdown (the staging channel is completed) stays a silent drop.
The worker event queue-depth gauge (`mxgateway.events.worker_queue.depth`) is
incremented at staging and decremented at consumer read, so it reports total
undelivered events across both channels rather than only the inbound channel.
### Write loop
+3 -2
View File
@@ -72,7 +72,7 @@ Observable gauges are pull-based; the `Meter` invokes the supplied callback when
|------------|--------------|-------------|
| `mxgateway.sessions.open` | `_openSessions` | Currently open sessions tracked by `SessionManager`. |
| `mxgateway.workers.running` | `_workersRunning` | Worker clients in a running state. |
| `mxgateway.events.worker_queue.depth` | `_workerEventQueueDepth` | Last reported depth of the worker-side event queue. |
| `mxgateway.events.worker_queue.depth` | `_workerEventQueueDepth` | Undelivered worker events held by `WorkerClient` — staged *and* queued (GWC-24). Incremented when the read loop stages an event, decremented when the consumer reads it, so a backlog stuck in the staging channel is visible rather than invisible. |
| `mxgateway.events.grpc_stream_queue.depth` | `_eventStreamBacklogSources` (summed on demand) | Live backlog buffered across every active `EventStreamService` subscriber, summed from the subscribers' channel `Count` at collection time. |
## Snapshot Shape
@@ -146,8 +146,9 @@ _metrics.RemoveSessionEvents(session.SessionId);
- `RecordWorkerStoppedOnce` calls `WorkerStopped(reason)` exactly once per worker, guarding against double-counting on simultaneous fault and exit signals.
- `WorkerKilled(reason)` when the client forcibly terminates the worker.
- `HeartbeatFailed(SessionId)` per missed heartbeat.
- `SetWorkerEventQueueDepth(queueDepth)` after each event ingest.
- `SetWorkerEventQueueDepth(queueDepth)` when the read loop stages an event and when the consumer reads one, so the gauge tracks staged + queued events.
- `EventReceived(SessionId, workerEvent.Event.Family.ToString())` for each worker event.
- `QueueOverflow("worker-events")` when the timed write into the bounded consumer channel exceeds `EventChannelFullModeTimeout`, and `QueueOverflow("worker-event-staging")` when the staging channel is full at its `2 × EventChannelCapacity` bound. The two labels distinguish a stalled consumer from one that merely drains too slowly; both fault the session with `ProtocolViolation`.
### Worker process launcher
+12
View File
@@ -623,6 +623,18 @@ queue fills:
Production coalescing may be added later, but it must be explicit and tested.
Do not drop or coalesce events in v1.
The gateway side of the event path is bounded to match. `WorkerClient` buffers
inbound events in a bounded consumer channel plus a staging channel bounded at
`2 ×` that capacity, so a session holds at most three times
`MxGateway:Events:QueueCapacity` undelivered events before it faults with
`ProtocolViolation` and kills this worker. Two faults cover the two failure
shapes — a consumer that stops entirely (the timed write past
`EventChannelFullModeTimeout`, metric `QueueOverflow("worker-events")`) and a
consumer that merely drains slower than this worker produces (the staging bound,
metric `QueueOverflow("worker-event-staging")`). See
`docs/GatewayProcessDesign.md`. A worker that outruns its consumer therefore
dies loudly rather than growing gateway memory silently.
## Heartbeat And Watchdog
`WorkerPipeSession` starts the heartbeat loop after the gateway validates