fix(GWC-24): bound the worker event staging channel and unify the depth gauge
The GWC-04 remediation decoupled the read loop from event backpressure by
staging events into an unbounded channel, so its TryWrite always succeeded and
the only overflow fault was a single timed WriteAsync exceeding
EventChannelFullModeTimeout. A consumer draining slower than the worker
produces — each individual write still completing inside the window — therefore
grew gateway memory without bound, without a fault, and without a metric: the
queue-depth gauge counted only the bounded consumer channel, so staged events
were invisible.
Bound _eventStaging at 2 x EventChannelCapacity (Wait, single reader/writer, no
synchronous continuations). A rejected staging TryWrite is the sustained
slow-drain signal and faults the client ProtocolViolation with
QueueOverflow("worker-event-staging"), guarded by IsTerminalState() so a
completed channel during shutdown stays a silent drop. SetFaulted is
non-blocking, so the read loop still never awaits behind events. The timed-write
fault is unchanged and still catches the full-stall case earlier.
Move the queue-depth increment from EnqueueWorkerEventAsync to StageWorkerEvent
so the single counter reports total undelivered events (staged + queued); the
decrement at consumer read was already correct. No new configuration key: the
bound is derived, and gateway-side buffering per session is now at most
3 x MxGateway:Events:QueueCapacity. Coordination with still-open GWC-21
(EventChannelFullModeTimeout configurability) remains open and was not blocked
on.
Tests: StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout (5-min
full-mode timeout so only the staging bound can fire; asserts an interleaved
command reply still completes) and WorkerEventQueueDepthGaugeCountsStagedEvents.
Docs updated in the same change: GatewayProcessDesign, MxAccessWorkerInstanceDesign,
GatewayConfiguration, Metrics. GWC-24 flipped to Done in both trackers.
This commit is contained in:
@@ -623,6 +623,18 @@ queue fills:
|
||||
Production coalescing may be added later, but it must be explicit and tested.
|
||||
Do not drop or coalesce events in v1.
|
||||
|
||||
The gateway side of the event path is bounded to match. `WorkerClient` buffers
|
||||
inbound events in a bounded consumer channel plus a staging channel bounded at
|
||||
`2 ×` that capacity, so a session holds at most three times
|
||||
`MxGateway:Events:QueueCapacity` undelivered events before it faults with
|
||||
`ProtocolViolation` and kills this worker. Two faults cover the two failure
|
||||
shapes — a consumer that stops entirely (the timed write past
|
||||
`EventChannelFullModeTimeout`, metric `QueueOverflow("worker-events")`) and a
|
||||
consumer that merely drains slower than this worker produces (the staging bound,
|
||||
metric `QueueOverflow("worker-event-staging")`). See
|
||||
`docs/GatewayProcessDesign.md`. A worker that outruns its consumer therefore
|
||||
dies loudly rather than growing gateway memory silently.
|
||||
|
||||
## Heartbeat And Watchdog
|
||||
|
||||
`WorkerPipeSession` starts the heartbeat loop after the gateway validates
|
||||
|
||||
Reference in New Issue
Block a user