dohertj2/mxaccessgw
1
0
Fork 0
Code Issues 14 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

fix(GWC-24): bound the worker event staging channel and unify the depth gauge

Browse Source 
The GWC-04 remediation decoupled the read loop from event backpressure by
staging events into an unbounded channel, so its TryWrite always succeeded and
the only overflow fault was a single timed WriteAsync exceeding
EventChannelFullModeTimeout. A consumer draining slower than the worker
produces — each individual write still completing inside the window — therefore
grew gateway memory without bound, without a fault, and without a metric: the
queue-depth gauge counted only the bounded consumer channel, so staged events
were invisible.

Bound _eventStaging at 2 x EventChannelCapacity (Wait, single reader/writer, no
synchronous continuations). A rejected staging TryWrite is the sustained
slow-drain signal and faults the client ProtocolViolation with
QueueOverflow("worker-event-staging"), guarded by IsTerminalState() so a
completed channel during shutdown stays a silent drop. SetFaulted is
non-blocking, so the read loop still never awaits behind events. The timed-write
fault is unchanged and still catches the full-stall case earlier.

Move the queue-depth increment from EnqueueWorkerEventAsync to StageWorkerEvent
so the single counter reports total undelivered events (staged + queued); the
decrement at consumer read was already correct. No new configuration key: the
bound is derived, and gateway-side buffering per session is now at most
3 x MxGateway:Events:QueueCapacity. Coordination with still-open GWC-21
(EventChannelFullModeTimeout configurability) remains open and was not blocked
on.

Tests: StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout (5-min
full-mode timeout so only the staging bound can fire; asserts an interleaved
command reply still completes) and WorkerEventQueueDepthGaugeCountsStagedEvents.
Docs updated in the same change: GatewayProcessDesign, MxAccessWorkerInstanceDesign,
GatewayConfiguration, Metrics. GWC-24 flipped to Done in both trackers.
This commit is contained in:
Joseph Doherty
2026-08-07 05:35:07 -04:00
parent ead921cace
commit d4154e340c
8 changed files with 254 additions and 37 deletions
Download Patch File Download Diff File Expand all files Collapse all files
archreview/2026-07-12/remediation/10-gateway-core.md
+2 -2
View File
@@ -8,7 +8,7 @@ This document turns the 2026-07-12 re-review's **new** Gateway Server Core findi
| ID | Sev | Tier | Eff | Dep | Status | Title |
|----|-----|------|-----|-----|--------|-------|
| GWC-24 | Medium | P1 | M | GWC-21 (coord) | Not started | Unbounded event staging channel: sustained slow drain grows memory silently and invisibly |
| GWC-24 | Medium | P1 | M | GWC-21 (coord) | Done | Unbounded event staging channel: sustained slow drain grows memory silently and invisibly |
| GWC-25 | Medium | P0 | S | CLI-35/36 (coord) | Not started | Empty-ring ReplayGap sentinel carries `oldest_available_sequence = 0`, dead-streaming a compliant client |
| GWC-26 | Low | P2 | M | GWC-27 | Not started | Alarm monitor attaches its subscriber after SubscribeAlarms; window transitions bypass the feed, missed Acknowledge never repaired |
| GWC-27 | Low | P2 | S | GWC-26 | Not started | `AttachInternalEventSubscriber` bypasses the readiness gate; premature attach poisons the distributor permanently |
@@ -20,7 +20,7 @@ Dependency notes: GWC-26 and GWC-27 both change the internal-subscriber attach p
---
## GWC-24 — Unbounded event staging channel: sustained slow drain grows memory silently and invisibly `Medium` · `P1`
## GWC-24 — Unbounded event staging channel: sustained slow drain grows memory silently and invisibly `Medium` · `P1` · **Done (2026-08-07)**
**Finding.** The GWC-04 remediation decoupled the read loop from event backpressure by staging events into `_eventStaging`, an **unbounded** channel (`Workers/WorkerClient.cs:93-100`). `StageWorkerEvent`'s `TryWrite` therefore always succeeds (`:565-573`), and the sustained-overflow `ProtocolViolation` fault fires only when a *single* timed `WriteAsync` against the bounded `_events` exceeds `EventChannelFullModeTimeout` (default 5 s, `:610-647`). The queue-depth gauge counts only `_events` — `_eventQueueDepth` is incremented in `EnqueueWorkerEventAsync` (`:616`, `:627`) and decremented in `ReadEventsCoreAsync` (`:289`), so staged-but-unqueued events are invisible to `SetWorkerEventQueueDepth`. The field comment (`:28-33`) claims staging "only fills during the bounded EventChannelFullModeTimeout window", which is true only for a full consumer stall, not for a consumer that drains slower than the worker produces while each individual write still completes inside the window.