feat(gateway): decouple worker event enqueue from the read loop (GWC-04)

The read loop awaited EnqueueWorkerEventAsync inline, which blocks in the bounded
event channel's timed WriteAsync (up to EventChannelFullModeTimeout, default 5s)
when the channel is full with no/slow consumer. A WorkerCommandReply or heartbeat
queued behind an event frame was then stalled, so an in-flight InvokeAsync could
hit CommandTimeout even though the worker replied in time.

Mirror the existing outbound WriteLoopAsync: add an unbounded event staging
channel and a dedicated EventWriteLoopAsync. DispatchEnvelope is now fully
synchronous — the WorkerEvent branch hands the event off with a non-blocking
TryWrite and the read loop never awaits. The event write loop owns the timed
WriteAsync into the bounded channel and the sustained-overflow ProtocolViolation
fault (unchanged contract). Registered in WaitForBackgroundTasks + completed on
close/fault/dispose.

Test: reply arriving after events with a full, consumer-less event channel is
dispatched promptly (no CommandTimeout) — pipe-harness, verified on windev.
Docs: GatewayProcessDesign read/write/event-loop section.
This commit is contained in:
Joseph Doherty
2026-07-09 09:28:13 -04:00
parent bfbfcdd112
commit 32d6b48910
3 changed files with 132 additions and 15 deletions
+18 -2
View File
@@ -481,7 +481,9 @@ Internally it owns:
- pipe stream,
- read loop,
- write loop,
- event write loop,
- outbound command/control channel serialized by the write loop,
- unbounded event staging channel drained by the event write loop,
- bounded inbound event channel,
- pending command dictionary keyed by correlation id,
- heartbeat monitor,
@@ -499,15 +501,29 @@ The read loop:
1. Reads one frame.
2. Parses `WorkerEnvelope`.
3. Validates protocol fields.
4. Dispatches by body type:
4. Dispatches by body type, **synchronously and without blocking**:
- `WorkerCommandReply`: completes pending command.
- `WorkerEvent`: enqueues event.
- `WorkerEvent`: hands the event to the event write loop via a non-blocking staging write.
- `WorkerHeartbeat`: updates heartbeat timestamp.
- `WorkerFault`: faults session.
5. Stops when pipe closes or cancellation is requested.
If the pipe closes while the session is not closing, fault the session.
The read loop never awaits event enqueue. Events are staged to the event write
loop with a non-blocking write, so a full inbound event channel (a slow or
absent `StreamEvents` consumer) cannot stall the read loop behind an event and
delay a command reply or heartbeat (GWC-04). The bounded-channel backpressure
window (`EventChannelFullModeTimeout`) and the sustained-overflow fault are
applied by the event write loop, not the read loop.
### Event write loop
The event write loop drains the staging channel and performs the timed write
into the bounded inbound event channel. When the inbound channel stays full past
`EventChannelFullModeTimeout` it faults the session (`ProtocolViolation`) — the
same overflow contract as before, moved off the read loop.
### Write loop
The write loop serializes all writes to the pipe. No other code should write to