fix(IPC-33): bound the DrainEvents byte budget against the gRPC send cap, not just the pipe frame max

The worker budgeted a DrainEvents reply against the negotiated pipe frame
maximum less one 64 KiB reserve, but the gateway returns that same
MxCommandReply verbatim through Invoke, where MaxSendMessageSize applies. The
two limits were coupled only from below (pipe >= gRPC + reserve), so any
configuration the validator permitted with a wider gap let the worker pack a
reply the pipe carried and the gRPC layer then refused with ResourceExhausted --
after Drain had already removed the events from the worker queue, losing them to
both the caller and the event stream.

Option (a) from the finding: close the coupling at both ends.

- GatewayOptionsValidator adds the upper half of the headroom rule
  (Worker:MaxMessageBytes <= Protocol:MaxGrpcMessageBytes + 2 x reserve), with a
  message naming the DrainEvents budget as the reason and the two knobs to move.
- WorkerPipeSession.ResolveDrainReplyByteBudget subtracts the reserve twice --
  once for the pipe frame, once so the MxCommandReply wrapper clears the gRPC
  send cap -- keeping the MaxMessageBytes/2 floor that makes small frame maxima
  usable.

The two changes are load-bearing for each other: the validator band is what
makes "frame max - 2 x reserve" provably at or below the gRPC cap, and the
doubled subtraction is what makes the band sufficient.

Docs updated in the same change (GatewayConfiguration.md both option rows plus
the headroom prose, WorkerFrameProtocol.md drain budget, gateway.md DrainEvents
byte axis). Tests: new GatewayOptionsValidatorTests cases for the upper bound
(reject) and both edges of the band (accept); GatewayOptionsTests' override case
now moves both message-size knobs together, since the band makes moving one
alone illegal.

The worker half cannot be built on macOS; verified on windev separately.
This commit is contained in:
Joseph Doherty
2026-08-20 09:30:39 -04:00
parent f4919ba97c
commit 577a0db862
9 changed files with 151 additions and 26 deletions
@@ -606,6 +606,15 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
// wrapped in a WorkerEnvelope and the outbound write faults the whole session. Fail fast
// at startup rather than mid-traffic. Only checked when both knobs are themselves in range so the
// message is not doubled up with the individual range errors.
//
// IPC-33: the coupling also has an upper half. The worker budgets a DrainEvents reply against the
// negotiated frame maximum less two reserves, and that same MxCommandReply is then returned
// verbatim through Invoke, where the gateway's gRPC MaxSendMessageSize (= MaxGrpcMessageBytes)
// applies. Letting the pipe maximum run arbitrarily far above the gRPC cap lets the worker pack a
// reply the pipe carries and the gRPC layer then refuses to send — with the drained events already
// removed from the worker queue, so they are lost to both the caller and the event stream. Keeping
// the pipe maximum within two reserves of the gRPC cap makes "frame max - 2x reserve" a budget that
// is always at or below the gRPC cap, so the wrapper always has a full reserve of margin.
private static void ValidateFrameSizeHeadroom(
WorkerOptions worker,
ProtocolOptions protocol,
@@ -628,6 +637,20 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
+ $"{WorkerFrameProtocolOptions.EnvelopeOverheadReserveBytes}-byte worker-frame envelope reserve "
+ $"(>= {requiredWorkerMax}).");
}
long permittedWorkerMax =
(long)protocol.MaxGrpcMessageBytes + (2L * WorkerFrameProtocolOptions.EnvelopeOverheadReserveBytes);
if (worker.MaxMessageBytes > permittedWorkerMax)
{
builder.Add(
$"MxGateway:Worker:MaxMessageBytes ({worker.MaxMessageBytes}) must be at most "
+ $"MxGateway:Protocol:MaxGrpcMessageBytes ({protocol.MaxGrpcMessageBytes}) plus twice the "
+ $"{WorkerFrameProtocolOptions.EnvelopeOverheadReserveBytes}-byte worker-frame envelope reserve "
+ $"(<= {permittedWorkerMax}). The worker budgets a DrainEvents reply against the negotiated "
+ "frame maximum less two reserves; a pipe maximum further above the gRPC cap lets the worker "
+ "pack a reply the pipe carries but the gRPC send cap refuses, losing the already-drained "
+ "events. Lower MxGateway:Worker:MaxMessageBytes or raise MxGateway:Protocol:MaxGrpcMessageBytes.");
}
}
private static void AddIfBlank(string? value, string message, ValidationBuilder builder)