fix(IPC-33): bound the DrainEvents byte budget against the gRPC send cap, not just the pipe frame max
The worker budgeted a DrainEvents reply against the negotiated pipe frame maximum less one 64 KiB reserve, but the gateway returns that same MxCommandReply verbatim through Invoke, where MaxSendMessageSize applies. The two limits were coupled only from below (pipe >= gRPC + reserve), so any configuration the validator permitted with a wider gap let the worker pack a reply the pipe carried and the gRPC layer then refused with ResourceExhausted -- after Drain had already removed the events from the worker queue, losing them to both the caller and the event stream. Option (a) from the finding: close the coupling at both ends. - GatewayOptionsValidator adds the upper half of the headroom rule (Worker:MaxMessageBytes <= Protocol:MaxGrpcMessageBytes + 2 x reserve), with a message naming the DrainEvents budget as the reason and the two knobs to move. - WorkerPipeSession.ResolveDrainReplyByteBudget subtracts the reserve twice -- once for the pipe frame, once so the MxCommandReply wrapper clears the gRPC send cap -- keeping the MaxMessageBytes/2 floor that makes small frame maxima usable. The two changes are load-bearing for each other: the validator band is what makes "frame max - 2 x reserve" provably at or below the gRPC cap, and the doubled subtraction is what makes the band sufficient. Docs updated in the same change (GatewayConfiguration.md both option rows plus the headroom prose, WorkerFrameProtocol.md drain budget, gateway.md DrainEvents byte axis). Tests: new GatewayOptionsValidatorTests cases for the upper bound (reject) and both edges of the band (accept); GatewayOptionsTests' override case now moves both message-size knobs together, since the band makes moving one alone illegal. The worker half cannot be built on macOS; verified on windev separately.
This commit is contained in:
@@ -27,7 +27,8 @@ public sealed class WorkerPipeSession
|
||||
// Headroom subtracted from the negotiated frame maximum when budgeting a DrainEvents reply. It
|
||||
// covers the WorkerEnvelope/WorkerCommandReply/MxCommandReply wrapper the drained events are
|
||||
// packed into — the same envelope-overhead reserve rationale docs/WorkerFrameProtocol.md
|
||||
// records for the frame max itself.
|
||||
// records for the frame max itself. ResolveDrainReplyByteBudget subtracts it twice: once for the
|
||||
// pipe frame, once so the reply also clears the public gRPC send cap it is forwarded through.
|
||||
private const int DrainReplyFrameHeadroomBytes = 64 * 1024;
|
||||
|
||||
private readonly WorkerFrameProtocolOptions _options;
|
||||
@@ -1021,20 +1022,33 @@ public sealed class WorkerPipeSession
|
||||
|
||||
/// <summary>
|
||||
/// Byte budget for the events packed into one DrainEvents reply: the negotiated frame
|
||||
/// maximum less a fixed reserve for the envelope/reply wrapper, but never below half the
|
||||
/// negotiated maximum. The lower bound must be a floor, not a step: a bare
|
||||
/// <c>subtract-then-guard-positive</c> collapses the budget to a handful of bytes just above
|
||||
/// the reserve (e.g. at the validator-permitted floor MaxMessageBytes = 1024 + 64 KiB the
|
||||
/// subtraction leaves 1024, too small to move even one byte-heavy event, so every drain
|
||||
/// truncates and the drain-until-empty caller never terminates). Taking the max with
|
||||
/// half the negotiated maximum keeps the budget monotonic across the reserve boundary while
|
||||
/// still leaving the full reserve for the wrapper whenever the frame max is large enough
|
||||
/// that the reserve is the smaller subtraction — which is every configuration above 128 KiB.
|
||||
/// maximum less <em>two</em> reserves, but never below half the negotiated maximum. The
|
||||
/// first reserve covers the envelope/reply wrapper the events are packed into so the reply
|
||||
/// fits one pipe frame. The second reserve exists because that same
|
||||
/// <c>MxCommandReply</c> does not stop at the pipe: the gateway returns it verbatim through
|
||||
/// <c>Invoke</c>, where the public gRPC send cap applies, and the gateway's validator
|
||||
/// couples the two limits so the pipe maximum never exceeds the gRPC cap by more than two
|
||||
/// reserves (IPC-33). Subtracting both therefore keeps the packed events at or below the
|
||||
/// gRPC cap and leaves the wrapper — correlation id, protocol status, kind, the
|
||||
/// <c>drain_events</c> length prefix and, in the truncated case, the
|
||||
/// <c>DiagnosticMessage</c> — a full reserve of margin under it. Budgeting against the pipe
|
||||
/// maximum alone let a legal configuration build a reply the pipe carried and the gRPC layer
|
||||
/// then refused to send, with the events already dequeued and therefore lost.
|
||||
/// <para>
|
||||
/// The lower bound must be a floor, not a step: a bare <c>subtract-then-guard-positive</c>
|
||||
/// collapses the budget to a handful of bytes (or below zero) once the reserves approach the
|
||||
/// frame maximum (e.g. at the validator-permitted floor MaxMessageBytes = 1024 + 64 KiB),
|
||||
/// too small to move even one byte-heavy event, so every drain truncates and the
|
||||
/// drain-until-empty caller never terminates. Taking the max with half the negotiated
|
||||
/// maximum keeps the budget monotonic across the reserve boundary while still leaving both
|
||||
/// reserves for the wrapper whenever the frame max is large enough that the subtraction is
|
||||
/// the smaller of the two — which is every configuration above 256 KiB.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
private int ResolveDrainReplyByteBudget()
|
||||
{
|
||||
return Math.Max(
|
||||
_options.MaxMessageBytes - DrainReplyFrameHeadroomBytes,
|
||||
_options.MaxMessageBytes - (2 * DrainReplyFrameHeadroomBytes),
|
||||
_options.MaxMessageBytes / 2);
|
||||
}
|
||||
|
||||
|
||||
Reference in New Issue
Block a user