fix(WRK-21): make drain budget monotonic at the reserve boundary; guard the reply-too-large fallback
Review follow-ups on the WRK-21 cluster. 1. ResolveDrainReplyByteBudget was a step function, not a floor: just above the 64 KiB reserve the budget collapsed to a few bytes (at the validator-permitted floor MaxMessageBytes = 1024 + 64 KiB it was exactly 1024), too small to move a byte-heavy event, so DrainEvents truncated on every call and the drain-until- empty loop never terminated. It now takes the max of (frameMax - reserve) and frameMax/2, so the budget is monotonic and never below half the frame max. New test DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates drives a byte-heavy queue at the exact validator floor and asserts it drains to empty with no head ever reported oversized. 2. The reply-too-large fallback write is now itself size-guarded (WriteReplyTooLargeFallbackAsync, used by both the control and STA reply seams): at a pathologically tiny negotiated max below the gateway's floor the fallback could also throw MessageTooLarge and — uncaught — kill the session, defeating the "no diagnostics command is session-fatal" invariant. It now log-and-swallows; comment notes WRK-24 adds the negotiated-max lower bound that makes it unreachable. 3. Corrected the RepeatedFieldOverheadBytes doc comments: WorkerEvent.CalculateSize() already includes the event's tag and length prefix (the same shape the reply's repeated events field packs), so the 8 bytes is pure slack over an already- conservative estimate, not compensation for a missing wrapper.
This commit is contained in:
@@ -633,6 +633,76 @@ public sealed class WorkerPipeSessionTests
|
||||
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Regression for the reserve-boundary budget bug. The gateway validator accepts a Worker
|
||||
/// frame maximum as low as 1024 + 64 KiB, and just above that boundary a naive
|
||||
/// subtract-then-guard budget collapses to ~1024 bytes — too small to move even one
|
||||
/// byte-heavy event, so every drain reports truncation with the same head blocked and the
|
||||
/// drain-until-empty loop never terminates. The budget is now a floor (never below half the
|
||||
/// negotiated maximum), so a byte-heavy queue drains to empty even at the validator floor.
|
||||
/// </summary>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
[Fact]
|
||||
public async Task DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates()
|
||||
{
|
||||
// The lowest Worker.MaxMessageBytes GatewayOptionsValidator permits: the public gRPC floor
|
||||
// (1024) plus the 64 KiB envelope-overhead reserve. The naive budget would be exactly 1024
|
||||
// here; the floored budget is half of the frame max (~33 KiB).
|
||||
const uint validatorFloorFrameMax = 1024 + (64 * 1024);
|
||||
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(60));
|
||||
using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
|
||||
FakeRuntimeSession runtime = new()
|
||||
{
|
||||
SuppressDrainForBatchSize = 128,
|
||||
BackingQueue = CreateByteHeavyQueue(200, ByteHeavyEventPayloadBytes),
|
||||
};
|
||||
WorkerPipeSession session = CreatePipeSession(pipePair.WorkerStream, runtime);
|
||||
Task runTask = session.RunAsync(cancellation.Token);
|
||||
await CompleteGatewayHandshakeAsync(pipePair, validatorFloorFrameMax, cancellation.Token);
|
||||
|
||||
int recovered = 0;
|
||||
int replyCount = 0;
|
||||
while (true)
|
||||
{
|
||||
string correlationId = $"floor-drain-{replyCount}";
|
||||
await pipePair.GatewayWriter
|
||||
.WriteAsync(
|
||||
CreateControlCommandEnvelope(
|
||||
correlationId,
|
||||
MxCommandKind.DrainEvents,
|
||||
command => command.DrainEvents = new DrainEventsCommand { MaxEvents = 0 }),
|
||||
cancellation.Token);
|
||||
|
||||
WorkerEnvelope replyEnvelope = await ReadUntilAsync(
|
||||
pipePair.GatewayReader,
|
||||
WorkerEnvelope.BodyOneofCase.WorkerCommandReply,
|
||||
envelope => envelope.WorkerCommandReply.Reply.CorrelationId == correlationId,
|
||||
cancellation.Token);
|
||||
replyCount++;
|
||||
|
||||
MxCommandReply reply = replyEnvelope.WorkerCommandReply.Reply;
|
||||
Assert.True(
|
||||
replyEnvelope.CalculateSize() <= validatorFloorFrameMax,
|
||||
$"reply {replyCount} serialized to {replyEnvelope.CalculateSize()} bytes.");
|
||||
|
||||
int drainedThisReply = reply.DrainEvents.Events.Count;
|
||||
if (drainedThisReply == 0)
|
||||
{
|
||||
break;
|
||||
}
|
||||
|
||||
// The head is never reported as oversized at this frame max: the ~33 KiB floored budget
|
||||
// comfortably fits the ~1.8 KiB events, so each reply makes real progress.
|
||||
Assert.DoesNotContain("alone exceeds", reply.DiagnosticMessage);
|
||||
recovered += drainedThisReply;
|
||||
Assert.True(replyCount < 200, "DrainEvents made no progress at the validator floor frame max.");
|
||||
}
|
||||
|
||||
Assert.Equal(200, recovered);
|
||||
|
||||
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Verifies the control-reply write seam is not session-fatal on size (WRK-21 backstop). The
|
||||
/// reply builders size their payloads, so this path needs a deliberately budget-blind drain
|
||||
|
||||
Reference in New Issue
Block a user