fix(WRK-21): make drain budget monotonic at the reserve boundary; guard the reply-too-large fallback
ci / windows-x86 (push) Successful in 1m16s
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m17s
ci / portable (push) Successful in 9m31s

Review follow-ups on the WRK-21 cluster.

1. ResolveDrainReplyByteBudget was a step function, not a floor: just above the
   64 KiB reserve the budget collapsed to a few bytes (at the validator-permitted
   floor MaxMessageBytes = 1024 + 64 KiB it was exactly 1024), too small to move a
   byte-heavy event, so DrainEvents truncated on every call and the drain-until-
   empty loop never terminated. It now takes the max of (frameMax - reserve) and
   frameMax/2, so the budget is monotonic and never below half the frame max. New
   test DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates drives a
   byte-heavy queue at the exact validator floor and asserts it drains to empty
   with no head ever reported oversized.

2. The reply-too-large fallback write is now itself size-guarded
   (WriteReplyTooLargeFallbackAsync, used by both the control and STA reply seams):
   at a pathologically tiny negotiated max below the gateway's floor the fallback
   could also throw MessageTooLarge and — uncaught — kill the session, defeating the
   "no diagnostics command is session-fatal" invariant. It now log-and-swallows;
   comment notes WRK-24 adds the negotiated-max lower bound that makes it unreachable.

3. Corrected the RepeatedFieldOverheadBytes doc comments: WorkerEvent.CalculateSize()
   already includes the event's tag and length prefix (the same shape the reply's
   repeated events field packs), so the 8 bytes is pure slack over an already-
   conservative estimate, not compensation for a missing wrapper.
This commit is contained in:
Joseph Doherty
2026-08-07 07:09:13 -04:00
parent c9925688f5
commit 6bc3f9b991
3 changed files with 134 additions and 17 deletions
@@ -633,6 +633,76 @@ public sealed class WorkerPipeSessionTests
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
}
/// <summary>
/// Regression for the reserve-boundary budget bug. The gateway validator accepts a Worker
/// frame maximum as low as 1024 + 64 KiB, and just above that boundary a naive
/// subtract-then-guard budget collapses to ~1024 bytes — too small to move even one
/// byte-heavy event, so every drain reports truncation with the same head blocked and the
/// drain-until-empty loop never terminates. The budget is now a floor (never below half the
/// negotiated maximum), so a byte-heavy queue drains to empty even at the validator floor.
/// </summary>
/// <returns>A task that represents the asynchronous operation.</returns>
[Fact]
public async Task DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates()
{
// The lowest Worker.MaxMessageBytes GatewayOptionsValidator permits: the public gRPC floor
// (1024) plus the 64 KiB envelope-overhead reserve. The naive budget would be exactly 1024
// here; the floored budget is half of the frame max (~33 KiB).
const uint validatorFloorFrameMax = 1024 + (64 * 1024);
using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(60));
using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);
FakeRuntimeSession runtime = new()
{
SuppressDrainForBatchSize = 128,
BackingQueue = CreateByteHeavyQueue(200, ByteHeavyEventPayloadBytes),
};
WorkerPipeSession session = CreatePipeSession(pipePair.WorkerStream, runtime);
Task runTask = session.RunAsync(cancellation.Token);
await CompleteGatewayHandshakeAsync(pipePair, validatorFloorFrameMax, cancellation.Token);
int recovered = 0;
int replyCount = 0;
while (true)
{
string correlationId = $"floor-drain-{replyCount}";
await pipePair.GatewayWriter
.WriteAsync(
CreateControlCommandEnvelope(
correlationId,
MxCommandKind.DrainEvents,
command => command.DrainEvents = new DrainEventsCommand { MaxEvents = 0 }),
cancellation.Token);
WorkerEnvelope replyEnvelope = await ReadUntilAsync(
pipePair.GatewayReader,
WorkerEnvelope.BodyOneofCase.WorkerCommandReply,
envelope => envelope.WorkerCommandReply.Reply.CorrelationId == correlationId,
cancellation.Token);
replyCount++;
MxCommandReply reply = replyEnvelope.WorkerCommandReply.Reply;
Assert.True(
replyEnvelope.CalculateSize() <= validatorFloorFrameMax,
$"reply {replyCount} serialized to {replyEnvelope.CalculateSize()} bytes.");
int drainedThisReply = reply.DrainEvents.Events.Count;
if (drainedThisReply == 0)
{
break;
}
// The head is never reported as oversized at this frame max: the ~33 KiB floored budget
// comfortably fits the ~1.8 KiB events, so each reply makes real progress.
Assert.DoesNotContain("alone exceeds", reply.DiagnosticMessage);
recovered += drainedThisReply;
Assert.True(replyCount < 200, "DrainEvents made no progress at the validator floor frame max.");
}
Assert.Equal(200, recovered);
await SendShutdownAndWaitAsync(pipePair, runTask, cancellation.Token);
}
/// <summary>
/// Verifies the control-reply write seam is not session-fatal on size (WRK-21 backstop). The
/// reply builders size their payloads, so this path needs a deliberately budget-blind drain