fix(WRK-21,WRK-28,WRK-23,IPC-30): byte-budget the DrainEvents reply, stop size errors from killing sessions
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m8s
ci / portable (push) Successful in 7m41s
ci / windows-x86 (push) Failing after 12m32s

WRK-21 — DrainEvents was bounded by event count only, so a byte-heavy queue
(large string/array MxValues) built a reply above the negotiated frame maximum:
the writer rejected the frame, the exception unwound the session, and the events
already dequeued were destroyed. The drain is now byte-budgeted inside the queue
lock, so an event is dequeued only once it is known to fit and one that does not
stays at the head. Truncation is reported through the reply's existing
DiagnosticMessage (no contract change); callers drain until an empty reply. Both
reply-write seams — the control-command path and ProcessCommandAsync — now catch
MessageTooLarge and answer the correlation with an InvalidRequest reply instead
of unwinding or faulting the session. Satisfies IPC-23 R1-R3.

WRK-28 — the 10,000 drain ceiling moves to GatewayContractInfo
.MaxDrainEventsPerCommand, referenced by both the gateway request validator and
the worker clamp, replacing a comment-only sync contract. C# const only; no
.proto change.

WRK-23 — WorkerFrameWriter now peek-stamps, validates, then commits the sequence
counter immediately before the stream write, so a per-frame rejection leaves no
phantom gap on the wire.

IPC-30 — an oversized event frame stays session-fatal (it is undeliverable end to
end and neither dropping nor synthesizing a replacement is allowed), but the
death is structured: the event's identity and sizes are logged (never its value),
a WorkerFault with category PROTOCOL_VIOLATION and command method EventDrain is
written, then the session exits as before.

Docs updated in the same change: MxAccessWorkerInstanceDesign.md (drain byte cap,
truncation contract, oversized-head behavior, oversized-event policy, no control
reply is session-fatal on size), WorkerFrameProtocol.md (reply pre-sizing,
non-fatal reply-size rule, oversized-event policy, rejected frames do not consume
sequence numbers), gateway.md (DrainEvents two-axis bound).
This commit is contained in:
Joseph Doherty
2026-08-07 05:38:23 -04:00
parent ead921cace
commit 33ba612ddd
18 changed files with 1118 additions and 57 deletions
@@ -98,6 +98,103 @@ public sealed class MxAccessEventQueueTests
Assert.Equal(0, queue.Count);
}
/// <summary>
/// Verifies the byte-budgeted drain stops before the budget is exceeded, leaves the
/// remainder queued in order, and reports the exact remaining count (WRK-21). Events that
/// do not fit must never be dequeued — dequeuing them is how the pre-fix drain lost events
/// when the reply frame was rejected.
/// </summary>
[Fact]
public void Drain_ByteBudget_StopsBeforeBudgetAndLeavesRemainderQueued()
{
MxAccessEventQueue queue = new(capacity: 8);
for (int itemHandle = 0; itemHandle < 5; itemHandle++)
{
queue.Enqueue(CreateEventWithPayload(itemHandle, payloadLength: 512));
}
int perEventCost = MeasureDrainCost(payloadLength: 512);
// Budget for exactly two events (plus a sliver too small for a third).
IReadOnlyList<WorkerEvent> drained =
queue.Drain(maxEvents: 0, maxTotalBytes: (perEventCost * 2) + (perEventCost / 2)).Events;
Assert.Equal(2, drained.Count);
Assert.Equal(0, drained[0].Event.ItemHandle);
Assert.Equal(1, drained[1].Event.ItemHandle);
Assert.Equal(3, queue.Count);
// The undrained remainder is still present, still in order.
IReadOnlyList<WorkerEvent> rest = queue.Drain(maxEvents: 0);
Assert.Equal(new[] { 2, 3, 4 }, new[] { rest[0].Event.ItemHandle, rest[1].Event.ItemHandle, rest[2].Event.ItemHandle });
}
/// <summary>
/// Verifies the byte-budgeted drain reports truncation and the exact remaining count so the
/// DrainEvents reply can tell the caller to drain again.
/// </summary>
[Fact]
public void Drain_ByteBudget_ReportsTruncationAndRemainingCount()
{
MxAccessEventQueue queue = new(capacity: 8);
for (int itemHandle = 0; itemHandle < 4; itemHandle++)
{
queue.Enqueue(CreateEventWithPayload(itemHandle, payloadLength: 256));
}
WorkerEventDrainResult result = queue.Drain(
maxEvents: 0,
maxTotalBytes: MeasureDrainCost(payloadLength: 256));
Assert.Single(result.Events);
Assert.True(result.TruncatedBySize);
Assert.Equal(3, result.RemainingCount);
Assert.Equal(0UL, result.OversizedHeadSequence);
}
/// <summary>
/// Verifies the degenerate case: a head event whose own serialized size exceeds the whole
/// budget is not drained (draining it would build an oversized reply or lose the event) and
/// its worker sequence is reported so an operator can find the offending tag.
/// </summary>
[Fact]
public void Drain_ByteBudget_OversizedHead_DrainsNothingAndReportsHeadSequence()
{
MxAccessEventQueue queue = new(capacity: 8);
queue.Enqueue(CreateEventWithPayload(itemHandle: 0, payloadLength: 4096));
queue.Enqueue(CreateEventWithPayload(itemHandle: 1, payloadLength: 8));
WorkerEventDrainResult result = queue.Drain(maxEvents: 0, maxTotalBytes: 1024);
Assert.Empty(result.Events);
Assert.True(result.TruncatedBySize);
Assert.Equal(2, result.RemainingCount);
Assert.Equal(1UL, result.OversizedHeadSequence);
// The blocked event is still queued — it was never removed.
Assert.Equal(2, queue.Count);
}
/// <summary>
/// Verifies the count cap still binds when the byte budget is generous: the byte cap is an
/// additional bound, not a replacement.
/// </summary>
[Fact]
public void Drain_ByteBudget_CountCapStillBinds()
{
MxAccessEventQueue queue = new(capacity: 8);
for (int itemHandle = 0; itemHandle < 5; itemHandle++)
{
queue.Enqueue(CreateEventWithPayload(itemHandle, payloadLength: 16));
}
WorkerEventDrainResult result = queue.Drain(maxEvents: 2, maxTotalBytes: 1024 * 1024);
Assert.Equal(2, result.Events.Count);
Assert.False(result.TruncatedBySize);
Assert.Equal(3, result.RemainingCount);
}
/// <summary>Verifies that Enqueue is rejected after a fault is recorded manually.</summary>
[Fact]
public void Enqueue_AfterRecordFault_ThrowsInvalidOperationException()
@@ -149,6 +246,40 @@ public sealed class MxAccessEventQueueTests
Assert.Equal(WorkerFaultCategory.MxaccessEventConversionFailed, queue.Fault?.Category);
}
// Mirrors MxAccessEventQueue's per-event repeated-field allowance. Kept local (and asserted
// through the budgets below) so a change to the queue's charge shows up as a failing bound
// rather than silently loosening these tests.
private const int RepeatedFieldOverheadBytes = 8;
/// <summary>
/// Measures what the queue charges one event of the given payload size against the byte budget:
/// the serialized <see cref="WorkerEvent"/> as it exists after Enqueue (sequence and timestamp
/// stamped) plus the repeated-field allowance.
/// </summary>
/// <param name="payloadLength">Length of the event's raw-status payload string.</param>
/// <returns>The per-event byte cost.</returns>
private static int MeasureDrainCost(int payloadLength)
{
MxAccessEventQueue probe = new(capacity: 1);
probe.Enqueue(CreateEventWithPayload(0, payloadLength));
Assert.True(probe.TryDequeue(out WorkerEvent? probeEvent));
return probeEvent!.CalculateSize() + RepeatedFieldOverheadBytes;
}
/// <summary>
/// Builds a byte-heavy event: a large string field is the cheapest stand-in for the array/string
/// <see cref="MxValue"/> payloads that make a count-capped drain overshoot the frame maximum.
/// </summary>
/// <param name="itemHandle">Item handle identifying the event in assertions.</param>
/// <param name="payloadLength">Length of the raw-status payload string.</param>
/// <returns>The constructed event.</returns>
private static MxEvent CreateEventWithPayload(int itemHandle, int payloadLength)
{
MxEvent mxEvent = CreateEvent(MxEventFamily.OnDataChange, itemHandle);
mxEvent.RawStatus = new string('x', payloadLength);
return mxEvent;
}
private static MxEvent CreateEvent(
MxEventFamily family,
int itemHandle)