fix(WRK-21,WRK-28,WRK-23,IPC-30): byte-budget the DrainEvents reply, stop size errors from killing sessions
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m8s
ci / portable (push) Successful in 7m41s
ci / windows-x86 (push) Failing after 12m32s

WRK-21 — DrainEvents was bounded by event count only, so a byte-heavy queue
(large string/array MxValues) built a reply above the negotiated frame maximum:
the writer rejected the frame, the exception unwound the session, and the events
already dequeued were destroyed. The drain is now byte-budgeted inside the queue
lock, so an event is dequeued only once it is known to fit and one that does not
stays at the head. Truncation is reported through the reply's existing
DiagnosticMessage (no contract change); callers drain until an empty reply. Both
reply-write seams — the control-command path and ProcessCommandAsync — now catch
MessageTooLarge and answer the correlation with an InvalidRequest reply instead
of unwinding or faulting the session. Satisfies IPC-23 R1-R3.

WRK-28 — the 10,000 drain ceiling moves to GatewayContractInfo
.MaxDrainEventsPerCommand, referenced by both the gateway request validator and
the worker clamp, replacing a comment-only sync contract. C# const only; no
.proto change.

WRK-23 — WorkerFrameWriter now peek-stamps, validates, then commits the sequence
counter immediately before the stream write, so a per-frame rejection leaves no
phantom gap on the wire.

IPC-30 — an oversized event frame stays session-fatal (it is undeliverable end to
end and neither dropping nor synthesizing a replacement is allowed), but the
death is structured: the event's identity and sizes are logged (never its value),
a WorkerFault with category PROTOCOL_VIOLATION and command method EventDrain is
written, then the session exits as before.

Docs updated in the same change: MxAccessWorkerInstanceDesign.md (drain byte cap,
truncation contract, oversized-head behavior, oversized-event policy, no control
reply is session-fatal on size), WorkerFrameProtocol.md (reply pre-sizing,
non-fatal reply-size rule, oversized-event policy, rejected frames do not consume
sequence numbers), gateway.md (DrainEvents two-axis bound).
This commit is contained in:
Joseph Doherty
2026-08-07 05:38:23 -04:00
parent ead921cace
commit 33ba612ddd
18 changed files with 1118 additions and 57 deletions
@@ -43,6 +43,13 @@ internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
/// <summary>Gets or sets whether ShutdownGracefullyAsync throws a TimeoutException.</summary>
public bool ThrowTimeoutOnShutdown { get; set; }
/// <summary>
/// Optional diagnostic message stuffed into every dispatched command reply. A long value
/// pushes the STA command reply past a small negotiated frame maximum, which is how a test
/// drives the <c>ProcessCommandAsync</c> reply-size backstop.
/// </summary>
public string? DispatchReplyDiagnosticMessage { get; set; }
/// <summary>Gets a value indicating whether Dispose was called.</summary>
public bool Disposed { get; private set; }
@@ -92,7 +99,7 @@ internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
throw new InvalidOperationException("Command failed after shutdown started.");
}
return new MxCommandReply
MxCommandReply reply = new()
{
SessionId = command.SessionId,
CorrelationId = command.CorrelationId,
@@ -103,6 +110,13 @@ internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
Message = "OK",
},
};
if (DispatchReplyDiagnosticMessage is not null)
{
reply.DiagnosticMessage = DispatchReplyDiagnosticMessage;
}
return reply;
});
}
@@ -133,6 +147,27 @@ internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
/// </summary>
public uint? LastDrainMaxEvents { get; private set; }
/// <summary>
/// Optional real event queue backing the drain paths. When set, both
/// <see cref="DrainEvents(uint)"/> and <see cref="DrainEvents(uint, int)"/> delegate to it
/// so a test can exercise the production byte-budgeting logic behind the fake session.
/// </summary>
public MxAccessEventQueue? BackingQueue { get; set; }
/// <summary>
/// When set, <see cref="DrainEvents(uint, int)"/> ignores the byte budget and drains purely
/// by count. Simulates the "sizing bug or future command" case the control-reply size
/// backstop exists for, so a test can drive an oversized reply without a real budgeting
/// defect.
/// </summary>
public bool IgnoreDrainByteBudget { get; set; }
/// <summary>
/// Records the <c>maxTotalBytes</c> argument of the most recent byte-budgeted
/// <see cref="DrainEvents(uint, int)"/> call.
/// </summary>
public int? LastDrainMaxTotalBytes { get; private set; }
/// <inheritdoc />
public IReadOnlyList<WorkerEvent> DrainEvents(uint maxEvents)
{
@@ -143,6 +178,76 @@ internal sealed class FakeRuntimeSession : IWorkerRuntimeSession
LastDrainMaxEvents = maxEvents;
if (BackingQueue is not null)
{
return BackingQueue.Drain(maxEvents);
}
lock (gate)
{
int drainCount = maxEvents == 0
? events.Count
: Math.Min(events.Count, checked((int)Math.Min(maxEvents, int.MaxValue)));
List<WorkerEvent> drained = new(drainCount);
for (int index = 0; index < drainCount; index++)
{
drained.Add(events.Dequeue());
}
return drained;
}
}
/// <inheritdoc />
public WorkerEventDrainResult DrainEvents(uint maxEvents, int maxTotalBytes)
{
if (SuppressDrainForBatchSize is uint suppressed && maxEvents == suppressed)
{
return new WorkerEventDrainResult(
Array.Empty<WorkerEvent>(),
truncatedBySize: false,
remainingCount: PendingEventCount,
oversizedHeadSequence: 0);
}
LastDrainMaxEvents = maxEvents;
LastDrainMaxTotalBytes = maxTotalBytes;
if (BackingQueue is not null && !IgnoreDrainByteBudget)
{
return BackingQueue.Drain(maxEvents, maxTotalBytes);
}
// Count-only drain: either no backing queue (the simple fakes) or a deliberately
// budget-blind drain used to exercise the reply-size backstop.
IReadOnlyList<WorkerEvent> drained = BackingQueue is not null
? BackingQueue.Drain(maxEvents)
: DrainByCount(maxEvents);
return new WorkerEventDrainResult(
drained,
truncatedBySize: false,
remainingCount: PendingEventCount,
oversizedHeadSequence: 0);
}
private int PendingEventCount
{
get
{
if (BackingQueue is not null)
{
return BackingQueue.Count;
}
lock (gate)
{
return events.Count;
}
}
}
private IReadOnlyList<WorkerEvent> DrainByCount(uint maxEvents)
{
lock (gate)
{
int drainCount = maxEvents == 0