fix(WRK-21,WRK-28,WRK-23,IPC-30): byte-budget the DrainEvents reply, stop size errors from killing sessions
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m8s
ci / portable (push) Successful in 7m41s
ci / windows-x86 (push) Failing after 12m32s

WRK-21 — DrainEvents was bounded by event count only, so a byte-heavy queue
(large string/array MxValues) built a reply above the negotiated frame maximum:
the writer rejected the frame, the exception unwound the session, and the events
already dequeued were destroyed. The drain is now byte-budgeted inside the queue
lock, so an event is dequeued only once it is known to fit and one that does not
stays at the head. Truncation is reported through the reply's existing
DiagnosticMessage (no contract change); callers drain until an empty reply. Both
reply-write seams — the control-command path and ProcessCommandAsync — now catch
MessageTooLarge and answer the correlation with an InvalidRequest reply instead
of unwinding or faulting the session. Satisfies IPC-23 R1-R3.

WRK-28 — the 10,000 drain ceiling moves to GatewayContractInfo
.MaxDrainEventsPerCommand, referenced by both the gateway request validator and
the worker clamp, replacing a comment-only sync contract. C# const only; no
.proto change.

WRK-23 — WorkerFrameWriter now peek-stamps, validates, then commits the sequence
counter immediately before the stream write, so a per-frame rejection leaves no
phantom gap on the wire.

IPC-30 — an oversized event frame stays session-fatal (it is undeliverable end to
end and neither dropping nor synthesizing a replacement is allowed), but the
death is structured: the event's identity and sizes are logged (never its value),
a WorkerFault with category PROTOCOL_VIOLATION and command method EventDrain is
written, then the session exits as before.

Docs updated in the same change: MxAccessWorkerInstanceDesign.md (drain byte cap,
truncation contract, oversized-head behavior, oversized-event policy, no control
reply is session-fatal on size), WorkerFrameProtocol.md (reply pre-sizing,
non-fatal reply-size rule, oversized-event policy, rejected frames do not consume
sequence numbers), gateway.md (DrainEvents two-axis bound).
This commit is contained in:
Joseph Doherty
2026-08-07 05:38:23 -04:00
parent ead921cace
commit 33ba612ddd
18 changed files with 1118 additions and 57 deletions
@@ -43,8 +43,9 @@ public sealed class WorkerFrameWriter
private readonly Queue<PendingFrame> _eventFrames = new Queue<PendingFrame>();
// Only ever read/written by the current write-lock holder while draining, so no interlock is
// needed. Starts at 0 and is pre-incremented, so the first written frame carries sequence 1
// (matching the previous behaviour).
// needed. Starts at 0 and is committed only immediately before the stream write, so the first
// written frame carries sequence 1 and a per-frame rejection leaves the counter untouched —
// the next accepted frame reuses the number and the wire sequence stays contiguous.
private ulong _nextSequence;
/// <summary>Initializes a new instance of the WorkerFrameWriter class.</summary>
@@ -237,7 +238,14 @@ public sealed class WorkerFrameWriter
// Stamp the sequence at the actual point of writing, under the write lock, so the wire order
// and the stamped sequence agree regardless of caller concurrency or priority.
envelope.Sequence = unchecked(++_nextSequence);
//
// Peek-stamp-commit (WRK-23): the sequence participates in CalculateSize() (varint width),
// so it must be stamped before the size checks — but a per-frame rejection must not burn a
// number, or the wire shows phantom gaps that an operator reads as lost frames. Stamp a
// candidate, validate the stamped envelope, and commit the counter only once the frame is
// certain to be written.
ulong candidateSequence = unchecked(_nextSequence + 1);
envelope.Sequence = candidateSequence;
int payloadLength = envelope.CalculateSize();
if (payloadLength == 0)
@@ -254,6 +262,8 @@ public sealed class WorkerFrameWriter
$"Worker envelope payload length {payloadLength} exceeds the configured maximum of {_options.MaxMessageBytes} bytes.");
}
_nextSequence = candidateSequence;
// Serialize once into a single buffer that carries the 4-byte length prefix followed by the
// payload, then issue one stream write. This avoids a second serialization pass, a separate
// prefix array, and a separate prefix write. The flush is deferred to the end of the drained