fix(WRK-21,WRK-28,WRK-23,IPC-30): byte-budget the DrainEvents reply, stop size errors from killing sessions
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m8s
ci / portable (push) Successful in 7m41s
ci / windows-x86 (push) Failing after 12m32s

WRK-21 — DrainEvents was bounded by event count only, so a byte-heavy queue
(large string/array MxValues) built a reply above the negotiated frame maximum:
the writer rejected the frame, the exception unwound the session, and the events
already dequeued were destroyed. The drain is now byte-budgeted inside the queue
lock, so an event is dequeued only once it is known to fit and one that does not
stays at the head. Truncation is reported through the reply's existing
DiagnosticMessage (no contract change); callers drain until an empty reply. Both
reply-write seams — the control-command path and ProcessCommandAsync — now catch
MessageTooLarge and answer the correlation with an InvalidRequest reply instead
of unwinding or faulting the session. Satisfies IPC-23 R1-R3.

WRK-28 — the 10,000 drain ceiling moves to GatewayContractInfo
.MaxDrainEventsPerCommand, referenced by both the gateway request validator and
the worker clamp, replacing a comment-only sync contract. C# const only; no
.proto change.

WRK-23 — WorkerFrameWriter now peek-stamps, validates, then commits the sequence
counter immediately before the stream write, so a per-frame rejection leaves no
phantom gap on the wire.

IPC-30 — an oversized event frame stays session-fatal (it is undeliverable end to
end and neither dropping nor synthesizing a replacement is allowed), but the
death is structured: the event's identity and sizes are logged (never its value),
a WorkerFault with category PROTOCOL_VIOLATION and command method EventDrain is
written, then the session exits as before.

Docs updated in the same change: MxAccessWorkerInstanceDesign.md (drain byte cap,
truncation contract, oversized-head behavior, oversized-event policy, no control
reply is session-fatal on size), WorkerFrameProtocol.md (reply pre-sizing,
non-fatal reply-size rule, oversized-event policy, rejected frames do not consume
sequence numbers), gateway.md (DrainEvents two-axis bound).
This commit is contained in:
Joseph Doherty
2026-08-07 05:38:23 -04:00
parent ead921cace
commit 33ba612ddd
18 changed files with 1118 additions and 57 deletions
@@ -44,6 +44,19 @@ public interface IWorkerRuntimeSession : IDisposable
/// <returns>List of drained events.</returns>
IReadOnlyList<WorkerEvent> DrainEvents(uint maxEvents);
/// <summary>
/// Drains pending events bounded by both a count cap and a byte budget, so a caller building a
/// single reply frame never removes an event it cannot ship.
/// </summary>
/// <remarks>
/// Declared as a second method rather than a default interface method: the worker targets
/// .NET Framework 4.8, which has no runtime support for default interface members.
/// </remarks>
/// <param name="maxEvents">Maximum number of events to drain; 0 means "no count limit".</param>
/// <param name="maxTotalBytes">Byte budget for the drained events' estimated serialized size.</param>
/// <returns>The drained events and the truncation facts describing what stayed queued.</returns>
WorkerEventDrainResult DrainEvents(uint maxEvents, int maxTotalBytes);
/// <summary>
/// Drains a pending fault from the queue, if any.
/// </summary>
@@ -26,6 +26,11 @@ public sealed class MxAccessEventQueue
/// </summary>
public const int DefaultCapacity = 10000;
// Per-event allowance added to WorkerEvent.CalculateSize() when charging the byte budget in
// Drain(maxEvents, maxTotalBytes): conservatively covers the repeated-field tag byte and the
// length varint the event costs once packed into DrainEventsReply.
private const int RepeatedFieldOverheadBytes = 8;
private readonly int capacity;
private readonly Queue<WorkerEvent> events;
private readonly object syncRoot = new();
@@ -209,6 +214,63 @@ public sealed class MxAccessEventQueue
}
}
/// <summary>
/// Drains from the head while both the count cap and a byte budget allow it, so the caller can
/// build a reply frame that is guaranteed to fit the negotiated frame maximum.
/// </summary>
/// <remarks>
/// The size decision happens inside the queue lock, so an event is dequeued only once it is
/// known to fit: an event that does not fit stays at the head for the next call and is never
/// lost (WRK-21). Per-event cost is <c>WorkerEvent.CalculateSize()</c> plus
/// <see cref="RepeatedFieldOverheadBytes"/>; the <see cref="WorkerEvent"/> wrapper slightly
/// overestimates the packed <c>MxEvent</c> and the constant conservatively covers the
/// repeated-field tag and length varint, so the estimate errs strictly on the safe side.
/// </remarks>
/// <param name="maxEvents">Maximum number of events to drain; 0 means "no count limit".</param>
/// <param name="maxTotalBytes">Byte budget for the drained events' estimated serialized size.</param>
/// <returns>The drained events plus the truncation facts the caller reports to the gateway.</returns>
public WorkerEventDrainResult Drain(uint maxEvents, int maxTotalBytes)
{
lock (syncRoot)
{
int countLimit = maxEvents == 0
? int.MaxValue
: checked((int)Math.Min(maxEvents, int.MaxValue));
List<WorkerEvent> drained = new();
int remainingBudget = maxTotalBytes;
bool truncatedBySize = false;
ulong oversizedHeadSequence = 0;
while (drained.Count < countLimit && events.Count > 0)
{
WorkerEvent head = events.Peek();
int cost = head.CalculateSize() + RepeatedFieldOverheadBytes;
if (cost > remainingBudget)
{
truncatedBySize = true;
if (cost > maxTotalBytes)
{
// The head alone cannot fit this budget, so repeating the call will not
// move it either. Report its sequence instead of silently stalling; the
// events that did fit are still returned.
oversizedHeadSequence = head.Event?.WorkerSequence ?? 0;
}
break;
}
remainingBudget -= cost;
drained.Add(events.Dequeue());
}
return new WorkerEventDrainResult(
drained,
truncatedBySize,
events.Count,
oversizedHeadSequence);
}
}
/// <summary>
/// Records a fault if one has not already been recorded.
/// </summary>
@@ -392,6 +392,12 @@ public sealed class MxAccessStaSession : IWorkerRuntimeSession
return eventQueue.Drain(maxEvents);
}
/// <inheritdoc />
public WorkerEventDrainResult DrainEvents(uint maxEvents, int maxTotalBytes)
{
return eventQueue.Drain(maxEvents, maxTotalBytes);
}
/// <inheritdoc />
public WorkerFault? DrainFault()
{
@@ -0,0 +1,56 @@
using System.Collections.Generic;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Worker.MxAccess;
/// <summary>
/// Outcome of a byte-budgeted drain from the MXAccess outbound event queue.
/// </summary>
/// <remarks>
/// A count cap alone cannot keep a <c>DrainEvents</c> reply inside the negotiated frame
/// maximum: byte-heavy events (large string or array <c>MxValue</c>s) overshoot the frame max
/// long before the count ceiling is reached, and the writer's per-frame rejection then
/// destroys events that were already removed from the queue. The byte-budgeted drain sizes
/// the reply while draining, so an event that does not fit is never dequeued (WRK-21), and
/// this result carries the truncation facts the reply's <c>DiagnosticMessage</c> reports —
/// no contract change is needed to express them.
/// Plain constructor and get-only properties: the worker targets .NET Framework 4.8, which
/// has no init-only members or positional records.
/// </remarks>
public sealed class WorkerEventDrainResult
{
/// <summary>Initializes a new instance of the <see cref="WorkerEventDrainResult"/> class.</summary>
/// <param name="events">Events removed from the queue, in enqueue order.</param>
/// <param name="truncatedBySize">Whether the byte budget, not the count cap, ended the drain.</param>
/// <param name="remainingCount">Number of events still queued after the drain.</param>
/// <param name="oversizedHeadSequence">
/// Worker sequence of a head event whose own serialized size exceeds the whole budget, so
/// no future call of the same budget can ship it; 0 when there is no such event.
/// </param>
public WorkerEventDrainResult(
IReadOnlyList<WorkerEvent> events,
bool truncatedBySize,
int remainingCount,
ulong oversizedHeadSequence)
{
Events = events;
TruncatedBySize = truncatedBySize;
RemainingCount = remainingCount;
OversizedHeadSequence = oversizedHeadSequence;
}
/// <summary>Gets the events removed from the queue, in enqueue order.</summary>
public IReadOnlyList<WorkerEvent> Events { get; }
/// <summary>Gets a value indicating whether the byte budget ended the drain early.</summary>
public bool TruncatedBySize { get; }
/// <summary>Gets the number of events still queued after the drain.</summary>
public int RemainingCount { get; }
/// <summary>
/// Gets the worker sequence of the head event that alone exceeds the byte budget, or 0 when
/// no single event blocks the drain. Naming it lets an operator find the offending tag.
/// </summary>
public ulong OversizedHeadSequence { get; }
}