fix(WRK-21,WRK-28,WRK-23,IPC-30): byte-budget the DrainEvents reply, stop size errors from killing sessions
ci / nightly-windev (push) Has been skipped
ci / java (push) Successful in 2m8s
ci / portable (push) Successful in 7m41s
ci / windows-x86 (push) Failing after 12m32s

WRK-21 — DrainEvents was bounded by event count only, so a byte-heavy queue
(large string/array MxValues) built a reply above the negotiated frame maximum:
the writer rejected the frame, the exception unwound the session, and the events
already dequeued were destroyed. The drain is now byte-budgeted inside the queue
lock, so an event is dequeued only once it is known to fit and one that does not
stays at the head. Truncation is reported through the reply's existing
DiagnosticMessage (no contract change); callers drain until an empty reply. Both
reply-write seams — the control-command path and ProcessCommandAsync — now catch
MessageTooLarge and answer the correlation with an InvalidRequest reply instead
of unwinding or faulting the session. Satisfies IPC-23 R1-R3.

WRK-28 — the 10,000 drain ceiling moves to GatewayContractInfo
.MaxDrainEventsPerCommand, referenced by both the gateway request validator and
the worker clamp, replacing a comment-only sync contract. C# const only; no
.proto change.

WRK-23 — WorkerFrameWriter now peek-stamps, validates, then commits the sequence
counter immediately before the stream write, so a per-frame rejection leaves no
phantom gap on the wire.

IPC-30 — an oversized event frame stays session-fatal (it is undeliverable end to
end and neither dropping nor synthesizing a replacement is allowed), but the
death is structured: the event's identity and sizes are logged (never its value),
a WorkerFault with category PROTOCOL_VIOLATION and command method EventDrain is
written, then the session exits as before.

Docs updated in the same change: MxAccessWorkerInstanceDesign.md (drain byte cap,
truncation contract, oversized-head behavior, oversized-event policy, no control
reply is session-fatal on size), WorkerFrameProtocol.md (reply pre-sizing,
non-fatal reply-size rule, oversized-event policy, rejected frames do not consume
sequence numbers), gateway.md (DrainEvents two-axis bound).
This commit is contained in:
Joseph Doherty
2026-08-07 05:38:23 -04:00
parent ead921cace
commit 33ba612ddd
18 changed files with 1118 additions and 57 deletions
@@ -1,17 +1,11 @@
using Grpc.Core;
using ZB.MOM.WW.MxGateway.Contracts;
using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Server.Grpc;
public sealed class MxAccessGrpcRequestValidator
{
// Upper bound on a single DrainEvents request. DrainEvents is a diagnostics RPC that returns
// buffered events in one non-streaming reply, so an unbounded max_events could pack the whole
// queue into a session-killing frame. The worker independently caps each reply at its
// own MaxDrainEventsPerReply; this public bound rejects an obviously-abusive request loudly at
// the boundary. max_events = 0 is allowed and means "the worker's default batch cap".
private const uint MaxDrainEventsPerRequest = 10_000;
/// <summary>Validates an open session request.</summary>
/// <param name="request">The request to validate.</param>
public void ValidateOpenSession(OpenSessionRequest request)
@@ -78,10 +72,18 @@ public sealed class MxAccessGrpcRequestValidator
}
// The payload case now matches the kind, so command.DrainEvents is non-null here.
if (command.Kind is MxCommandKind.DrainEvents && command.DrainEvents.MaxEvents > MaxDrainEventsPerRequest)
// DrainEvents is a diagnostics RPC that returns buffered events in one non-streaming
// reply, so an unbounded max_events could pack the whole queue into a session-killing
// frame. The worker independently clamps every reply to the same shared ceiling and
// additionally caps it by serialized bytes; this public bound rejects an obviously-abusive
// request loudly at the boundary. max_events = 0 is allowed and means "the worker's
// default batch cap".
if (command.Kind is MxCommandKind.DrainEvents
&& command.DrainEvents.MaxEvents > GatewayContractInfo.MaxDrainEventsPerCommand)
{
throw InvalidArgument(
$"DrainEvents max_events ({command.DrainEvents.MaxEvents}) must not exceed {MaxDrainEventsPerRequest}; "
$"DrainEvents max_events ({command.DrainEvents.MaxEvents}) must not exceed "
+ $"{GatewayContractInfo.MaxDrainEventsPerCommand}; "
+ "use 0 to request the worker default batch cap.");
}
}