fix(WRK-21,WRK-28,WRK-23,IPC-30): byte-budget the DrainEvents reply, stop size errors from killing sessions

WRK-21 — DrainEvents was bounded by event count only, so a byte-heavy queue
(large string/array MxValues) built a reply above the negotiated frame maximum:
the writer rejected the frame, the exception unwound the session, and the events
already dequeued were destroyed. The drain is now byte-budgeted inside the queue
lock, so an event is dequeued only once it is known to fit and one that does not
stays at the head. Truncation is reported through the reply's existing
DiagnosticMessage (no contract change); callers drain until an empty reply. Both
reply-write seams — the control-command path and ProcessCommandAsync — now catch
MessageTooLarge and answer the correlation with an InvalidRequest reply instead
of unwinding or faulting the session. Satisfies IPC-23 R1-R3.

WRK-28 — the 10,000 drain ceiling moves to GatewayContractInfo
.MaxDrainEventsPerCommand, referenced by both the gateway request validator and
the worker clamp, replacing a comment-only sync contract. C# const only; no
.proto change.

WRK-23 — WorkerFrameWriter now peek-stamps, validates, then commits the sequence
counter immediately before the stream write, so a per-frame rejection leaves no
phantom gap on the wire.

IPC-30 — an oversized event frame stays session-fatal (it is undeliverable end to
end and neither dropping nor synthesizing a replacement is allowed), but the
death is structured: the event's identity and sizes are logged (never its value),
a WorkerFault with category PROTOCOL_VIOLATION and command method EventDrain is
written, then the session exits as before.

Docs updated in the same change: MxAccessWorkerInstanceDesign.md (drain byte cap,
truncation contract, oversized-head behavior, oversized-event policy, no control
reply is session-fatal on size), WorkerFrameProtocol.md (reply pre-sizing,
non-fatal reply-size rule, oversized-event policy, rejected frames do not consume
sequence numbers), gateway.md (DrainEvents two-axis bound).

src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs

@@ -5,6 +5,7 @@ using System.IO;
 using System.Threading;
 using System.Threading.Tasks;
 using Google.Protobuf.WellKnownTypes;
+using ZB.MOM.WW.MxGateway.Contracts;
 using ZB.MOM.WW.MxGateway.Contracts.Proto;
 using ZB.MOM.WW.MxGateway.Worker.Bootstrap;
 using ZB.MOM.WW.MxGateway.Worker.MxAccess;
@@ -18,12 +19,11 @@ public sealed class WorkerPipeSession
     private static readonly TimeSpan BackgroundTaskStopTimeout = TimeSpan.FromSeconds(1);
     private const uint EventDrainBatchSize = 128;
 
-    // Hard cap on how many events a single DrainEvents diagnostic reply may carry. DrainEvents is a
-    // non-streaming control command, so an unbounded drain (including the max_events = 0 "as many as
-    // available" request) could pack the whole queue into one session-killing reply frame.
-    // The gateway request validator rejects requests above its public ceiling; this worker-side cap is
-    // the backstop and defines the effective per-reply maximum. Kept in step with that public ceiling.
-    private const uint MaxDrainEventsPerReply = 10_000;
+    // Headroom subtracted from the negotiated frame maximum when budgeting a DrainEvents reply. It
+    // covers the WorkerEnvelope/WorkerCommandReply/MxCommandReply wrapper the drained events are
+    // packed into — the same envelope-overhead reserve rationale docs/WorkerFrameProtocol.md
+    // records for the frame max itself.
+    private const int DrainReplyFrameHeadroomBytes = 64 * 1024;
 
     private readonly WorkerFrameProtocolOptions _options;
     private readonly Func<int> _processIdProvider;
@@ -376,13 +376,83 @@ public sealed class WorkerPipeSession
                 // Events are the low-priority frame class: the writer holds them behind any pending
                 // control frame (reply, fault, heartbeat, shutdown ack) so those are not delayed
                 // behind an event backlog.
-                await _writer
-                    .WriteAsync(CreateEnvelope(workerEvent), WorkerFrameWritePriority.Event, cancellationToken)
-                    .ConfigureAwait(false);
+                try
+                {
+                    await _writer
+                        .WriteAsync(CreateEnvelope(workerEvent), WorkerFrameWritePriority.Event, cancellationToken)
+                        .ConfigureAwait(false);
+                }
+                catch (WorkerFrameProtocolException exception)
+                    when (exception.ErrorCode == WorkerFrameProtocolErrorCode.MessageTooLarge)
+                {
+                    await FaultOnOversizedEventAsync(workerEvent, exception, cancellationToken)
+                        .ConfigureAwait(false);
+                }
             }
         }
     }
 
+    /// <summary>
+    ///     Ends the session on an event that cannot be framed, but deliberately and diagnosably
+    ///     (IPC-30). An event above the negotiated frame maximum is undeliverable end to end — the
+    ///     pipe maximum sits only an envelope reserve above the public gRPC cap — so dropping it
+    ///     would silently make the event stream unfaithful, and synthesizing a placeholder is barred
+    ///     by the no-synthesized-events rule. Instead the worker records which event blocked (never
+    ///     its value: the redaction rule), writes a structured fault the gateway and dashboard can
+    ///     surface, and then exits as it did before. Remediation is configuration:
+    ///     <c>MxGateway:Worker:MaxMessageBytes</c>. Other per-frame rejection codes keep the previous
+    ///     behavior — they indicate worker bugs, not workload size.
+    /// </summary>
+    private async Task FaultOnOversizedEventAsync(
+        WorkerEvent workerEvent,
+        WorkerFrameProtocolException exception,
+        CancellationToken cancellationToken)
+    {
+        MxEvent? mxEvent = workerEvent.Event;
+        string family = (mxEvent?.Family ?? MxEventFamily.Unspecified).ToString();
+        ulong workerSequence = mxEvent?.WorkerSequence ?? 0;
+        int serverHandle = mxEvent?.ServerHandle ?? 0;
+        int itemHandle = mxEvent?.ItemHandle ?? 0;
+
+        _logger?.Error(
+            "WorkerEventFrameTooLarge",
+            new Dictionary<string, object?>
+            {
+                ["session_id"] = _options.SessionId,
+                ["event_family"] = family,
+                ["worker_sequence"] = workerSequence,
+                ["server_handle"] = serverHandle,
+                ["item_handle"] = itemHandle,
+                ["max_message_bytes"] = _options.MaxMessageBytes,
+                // Sizes only — the event value never reaches the log.
+                ["reason"] = exception.Message,
+            });
+
+        string diagnosticMessage =
+            $"{family} event for server handle {serverHandle}, item handle {itemHandle} "
+            + $"(worker sequence {workerSequence}) exceeds the negotiated frame maximum of "
+            + $"{_options.MaxMessageBytes} bytes and cannot be delivered; raise "
+            + "MxGateway:Worker:MaxMessageBytes for this workload.";
+
+        _state = WorkerState.Faulted;
+        await TryWriteFaultAsync(
+            new WorkerFault
+            {
+                Category = WorkerFaultCategory.ProtocolViolation,
+                CommandMethod = "EventDrain",
+                ExceptionType = exception.GetType().FullName ?? string.Empty,
+                DiagnosticMessage = diagnosticMessage,
+                ProtocolStatus = new ProtocolStatus
+                {
+                    Code = ProtocolStatusCode.ProtocolViolation,
+                    Message = diagnosticMessage,
+                },
+            },
+            cancellationToken).ConfigureAwait(false);
+
+        throw new InvalidOperationException(diagnosticMessage, exception);
+    }
+
     private async Task<bool> DispatchGatewayEnvelopeAsync(
         WorkerEnvelope envelope,
         CancellationToken cancellationToken)
@@ -478,7 +548,8 @@ public sealed class WorkerPipeSession
             _ => CreateControlOkReply(correlationId, command.Kind),
         };
 
-        await WriteControlReplyAsync(reply, cancellationToken).ConfigureAwait(false);
+        await WriteControlReplyWithSizeBackstopAsync(reply, correlationId, command.Kind, cancellationToken)
+            .ConfigureAwait(false);
         return true;
     }
 
@@ -495,6 +566,70 @@ public sealed class WorkerPipeSession
             cancellationToken);
     }
 
+    /// <summary>
+    ///     Writes a control reply, answering the correlation with a small error reply instead of
+    ///     unwinding the session if the reply does not fit the negotiated frame maximum. Reply
+    ///     builders already size their payloads (see <see cref="CreateDrainEventsReply"/>), so this
+    ///     is a backstop against a future command or a sizing bug — but without it a single
+    ///     oversized diagnostic reply is session-fatal, which no diagnostics command may be.
+    /// </summary>
+    private async Task WriteControlReplyWithSizeBackstopAsync(
+        MxCommandReply reply,
+        string correlationId,
+        MxCommandKind kind,
+        CancellationToken cancellationToken)
+    {
+        try
+        {
+            await WriteControlReplyAsync(reply, cancellationToken).ConfigureAwait(false);
+        }
+        catch (WorkerFrameProtocolException exception)
+            when (exception.ErrorCode == WorkerFrameProtocolErrorCode.MessageTooLarge)
+        {
+            LogControlReplyTooLarge(correlationId, kind, exception);
+            await WriteControlReplyAsync(
+                CreateReplyTooLargeReply(correlationId, kind),
+                cancellationToken).ConfigureAwait(false);
+        }
+    }
+
+    private void LogControlReplyTooLarge(
+        string correlationId,
+        MxCommandKind kind,
+        WorkerFrameProtocolException exception)
+    {
+        _logger?.Error(
+            "WorkerControlReplyTooLarge",
+            new Dictionary<string, object?>
+            {
+                ["correlation_id"] = correlationId,
+                ["command_kind"] = kind.ToString(),
+                ["max_message_bytes"] = _options.MaxMessageBytes,
+                // The writer's message carries the rejected payload length; it names sizes only,
+                // never reply content.
+                ["reason"] = exception.Message,
+            });
+    }
+
+    private MxCommandReply CreateReplyTooLargeReply(string correlationId, MxCommandKind kind)
+    {
+        const string message =
+            "Worker reply exceeded the negotiated frame maximum; retry with a smaller request.";
+        return new MxCommandReply
+        {
+            SessionId = _options.SessionId,
+            CorrelationId = correlationId,
+            Kind = kind,
+            Hresult = 0,
+            DiagnosticMessage = message,
+            ProtocolStatus = new ProtocolStatus
+            {
+                Code = ProtocolStatusCode.InvalidRequest,
+                Message = message,
+            },
+        };
+    }
+
     private MxCommandReply CreatePingReply(string correlationId, MxCommand command)
     {
         MxCommandReply reply = CreateControlOkReply(correlationId, command.Kind);
@@ -543,24 +678,71 @@ public sealed class WorkerPipeSession
         if (runtimeSession is not null)
         {
             // Bound the diagnostic drain so max_events = 0 ("as many as available") or an over-large
-            // request cannot pack the whole queue into one session-killing reply frame.
+            // request cannot pack the whole queue into one session-killing reply frame. The count cap
+            // alone is not enough: byte-heavy events overshoot the negotiated frame maximum long
+            // before the count ceiling, so the drain is also byte-budgeted and sizes the reply while
+            // draining — an event that does not fit is left queued rather than dequeued and lost.
             uint requested = command.DrainEvents?.MaxEvents ?? 0;
-            uint maxEvents = requested == 0 || requested > MaxDrainEventsPerReply
-                ? MaxDrainEventsPerReply
+            uint maxEvents = requested == 0 || requested > GatewayContractInfo.MaxDrainEventsPerCommand
+                ? GatewayContractInfo.MaxDrainEventsPerCommand
                 : requested;
-            foreach (WorkerEvent workerEvent in runtimeSession.DrainEvents(maxEvents))
+            WorkerEventDrainResult drainResult = runtimeSession.DrainEvents(
+                maxEvents,
+                ResolveDrainReplyByteBudget());
+            foreach (WorkerEvent workerEvent in drainResult.Events)
             {
                 if (workerEvent.Event is not null)
                 {
                     drainReply.Events.Add(workerEvent.Event);
                 }
             }
+
+            if (drainResult.TruncatedBySize)
+            {
+                // DrainEventsReply has no truncation field, and adding one would regenerate every
+                // language client for a diagnostic nicety. The reply's existing DiagnosticMessage
+                // carries the same information at zero contract cost; the caller contract is to
+                // repeat DrainEvents until it comes back empty.
+                reply.DiagnosticMessage = CreateDrainTruncationMessage(
+                    drainReply.Events.Count,
+                    drainResult);
+            }
         }
 
         reply.DrainEvents = drainReply;
         return reply;
     }
 
+    /// <summary>
+    ///     Byte budget for the events packed into one DrainEvents reply: the negotiated frame
+    ///     maximum less a fixed reserve for the envelope/reply wrapper. A negotiated maximum below
+    ///     the reserve would otherwise yield a non-positive budget and stall the drain forever, so
+    ///     a tiny frame maximum falls back to half of itself — still ample headroom for a wrapper
+    ///     measured in tens of bytes.
+    /// </summary>
+    private int ResolveDrainReplyByteBudget()
+    {
+        int budget = _options.MaxMessageBytes - DrainReplyFrameHeadroomBytes;
+        return budget > 0 ? budget : _options.MaxMessageBytes / 2;
+    }
+
+    private static string CreateDrainTruncationMessage(
+        int returnedCount,
+        WorkerEventDrainResult drainResult)
+    {
+        string message =
+            $"{returnedCount} events returned, {drainResult.RemainingCount} remain; "
+            + "repeat DrainEvents for the rest.";
+        if (drainResult.OversizedHeadSequence != 0)
+        {
+            message +=
+                $" The next event (worker sequence {drainResult.OversizedHeadSequence}) alone exceeds "
+                + "the negotiated frame maximum and cannot be drained; raise MxGateway:Worker:MaxMessageBytes.";
+        }
+
+        return message;
+    }
+
     private MxCommandReply CreateControlOkReply(string correlationId, MxCommandKind kind)
     {
         return new MxCommandReply
@@ -627,15 +809,29 @@ public sealed class WorkerPipeSession
                 return;
             }
 
-            await _writer
-                .WriteAsync(
-                    CreateEnvelope(new WorkerCommandReply
-                    {
-                        Reply = reply,
-                        CompletedTimestamp = Timestamp.FromDateTime(DateTime.UtcNow),
-                    }),
-                    cancellationToken)
-                .ConfigureAwait(false);
+            try
+            {
+                await _writer
+                    .WriteAsync(
+                        CreateEnvelope(new WorkerCommandReply
+                        {
+                            Reply = reply,
+                            CompletedTimestamp = Timestamp.FromDateTime(DateTime.UtcNow),
+                        }),
+                        cancellationToken)
+                    .ConfigureAwait(false);
+            }
+            catch (WorkerFrameProtocolException sizeException)
+                when (sizeException.ErrorCode == WorkerFrameProtocolErrorCode.MessageTooLarge)
+            {
+                // An oversized STA command reply is a property of that one command, not of the
+                // session. Answer the correlation with an error reply instead of falling into the
+                // generic catch below, which would fault the whole session for it.
+                LogControlReplyTooLarge(envelope.CorrelationId, command.Kind, sizeException);
+                await WriteControlReplyAsync(
+                    CreateReplyTooLargeReply(envelope.CorrelationId, command.Kind),
+                    cancellationToken).ConfigureAwait(false);
+            }
         }
         catch (Exception exception) when (exception is not OperationCanceledException)
         {