fix(IPC-33): bound the DrainEvents byte budget against the gRPC send cap, not just the pipe frame max

The worker budgeted a DrainEvents reply against the negotiated pipe frame
maximum less one 64 KiB reserve, but the gateway returns that same
MxCommandReply verbatim through Invoke, where MaxSendMessageSize applies. The
two limits were coupled only from below (pipe >= gRPC + reserve), so any
configuration the validator permitted with a wider gap let the worker pack a
reply the pipe carried and the gRPC layer then refused with ResourceExhausted --
after Drain had already removed the events from the worker queue, losing them to
both the caller and the event stream.

Option (a) from the finding: close the coupling at both ends.

- GatewayOptionsValidator adds the upper half of the headroom rule
  (Worker:MaxMessageBytes <= Protocol:MaxGrpcMessageBytes + 2 x reserve), with a
  message naming the DrainEvents budget as the reason and the two knobs to move.
- WorkerPipeSession.ResolveDrainReplyByteBudget subtracts the reserve twice --
  once for the pipe frame, once so the MxCommandReply wrapper clears the gRPC
  send cap -- keeping the MaxMessageBytes/2 floor that makes small frame maxima
  usable.

The two changes are load-bearing for each other: the validator band is what
makes "frame max - 2 x reserve" provably at or below the gRPC cap, and the
doubled subtraction is what makes the band sufficient.

Docs updated in the same change (GatewayConfiguration.md both option rows plus
the headroom prose, WorkerFrameProtocol.md drain budget, gateway.md DrainEvents
byte axis). Tests: new GatewayOptionsValidatorTests cases for the upper bound
(reject) and both edges of the band (accept); GatewayOptionsTests' override case
now moves both message-size knobs together, since the band makes moving one
alone illegal.

The worker half cannot be built on macOS; verified on windev separately.

src/ZB.MOM.WW.MxGateway.Server/Configuration/GatewayOptionsValidator.cs

@@ -606,6 +606,15 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
     // wrapped in a WorkerEnvelope and the outbound write faults the whole session. Fail fast
     // at startup rather than mid-traffic. Only checked when both knobs are themselves in range so the
     // message is not doubled up with the individual range errors.
+    //
+    // IPC-33: the coupling also has an upper half. The worker budgets a DrainEvents reply against the
+    // negotiated frame maximum less two reserves, and that same MxCommandReply is then returned
+    // verbatim through Invoke, where the gateway's gRPC MaxSendMessageSize (= MaxGrpcMessageBytes)
+    // applies. Letting the pipe maximum run arbitrarily far above the gRPC cap lets the worker pack a
+    // reply the pipe carries and the gRPC layer then refuses to send — with the drained events already
+    // removed from the worker queue, so they are lost to both the caller and the event stream. Keeping
+    // the pipe maximum within two reserves of the gRPC cap makes "frame max - 2x reserve" a budget that
+    // is always at or below the gRPC cap, so the wrapper always has a full reserve of margin.
     private static void ValidateFrameSizeHeadroom(
         WorkerOptions worker,
         ProtocolOptions protocol,
@@ -628,6 +637,20 @@ public sealed class GatewayOptionsValidator : OptionsValidatorBase<GatewayOption
                 + $"{WorkerFrameProtocolOptions.EnvelopeOverheadReserveBytes}-byte worker-frame envelope reserve "
                 + $"(>= {requiredWorkerMax}).");
         }
+
+        long permittedWorkerMax =
+            (long)protocol.MaxGrpcMessageBytes + (2L * WorkerFrameProtocolOptions.EnvelopeOverheadReserveBytes);
+        if (worker.MaxMessageBytes > permittedWorkerMax)
+        {
+            builder.Add(
+                $"MxGateway:Worker:MaxMessageBytes ({worker.MaxMessageBytes}) must be at most "
+                + $"MxGateway:Protocol:MaxGrpcMessageBytes ({protocol.MaxGrpcMessageBytes}) plus twice the "
+                + $"{WorkerFrameProtocolOptions.EnvelopeOverheadReserveBytes}-byte worker-frame envelope reserve "
+                + $"(<= {permittedWorkerMax}). The worker budgets a DrainEvents reply against the negotiated "
+                + "frame maximum less two reserves; a pipe maximum further above the gRPC cap lets the worker "
+                + "pack a reply the pipe carries but the gRPC send cap refuses, losing the already-drained "
+                + "events. Lower MxGateway:Worker:MaxMessageBytes or raise MxGateway:Protocol:MaxGrpcMessageBytes.");
+        }
     }
 
     private static void AddIfBlank(string? value, string message, ValidationBuilder builder)

src/ZB.MOM.WW.MxGateway.Server/Configuration/WorkerOptions.cs

@@ -58,9 +58,12 @@ public sealed class WorkerOptions
     /// The maximum worker-frame (pipe) message size in bytes. Must stay at least
     /// <see cref="Workers.WorkerFrameProtocolOptions.EnvelopeOverheadReserveBytes"/> above
     /// <see cref="ProtocolOptions.MaxGrpcMessageBytes"/> so a maximally-sized accepted gRPC payload
-    /// still fits one worker frame; the gateway conveys this value to the worker in the
-    /// handshake (<c>GatewayHello.max_frame_bytes</c>). Default is the 16 MB public gRPC cap
-    /// plus that reserve.
+    /// still fits one worker frame, and at most <em>twice</em> that reserve above it so a
+    /// <c>DrainEvents</c> reply budgeted against this maximum (frame max less two reserves) still
+    /// fits the gRPC send cap it is returned through — otherwise the worker can pack a reply the
+    /// pipe carries and the gRPC layer refuses, losing the already-drained events (IPC-33). The
+    /// gateway conveys this value to the worker in the handshake
+    /// (<c>GatewayHello.max_frame_bytes</c>). Default is the 16 MB public gRPC cap plus one reserve.
     /// </summary>
     public int MaxMessageBytes { get; init; } =
         (16 * 1024 * 1024) + Workers.WorkerFrameProtocolOptions.EnvelopeOverheadReserveBytes;

src/ZB.MOM.WW.MxGateway.Tests/Configuration/GatewayOptionsTests.cs

@@ -86,6 +86,11 @@ public sealed class GatewayOptionsTests
                 ["MxGateway:Events:QueueCapacity"] = "256",
                 ["MxGateway:Dashboard:Enabled"] = "false",
                 ["MxGateway:Protocol:MaxGrpcMessageBytes"] = "8388608",
+                // The frame-headroom rule is two-sided (IPC-33): the worker-frame maximum must sit
+                // between one and two 64 KiB envelope reserves above the gRPC cap, so moving the
+                // gRPC cap without moving this one no longer validates. Kept at one reserve above,
+                // the same shape as the shipped default.
+                ["MxGateway:Worker:MaxMessageBytes"] = "8454144",
                 // Blank-by-default bind password must be supplied so enabled LDAP validates on bind.
                 ["MxGateway:Ldap:ServiceAccountPassword"] = "test-bind-password"
             });

src/ZB.MOM.WW.MxGateway.Tests/Configuration/GatewayOptionsValidatorTests.cs

@@ -1,5 +1,6 @@
 using Microsoft.Extensions.Options;
 using ZB.MOM.WW.MxGateway.Server.Configuration;
+using ZB.MOM.WW.MxGateway.Server.Workers;
 using LdapTransport = ZB.MOM.WW.Auth.Abstractions.Ldap.LdapTransport;
 
 namespace ZB.MOM.WW.MxGateway.Tests.Configuration;
@@ -1256,6 +1257,57 @@ public sealed class GatewayOptionsValidatorTests
         Assert.Contains(result.Failures!, f => f.Contains("MaxMessageBytes") && f.Contains("reserve"));
     }
 
+    /// <summary>
+    /// Verifies the headroom rule's upper half (IPC-33): a worker-frame maximum more than two
+    /// envelope reserves above the gRPC cap is rejected. The worker budgets a DrainEvents reply at
+    /// the frame maximum less two reserves and the gateway returns that same reply through
+    /// <c>Invoke</c>, so a wider gap lets the worker pack a reply the pipe carries and the gRPC send
+    /// cap refuses — with the drained events already removed from the worker queue.
+    /// </summary>
+    [Fact]
+    public void Validate_Fails_WhenWorkerFrameMaxExceedsGrpcMaxByMoreThanTwoReserves()
+    {
+        const int grpcMax = 16 * 1024 * 1024;
+        int tooLarge = grpcMax + (2 * WorkerFrameProtocolOptions.EnvelopeOverheadReserveBytes) + 1;
+
+        ValidateOptionsResult result = new GatewayOptionsValidator().Validate(
+            null,
+            WithWorkerAndProtocol(
+                new WorkerOptions { MaxMessageBytes = tooLarge },
+                new ProtocolOptions { MaxGrpcMessageBytes = grpcMax }));
+
+        Assert.True(result.Failed);
+        Assert.Contains(
+            result.Failures!,
+            f => f.Contains("MaxMessageBytes", StringComparison.Ordinal)
+                && f.Contains("at most", StringComparison.Ordinal)
+                && f.Contains("DrainEvents", StringComparison.Ordinal));
+    }
+
+    /// <summary>
+    /// Verifies both edges of the two-sided headroom rule are themselves accepted: exactly one
+    /// reserve above the gRPC cap (the shipped default's shape) and exactly two reserves above it
+    /// (the widest legal gap). Pinning the boundaries keeps a future retune of the reserve from
+    /// silently making the shipped default illegal.
+    /// </summary>
+    /// <param name="reserveMultiple">Number of envelope reserves between the gRPC cap and the frame maximum.</param>
+    [Theory]
+    [InlineData(1)]
+    [InlineData(2)]
+    public void Validate_Succeeds_AtBothEdgesOfTheFrameHeadroomBand(int reserveMultiple)
+    {
+        const int grpcMax = 16 * 1024 * 1024;
+        int workerMax = grpcMax + (reserveMultiple * WorkerFrameProtocolOptions.EnvelopeOverheadReserveBytes);
+
+        ValidateOptionsResult result = new GatewayOptionsValidator().Validate(
+            null,
+            WithWorkerAndProtocol(
+                new WorkerOptions { MaxMessageBytes = workerMax },
+                new ProtocolOptions { MaxGrpcMessageBytes = grpcMax }));
+
+        Assert.True(result.Succeeded);
+    }
+
     /// <summary>
     /// Verifies the shipped worker event-queue capacity default passes validation and still matches
     /// the worker-side <c>MxAccessEventQueue.DefaultCapacity</c> the environment variable falls back

src/ZB.MOM.WW.MxGateway.Worker.Tests/Ipc/WorkerPipeSessionTests.cs

@@ -692,7 +692,7 @@ public sealed class WorkerPipeSessionTests
     /// <summary>
     ///     Regression for the reserve-boundary budget bug. The gateway validator accepts a Worker
     ///     frame maximum as low as 1024 + 64 KiB, and just above that boundary a naive
-    ///     subtract-then-guard budget collapses to ~1024 bytes — too small to move even one
+    ///     subtract-then-guard budget collapses to nothing — too small to move even one
     ///     byte-heavy event, so every drain reports truncation with the same head blocked and the
     ///     drain-until-empty loop never terminates. The budget is now a floor (never below half the
     ///     negotiated maximum), so a byte-heavy queue drains to empty even at the validator floor.
@@ -702,8 +702,9 @@ public sealed class WorkerPipeSessionTests
     public async Task DrainEvents_AtValidatorFloorFrameMax_MakesProgressAndTerminates()
     {
         // The lowest Worker.MaxMessageBytes GatewayOptionsValidator permits: the public gRPC floor
-        // (1024) plus the 64 KiB envelope-overhead reserve. The naive budget would be exactly 1024
-        // here; the floored budget is half of the frame max (~33 KiB).
+        // (1024) plus the 64 KiB envelope-overhead reserve. The naive budget — frame max less the
+        // two reserves the DrainEvents budget subtracts (IPC-33) — would be negative here; the
+        // floored budget is half of the frame max (~33 KiB).
         const uint validatorFloorFrameMax = 1024 + (64 * 1024);
         using CancellationTokenSource cancellation = new(TimeSpan.FromSeconds(60));
         using PipePair pipePair = await PipePair.CreateAsync(cancellation.Token);

src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerPipeSession.cs

@@ -27,7 +27,8 @@ public sealed class WorkerPipeSession
     // Headroom subtracted from the negotiated frame maximum when budgeting a DrainEvents reply. It
     // covers the WorkerEnvelope/WorkerCommandReply/MxCommandReply wrapper the drained events are
     // packed into — the same envelope-overhead reserve rationale docs/WorkerFrameProtocol.md
-    // records for the frame max itself.
+    // records for the frame max itself. ResolveDrainReplyByteBudget subtracts it twice: once for the
+    // pipe frame, once so the reply also clears the public gRPC send cap it is forwarded through.
     private const int DrainReplyFrameHeadroomBytes = 64 * 1024;
 
     private readonly WorkerFrameProtocolOptions _options;
@@ -1021,20 +1022,33 @@ public sealed class WorkerPipeSession
 
     /// <summary>
     ///     Byte budget for the events packed into one DrainEvents reply: the negotiated frame
-    ///     maximum less a fixed reserve for the envelope/reply wrapper, but never below half the
-    ///     negotiated maximum. The lower bound must be a floor, not a step: a bare
-    ///     <c>subtract-then-guard-positive</c> collapses the budget to a handful of bytes just above
-    ///     the reserve (e.g. at the validator-permitted floor MaxMessageBytes = 1024 + 64 KiB the
-    ///     subtraction leaves 1024, too small to move even one byte-heavy event, so every drain
-    ///     truncates and the drain-until-empty caller never terminates). Taking the max with
-    ///     half the negotiated maximum keeps the budget monotonic across the reserve boundary while
-    ///     still leaving the full reserve for the wrapper whenever the frame max is large enough
-    ///     that the reserve is the smaller subtraction — which is every configuration above 128 KiB.
+    ///     maximum less <em>two</em> reserves, but never below half the negotiated maximum. The
+    ///     first reserve covers the envelope/reply wrapper the events are packed into so the reply
+    ///     fits one pipe frame. The second reserve exists because that same
+    ///     <c>MxCommandReply</c> does not stop at the pipe: the gateway returns it verbatim through
+    ///     <c>Invoke</c>, where the public gRPC send cap applies, and the gateway's validator
+    ///     couples the two limits so the pipe maximum never exceeds the gRPC cap by more than two
+    ///     reserves (IPC-33). Subtracting both therefore keeps the packed events at or below the
+    ///     gRPC cap and leaves the wrapper — correlation id, protocol status, kind, the
+    ///     <c>drain_events</c> length prefix and, in the truncated case, the
+    ///     <c>DiagnosticMessage</c> — a full reserve of margin under it. Budgeting against the pipe
+    ///     maximum alone let a legal configuration build a reply the pipe carried and the gRPC layer
+    ///     then refused to send, with the events already dequeued and therefore lost.
+    ///     <para>
+    ///     The lower bound must be a floor, not a step: a bare <c>subtract-then-guard-positive</c>
+    ///     collapses the budget to a handful of bytes (or below zero) once the reserves approach the
+    ///     frame maximum (e.g. at the validator-permitted floor MaxMessageBytes = 1024 + 64 KiB),
+    ///     too small to move even one byte-heavy event, so every drain truncates and the
+    ///     drain-until-empty caller never terminates. Taking the max with half the negotiated
+    ///     maximum keeps the budget monotonic across the reserve boundary while still leaving both
+    ///     reserves for the wrapper whenever the frame max is large enough that the subtraction is
+    ///     the smaller of the two — which is every configuration above 256 KiB.
+    ///     </para>
     /// </summary>
     private int ResolveDrainReplyByteBudget()
     {
         return Math.Max(
-            _options.MaxMessageBytes - DrainReplyFrameHeadroomBytes,
+            _options.MaxMessageBytes - (2 * DrainReplyFrameHeadroomBytes),
             _options.MaxMessageBytes / 2);
     }