fix(IPC-33): bound the DrainEvents byte budget against the gRPC send cap, not just the pipe frame max
The worker budgeted a DrainEvents reply against the negotiated pipe frame maximum less one 64 KiB reserve, but the gateway returns that same MxCommandReply verbatim through Invoke, where MaxSendMessageSize applies. The two limits were coupled only from below (pipe >= gRPC + reserve), so any configuration the validator permitted with a wider gap let the worker pack a reply the pipe carried and the gRPC layer then refused with ResourceExhausted -- after Drain had already removed the events from the worker queue, losing them to both the caller and the event stream. Option (a) from the finding: close the coupling at both ends. - GatewayOptionsValidator adds the upper half of the headroom rule (Worker:MaxMessageBytes <= Protocol:MaxGrpcMessageBytes + 2 x reserve), with a message naming the DrainEvents budget as the reason and the two knobs to move. - WorkerPipeSession.ResolveDrainReplyByteBudget subtracts the reserve twice -- once for the pipe frame, once so the MxCommandReply wrapper clears the gRPC send cap -- keeping the MaxMessageBytes/2 floor that makes small frame maxima usable. The two changes are load-bearing for each other: the validator band is what makes "frame max - 2 x reserve" provably at or below the gRPC cap, and the doubled subtraction is what makes the band sufficient. Docs updated in the same change (GatewayConfiguration.md both option rows plus the headroom prose, WorkerFrameProtocol.md drain budget, gateway.md DrainEvents byte axis). Tests: new GatewayOptionsValidatorTests cases for the upper bound (reject) and both edges of the band (accept); GatewayOptionsTests' override case now moves both message-size knobs together, since the band makes moving one alone illegal. The worker half cannot be built on macOS; verified on windev separately.
This commit is contained in:
@@ -123,7 +123,7 @@ launch CWD (SEC-01, SEC-33).
|
||||
| `MxGateway:Worker:ShutdownTimeoutSeconds` | `10` | Grace period for worker shutdown before the gateway treats shutdown as failed and may kill the worker process tree. |
|
||||
| `MxGateway:Worker:HeartbeatIntervalSeconds` | `5` | Worker heartbeat send interval and gateway heartbeat check cadence input. |
|
||||
| `MxGateway:Worker:HeartbeatGraceSeconds` | `15` | Maximum age of the last worker heartbeat before the gateway faults the worker. This must be greater than or equal to `HeartbeatIntervalSeconds`. |
|
||||
| `MxGateway:Worker:MaxMessageBytes` | `16842752` | Maximum worker IPC frame payload size in bytes. The validator allows values from `1024` through `268435456`, and additionally requires this to be at least `MxGateway:Protocol:MaxGrpcMessageBytes` plus a 65536-byte (64 KiB) envelope-overhead reserve. The default is the 16 MiB public gRPC cap plus that reserve. The gateway conveys the negotiated value to the worker in the handshake (`GatewayHello.max_frame_bytes`), so the worker frames to the same limit rather than a hard-coded default. |
|
||||
| `MxGateway:Worker:MaxMessageBytes` | `16842752` | Maximum worker IPC frame payload size in bytes. The validator allows values from `1024` through `268435456`, and additionally requires this to be at least `MxGateway:Protocol:MaxGrpcMessageBytes` plus a 65536-byte (64 KiB) envelope-overhead reserve **and at most that cap plus twice the reserve**. The default is the 16 MiB public gRPC cap plus one reserve. The gateway conveys the negotiated value to the worker in the handshake (`GatewayHello.max_frame_bytes`), so the worker frames to the same limit rather than a hard-coded default. The band is narrow by design, so changing either message-size knob alone no longer validates — move `MxGateway:Protocol:MaxGrpcMessageBytes` and this one together, keeping this one one to two reserves above it. |
|
||||
|
||||
`StartupProbeRetryAttempts`, `StartupProbeRetryDelayMilliseconds`,
|
||||
`PipeConnectAttemptTimeoutMilliseconds`, timeout values, heartbeat values, and
|
||||
@@ -132,7 +132,15 @@ to avoid accidental large allocations from malformed or oversized frames. It
|
||||
must also stay above `MaxGrpcMessageBytes` by the envelope-overhead reserve so a
|
||||
maximally-sized accepted gRPC payload always fits one worker frame once wrapped
|
||||
in a `WorkerEnvelope`; otherwise the oversized outbound write would fault the
|
||||
whole session. Startup validation fails fast when the headroom is violated.
|
||||
whole session. The coupling is two-sided: `MaxMessageBytes` may exceed
|
||||
`MaxGrpcMessageBytes` by at most *twice* the reserve, because the worker budgets
|
||||
a `DrainEvents` reply against the frame maximum less two reserves and the
|
||||
gateway then returns that same `MxCommandReply` through `Invoke`, where the gRPC
|
||||
send cap applies. Without the upper bound a legal configuration (large pipe
|
||||
maximum, stock gRPC cap) lets the worker pack a reply the pipe carries and the
|
||||
gRPC layer refuses to send — with the drained events already removed from the
|
||||
worker queue and therefore lost to both the caller and the event stream. Startup
|
||||
validation fails fast when either half of the headroom rule is violated.
|
||||
|
||||
## Session Options
|
||||
|
||||
@@ -385,7 +393,7 @@ is **not yet available**.
|
||||
| Option | Default | Description |
|
||||
|--------|---------|-------------|
|
||||
| `MxGateway:Protocol:WorkerProtocolVersion` | `1` | Worker IPC protocol version expected by the gateway and worker. This must match `GatewayContractInfo.WorkerProtocolVersion`. |
|
||||
| `MxGateway:Protocol:MaxGrpcMessageBytes` | `16777216` | Public gRPC max send and receive message size in bytes. The same default is used by official clients. The validator allows values from `1024` through `268435456`, and requires `MxGateway:Worker:MaxMessageBytes` to exceed this by at least the 64 KiB worker-frame envelope reserve (see the Worker options). |
|
||||
| `MxGateway:Protocol:MaxGrpcMessageBytes` | `16777216` | Public gRPC max send and receive message size in bytes. The same default is used by official clients. The validator allows values from `1024` through `268435456`, and requires `MxGateway:Worker:MaxMessageBytes` to exceed this by at least the 64 KiB worker-frame envelope reserve and by no more than twice that reserve, so a `DrainEvents` reply budgeted at the pipe maximum still fits this send cap (see the Worker options). |
|
||||
|
||||
The protocol option is exposed for diagnostics and explicit deployment
|
||||
configuration, not for compatibility negotiation. A mismatch fails validation
|
||||
|
||||
@@ -43,9 +43,23 @@ guarantees hellos, heartbeats, acks, and faults fit.
|
||||
Every worker-to-gateway frame must serialize within this limit, control replies
|
||||
included, so reply builders truncate to fit rather than emit a frame the writer
|
||||
will reject. `WorkerPipeSession` pre-sizes a `DrainEvents` reply below the
|
||||
negotiated maximum (less a fixed envelope/reply-wrapper reserve) and reports the
|
||||
truncation in the reply's `DiagnosticMessage`; the caller contract is to repeat
|
||||
`DrainEvents` until it returns an empty reply. Should a reply still overshoot,
|
||||
negotiated maximum and reports the truncation in the reply's
|
||||
`DiagnosticMessage`; the caller contract is to repeat `DrainEvents` until it
|
||||
returns an empty reply. The event budget is `max(MaxMessageBytes - 2 x 64 KiB,
|
||||
MaxMessageBytes / 2)`: the reserve is subtracted **twice** because the reply has
|
||||
two boundaries to clear, not one. The first reserve covers the
|
||||
`WorkerEnvelope`/`WorkerCommandReply`/`MxCommandReply` wrapper on the pipe; the
|
||||
second exists because the gateway returns that same `MxCommandReply` verbatim
|
||||
through `Invoke`, where the public gRPC send cap applies. `DrainEvents` removes
|
||||
the events it packs from the worker queue, so a reply the pipe carries and the
|
||||
gRPC layer then refuses would lose them for good — the gateway's validator
|
||||
therefore also caps `MxGateway:Worker:MaxMessageBytes` at
|
||||
`MxGateway:Protocol:MaxGrpcMessageBytes` plus **two** reserves, which is what
|
||||
makes the doubled subtraction sufficient rather than merely generous. The
|
||||
`MaxMessageBytes / 2` floor keeps the budget usable at frame maxima small enough
|
||||
that two reserves would consume them (below 256 KiB), where a collapsed budget
|
||||
would truncate every drain and never let the drain-until-empty caller finish.
|
||||
Should a reply still overshoot,
|
||||
`MessageTooLarge` at the reply-write seam is answered with a small
|
||||
`InvalidRequest` reply for that correlation, not with session teardown — no
|
||||
diagnostics command may kill a session.
|
||||
|
||||
Reference in New Issue
Block a user