Joseph Doherty
33ba612ddd
WRK-21 — DrainEvents was bounded by event count only, so a byte-heavy queue (large string/array MxValues) built a reply above the negotiated frame maximum: the writer rejected the frame, the exception unwound the session, and the events already dequeued were destroyed. The drain is now byte-budgeted inside the queue lock, so an event is dequeued only once it is known to fit and one that does not stays at the head. Truncation is reported through the reply's existing DiagnosticMessage (no contract change); callers drain until an empty reply. Both reply-write seams — the control-command path and ProcessCommandAsync — now catch MessageTooLarge and answer the correlation with an InvalidRequest reply instead of unwinding or faulting the session. Satisfies IPC-23 R1-R3. WRK-28 — the 10,000 drain ceiling moves to GatewayContractInfo .MaxDrainEventsPerCommand, referenced by both the gateway request validator and the worker clamp, replacing a comment-only sync contract. C# const only; no .proto change. WRK-23 — WorkerFrameWriter now peek-stamps, validates, then commits the sequence counter immediately before the stream write, so a per-frame rejection leaves no phantom gap on the wire. IPC-30 — an oversized event frame stays session-fatal (it is undeliverable end to end and neither dropping nor synthesizing a replacement is allowed), but the death is structured: the event's identity and sizes are logged (never its value), a WorkerFault with category PROTOCOL_VIOLATION and command method EventDrain is written, then the session exits as before. Docs updated in the same change: MxAccessWorkerInstanceDesign.md (drain byte cap, truncation contract, oversized-head behavior, oversized-event policy, no control reply is session-fatal on size), WorkerFrameProtocol.md (reply pre-sizing, non-fatal reply-size rule, oversized-event policy, rejected frames do not consume sequence numbers), gateway.md (DrainEvents two-axis bound).
4.2 KiB
Worker Frame Protocol
The gateway uses the worker frame protocol to move WorkerEnvelope protobuf
messages over a bidirectional named pipe. The frame layer is deliberately small:
it handles message boundaries, size limits, protobuf parsing, and envelope
validation before higher-level worker client code routes commands, replies,
events, and faults.
Frame Format
Each frame starts with a four-byte little-endian unsigned payload length,
followed by the serialized WorkerEnvelope payload:
uint32 little-endian payload_length
payload_length bytes protobuf WorkerEnvelope
The reader rejects zero-length payloads and payloads larger than the configured
maximum before allocating the payload buffer. The default maximum is the 16 MiB
public gRPC cap plus a 64 KiB envelope-overhead reserve (16842752 bytes) so a
maximally-sized accepted gRPC payload always fits one worker frame once wrapped
in a WorkerEnvelope.
The gateway is the source of truth for this maximum: it conveys the negotiated
value in the handshake as GatewayHello.max_frame_bytes, and the worker adopts
it as its WorkerFrameProtocolOptions.MaxMessageBytes instead of a hard-coded
default. A max_frame_bytes of 0 (an older gateway that never set the field)
means "use the worker's built-in default". This keeps both ends framing to the
same limit rather than depending on matched compile-time constants.
Every worker-to-gateway frame must serialize within this limit, control replies
included, so reply builders truncate to fit rather than emit a frame the writer
will reject. WorkerPipeSession pre-sizes a DrainEvents reply below the
negotiated maximum (less a fixed envelope/reply-wrapper reserve) and reports the
truncation in the reply's DiagnosticMessage; the caller contract is to repeat
DrainEvents until it returns an empty reply. Should a reply still overshoot,
MessageTooLarge at the reply-write seam is answered with a small
InvalidRequest reply for that correlation, not with session teardown — no
diagnostics command may kill a session.
An oversized event frame is the deliberate exception. Such an event is
undeliverable end to end (the pipe maximum sits only the envelope-overhead
reserve above the public gRPC cap), so the session faults: the worker logs the
event's identity and sizes — never its value — writes a WorkerFault with
category ProtocolViolation and command method EventDrain, then exits.
Remediation is raising MxGateway:Worker:MaxMessageBytes for that workload.
A per-frame rejection does not consume an envelope sequence. The writer stamps
a candidate sequence, runs the empty-payload and size checks against the stamped
envelope, and commits the counter only immediately before the stream write, so
the sequences observed on the wire stay contiguous across rejections and an
operator reading a pipe capture never sees a phantom gap.
Envelope Validation
WorkerFrameReader and WorkerFrameWriter validate each envelope against the
owning session before returning or writing it:
protocol_versionmust match the configured worker protocol version,session_idmust match the owning gateway session,- the envelope must contain one typed
bodyvalue.
Protocol violations throw WorkerFrameProtocolException with a
WorkerFrameProtocolErrorCode so callers can distinguish malformed frames,
oversized frames, protocol version mismatches, and session mismatches.
Verification
The frame protocol lives in ZB.MOM.WW.MxGateway.Worker.Ipc (WorkerFrameReader,
WorkerFrameWriter, WorkerFrameProtocolOptions) and is covered by
src/ZB.MOM.WW.MxGateway.Worker.Tests/Ipc/WorkerFrameProtocolTests.cs. The worker is an
x86 process, so build and test it with -p:Platform=x86.
Run the focused tests after changing the frame protocol:
dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj -p:Platform=x86 --filter WorkerFrameProtocolTests
Run the x86 worker build because the frame protocol is part of
ZB.MOM.WW.MxGateway.Worker:
dotnet build src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj -p:Platform=x86