Joseph Doherty
33ba612ddd
WRK-21 — DrainEvents was bounded by event count only, so a byte-heavy queue (large string/array MxValues) built a reply above the negotiated frame maximum: the writer rejected the frame, the exception unwound the session, and the events already dequeued were destroyed. The drain is now byte-budgeted inside the queue lock, so an event is dequeued only once it is known to fit and one that does not stays at the head. Truncation is reported through the reply's existing DiagnosticMessage (no contract change); callers drain until an empty reply. Both reply-write seams — the control-command path and ProcessCommandAsync — now catch MessageTooLarge and answer the correlation with an InvalidRequest reply instead of unwinding or faulting the session. Satisfies IPC-23 R1-R3. WRK-28 — the 10,000 drain ceiling moves to GatewayContractInfo .MaxDrainEventsPerCommand, referenced by both the gateway request validator and the worker clamp, replacing a comment-only sync contract. C# const only; no .proto change. WRK-23 — WorkerFrameWriter now peek-stamps, validates, then commits the sequence counter immediately before the stream write, so a per-frame rejection leaves no phantom gap on the wire. IPC-30 — an oversized event frame stays session-fatal (it is undeliverable end to end and neither dropping nor synthesizing a replacement is allowed), but the death is structured: the event's identity and sizes are logged (never its value), a WorkerFault with category PROTOCOL_VIOLATION and command method EventDrain is written, then the session exits as before. Docs updated in the same change: MxAccessWorkerInstanceDesign.md (drain byte cap, truncation contract, oversized-head behavior, oversized-event policy, no control reply is session-fatal on size), WorkerFrameProtocol.md (reply pre-sizing, non-fatal reply-size rule, oversized-event policy, rejected frames do not consume sequence numbers), gateway.md (DrainEvents two-axis bound).
92 lines
4.2 KiB
Markdown
92 lines
4.2 KiB
Markdown
# Worker Frame Protocol
|
|
|
|
The gateway uses the worker frame protocol to move `WorkerEnvelope` protobuf
|
|
messages over a bidirectional named pipe. The frame layer is deliberately small:
|
|
it handles message boundaries, size limits, protobuf parsing, and envelope
|
|
validation before higher-level worker client code routes commands, replies,
|
|
events, and faults.
|
|
|
|
## Frame Format
|
|
|
|
Each frame starts with a four-byte little-endian unsigned payload length,
|
|
followed by the serialized `WorkerEnvelope` payload:
|
|
|
|
```text
|
|
uint32 little-endian payload_length
|
|
payload_length bytes protobuf WorkerEnvelope
|
|
```
|
|
|
|
The reader rejects zero-length payloads and payloads larger than the configured
|
|
maximum before allocating the payload buffer. The default maximum is the 16 MiB
|
|
public gRPC cap plus a 64 KiB envelope-overhead reserve (16842752 bytes) so a
|
|
maximally-sized accepted gRPC payload always fits one worker frame once wrapped
|
|
in a `WorkerEnvelope`.
|
|
|
|
The gateway is the source of truth for this maximum: it conveys the negotiated
|
|
value in the handshake as `GatewayHello.max_frame_bytes`, and the worker adopts
|
|
it as its `WorkerFrameProtocolOptions.MaxMessageBytes` instead of a hard-coded
|
|
default. A `max_frame_bytes` of 0 (an older gateway that never set the field)
|
|
means "use the worker's built-in default". This keeps both ends framing to the
|
|
same limit rather than depending on matched compile-time constants.
|
|
|
|
Every worker-to-gateway frame must serialize within this limit, control replies
|
|
included, so reply builders truncate to fit rather than emit a frame the writer
|
|
will reject. `WorkerPipeSession` pre-sizes a `DrainEvents` reply below the
|
|
negotiated maximum (less a fixed envelope/reply-wrapper reserve) and reports the
|
|
truncation in the reply's `DiagnosticMessage`; the caller contract is to repeat
|
|
`DrainEvents` until it returns an empty reply. Should a reply still overshoot,
|
|
`MessageTooLarge` at the reply-write seam is answered with a small
|
|
`InvalidRequest` reply for that correlation, not with session teardown — no
|
|
diagnostics command may kill a session.
|
|
|
|
An oversized *event* frame is the deliberate exception. Such an event is
|
|
undeliverable end to end (the pipe maximum sits only the envelope-overhead
|
|
reserve above the public gRPC cap), so the session faults: the worker logs the
|
|
event's identity and sizes — never its value — writes a `WorkerFault` with
|
|
category `ProtocolViolation` and command method `EventDrain`, then exits.
|
|
Remediation is raising `MxGateway:Worker:MaxMessageBytes` for that workload.
|
|
|
|
A per-frame rejection does not consume an envelope `sequence`. The writer stamps
|
|
a candidate sequence, runs the empty-payload and size checks against the stamped
|
|
envelope, and commits the counter only immediately before the stream write, so
|
|
the sequences observed on the wire stay contiguous across rejections and an
|
|
operator reading a pipe capture never sees a phantom gap.
|
|
|
|
## Envelope Validation
|
|
|
|
`WorkerFrameReader` and `WorkerFrameWriter` validate each envelope against the
|
|
owning session before returning or writing it:
|
|
|
|
- `protocol_version` must match the configured worker protocol version,
|
|
- `session_id` must match the owning gateway session,
|
|
- the envelope must contain one typed `body` value.
|
|
|
|
Protocol violations throw `WorkerFrameProtocolException` with a
|
|
`WorkerFrameProtocolErrorCode` so callers can distinguish malformed frames,
|
|
oversized frames, protocol version mismatches, and session mismatches.
|
|
|
|
## Verification
|
|
|
|
The frame protocol lives in `ZB.MOM.WW.MxGateway.Worker.Ipc` (`WorkerFrameReader`,
|
|
`WorkerFrameWriter`, `WorkerFrameProtocolOptions`) and is covered by
|
|
`src/ZB.MOM.WW.MxGateway.Worker.Tests/Ipc/WorkerFrameProtocolTests.cs`. The worker is an
|
|
x86 process, so build and test it with `-p:Platform=x86`.
|
|
|
|
Run the focused tests after changing the frame protocol:
|
|
|
|
```powershell
|
|
dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj -p:Platform=x86 --filter WorkerFrameProtocolTests
|
|
```
|
|
|
|
Run the x86 worker build because the frame protocol is part of
|
|
`ZB.MOM.WW.MxGateway.Worker`:
|
|
|
|
```powershell
|
|
dotnet build src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj -p:Platform=x86
|
|
```
|
|
|
|
## Related Documentation
|
|
|
|
- [Gateway Process Detailed Design](./GatewayProcessDesign.md)
|
|
- [Protobuf Contracts](./Contracts.md)
|