mxaccessgw/docs/WorkerFrameProtocol.md

# Worker Frame Protocol

The gateway uses the worker frame protocol to move `WorkerEnvelope` protobuf
messages over a bidirectional named pipe. The frame layer is deliberately small:
it handles message boundaries, size limits, protobuf parsing, and envelope
validation before higher-level worker client code routes commands, replies,
events, and faults.

## Frame Format

Each frame starts with a four-byte little-endian unsigned payload length,
followed by the serialized `WorkerEnvelope` payload:

```text
uint32 little-endian payload_length
payload_length bytes protobuf WorkerEnvelope
```

The reader rejects zero-length payloads and payloads larger than the configured
maximum before allocating the payload buffer. The default maximum is the 16 MiB
public gRPC cap plus a 64 KiB envelope-overhead reserve (16842752 bytes) so a
maximally-sized accepted gRPC payload always fits one worker frame once wrapped
in a `WorkerEnvelope`.

The gateway is the source of truth for this maximum: it conveys the negotiated
value in the handshake as `GatewayHello.max_frame_bytes`, and the worker adopts
it as its `WorkerFrameProtocolOptions.MaxMessageBytes` instead of a hard-coded
default. A `max_frame_bytes` of 0 (an older gateway that never set the field)
means "use the worker's built-in default". This keeps both ends framing to the
same limit rather than depending on matched compile-time constants.

The worker accepts a negotiated value in the closed range [1024, 256 MiB]
(`MinNegotiableFrameBytes` .. `MaxNegotiableFrameBytes`); 0 keeps the default.
A value outside that range is rejected at the handshake with a fault frame
rather than adopted, because a nonsensical maximum — a gateway bug or a
foreign/old peer — would otherwise leave a session that handshakes cleanly and
then fails every subsequent frame with per-frame size errors, the worst
diagnostic shape for an operator. The 1024-byte floor matches the gateway's own
`GatewayOptionsValidator.MinimumMaxMessageBytes`, so the worker never rejects a
value the gateway's validator accepts as legal configuration, and 1024 still
guarantees hellos, heartbeats, acks, and faults fit.

Every worker-to-gateway frame must serialize within this limit, control replies
included, so reply builders truncate to fit rather than emit a frame the writer
will reject. `WorkerPipeSession` pre-sizes a `DrainEvents` reply below the
negotiated maximum and reports the truncation in the reply's
`DiagnosticMessage`; the caller contract is to repeat `DrainEvents` until it
returns an empty reply. The event budget is `max(MaxMessageBytes - 2 x 64 KiB,
MaxMessageBytes / 2)`: the reserve is subtracted **twice** because the reply has
two boundaries to clear, not one. The first reserve covers the
`WorkerEnvelope`/`WorkerCommandReply`/`MxCommandReply` wrapper on the pipe; the
second exists because the gateway returns that same `MxCommandReply` verbatim
through `Invoke`, where the public gRPC send cap applies. `DrainEvents` removes
the events it packs from the worker queue, so a reply the pipe carries and the
gRPC layer then refuses would lose them for good — the gateway's validator
therefore also caps `MxGateway:Worker:MaxMessageBytes` at
`MxGateway:Protocol:MaxGrpcMessageBytes` plus **two** reserves, which is what
makes the doubled subtraction sufficient rather than merely generous. The
`MaxMessageBytes / 2` floor keeps the budget usable at frame maxima small enough
that two reserves would consume them (below 256 KiB), where a collapsed budget
would truncate every drain and never let the drain-until-empty caller finish.
Should a reply still overshoot,
`MessageTooLarge` at the reply-write seam is answered with a small
`InvalidRequest` reply for that correlation, not with session teardown — no
diagnostics command may kill a session.

An oversized *event* frame is the deliberate exception. Such an event is
undeliverable end to end (the pipe maximum sits only the envelope-overhead
reserve above the public gRPC cap), so the session faults: the worker logs the
event's identity and sizes — never its value — writes a `WorkerFault` with
category `ProtocolViolation` and command method `EventDrain`, then exits.
Remediation is raising `MxGateway:Worker:MaxMessageBytes` for that workload.

A per-frame rejection does not consume an envelope `sequence`. The writer stamps
a candidate sequence, runs the empty-payload and size checks against the stamped
envelope, and commits the counter only immediately before the stream write, so
the sequences observed on the wire stay contiguous across rejections and an
operator reading a pipe capture never sees a phantom gap.

## Envelope Validation

`WorkerFrameReader` and `WorkerFrameWriter` validate each envelope against the
owning session before returning or writing it:

- `protocol_version` must match the configured worker protocol version,
- `session_id` must match the owning gateway session,
- the envelope must contain one typed `body` value.

Protocol violations throw `WorkerFrameProtocolException` with a
`WorkerFrameProtocolErrorCode` so callers can distinguish malformed frames,
oversized frames, protocol version mismatches, and session mismatches.

## Write Scheduling And Sequencing

This section covers write scheduling (priority classes, enqueue-then-contend,
flush coalescing) and sequencing (write-time stamping) together, because both
are properties of the same single write lock.

`WorkerFrameWriter` is a two-class cooperative priority scheduler
(`WorkerFrameWritePriority.Control` and `.Event`), not a strict per-kind
priority order. A caller enqueues its frame into the control or event queue
under a lock, then contends for a single write lock; whichever caller wins
drains every frame queued at that moment, control frames first and each class
in FIFO order, so a command reply, fault, heartbeat, or shutdown
acknowledgement is never delayed behind a backlog of queued events — neither
in the bytes written nor in the flush that marks them delivered (see the
class-boundary flush under flush coalescing below). Priority
only reorders *which frame writes next* — it does not affect the sequence
value a frame receives (see below), so a caller cannot infer priority class
from the wire sequence.

The envelope `Sequence` is stamped by the draining lock-holder at the actual
moment of writing, not when the frame is enqueued, so the on-wire order and
the stamped sequence always agree regardless of caller concurrency or
priority reordering. Stamping uses peek-stamp-commit: a candidate sequence is
assigned and the frame is validated (size, non-empty payload) against that
stamped value, but the counter is committed only immediately before the
stream write. A per-frame rejection therefore leaves the counter untouched —
the next accepted frame reuses the candidate number, so the wire sequence
stays contiguous across rejections and an operator reading a pipe capture
never sees a phantom gap from a rejected frame.

Two failure shapes are distinguished during a drain pass:

- **Per-frame rejection** (`InvalidEnvelope`, `MessageTooLarge`,
  `ProtocolVersionMismatch`, `SessionMismatch`) is specific to the one frame
  that failed validation or sizing. Nothing was written for it, so it fails
  only that frame's completion and draining continues with the next queued
  frame.
- **Stream failure** (anything else — a broken pipe, an I/O error) means the
  underlying stream itself is no longer trustworthy. It fails the frame that
  triggered it, every frame already written this batch but not yet flushed,
  and every frame still queued, then stops draining entirely so no caller
  waits forever on a stream that will not recover.

Flushes are coalesced across a *run of same-class frames* inside a drain
pass: each frame in the run is written to the stream without an individual
flush, then one `FlushAsync` runs — at the end of the pass, and additionally
at every control-to-event boundary — and only then does every
successfully-written frame of that run resolve its completion. A caller's
`WriteAsync` therefore still does not complete until its bytes are both
written *and* flushed; what changed is *when* that moment arrives
for a control frame that a pass writes ahead of queued events. It used to be
the end of the pass, so a heartbeat, command reply, fault, or shutdown
acknowledgement was written first but only counted as delivered after up to a
full event batch had been written and flushed behind it. The boundary flush
closes the control run out before the events are written, so the priority
class governs the frame's delivery point and not just its byte order. The
cost stays bounded: a pure-event pass — the event hot path — still pays
exactly one flush however many frames drain together, a run of control
frames still pays one for the whole run (never one per heartbeat, the
syscall-per-frame cost the coalescing removed), and only a pass that actually
mixes both classes pays a second. Mixed passes are not exotic: command replies
travel at Control priority alongside heartbeats and faults, so a session under
sustained command traffic concurrent with event streaming can hit them
routinely. The cost stays bounded either way — one extra flush per class
transition present in the pass, not per frame.

One consequence of the boundary flush is worth stating: a control frame whose
run has already been flushed and completed is out of the drain's
written-but-unflushed set, so a *later* failure in the same pass — a broken
write, or a failed end-of-pass flush — no longer reaches back and fails it.
That is the honest outcome: its bytes were flushed, so it was delivered. A
failure of the boundary flush itself is treated exactly like a failed
end-of-pass flush, and additionally fails the event frame the drain had
already claimed off its queue (nothing else would ever complete it) along
with every frame still queued.

Note the ordering all of this implies at the peer: the frames reach the pipe
before the flush that follows them, so the gateway can read a whole batch
while the writer has not yet flushed it. Anything observing the flush itself
(a test counting flushes, for instance) must wait for the flush, not infer it
from frames arriving. The event drain loop (`WorkerPipeSession.RunEventDrainLoopAsync`)
submits a whole drained event batch through `WriteBatchAsync`, which enqueues
every frame under one `_gate` acquisition, takes the write lock once, and
drains them together, so a burst of N events costs one flush rather than N —
the coalescing the batch machinery was built for now engages on the event hot
path, not only when independent producers happen to queue behind a blocked
write. Intra-batch order is preserved (FIFO enqueue under one lock), and a
concurrently queued control frame is still drained — and now flushed and
completed — ahead of the batch's remaining events, which is why a batch a
control frame cuts into pays one extra flush while an uninterrupted batch
still pays exactly one. A per-frame rejection inside a batch (for example one
oversized event) surfaces from the batch's awaited completions as that frame's
`WorkerFrameProtocolException`; the remaining completions are still observed
so none faults unobserved.

The completion is the frame's delivery point, and a `WriteAsync` caller now
returns at it. The boundary flush alone only made the delivery point honest:
a caller that lost the race for the write lock still sat in the lock wait
until the winning drainer released it, so its awaited task was charged for the
whole event backlog its control frame had just been flushed ahead of. To close
that, `WriteAsync` awaits its own frame's completion *racing* the lock
acquisition instead of the acquisition alone. Whichever settles first decides:

- **Completion first** — the winning drainer wrote and flushed this frame at
  the class boundary, so the caller returns immediately. The lock acquisition
  it leaves outstanding is *detached*, not dropped: a continuation drains
  whatever is queued and then releases, so the lock is never acquired and
  silently held, and a frame enqueued between the previous drainer's last
  dequeue and its release is still written by someone. Draining an empty queue
  is a no-op, so the common case is acquire-nothing-release.
- **Lock first** — the caller drains the pass itself, exactly as before.
- **Cancellation** — the wait ends without the lock (`SemaphoreSlim` hands no
  count to a wait it cancels, so the detached continuation releases nothing on
  that path) and the tombstone rules below apply unchanged. A token that fires
  *after* the frame's completion won the race changes nothing: the frame was
  delivered, and the caller returns normally.

`WriteBatchAsync` deliberately keeps the plain wait-then-drain shape. A batch
caller's result is its whole set of completions and the last of those resolves
at the end-of-pass flush — the instant before the drainer releases the lock —
so racing the acquisition would buy it nothing while adding one detached
acquisition per call.

Cancellation of a `WriteAsync`/`WriteBatchAsync` call that is still waiting
for the write lock when its token fires tombstones the queued frame: the
cancelled caller marks its frame under `_gate`, and the draining lock-holder's
`DequeueNext` skips any tombstoned frame, so a cancelled call is guaranteed
never to reach the wire — *unless* a lock-holder has already claimed the frame
to write it. Claiming and cancelling are interlocked under `_gate`, so exactly
one wins; a frame already claimed is mid-write and can no longer be recalled,
so the caller observes `OperationCanceledException` while that one frame still
reaches the wire. That residual window is by design: blocking the canceller
behind the very write it is abandoning would defeat the point of cancellation.

Two hygiene notes on that residual (NEXT-04/NEXT-05). First, a frame the
cancelled caller abandons — claimed mid-write, or already faulted by a
concurrent queue-wide failure — completes on a task nobody awaits; the
tombstone path attaches a fault-observing continuation to it so a later write
failure never surfaces as a `TaskScheduler.UnobservedTaskException`. Second,
tombstoned entries stay in the class queues until a future `DequeueNext` pops
and skips them; that lazy purge is deliberate. Eagerly rebuilding a `Queue<T>`
under `_gate` on every cancellation would add ordering-invariant surface next
to the claim/cancel interlock for no real gain: any subsequent write of either
class drains both queues to empty, and the heartbeat loop guarantees one
arrives within a heartbeat interval, so worst-case residency is a few envelope
references for seconds — not a leak.

## Pipe Buffers

The gateway creates each worker pipe with an explicit 128 KiB kernel buffer per
direction (`SessionWorkerClientFactory.PipeBufferSizeBytes`) rather than the zero
quota the short `NamedPipeServerStream` overloads request. A zero-quota byte-mode
pipe makes every write rendezvous with a pending read, so a writer with no reader
parked blocks until one arrives — the failure class behind the historical windev
full-suite wedge. A real quota decouples writer latency from reader scheduling and
lets the flush coalescing above actually pay off. On Unix hosts, where named pipes
are Unix domain sockets, the sizes are advisory.

## Verification

The frame protocol lives in `ZB.MOM.WW.MxGateway.Worker.Ipc` (`WorkerFrameReader`,
`WorkerFrameWriter`, `WorkerFrameProtocolOptions`) and is covered by
`src/ZB.MOM.WW.MxGateway.Worker.Tests/Ipc/WorkerFrameProtocolTests.cs`. The worker is an
x86 process, so build and test it with `-p:Platform=x86`.

Run the focused tests after changing the frame protocol:

```powershell
dotnet test src/ZB.MOM.WW.MxGateway.Worker.Tests/ZB.MOM.WW.MxGateway.Worker.Tests.csproj -p:Platform=x86 --filter WorkerFrameProtocolTests
```

Run the x86 worker build because the frame protocol is part of
`ZB.MOM.WW.MxGateway.Worker`:

```powershell
dotnet build src/ZB.MOM.WW.MxGateway.Worker/ZB.MOM.WW.MxGateway.Worker.csproj -p:Platform=x86
```

## Related Documentation

- [Gateway Process Detailed Design](./GatewayProcessDesign.md)
- [Protobuf Contracts](./Contracts.md)