dohertj2/mxaccessgw
1
0
Fork 0
Code Issues 14 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

perf(worker): control-frame completions resolve at the class-transition flush, not after the event batch

Browse Source 
The two-class writer already got control bytes out ahead of a queued event
backlog, but a frame counts as delivered only once flushed, and the drain
deferred its single FlushAsync — and every TrySetResult — to the end of the
pass. A heartbeat, command reply, fault, or shutdown ack was therefore written
first and completed last, behind up to a full 128-frame event batch.

The drain now records each frame's priority class on PendingFrame and flushes
at every control-to-event boundary, completing and clearing the written set
there. Cost stays bounded: a pure-event pass still pays exactly one flush, a
run of control frames still pays one for the run, and only a pass that mixes
both classes pays a second — never one flush per control frame, the
syscall-per-heartbeat cost WRK-12 removed.

A boundary flush that itself fails is a new failure window and is handled like
the end-of-pass flush failure, additionally failing the event frame the drain
had already claimed off its queue and every frame still queued. Frames a
boundary flush completed leave the written set, so a later failure in the same
pass can no longer reach back and fail an already-delivered control frame.

The awaited task of a caller that lost the write-lock race is still bounded by
the winning drainer's pass — that enqueue-then-contend parking is unchanged and
now documented on WriteAsync and in docs/WorkerFrameProtocol.md.
This commit is contained in:
Joseph Doherty
2026-08-15 21:05:57 -04:00
parent 9871d4772d
commit aac79579ab
3 changed files with 401 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameWriter.cs
+87 -16
View File
@@ -14,9 +14,11 @@ namespace ZB.MOM.WW.MxGateway.Worker.Ipc;
/// Writes worker frames to a stream with length-prefixed protobuf serialization. Callers enqueue a
/// frame at a <see cref="WorkerFrameWritePriority"/> and then contend for a single write lock; whoever
/// holds the lock drains every queued frame, control frames first, so a reply, fault, or heartbeat is
/// never delayed behind an event backlog. The envelope <c>Sequence</c> is stamped by the
/// draining lock-holder at the moment of writing, so the on-wire order and the stamped sequence always
/// agree even under concurrent callers and priority reordering.
/// never delayed behind an event backlog — neither in the bytes it writes nor in the flush that
/// delivers them, because the drain flushes at every control-to-event boundary rather than only at the
/// end of the pass. The envelope <c>Sequence</c> is stamped by the draining lock-holder at the moment
/// of writing, so the on-wire order and the stamped sequence always agree even under concurrent callers
/// and priority reordering.
/// </summary>
public sealed class WorkerFrameWriter
{
@@ -24,15 +26,25 @@ public sealed class WorkerFrameWriter
{
/// <summary>Initializes a new instance of the PendingFrame class.</summary>
/// <param name="envelope">Worker envelope awaiting write.</param>
public PendingFrame(WorkerEnvelope envelope)
/// <param name="priority">Priority class the frame was queued at.</param>
public PendingFrame(WorkerEnvelope envelope, WorkerFrameWritePriority priority)
{
Envelope = envelope;
IsControl = priority != WorkerFrameWritePriority.Event;
Completion = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
}
/// <summary>Gets the worker envelope awaiting write.</summary>
public WorkerEnvelope Envelope { get; }
/// <summary>
/// Gets a value indicating whether this frame was queued as control-plane traffic. Recorded at
/// construction from the same expression that picks the queue, so the class the drain sees can
/// never disagree with the queue the frame sits in. The drain uses it to flush and complete
/// written control frames at the moment it turns to events (see <see cref="DrainQueuedFramesAsync"/>).
/// </summary>
public bool IsControl { get; }
/// <summary>Gets the completion source signaled once the frame has been written or has failed.</summary>
public TaskCompletionSource<bool> Completion { get; }
@@ -95,6 +107,15 @@ public sealed class WorkerFrameWriter
/// the canceller behind the very write it is abandoning would defeat the point of cancellation.
/// The abandoned frame's completion gets a fault-observing continuation so a write failure after
/// the caller unwinds never raises an unobserved-task exception (NEXT-04).
/// <para>
/// Latency contract: a control frame's bytes are written, flushed, and its completion
/// resolved before the events a drain pass writes after it — the delivery point of a heartbeat,
/// reply, fault, or shutdown ack is never charged for the event backlog behind it. The returned
/// task can still be later than that instant for a caller that lost the write-lock race: it only
/// observes its completion after the winning drainer releases the lock, so its own return remains
/// bounded by that pass. That parking is deliberate — the alternative is to race the lock wait
/// against the completion, which buys nothing for the frame's delivery.
/// </para>
/// </remarks>
public async Task WriteAsync(
WorkerEnvelope envelope,
@@ -106,7 +127,7 @@ public sealed class WorkerFrameWriter
throw new ArgumentNullException(nameof(envelope));
}
PendingFrame frame = new PendingFrame(envelope);
PendingFrame frame = new PendingFrame(envelope, priority);
lock (_gate)
{
if (priority == WorkerFrameWritePriority.Event)
@@ -153,8 +174,12 @@ public sealed class WorkerFrameWriter
/// rather than one per frame (WRK-25, realizing the WRK-12 coalescing on the path it was built
/// for). Intra-batch order is preserved because the enqueue is atomic under <c>_gate</c> and each
/// class queue is FIFO; the control-before-event guarantee still holds because any concurrently
/// queued control frame is drained ahead of this batch by <see cref="DequeueNext"/>. Every frame's
/// "written and flushed before completion" contract is unchanged.
/// queued control frame is drained ahead of this batch by <see cref="DequeueNext"/>, and — since
/// the control-frame completion decoupling — is also flushed and completed before this batch's
/// remaining events are written, so a batch in flight does not delay a control frame's delivery.
/// Every frame's "written and flushed before completion" contract is unchanged. An event batch that
/// a control frame cuts into therefore pays one extra flush; an uninterrupted batch still pays
/// exactly one.
/// </summary>
/// <param name="envelopes">Envelopes to write, in order.</param>
/// <param name="priority">Scheduling priority for the whole batch.</param>
@@ -189,7 +214,7 @@ public sealed class WorkerFrameWriter
{
WorkerEnvelope envelope = envelopes[index]
?? throw new ArgumentException("Batch envelopes must not contain null.", nameof(envelopes));
frames[index] = new PendingFrame(envelope);
frames[index] = new PendingFrame(envelope, priority);
}
lock (_gate)
@@ -296,14 +321,24 @@ public sealed class WorkerFrameWriter
// The stream write itself is not cancellable: a frame is written atomically or fails, never left
// half-written on the pipe because a caller gave up waiting.
//
// Flushes are coalesced across the whole drained batch (WRK-12 / IPC-15): each frame is written to
// the stream but not flushed individually; a single FlushAsync runs after the batch, then every
// successfully-written frame is completed. A caller's Completion therefore still signals only after
// its bytes have been written AND flushed, so the "written and flushed" contract is unchanged — but
// a burst of N events now costs one flush syscall instead of N.
// Flushes are coalesced within a priority class rather than blindly across the whole pass (WRK-12 /
// IPC-15, narrowed by the control-frame completion decoupling): each frame is written to the stream
// but not flushed individually, and one FlushAsync runs at the end of the pass — plus one at each
// control-to-event boundary, which flushes and completes the control frames written so far before
// the event backlog behind them is written, instead of after it. Without that boundary flush the
// priority scheduler only got control *bytes* out early: their delivery point, and every waiting
// caller's completion, still sat behind up to a full event batch.
//
// A caller's Completion therefore still signals only after its bytes have been written AND flushed —
// the contract is unchanged, the moment it is reached simply stops being pinned to the end of the
// pass. Cost is bounded: a pure-event pass (the event hot path) still pays exactly one flush, a burst
// of control frames still pays one for the whole burst, and only a pass that actually mixes both
// classes pays a second — never one flush per control frame, which is the syscall-per-heartbeat cost
// WRK-12 removed.
private async Task DrainQueuedFramesAsync()
{
List<PendingFrame> written = new List<PendingFrame>();
bool writtenHoldsControl = false;
while (true)
{
PendingFrame? frame = DequeueNext();
@@ -312,10 +347,40 @@ public sealed class WorkerFrameWriter
break;
}
if (writtenHoldsControl && !frame.IsControl)
{
// Class transition: the frames written so far include at least one control frame whose
// caller is waiting on delivery. Flush and complete them here rather than parking them
// behind the events this pass is about to write. Charged once per transition, not once
// per control frame. Event frames already in the list ride along — they too are written
// and now flushed, so completing them early is the same contract, earlier.
try
{
await _stream.FlushAsync(CancellationToken.None).ConfigureAwait(false);
}
catch (Exception exception)
{
// Same shape as the end-of-pass flush failure: the bytes reached the stream but the
// flush that guarantees delivery failed, so the pipe is broken. Fail the frame just
// claimed (it is out of its queue and nothing else will ever complete it), every
// written-but-unflushed frame, and everything still queued, then stop draining.
frame.Completion.TrySetException(exception);
FailFrames(written, exception);
FailAllQueued(exception);
return;
}
// Completed frames leave the list, so a later failure in this pass cannot fail them.
CompleteFrames(written);
written.Clear();
writtenHoldsControl = false;
}
try
{
await WriteFrameAsync(frame.Envelope).ConfigureAwait(false);
written.Add(frame);
writtenHoldsControl |= frame.IsControl;
}
catch (WorkerFrameProtocolException exception) when (IsPerFrameRejection(exception))
{
@@ -348,13 +413,19 @@ public sealed class WorkerFrameWriter
catch (Exception exception)
{
// The batch reached the stream but the flush that guarantees delivery failed: the pipe is
// broken. Fail every frame in the batch (the queue was already drained) so no caller treats
// an unflushed write as delivered.
// broken. Fail every frame still in the batch (the queue was already drained) so no caller
// treats an unflushed write as delivered. Frames a boundary flush already completed are not
// in the list — their bytes were flushed, so this later failure does not reach back to them.
FailFrames(written, exception);
return;
}
foreach (PendingFrame frame in written)
CompleteFrames(written);
}
private static void CompleteFrames(List<PendingFrame> frames)
{
foreach (PendingFrame frame in frames)
{
frame.Completion.TrySetResult(true);
}