dohertj2/mxaccessgw
1
0
Fork 0
Code Issues 14 Pull Requests 2 Actions Packages Projects Releases Wiki Activity

feat(worker): adopt negotiated frame max, bound drain, priority write scheduler (IPC-02/04 + WRK-04/07 worker half)

Browse Source 
Worker half of the Wave 3 size/backpressure + write-ordering pass:

- IPC-02: the worker adopts GatewayHello.max_frame_bytes during the handshake
  (WorkerFrameProtocolOptions.AdoptNegotiatedMaxMessageBytes) instead of a
  hard-coded default; 0 keeps the default, a value above a 256 MiB ceiling is
  rejected. Reader and writer share the options instance, applied before the
  message loop.
- IPC-04: DrainEvents caps each reply at MaxDrainEventsPerReply (10_000) and
  treats max_events = 0 as that cap rather than 'drain the entire queue', so one
  diagnostic drain cannot pack a session-killing reply frame.
- WRK-04: WorkerFrameWriter stamps the envelope Sequence at the actual point of
  writing (under the write lock) instead of at envelope creation, so the on-wire
  order and the stamped sequence always agree under concurrent producers.
- WRK-07: the writer is now a cooperative priority scheduler — callers enqueue at
  Control or Event priority and the draining lock-holder writes all control
  frames before any event frame, so replies/faults/heartbeats jump ahead of an
  event backlog. Per-frame validation/size rejections fail only that frame; a
  stream write failure fails all queued frames.

Tests: monotonic gap-free sequence under concurrency, control-before-event
priority (gated stream), negotiated-max adoption, DrainEvents zero-bound.
Worker builds x86 only — verified on windev.
This commit is contained in:
Joseph Doherty
2026-07-09 09:09:14 -04:00
parent c8b3a2281a
commit ebe6aeac98
7 changed files with 450 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameWriter.cs
+163 -19
View File
@@ -1,4 +1,5 @@
using System;
using System.Collections.Generic;
using System.IO;
using System.Threading;
using System.Threading.Tasks;
@@ -7,12 +8,40 @@ using ZB.MOM.WW.MxGateway.Contracts.Proto;
namespace ZB.MOM.WW.MxGateway.Worker.Ipc;
/// <summary>Writes worker frames to a stream with length-prefixed protobuf serialization.</summary>
/// <summary>
/// Writes worker frames to a stream with length-prefixed protobuf serialization. Callers enqueue a
/// frame at a <see cref="WorkerFrameWritePriority"/> and then contend for a single write lock; whoever
/// holds the lock drains every queued frame, control frames first, so a reply, fault, or heartbeat is
/// never delayed behind an event backlog (WRK-07). The envelope <c>Sequence</c> is stamped by the
/// draining lock-holder at the moment of writing, so the on-wire order and the stamped sequence always
/// agree even under concurrent callers and priority reordering (WRK-04).
/// </summary>
public sealed class WorkerFrameWriter
{
private sealed class PendingFrame
{
public PendingFrame(WorkerEnvelope envelope)
{
Envelope = envelope;
Completion = new TaskCompletionSource<bool>(TaskCreationOptions.RunContinuationsAsynchronously);
}
public WorkerEnvelope Envelope { get; }
public TaskCompletionSource<bool> Completion { get; }
}
private readonly WorkerFrameProtocolOptions _options;
private readonly SemaphoreSlim _writeLock = new(1, 1);
private readonly SemaphoreSlim _writeLock = new SemaphoreSlim(1, 1);
private readonly Stream _stream;
private readonly object _gate = new object();
private readonly Queue<PendingFrame> _controlFrames = new Queue<PendingFrame>();
private readonly Queue<PendingFrame> _eventFrames = new Queue<PendingFrame>();
// Only ever read/written by the current write-lock holder while draining, so no interlock is
// needed. Starts at 0 and is pre-incremented, so the first written frame carries sequence 1
// (matching the previous behaviour).
private ulong _nextSequence;
/// <summary>Initializes a new instance of the WorkerFrameWriter class.</summary>
/// <param name="stream">Stream to write frames to.</param>
@@ -25,12 +54,25 @@ public sealed class WorkerFrameWriter
_options = options ?? throw new ArgumentNullException(nameof(options));
}
/// <summary>Writes a worker envelope frame to the stream with length prefix.</summary>
/// <summary>Writes a control-priority worker envelope frame to the stream with length prefix.</summary>
/// <param name="envelope">Worker envelope to write.</param>
/// <param name="cancellationToken">Token to cancel the asynchronous operation.</param>
/// <returns>A task that represents the asynchronous operation.</returns>
/// <returns>A task that completes when the frame has been written and flushed.</returns>
public Task WriteAsync(
WorkerEnvelope envelope,
CancellationToken cancellationToken = default)
{
return WriteAsync(envelope, WorkerFrameWritePriority.Control, cancellationToken);
}
/// <summary>Queues a worker envelope frame for writing at the given priority and drains the queue.</summary>
/// <param name="envelope">Worker envelope to write.</param>
/// <param name="priority">Scheduling priority; control frames are written ahead of event frames.</param>
/// <param name="cancellationToken">Token to cancel waiting for the write lock.</param>
/// <returns>A task that completes when the frame has been written and flushed.</returns>
public async Task WriteAsync(
WorkerEnvelope envelope,
WorkerFrameWritePriority priority,
CancellationToken cancellationToken = default)
{
if (envelope is null)
@@ -38,8 +80,120 @@ public sealed class WorkerFrameWriter
throw new ArgumentNullException(nameof(envelope));
}
PendingFrame frame = new PendingFrame(envelope);
lock (_gate)
{
if (priority == WorkerFrameWritePriority.Event)
{
_eventFrames.Enqueue(frame);
}
else
{
_controlFrames.Enqueue(frame);
}
}
// Contend for the single writer: whoever wins drains every currently-queued frame in priority
// order, so this frame is written by this call or by a concurrent caller that got the lock
// first. Either way it completes via its own TaskCompletionSource.
await _writeLock.WaitAsync(cancellationToken).ConfigureAwait(false);
try
{
await DrainQueuedFramesAsync().ConfigureAwait(false);
}
finally
{
_writeLock.Release();
}
await frame.Completion.Task.ConfigureAwait(false);
}
// Runs only under _writeLock. Drains control frames before event frames, stamping and writing each.
// The stream write itself is not cancellable: a frame is written atomically or fails, never left
// half-written on the pipe because a caller gave up waiting.
private async Task DrainQueuedFramesAsync()
{
while (true)
{
PendingFrame? frame = DequeueNext();
if (frame is null)
{
return;
}
try
{
await WriteFrameAsync(frame.Envelope).ConfigureAwait(false);
frame.Completion.TrySetResult(true);
}
catch (WorkerFrameProtocolException exception) when (IsPerFrameRejection(exception))
{
// Validation, empty-payload, and oversized-frame errors are specific to this frame and
// do not damage the stream; fail only this frame and keep draining the rest.
frame.Completion.TrySetException(exception);
}
catch (Exception exception)
{
// A stream write/flush failure means the pipe is broken; fail this frame and every frame
// still queued so no caller awaits forever, then stop draining.
frame.Completion.TrySetException(exception);
FailAllQueued(exception);
return;
}
}
}
private static bool IsPerFrameRejection(WorkerFrameProtocolException exception)
{
return exception.ErrorCode is WorkerFrameProtocolErrorCode.InvalidEnvelope
or WorkerFrameProtocolErrorCode.MessageTooLarge
or WorkerFrameProtocolErrorCode.ProtocolVersionMismatch
or WorkerFrameProtocolErrorCode.SessionMismatch;
}
private PendingFrame? DequeueNext()
{
lock (_gate)
{
if (_controlFrames.Count > 0)
{
return _controlFrames.Dequeue();
}
if (_eventFrames.Count > 0)
{
return _eventFrames.Dequeue();
}
return null;
}
}
private void FailAllQueued(Exception exception)
{
lock (_gate)
{
while (_controlFrames.Count > 0)
{
_controlFrames.Dequeue().Completion.TrySetException(exception);
}
while (_eventFrames.Count > 0)
{
_eventFrames.Dequeue().Completion.TrySetException(exception);
}
}
}
private async Task WriteFrameAsync(WorkerEnvelope envelope)
{
WorkerEnvelopeValidator.Validate(envelope, _options);
// Stamp the sequence at the actual point of writing, under the write lock, so the wire order
// and the stamped sequence agree regardless of caller concurrency or priority (WRK-04).
envelope.Sequence = unchecked(++_nextSequence);
int payloadLength = envelope.CalculateSize();
if (payloadLength == 0)
{
@@ -55,26 +209,16 @@ public sealed class WorkerFrameWriter
$"Worker envelope payload length {payloadLength} exceeds the configured maximum of {_options.MaxMessageBytes} bytes.");
}
// Serialize once into a single buffer that carries the 4-byte
// length prefix followed by the payload, then issue one stream write.
// This avoids a second serialization pass (envelope.ToByteArray()
// would re-run CalculateSize internally), a separate prefix array,
// and a separate prefix write.
// Serialize once into a single buffer that carries the 4-byte length prefix followed by the
// payload, then issue one stream write. This avoids a second serialization pass, a separate
// prefix array, and a separate prefix write.
int frameLength = sizeof(uint) + payloadLength;
byte[] frame = new byte[frameLength];
WriteUInt32LittleEndian(frame, (uint)payloadLength);
envelope.WriteTo(new Span<byte>(frame, sizeof(uint), payloadLength));
await _writeLock.WaitAsync(cancellationToken).ConfigureAwait(false);
try
{
await _stream.WriteAsync(frame, 0, frameLength, cancellationToken).ConfigureAwait(false);
await _stream.FlushAsync(cancellationToken).ConfigureAwait(false);
}
finally
{
_writeLock.Release();
}
await _stream.WriteAsync(frame, 0, frameLength, CancellationToken.None).ConfigureAwait(false);
await _stream.FlushAsync(CancellationToken.None).ConfigureAwait(false);
}
private static void WriteUInt32LittleEndian(