dohertj2/mxaccessgw
1
0
Fork 0
Code Issues 3 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

perf(worker): unpark awaited control-frame writers from the winning drain pass

Browse Source 
WriteAsync enqueued its frame and then contended unconditionally for the write
lock, so a caller that lost the race stayed in WaitAsync until the winning
drainer released — even though that winner writes, flushes, and completes the
loser's control frame at the control-to-event class boundary, part-way through
its pass. The boundary flush made the delivery point honest; the awaited task
was still charged for the whole event backlog it had just been flushed ahead of.

WriteAsync now awaits its own frame's completion racing the lock acquisition.
Completion first: the caller returns at its frame's delivery point and the
outstanding acquisition is detached, not dropped — a continuation drains
whatever is queued and releases, so the lock is never acquired and silently
held and a frame enqueued between the previous drainer's last dequeue and its
release is still written. Lock first: drain as before. Cancellation keeps the
WRK-22 tombstone semantics exactly, and a wait cancelled after the caller has
already detached releases nothing (SemaphoreSlim hands no count to a wait it
cancels), so no count leaks and no queued frame is stranded. A token that fires
after the frame's completion won the race changes nothing — the frame was
delivered. WriteBatchAsync deliberately keeps the plain wait-then-drain shape:
its last completion resolves at the end-of-pass flush anyway.

Three tests: the latency win (a control caller returning while the winning
WriteBatchAsync event burst is demonstrably still blocked mid-pass), a
mixed-priority concurrency soak pinning exactly-once writes and a single
drainer, and the cancel-after-detach corner (a wrongly released count would
surface as the drainer's own Release throwing SemaphoreFullException).

edited on macOS, windev verification pending (plan Task 11). Verified here by
compiling and running WorkerFrameWriter plus the writer suite against net10.0
in a scratch harness: 31/31 pass, and the two behaviour-pinning tests fail
against the pre-change parked implementation.
This commit is contained in:
Joseph Doherty
2026-08-17 03:55:05 -04:00
parent c79aaaf9eb
commit 9130994736
4 changed files with 354 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.MxGateway.Worker/Ipc/WorkerFrameWriter.cs
+125 -21
View File
@@ -16,9 +16,12 @@ namespace ZB.MOM.WW.MxGateway.Worker.Ipc;
/// holds the lock drains every queued frame, control frames first, so a reply, fault, or heartbeat is
/// never delayed behind an event backlog — neither in the bytes it writes nor in the flush that
/// delivers them, because the drain flushes at every control-to-event boundary rather than only at the
/// end of the pass. The envelope <c>Sequence</c> is stamped by the draining lock-holder at the moment
/// of writing, so the on-wire order and the stamped sequence always agree even under concurrent callers
/// and priority reordering.
/// end of the pass. A caller that loses the lock race does not wait for the winner's pass to end: it
/// awaits its own frame's completion racing its lock acquisition, so it returns at the boundary flush
/// that delivered its frame and the acquisition it walks away from is detached rather than dropped
/// (see <see cref="DetachLockWait"/>). The envelope <c>Sequence</c> is stamped by the draining
/// lock-holder at the moment of writing, so the on-wire order and the stamped sequence always agree
/// even under concurrent callers and priority reordering.
/// </summary>
public sealed class WorkerFrameWriter
{
@@ -111,10 +114,12 @@ public sealed class WorkerFrameWriter
/// Latency contract: a control frame's bytes are written, flushed, and its completion
/// resolved before the events a drain pass writes after it — the delivery point of a heartbeat,
/// reply, fault, or shutdown ack is never charged for the event backlog behind it. The returned
/// task can still be later than that instant for a caller that lost the write-lock race: it only
/// observes its completion after the winning drainer releases the lock, so its own return remains
/// bounded by that pass. That parking is deliberate — the alternative is to race the lock wait
/// against the completion, which buys nothing for the frame's delivery.
/// task resolves at that same instant even for a caller that lost the write-lock race, because
/// this call awaits its own frame's completion racing the lock acquisition rather than the lock
/// alone: the completion resolving first returns the caller immediately and hands the outstanding
/// acquisition to <see cref="DetachLockWait"/>, which keeps its drain responsibility. Without that,
/// a control frame delivered at a class boundary still reported back only after the winning
/// drainer finished writing and flushing the entire event backlog behind it.
/// </para>
/// </remarks>
public async Task WriteAsync(
@@ -140,20 +145,39 @@ public sealed class WorkerFrameWriter
}
}
// Contend for the single writer: whoever wins drains every currently-queued frame in priority
// order, so this frame is written by this call or by a concurrent caller that got the lock
// first. Either way it completes via its own TaskCompletionSource.
try
// Contend for the single writer, but race that contention against this frame's own completion.
// Whoever wins the lock drains every currently-queued frame in priority order, so this frame is
// written by this call or by a concurrent caller that got the lock first — and in the latter
// case the winner writes, flushes, and completes it at the control-to-event boundary, part-way
// through its pass. Racing the two is what lets this call return at that instant instead of at
// the winner's release; the acquisition it then walks away from is detached, never dropped, so
// no drain responsibility leaves with it.
Task lockWait = _writeLock.WaitAsync(cancellationToken);
Task completion = frame.Completion.Task;
await Task.WhenAny(completion, lockWait).ConfigureAwait(false);
if (lockWait.Status != TaskStatus.RanToCompletion)
{
await _writeLock.WaitAsync(cancellationToken).ConfigureAwait(false);
}
catch (OperationCanceledException)
{
// Tombstone the queued frame so DequeueNext skips it — but only if a draining lock-holder
// has not already claimed it. If it is claimed it is mid-write and cannot be recalled; the
// caller still observes cancellation while the frame reaches the wire (documented above).
TombstoneIfUnclaimed(frame, cancellationToken);
throw;
// This call does not hold the lock: either the completion won the race and the acquisition
// is still outstanding, or the wait ended without the lock because the token fired. Hand the
// wait over either way — an acquisition nobody is waiting on must still drain and release,
// and a wait that ends without the lock must still have its outcome observed.
DetachLockWait(lockWait);
if (!completion.IsCompleted)
{
// The frame was not delivered, so the wait must have ended in cancellation. Tombstone
// the queued frame so DequeueNext skips it — but only if a draining lock-holder has not
// already claimed it. If it is claimed it is mid-write and cannot be recalled; the
// caller still observes cancellation while the frame reaches the wire (documented
// above). Rethrow from the wait rather than from the completion, so a claimed frame's
// canceller is not held behind the very write it is abandoning.
TombstoneIfUnclaimed(frame, cancellationToken);
await lockWait.ConfigureAwait(false);
}
await completion.ConfigureAwait(false);
return;
}
try
@@ -165,7 +189,7 @@ public sealed class WorkerFrameWriter
_writeLock.Release();
}
await frame.Completion.Task.ConfigureAwait(false);
await completion.ConfigureAwait(false);
}
/// <summary>
@@ -195,6 +219,13 @@ public sealed class WorkerFrameWriter
/// <see cref="WriteAsync(WorkerEnvelope, WorkerFrameWritePriority, CancellationToken)"/>; frames
/// the cancelled caller abandons (claimed mid-write, or already faulted) get a fault-observing
/// continuation so a later write failure never raises an unobserved-task exception (NEXT-04).
/// <para>
/// This path deliberately keeps the plain wait-then-drain shape rather than the single-frame
/// path's completion-versus-lock race. A batch caller's result is its whole set of completions,
/// and the last of those resolves at the end-of-pass flush — the instant before the drainer
/// releases the lock — so racing the acquisition would buy a batch caller nothing while adding a
/// detached acquisition per call. Semantics here are unchanged by that race.
/// </para>
/// </remarks>
public async Task WriteBatchAsync(
IReadOnlyList<WorkerEnvelope> envelopes,
@@ -319,6 +350,79 @@ public sealed class WorkerFrameWriter
TaskScheduler.Default);
}
/// <summary>
/// Keeps a write-lock acquisition whose caller has stopped waiting for it — its frame was
/// delivered inside the winning drainer's pass, or its token fired — from losing the drain
/// responsibility that comes with the lock. The wait is never simply dropped: if it goes on to
/// acquire the lock, the continuation drains whatever is queued and then releases, so the lock
/// is never acquired and silently held, and a frame enqueued between the previous drainer's
/// last <see cref="DequeueNext"/> and its release is still written by someone. A wait that ends
/// without the lock took no semaphore count and so has nothing to release.
/// <para>
/// The continuation is queued to the thread pool rather than run inline, because it starts a
/// drain pass: running that synchronously would charge whichever thread called <c>Release</c>
/// for the next caller's writes.
/// </para>
/// </summary>
/// <param name="lockWait">Outstanding write-lock acquisition the caller has walked away from.</param>
private void DetachLockWait(Task lockWait)
{
_ = lockWait.ContinueWith(
OnDetachedLockWaitSettled,
CancellationToken.None,
TaskContinuationOptions.None,
TaskScheduler.Default);
}
/// <summary>
/// Settles a detached write-lock acquisition: drain and release if it took the lock, otherwise
/// observe its outcome and stop. Acquisition and cancellation are mutually exclusive —
/// <see cref="SemaphoreSlim"/> never hands a count to a wait it cancels — so neither branch can
/// leak a count, and neither can strand a queued frame: a cancelled wait never held the lock,
/// so whatever is queued is still owned by the next caller to acquire it.
/// </summary>
/// <param name="lockWait">Settled write-lock acquisition task.</param>
private void OnDetachedLockWaitSettled(Task lockWait)
{
if (lockWait.Status != TaskStatus.RanToCompletion)
{
// Cancelled by the originating caller's token, or — only pathologically, a disposed
// semaphore — faulted. No count was taken, so there is nothing to release and no drain to
// inherit. Touch Exception so a fault on a task nobody awaits any more cannot surface as an
// unobserved-task exception.
_ = lockWait.Exception;
return;
}
_ = DrainDetachedAsync();
}
/// <summary>
/// Runs a drain pass under a write lock this writer acquired on behalf of a caller that has
/// already returned, then releases it. An empty queue makes <see cref="DrainQueuedFramesAsync"/>
/// a no-op, so the common case is acquire-drain-nothing-release; the pass earns its keep for a
/// frame enqueued after the previous drainer's last dequeue but before its release.
/// </summary>
/// <returns>A task that completes once the drain pass has ended and the lock has been released.</returns>
private async Task DrainDetachedAsync()
{
try
{
await DrainQueuedFramesAsync().ConfigureAwait(false);
}
catch (Exception)
{
// DrainQueuedFramesAsync routes every write and flush failure onto the affected frames'
// completions, so nothing is expected to escape it. If anything ever does, there is no
// caller left on this pass to receive it and letting it out would only raise an
// unobserved-task exception. The release below is the part that must not be skipped.
}
finally
{
_writeLock.Release();
}
}
// Runs only under _writeLock. Drains control frames before event frames, stamping and writing each.
// The stream write itself is not cancellable: a frame is written atomically or fails, never left
// half-written on the pipe because a caller gave up waiting.