dohertj2/mxaccessgw
1
0
Fork 0
Code Issues 3 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

fix(IPC-34): forward a WorkerCancel when the caller abandons a command, not only on gateway timeout

Browse Source 
IPC-08 closed with WorkerCancel sent from the gateway's own timeout arm only.
Its rationale -- the gateway has stopped waiting but the worker has not stopped
working, so a queued-but-not-started command should be dropped instead of
occupying the single STA behind a caller that is already gone -- applies exactly
as well to caller cancellation, which is the common abandonment path: gRPC
clients routinely set per-call deadlines shorter than the gateway's
CommandTimeout, and a client disconnect cancels the RPC's token. Both landed in
arms that removed the pending entry and returned silently, leaving the
correlation to execute in its turn. Under a retry storm against a slow provider
the STA works through dead commands while live callers queue behind them --
precisely the backlog the cancel exists to shorten.

- TrySendCancelForTimedOutCommand is now TrySendCancelForAbandonedCommand and
  takes the reason string, since it serves both arms.
- Both caller-cancellation arms route through AbandonCommandOnCallerCancellation,
  which sends the cancel under the same `removed` ownership gate as the timeout
  arm (so a reply that won the race is never chased) and skips it once _stopCts
  has fired, where the outbound channel is being completed and every pending
  command is failed anyway.
- New WorkerClientErrorCode.CallerCancelled replaces the GatewayShutdown label
  those arms used, so metrics and logs stop recording a shutdown that did not
  happen. It maps to StatusCode.Cancelled, the same status GatewayShutdown maps
  to, so nothing changes for clients.

Docs: docs/Grpc.md mapping line, gateway.md cancellation rules,
docs/GatewayProcessDesign.md pending-command section.

Tests: InvokeAsync_WhenCallerCancels_SendsWorkerCancelForThatCorrelation (twin
of the timeout test, driven by the caller token; verified failing without the
send) and InvokeAsync_WhenClientStopsUnderInFlightCommand_DoesNotSendWorkerCancel.
This commit is contained in:
Joseph Doherty
2026-08-20 09:31:29 -04:00
parent a71663a2a2
commit b27035b3a6
7 changed files with 202 additions and 32 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/ZB.MOM.WW.MxGateway.Server/Workers/WorkerClient.cs
+59 -24
View File
@@ -291,7 +291,10 @@ public sealed class WorkerClient : IWorkerClient
// throw, so it can never replace the timeout the caller is owed.
if (removed)
{
TrySendCancelForTimedOutCommand(correlationId, method, timeout);
TrySendCancelForAbandonedCommand(
correlationId,
method,
$"gateway command timeout after {timeout}");
}
throw new WorkerClientException(
@@ -300,11 +303,7 @@ public sealed class WorkerClient : IWorkerClient
}
catch (OperationCanceledException)
{
RemovePendingCommandAsFailed(
correlationId,
pendingCommand,
WorkerClientErrorCode.GatewayShutdown,
"Command wait was canceled.");
AbandonCommandOnCallerCancellation(correlationId, pendingCommand, method);
// WaitAsync surfaces TaskCanceledException; throwing through the token keeps the
// exception the caller observes exactly what the hand-rolled wait produced.
@@ -316,11 +315,7 @@ public sealed class WorkerClient : IWorkerClient
// The deadline and the caller's cancellation raced and WaitAsync picked the timer.
// The old wait classified this as cancellation, so this clause — reached only when
// the filter above saw a canceled token — reproduces that treatment exactly.
RemovePendingCommandAsFailed(
correlationId,
pendingCommand,
WorkerClientErrorCode.GatewayShutdown,
"Command wait was canceled.");
AbandonCommandOnCallerCancellation(correlationId, pendingCommand, method);
cancellationToken.ThrowIfCancellationRequested();
throw;
}
@@ -822,6 +817,45 @@ public sealed class WorkerClient : IWorkerClient
return true;
}
/// <summary>
/// Fails the pending command for a caller whose own token fired — a client deadline shorter
/// than the gateway's command timeout, or a disconnect that cancelled the RPC's token — and
/// forwards a <c>WorkerCancel</c> for the correlation the caller has abandoned (IPC-34).
/// <para>
/// Caller cancellation is the common abandonment path, not the rare one, and it leaves the
/// worker in exactly the state the timeout arm's cancel exists for: the correlation is still
/// on the worker's single STA queue and would execute in its turn, behind live callers, for
/// a caller that is already gone. Under a burst of deadline-cancelled calls that backlog is
/// precisely what the cancel was added to shorten, so both arms send it.
/// </para>
/// <para>
/// Gated the same way as the timeout arm: only the caller that owned the removal sends, so a
/// reply that won the race is never chased. Additionally skipped once <c>_stopCts</c> has
/// fired — the client is closing, the outbound channel is being completed and every pending
/// command failed anyway, so a cancel would be at best pointless and at worst a write racing
/// teardown.
/// </para>
/// </summary>
/// <param name="correlationId">Correlation id of the abandoned command.</param>
/// <param name="pendingCommand">The pending command entry to fail.</param>
/// <param name="method">Command method name, for the cancel reason and diagnostics.</param>
private void AbandonCommandOnCallerCancellation(
string correlationId,
PendingCommand pendingCommand,
string method)
{
bool removed = RemovePendingCommandAsFailed(
correlationId,
pendingCommand,
WorkerClientErrorCode.CallerCancelled,
"Command wait was canceled.");
if (removed && !_stopCts.IsCancellationRequested)
{
TrySendCancelForAbandonedCommand(correlationId, method, "caller cancelled the command");
}
}
/// <summary>
/// Forwards a <c>WorkerCancel</c> for a correlation the gateway has given up waiting for, so the
/// worker can drop it from its STA queue (<c>WorkerPipeSession</c> routes the envelope to
@@ -838,9 +872,10 @@ public sealed class WorkerClient : IWorkerClient
/// <remarks>
/// The whole body sits under one catch-all that debug-logs, because the guarantee this method
/// owes its caller is structural, not incidental: the caller is on the throw path for the
/// timeout, so anything escaping here — an envelope that failed to build, a <c>TryWrite</c>
/// against a disposed channel, a scheduler refusing the detached task — would replace the
/// <see cref="WorkerClientErrorCode.CommandTimeout"/> the caller is owed with an unrelated
/// timeout or the cancellation, so anything escaping here — an envelope that failed to build,
/// a <c>TryWrite</c> against a disposed channel, a scheduler refusing the detached task —
/// would replace the <see cref="WorkerClientErrorCode.CommandTimeout"/> (or the
/// <see cref="OperationCanceledException"/>) the caller is owed with an unrelated
/// exception. Losing the cancel costs the worker one wasted command; losing the timeout
/// misreports why the call failed. The detached task carries its own handler for the same
/// reason: its failures (including the <see cref="ObjectDisposedException"/> from
@@ -848,13 +883,13 @@ public sealed class WorkerClient : IWorkerClient
/// and would otherwise be unobserved. It is deliberately not tracked or awaited — it holds no
/// resource the shutdown path needs back, and the outbound channel is completed on close.
/// </remarks>
/// <param name="correlationId">Correlation id of the command that timed out.</param>
/// <param name="method">Command method name, for the cancel reason and diagnostics.</param>
/// <param name="timeout">The elapsed command timeout, for the cancel reason.</param>
private void TrySendCancelForTimedOutCommand(
/// <param name="correlationId">Correlation id of the abandoned command.</param>
/// <param name="method">Command method name, for diagnostics.</param>
/// <param name="reason">Why the gateway abandoned the command, carried on the <c>WorkerCancel</c>.</param>
private void TrySendCancelForAbandonedCommand(
string correlationId,
string method,
TimeSpan timeout)
string reason)
{
try
{
@@ -862,7 +897,7 @@ public sealed class WorkerClient : IWorkerClient
correlationId,
envelope => envelope.WorkerCancel = new WorkerCancel
{
Reason = $"gateway command timeout after {timeout}",
Reason = reason,
});
if (_outboundEnvelopes.Writer.TryWrite(cancelEnvelope))
@@ -888,10 +923,10 @@ public sealed class WorkerClient : IWorkerClient
}
}
/// <summary>Records a cancel that could not be forwarded for a timed-out command.</summary>
/// <summary>Records a cancel that could not be forwarded for an abandoned command.</summary>
/// <param name="exception">The failure that stopped the cancel from being sent.</param>
/// <param name="method">Command method name of the timed-out command.</param>
/// <param name="correlationId">Correlation id of the timed-out command.</param>
/// <param name="method">Command method name of the abandoned command.</param>
/// <param name="correlationId">Correlation id of the abandoned command.</param>
private void LogCancelNotForwarded(
Exception exception,
string method,
@@ -899,7 +934,7 @@ public sealed class WorkerClient : IWorkerClient
{
_logger.LogDebug(
exception,
"Could not forward a cancel for timed-out worker command {Method} on session {SessionId} "
"Could not forward a cancel for abandoned worker command {Method} on session {SessionId} "
+ "and correlation {CorrelationId}.",
method,
SessionId,