fix(IPC-34): forward a WorkerCancel when the caller abandons a command, not only on gateway timeout
IPC-08 closed with WorkerCancel sent from the gateway's own timeout arm only. Its rationale -- the gateway has stopped waiting but the worker has not stopped working, so a queued-but-not-started command should be dropped instead of occupying the single STA behind a caller that is already gone -- applies exactly as well to caller cancellation, which is the common abandonment path: gRPC clients routinely set per-call deadlines shorter than the gateway's CommandTimeout, and a client disconnect cancels the RPC's token. Both landed in arms that removed the pending entry and returned silently, leaving the correlation to execute in its turn. Under a retry storm against a slow provider the STA works through dead commands while live callers queue behind them -- precisely the backlog the cancel exists to shorten. - TrySendCancelForTimedOutCommand is now TrySendCancelForAbandonedCommand and takes the reason string, since it serves both arms. - Both caller-cancellation arms route through AbandonCommandOnCallerCancellation, which sends the cancel under the same `removed` ownership gate as the timeout arm (so a reply that won the race is never chased) and skips it once _stopCts has fired, where the outbound channel is being completed and every pending command is failed anyway. - New WorkerClientErrorCode.CallerCancelled replaces the GatewayShutdown label those arms used, so metrics and logs stop recording a shutdown that did not happen. It maps to StatusCode.Cancelled, the same status GatewayShutdown maps to, so nothing changes for clients. Docs: docs/Grpc.md mapping line, gateway.md cancellation rules, docs/GatewayProcessDesign.md pending-command section. Tests: InvokeAsync_WhenCallerCancels_SendsWorkerCancelForThatCorrelation (twin of the timeout test, driven by the caller token; verified failing without the send) and InvokeAsync_WhenClientStopsUnderInFlightCommand_DoesNotSendWorkerCancel.
This commit is contained in:
@@ -1051,6 +1051,7 @@ public sealed class MxAccessGatewayService(
|
||||
{
|
||||
WorkerClientErrorCode.CommandTimeout => StatusCode.DeadlineExceeded,
|
||||
WorkerClientErrorCode.GatewayShutdown => StatusCode.Cancelled,
|
||||
WorkerClientErrorCode.CallerCancelled => StatusCode.Cancelled,
|
||||
WorkerClientErrorCode.InvalidState => StatusCode.FailedPrecondition,
|
||||
WorkerClientErrorCode.ProtocolViolation => StatusCode.Internal,
|
||||
WorkerClientErrorCode.CommandTooLarge => StatusCode.ResourceExhausted,
|
||||
|
||||
Reference in New Issue
Block a user