fix(IPC-34): forward a WorkerCancel when the caller abandons a command, not only on gateway timeout
IPC-08 closed with WorkerCancel sent from the gateway's own timeout arm only. Its rationale -- the gateway has stopped waiting but the worker has not stopped working, so a queued-but-not-started command should be dropped instead of occupying the single STA behind a caller that is already gone -- applies exactly as well to caller cancellation, which is the common abandonment path: gRPC clients routinely set per-call deadlines shorter than the gateway's CommandTimeout, and a client disconnect cancels the RPC's token. Both landed in arms that removed the pending entry and returned silently, leaving the correlation to execute in its turn. Under a retry storm against a slow provider the STA works through dead commands while live callers queue behind them -- precisely the backlog the cancel exists to shorten. - TrySendCancelForTimedOutCommand is now TrySendCancelForAbandonedCommand and takes the reason string, since it serves both arms. - Both caller-cancellation arms route through AbandonCommandOnCallerCancellation, which sends the cancel under the same `removed` ownership gate as the timeout arm (so a reply that won the race is never chased) and skips it once _stopCts has fired, where the outbound channel is being completed and every pending command is failed anyway. - New WorkerClientErrorCode.CallerCancelled replaces the GatewayShutdown label those arms used, so metrics and logs stop recording a shutdown that did not happen. It maps to StatusCode.Cancelled, the same status GatewayShutdown maps to, so nothing changes for clients. Docs: docs/Grpc.md mapping line, gateway.md cancellation rules, docs/GatewayProcessDesign.md pending-command section. Tests: InvokeAsync_WhenCallerCancels_SendsWorkerCancelForThatCorrelation (twin of the timeout test, driven by the caller token; verified failing without the send) and InvokeAsync_WhenClientStopsUnderInFlightCommand_DoesNotSendWorkerCancel.
This commit is contained in:
@@ -593,9 +593,19 @@ Pending command handling:
|
||||
Timeouts should not assume the COM call stopped. A timed-out command may still
|
||||
finish inside the worker.
|
||||
|
||||
On timeout the client also forwards a `WorkerCancel` carrying the abandoned
|
||||
correlation id, best-effort: the gateway has stopped waiting, but the worker has
|
||||
not stopped working, and the worker owns a single STA. `WorkerPipeSession` routes
|
||||
On timeout **and on caller cancellation** the client also forwards a
|
||||
`WorkerCancel` carrying the abandoned correlation id, best-effort: the gateway
|
||||
has stopped waiting, but the worker has not stopped working, and the worker owns
|
||||
a single STA. Caller cancellation — a per-call deadline shorter than the
|
||||
gateway's command timeout, or a client disconnect cancelling the RPC's token — is
|
||||
the more common of the two paths, and under a retry storm it is exactly the
|
||||
backlog the cancel exists to shorten, so both arms send it; the pending command
|
||||
is failed as `CallerCancelled` rather than `GatewayShutdown` so metrics and logs
|
||||
name the real cause (both still map to gRPC `Cancelled`). The send is gated on
|
||||
the same ownership check as the timeout arm — only the arm that removed the
|
||||
pending entry sends, so a reply that won the race is never chased — and is
|
||||
skipped when the client's own stop token has fired, since the outbound channel is
|
||||
being completed and every pending command is failed anyway. `WorkerPipeSession` routes
|
||||
the cancel to `CancelCommand`, which drops the correlation from the STA queue if
|
||||
it has not started and replies `Canceled` for it. A cancel that arrives after the
|
||||
command reached MXAccess is a no-op — there is no way to abort an in-flight COM
|
||||
@@ -606,9 +616,10 @@ the worker reads the command and then its cancel and drops it before execution.
|
||||
|
||||
Cancels ride the same outbound channel as commands, whose capacity is
|
||||
`MaxPendingCommands + 4`: the reserve above the pending-command limit is what
|
||||
absorbs them, so a burst of timeouts stays bounded and cannot deadlock the
|
||||
enqueue path. Failing to send the cancel is logged at debug and never replaces
|
||||
the `CommandTimeout` the caller is owed.
|
||||
absorbs them, so a burst of timeouts or cancellations stays bounded and cannot
|
||||
deadlock the enqueue path. Failing to send the cancel is logged at debug and
|
||||
never replaces the `CommandTimeout` — or the `OperationCanceledException` — the
|
||||
caller is owed.
|
||||
|
||||
Cancellation outranks the deadline. When a caller's token is canceled around the
|
||||
same time the timeout fires, the command is reported as canceled
|
||||
|
||||
+1
-1
@@ -282,7 +282,7 @@ StatusCode statusCode = exception.ErrorCode switch
|
||||
};
|
||||
```
|
||||
|
||||
`WorkerClientException` follows the same pattern: `CommandTimeout` becomes `DeadlineExceeded`, `GatewayShutdown` becomes `Cancelled`, `InvalidState` becomes `FailedPrecondition`, `ProtocolViolation` becomes `Internal`, `CommandTooLarge` becomes `ResourceExhausted`, and unmapped codes fall through to `Unavailable`.
|
||||
`WorkerClientException` follows the same pattern: `CommandTimeout` becomes `DeadlineExceeded`, `GatewayShutdown` and `CallerCancelled` both become `Cancelled`, `InvalidState` becomes `FailedPrecondition`, `ProtocolViolation` becomes `Internal`, `CommandTooLarge` becomes `ResourceExhausted`, and unmapped codes fall through to `Unavailable`. `CallerCancelled` is the caller's own token firing — a per-call deadline shorter than the gateway command timeout, or a disconnect — and is separate from `GatewayShutdown` only so metrics and logs stop attributing those calls to a shutdown that did not happen; the status the client observes is the same.
|
||||
|
||||
## Event Streaming Model
|
||||
|
||||
|
||||
Reference in New Issue
Block a user