fix(IPC-34): forward a WorkerCancel when the caller abandons a command, not only on gateway timeout

IPC-08 closed with WorkerCancel sent from the gateway's own timeout arm only.
Its rationale -- the gateway has stopped waiting but the worker has not stopped
working, so a queued-but-not-started command should be dropped instead of
occupying the single STA behind a caller that is already gone -- applies exactly
as well to caller cancellation, which is the common abandonment path: gRPC
clients routinely set per-call deadlines shorter than the gateway's
CommandTimeout, and a client disconnect cancels the RPC's token. Both landed in
arms that removed the pending entry and returned silently, leaving the
correlation to execute in its turn. Under a retry storm against a slow provider
the STA works through dead commands while live callers queue behind them --
precisely the backlog the cancel exists to shorten.

- TrySendCancelForTimedOutCommand is now TrySendCancelForAbandonedCommand and
  takes the reason string, since it serves both arms.
- Both caller-cancellation arms route through AbandonCommandOnCallerCancellation,
  which sends the cancel under the same `removed` ownership gate as the timeout
  arm (so a reply that won the race is never chased) and skips it once _stopCts
  has fired, where the outbound channel is being completed and every pending
  command is failed anyway.
- New WorkerClientErrorCode.CallerCancelled replaces the GatewayShutdown label
  those arms used, so metrics and logs stop recording a shutdown that did not
  happen. It maps to StatusCode.Cancelled, the same status GatewayShutdown maps
  to, so nothing changes for clients.

Docs: docs/Grpc.md mapping line, gateway.md cancellation rules,
docs/GatewayProcessDesign.md pending-command section.

Tests: InvokeAsync_WhenCallerCancels_SendsWorkerCancelForThatCorrelation (twin
of the timeout test, driven by the caller token; verified failing without the
send) and InvokeAsync_WhenClientStopsUnderInFlightCommand_DoesNotSendWorkerCancel.
This commit is contained in:
Joseph Doherty
2026-08-20 09:31:29 -04:00
parent a71663a2a2
commit b27035b3a6
7 changed files with 202 additions and 32 deletions
+17 -6
View File
@@ -593,9 +593,19 @@ Pending command handling:
Timeouts should not assume the COM call stopped. A timed-out command may still
finish inside the worker.
On timeout the client also forwards a `WorkerCancel` carrying the abandoned
correlation id, best-effort: the gateway has stopped waiting, but the worker has
not stopped working, and the worker owns a single STA. `WorkerPipeSession` routes
On timeout **and on caller cancellation** the client also forwards a
`WorkerCancel` carrying the abandoned correlation id, best-effort: the gateway
has stopped waiting, but the worker has not stopped working, and the worker owns
a single STA. Caller cancellation — a per-call deadline shorter than the
gateway's command timeout, or a client disconnect cancelling the RPC's token — is
the more common of the two paths, and under a retry storm it is exactly the
backlog the cancel exists to shorten, so both arms send it; the pending command
is failed as `CallerCancelled` rather than `GatewayShutdown` so metrics and logs
name the real cause (both still map to gRPC `Cancelled`). The send is gated on
the same ownership check as the timeout arm — only the arm that removed the
pending entry sends, so a reply that won the race is never chased — and is
skipped when the client's own stop token has fired, since the outbound channel is
being completed and every pending command is failed anyway. `WorkerPipeSession` routes
the cancel to `CancelCommand`, which drops the correlation from the STA queue if
it has not started and replies `Canceled` for it. A cancel that arrives after the
command reached MXAccess is a no-op — there is no way to abort an in-flight COM
@@ -606,9 +616,10 @@ the worker reads the command and then its cancel and drops it before execution.
Cancels ride the same outbound channel as commands, whose capacity is
`MaxPendingCommands + 4`: the reserve above the pending-command limit is what
absorbs them, so a burst of timeouts stays bounded and cannot deadlock the
enqueue path. Failing to send the cancel is logged at debug and never replaces
the `CommandTimeout` the caller is owed.
absorbs them, so a burst of timeouts or cancellations stays bounded and cannot
deadlock the enqueue path. Failing to send the cancel is logged at debug and
never replaces the `CommandTimeout` — or the `OperationCanceledException` — the
caller is owed.
Cancellation outranks the deadline. When a caller's token is canceled around the
same time the timeout fires, the command is reported as canceled
+1 -1
View File
@@ -282,7 +282,7 @@ StatusCode statusCode = exception.ErrorCode switch
};
```
`WorkerClientException` follows the same pattern: `CommandTimeout` becomes `DeadlineExceeded`, `GatewayShutdown` becomes `Cancelled`, `InvalidState` becomes `FailedPrecondition`, `ProtocolViolation` becomes `Internal`, `CommandTooLarge` becomes `ResourceExhausted`, and unmapped codes fall through to `Unavailable`.
`WorkerClientException` follows the same pattern: `CommandTimeout` becomes `DeadlineExceeded`, `GatewayShutdown` and `CallerCancelled` both become `Cancelled`, `InvalidState` becomes `FailedPrecondition`, `ProtocolViolation` becomes `Internal`, `CommandTooLarge` becomes `ResourceExhausted`, and unmapped codes fall through to `Unavailable`. `CallerCancelled` is the caller's own token firing — a per-call deadline shorter than the gateway command timeout, or a disconnect — and is separate from `GatewayShutdown` only so metrics and logs stop attributing those calls to a shutdown that did not happen; the status the client observes is the same.
## Event Streaming Model