fix(IPC-34): forward a WorkerCancel when the caller abandons a command, not only on gateway timeout

IPC-08 closed with WorkerCancel sent from the gateway's own timeout arm only.
Its rationale -- the gateway has stopped waiting but the worker has not stopped
working, so a queued-but-not-started command should be dropped instead of
occupying the single STA behind a caller that is already gone -- applies exactly
as well to caller cancellation, which is the common abandonment path: gRPC
clients routinely set per-call deadlines shorter than the gateway's
CommandTimeout, and a client disconnect cancels the RPC's token. Both landed in
arms that removed the pending entry and returned silently, leaving the
correlation to execute in its turn. Under a retry storm against a slow provider
the STA works through dead commands while live callers queue behind them --
precisely the backlog the cancel exists to shorten.

- TrySendCancelForTimedOutCommand is now TrySendCancelForAbandonedCommand and
  takes the reason string, since it serves both arms.
- Both caller-cancellation arms route through AbandonCommandOnCallerCancellation,
  which sends the cancel under the same `removed` ownership gate as the timeout
  arm (so a reply that won the race is never chased) and skips it once _stopCts
  has fired, where the outbound channel is being completed and every pending
  command is failed anyway.
- New WorkerClientErrorCode.CallerCancelled replaces the GatewayShutdown label
  those arms used, so metrics and logs stop recording a shutdown that did not
  happen. It maps to StatusCode.Cancelled, the same status GatewayShutdown maps
  to, so nothing changes for clients.

Docs: docs/Grpc.md mapping line, gateway.md cancellation rules,
docs/GatewayProcessDesign.md pending-command section.

Tests: InvokeAsync_WhenCallerCancels_SendsWorkerCancelForThatCorrelation (twin
of the timeout test, driven by the caller token; verified failing without the
send) and InvokeAsync_WhenClientStopsUnderInFlightCommand_DoesNotSendWorkerCancel.
This commit is contained in:
Joseph Doherty
2026-08-20 09:31:29 -04:00
parent a71663a2a2
commit b27035b3a6
7 changed files with 202 additions and 32 deletions
+10 -1
View File
@@ -917,10 +917,19 @@ Timeouts:
Cancellation:
- canceling the gRPC call should stop waiting in the gateway,
- canceling the gRPC call stops the gateway waiting **and** forwards a
`WorkerCancel` for the abandoned correlation, the same best-effort frame the
command timeout sends — a client deadline or a disconnect is the common
abandonment path, and leaving the correlation on the worker's single STA queue
makes live callers queue behind work nobody is waiting for,
- the cancel is sent only by the caller that owned the pending-entry removal (so
a reply that won the race is never chased) and is skipped once the client is
stopping, where every pending command fails anyway,
- it cannot safely abort an in-flight COM call on the STA,
- the worker should finish the COM call and discard or log the late reply if
the correlation was canceled,
- caller cancellation is reported as `CallerCancelled` (mapped to gRPC
`Cancelled`), distinct from a gateway shutdown, so metrics name the real cause,
- hard cancellation means killing the worker process.
## Event Delivery And Backpressure