fix(IPC-34): forward a WorkerCancel when the caller abandons a command, not only on gateway timeout
IPC-08 closed with WorkerCancel sent from the gateway's own timeout arm only. Its rationale -- the gateway has stopped waiting but the worker has not stopped working, so a queued-but-not-started command should be dropped instead of occupying the single STA behind a caller that is already gone -- applies exactly as well to caller cancellation, which is the common abandonment path: gRPC clients routinely set per-call deadlines shorter than the gateway's CommandTimeout, and a client disconnect cancels the RPC's token. Both landed in arms that removed the pending entry and returned silently, leaving the correlation to execute in its turn. Under a retry storm against a slow provider the STA works through dead commands while live callers queue behind them -- precisely the backlog the cancel exists to shorten. - TrySendCancelForTimedOutCommand is now TrySendCancelForAbandonedCommand and takes the reason string, since it serves both arms. - Both caller-cancellation arms route through AbandonCommandOnCallerCancellation, which sends the cancel under the same `removed` ownership gate as the timeout arm (so a reply that won the race is never chased) and skips it once _stopCts has fired, where the outbound channel is being completed and every pending command is failed anyway. - New WorkerClientErrorCode.CallerCancelled replaces the GatewayShutdown label those arms used, so metrics and logs stop recording a shutdown that did not happen. It maps to StatusCode.Cancelled, the same status GatewayShutdown maps to, so nothing changes for clients. Docs: docs/Grpc.md mapping line, gateway.md cancellation rules, docs/GatewayProcessDesign.md pending-command section. Tests: InvokeAsync_WhenCallerCancels_SendsWorkerCancelForThatCorrelation (twin of the timeout test, driven by the caller token; verified failing without the send) and InvokeAsync_WhenClientStopsUnderInFlightCommand_DoesNotSendWorkerCancel.
This commit is contained in:
+10
-1
@@ -917,10 +917,19 @@ Timeouts:
|
||||
|
||||
Cancellation:
|
||||
|
||||
- canceling the gRPC call should stop waiting in the gateway,
|
||||
- canceling the gRPC call stops the gateway waiting **and** forwards a
|
||||
`WorkerCancel` for the abandoned correlation, the same best-effort frame the
|
||||
command timeout sends — a client deadline or a disconnect is the common
|
||||
abandonment path, and leaving the correlation on the worker's single STA queue
|
||||
makes live callers queue behind work nobody is waiting for,
|
||||
- the cancel is sent only by the caller that owned the pending-entry removal (so
|
||||
a reply that won the race is never chased) and is skipped once the client is
|
||||
stopping, where every pending command fails anyway,
|
||||
- it cannot safely abort an in-flight COM call on the STA,
|
||||
- the worker should finish the COM call and discard or log the late reply if
|
||||
the correlation was canceled,
|
||||
- caller cancellation is reported as `CallerCancelled` (mapped to gRPC
|
||||
`Cancelled`), distinct from a gateway shutdown, so metrics name the real cause,
|
||||
- hard cancellation means killing the worker process.
|
||||
|
||||
## Event Delivery And Backpressure
|
||||
|
||||
Reference in New Issue
Block a user