fix(worker): bound ShutdownAsync's envelope enqueue by the shutdown timeout
WorkerClient.ShutdownAsync marked the client Closing and then queued the WorkerShutdown envelope on the caller's token alone; the shutdown-timeout CTS was only created afterwards. The outbound channel is bounded, and a worker that has stopped reading the pipe parks the write loop in a pipe write it can only leave through _stopCts while the channel behind it fills with commands and their cancels. Once Closing, the heartbeat watchdog skips every tick, so nothing cancelled _stopCts and the enqueue waited for the caller's token - the lease sweep's stopping token, which fires at host stop and never before - with GatewaySession.CloseAsync holding its close lock throughout. Create the linked timeout CTS before the enqueue and run the enqueue, the loop waits and the process-exit wait on it, so the whole call is bounded by the shutdown timeout. A stalled queue now surfaces as the existing ShutdownTimeout fault, which cancels the loops, kills the worker process tree and hands GatewaySession its kill fallback. Test: WorkerClientTests.ShutdownAsync_WhenOutboundQueueIsFullBehindStalledPipeWrite_TimesOutAndKillsWorker stalls the gateway-side stream (a worker that stopped reading), fills the outbound channel with abandoned commands and their cancels, and asserts that ShutdownAsync throws ShutdownTimeout inside the test bound and kills the fake worker. It hung past the bound before the fix.
This commit is contained in:
@@ -177,9 +177,17 @@ return a successful close result with the final known state.
|
||||
|
||||
`WorkerClient.ShutdownAsync` sends `WorkerShutdown`, waits for the worker read,
|
||||
write, and heartbeat loops to stop, and waits for the launched worker process to
|
||||
exit within the same shutdown timeout. If the pipe loops or process exit exceed
|
||||
the timeout, the close operation fails with `ShutdownTimeout`; `GatewaySession`
|
||||
then kills the worker process tree before surfacing the close failure.
|
||||
exit within the same shutdown timeout. The timeout starts before the
|
||||
`WorkerShutdown` envelope is queued, not after: the outbound channel is bounded,
|
||||
and a worker that has stopped reading the pipe parks the write loop in a pipe
|
||||
write while the channel behind it fills with commands and their cancels. Because
|
||||
the heartbeat watchdog only watches a `Ready` client, nothing else would end that
|
||||
stall once the client is `Closing`, so an enqueue bound to the caller's token
|
||||
alone waited for that token — for the lease sweep, until host stop — with the
|
||||
session's close lock held. If queuing the envelope, the pipe loops, or process
|
||||
exit exceed the timeout, the close operation fails with `ShutdownTimeout` (the
|
||||
fault cancels the loops and kills the worker); `GatewaySession` then runs its
|
||||
kill fallback before surfacing the close failure.
|
||||
|
||||
### Invoke
|
||||
|
||||
|
||||
Reference in New Issue
Block a user