fix(worker): bound ShutdownAsync's envelope enqueue by the shutdown timeout

WorkerClient.ShutdownAsync marked the client Closing and then queued the
WorkerShutdown envelope on the caller's token alone; the shutdown-timeout CTS
was only created afterwards. The outbound channel is bounded, and a worker that
has stopped reading the pipe parks the write loop in a pipe write it can only
leave through _stopCts while the channel behind it fills with commands and
their cancels. Once Closing, the heartbeat watchdog skips every tick, so nothing
cancelled _stopCts and the enqueue waited for the caller's token - the lease
sweep's stopping token, which fires at host stop and never before - with
GatewaySession.CloseAsync holding its close lock throughout.

Create the linked timeout CTS before the enqueue and run the enqueue, the loop
waits and the process-exit wait on it, so the whole call is bounded by the
shutdown timeout. A stalled queue now surfaces as the existing ShutdownTimeout
fault, which cancels the loops, kills the worker process tree and hands
GatewaySession its kill fallback.

Test: WorkerClientTests.ShutdownAsync_WhenOutboundQueueIsFullBehindStalledPipeWrite_TimesOutAndKillsWorker
stalls the gateway-side stream (a worker that stopped reading), fills the
outbound channel with abandoned commands and their cancels, and asserts that
ShutdownAsync throws ShutdownTimeout inside the test bound and kills the fake
worker. It hung past the bound before the fix.
This commit is contained in:
Joseph Doherty
2026-10-22 06:10:52 -04:00
parent 488649ce99
commit b0100e738a
4 changed files with 228 additions and 8 deletions
+11 -3
View File
@@ -177,9 +177,17 @@ return a successful close result with the final known state.
`WorkerClient.ShutdownAsync` sends `WorkerShutdown`, waits for the worker read,
write, and heartbeat loops to stop, and waits for the launched worker process to
exit within the same shutdown timeout. If the pipe loops or process exit exceed
the timeout, the close operation fails with `ShutdownTimeout`; `GatewaySession`
then kills the worker process tree before surfacing the close failure.
exit within the same shutdown timeout. The timeout starts before the
`WorkerShutdown` envelope is queued, not after: the outbound channel is bounded,
and a worker that has stopped reading the pipe parks the write loop in a pipe
write while the channel behind it fills with commands and their cancels. Because
the heartbeat watchdog only watches a `Ready` client, nothing else would end that
stall once the client is `Closing`, so an enqueue bound to the caller's token
alone waited for that token — for the lease sweep, until host stop — with the
session's close lock held. If queuing the envelope, the pipe loops, or process
exit exceed the timeout, the close operation fails with `ShutdownTimeout` (the
fault cancels the loops and kills the worker); `GatewaySession` then runs its
kill fallback before surfacing the close failure.
### Invoke