fix(sessions): fault a Ready session whose worker reached a terminal state on its own

Live windev finding: after a client sent the ShutdownWorker control command the worker exited 0 and WorkerClient moved to Closed on the shutdown ack, but GatewaySession.State stayed Ready (even with an event stream attached) until its lease expired. IWorkerClient has no state-change callback, so the session observes the worker from its own worker-event pump source: WorkerClient enters Closed/Faulted before completing the event channel, so when the enumeration ends ObserveWorkerTerminalState sees the terminal state and faults a Ready session. Faulted rather than Closed: Closed means the gateway closed it and would linger in the registry; a faulted session stays observable with the reason and the faulted-session sweep reaps it, accounting SessionClosed exactly once. Only a Ready session follows its worker, so a gateway- initiated close is unaffected. FakeWorkerHarness-based tests cover the ShutdownWorker exit and a broken worker pipe.
2026-10-22 06:48:31 -04:00
parent ec9f26400a
commit cc1e895474
3 changed files with 204 additions and 5 deletions
@@ -1004,12 +1004,54 @@ public sealed class GatewaySession
        IWorkerClient workerClient = await GetReadyWorkerClientAsync(cancellationToken).ConfigureAwait(false);
        TouchClientActivity(_eventStreaming.TimeProvider.GetUtcNow());

-        await foreach (WorkerEvent workerEvent in workerClient
-            .ReadEventsAsync(cancellationToken)
-            .WithCancellation(cancellationToken)
-            .ConfigureAwait(false))
+        try
        {
-            yield return mapper.MapEvent(workerEvent);
+            await foreach (WorkerEvent workerEvent in workerClient
+                .ReadEventsAsync(cancellationToken)
+                .WithCancellation(cancellationToken)
+                .ConfigureAwait(false))
+            {
+                yield return mapper.MapEvent(workerEvent);
+            }
+        }
+        finally
+        {
+            // The worker event stream only ends when the worker client has reached a terminal
+            // state (it enters Closed/Faulted BEFORE completing the channel) or the pump is being
+            // torn down. This is the session's one always-on view of the worker, so it is where
+            // a worker that went away on its own is noticed.
+            ObserveWorkerTerminalState();
+        }
+    }
+
+    // IWorkerClient has no state-change callback, so a worker that reached Closed/Faulted
+    // without the gateway closing it — the client sent ShutdownWorker and the worker exited,
+    // the worker crashed, its pipe broke, a heartbeat timed out — used to leave the session
+    // reporting Ready until its lease expired (live finding). The session follows the worker
+    // into Faulted: it is permanently unusable (every command fails the readiness check), stays
+    // observable via GetSessionStatus with the reason, and the faulted-session sweep reaps it
+    // after FaultedGraceSeconds, accounting SessionClosed exactly once there. Not Closed: Closed
+    // means the gateway closed it, and a Closed session still in the registry would linger until
+    // its lease expired with nothing reaping it. Only a Ready session follows — during a
+    // gateway-initiated close the worker closing is the expected outcome, not a fault.
+    private void ObserveWorkerTerminalState()
+    {
+        lock (_syncRoot)
+        {
+            if (_state is not SessionState.Ready || _workerClient is null)
+            {
+                return;
+            }
+
+            WorkerClientState workerState = _workerClient.State;
+            if (workerState is not (WorkerClientState.Closed or WorkerClientState.Faulted))
+            {
+                return;
+            }
+
+            MarkFaulted(
+                $"Worker for session {SessionId} reached {workerState} outside a gateway-initiated close "
+                + "(worker exited or faulted).");
        }
    }