fix(sessions): refuse OpenSession while the gateway is stopping; drain until the registry is empty
ShutdownAsync iterated a single registry snapshot while Kestrel still accepted OpenSession calls (ApplicationStopping fires before hosted services stop), so a session opened behind the snapshot had nobody left to close it and its x86 worker outlived the gateway as an orphan. SessionManager now observes IHostApplicationLifetime.ApplicationStopping (and sets the same flag at the start of ShutdownAsync) and rejects OpenSessionAsync with the new SessionManagerErrorCode.GatewayStopping, re-checked after registration so the flag write before the drain's snapshot closes the race. GatewayStopping maps to gRPC Unavailable and is not a fault. ShutdownAsync re-snapshots until the registry is empty, bounded at three passes.
This commit is contained in:
@@ -306,6 +306,8 @@ StatusCode statusCode = exception.ErrorCode switch
|
||||
SessionManagerErrorCode.SessionLimitExceeded => StatusCode.ResourceExhausted,
|
||||
SessionManagerErrorCode.OpenFailed => StatusCode.Unavailable,
|
||||
SessionManagerErrorCode.CloseFailed => StatusCode.Unavailable,
|
||||
SessionManagerErrorCode.PermissionDenied => StatusCode.PermissionDenied,
|
||||
SessionManagerErrorCode.GatewayStopping => StatusCode.Unavailable,
|
||||
_ => StatusCode.Unavailable,
|
||||
};
|
||||
```
|
||||
|
||||
+41
-24
@@ -164,6 +164,8 @@ public sealed record SessionCloseResult(
|
||||
| `SessionLimitExceeded` | `MaxSessions` is in use. |
|
||||
| `OpenFailed` | `OpenSessionAsync` failed; the inner exception carries the cause. |
|
||||
| `CloseFailed` | A close started but did not complete cleanly; the session is removed and faulted. |
|
||||
| `PermissionDenied` | The caller may not perform the operation on this session (for example, attaching `StreamEvents` to a session opened by a different API key). |
|
||||
| `GatewayStopping` | The gateway is stopping and refuses new sessions (see [Shutdown Coordination](#shutdown-coordination)). Maps to gRPC `Unavailable`; not recorded as a fault. |
|
||||
|
||||
## Lifecycle
|
||||
|
||||
@@ -306,40 +308,55 @@ If both graceful shutdown and the kill fall-back fail, the original and kill exc
|
||||
`SessionShutdownHostedService.StopAsync` calls `SessionManager.ShutdownAsync`, which closes every registered session with `GatewayShutdownReason`. Sessions are drained with the same bounded fan-out the lease sweep uses (`MaxParallelSessionCloses`), because a one-at-a-time drain of a full registry at a worst-case worker shutdown timeout each outruns any host stop-timeout and leaves the tail to the orphan killer. Each iteration catches its own exceptions — *every* exception, including from the fallback — calls `KillWorkerAsync` on an uncancellable token, and removes the session, so that neither one stuck worker nor one failing teardown can block or abort the rest of the host's drain:
|
||||
|
||||
```csharp
|
||||
await Parallel.ForEachAsync(
|
||||
_registry.Snapshot(),
|
||||
new ParallelOptions { MaxDegreeOfParallelism = MaxParallelSessionCloses },
|
||||
async (session, _) =>
|
||||
MarkStopping();
|
||||
for (int pass = 0; pass < MaxShutdownDrainPasses; pass++)
|
||||
{
|
||||
if (_registry.Count == 0)
|
||||
{
|
||||
try
|
||||
return;
|
||||
}
|
||||
|
||||
await Parallel.ForEachAsync(
|
||||
_registry.Snapshot(),
|
||||
new ParallelOptions { MaxDegreeOfParallelism = MaxParallelSessionCloses },
|
||||
async (session, _) =>
|
||||
{
|
||||
await CloseSessionCoreAsync(session, GatewayShutdownReason, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
_logger.LogWarning(
|
||||
exception,
|
||||
"Graceful shutdown failed for session {SessionId}; killing worker.",
|
||||
session.SessionId);
|
||||
if (_registry.TryGet(session.SessionId, out GatewaySession? registeredSession)
|
||||
&& registeredSession is not null)
|
||||
try
|
||||
{
|
||||
try
|
||||
await CloseSessionCoreAsync(session, GatewayShutdownReason, cancellationToken).ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception exception)
|
||||
{
|
||||
_logger.LogWarning(
|
||||
exception,
|
||||
"Graceful shutdown failed for session {SessionId}; killing worker.",
|
||||
session.SessionId);
|
||||
if (_registry.TryGet(session.SessionId, out GatewaySession? registeredSession)
|
||||
&& registeredSession is not null)
|
||||
{
|
||||
// Not the caller's token: the kill is the last-resort orphan preventer.
|
||||
await KillWorkerAsync(session.SessionId, GatewayShutdownReason, CancellationToken.None).ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception killException)
|
||||
{
|
||||
_logger.LogWarning(killException, "Worker kill fallback failed for session {SessionId}.", session.SessionId);
|
||||
try
|
||||
{
|
||||
// Not the caller's token: the kill is the last-resort orphan preventer.
|
||||
await KillWorkerAsync(session.SessionId, GatewayShutdownReason, CancellationToken.None).ConfigureAwait(false);
|
||||
}
|
||||
catch (Exception killException)
|
||||
{
|
||||
_logger.LogWarning(killException, "Worker kill fallback failed for session {SessionId}.", session.SessionId);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}).ConfigureAwait(false);
|
||||
}).ConfigureAwait(false);
|
||||
}
|
||||
```
|
||||
|
||||
Iterating over `Snapshot` rather than the live dictionary lets `RemoveSessionAsync` mutate the registry from inside the loop without throwing, and gives the parallel drain a stable, already-materialized source.
|
||||
|
||||
### Stopping gate
|
||||
|
||||
A drain that only ever looked at one snapshot could be outrun by `OpenSession`: the host raises `ApplicationStopping` *before* hosted services stop, and Kestrel keeps accepting gRPC calls until its own `StopAsync` runs, so a session opened behind the snapshot would have nobody left to close it and its x86 worker would survive the gateway as an orphan. `SessionManager` therefore keeps a stopping flag that is set by the host's `IHostApplicationLifetime.ApplicationStopping` (the manager registers for it in its constructor when the lifetime is injected) and, as a fallback for hosts without one, at the start of `ShutdownAsync`. Once set it is never cleared. `OpenSessionAsync` checks it twice: on entry, before a slot is taken, and again right after the session is registered. The second check is what closes the race — the flag is written before the drain takes its registry snapshot, so an open that does not observe it there registered its session before that snapshot and is drained, while one that does observe it rolls its own session back. Either way the caller receives `SessionManagerException` with `GatewayStopping`, which the gRPC layer maps to `Unavailable` so the client retries against the restarted gateway; it is not an open failure and records no fault metric.
|
||||
|
||||
The drain loop above is the backstop to that gate: it re-snapshots until the registry is empty, bounded at `MaxShutdownDrainPasses` (`3`) so a session that somehow cannot be removed cannot hold the host stop forever. Anything still registered after the last pass is logged and left to the orphan terminator on the next start.
|
||||
|
||||
## Dependency Injection
|
||||
|
||||
`SessionServiceCollectionExtensions.AddGatewaySessions` registers the four singletons and the hosted service:
|
||||
|
||||
Reference in New Issue
Block a user