perf(sessions): bounded-parallel teardown in lease sweep and shutdown

This commit is contained in:
Joseph Doherty
2026-08-15 12:43:30 -04:00
parent a1a38b5538
commit 0bc13b5292
3 changed files with 267 additions and 36 deletions
+19 -10
View File
@@ -207,6 +207,12 @@ The repair transitions the monitor's reconcile broadcasts on the alarm feed (Rai
Sessions open with `MxGateway:Sessions:DefaultLeaseSeconds` (default 1800) added to the open timestamp. Unary client activity refreshes the lease by the same duration. `ExtendLease` and `IsLeaseExpired` cooperate with `SessionManager.CloseExpiredLeasesAsync`, which iterates a registry snapshot and closes any session whose lease has expired with `LeaseExpiredReason`. `SessionLeaseMonitorHostedService` runs that sweep every `MxGateway:Sessions:LeaseSweepIntervalSeconds` seconds (default 30).
#### Teardown parallelism
A sweep pass is two phases. *Selection* stays a single sequential pass over the snapshot, because that is what gives the precedence rule (lease-expiry, then faulted, then detach-grace) and the `TryBeginCloseIfExpired` TOCTOU re-check their meaning. *Closing* then runs over the already-selected set with `Parallel.ForEachAsync` at `MaxParallelSessionCloses` (4). Each close is bounded by `MxGateway:Worker:ShutdownTimeoutSeconds` (default 10), so a one-at-a-time sweep lets a few hung workers serialize reaping and starve session slots for the rest. Parallel closing is safe because `TryBeginCloseIfExpired` already flipped each selected session to `Closing` under its own lock — that idempotent begin-close is the per-session exclusivity invariant, so no two teardowns can ever run against one session. The degree is bounded rather than unlimited because every concurrent close is one x86 worker process being shut down or killed; the fan-out exists to hide a few hung workers, not to tear the whole registry down at once.
A close that throws no longer abandons the rest of the selected set: the sweep captures the first failure, finishes the remaining closes, then rethrows it so `SessionLeaseMonitorHostedService` still logs the sweep failure as before. `ShutdownAsync` drains sessions with the same bounded fan-out, keeping its existing per-session catch → `KillWorkerAsync` fallback.
#### Detach-grace retention
`MxGateway:Sessions:DetachGraceSeconds` (default 30) is a bounded retention window kept after a session's *last external (gRPC) event-stream subscriber* drops, so a client can reconnect to the same session instead of having it torn down on the first stream disconnect. While the window is open the session stays `Ready` and fully usable — worker commands continue to work and a reconnecting subscriber re-attaches normally. Because retention is keyed on the *external* subscriber count (`_activeEventSubscriberCount`), and the gateway-owned internal dashboard mirror registers directly on the distributor with `isInternal: true` and is therefore *not* counted, a session whose only remaining subscriber is the dashboard mirror still enters detach-grace.
@@ -276,16 +282,21 @@ If both graceful shutdown and the kill fall-back fail, the original and kill exc
## Shutdown Coordination
`SessionShutdownHostedService.StopAsync` calls `SessionManager.ShutdownAsync`, which closes every registered session with `GatewayShutdownReason`. The shutdown loop catches per-session exceptions, calls `KillWorker`, and removes the session so that one stuck worker cannot block the rest of the host:
`SessionShutdownHostedService.StopAsync` calls `SessionManager.ShutdownAsync`, which closes every registered session with `GatewayShutdownReason`. Sessions are drained with the same bounded fan-out the lease sweep uses (`MaxParallelSessionCloses`), because a one-at-a-time drain of a full registry at a worst-case worker shutdown timeout each outruns any host stop-timeout and leaves the tail to the orphan killer. Each iteration catches its own exceptions, calls `KillWorkerAsync`, and removes the session so that one stuck worker cannot block the rest of the host:
```csharp
public async Task ShutdownAsync(CancellationToken cancellationToken)
{
foreach (GatewaySession session in _registry.Snapshot())
await Parallel.ForEachAsync(
_registry.Snapshot(),
new ParallelOptions
{
MaxDegreeOfParallelism = MaxParallelSessionCloses,
CancellationToken = cancellationToken,
},
async (session, closeToken) =>
{
try
{
await CloseSessionCoreAsync(session, GatewayShutdownReason, cancellationToken).ConfigureAwait(false);
await CloseSessionCoreAsync(session, GatewayShutdownReason, closeToken).ConfigureAwait(false);
}
catch (Exception exception)
{
@@ -295,15 +306,13 @@ public async Task ShutdownAsync(CancellationToken cancellationToken)
session.SessionId);
if (_registry.TryGet(session.SessionId, out _))
{
session.KillWorker(GatewayShutdownReason);
await RemoveSessionAsync(session).ConfigureAwait(false);
await KillWorkerAsync(session.SessionId, GatewayShutdownReason, closeToken).ConfigureAwait(false);
}
}
}
}
}).ConfigureAwait(false);
```
Iterating over `Snapshot` rather than the live dictionary lets `RemoveSessionAsync` mutate the registry inside the loop without throwing.
Iterating over `Snapshot` rather than the live dictionary lets `RemoveSessionAsync` mutate the registry from inside the loop without throwing, and gives the parallel drain a stable, already-materialized source.
## Dependency Injection