dohertj2/mxaccessgw
1
0
Fork 0
Code Issues 3 Pull Requests 1 Actions Packages Projects Releases Wiki Activity

fix(sessions): refuse late worker attach after close during startup; route open rollback through removal path

Browse Source 
A session is registered before its worker is created, so an admin Kill or
the host drain could close it during the startup window and the worker
client that arrived afterwards was stored on the dead session, MarkReady
silently refused, SessionOpened recorded and the Closed session returned
to the caller with a live x86 worker nothing would ever stop.

AttachWorkerClient now returns false under _syncRoot when the session is
Closing/Closed/Faulted or disposed; the manager runs attach + MarkReady +
SessionOpened under the per-session close gate, discards (kills and
disposes) a refused worker client and fails the open with OpenFailed.
OpenRecorded gates the closed-side accounting so a never-opened session is
not counted as closed.

The open rollback goes through RemoveSessionAsync so per-session event
metrics are dropped like every other teardown.
This commit is contained in:
Joseph Doherty
2026-10-22 06:16:53 -04:00
parent 488649ce99
commit 87f23d9712
5 changed files with 389 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Sessions.md
+10 -9
View File
@@ -174,14 +174,7 @@ public sealed record SessionCloseResult(
```csharp
catch (Exception exception)
{
session?.MarkFaulted(exception.Message);
if (session is not null)
{
_registry.TryRemove(session.SessionId, out _);
await session.DisposeAsync().ConfigureAwait(false);
}
ReleaseSessionSlot();
await RollBackOpenAsync(session, registered, exception).ConfigureAwait(false);
_metrics.Fault(SessionManagerErrorCode.OpenFailed.ToString());
_logger.LogWarning(
exception,
@@ -195,7 +188,15 @@ catch (Exception exception)
}
```
The order — fault, deregister, dispose, release slot, record metric, log, rethrow — matters because releasing the semaphore before disposal would let the next open race the worker process tear-down on the same machine.
`RollBackOpenAsync` faults the session and, when it had been registered, routes it through `RemoveSessionAsync` — the single removal path every teardown uses, which deregisters, drops the session's per-session event metrics, releases the slot and disposes, in that order. The order matters because releasing the semaphore before disposal would let the next open race the worker process tear-down on the same machine. A session that was never registered (id collision) releases the slot and disposes directly. The open-session gauge is not touched by the rollback: `SessionOpened` is recorded inside the attach gate described next, after which nothing in the open sequence can fail.
#### Close or kill during worker startup
The session is registered — and therefore visible to the dashboard's Kill button, the lease sweep and the host drain — for the whole worker startup, which can last up to `MxGateway:Worker:StartupTimeoutSeconds`. A kill or drain that lands in that window finds no worker client to stop, flips the session to `Closed`, removes it and disposes it. The worker client that `SessionWorkerClientFactory.CreateAsync` then returns must not be stored on that dead session and handed back to the caller as a successful open: nothing would ever shut the worker down (the close already ran) and the client would hold a `Closed` session.
`GatewaySession.AttachWorkerClient` therefore returns `false`, under `_syncRoot`, when the session is already `Closing`, `Closed` or `Faulted` or has been disposed. The manager runs the end of the open sequence — attach, `MarkReady`, `SessionOpened` — through `TryAttachWorkerClientAndMarkReadyAsync`, which holds the per-session close gate (`_closeLock`) so a concurrent `CloseAsync` / `KillWorkerWithCloseGateAsync` runs either entirely before it (the attach is refused) or entirely after it (an ordinary close of a Ready session). On a refused attach the manager kills and disposes the late worker client and the open fails with `OpenFailed`; the registry is already empty and the slot was released by whichever teardown closed the session.
Because such a session was never counted as opened, it must not be counted as closed either. `GatewaySession.OpenRecorded` is set inside the same gate right after `SessionOpened` is recorded, and every teardown path (`KillWorkerAsync`, `CloseSessionCoreAsync`, the close-failed branch) records `SessionClosed` / `SessionRemoved` only when it is set — so `sessions.opened`, `sessions.closed` and the `sessions.open` gauge stay consistent across the startup window.
### Run