mxaccessgw/docs/Sessions.md

Gateway Sessions

The sessions subsystem owns the in-memory representation of an active gateway-to-worker pairing and coordinates its lifecycle from open through close. Each GatewaySession corresponds to exactly one MXAccess worker process connected over a dedicated named pipe.

Overview

A session is the gateway-side handle that callers use to invoke worker commands, stream worker events, and tear the worker down. The subsystem is split between the per-session state machine (GatewaySession), an in-memory directory (SessionRegistry), the orchestrator that opens and closes sessions (SessionManager), the worker construction step (SessionWorkerClientFactory), and a hosted service that drains sessions during host shutdown (SessionShutdownHostedService).

All four interfaces (ISessionManager, ISessionRegistry, ISessionWorkerClientFactory) plus SessionShutdownHostedService are wired as singletons by SessionServiceCollectionExtensions.AddGatewaySessions.

Key Types

GatewaySession

GatewaySession is a sealed class that holds the identity, configured timeouts, worker client reference, and current SessionState for one session. State is protected by a private _syncRoot lock so that property reads and transitions are observed atomically by concurrent gRPC calls and the lease sweeper.

The session id is an opaque string in the form session-{guid:N} and the per-session pipe name is mxaccess-gateway-{ProcessId}-{SessionId}. Encoding the gateway PID into the pipe name avoids collisions when an old gateway process leaks pipes that the OS has not yet reclaimed.

SessionState itself is the protobuf-generated enum from ZB.MOM.WW.MxGateway.Contracts.Proto, so it is shared between the gateway and clients on the wire.

public void TransitionTo(SessionState nextState)
{
    lock (_syncRoot)
    {
        if (_state is SessionState.Closed)
        {
            return;
        }

        if (_state is SessionState.Faulted && nextState is not SessionState.Closed)
        {
            return;
        }

        if (_state is SessionState.Closing
            && nextState is not SessionState.Closed
            && nextState is not SessionState.Faulted)
        {
            return;
        }

        _state = nextState;
    }
}

Closed is terminal, Faulted only allows a transition to Closed, and Closing only allows a transition to Closed or Faulted. This guards against late callbacks (worker exit, heartbeat timeout) re-animating a session that is already tearing down or torn down — once CloseAsync has set Closing under _syncRoot, no TransitionTo(Ready) from another thread can walk the session back to Ready. Both close-related writes (Closing and Closed) go through _syncRoot exactly like every other state write; _closeLock only serializes concurrent close attempts.

SessionManager (ISessionManager)

SessionManager is the orchestrator. It exposes OpenSessionAsync, TryGetSession, InvokeAsync, ReadEventsAsync, CloseSessionAsync, KillWorkerAsync, CloseExpiredLeasesAsync, and ShutdownAsync. It composes ISessionRegistry, ISessionWorkerClientFactory, GatewayMetrics, and GatewayOptions.

CloseSessionAsync and KillWorkerAsync are both end-of-life paths but differ in what they offer the worker:

• CloseSessionAsync is the graceful path: it calls GatewaySession.CloseAsync, which asks the worker to shut down via IWorkerClient.ShutdownAsync and only kills the process as a fallback if shutdown fails.
• KillWorkerAsync is the forceful path used by the dashboard's admin Kill button: it calls GatewaySession.KillWorker directly, which kills the worker process immediately with no graceful-shutdown attempt and transitions the session to Closed.

Both paths converge on the same registry/metrics cleanup, so the open-session slot is released and mxgateway.sessions.closed is incremented either way.

Concurrency is bounded by a SemaphoreSlim initialized to GatewayOptions.Sessions.MaxSessions. Open requests that exceed the bound throw SessionManagerException with SessionLimitExceeded rather than queuing; the caller is expected to retry.

private void EnsureSessionCapacity()
{
    if (!_sessionSlots.Wait(0))
    {
        throw new SessionManagerException(
            SessionManagerErrorCode.SessionLimitExceeded,
            $"Gateway session limit {_options.Sessions.MaxSessions} has been reached.");
    }
}

SessionManager also defines three close-reason constants — DefaultCloseReason ("client-close"), GatewayShutdownReason ("gateway-shutdown"), and LeaseExpiredReason ("lease-expired") — so that the metrics and worker shutdown paths agree on a fixed vocabulary.

SessionRegistry (ISessionRegistry)

SessionRegistry is a thin wrapper over a ConcurrentDictionary<string, GatewaySession> keyed by session id with StringComparer.Ordinal. Snapshot materializes the values into an array so iteration callers (lease sweeper, shutdown) do not race with concurrent TryAdd and TryRemove calls.

ActiveCount filters out sessions whose state is Closed; this is consumed by metrics and the dashboard, where Count would otherwise momentarily over-report during teardown.

SessionWorkerClientFactory (ISessionWorkerClientFactory)

SessionWorkerClientFactory.CreateAsync is the only path that builds an IWorkerClient. It drives the session through the protobuf SessionState substates in order: StartingWorker, WaitingForPipe, Handshaking, InitializingWorker. The substates are wire-visible so the dashboard and clients can render startup progress.

A linked CancellationTokenSource enforces session.StartupTimeout. If startup fails or times out, the factory either kills the partially-constructed WorkerClient or, if the client was never built, kills the launched process and disposes the named pipe before rethrowing. A pure timeout is rewritten as TimeoutException so callers can distinguish it from caller-driven cancellation:

if (exception is OperationCanceledException
    && startupCancellation.IsCancellationRequested
    && !cancellationToken.IsCancellationRequested)
{
    throw new TimeoutException(
        $"Worker session {session.SessionId} did not complete startup within {session.StartupTimeout}.",
        exception);
}

The named pipe is created with maxNumberOfServerInstances: 1 so a second worker cannot connect to the same pipe name even if the first launch is still pending. Combined with the per-session nonce passed to the worker, this is the gateway's defense against a foreign process answering a pipe.

SessionShutdownHostedService

SessionShutdownHostedService is an IHostedService whose only job is to call ISessionManager.ShutdownAsync from StopAsync. It catches OperationCanceledException triggered by the host shutdown timeout and logs a warning so that an over-running shutdown does not surface as an unhandled exception.

SessionOpenRequest

SessionOpenRequest is the gateway-internal record passed to OpenSessionAsync. It is constructed from the wire-level OpenSessionRequest via SessionOpenRequest.FromContract. Keeping a separate internal record means the gRPC layer can normalize input (defaulting backend, sanitizing strings) without leaking generated proto types into SessionManager.

public sealed record SessionOpenRequest(
    string? RequestedBackend,
    string? ClientSessionName,
    string? ClientCorrelationId,
    Duration? CommandTimeout)
{
    public static SessionOpenRequest FromContract(OpenSessionRequest request)
    {
        ArgumentNullException.ThrowIfNull(request);

        return new SessionOpenRequest(
            request.RequestedBackend,
            request.ClientSessionName,
            request.ClientCorrelationId,
            request.CommandTimeout);
    }
}

SessionCloseResult

SessionCloseResult is the record returned from a successful close. AlreadyClosed distinguishes "this call closed the session" from "the session was already closed when we acquired the close lock", which the metrics layer uses to avoid double-counting.

public sealed record SessionCloseResult(
    string SessionId,
    SessionState FinalState,
    bool AlreadyClosed);

SessionCloseStartedException

SessionCloseStartedException is internal and is only thrown from inside GatewaySession.CloseAsync when the close path has already begun mutating worker state and a subsequent step fails. SessionManager.CloseSessionCoreAsync catches it, marks the session faulted, increments the close-failed metric, removes the session from the registry, and rethrows it wrapped as SessionManagerException with CloseFailed. The intermediate type exists so the public API surface only ever exposes SessionManagerException.

SessionManagerException and SessionManagerErrorCode

SessionManagerException is the single public error type emitted from this subsystem; the code is carried in the ErrorCode property and is also surfaced to metrics tags via SessionManagerErrorCode.ToString().

Code	Meaning
SessionNotFound	The session id is not in the registry.
SessionNotReady	The session or its IWorkerClient is not in Ready state.
EventSubscriberAlreadyActive	A second event subscriber attached when only one is allowed.
EventQueueOverflow	Reserved for the worker event channel overflow path.
SessionLimitExceeded	MaxSessions is in use.
OpenFailed	OpenSessionAsync failed; the inner exception carries the cause.
CloseFailed	A close started but did not complete cleanly; the session is removed and faulted.

Lifecycle

Open

SessionManager.OpenSessionAsync allocates a session slot, builds the GatewaySession, registers it, and asks the factory to bring up the worker. Failures roll back every preceding step:

catch (Exception exception)
{
    session?.MarkFaulted(exception.Message);
    if (session is not null)
    {
        _registry.TryRemove(session.SessionId, out _);
        await session.DisposeAsync().ConfigureAwait(false);
    }

    ReleaseSessionSlot();
    _metrics.Fault(SessionManagerErrorCode.OpenFailed.ToString());
    _logger.LogWarning(
        exception,
        "Failed to open gateway session {SessionId}.",
        session?.SessionId ?? "<not-created>");

    throw new SessionManagerException(
        SessionManagerErrorCode.OpenFailed,
        session is null ? "Failed to create session." : $"Failed to open session {session.SessionId}.",
        exception);
}

The order — fault, deregister, dispose, release slot, record metric, log, rethrow — matters because releasing the semaphore before disposal would let the next open race the worker process tear-down on the same machine.

Run

While Ready, callers reach the worker through SessionManager.InvokeAsync or ReadEventsAsync. Both delegate to GatewaySession, which checks the state under lock and updates LastClientActivityAt on every invocation. GatewaySession also exposes typed bulk helpers (AddItemBulkAsync, SubscribeBulkAsync, etc.) that wrap WorkerCommand round-trips and translate non-Ok ProtocolStatus replies into SessionManagerException with SessionNotReady.

Event streaming uses AttachEventSubscriber which returns a disposable lease. When allowMultipleSubscribers is false the second attach throws EventSubscriberAlreadyActive; this prevents two gRPC streams from racing on the same worker event channel. Active event subscribers keep the session lease from expiring until the stream is disposed.

Sessions open with MxGateway:Sessions:DefaultLeaseSeconds (default 1800) added to the open timestamp. Unary client activity refreshes the lease by the same duration. ExtendLease and IsLeaseExpired cooperate with SessionManager.CloseExpiredLeasesAsync, which iterates a registry snapshot and closes any session whose lease has expired with LeaseExpiredReason. SessionLeaseMonitorHostedService runs that sweep every MxGateway:Sessions:LeaseSweepIntervalSeconds seconds (default 30).

Close

GatewaySession.CloseAsync is serialized by a per-session SemaphoreSlim (_closeLock) so only one close runs at a time, but every read/write of _state still passes through _syncRoot (via TryBeginClose and MarkClosed). The close path therefore obeys the same lock discipline as TransitionTo / MarkFaulted: it transitions to Closing, asks the worker client to shut down within ShutdownTimeout, and on success transitions to Closed. DisposeAsync waits on _closeLock once before disposing the semaphore so an in-flight close's Release() cannot race against the dispose. If WorkerClient.ShutdownAsync throws, the session falls back to IWorkerClient.Kill (forced close):

if (_workerClient is not null)
{
    try
    {
        await _workerClient.ShutdownAsync(ShutdownTimeout, cancellationToken).ConfigureAwait(false);
    }
    catch (Exception exception)
    {
        try
        {
            _workerClient.Kill(reason);
        }
        catch (Exception killException)
        {
            throw new SessionCloseStartedException(
                $"Session {SessionId} close failed after worker shutdown started.",
                new AggregateException(exception, killException));
        }

        throw;
    }
}

If both graceful shutdown and the kill fall-back fail, the original and kill exceptions are bundled into an AggregateException and surfaced as SessionCloseStartedException. SessionManager.CloseSessionCoreAsync then translates that into a SessionManagerException with CloseFailed and removes the session.

GatewaySession.KillWorker is the unconditional forced-close path used by shutdown when graceful close itself throws, and also by SessionManager.KillWorkerAsync — the explicit kill path that the dashboard's admin Kill button invokes. KillWorkerAsync skips WorkerClient.ShutdownAsync entirely, so KillCount increments while ShutdownCount does not; the session is then removed from the registry and the open-session slot is released, identical to the cleanup that follows a successful CloseSessionAsync.

Shutdown Coordination

SessionShutdownHostedService.StopAsync calls SessionManager.ShutdownAsync, which closes every registered session with GatewayShutdownReason. The shutdown loop catches per-session exceptions, calls KillWorker, and removes the session so that one stuck worker cannot block the rest of the host:

public async Task ShutdownAsync(CancellationToken cancellationToken)
{
    foreach (GatewaySession session in _registry.Snapshot())
    {
        try
        {
            await CloseSessionCoreAsync(session, GatewayShutdownReason, cancellationToken).ConfigureAwait(false);
        }
        catch (Exception exception)
        {
            _logger.LogWarning(
                exception,
                "Graceful shutdown failed for session {SessionId}; killing worker.",
                session.SessionId);
            if (_registry.TryGet(session.SessionId, out _))
            {
                session.KillWorker(GatewayShutdownReason);
                await RemoveSessionAsync(session).ConfigureAwait(false);
            }
        }
    }
}

Iterating over Snapshot rather than the live dictionary lets RemoveSessionAsync mutate the registry inside the loop without throwing.

Dependency Injection

SessionServiceCollectionExtensions.AddGatewaySessions registers the four singletons and the hosted service:

public static IServiceCollection AddGatewaySessions(this IServiceCollection services)
{
    services.AddSingleton<ISessionRegistry, SessionRegistry>();
    services.AddSingleton<ISessionWorkerClientFactory, SessionWorkerClientFactory>();
    services.AddSingleton<ISessionManager, SessionManager>();
    services.AddHostedService<SessionShutdownHostedService>();

    return services;
}

The registry must be a singleton because its ConcurrentDictionary is the source of truth for session state across the gRPC service, the lease sweeper, the dashboard, and the shutdown hosted service. Registering SessionShutdownHostedService last ensures it is constructed after ISessionManager and therefore drains sessions during host stop.

Related Documentation

• Gateway Process Design
• Gateway Configuration
• Worker Process Launcher