mxaccessgw/docs/Sessions.md

Gateway Sessions

The sessions subsystem owns the in-memory representation of an active gateway-to-worker pairing and coordinates its lifecycle from open through close. Each GatewaySession corresponds to exactly one MXAccess worker process connected over a dedicated named pipe.

Overview

A session is the gateway-side handle that callers use to invoke worker commands, stream worker events, and tear the worker down. The subsystem is split between the per-session state machine (GatewaySession), an in-memory directory (SessionRegistry), the orchestrator that opens and closes sessions (SessionManager), the worker construction step (SessionWorkerClientFactory), and a hosted service that drains sessions during host shutdown (SessionShutdownHostedService).

All three interfaces (ISessionManager, ISessionRegistry, ISessionWorkerClientFactory) plus SessionShutdownHostedService are wired as singletons by SessionServiceCollectionExtensions.AddGatewaySessions.

Key Types

GatewaySession

GatewaySession is a sealed class that holds the identity, configured timeouts, worker client reference, and current SessionState for one session. State is protected by a private _syncRoot lock so that property reads and transitions are observed atomically by concurrent gRPC calls and the lease sweeper.

The session id is an opaque string in the form session-{guid:N} and the per-session pipe name is mxgw-{ProcessId}-{guid:N} (the same guid hex, without the session- prefix). Encoding the gateway PID into the pipe name avoids collisions when an old gateway process leaks pipes that the OS has not yet reclaimed. The name is kept short because .NET named pipes on Unix-like hosts are Unix domain sockets at $TMPDIR/CoreFxPipe_{name}, and macOS caps that path at 104 bytes while its default per-user TMPDIR is already ~49 — the old mxaccess-gateway-{pid}-{sessionId} form overflowed it and broke the fake-worker/e2e tests on macOS.

SessionState itself is the protobuf-generated enum from ZB.MOM.WW.MxGateway.Contracts.Proto, so it is shared between the gateway and clients on the wire.

public void TransitionTo(SessionState nextState)
{
    lock (_syncRoot)
    {
        if (_state is SessionState.Closed)
        {
            return;
        }

        if (_state is SessionState.Faulted && nextState is not SessionState.Closed)
        {
            return;
        }

        if (_state is SessionState.Closing
            && nextState is not SessionState.Closed
            && nextState is not SessionState.Faulted)
        {
            return;
        }

        _state = nextState;
    }
}

Closed is terminal, Faulted only allows a transition to Closed, and Closing only allows a transition to Closed or Faulted. This guards against late callbacks (worker exit, heartbeat timeout) re-animating a session that is already tearing down or torn down — once CloseAsync has set Closing under _syncRoot, no TransitionTo(Ready) from another thread can walk the session back to Ready. Both close-related writes (Closing and Closed) go through _syncRoot exactly like every other state write; _closeLock only serializes concurrent close attempts.

Session tags and dashboard event visibility

GatewaySession.Tags is an immutable, ordinal-ignore-case set of dashboard visibility tags, stamped once at construction from the ownerDashboardTags argument and never mutated for the session's life. The values come from the owning API key's ApiKeyConstraints.DashboardTags (set with apikey --dashboard-tags), which MxAccessGatewayService reads off the authenticated caller and passes to the tagged OpenSessionAsync overload. They are never read from the client's wire request, so a client cannot label its own session with another tenant's tag. A session whose owner key declared no tags — and every session opened through the tagless OpenSessionAsync overload, which unit-test fakes inherit by default — is untagged.

Tags gate visibility only: which sessions' event metadata a dashboard user may observe. They are not a data-access constraint, so they neither widen nor narrow what the owning key can read or write, and they play no part in the gRPC event stream, whose attach check is owner-key identity (see Reconnect and replay).

IDashboardSessionAcl.CanViewSession is the single decision both dashboard subscribe seams consult — EventsHub.SubscribeSession for remote hub clients and the session-details page's in-process subscribe. An authenticated Administrator is allowed first, before the session is even looked up; otherwise an unknown session id is denied, an untagged session follows MxGateway:Dashboard:UntaggedSessionVisibility (AdminOnly by default), and a tagged session is allowed only when its tags intersect the caller's granted tags. A Viewer's grant comes from MxGateway:Dashboard:GroupToTag applied to their LDAP groups; a principal carrying no tag claims — anonymous localhost included — is an empty-grant Viewer and sees no tagged session. Because Tags is immutable, the decision taken at subscribe time cannot go stale while the subscription lives, so there is no per-event re-check. See docs/GatewayDashboardDesign.md.

SessionManager (ISessionManager)

SessionManager is the orchestrator. It exposes OpenSessionAsync, TryGetSession, InvokeAsync, CloseSessionAsync, KillWorkerAsync, CloseExpiredLeasesAsync, and ShutdownAsync. It composes ISessionRegistry, ISessionWorkerClientFactory, GatewayMetrics, and GatewayOptions.

CloseSessionAsync and KillWorkerAsync are both end-of-life paths but differ in what they offer the worker:

• CloseSessionAsync is the graceful path: it calls GatewaySession.CloseAsync, which asks the worker to shut down via IWorkerClient.ShutdownAsync and only kills the process as a fallback if shutdown fails.
• KillWorkerAsync is the forceful path used by the dashboard's admin Kill button: it calls GatewaySession.KillWorker directly, which kills the worker process immediately with no graceful-shutdown attempt and transitions the session to Closed.

Both paths converge on the same registry/metrics cleanup, so the open-session slot is released and mxgateway.sessions.closed is incremented either way.

Concurrency is bounded by a SemaphoreSlim initialized to GatewayOptions.Sessions.MaxSessions. Open requests that exceed the bound throw SessionManagerException with SessionLimitExceeded rather than queuing; the caller is expected to retry.

private void EnsureSessionCapacity()
{
    if (!_sessionSlots.Wait(0))
    {
        throw new SessionManagerException(
            SessionManagerErrorCode.SessionLimitExceeded,
            $"Gateway session limit {_options.Sessions.MaxSessions} has been reached.");
    }
}

SessionManager also defines six close-reason constants — DefaultCloseReason ("client-close"), GatewayShutdownReason ("gateway-shutdown"), LeaseExpiredReason ("lease-expired"), FaultedReason ("faulted-reaped"), DetachGraceExpiredReason ("detach-grace-expired"), and UnattachedGraceExpiredReason ("unattached-grace-expired") — so that the metrics and worker shutdown paths agree on a fixed vocabulary.

SessionRegistry (ISessionRegistry)

SessionRegistry is a thin wrapper over a ConcurrentDictionary<string, GatewaySession> keyed by session id with StringComparer.Ordinal. Snapshot materializes the values into an array so iteration callers (lease sweeper, shutdown) do not race with concurrent TryAdd and TryRemove calls.

ActiveCount filters out sessions whose state is Closed; this is consumed by metrics and the dashboard, where Count would otherwise momentarily over-report during teardown.

SessionWorkerClientFactory (ISessionWorkerClientFactory)

SessionWorkerClientFactory.CreateAsync is the only path that builds an IWorkerClient. It drives the session through the protobuf SessionState substates in order: StartingWorker, WaitingForPipe, Handshaking, InitializingWorker. The substates are wire-visible so the dashboard and clients can render startup progress.

A linked CancellationTokenSource enforces session.StartupTimeout. If startup fails or times out, the factory either kills the partially-constructed WorkerClient or, if the client was never built, kills the launched process and disposes the named pipe before rethrowing. A pure timeout is rewritten as TimeoutException so callers can distinguish it from caller-driven cancellation:

if (exception is OperationCanceledException
    && startupCancellation.IsCancellationRequested
    && !cancellationToken.IsCancellationRequested)
{
    throw new TimeoutException(
        $"Worker session {session.SessionId} did not complete startup within {session.StartupTimeout}.",
        exception);
}

The named pipe is created with maxNumberOfServerInstances: 1 so a second worker cannot connect to the same pipe name even if the first launch is still pending. Combined with the per-session nonce passed to the worker, this is the gateway's defense against a foreign process answering a pipe.

SessionShutdownHostedService

SessionShutdownHostedService is an IHostedService whose only job is to call ISessionManager.ShutdownAsync from StopAsync. It catches OperationCanceledException triggered by the host shutdown timeout and logs a warning so that an over-running shutdown does not surface as an unhandled exception.

SessionOpenRequest

SessionOpenRequest is the gateway-internal record passed to OpenSessionAsync. It is constructed from the wire-level OpenSessionRequest via SessionOpenRequest.FromContract. Keeping a separate internal record means the gRPC layer can normalize input (defaulting backend, sanitizing strings) without leaking generated proto types into SessionManager.

public sealed record SessionOpenRequest(
    string? RequestedBackend,
    string? ClientSessionName,
    string? ClientCorrelationId,
    Duration? CommandTimeout)
{
    public static SessionOpenRequest FromContract(OpenSessionRequest request)
    {
        ArgumentNullException.ThrowIfNull(request);

        return new SessionOpenRequest(
            request.RequestedBackend,
            request.ClientSessionName,
            request.ClientCorrelationId,
            request.CommandTimeout);
    }
}

SessionCloseResult

SessionCloseResult is the record returned from a successful close. AlreadyClosed distinguishes "this call closed the session" from "the session was already closed when we acquired the close lock", which the metrics layer uses to avoid double-counting. WorkerShutdownFailure is set when the worker refused a graceful shutdown and was killed instead — still a successful close (see Close); the manager logs it and the gRPC reply does not expose it.

public sealed record SessionCloseResult(
    string SessionId,
    SessionState FinalState,
    bool AlreadyClosed,
    Exception? WorkerShutdownFailure = null);

SessionCloseStartedException

SessionCloseStartedException is internal and is only thrown from inside GatewaySession.CloseAsync when the close path has already begun mutating worker state and a subsequent step fails — in practice, when the worker's graceful shutdown failed and the kill fall-back failed too, so a worker may still be running. SessionManager.CloseSessionCoreAsync catches it, marks the session faulted, increments the close-failed metric, removes the session from the registry, and rethrows it wrapped as SessionManagerException with CloseFailed. The intermediate type exists so the public API surface only ever exposes SessionManagerException.

SessionManagerException and SessionManagerErrorCode

SessionManagerException is the single public error type emitted from this subsystem; the code is carried in the ErrorCode property and is also surfaced to metrics tags via SessionManagerErrorCode.ToString().

Code	Meaning
SessionNotFound	The session id is not in the registry.
SessionNotReady	The session or its IWorkerClient is not in Ready state.
EventSubscriberAlreadyActive	A second event subscriber attached in single-subscriber mode (AllowMultipleEventSubscribers is false).
EventSubscriberLimitReached	In multi-subscriber mode, an attach exceeded MaxEventSubscribersPerSession concurrent external subscribers.
EventQueueOverflow	Reserved for the worker event channel overflow path.
SessionLimitExceeded	MaxSessions is in use.
OpenFailed	OpenSessionAsync failed; the inner exception carries the cause.
CloseFailed	A close started but did not complete cleanly — the worker's graceful shutdown and the kill fall-back both failed (or a kill threw); the session is removed and faulted. A shutdown failure whose kill succeeded is a successful close, not this.
PermissionDenied	The caller may not perform the operation on this session (for example, attaching StreamEvents to a session opened by a different API key).
GatewayStopping	The gateway is stopping and refuses new sessions (see Shutdown Coordination). Maps to gRPC Unavailable; not recorded as a fault.

Lifecycle

Open

SessionManager.OpenSessionAsync allocates a session slot, builds the GatewaySession, registers it, and asks the factory to bring up the worker. Failures roll back every preceding step:

catch (Exception exception)
{
    await RollBackOpenAsync(session, registered, exception).ConfigureAwait(false);
    _metrics.Fault(SessionManagerErrorCode.OpenFailed.ToString());
    _logger.LogWarning(
        exception,
        "Failed to open gateway session {SessionId}.",
        session?.SessionId ?? "<not-created>");

    throw new SessionManagerException(
        SessionManagerErrorCode.OpenFailed,
        session is null ? "Failed to create session." : $"Failed to open session {session.SessionId}.",
        exception);
}

RollBackOpenAsync faults the session and, when it had been registered, routes it through RemoveSessionAsync — the single removal path every teardown uses, which deregisters, drops the session's per-session event metrics, releases the slot and disposes, in that order. The order matters because releasing the semaphore before disposal would let the next open race the worker process tear-down on the same machine. A session that was never registered (id collision) releases the slot and disposes directly. The open-session gauge is not touched by the rollback: SessionOpened is recorded inside the attach gate described next, after which nothing in the open sequence can fail.

Close or kill during worker startup

The session is registered — and therefore visible to the dashboard's Kill button, the lease sweep and the host drain — for the whole worker startup, which can last up to MxGateway:Worker:StartupTimeoutSeconds. A kill or drain that lands in that window finds no worker client to stop, flips the session to Closed, removes it and disposes it. The worker client that SessionWorkerClientFactory.CreateAsync then returns must not be stored on that dead session and handed back to the caller as a successful open: nothing would ever shut the worker down (the close already ran) and the client would hold a Closed session.

GatewaySession.AttachWorkerClient therefore returns false, under _syncRoot, when the session is already Closing, Closed or Faulted or has been disposed. The manager runs the end of the open sequence — attach, MarkReady, SessionOpened — through TryAttachWorkerClientAndMarkReadyAsync, which holds the per-session close gate (_closeLock) so a concurrent CloseAsync / KillWorkerWithCloseGateAsync runs either entirely before it (the attach is refused) or entirely after it (an ordinary close of a Ready session). On a refused attach the manager kills and disposes the late worker client and the open fails with OpenFailed; the registry is already empty and the slot was released by whichever teardown closed the session.

Once CreateAsync has returned, a live worker exists that only the open path knows about, so the gate wait is not cancellable (it waits on CancellationToken.None; the gate is only ever held for the duration of one close or kill) and the manager discards the client on any exception out of the attach step — the caller's own cancellation (checked before the attach, so an abandoned open never yields a session), a disposed gate, anything thrown while becoming Ready. The failure then surfaces as OpenFailed with the cause as inner exception. An exception that escaped before the discard would be an orphaned x86 worker, which a restarted gateway terminates rather than adopts.

Because such a session was never counted as opened, it must not be counted as closed either. GatewaySession.OpenRecorded is set inside the same gate right after SessionOpened is recorded, and every teardown path (KillWorkerAsync, CloseSessionCoreAsync, the close-failed branch) records SessionClosed / SessionRemoved only when it is set — so sessions.opened, sessions.closed and the sessions.open gauge stay consistent across the startup window.

Run

While Ready, callers reach the worker through SessionManager.InvokeAsync, which delegates to GatewaySession, which checks the state under lock and updates LastClientActivityAt on every invocation. Events do not travel this path: every consumer attaches to the session's SessionEventDistributor instead (see below), so the manager exposes no event-read member. GatewaySession also exposes typed bulk helpers (AddItemBulkAsync, SubscribeBulkAsync, etc.) that wrap WorkerCommand round-trips and translate non-Ok ProtocolStatus replies into SessionManagerException with SessionNotReady.

Event streaming uses AttachEventSubscriber which returns a disposable lease. When allowMultipleSubscribers is false (single-subscriber mode) a second attach throws EventSubscriberAlreadyActive; this prevents two gRPC streams from racing on the same worker event channel. When it is true, up to MaxEventSubscribersPerSession concurrent external subscribers are allowed and the next attach throws EventSubscriberLimitReached. The count-check-and-increment is atomic under the session lock, so concurrent attaches can never exceed the cap. The gateway-owned internal dashboard mirror subscriber is registered directly on the distributor and does not count toward the cap. Active event subscribers keep the session lease from expiring until the stream is disposed.

FailFast event backpressure faults the whole session only in single-subscriber mode; in multi-subscriber mode it degrades to a per-subscriber disconnect so one slow consumer never faults a session shared by others. The session passes its mode to the SessionEventDistributor at construction, so this decision is made on the fixed mode rather than a live subscriber-count snapshot.

The single worker event channel has exactly one direct reader: the SessionEventDistributor pump (MapWorkerEventsAsync). Both gateway-owned internal consumers — the dashboard mirror and the central alarm monitor — attach as distributor subscribers rather than draining the worker channel themselves. GatewaySession.AttachInternalEventSubscriber mirrors the dashboard-mirror lease (isInternal: true): the alarm monitor calls it directly on its session so it consumes the same mapped MxEvents the pump fans to every subscriber, without counting against MaxEventSubscribersPerSession and without a slow reconcile faulting the session. This is what keeps the alarm feed and the dashboard from splitting the stream between two raw drains (which would silently lose Acknowledge and provider-mode transitions); the worker channel is single-reader and a second WorkerClient.ReadEventsAsync consumer throws so a regression fails loudly.

The monitor takes that lease before it issues SubscribeAlarms, so no transition window is missed: the pump has been running since MarkReady started the dashboard mirror, and the distributor only fans to subscribers registered at fan-out time, so a lease taken after the subscribe + first-reconcile round trips would lose every transition raised inside that window. Transitions arriving while the monitor subscribes and reconciles buffer in the lease's bounded channel and are applied after the reconcile, which is order-safe because a live transition can update an alarm the reconciled snapshot already holds.

The repair transitions the monitor's reconcile broadcasts on the alarm feed (Raise/Clear presence deltas and the acked-state delta) are at-least-once: a reconcile reads the worker's current state while the matching live transition may still be buffered in the lease, so both can be broadcast and the duplicates are indistinguishable — alarm-feed consumers must apply transitions idempotently. This is a property of the reconcile design, not of the buffering above. The monitor dedups the common case best-effort (NEXT-03): a buffered live transition that positively matches the cache's worker timestamp and resulting state — or, for Clear, a tombstone keyed on the cleared instance's original raise timestamp — was already broadcast as a repair and is suppressed; unset timestamps never suppress, so the consumer contract is unchanged.

AttachInternalEventSubscriber enforces the same readiness gate as AttachEventSubscriber — a session (or worker) that is not Ready throws SessionNotReady before the distributor is constructed. A premature attach would otherwise start the pump against a source that throws, completing every subscriber with that error and latching the distributor for the session's whole lifetime.

The same one-distributor-per-session rule is enforced at the other end of the lifecycle. Session disposal disposes the distributor and latches the fact, so an attach that passed its readiness gate and then lost the CPU until the session was torn down throws SessionNotReady rather than constructing a replacement distributor. A replacement can never be pumped — the pump-start flag is a one-way latch — so its subscriber would block on an empty channel that is never completed until the client's own deadline expired, with no error, log or metric to explain it. Every attach entry point (AttachEventSubscriber, AttachEventSubscriberWithReplay, AttachInternalEventSubscriber) fails this way, and each rolls back the subscriber-count slot it took, so a caller that loses the race to teardown sees the same failure it would have seen a moment later from the readiness gate.

Sessions open with MxGateway:Sessions:DefaultLeaseSeconds (default 1800) added to the open timestamp. Unary client activity refreshes the lease by the same duration (TouchClientActivity is the only path that moves the lease). IsLeaseExpired cooperates with SessionManager.CloseExpiredLeasesAsync, which iterates a registry snapshot and closes any session whose lease has expired with LeaseExpiredReason. SessionLeaseMonitorHostedService runs that sweep every MxGateway:Sessions:LeaseSweepIntervalSeconds seconds (default 30).

Teardown parallelism

A sweep pass is two phases. Selection stays a single sequential pass over the snapshot, because that is what gives the precedence rule (lease-expiry, then faulted, then detach-grace, then unattached-grace) and the TryBeginCloseIfExpired TOCTOU re-check their meaning. Closing then runs over the already-selected set with Parallel.ForEachAsync at MaxParallelSessionCloses, a compile-time constant of 4 in SessionManager. Each close is bounded by MxGateway:Worker:ShutdownTimeoutSeconds (default 10), so a one-at-a-time sweep lets a few hung workers serialize reaping and starve session slots for the rest. Parallel closing is safe because TryBeginCloseIfExpired already flipped each selected session to Closing under its own lock — that idempotent begin-close is the per-session exclusivity invariant, so no two teardowns can ever run against one session. The degree is a fixed constant rather than an option, and bounded rather than unlimited, because every concurrent close is one x86 worker process being shut down or killed; the fan-out exists to hide a few hung workers, not to tear the whole registry down at once.

TryBeginCloseIfExpired is a reservation, not the close: it writes Closing but leaves the session's close-started flag alone, so the CloseAsync the sweep issues next is the first close to run and reports AlreadyClosed: false. That is what lets CloseSessionCoreAsync account SessionClosed for lease-expired, faulted-reaped and detach-grace-expired closes exactly once; a reservation that also set the flag made every sweep-driven close look like a second concurrent close, and swept sessions never left the sessions.open gauge. A client close that races the reservation still counts once: whichever CloseAsync takes the close lock first does the accounting, and the other finds the session Closed.

Splitting the phases moves the TOCTOU re-check earlier, and that is an accepted trade rather than an unchanged behavior: selection now flips every chosen session to Closing up front, before any teardown runs, whereas the sequential sweep re-checked session N only after sessions 1..N-1 had finished closing. A client that re-attaches a subscriber while the close phase is running therefore loses a race it could previously win — the eligibility snapshot is taken at one instant for the whole pass. Expiry evaluation itself is unaffected, because now is a parameter and is not re-read per session.

A close that throws no longer abandons the rest of the selected set: the sweep attempts every selected session, captures the first failure, and rethrows it once the pass is done so SessionLeaseMonitorHostedService still logs the sweep failure as before. Because the closes run concurrently, which failure surfaces when several fail in one pass is nondeterministic; the log line is the diagnostic, not the identity of the exception.

ShutdownAsync drains sessions with the same bounded fan-out and the same per-session catch → KillWorkerAsync fallback, with two rules that keep a stop deadline from turning into leaked workers. First, its body is exception-total — nothing escapes it — because Parallel.ForEachAsync cancels the token handed to the sibling bodies as soon as one body throws, which would abort in-flight graceful shutdowns and make their kill fallback fail instantly on the freshly cancelled token. Second, the drain loop is deliberately not bound to the caller's CancellationToken and the kill fallback runs on CancellationToken.None: a cancelled ParallelOptions token stops dispatching the remaining sessions entirely, so the untried tail would be neither closed nor killed. This fixes a leak the sequential drain also had, rather than restoring the sequential drain's behavior — there the kill fallback ran on the caller's already-cancelled token, and KillWorkerAsync's entry ThrowIfCancellationRequested threw out of the loop on the very first session, producing zero kills. The token is passed to the graceful close only, so a host stop deadline turns the drain into a kill sweep rather than into a leak. This matters because nothing reattaches to a leaked worker — a restarted gateway terminates orphans (see Design Decisions).

The asymmetry with CloseExpiredLeasesAsync — whose ParallelOptions is token-bound — is intentional: the sweep is periodic maintenance, so a pass abandoned on cancellation loses nothing permanently (the next pass re-selects, and ShutdownAsync backstops it), whereas the shutdown drain is terminal and must not be abandoned partway.

Stranded-Closing bound. A sweep pass that is cancelled after selection leaves its unclosed selections in Closing with close already started. IsFaultedReapableCore requires state == Faulted, so a session selected under FaultedReason and stranded this way is not re-selected as faulted; it is swept only when its normal lease expires (up to MxGateway:Sessions:DefaultLeaseSeconds, default 1800 s), since IsLeaseExpiredCore and IsDetachGraceExpiredCore are state-agnostic. This bound is documented rather than closed with a re-selection clause: the sweep's only caller cancels on the host's stoppingToken, so the very next thing that runs is ShutdownAsync, which drains (or kills) the whole registry — and any worker that still survives that is terminated as an orphan on the next gateway start. Adding a "Closing and close-started" re-selection clause would also have to distinguish an abandoned close from one that is merely still in flight, which would weaken the single invariant that makes the parallel close phase safe.

Detach-grace retention

MxGateway:Sessions:DetachGraceSeconds (default 30) is a bounded retention window kept after a session's last external (gRPC) event-stream subscriber drops, so a client can reconnect to the same session instead of having it torn down on the first stream disconnect. While the window is open the session stays Ready and fully usable — worker commands continue to work and a reconnecting subscriber re-attaches normally. Because retention is keyed on the external subscriber count (_activeEventSubscriberCount), and the gateway-owned internal dashboard mirror registers directly on the distributor with isInternal: true and is therefore not counted, a session whose only remaining subscriber is the dashboard mirror still enters detach-grace.

Mechanically: when the last external subscriber detaches and DetachGraceSeconds > 0, DetachEventSubscriber stamps DetachedAtUtc from the session's TimeProvider under _syncRoot (the detach→grace-start transition). AttachEventSubscriber clears DetachedAtUtc under the same lock when a subscriber re-attaches (the reattach→grace-cancel transition), so the two races and the sweeper's read all serialize on _syncRoot. SessionManager.CloseExpiredLeasesAsync checks IsDetachGraceExpired(now) alongside IsLeaseExpired(now): a session detached for at least DetachGraceSeconds with no active external subscriber is closed by the same lease sweep, with the distinct DetachGraceExpiredReason ("detach-grace-expired") so operators can tell a short reconnect-window expiry from a long idle-lease expiry. Setting DetachGraceSeconds to 0 disables retention and reverts to the original behavior: a detached session is retained only until its normal lease expires.

DetachGraceSeconds controls retention and expiry only; the reconnect/replay path that re-attaches a dropped client to a retained session is described in Reconnect and replay.

Worker exit outside a gateway close

IWorkerClient has no state-change callback, so a worker that reached Closed or Faulted without the gateway closing it — the client sent the ShutdownWorker control command and the worker exited cleanly, the worker crashed, its pipe broke, a heartbeat timed out — used to leave the session reporting Ready until its lease expired (a live finding: WorkerClient was Closed on the shutdown ack, the session still Ready, even with an event stream attached). The session now observes it from its own worker-event pump source (MapWorkerEventsAsync): the single distributor pump is the session's one always-on view of the worker (it runs from MarkReady via the dashboard mirror), and WorkerClient enters its terminal state before it completes the event channel, so when the enumeration ends the terminal state is visible. ObserveWorkerTerminalState then faults a Ready session whose worker is Closed/Faulted.

The choice is Faulted, not Closed: Closed means the gateway closed the session, and a Closed session left in the registry would linger until its lease expired with nothing reaping it. A faulted session is permanently unusable (every command fails the readiness check, as the worker is gone), stays observable via GetSessionStatus with FinalFault naming the worker state, and is reaped by the faulted-session sweep below after FaultedGraceSeconds — which is where SessionClosed is accounted, exactly once, through the ordinary CloseSessionCoreAsync path (WorkerClient.ShutdownAsync on a Closed client returns immediately, so no kill fallback or warning). Only a Ready session follows its worker: during a gateway-initiated close the worker closing is the expected outcome, not a fault.

Faulted-session reaping

A session that faults (for example, a slow single-subscriber client that overflows its event queue under the FailFast backpressure policy) is left permanently unusable — every command fails EvaluateReadyUnderLock — but its worker keeps running and it still pins one of MxGateway:Sessions:MaxSessions slots. Rather than waiting up to DefaultLeaseSeconds for the lease to expire, SessionManager.CloseExpiredLeasesAsync also checks IsFaultedReapable(now) and reaps a faulted session through the same TryBeginCloseIfExpired → CloseSessionCoreAsync teardown, with the distinct FaultedReason ("faulted-reaped"). MarkFaulted stamps a fault timestamp from the session's TimeProvider; MxGateway:Sessions:FaultedGraceSeconds (default 0) bounds how long the faulted session is retained before the next sweep reaps it — 0 reaps it on the next sweep, a positive value keeps it observable via GetSessionStatus for that window first. Sweep precedence when several conditions fire at once is lease-expiry, then faulted, then detach-grace, then unattached-grace.

Never-subscribed session reaping

Detach-grace only helps a session that had a subscriber. A client that opens a session and then disappears before it ever calls StreamEvents — OtOpcUa's GalaxyDriver failing part-way through its start-up is the motivating case — leaves a Ready session with no subscriber, no detach timestamp and a lease 30 minutes out, pinning one of MaxSessions and a live x86 worker for all of DefaultLeaseSeconds. MxGateway:Sessions:UnattachedSessionGraceSeconds (default 0, off) closes that gap: when positive, CloseExpiredLeasesAsync also checks IsUnattachedGraceExpired(now) and reaps, with the distinct UnattachedGraceExpiredReason ("unattached-grace-expired"), any Ready session that has never had a subscriber of either kind and has been idle for the window.

Four deliberate limits keep the rule from catching legitimate sessions. It applies only to Ready sessions, and its idle clock starts at Ready (MarkReady stamps _readyAtUtc), so a slow worker start neither reaps the session mid-startup nor eats the window before the client even has the session id. Idle is measured from the later of that Ready stamp and the session's last client activity (its most recent unary command), so a unary-only client (reads and writes, never streams) is not reaped while it keeps calling; the rule is an idle timeout for sessions that never streamed, not a deadline to stream. It stops applying the moment the first subscriber attaches, after which detach-grace and the normal lease govern the session. And an internal in-process subscriber counts as attached for this rule (AttachInternalEventSubscriber sets a separate _everHadInternalEventSubscriber latch): the gateway's own GatewayAlarmMonitor session only ever attaches that way, and it must not be reaped by its own gateway every cycle. The latch is separate from _everHadEventSubscriber because that one drives detach-grace, which is keyed on external subscribers only. The dashboard mirror, which every session registers at MarkReady, is deliberately not a subscriber for this rule. Like the other sweep conditions it is re-verified inside TryBeginCloseIfExpired, so a subscriber that attaches between selection and the close wins the race. Operators should set the window comfortably above a well-behaved client's open-to-first-StreamEvents time; the close lands on the next sweep after the window elapses, up to LeaseSweepIntervalSeconds later.

Reconnect and replay

A client that drops mid-stream reconnects by re-issuing StreamEvents with StreamEventsRequest.after_worker_sequence set to the last worker_sequence it observed. A non-zero after_worker_sequence means resume; 0 means fresh stream and behaves exactly as a first-time subscribe — no replay, no sentinel.

Owner-scoped attach (security control). OpenSession records the caller's API key id on the session as GatewaySession.OwnerKeyId. Every StreamEvents attach and reattach is owner-checked: EventStreamService.StreamEventsAsync compares the caller's key id (threaded from GatewayGrpcAuthorizationInterceptor via MxAccessGatewayService.StreamEvents) to the session owner and throws SessionManagerException(PermissionDenied) — surfaced as gRPC PermissionDenied — when they differ, before any subscriber is attached and before any replayed or live event is delivered. Possessing the events:read scope and knowing a session id is not enough. This is what makes the default-on detach-grace and replay-retention windows safe: without the check, any events:read-scoped key that learned a session id could attach to another key's retained session and receive its replayed and live data. A null owner (session opened with authentication disabled) matches only a null caller key.

On a resume, EventStreamService.StreamEventsAsync attaches through GatewaySession.AttachEventSubscriberWithReplay, which calls SessionEventDistributor.RegisterWithReplay. That method snapshots the session's replay ring for events newer than after_worker_sequence and registers the live subscriber inside a single _replayLock critical section. This atomicity is what makes the replay→live handoff free of gaps and duplicates: the pump appends each event to the replay ring (under _replayLock) before fanning it to subscriber channels, so relative to that one critical section every event is either in the replay snapshot or fanned into the freshly-registered live channel — never both observably, never neither.

The handoff is sealed by a watermark. RegisterWithReplay returns LiveResumeSequence (the highest replayed sequence, or after_worker_sequence when nothing was replayed); EventStreamService then filters the live channel to events strictly greater than that watermark. An event that was both included in the replay snapshot and — racing the registration — also written to the live channel has worker_sequence <= LiveResumeSequence, so the live filter drops it exactly once (no duplicate), while every newer event is delivered (no gap). The same per-item filter governs replayed and live events identically, so a constrained or resuming caller never sees a replayed event it could not have seen live.

Emit order on a resumed stream:

1. ReplayGap sentinel (only when events were evicted). If the requested after_worker_sequence predates the oldest event still retained — i.e. events in the open interval were dropped by capacity or age eviction and are unrecoverable — the gateway first yields a single sentinel MxEvent with replay_gap populated (requested_after_sequence = the requested watermark, oldest_available_sequence = the resume anchor described below). The sentinel carries the session id; its family is UNSPECIFIED, its body oneof is unset, and no per-item fields are populated. It is an explicit, documented control signal — not a synthesized MXAccess event — telling the client to discard local state and re-snapshot. A client that wants to resume without another gap should set after_worker_sequence = oldest_available_sequence - 1 on its next request.
2. Retained replay batch. The still-retained events newer than the requested watermark, in ascending worker_sequence order.
3. Live events, resuming strictly after LiveResumeSequence.

When after_worker_sequence is inside the retained window (nothing was evicted), step 1 is skipped: the stream replays the retained tail then resumes live with no sentinel.

oldest_available_sequence when the ring is empty. Age eviction (ReplayRetentionSeconds, default 300) and a disabled ring (ReplayBufferCapacity = 0) both leave nothing retained, so there is no oldest-retained sequence to report. In that case the sentinel carries the next sequence that can possibly be delivered — the highest sequence the distributor has observed plus one — rather than 0. That keeps after_worker_sequence = oldest_available_sequence - 1 the single universal resume formula: the follow-up resume lands exactly on the highest observed sequence, replays nothing, reports no gap, and receives every subsequent live event. Reporting 0 here would make an unsigned client compute 2^64 - 1 and then silently receive nothing, because the live filter drops every event at or below that watermark. Nothing is lost relative to reporting 0 — the evicted interval is unrecoverable either way, and the sentinel's job is to tell the client to re-snapshot. 0 remains the value when there is no gap, where the field is meaningless and never emitted.

The ReplayGap sentinel is emitted only on the StreamEvents server stream and only to the resuming subscriber — it is never fanned to other subscribers and never appears in DrainEventsReply (the diagnostic drain path is untouched). Replay retention itself is bounded by MxGateway:Events:ReplayBufferCapacity (count) and ReplayRetentionSeconds (age); see Configuration.

Close

GatewaySession.CloseAsync is serialized by a per-session SemaphoreSlim (_closeLock) so only one close runs at a time, but every read/write of _state still passes through _syncRoot (via TryBeginClose and MarkClosed). The close path therefore obeys the same lock discipline as TransitionTo / MarkFaulted: it transitions to Closing, asks the worker client to shut down within ShutdownTimeout, and on success transitions to Closed. DisposeAsync waits on _closeLock once before disposing the semaphore so an in-flight close's Release() cannot race against the dispose. WorkerClient.ShutdownAsync is bounded by ShutdownTimeout end to end — queuing the WorkerShutdown envelope included, since a worker that stopped reading the pipe can leave the bounded outbound channel full with no watchdog left to clear it once the client is Closing — so a close never holds _closeLock longer than the timeout before reaching the fallback. If WorkerClient.ShutdownAsync throws, the session falls back to IWorkerClient.Kill (forced close):

Exception? workerShutdownFailure = null;
if (_workerClient is not null)
{
    try
    {
        await _workerClient.ShutdownAsync(ShutdownTimeout, cancellationToken).ConfigureAwait(false);
    }
    catch (Exception exception)
    {
        try
        {
            _workerClient.Kill(reason);
        }
        catch (Exception killException)
        {
            throw new SessionCloseStartedException(
                $"Session {SessionId} close failed after worker shutdown started.",
                new AggregateException(exception, killException));
        }

        workerShutdownFailure = exception;
    }
}

MarkClosed();
return new SessionCloseResult(SessionId, SessionState.Closed, alreadyClosing, workerShutdownFailure);

A kill that succeeds completes the close successfully: the worker is gone and the session is Closed, exactly as if the shutdown had worked, and the caller's CloseSession reply says so. The graceful failure is not lost — it travels back as SessionCloseResult.WorkerShutdownFailure, and SessionManager.CloseSessionCoreAsync logs it at warning level ("did not shut down gracefully and was killed"), which is the operator signal that workers are hanging on shutdown. The kill itself is counted by WorkerClient.Kill (mxgateway.workers.killed). Nothing is faulted and no CloseFailed fault is recorded, because nothing failed from the session's point of view; reporting the common hung-worker case that way had made every such close look like a teardown failure.

Only if both the graceful shutdown and the kill fall-back fail are the original and kill exceptions bundled into an AggregateException and surfaced as SessionCloseStartedException. SessionManager.CloseSessionCoreAsync then translates that into a SessionManagerException with CloseFailed, faults the session, records the fault metric and removes the session — a worker may still be running, which is what CloseFailed is reserved for.

Losing a race to a concurrent close. A caller that looked a session up (a kill, a client close, a sweep's snapshot) and then found its close gate already disposed — because a concurrent close finished, removed the session, accounted SessionClosed and disposed it in between — sees ObjectDisposedException from the gate. Both KillWorkerAsync and CloseSessionCoreAsync catch that case when the session is no longer registered and return AlreadyClosed: true with no accounting, no fault and no log: the close that won already did all of it. (The old kill catch recorded a fault, faulted the session and decremented the open gauge a second time; the sweep rethrew it as a sweep failure.) A sweep does not count such a session among the sessions it closed. A disposed gate on a session that is still registered is a real failure and takes the normal CloseFailed path.

SessionManager.KillWorkerAsync is the explicit kill path that the dashboard's admin Kill button invokes and that ShutdownAsync falls back to when a graceful close throws. It goes through GatewaySession.KillWorkerWithCloseGateAsync, which skips WorkerClient.ShutdownAsync entirely, so KillCount increments while ShutdownCount does not; the session is then removed from the registry and the open-session slot is released, identical to the cleanup that follows a successful CloseSessionAsync.

Shutdown Coordination

SessionShutdownHostedService.StopAsync calls SessionManager.ShutdownAsync, which closes every registered session with GatewayShutdownReason. Sessions are drained with the same bounded fan-out the lease sweep uses (MaxParallelSessionCloses), because a one-at-a-time drain of a full registry at a worst-case worker shutdown timeout each outruns any host stop-timeout and leaves the tail to the orphan killer. Each iteration catches its own exceptions — every exception, including from the fallback — calls KillWorkerAsync on an uncancellable token, and removes the session, so that neither one stuck worker nor one failing teardown can block or abort the rest of the host's drain:

MarkStopping();
for (int pass = 0; pass < MaxShutdownDrainPasses; pass++)
{
    if (_registry.Count == 0)
    {
        return;
    }

    await Parallel.ForEachAsync(
        _registry.Snapshot(),
        new ParallelOptions { MaxDegreeOfParallelism = MaxParallelSessionCloses },
        async (session, _) =>
        {
            try
            {
                await CloseSessionCoreAsync(session, GatewayShutdownReason, cancellationToken).ConfigureAwait(false);
            }
            catch (Exception exception)
            {
                _logger.LogWarning(
                    exception,
                    "Graceful shutdown failed for session {SessionId}; killing worker.",
                    session.SessionId);
                if (_registry.TryGet(session.SessionId, out GatewaySession? registeredSession)
                    && registeredSession is not null)
                {
                    try
                    {
                        // Not the caller's token: the kill is the last-resort orphan preventer.
                        await KillWorkerAsync(session.SessionId, GatewayShutdownReason, CancellationToken.None).ConfigureAwait(false);
                    }
                    catch (Exception killException)
                    {
                        _logger.LogWarning(killException, "Worker kill fallback failed for session {SessionId}.", session.SessionId);
                    }
                }
            }
        }).ConfigureAwait(false);
}

Iterating over Snapshot rather than the live dictionary lets RemoveSessionAsync mutate the registry from inside the loop without throwing, and gives the parallel drain a stable, already-materialized source.

Stopping gate

A drain that only ever looked at one snapshot could be outrun by OpenSession: the host raises ApplicationStopping before hosted services stop, and Kestrel keeps accepting gRPC calls until its own StopAsync runs, so a session opened behind the snapshot would have nobody left to close it and its x86 worker would survive the gateway as an orphan. SessionManager therefore keeps a stopping flag that is set by the host's IHostApplicationLifetime.ApplicationStopping (the manager registers for it in its constructor when the lifetime is injected) and, as a fallback for hosts without one, at the start of ShutdownAsync. Once set it is never cleared. OpenSessionAsync checks it twice: on entry, before a slot is taken, and again right after the session is registered. The second check is what closes the race — the flag is written before the drain takes its registry snapshot, so an open that does not observe it there registered its session before that snapshot and is drained, while one that does observe it rolls its own session back. Either way the caller receives SessionManagerException with GatewayStopping, which the gRPC layer maps to Unavailable so the client retries against the restarted gateway; it is not an open failure and records no fault metric.

The drain loop above is the backstop to that gate: it re-snapshots until the registry is empty, bounded at MaxShutdownDrainPasses (3) so a session that somehow cannot be removed cannot hold the host stop forever. Anything still registered after the last pass is logged and left to the orphan terminator on the next start.

Dependency Injection

SessionServiceCollectionExtensions.AddGatewaySessions registers the four singletons and the hosted service:

public static IServiceCollection AddGatewaySessions(this IServiceCollection services)
{
    services.AddSingleton<ISessionRegistry, SessionRegistry>();
    services.AddSingleton<ISessionWorkerClientFactory, SessionWorkerClientFactory>();
    services.AddSingleton<ISessionManager, SessionManager>();
    services.AddHostedService<SessionShutdownHostedService>();

    return services;
}

The registry must be a singleton because its ConcurrentDictionary is the source of truth for session state across the gRPC service, the lease sweeper, the dashboard, and the shutdown hosted service. Registering SessionShutdownHostedService last ensures it is constructed after ISessionManager and therefore drains sessions during host stop.

Related Documentation

• Gateway Process Design
• Gateway Configuration
• Worker Process Launcher