From 3ffba83723922ab9a451f9eaf1b879bac6929fce Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Thu, 22 Oct 2026 08:43:41 -0400 Subject: [PATCH] docs: dashboard live-data session reaping, subscriber-refusal code, distributor-based StreamEvents model - Sessions.md / GatewayConfiguration.md: the dashboard Browse live-data session is unary-only, is reaped by UnattachedSessionGraceSeconds when idle, and reopens via EnsureReadyAsync; only the alarm monitor is exempt. - GatewayConfiguration.md: a refused second subscriber gets ResourceExhausted, not AlreadyExists. - Grpc.md: rewrite the event streaming model around the SessionEventDistributor pump and replay ring; add EventSubscriberLimitReached to the mapping snippet; describe what clients see while the gateway is stopping. --- docs/GatewayConfiguration.md | 4 +-- docs/Grpc.md | 57 ++++++++---------------------------- docs/Sessions.md | 2 +- 3 files changed, 15 insertions(+), 48 deletions(-) diff --git a/docs/GatewayConfiguration.md b/docs/GatewayConfiguration.md index 6b2a62e..62e85e8 100644 --- a/docs/GatewayConfiguration.md +++ b/docs/GatewayConfiguration.md @@ -154,8 +154,8 @@ validation fails fast when either half of the headroom rule is violated. | `MxGateway:Sessions:LeaseSweepIntervalSeconds` | `30` | Hosted monitor interval for closing expired leases. Active event-stream subscribers keep a session from expiring while the stream remains attached. | | `MxGateway:Sessions:DetachGraceSeconds` | `30` | Detach-grace retention window. When positive, a session whose last external (gRPC) event-stream subscriber drops is retained in `Ready` for this many seconds so a client can reconnect; if no external subscriber re-attaches within the window, the lease monitor closes it with `detach-grace-expired`. The internal dashboard mirror does not count as an external subscriber, so a dashboard-only session still enters detach-grace. `0` disables retention and reverts to closing only on normal lease expiry. Must be zero or greater. Reconnect/replay itself is implemented separately (Task 12); this option controls retention and expiry only. The effective close happens within the next sweep cycle after the window elapses — up to `LeaseSweepIntervalSeconds` after expiry. Operators wanting a firm minimum retention bound should set `DetachGraceSeconds` greater than `LeaseSweepIntervalSeconds`. | | `MxGateway:Sessions:FaultedGraceSeconds` | `0` | Grace window before the lease monitor reaps a faulted session (killing its worker and freeing the slot). A faulted session is permanently unusable yet otherwise pins a session slot and a live x86 worker until its normal lease expires. `0` (the default) reaps it on the next sweep cycle, bounding blast radius; a positive value keeps the faulted session observable via `GetSessionStatus` for that window before it is reclaimed, with the close reason `faulted-reaped`. Must be zero or greater. | -| `MxGateway:Sessions:UnattachedSessionGraceSeconds` | `0` | Idle window after which the lease monitor reaps a `Ready` session that has **never** attached a `StreamEvents` subscriber, with the close reason `unattached-grace-expired`. Exists because a client that opens a session and vanishes before it ever streams — OtOpcUa's `GalaxyDriver` failing part-way through start-up, for example — otherwise pins a session slot and a live x86 worker for the full `DefaultLeaseSeconds` (30 minutes). The window is measured from the later of the session becoming `Ready` and its last unary command, so a slow worker start does not eat it and a unary-only client that keeps calling is never reaped by this rule; it stops applying once the session's first subscriber attaches (`DetachGraceSeconds` and the normal lease govern it from then on), and the gateway's own alarm-monitor session, whose subscriber is internal, is exempt the same way. `0` (the default) disables it. Must be zero or greater. Set it comfortably above the time a well-behaved client needs between `OpenSession` and its first `StreamEvents`; the close lands on the next sweep after the window elapses, up to `LeaseSweepIntervalSeconds` later. | -| `MxGateway:Sessions:AllowMultipleEventSubscribers` | `false` | Controls whether multiple `StreamEvents` subscribers may attach to one session. When `false` the session refuses a second subscriber with `AlreadyExists`. Set to `true` to enable fan-out via the `SessionEventDistributor`. | +| `MxGateway:Sessions:UnattachedSessionGraceSeconds` | `0` | Idle window after which the lease monitor reaps a `Ready` session that has **never** attached a `StreamEvents` subscriber, with the close reason `unattached-grace-expired`. Exists because a client that opens a session and vanishes before it ever streams — OtOpcUa's `GalaxyDriver` failing part-way through start-up, for example — otherwise pins a session slot and a live x86 worker for the full `DefaultLeaseSeconds` (30 minutes). The window is measured from the later of the session becoming `Ready` and its last unary command, so a slow worker start does not eat it and a unary-only client that keeps calling is never reaped by this rule; it stops applying once the session's first subscriber attaches (`DetachGraceSeconds` and the normal lease govern it from then on), and the gateway's own alarm-monitor session, whose subscriber is internal, is exempt the same way. The dashboard Browse live-data session (`DashboardLiveDataService`) is not exempt: it is unary-only, so it is reaped once idle for the window, and it opens a new session through `EnsureReadyAsync` on its next use. `0` (the default) disables it. Must be zero or greater. Set it comfortably above the time a well-behaved client needs between `OpenSession` and its first `StreamEvents`; the close lands on the next sweep after the window elapses, up to `LeaseSweepIntervalSeconds` later. | +| `MxGateway:Sessions:AllowMultipleEventSubscribers` | `false` | Controls whether multiple `StreamEvents` subscribers may attach to one session. When `false` the session refuses a second subscriber with `ResourceExhausted` (`EventSubscriberAlreadyActive`). Set to `true` to enable fan-out via the `SessionEventDistributor`. | | `MxGateway:Sessions:MaxEventSubscribersPerSession` | `8` | Maximum number of concurrent `StreamEvents` subscribers per session when `AllowMultipleEventSubscribers` is `true`. Effectively 1 when `AllowMultipleEventSubscribers` is `false`. Must be greater than zero. | | `MxGateway:Sessions:WorkerReadyWaitTimeoutMs` | `0` | Bounded time, in milliseconds, the gateway will wait for a worker to reach `Ready` when the session is already `Ready` but the worker state has transiently diverged (e.g. `Handshaking` after a heartbeat blip). Applies only to transient worker states; terminal states (`Faulted`/`Closing`/`Closed`/no worker) fail fast immediately regardless of this setting. `0` (the default) disables the wait and preserves the original fail-fast behavior. Must be greater than or equal to zero. | diff --git a/docs/Grpc.md b/docs/Grpc.md index d8ecad0..d5a7849 100644 --- a/docs/Grpc.md +++ b/docs/Grpc.md @@ -311,6 +311,7 @@ StatusCode statusCode = exception.ErrorCode switch SessionManagerErrorCode.SessionNotFound => StatusCode.NotFound, SessionManagerErrorCode.SessionNotReady => StatusCode.FailedPrecondition, SessionManagerErrorCode.EventSubscriberAlreadyActive => StatusCode.ResourceExhausted, + SessionManagerErrorCode.EventSubscriberLimitReached => StatusCode.ResourceExhausted, SessionManagerErrorCode.EventQueueOverflow => StatusCode.ResourceExhausted, SessionManagerErrorCode.SessionLimitExceeded => StatusCode.ResourceExhausted, SessionManagerErrorCode.OpenFailed => StatusCode.Unavailable, @@ -323,64 +324,30 @@ StatusCode statusCode = exception.ErrorCode switch `WorkerClientException` follows the same pattern: `CommandTimeout` becomes `DeadlineExceeded`, `GatewayShutdown` and `CallerCancelled` both become `Cancelled`, `InvalidState` becomes `FailedPrecondition`, `ProtocolViolation` becomes `Internal`, `CommandTooLarge` becomes `ResourceExhausted`, and unmapped codes fall through to `Unavailable`. `CallerCancelled` is the caller's own token firing — a per-call deadline shorter than the gateway command timeout, or a disconnect — and is separate from `GatewayShutdown` only so metrics and logs stop attributing those calls to a shutdown that did not happen; the status the client observes is the same. +While the gateway is stopping, clients see three different outcomes. An in-flight `Invoke` gets `Cancelled` (`GatewayShutdown`). An active `StreamEvents` ends cleanly with `OK`, because the drain closes its session and completes the subscriber channel. A new `OpenSession` gets `Unavailable` (`GatewayStopping`), including an open whose session the drain closed while its worker was still starting. An `OpenSession` the caller cancels while its worker is starting gets `Cancelled`, not `Unavailable`, and is not counted as an open fault. + ## Event Streaming Model -`EventStreamService` implements the `StreamEvents` pipeline. It exists as a separate service (rather than living inside `MxAccessGatewayService`) so that the channel, backpressure policy, and metric bookkeeping can be unit-tested without spinning up a Kestrel host. +`EventStreamService` implements the `StreamEvents` pipeline. It exists as a separate service (rather than living inside `MxAccessGatewayService`) so that the attach, replay, filtering and metric bookkeeping can be unit-tested without spinning up a Kestrel host. It does not read the worker itself: each session owns one `SessionEventDistributor` whose single pump drains the worker event channel, maps each `WorkerEvent` to the public `MxEvent`, appends it to the session's replay ring, and fans it to every registered subscriber. A `StreamEvents` call is one of those subscribers. [Sessions](./Sessions.md) describes the distributor, the replay ring and detach-grace in full; this section covers the per-RPC boundary. -The pipeline has three stages: +The pipeline has four stages: 1. The session is resolved via `sessionManager.TryGetSession`. A miss raises `SessionManagerException(SessionNotFound)` so the handler reports `StatusCode.NotFound`. -2. `session.AttachEventSubscriber(...)` enforces the single-subscriber-per-session rule (or allows multiple subscribers if `Sessions:AllowMultipleEventSubscribers` is enabled). The returned `IDisposable` is released in the `finally` block, ensuring the subscriber slot is freed even when the client cancels mid-stream. -3. A `Channel` decouples the worker-side producer from the gRPC writer. The channel is bounded by `Events:QueueCapacity` and configured for a single reader and writer: +2. The attach is owner-scoped: the caller's API key id must match the key that opened the session, or the call fails with `PermissionDenied` before anything is attached or delivered. +3. The call registers a subscriber on the distributor. A fresh stream (`after_worker_sequence == 0`) uses `session.AttachEventSubscriber(...)`. A resume uses `session.AttachEventSubscriberWithReplay(...)`, which snapshots the replay ring for events newer than the cursor and registers the live subscriber under one lock, so the replay-to-live handoff has no gap and no duplicate. In single-subscriber mode (the default) a second attach fails with `EventSubscriberAlreadyActive`; with `Sessions:AllowMultipleEventSubscribers` enabled, an attach beyond `MaxEventSubscribersPerSession` fails with `EventSubscriberLimitReached`. Both map to `ResourceExhausted`. The lease is released in the `finally` block, so the subscriber slot is freed even when the client cancels mid-stream. +4. The call yields events from the lease's bounded channel (capacity `Events:QueueCapacity`). On a resume it first yields a `ReplayGap` sentinel when the cursor predates the oldest retained event, then the replayed batch, then live events filtered to sequences above the last replayed one. Per-session ordering is preserved end-to-end, because the pump is the single writer to each subscriber channel and writes in `WorkerSequence` order. -```csharp -Channel eventQueue = Channel.CreateBounded( - new BoundedChannelOptions(options.Value.Events.QueueCapacity) - { - SingleReader = true, - SingleWriter = true, - FullMode = BoundedChannelFullMode.Wait, - AllowSynchronousContinuations = false, - }); -``` - -The producer reads `WorkerEvent`s from the session, maps them, and writes to the channel. Per-session ordering is preserved end-to-end: events arrive from the worker in `WorkerSequence` order, the producer is the single writer, and the consumer drains FIFO. The producer also honours the request's `AfterWorkerSequence` cursor by skipping any event whose `WorkerSequence` is at or before the requested cutoff, which lets clients resume after a disconnect without server-side replay state. +The replay ring is server-side state that outlives a single RPC: with the default `DetachGraceSeconds` and `ReplayRetentionSeconds`, a client that drops can reconnect with its last `worker_sequence` and receive what it missed, as long as the events are still retained. ### Backpressure -When `TryWrite` fails the queue is full. The handling depends on `Events:BackpressurePolicy`: - -```csharp -if (!writer.TryWrite(publicEvent)) -{ - string message = $"Session {session.SessionId} event stream queue overflowed."; - metrics.QueueOverflow("grpc-event-stream"); - if (options.Value.Events.BackpressurePolicy == EventBackpressurePolicy.FailFast) - { - session.MarkFaulted(message); - metrics.Fault(SessionManagerErrorCode.EventQueueOverflow.ToString()); - } - else - { - logger.LogDebug( - "Disconnecting event stream for session {SessionId} after queue overflow.", - session.SessionId); - } - - writer.TryComplete(new SessionManagerException( - SessionManagerErrorCode.EventQueueOverflow, - message)); - return; -} -``` - -Under `FailFast` the session is faulted so subsequent commands return `FailedPrecondition`; the client must reopen. Under the default policy only the stream is dropped and the session continues to accept commands, leaving recovery to the client (typically a fresh `StreamEvents` call with an updated `AfterWorkerSequence`). Either way, the consumer side observes `StatusCode.ResourceExhausted` via the `EventQueueOverflow` mapping above. +The pump never blocks on a slow subscriber: it writes to each subscriber channel without waiting. When one subscriber's channel is full, the distributor's overflow handler records `QueueOverflow("grpc-event-stream")` and completes that subscriber's channel with `SessionManagerException(EventQueueOverflow)`, which the client observes as `ResourceExhausted`. Under `Events:BackpressurePolicy = FailFast` (the default) in single-subscriber mode the session is also faulted (and `Fault(EventQueueOverflow)` recorded), so subsequent commands return `FailedPrecondition` and the client must reopen. In multi-subscriber mode, or under `DisconnectSubscriber`, only that stream is dropped and the session keeps serving commands and other subscribers; the client recovers with a fresh `StreamEvents` that resumes from its last `worker_sequence`. The internal dashboard mirror is registered as an internal subscriber, so a slow dashboard only disconnects itself and never faults the session. ### Cancellation and cleanup -The handler creates a linked cancellation token (`streamCts`) so that completing the consumer (client disconnect, error, or graceful end-of-stream) also cancels the producer. The `finally` block cancels the source, disposes the subscriber slot, awaits the producer (swallowing the expected cancellation), and emits `StreamDisconnected(reason)` so dashboards see every disconnection. The reason is the bounded constant (`EventStreamDisconnectReasons`) for how the stream ended — `SessionClosed`, `Overflow`, `WorkerFault`, `ClientCancelled`, `Error`, or `Detached` — rather than one constant for every cause, so an operator can tell slow consumers from worker faults from routine client disconnects. +The `finally` block removes the subscriber's backlog-gauge contribution, disposes the lease, and emits `StreamDisconnected(reason)` so dashboards see every disconnection. The reason is the bounded constant (`EventStreamDisconnectReasons`) for how the stream ended — `SessionClosed`, `Overflow`, `WorkerFault`, `ClientCancelled`, `Error`, or `Detached` — rather than one constant for every cause, so an operator can tell slow consumers from worker faults from routine client disconnects. -`WorkerClientException` thrown by the producer marks the session as faulted before completing the channel — the worker is presumed gone, and any subsequent command on that session must observe the fault rather than silently retry. +When the worker event stream ends with a `WorkerClientException`, the pump completes every subscriber channel with it. Each subscriber marks the session faulted, so any subsequent command observes the fault rather than silently retrying, and records the `WorkerFault` disconnect reason. The fault itself is counted once in `mxgateway.faults`, by `WorkerClient` when it faults, however many subscribers observe it. ## Authorization Interceptor Integration diff --git a/docs/Sessions.md b/docs/Sessions.md index 200da7e..f00fc95 100644 --- a/docs/Sessions.md +++ b/docs/Sessions.md @@ -268,7 +268,7 @@ A session that faults (for example, a slow single-subscriber client that overflo Detach-grace only helps a session that *had* a subscriber. A client that opens a session and then disappears before it ever calls `StreamEvents` — OtOpcUa's `GalaxyDriver` failing part-way through its start-up is the motivating case — leaves a `Ready` session with no subscriber, no detach timestamp and a lease 30 minutes out, pinning one of `MaxSessions` and a live x86 worker for all of `DefaultLeaseSeconds`. `MxGateway:Sessions:UnattachedSessionGraceSeconds` (default `0`, off) closes that gap: when positive, `CloseExpiredLeasesAsync` also checks `IsUnattachedGraceExpired(now)` and reaps, with the distinct `UnattachedGraceExpiredReason` (`"unattached-grace-expired"`), any `Ready` session that has never had a subscriber of either kind and has been idle for the window. -Four deliberate limits keep the rule from catching legitimate sessions. It applies only to `Ready` sessions, and its idle clock starts at `Ready` (`MarkReady` stamps `_readyAtUtc`), so a slow worker start neither reaps the session mid-startup nor eats the window before the client even has the session id. Idle is measured from the later of that Ready stamp and the session's *last client activity* (its most recent unary command), so a unary-only client (reads and writes, never streams) is not reaped while it keeps calling; the rule is an idle timeout for sessions that never streamed, not a deadline to stream. It stops applying the moment the first subscriber attaches, after which detach-grace and the normal lease govern the session. And an *internal* in-process subscriber counts as attached for this rule (`AttachInternalEventSubscriber` sets a separate `_everHadInternalEventSubscriber` latch): the gateway's own `GatewayAlarmMonitor` session only ever attaches that way, and it must not be reaped by its own gateway every cycle. The latch is separate from `_everHadEventSubscriber` because that one drives detach-grace, which is keyed on external subscribers only. The dashboard mirror, which every session registers at `MarkReady`, is deliberately *not* a subscriber for this rule. Like the other sweep conditions it is re-verified inside `TryBeginCloseIfExpired`, so a subscriber that attaches between selection and the close wins the race. Operators should set the window comfortably above a well-behaved client's open-to-first-`StreamEvents` time; the close lands on the next sweep after the window elapses, up to `LeaseSweepIntervalSeconds` later. +Four deliberate limits keep the rule from catching legitimate sessions. It applies only to `Ready` sessions, and its idle clock starts at `Ready` (`MarkReady` stamps `_readyAtUtc`), so a slow worker start neither reaps the session mid-startup nor eats the window before the client even has the session id. Idle is measured from the later of that Ready stamp and the session's *last client activity* (its most recent unary command), so a unary-only client (reads and writes, never streams) is not reaped while it keeps calling; the rule is an idle timeout for sessions that never streamed, not a deadline to stream. It stops applying the moment the first subscriber attaches, after which detach-grace and the normal lease govern the session. And an *internal* in-process subscriber counts as attached for this rule (`AttachInternalEventSubscriber` sets a separate `_everHadInternalEventSubscriber` latch): the gateway's own `GatewayAlarmMonitor` session only ever attaches that way, and it must not be reaped by its own gateway every cycle. The latch is separate from `_everHadEventSubscriber` because that one drives detach-grace, which is keyed on external subscribers only. The dashboard mirror, which every session registers at `MarkReady`, is deliberately *not* a subscriber for this rule. The alarm monitor is the only gateway-owned session exempted. The dashboard Browse page's live-data session (`DashboardLiveDataService`) is unary-only: it registers and reads but never attaches a subscriber, so once it has been idle for the window the rule reaps it. That is expected. The service checks its cached session on every call (`EnsureReadyAsync`) and opens and registers a new one when the old one is gone. Like the other sweep conditions it is re-verified inside `TryBeginCloseIfExpired`, so a subscriber that attaches between selection and the close wins the race. Operators should set the window comfortably above a well-behaved client's open-to-first-`StreamEvents` time; the close lands on the next sweep after the window elapses, up to `LeaseSweepIntervalSeconds` later. #### Reconnect and replay