Merge stream C (gRPC surface: audit aggregation, streaming auth, alarm overflow/snapshot, write cancellation, correlation echo, disconnect reasons, verifier single-flight, opt-in /metrics auth)
This commit is contained in:
Joseph Doherty
@@ -113,6 +113,16 @@ library:
|
||||
library returns carries no expiry timestamp, so the cache cannot cap an entry at the
|
||||
key's expiry (capping it needs the donor library to surface expiry on the
|
||||
verification identity). The default 15 s TTL bounds both windows.
|
||||
Misses are **single-flight**: concurrent misses for the same token share one inner
|
||||
verification, so a TTL expiry under load costs one store read instead of one per
|
||||
in-flight RPC. The flight is keyed on the token hash *and* the key's generation, so
|
||||
a miss that starts after an `Invalidate` never joins a verification that started
|
||||
before it and cannot receive the pre-revoke identity. The shared inner call is not
|
||||
bound to any one caller's cancellation token; a caller that gives up stops waiting
|
||||
without cancelling the others. The per-key index `Invalidate` uses to find every
|
||||
cached secret variant is pruned by a post-eviction callback when the cache drops an
|
||||
entry on its own (TTL expiry, compaction), so it tracks the live cache instead of
|
||||
growing with every token ever verified.
|
||||
- **`CoalescingMarkApiKeyStore`** wraps the library `IApiKeyStore` and forwards at
|
||||
most one `MarkUsed` write per key per
|
||||
`MxGateway:Security:ApiKeyLastUsedCoalesceSeconds` (default 60 s), so even under a
|
||||
|
||||
+18
-7
@@ -10,7 +10,7 @@ Authorization runs as a single gRPC server interceptor registered for every call
|
||||
|
||||
The participating types live under `src/ZB.MOM.WW.MxGateway.Server/Security/Authorization/`:
|
||||
|
||||
- `GatewayGrpcAuthorizationInterceptor` runs the authenticate-then-authorize pipeline for unary and server-streaming calls.
|
||||
- `GatewayGrpcAuthorizationInterceptor` runs the authenticate-then-authorize pipeline for every call shape: unary, server-streaming, client-streaming, and duplex-streaming.
|
||||
- `GatewayGrpcScopeResolver` maps a request message (and, for `MxCommandRequest`, the inner `MxCommandKind`) to the scope string that must be present on the caller.
|
||||
- `GatewayScopes` exposes the canonical scope constants used by the resolver and any downstream consumer.
|
||||
- `GatewayRequestIdentityAccessor` and `IGatewayRequestIdentityAccessor` expose the verified identity to handlers and any service code that runs inside the call.
|
||||
@@ -32,7 +32,7 @@ Centralizing the policy in `GatewayGrpcAuthorizationInterceptor` produces three
|
||||
|
||||
## Interceptor Flow
|
||||
|
||||
`GatewayGrpcAuthorizationInterceptor` overrides both `UnaryServerHandler` and `ServerStreamingServerHandler`. Both call the same private `AuthenticateAndAuthorizeAsync` helper before invoking the continuation, then push the resolved identity onto the accessor for the duration of the call.
|
||||
`GatewayGrpcAuthorizationInterceptor` overrides all four server handlers: `UnaryServerHandler`, `ServerStreamingServerHandler`, `ClientStreamingServerHandler`, and `DuplexStreamingServerHandler`. Each calls the same private `AuthenticateAndAuthorizeAsync` helper before invoking the continuation, then pushes the resolved identity onto the accessor for the duration of the call. The base `Interceptor` passes any handler it does not override straight through, so a client- or duplex-streaming RPC added to a mapped service would otherwise run with no API key check at all. The unary and server-streaming handlers resolve the required scope from the request message; the client- and duplex-streaming handlers have no single request to inspect before the handler reads the stream, so they fail closed to `admin` — the same scope the resolver assigns an unrecognized request type. No public RPC uses either shape today.
|
||||
|
||||
```csharp
|
||||
public override async Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
|
||||
@@ -40,7 +40,10 @@ public override async Task<TResponse> UnaryServerHandler<TRequest, TResponse>(
|
||||
ServerCallContext context,
|
||||
UnaryServerMethod<TRequest, TResponse> continuation)
|
||||
{
|
||||
ApiKeyIdentity? identity = await AuthenticateAndAuthorizeAsync(request, context).ConfigureAwait(false);
|
||||
ApiKeyIdentity? identity = await AuthenticateAndAuthorizeAsync(
|
||||
scopeResolver.ResolveRequiredScope(request),
|
||||
context)
|
||||
.ConfigureAwait(false);
|
||||
IDisposable? identityScope = identity is null ? null : identityAccessor.Push(identity);
|
||||
using (identityScope)
|
||||
{
|
||||
@@ -96,7 +99,7 @@ Before the verification store read, the helper asks a cheap in-process failure c
|
||||
- **Composite `(transport peer, key id)` partitions.** Reaching `MxGateway:Security:ApiKeyFailureLimit` failures binds the throttle to the address that produced them. The key id alone is never the partition: key ids are not secret — they ride in every token and are listed on the dashboard — so keying on them let any network peer deny a key to its legitimate holder. The key id joins the partition only after a token-shape check (literal `mxgw` prefix, at least three non-empty `_` segments, key id of at most 64 characters), and one address may mint at most 32 key-id partitions before the overflow collapses onto that address's fallback partition.
|
||||
- **A per-key-id aggregate** across all peers (`ApiKeyFailureAggregateLimit`, default 30), which bounds a distributed or source-rotating sprayer that never trips any single partition.
|
||||
|
||||
An over-limit state is a valve, not a wall: one request per `ApiKeyFailureProbeIntervalSeconds` (default 5 s) is admitted through to the real verifier, and everything else is refused with `StatusCode.ResourceExhausted` before the store read. The slot is claimed atomically, so a burst arriving together at an interval boundary still yields exactly one admission, and a slot claimed for a request that a later layer then refuses is handed back under a per-state version stamp — never by timestamp comparison, which collides whenever a concurrent failure re-arms the same state on the same clock tick. A successful verification resets both layers — which is why the reset path stays reachable while a key is under active spray. One exception: when the caller's key id was collapsed into its address's shared fallback partition by the per-peer cap, a success clears the key's aggregate but leaves that shared partition alone, since it also holds failures contributed by other key ids from the same address. The tracked partitions form a bounded LRU (`ApiKeyFailureTrackedPeers`) whose eviction prefers fully expired windows and never removes an over-limit partition below a 2x transient overshoot ceiling, so the cap bounds memory without becoming a reset button for an active block. `ResourceExhausted` reveals only that throttling is in effect, not whether any particular secret was valid, preserving the opaque-failure property. Refusals increment `mxgateway.auth.throttled`, tagged `stage=peer|aggregate` and nothing else — `/metrics` is unauthenticated, so neither key ids nor peer addresses may appear there.
|
||||
An over-limit state is a valve, not a wall: one request per `ApiKeyFailureProbeIntervalSeconds` (default 5 s) is admitted through to the real verifier, and everything else is refused with `StatusCode.ResourceExhausted` before the store read. The slot is claimed atomically, so a burst arriving together at an interval boundary still yields exactly one admission, and a slot claimed for a request that a later layer then refuses is handed back under a per-state version stamp — never by timestamp comparison, which collides whenever a concurrent failure re-arms the same state on the same clock tick. A successful verification resets both layers — which is why the reset path stays reachable while a key is under active spray. One exception: when the caller's key id was collapsed into its address's shared fallback partition by the per-peer cap, a success clears the key's aggregate but leaves that shared partition alone, since it also holds failures contributed by other key ids from the same address. The tracked partitions form a bounded LRU (`ApiKeyFailureTrackedPeers`) whose eviction prefers fully expired windows and never removes an over-limit partition below a 2x transient overshoot ceiling, so the cap bounds memory without becoming a reset button for an active block. `ResourceExhausted` reveals only that throttling is in effect, not whether any particular secret was valid, preserving the opaque-failure property. Refusals increment `mxgateway.auth.throttled`, tagged `stage=peer|aggregate` and nothing else — `/metrics` is anonymous by default (`MxGateway:Metrics:RequireAuthentication` is opt-in), so neither key ids nor peer addresses may appear there.
|
||||
|
||||
The dashboard login surface is throttled independently: `POST /auth/login` carries a fixed-window ASP.NET Core rate-limiter policy keyed per remote IP (`MxGateway:Security:LoginRateLimit*`), rejecting a burst with HTTP 429 before the LDAP bind is relayed to the directory. See [GatewayConfiguration](./GatewayConfiguration.md#security-options).
|
||||
|
||||
@@ -262,9 +265,17 @@ the gateway fails closed.
|
||||
|
||||
Non-bulk constraint failures return gRPC `PermissionDenied`. Bulk read
|
||||
commands preserve input order and return a failed `SubscribeResult` for each
|
||||
denied item while still forwarding allowed items to the worker. Every denial
|
||||
adds an `api_key_audit` entry with the key id, command kind, target, and
|
||||
blocking constraint; secured values and raw credentials are never logged.
|
||||
denied item while still forwarding allowed items to the worker. A non-bulk
|
||||
denial adds one `constraint-denied` audit event (canonical `audit_event` table)
|
||||
with the key id, command kind, target, and blocking constraint. A bulk command
|
||||
adds **one aggregated** `constraint-denied` event per command, not one per
|
||||
denied item: its target is `<commandKind>:bulk` and its details carry the
|
||||
denied and total item counts, the distinct blocking constraint names, a sample
|
||||
of at most 20 denied targets (`sampleTruncated` says whether more were
|
||||
denied), and the client correlation id. The audit writer is a bounded channel
|
||||
that drops on full, so per-item events let a single low-privilege key flush
|
||||
every other pending audit record with one large bulk command. Secured values
|
||||
and raw credentials are never logged.
|
||||
|
||||
## Scope Catalog
|
||||
|
||||
|
||||
@@ -430,6 +430,44 @@ Suggested mapping:
|
||||
- Galaxy SQL metadata endpoints if added: `metadata:read`
|
||||
- worker shutdown diagnostics and key management: `admin`
|
||||
|
||||
## Metrics Endpoint Authentication (SEC-14)
|
||||
|
||||
Decision: `/metrics` stays anonymous by default and can be put behind
|
||||
authentication with `MxGateway:Metrics:RequireAuthentication=true`; `/health`
|
||||
stays anonymous unconditionally.
|
||||
|
||||
- **Opt-in rather than on by default.** Existing Prometheus scrapers on the
|
||||
deployed hosts scrape anonymously; flipping the default would silently break
|
||||
them on upgrade. Hosts whose network exposure makes the counters sensitive
|
||||
enable it explicitly.
|
||||
- **Health is never gated.** The family Overview dashboard (another repo) polls
|
||||
`/health/active` without credentials, and liveness/readiness probes are
|
||||
conventionally anonymous. Health responses carry status only.
|
||||
- **Credentials reuse existing plumbing.** One `MxGateway.Metrics` scheme
|
||||
accepts an API key with the `admin` scope (verified through the same cached
|
||||
`IApiKeyVerifier` and the same per-peer failure limiter as gRPC, so `/metrics`
|
||||
is not a way around the gRPC brute-force throttle) or, when no authorization
|
||||
header is sent, the dashboard login cookie with a real dashboard role. No new
|
||||
credential type or store is introduced.
|
||||
- **`admin` scope, any dashboard role.** Metrics are an operator surface, so a
|
||||
data-plane client key (`invoke:*`, `events:read`) is not enough. A dashboard
|
||||
`Viewer` qualifies because every dashboard role already sees the same aggregate
|
||||
counters on the dashboard; the exported metrics carry no session ids or tag
|
||||
values.
|
||||
- **No environment bypass.** The requirement reads no options, so anonymous
|
||||
localhost and `Authentication:Mode=Disabled` — both of which satisfy the
|
||||
dashboard Viewer policy — do not satisfy it. Nor does dev-only
|
||||
`Dashboard:DisableLogin`: its auto-login handler sits under the cookie scheme
|
||||
name, so the metrics scheme skips the cookie fallback while it is on rather than
|
||||
letting a dev flag silently open an endpoint the operator asked to protect.
|
||||
- **Throttle sharing is partial.** The `/metrics` per-peer limiter partition is
|
||||
the remote IP, while the gRPC one is the gRPC peer string (per connection), so
|
||||
only the per-key-id aggregate is shared between the two paths.
|
||||
- **One scheme, plain 401.** Listing the cookie scheme on the policy directly
|
||||
would challenge an anonymous scrape with a 302 redirect to `/login`; a single
|
||||
scheme that falls back to the cookie internally keeps the challenge a `401`
|
||||
with `WWW-Authenticate: Bearer`, which scrapers understand.
|
||||
|
||||
## Worker Process Identity
|
||||
|
||||
Decision: run workers as the gateway service identity for v1.
|
||||
|
||||
@@ -419,6 +419,45 @@ model requires otherwise.
|
||||
| `MxGateway:Security:AuditRetentionDays` | `90` | Days of canonical audit history kept in the `audit_event` table. The audit drain sweeps once at startup and hourly thereafter, deleting rows older than this window; without it the table grows without bound inside the same SQLite file the authentication hot path reads. Rows whose timestamp SQLite cannot parse are never swept. Must be greater than zero — retention can be widened but not switched off. |
|
||||
| `MxGateway:Security:ApiKeyFailureTrackedPeers` | `4096` | Maximum distinct partitions tracked by the failure counter (a bounded LRU) so a spray of unique tokens cannot grow memory without limit. It cannot be used to flush an active block either: only a validly shaped `mxgw_<keyId>_<secret>` token mints a key-id partition (everything else lands on the sender's transport-peer partition), each address may mint at most 32 key-id partitions before the overflow collapses onto that address's fallback partition, and eviction prefers fully expired windows, never removing an over-limit partition until the map exceeds twice this cap. Must be greater than zero. |
|
||||
|
||||
## Metrics Options
|
||||
|
||||
Bound from `MxGateway:Metrics`. Governs the Prometheus `/metrics` scrape endpoint
|
||||
(mounted by the shared `ZB.MOM.WW.Telemetry` package). The health endpoints
|
||||
(`/healthz`, `/health/ready`, `/health/active`) are not affected by this section and
|
||||
always stay anonymous: the family Overview dashboard polls `/health/active` from
|
||||
another repo without credentials, so gating it would break that consumer.
|
||||
|
||||
| Option | Default | Description |
|
||||
|--------|---------|-------------|
|
||||
| `MxGateway:Metrics:RequireAuthentication` | `false` | When `true`, `/metrics` requires an authenticated caller (SEC-14). A scrape is accepted when it presents an API key carrying the `admin` scope as `Authorization: Bearer mxgw_<keyId>_<secret>`, or — with no authorization header — a dashboard login cookie holding a real dashboard role (`Administrator` or `Viewer`). Anonymous localhost access, `Authentication:Mode=Disabled`, and `Dashboard:DisableLogin` do **not** satisfy it. An anonymous scrape gets `401` with `WWW-Authenticate: Bearer`; an authenticated caller without `admin` scope or a dashboard role gets `403`; a peer throttled by the API-key failure limiter (`MxGateway:Security:ApiKeyFailure*`, shared with the gRPC path) gets `429`. The default keeps the endpoint anonymous so existing scrapers keep working. |
|
||||
|
||||
Configure a Prometheus scraper against an authenticated endpoint with a dedicated
|
||||
admin-scoped key, for example:
|
||||
|
||||
```yaml
|
||||
scrape_configs:
|
||||
- job_name: mxgateway
|
||||
authorization:
|
||||
type: Bearer
|
||||
credentials_file: /etc/prometheus/mxgateway.key # contains mxgw_<keyId>_<secret>
|
||||
static_configs:
|
||||
- targets: ["gateway-host:5130"]
|
||||
```
|
||||
|
||||
The setting is read from configuration when endpoints are mapped, so changing it
|
||||
requires a restart. Dev-only `Dashboard:DisableLogin=true` registers an
|
||||
always-succeeding auto-login handler under the dashboard cookie scheme; the
|
||||
metrics scheme deliberately skips its cookie fallback while that flag is on, so
|
||||
`DisableLogin` never silently opens a `/metrics` endpoint that
|
||||
`RequireAuthentication` protects — only an admin-scoped API key is accepted then.
|
||||
|
||||
The `/metrics` path reuses the gRPC API-key failure limiter
|
||||
(`MxGateway:Security:ApiKeyFailure*`), but its per-peer partition is keyed on the
|
||||
remote **IP address**, whereas the gRPC interceptor's is the gRPC peer string
|
||||
(address and port — effectively per connection). The two paths therefore share
|
||||
only the per-key-id aggregate layer (`ApiKeyFailureAggregateLimit`); per-peer
|
||||
failure counts on one path do not throttle the other.
|
||||
|
||||
## Galaxy Options
|
||||
|
||||
| Option | Default | Description |
|
||||
|
||||
+9
-1
@@ -82,6 +82,8 @@ return mapper.MapCommandReply(workerReply);
|
||||
|
||||
Carrying the enqueue timestamp into the worker layer is what lets queue-wait time be measured separately from worker-side execution time when troubleshooting timeouts.
|
||||
|
||||
Every `MxCommandReply` echoes the request's `client_correlation_id` in `correlation_id` — success, MXAccess failure, and constraint-denied replies alike, and an empty string when the caller sent none. The handler stamps it after mapping the worker reply, overwriting the internal gateway↔worker envelope id the worker reply carries: that id is a pipe-routing detail with no meaning to a client, and clients that sent no correlation id previously received it while clients that sent one received it only on denied replies. To keep the two ids joinable, `Invoke` logs the pair (command kind, session id, client correlation id, worker envelope correlation id — no values) once per call at `Debug`, so a client error report quoting its correlation id can be matched to gateway and worker log lines.
|
||||
|
||||
An accepted gRPC command payload can still be too large for the worker pipe: the envelope built around it must fit `MxGateway:Worker:MaxMessageBytes`, which is validated at startup to sit at least a fixed envelope-overhead reserve above `MaxGrpcMessageBytes` (see the headroom rule in [Gateway Configuration](./GatewayConfiguration.md)) so this should not occur for a conformant payload, but if it does, `WorkerClient` raises `WorkerClientException(CommandTooLarge)` and `Invoke` reports `ResourceExhausted` for that command — the session is not faulted, so a client can retry with a smaller payload without reopening the session.
|
||||
|
||||
Selected control commands have additional public behavior beyond their protobuf shape:
|
||||
@@ -94,6 +96,8 @@ Selected control commands have additional public behavior beyond their protobuf
|
||||
|
||||
`StreamEvents` is a server-streaming RPC. The handler delegates the full pipeline to `IEventStreamService` and just forwards each `MxEvent` onto the response stream. Keeping the channel and producer/consumer machinery out of the handler means cancellation, exception mapping, and metric bookkeeping live in one place.
|
||||
|
||||
Each response write receives the call's cancellation token, so a write parked on a slow or vanished client is released when the call is cancelled rather than waiting on the transport. A write that then fails with an `IOException` or `OperationCanceledException` while the call token is cancelled is a routine client disconnect: it maps to `Cancelled` and logs nothing. The same `IOException` on a live call keeps the generic `Unavailable` + warning mapping, because that is a real transport failure. `StreamAlarms` applies the same rule.
|
||||
|
||||
### `AcknowledgeAlarm`
|
||||
|
||||
`AcknowledgeAlarm` is a unary, **session-less** RPC that acknowledges a single alarm. The handler validates `alarm_full_reference` and `operator_user` inline (it does not run through `MxAccessGrpcRequestValidator`) and delegates to `IGatewayAlarmService.AcknowledgeAsync`. The always-on `GatewayAlarmMonitor` routes the ack over its own gateway-managed worker session — clients no longer open a session to acknowledge an alarm. A reference that parses as a canonical GUID forwards to `AcknowledgeAlarmCommand`; a `Provider!Group.Tag` reference forwards to `AcknowledgeAlarmByNameCommand`.
|
||||
@@ -125,6 +129,8 @@ The worker calls `AuthenticateUser` as the operator, then writes `operator_comme
|
||||
|
||||
`StreamAlarms` is a server-streaming, **session-less** RPC that attaches to the gateway's central alarm feed. The handler delegates to `IGatewayAlarmService.StreamAsync`. The stream opens with a `provider_status` and a `snapshot_status` `AlarmFeedMessage` (the current provider mode and snapshot-completeness verdict), then one `AlarmFeedMessage` carrying an `active_alarm` per currently-active alarm (the ConditionRefresh snapshot), then a single `snapshot_complete`, then a `transition` for every subsequent raise / acknowledge / clear — interleaved with a further `provider_status` on each failover/failback and a further `snapshot_status` on each change of the truncation verdict. It is served by the always-on `GatewayAlarmMonitor`, which owns a single gateway-managed worker session and fans out to every attached client — clients no longer open a session of their own. `alarm_filter_prefix`, when set, scopes the stream to a sub-tree.
|
||||
|
||||
A subscriber's opening snapshot is taken from the monitor's memoized alarm-set projection (the same one `QueryActiveAlarms` and the dashboard read), captured under the broadcast lock in the same critical section that registers the subscriber, so no transition can fall between the snapshot and the live stream; the prefix filter runs afterwards, outside the lock. Each subscriber has a bounded queue (2048 messages). A subscriber that falls that far behind is dropped and its stream ends with `ResourceExhausted` ("Alarm feed subscriber fell behind and was dropped; reconnect to re-snapshot."), the same status a `StreamEvents` overflow carries; the remedy is to reconnect, which re-sends the snapshot. The drop is not logged as a gateway failure.
|
||||
|
||||
### `QueryActiveAlarms`
|
||||
|
||||
`QueryActiveAlarms` is a server-streaming, **session-less** RPC that returns a point-in-time snapshot of the currently-active alarm set. Unlike `StreamAlarms`, it does not stay attached for live transitions — it writes the snapshot and completes, which clients use after a reconnect to reseed alarm state or to reconcile alarms that may have been missed during a transport blip. It is served from the always-on `GatewayAlarmMonitor` cache (`IGatewayAlarmService.CurrentAlarms`), so no worker session is opened. The handler does not run through `MxAccessGrpcRequestValidator`; it only null-checks the request inline, then streams one `ActiveAlarmSnapshot` per cached alarm, honouring cancellation between writes. When `alarm_filter_prefix` is non-empty, only alarms whose `alarm_full_reference` starts with that prefix (ordinal `StartsWith`) are emitted; an empty prefix returns the full set. Exceptions are translated by the same `MapException` path as the other handlers. The RPC requires the `events:read` scope (`GatewayScopes.EventsRead`), the same scope as `StreamEvents` and `StreamAlarms`.
|
||||
@@ -294,6 +300,8 @@ Every handler wraps its body in `try { ... } catch (Exception exception) when (e
|
||||
- `SessionManagerException` is mapped by its `ErrorCode`.
|
||||
- `WorkerClientException` is mapped by its `ErrorCode`.
|
||||
|
||||
Two stream-specific cases are handled before `MapException`: on `StreamEvents` and `StreamAlarms` a write failure after the client cancelled becomes `Cancelled` without a log entry, and on `StreamAlarms` an `AlarmFeedSubscriberOverflowException` (slow-subscriber drop) becomes `ResourceExhausted` without a log entry.
|
||||
|
||||
Anything else is logged at warning and surfaced as `Unavailable` with a generic message — clients see a retryable status, and the unexpected exception is captured in gateway logs rather than leaked over the wire.
|
||||
|
||||
```csharp
|
||||
@@ -367,7 +375,7 @@ Under `FailFast` the session is faulted so subsequent commands return `FailedPre
|
||||
|
||||
### Cancellation and cleanup
|
||||
|
||||
The handler creates a linked cancellation token (`streamCts`) so that completing the consumer (client disconnect, error, or graceful end-of-stream) also cancels the producer. The `finally` block cancels the source, disposes the subscriber slot, awaits the producer (swallowing the expected cancellation), and emits `StreamDisconnected("Detached")` so dashboards see the disconnection regardless of cause.
|
||||
The handler creates a linked cancellation token (`streamCts`) so that completing the consumer (client disconnect, error, or graceful end-of-stream) also cancels the producer. The `finally` block cancels the source, disposes the subscriber slot, awaits the producer (swallowing the expected cancellation), and emits `StreamDisconnected(reason)` so dashboards see every disconnection. The reason is the bounded constant (`EventStreamDisconnectReasons`) for how the stream ended — `SessionClosed`, `Overflow`, `WorkerFault`, `ClientCancelled`, `Error`, or `Detached` — rather than one constant for every cause, so an operator can tell slow consumers from worker faults from routine client disconnects.
|
||||
|
||||
`WorkerClientException` thrown by the producer marks the session as faulted before completing the channel — the worker is presumed gone, and any subsequent command on that session must observe the fault rather than silently retry.
|
||||
|
||||
|
||||
+2
-2
@@ -45,7 +45,7 @@ All counters are `Counter<long>`. Tag values come from the call sites listed und
|
||||
| `mxgateway.workers.killed` | `reason` | Forced terminations of worker processes. |
|
||||
| `mxgateway.workers.exited` | `reason` | Clean or fault-driven worker exits. |
|
||||
| `mxgateway.heartbeats.failed` | *(none)* | Aggregate worker heartbeat misses. The exported counter carries **no** `session_id` tag — per-session attribution is unbounded cardinality (every session mints a new time series), so it is kept out of the exporter; use the dashboard snapshot or the log scope for per-session detail. |
|
||||
| `mxgateway.grpc.streams.disconnected` | `reason` | Detachments of the dashboard or client gRPC event stream. |
|
||||
| `mxgateway.grpc.streams.disconnected` | `reason` | Ends of a client `StreamEvents` stream. `reason` is one of a fixed set (`EventStreamDisconnectReasons`): `SessionClosed` (the session event stream completed), `Overflow` (subscriber fell behind its bounded queue), `WorkerFault` (the worker event stream faulted), `ClientCancelled` (the client cancelled or disconnected), `Error` (unclassified), `Detached` (the consumer stopped reading without cancelling). |
|
||||
| `mxgateway.retries.attempted` | `area` | Resilience retries executed by gateway components. |
|
||||
|
||||
### Histograms
|
||||
@@ -181,7 +181,7 @@ IDisposable backlogRegistration = metrics.RegisterEventStreamBacklogSource(
|
||||
() => subscriber.Reader.CanCount ? subscriber.Reader.Count : 0);
|
||||
...
|
||||
backlogRegistration.Dispose();
|
||||
metrics.StreamDisconnected("Detached");
|
||||
metrics.StreamDisconnected(disconnectReason); // an EventStreamDisconnectReasons constant
|
||||
...
|
||||
metrics.QueueOverflow("grpc-event-stream");
|
||||
metrics.Fault(SessionManagerErrorCode.EventQueueOverflow.ToString());
|
||||
|
||||
Reference in New Issue
Block a user