Joseph Doherty
d4154e340c
The GWC-04 remediation decoupled the read loop from event backpressure by
staging events into an unbounded channel, so its TryWrite always succeeded and
the only overflow fault was a single timed WriteAsync exceeding
EventChannelFullModeTimeout. A consumer draining slower than the worker
produces — each individual write still completing inside the window — therefore
grew gateway memory without bound, without a fault, and without a metric: the
queue-depth gauge counted only the bounded consumer channel, so staged events
were invisible.
Bound _eventStaging at 2 x EventChannelCapacity (Wait, single reader/writer, no
synchronous continuations). A rejected staging TryWrite is the sustained
slow-drain signal and faults the client ProtocolViolation with
QueueOverflow("worker-event-staging"), guarded by IsTerminalState() so a
completed channel during shutdown stays a silent drop. SetFaulted is
non-blocking, so the read loop still never awaits behind events. The timed-write
fault is unchanged and still catches the full-stall case earlier.
Move the queue-depth increment from EnqueueWorkerEventAsync to StageWorkerEvent
so the single counter reports total undelivered events (staged + queued); the
decrement at consumer read was already correct. No new configuration key: the
bound is derived, and gateway-side buffering per session is now at most
3 x MxGateway:Events:QueueCapacity. Coordination with still-open GWC-21
(EventChannelFullModeTimeout configurability) remains open and was not blocked
on.
Tests: StagingChannelOverflowFaultsWorkerWithoutWaitingForFullModeTimeout (5-min
full-mode timeout so only the staging bound can fire; asserts an interleaved
command reply still completes) and WorkerEventQueueDepthGaugeCountsStagedEvents.
Docs updated in the same change: GatewayProcessDesign, MxAccessWorkerInstanceDesign,
GatewayConfiguration, Metrics. GWC-24 flipped to Done in both trackers.
225 lines
13 KiB
Markdown
225 lines
13 KiB
Markdown
# Gateway Metrics
|
||
|
||
The metrics subsystem exposes counters, histograms, and observable gauges that describe gateway throughput, queue health, and worker lifecycle. Both the `System.Diagnostics.Metrics` pipeline and the in-memory `GatewayMetricsSnapshot` consume the same underlying state, so external collectors and the dashboard see consistent numbers.
|
||
|
||
## Overview
|
||
|
||
`GatewayMetrics` is a singleton (registered in `GatewayApplication.cs`) that owns a single `Meter` named `ZB.MOM.WW.MxGateway` and a set of synchronised counters, histograms, and observable gauges. Subsystems call typed mutator methods (`SessionOpened`, `CommandFailed`, `EventReceived`, etc.) rather than touching the `Meter` directly, which keeps the OpenTelemetry instrument names and tag conventions in one place. A `lock (_syncRoot)` block guards the scalar fields used by `GetSnapshot`, while per-event maps use `ConcurrentDictionary<string, long>` so the hot event path avoids the lock.
|
||
|
||
## Meter and OpenTelemetry Compatibility
|
||
|
||
The meter name is exposed as a constant so that hosting code can register it with an OpenTelemetry exporter:
|
||
|
||
```csharp
|
||
public sealed class GatewayMetrics : IDisposable
|
||
{
|
||
public const string MeterName = "ZB.MOM.WW.MxGateway";
|
||
|
||
public GatewayMetrics()
|
||
{
|
||
_meter = new Meter(MeterName, typeof(GatewayMetrics).Assembly.GetName().Version?.ToString());
|
||
_sessionsOpenedCounter = _meter.CreateCounter<long>("mxgateway.sessions.opened");
|
||
...
|
||
}
|
||
}
|
||
```
|
||
|
||
The meter version is the gateway assembly version, which gives exporters a stable identifier per build. All instrument names use the dotted `mxgateway.<area>.<event>` convention so they group cleanly under a single namespace in tools such as Prometheus, OTLP collectors, or `dotnet-counters`.
|
||
|
||
## Instrument Inventory
|
||
|
||
### Counters
|
||
|
||
All counters are `Counter<long>`. Tag values come from the call sites listed under [Recording Sites](#recording-sites).
|
||
|
||
| Instrument | Tags | What it measures |
|
||
|------------|------|------------------|
|
||
| `mxgateway.sessions.opened` | none | Successful `SessionManager.OpenSession` completions. |
|
||
| `mxgateway.sessions.closed` | none | Sessions closed cleanly via `SessionManager`. |
|
||
| `mxgateway.commands.started` | `method` | Command dispatches initiated by `WorkerClient`. |
|
||
| `mxgateway.commands.succeeded` | `method` | Commands acknowledged with success by the worker. |
|
||
| `mxgateway.commands.failed` | `method`, `category` | Command failures, where `category` is the `WorkerClientErrorCode` or exception type name. |
|
||
| `mxgateway.events.received` | `family` | Worker events accepted into the event pipeline. |
|
||
| `mxgateway.queues.overflows` | `queue` | Drops when a bounded queue rejects a message (e.g. `grpc-event-stream`). |
|
||
| `mxgateway.faults` | `category` | Faults reported by session, event, or worker code paths. The category is a `SessionManagerErrorCode` or `WorkerClientErrorCode` name. |
|
||
| `mxgateway.workers.killed` | `reason` | Forced terminations of worker processes. |
|
||
| `mxgateway.workers.exited` | `reason` | Clean or fault-driven worker exits. |
|
||
| `mxgateway.heartbeats.failed` | *(none)* | Aggregate worker heartbeat misses. The exported counter carries **no** `session_id` tag — per-session attribution is unbounded cardinality (every session mints a new time series), so it is kept out of the exporter; use the dashboard snapshot or the log scope for per-session detail. |
|
||
| `mxgateway.grpc.streams.disconnected` | `reason` | Detachments of the dashboard or client gRPC event stream. |
|
||
| `mxgateway.retries.attempted` | `area` | Resilience retries executed by gateway components. |
|
||
|
||
### Histograms
|
||
|
||
Histograms record durations in seconds (the `unit` argument on `CreateHistogram`):
|
||
|
||
```csharp
|
||
_workerStartupLatencyHistogram = _meter.CreateHistogram<double>("mxgateway.workers.startup.duration", "s");
|
||
_commandLatencyHistogram = _meter.CreateHistogram<double>("mxgateway.commands.duration", "s");
|
||
_eventStreamSendLatencyHistogram = _meter.CreateHistogram<double>("mxgateway.events.stream_send.duration", "s");
|
||
```
|
||
|
||
| Instrument | Tags | What it measures |
|
||
|------------|------|------------------|
|
||
| `mxgateway.workers.startup.duration` | none | Time from `WorkerClient` launch to worker-ready. |
|
||
| `mxgateway.commands.duration` | `method`, optional `category` | Command round-trip time. The `category` tag is added on failure so success and failure latencies stay distinguishable. |
|
||
| `mxgateway.events.stream_send.duration` | `family` | Time spent writing each public event to the gRPC response stream in `MxAccessGatewayService.StreamEvents`. |
|
||
|
||
### Observable gauges
|
||
|
||
Observable gauges are pull-based; the `Meter` invokes the supplied callback whenever a listener samples it. Each callback re-acquires `_syncRoot` so the gauge value matches the snapshot taken at the same instant.
|
||
|
||
| Instrument | Source field | Description |
|
||
|------------|--------------|-------------|
|
||
| `mxgateway.sessions.open` | `_openSessions` | Currently open sessions tracked by `SessionManager`. |
|
||
| `mxgateway.workers.running` | `_workersRunning` | Worker clients in a running state. |
|
||
| `mxgateway.events.worker_queue.depth` | `_workerEventQueueDepth` | Undelivered worker events held by `WorkerClient` — staged *and* queued (GWC-24). Incremented when the read loop stages an event, decremented when the consumer reads it, so a backlog stuck in the staging channel is visible rather than invisible. |
|
||
| `mxgateway.events.grpc_stream_queue.depth` | `_eventStreamBacklogSources` (summed on demand) | Live backlog buffered across every active `EventStreamService` subscriber, summed from the subscribers' channel `Count` at collection time. |
|
||
|
||
## Snapshot Shape
|
||
|
||
`GatewayMetricsSnapshot` is the immutable view of the same state, returned by `GatewayMetrics.GetSnapshot()` while holding `_syncRoot`. The dictionaries are copied so the caller can iterate without further synchronisation. The dashboard service is the primary consumer.
|
||
|
||
```csharp
|
||
public sealed record GatewayMetricsSnapshot(
|
||
int OpenSessions,
|
||
int WorkersRunning,
|
||
int WorkerEventQueueDepth,
|
||
int GrpcEventStreamQueueDepth,
|
||
long SessionsOpened,
|
||
long SessionsClosed,
|
||
long CommandsStarted,
|
||
long CommandsSucceeded,
|
||
long CommandsFailed,
|
||
long EventsReceived,
|
||
long QueueOverflows,
|
||
long Faults,
|
||
long WorkerKills,
|
||
long WorkerExits,
|
||
long HeartbeatFailures,
|
||
long StreamDisconnects,
|
||
long RetryAttempts,
|
||
IReadOnlyDictionary<string, long> CommandFailuresByMethod,
|
||
IReadOnlyDictionary<string, long> EventsByFamily,
|
||
IReadOnlyDictionary<string, long> EventsBySession,
|
||
IReadOnlyDictionary<string, long> RetryAttemptsByArea);
|
||
```
|
||
|
||
The scalar fields mirror the counters and gauges. The four dictionaries provide the breakdowns that counter tags would otherwise require an exporter to aggregate:
|
||
|
||
- `CommandFailuresByMethod` keys by gRPC method name.
|
||
- `EventsByFamily` keys by event family (the `Family` enum on a worker event).
|
||
- `EventsBySession` keys by `sessionId`; entries are removed via `RemoveSessionEvents` when a session closes so the map does not grow without bound.
|
||
- `RetryAttemptsByArea` keys by the resilience `area` tag, e.g. `worker_startup`.
|
||
|
||
`EventsReceived` is read with `Interlocked.Read(ref _eventsReceived)` because `EventReceived` increments it via `Interlocked.Increment` outside the lock to keep the event-ingestion path non-blocking.
|
||
|
||
## Recording Sites
|
||
|
||
The recording call sites describe the code paths that write into each instrument. This mapping makes it easier to trace an unexpected counter reading back to a subsystem.
|
||
|
||
### Session manager
|
||
|
||
`Sessions/SessionManager.cs` emits session lifecycle and fault counters:
|
||
|
||
```csharp
|
||
_metrics.SessionOpened();
|
||
...
|
||
_metrics.Fault(SessionManagerErrorCode.OpenFailed.ToString());
|
||
...
|
||
_metrics.SessionClosed();
|
||
...
|
||
_metrics.SessionRemoved();
|
||
...
|
||
_metrics.Fault(SessionManagerErrorCode.CloseFailed.ToString());
|
||
...
|
||
_metrics.RemoveSessionEvents(session.SessionId);
|
||
```
|
||
|
||
`SessionRemoved` decrements the open-session gauge without incrementing the closed counter, which covers cases where a session is evicted rather than closed by the client.
|
||
|
||
### Worker client
|
||
|
||
`Workers/WorkerClient.cs` records command throughput, worker lifecycle, heartbeat failures, and the worker-side event queue depth:
|
||
|
||
- `CommandStarted(method)` and `CommandSucceeded(method, duration)` / `CommandFailed(method, category, duration)` around the worker request/response pair.
|
||
- `WorkerStarted(startupDuration)` once the worker reports ready.
|
||
- `RecordWorkerStoppedOnce` calls `WorkerStopped(reason)` exactly once per worker, guarding against double-counting on simultaneous fault and exit signals.
|
||
- `WorkerKilled(reason)` when the client forcibly terminates the worker.
|
||
- `HeartbeatFailed(SessionId)` per missed heartbeat.
|
||
- `SetWorkerEventQueueDepth(queueDepth)` when the read loop stages an event and when the consumer reads one, so the gauge tracks staged + queued events.
|
||
- `EventReceived(SessionId, workerEvent.Event.Family.ToString())` for each worker event.
|
||
- `QueueOverflow("worker-events")` when the timed write into the bounded consumer channel exceeds `EventChannelFullModeTimeout`, and `QueueOverflow("worker-event-staging")` when the staging channel is full at its `2 × EventChannelCapacity` bound. The two labels distinguish a stalled consumer from one that merely drains too slowly; both fault the session with `ProtocolViolation`.
|
||
|
||
### Worker process launcher
|
||
|
||
`Workers/WorkerProcessLauncher.cs` records process-level kills and startup retries:
|
||
|
||
```csharp
|
||
_metrics.WorkerKilled(reason);
|
||
...
|
||
_metrics.RetryAttempted("worker_startup");
|
||
```
|
||
|
||
The `worker_startup` tag is hard-coded so the `RetryAttemptsByArea` snapshot reports launcher retries distinctly from other resilience areas.
|
||
|
||
### Session worker client factory
|
||
|
||
`Sessions/SessionWorkerClientFactory.cs` records the worker kill that follows a failed `OpenSession` handshake:
|
||
|
||
```csharp
|
||
_metrics.WorkerKilled("OpenSessionFailed");
|
||
```
|
||
|
||
This is the only fault path where the factory itself owns the kill decision; once the worker is bound to a session, the `WorkerClient` becomes responsible for lifecycle metrics.
|
||
|
||
### gRPC event stream service
|
||
|
||
`Grpc/EventStreamService.cs` reports the event-stream backlog through a live backlog source and records the disconnect/overflow counters:
|
||
|
||
```csharp
|
||
IDisposable backlogRegistration = metrics.RegisterEventStreamBacklogSource(
|
||
() => subscriber.Reader.CanCount ? subscriber.Reader.Count : 0);
|
||
...
|
||
backlogRegistration.Dispose();
|
||
metrics.StreamDisconnected("Detached");
|
||
...
|
||
metrics.QueueOverflow("grpc-event-stream");
|
||
metrics.Fault(SessionManagerErrorCode.EventQueueOverflow.ToString());
|
||
...
|
||
metrics.Fault(WorkerClientErrorCode.WorkerFaulted.ToString());
|
||
```
|
||
|
||
Rather than pushing a running total on every streamed event (which took the subscriber channel's `Count` lock and the metric lock per event), each subscriber registers its channel as a backlog source for the duration of the stream. The gauge callback and `GetSnapshot` sum those sources' `Count` on demand — only when the metric is scraped or projected — so the streaming hot path does no per-event gauge bookkeeping (GWC-15). Disposing the registration in the `finally` removes the subscriber's contribution, so the gauge reflects only the remaining subscribers and returns to zero once the last stream detaches. Negative source returns are clamped to zero when summed.
|
||
|
||
`Grpc/MxAccessGatewayService.cs` records gRPC event send latency around each response-stream write using the allocation-free timestamp API (no per-event `Stopwatch` object):
|
||
|
||
```csharp
|
||
long sendStartTimestamp = Stopwatch.GetTimestamp();
|
||
await responseStream.WriteAsync(publicEvent).ConfigureAwait(false);
|
||
metrics.RecordEventStreamSend(
|
||
publicEvent.Family.ToString(),
|
||
Stopwatch.GetElapsedTime(sendStartTimestamp));
|
||
```
|
||
|
||
## Dashboard Consumption
|
||
|
||
`Dashboard/DashboardSnapshotService.cs` calls `_metrics.GetSnapshot()` once per `GetSnapshot` invocation and projects it into the dashboard transport types together with the session registry view. The dashboard receives a single, internally consistent snapshot per tick rather than reading individual counters at separate times. See [Gateway Dashboard Design](./GatewayDashboardDesign.md) and [Dashboard Interface Design](./DashboardInterfaceDesign.md) for the projection rules and wire format.
|
||
|
||
## Auth Store Write Churn
|
||
|
||
The per-RPC authentication path is a hidden write source: the shared verifier
|
||
refreshes `last_used_utc` on every authenticated call. To keep that off the
|
||
auth-store throughput ceiling, the gateway caches successful verifications for a
|
||
short TTL and coalesces the `last_used_utc` write to at most one per key per
|
||
window (`MxGateway:Security:ApiKeyVerificationCacheSeconds` /
|
||
`ApiKeyLastUsedCoalesceSeconds`, defaults 15 s / 60 s). This bounds WAL churn on
|
||
the bulk-read workload without dropping any metric — auth activity is still
|
||
observable via command-throughput counters and audit rows. See
|
||
[Authentication](./Authentication.md#hot-path-caching-and-last-used-coalescing).
|
||
|
||
## Related Documentation
|
||
|
||
- [Gateway Dashboard Design](./GatewayDashboardDesign.md)
|
||
- [Dashboard Interface Design](./DashboardInterfaceDesign.md)
|
||
- [Sessions](./Sessions.md)
|