mxaccessgw/docs/Metrics.md

# Gateway Metrics

The metrics subsystem exposes counters, histograms, and observable gauges that describe gateway throughput, queue health, and worker lifecycle. Both the `System.Diagnostics.Metrics` pipeline and the in-memory `GatewayMetricsSnapshot` consume the same underlying state, so external collectors and the dashboard see consistent numbers.

## Overview

`GatewayMetrics` is a singleton (registered in `GatewayApplication.cs`) that owns a single `Meter` named `ZB.MOM.WW.MxGateway` and a set of synchronised counters, histograms, and observable gauges. Subsystems call typed mutator methods (`SessionOpened`, `CommandFailed`, `EventReceived`, etc.) rather than touching the `Meter` directly, which keeps the OpenTelemetry instrument names and tag conventions in one place. A `lock (_syncRoot)` block guards the scalar fields used by `GetSnapshot`, while per-event maps use `ConcurrentDictionary<string, long>` so the hot event path avoids the lock.

## Meter and OpenTelemetry Compatibility

The meter name is exposed as a constant so that hosting code can register it with an OpenTelemetry exporter:

```csharp
public sealed class GatewayMetrics : IDisposable
{
    public const string MeterName = "ZB.MOM.WW.MxGateway";

    public GatewayMetrics()
    {
        _meter = new Meter(MeterName, typeof(GatewayMetrics).Assembly.GetName().Version?.ToString());
        _sessionsOpenedCounter = _meter.CreateCounter<long>("mxgateway.sessions.opened");
        ...
    }
}
```

The meter version is the gateway assembly version, which gives exporters a stable identifier per build. All instrument names use the dotted `mxgateway.<area>.<event>` convention so they group cleanly under a single namespace in tools such as Prometheus, OTLP collectors, or `dotnet-counters`.

## Instrument Inventory

### Counters

All counters are `Counter<long>`. Tag values come from the call sites listed under [Recording Sites](#recording-sites).

| Instrument | Tags | What it measures |
|------------|------|------------------|
| `mxgateway.sessions.opened` | none | Successful `SessionManager.OpenSession` completions. |
| `mxgateway.sessions.closed` | none | Sessions closed cleanly via `SessionManager`. |
| `mxgateway.commands.started` | `method` | Command dispatches initiated by `WorkerClient`. |
| `mxgateway.commands.succeeded` | `method` | Commands acknowledged with success by the worker. |
| `mxgateway.commands.failed` | `method`, `category` | Command failures, where `category` is the `WorkerClientErrorCode` or exception type name. |
| `mxgateway.events.received` | `family` | Worker events accepted into the event pipeline. |
| `mxgateway.queues.overflows` | `queue` | Drops when a bounded queue rejects a message (e.g. `grpc-event-stream`). |
| `mxgateway.faults` | `category` | Faults reported by session, event, or worker code paths. The category is a `SessionManagerErrorCode` or `WorkerClientErrorCode` name. |
| `mxgateway.workers.killed` | `reason` | Forced terminations of worker processes. |
| `mxgateway.workers.exited` | `reason` | Clean or fault-driven worker exits. |
| `mxgateway.heartbeats.failed` | *(none)* | Aggregate worker heartbeat misses. The exported counter carries **no** `session_id` tag — per-session attribution is unbounded cardinality (every session mints a new time series), so it is kept out of the exporter; use the dashboard snapshot or the log scope for per-session detail. |
| `mxgateway.grpc.streams.disconnected` | `reason` | Detachments of the dashboard or client gRPC event stream. |
| `mxgateway.retries.attempted` | `area` | Resilience retries executed by gateway components. |

### Histograms

Histograms record durations in seconds (the `unit` argument on `CreateHistogram`):

```csharp
_workerStartupLatencyHistogram = _meter.CreateHistogram<double>("mxgateway.workers.startup.duration", "s");
_commandLatencyHistogram = _meter.CreateHistogram<double>("mxgateway.commands.duration", "s");
_eventStreamSendLatencyHistogram = _meter.CreateHistogram<double>("mxgateway.events.stream_send.duration", "s");
```

| Instrument | Tags | What it measures |
|------------|------|------------------|
| `mxgateway.workers.startup.duration` | none | Time from `WorkerClient` launch to worker-ready. |
| `mxgateway.commands.duration` | `method`, optional `category` | Command round-trip time. The `category` tag is added on failure so success and failure latencies stay distinguishable. |
| `mxgateway.events.stream_send.duration` | `family` | Time spent writing each public event to the gRPC response stream in `MxAccessGatewayService.StreamEvents`. |

### Observable gauges

Observable gauges are pull-based; the `Meter` invokes the supplied callback whenever a listener samples it. Each callback re-acquires `_syncRoot` so the gauge value matches the snapshot taken at the same instant.

| Instrument | Source field | Description |
|------------|--------------|-------------|
| `mxgateway.sessions.open` | `_openSessions` | Currently open sessions tracked by `SessionManager`. |
| `mxgateway.workers.running` | `_workersRunning` | Worker clients in a running state. |
| `mxgateway.events.worker_queue.depth` | `_workerEventQueueDepth` | Last reported depth of the worker-side event queue. |
| `mxgateway.events.grpc_stream_queue.depth` | `_eventStreamBacklogSources` (summed on demand) | Live backlog buffered across every active `EventStreamService` subscriber, summed from the subscribers' channel `Count` at collection time. |

## Snapshot Shape

`GatewayMetricsSnapshot` is the immutable view of the same state, returned by `GatewayMetrics.GetSnapshot()` while holding `_syncRoot`. The dictionaries are copied so the caller can iterate without further synchronisation. The dashboard service is the primary consumer.

```csharp
public sealed record GatewayMetricsSnapshot(
    int OpenSessions,
    int WorkersRunning,
    int WorkerEventQueueDepth,
    int GrpcEventStreamQueueDepth,
    long SessionsOpened,
    long SessionsClosed,
    long CommandsStarted,
    long CommandsSucceeded,
    long CommandsFailed,
    long EventsReceived,
    long QueueOverflows,
    long Faults,
    long WorkerKills,
    long WorkerExits,
    long HeartbeatFailures,
    long StreamDisconnects,
    long RetryAttempts,
    IReadOnlyDictionary<string, long> CommandFailuresByMethod,
    IReadOnlyDictionary<string, long> EventsByFamily,
    IReadOnlyDictionary<string, long> EventsBySession,
    IReadOnlyDictionary<string, long> RetryAttemptsByArea);
```

The scalar fields mirror the counters and gauges. The four dictionaries provide the breakdowns that counter tags would otherwise require an exporter to aggregate:

- `CommandFailuresByMethod` keys by gRPC method name.
- `EventsByFamily` keys by event family (the `Family` enum on a worker event).
- `EventsBySession` keys by `sessionId`; entries are removed via `RemoveSessionEvents` when a session closes so the map does not grow without bound.
- `RetryAttemptsByArea` keys by the resilience `area` tag, e.g. `worker_startup`.

`EventsReceived` is read with `Interlocked.Read(ref _eventsReceived)` because `EventReceived` increments it via `Interlocked.Increment` outside the lock to keep the event-ingestion path non-blocking.

## Recording Sites

The recording call sites describe the code paths that write into each instrument. This mapping makes it easier to trace an unexpected counter reading back to a subsystem.

### Session manager

`Sessions/SessionManager.cs` emits session lifecycle and fault counters:

```csharp
_metrics.SessionOpened();
...
_metrics.Fault(SessionManagerErrorCode.OpenFailed.ToString());
...
_metrics.SessionClosed();
...
_metrics.SessionRemoved();
...
_metrics.Fault(SessionManagerErrorCode.CloseFailed.ToString());
...
_metrics.RemoveSessionEvents(session.SessionId);
```

`SessionRemoved` decrements the open-session gauge without incrementing the closed counter, which covers cases where a session is evicted rather than closed by the client.

### Worker client

`Workers/WorkerClient.cs` records command throughput, worker lifecycle, heartbeat failures, and the worker-side event queue depth:

- `CommandStarted(method)` and `CommandSucceeded(method, duration)` / `CommandFailed(method, category, duration)` around the worker request/response pair.
- `WorkerStarted(startupDuration)` once the worker reports ready.
- `RecordWorkerStoppedOnce` calls `WorkerStopped(reason)` exactly once per worker, guarding against double-counting on simultaneous fault and exit signals.
- `WorkerKilled(reason)` when the client forcibly terminates the worker.
- `HeartbeatFailed(SessionId)` per missed heartbeat.
- `SetWorkerEventQueueDepth(queueDepth)` after each event ingest.
- `EventReceived(SessionId, workerEvent.Event.Family.ToString())` for each worker event.

### Worker process launcher

`Workers/WorkerProcessLauncher.cs` records process-level kills and startup retries:

```csharp
_metrics.WorkerKilled(reason);
...
_metrics.RetryAttempted("worker_startup");
```

The `worker_startup` tag is hard-coded so the `RetryAttemptsByArea` snapshot reports launcher retries distinctly from other resilience areas.

### Session worker client factory

`Sessions/SessionWorkerClientFactory.cs` records the worker kill that follows a failed `OpenSession` handshake:

```csharp
_metrics.WorkerKilled("OpenSessionFailed");
```

This is the only fault path where the factory itself owns the kill decision; once the worker is bound to a session, the `WorkerClient` becomes responsible for lifecycle metrics.

### gRPC event stream service

`Grpc/EventStreamService.cs` reports the event-stream backlog through a live backlog source and records the disconnect/overflow counters:

```csharp
IDisposable backlogRegistration = metrics.RegisterEventStreamBacklogSource(
    () => subscriber.Reader.CanCount ? subscriber.Reader.Count : 0);
...
backlogRegistration.Dispose();
metrics.StreamDisconnected("Detached");
...
metrics.QueueOverflow("grpc-event-stream");
metrics.Fault(SessionManagerErrorCode.EventQueueOverflow.ToString());
...
metrics.Fault(WorkerClientErrorCode.WorkerFaulted.ToString());
```

Rather than pushing a running total on every streamed event (which took the subscriber channel's `Count` lock and the metric lock per event), each subscriber registers its channel as a backlog source for the duration of the stream. The gauge callback and `GetSnapshot` sum those sources' `Count` on demand — only when the metric is scraped or projected — so the streaming hot path does no per-event gauge bookkeeping (GWC-15). Disposing the registration in the `finally` removes the subscriber's contribution, so the gauge reflects only the remaining subscribers and returns to zero once the last stream detaches. Negative source returns are clamped to zero when summed.

`Grpc/MxAccessGatewayService.cs` records gRPC event send latency around each response-stream write using the allocation-free timestamp API (no per-event `Stopwatch` object):

```csharp
long sendStartTimestamp = Stopwatch.GetTimestamp();
await responseStream.WriteAsync(publicEvent).ConfigureAwait(false);
metrics.RecordEventStreamSend(
    publicEvent.Family.ToString(),
    Stopwatch.GetElapsedTime(sendStartTimestamp));
```

## Dashboard Consumption

`Dashboard/DashboardSnapshotService.cs` calls `_metrics.GetSnapshot()` once per `GetSnapshot` invocation and projects it into the dashboard transport types together with the session registry view. The dashboard receives a single, internally consistent snapshot per tick rather than reading individual counters at separate times. See [Gateway Dashboard Design](./GatewayDashboardDesign.md) and [Dashboard Interface Design](./DashboardInterfaceDesign.md) for the projection rules and wire format.

## Auth Store Write Churn

The per-RPC authentication path is a hidden write source: the shared verifier
refreshes `last_used_utc` on every authenticated call. To keep that off the
auth-store throughput ceiling, the gateway caches successful verifications for a
short TTL and coalesces the `last_used_utc` write to at most one per key per
window (`MxGateway:Security:ApiKeyVerificationCacheSeconds` /
`ApiKeyLastUsedCoalesceSeconds`, defaults 15 s / 60 s). This bounds WAL churn on
the bulk-read workload without dropping any metric — auth activity is still
observable via command-throughput counters and audit rows. See
[Authentication](./Authentication.md#hot-path-caching-and-last-used-coalescing).

## Related Documentation

- [Gateway Dashboard Design](./GatewayDashboardDesign.md)
- [Dashboard Interface Design](./DashboardInterfaceDesign.md)
- [Sessions](./Sessions.md)