fix(gateway): remove the orphaned mxgateway.retries.attempted instrument (mxaccessgw#215)

The launcher's Polly startup-probe pipeline was its only recorder and was
removed in mxaccessgw#188, so the counter, the RetryAttempted method and the
snapshot's RetryAttempts / RetryAttemptsByArea fields carried a permanent
zero. No dashboard page or in-repo alert rule references them. Metrics.md
drops the row and the snapshot shape (which also gains the missing
AlarmProviderSwitchCount field).

Claude-Session: https://claude.ai/code/session_01GT67AEHd2Acv1YPwGfXEoV
This commit is contained in:
Joseph Doherty
2026-11-09 12:27:42 -05:00
parent f829df8087
commit d56353bf08
3 changed files with 6 additions and 38 deletions
+4 -7
View File
@@ -46,7 +46,6 @@ All counters are `Counter<long>`. Tag values come from the call sites listed und
| `mxgateway.workers.exited` | `reason` | Clean or fault-driven worker exits. |
| `mxgateway.heartbeats.failed` | *(none)* | Aggregate worker heartbeat misses. The exported counter carries **no** `session_id` tag — per-session attribution is unbounded cardinality (every session mints a new time series), so it is kept out of the exporter; use the dashboard snapshot or the log scope for per-session detail. |
| `mxgateway.grpc.streams.disconnected` | `reason` | Ends of a client `StreamEvents` stream. `reason` is one of a fixed set (`EventStreamDisconnectReasons`): `SessionClosed` (the session event stream completed), `Overflow` (subscriber fell behind its bounded queue), `WorkerFault` (the worker event stream faulted), `ClientCancelled` (the client cancelled or disconnected), `Error` (unclassified), `Detached` (the consumer stopped reading without cancelling). |
| `mxgateway.retries.attempted` | `area` | Resilience retries executed by gateway components. |
### Histograms
@@ -97,19 +96,17 @@ public sealed record GatewayMetricsSnapshot(
long WorkerExits,
long HeartbeatFailures,
long StreamDisconnects,
long RetryAttempts,
long AlarmProviderSwitchCount,
IReadOnlyDictionary<string, long> CommandFailuresByMethod,
IReadOnlyDictionary<string, long> EventsByFamily,
IReadOnlyDictionary<string, long> EventsBySession,
IReadOnlyDictionary<string, long> RetryAttemptsByArea);
IReadOnlyDictionary<string, long> EventsBySession);
```
The scalar fields mirror the counters and gauges. The four dictionaries provide the breakdowns that counter tags would otherwise require an exporter to aggregate:
The scalar fields mirror the counters and gauges. The three dictionaries provide the breakdowns that counter tags would otherwise require an exporter to aggregate:
- `CommandFailuresByMethod` keys by gRPC method name.
- `EventsByFamily` keys by event family (the `Family` enum on a worker event).
- `EventsBySession` keys by `sessionId`; entries are removed via `RemoveSessionEvents` when a session closes so the map does not grow without bound.
- `RetryAttemptsByArea` keys by the resilience `area` tag, (no area is currently recorded by the launcher).
`EventsReceived` is read with `Interlocked.Read(ref _eventsReceived)` because `EventReceived` increments it via `Interlocked.Increment` outside the lock to keep the event-ingestion path non-blocking. `CommandsStarted`, `CommandsSucceeded`, `CommandsFailed`, and `CommandFailuresByMethod` are read the same way: the command counters run two-to-three times per gRPC call, so they are recorded with `Interlocked` and a `ConcurrentDictionary` rather than under `_syncRoot` (PERF-02). The two queue depths are pulled from their registered sources before the lock is taken, since those delegates reach into subscriber channels and worker clients.
@@ -158,7 +155,7 @@ _metrics.RemoveSessionEvents(session.SessionId);
_metrics.WorkerKilled(reason);
```
The launcher no longer retries the startup probe (mxaccessgw#188), so it no longer records `RetryAttempted("worker_startup")`.
The launcher no longer retries the startup probe (mxaccessgw#188). The `mxgateway.retries.attempted` counter that the retry pipeline fed, together with the snapshot's `RetryAttempts` / `RetryAttemptsByArea` fields, was removed in mxaccessgw#215 because nothing recorded it any more; an exporter query on that name now finds no series rather than a flat zero.
### Session worker client factory
@@ -22,7 +22,6 @@ public sealed class GatewayMetrics : IDisposable
private readonly Counter<long> _workerExitsCounter;
private readonly Counter<long> _heartbeatFailuresCounter;
private readonly Counter<long> _streamDisconnectsCounter;
private readonly Counter<long> _retryAttemptsCounter;
private readonly Counter<long> _alarmProviderSwitchesCounter;
private readonly Counter<long> _authThrottledCounter;
private readonly Histogram<double> _workerStartupLatencyHistogram;
@@ -33,7 +32,6 @@ public sealed class GatewayMetrics : IDisposable
private readonly ConcurrentDictionary<string, long> _commandFailuresByMethod = new(StringComparer.OrdinalIgnoreCase);
private readonly ConcurrentDictionary<string, long> _eventsByFamily = new(StringComparer.OrdinalIgnoreCase);
private readonly ConcurrentDictionary<string, long> _eventsBySession = new(StringComparer.Ordinal);
private readonly Dictionary<string, long> _retryAttemptsByArea = new(StringComparer.OrdinalIgnoreCase);
// live backlog sources for the gRPC event-stream queue-depth gauge. Each active
// StreamEvents subscriber registers a delegate returning its channel's current backlog; the
@@ -66,7 +64,6 @@ public sealed class GatewayMetrics : IDisposable
private long _workerExits;
private long _heartbeatFailures;
private long _streamDisconnects;
private long _retryAttempts;
private long _alarmProviderSwitches;
private bool _disposed;
@@ -88,7 +85,6 @@ public sealed class GatewayMetrics : IDisposable
_workerExitsCounter = _meter.CreateCounter<long>("mxgateway.workers.exited");
_heartbeatFailuresCounter = _meter.CreateCounter<long>("mxgateway.heartbeats.failed");
_streamDisconnectsCounter = _meter.CreateCounter<long>("mxgateway.grpc.streams.disconnected");
_retryAttemptsCounter = _meter.CreateCounter<long>("mxgateway.retries.attempted");
_alarmProviderSwitchesCounter = _meter.CreateCounter<long>("mxgateway.alarms.provider_switches");
_authThrottledCounter = _meter.CreateCounter<long>("mxgateway.auth.throttled");
_workerStartupLatencyHistogram = _meter.CreateHistogram<double>("mxgateway.workers.startup.duration", "s");
@@ -401,21 +397,6 @@ public sealed class GatewayMetrics : IDisposable
_streamDisconnectsCounter.Add(1, new KeyValuePair<string, object?>("reason", reason));
}
/// <summary>
/// Records that a retry was attempted in the given area.
/// </summary>
/// <param name="area">Area in which the retry was attempted.</param>
public void RetryAttempted(string area)
{
lock (_syncRoot)
{
_retryAttempts++;
Increment(_retryAttemptsByArea, area);
}
_retryAttemptsCounter.Add(1, new KeyValuePair<string, object?>("area", area));
}
/// <summary>
/// Records that the alarm provider switched modes, increments the switch count, and updates the
/// current provider mode gauge.
@@ -482,12 +463,10 @@ public sealed class GatewayMetrics : IDisposable
WorkerExits: _workerExits,
HeartbeatFailures: _heartbeatFailures,
StreamDisconnects: _streamDisconnects,
RetryAttempts: _retryAttempts,
AlarmProviderSwitchCount: _alarmProviderSwitches,
CommandFailuresByMethod: new Dictionary<string, long>(_commandFailuresByMethod, StringComparer.OrdinalIgnoreCase),
EventsByFamily: new Dictionary<string, long>(_eventsByFamily, StringComparer.OrdinalIgnoreCase),
EventsBySession: new Dictionary<string, long>(_eventsBySession, StringComparer.Ordinal),
RetryAttemptsByArea: new Dictionary<string, long>(_retryAttemptsByArea, StringComparer.OrdinalIgnoreCase));
EventsBySession: new Dictionary<string, long>(_eventsBySession, StringComparer.Ordinal));
}
}
@@ -553,12 +532,6 @@ public sealed class GatewayMetrics : IDisposable
}
}
private static void Increment(Dictionary<string, long> values, string key)
{
values.TryGetValue(key, out long currentValue);
values[key] = currentValue + 1;
}
private static void Increment(ConcurrentDictionary<string, long> values, string key)
{
values.AddOrUpdate(key, 1, static (_, currentValue) => currentValue + 1);
@@ -17,9 +17,7 @@ public sealed record GatewayMetricsSnapshot(
long WorkerExits,
long HeartbeatFailures,
long StreamDisconnects,
long RetryAttempts,
long AlarmProviderSwitchCount,
IReadOnlyDictionary<string, long> CommandFailuresByMethod,
IReadOnlyDictionary<string, long> EventsByFamily,
IReadOnlyDictionary<string, long> EventsBySession,
IReadOnlyDictionary<string, long> RetryAttemptsByArea);
IReadOnlyDictionary<string, long> EventsBySession);