fix(grpc): count a worker fault once, not once per StreamEvents subscriber
WorkerClient.SetFaulted already records mxgateway.faults; EventStreamService added a WorkerFaulted fault per subscriber, so one fault read as 1+K. The subscriber now records only its WorkerFault disconnect reason. docs/Metrics.md updated.
This commit is contained in:
+3
-3
@@ -124,7 +124,7 @@ The recording call sites describe the code paths that write into each instrument
|
||||
```csharp
|
||||
_metrics.SessionOpened();
|
||||
...
|
||||
_metrics.Fault(SessionManagerErrorCode.OpenFailed.ToString());
|
||||
_metrics.Fault(SessionManagerErrorCode.OpenFailed.ToString()); // not for a cancelled open or GatewayStopping
|
||||
...
|
||||
_metrics.SessionClosed();
|
||||
...
|
||||
@@ -185,10 +185,10 @@ metrics.StreamDisconnected(disconnectReason); // an EventStreamDisconnectReasons
|
||||
...
|
||||
metrics.QueueOverflow("grpc-event-stream");
|
||||
metrics.Fault(SessionManagerErrorCode.EventQueueOverflow.ToString());
|
||||
...
|
||||
metrics.Fault(WorkerClientErrorCode.WorkerFaulted.ToString());
|
||||
```
|
||||
|
||||
A worker fault seen by a subscriber records only the `WorkerFault` disconnect reason, not a fault. `WorkerClient` records the fault once when it faults, and every subscriber of the session observes that same fault, so counting it per subscriber made one worker fault read as 1+K faults for K subscribers.
|
||||
|
||||
Rather than pushing a running total on every streamed event (which took the subscriber channel's `Count` lock and the metric lock per event), each subscriber registers its channel as a backlog source for the duration of the stream. The gauge callback and `GetSnapshot` sum those sources' `Count` on demand — only when the metric is scraped or projected — so the streaming hot path does no per-event gauge bookkeeping (GWC-15). Disposing the registration in the `finally` removes the subscriber's contribution, so the gauge reflects only the remaining subscribers and returns to zero once the last stream detaches. Negative source returns are clamped to zero when summed.
|
||||
|
||||
`Grpc/MxAccessGatewayService.cs` records gRPC event send latency around each response-stream write using the allocation-free timestamp API (no per-event `Stopwatch` object):
|
||||
|
||||
Reference in New Issue
Block a user