fix(grpc): count a worker fault once, not once per StreamEvents subscriber

WorkerClient.SetFaulted already records mxgateway.faults; EventStreamService added a
WorkerFaulted fault per subscriber, so one fault read as 1+K. The subscriber now records
only its WorkerFault disconnect reason. docs/Metrics.md updated.
This commit is contained in:
Joseph Doherty
2026-10-22 08:42:35 -04:00
parent a178d68b60
commit a0b5e91227
3 changed files with 73 additions and 7 deletions
@@ -174,12 +174,13 @@ public sealed class EventStreamService(
{
// The distributor pump completes every subscriber channel with the source
// fault when the worker event stream terminates abnormally; that surfaces
// here. Mirror the original ProduceEventsAsync behavior: fault the
// session and record the metric, then propagate the terminal fault to the
// gRPC client.
// here. Fault the session and propagate the terminal fault to the gRPC
// client. The fault itself is NOT counted here: WorkerClient.SetFaulted
// already recorded it once, and every subscriber of the session sees the
// same fault, so counting it per subscriber inflated mxgateway.faults to
// 1+K. This subscriber's share is the WorkerFault disconnect reason.
disconnectReason = EventStreamDisconnectReasons.WorkerFault;
session.MarkFaulted(workerException.Message);
metrics.Fault(WorkerClientErrorCode.WorkerFaulted.ToString());
throw;
}
catch (SessionManagerException sessionException)