perf(gateway): trim event/command hot-path allocations (GWC-06/07/15, IPC-05)

Behavior-preserving allocation cuts on the per-event/per-command path:
- GWC-06: StreamEvents send-timing uses Stopwatch.GetTimestamp() +
  GetElapsedTime() instead of a per-event Stopwatch allocation (same measured
  span).
- GWC-07/IPC-05 (event): MapEvent transfers ownership of the inner MxEvent
  instead of .Clone()-ing it. Safe: WorkerEvent is parsed fresh per pipe frame
  with the distributor pump as its single consumer (GWC-01), MapEvent runs once
  before fan-out, and every downstream consumer (subscribers, replay ring)
  only READS the event (WorkerSequence is stamped upstream; verified no
  post-mapping mutation). Comment documents the invariant + restore-clone
  caveat if a second consumer is added.
- IPC-05 (command): CreateCommandEnvelope no longer re-clones; MapCommand
  already isolated the graph from the caller-owned gRPC message.
- GWC-15: grpc_stream_queue.depth converts from a per-event push counter to an
  ObservableGauge summing registered channel sources at scrape time only
  (name/semantics unchanged); removes all per-event .Count/lock work.

Kept every load-bearing isolation clone (MapCommand, Invoke, bulk filters,
MapCommandReply). Server build clean (0 warnings); EventStream/Metrics/
Distributor/Mapper tests 62/62 (incl. formerly-flaky queue-depth tests, now
green under the lazy gauge, + 2 new MapEvent ownership tests). Docs: Metrics.md,
Grpc.md.

Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
This commit is contained in:
Joseph Doherty
2026-07-09 15:12:43 -04:00
parent baae59ce3b
commit 5e2e40a927
9 changed files with 155 additions and 46 deletions
@@ -116,11 +116,23 @@ public sealed class EventStreamService(
options.Value.Sessions.MaxEventSubscribersPerSession);
}
int streamQueueDepth = 0;
IAsyncEnumerator<MxEvent> reader = subscriber.Reader
.ReadAllAsync(cancellationToken)
.GetAsyncEnumerator(cancellationToken);
// GWC-15: register this subscriber's channel as a live backlog source instead of
// reconciling the queue-depth gauge on every event. The gauge previously read the
// bounded channel's Count (which takes the channel's internal lock) and adjusted the
// metric under its own lock on every streamed event. Now the metric reads Count only
// when it is scraped (ObservableGauge callback) or projected (GetSnapshot), summing the
// live backlog across every registered subscriber — the same "buffered, not yet
// delivered" aggregate the per-event push reported, but with no per-event lock traffic.
// Disposing the registration in the finally removes this subscriber's contribution, so
// the gauge returns to the other subscribers' backlog (zero when none remain) on
// disconnect. CanCount guards a channel that ever cannot report Count (contributes 0).
IDisposable backlogRegistration = metrics.RegisterEventStreamBacklogSource(
() => subscriber.Reader.CanCount ? subscriber.Reader.Count : 0);
try
{
// Emit order for a resume: the ReplayGap sentinel FIRST (only when events were
@@ -179,32 +191,21 @@ public sealed class EventStreamService(
continue;
}
// Queue-depth gauge tracks events the pump has fanned into this subscriber's
// channel but the client has not yet consumed — the same "buffered, not yet
// delivered" quantity the original per-RPC channel reported. The bounded
// subscriber channel supports counting, so reconcile the gauge to the current
// backlog; falling back to a no-op delta if a channel ever cannot count.
int backlog = subscriber.Reader.CanCount ? subscriber.Reader.Count : streamQueueDepth;
int delta = backlog - streamQueueDepth;
if (delta != 0)
{
streamQueueDepth = backlog;
metrics.AdjustGrpcEventStreamQueueDepth(delta);
}
// The queue-depth gauge is maintained lazily via the backlog registration above
// (GWC-15): the metric reads this subscriber's channel Count only when scraped,
// so there is no per-event gauge bookkeeping on this hot path.
yield return mxEvent;
}
}
finally
{
await reader.DisposeAsync().ConfigureAwait(false);
subscriber.Dispose();
if (streamQueueDepth != 0)
{
metrics.AdjustGrpcEventStreamQueueDepth(-streamQueueDepth);
streamQueueDepth = 0;
}
// Remove this subscriber's live backlog contribution before disposing the lease so
// the gauge stops counting a channel that is about to be completed; after this the
// gauge reflects only the remaining subscribers (zero when none remain).
backlogRegistration.Dispose();
subscriber.Dispose();
metrics.StreamDisconnected("Detached");
}