perf(gateway): trim event/command hot-path allocations (GWC-06/07/15, IPC-05)

Behavior-preserving allocation cuts on the per-event/per-command path:
- GWC-06: StreamEvents send-timing uses Stopwatch.GetTimestamp() +
  GetElapsedTime() instead of a per-event Stopwatch allocation (same measured
  span).
- GWC-07/IPC-05 (event): MapEvent transfers ownership of the inner MxEvent
  instead of .Clone()-ing it. Safe: WorkerEvent is parsed fresh per pipe frame
  with the distributor pump as its single consumer (GWC-01), MapEvent runs once
  before fan-out, and every downstream consumer (subscribers, replay ring)
  only READS the event (WorkerSequence is stamped upstream; verified no
  post-mapping mutation). Comment documents the invariant + restore-clone
  caveat if a second consumer is added.
- IPC-05 (command): CreateCommandEnvelope no longer re-clones; MapCommand
  already isolated the graph from the caller-owned gRPC message.
- GWC-15: grpc_stream_queue.depth converts from a per-event push counter to an
  ObservableGauge summing registered channel sources at scrape time only
  (name/semantics unchanged); removes all per-event .Count/lock work.

Kept every load-bearing isolation clone (MapCommand, Invoke, bulk filters,
MapCommandReply). Server build clean (0 warnings); EventStream/Metrics/
Distributor/Mapper tests 62/62 (incl. formerly-flaky queue-depth tests, now
green under the lazy gauge, + 2 new MapEvent ownership tests). Docs: Metrics.md,
Grpc.md.

Claude-Session: https://claude.ai/code/session_01DMXXvNuPekkkrTEyPNxEkW
This commit is contained in:
Joseph Doherty
2026-07-09 15:12:43 -04:00
parent baae59ce3b
commit 5e2e40a927
9 changed files with 155 additions and 46 deletions
+4
View File
@@ -215,6 +215,10 @@ public WorkerCommand MapCommand(MxCommandRequest request)
}
```
The command clone above is the only isolating copy on the command path. `WorkerClient.CreateCommandEnvelope` no longer re-clones the `WorkerCommand` into the outbound envelope: the mapper's clone already produced a fresh graph owned by the invoke pipeline, and the envelope is built and owned entirely inside `WorkerClient`, so a second copy would be redundant.
`MapEvent` takes the opposite stance from `MapCommand`: it transfers ownership of the inner `MxEvent` rather than cloning it. The enclosing `WorkerEvent` is parsed fresh from a single pipe frame and the `SessionEventDistributor` pump is its single consumer, so nothing else aliases or mutates it. The pump then shares that one `MxEvent` across every subscriber and the replay ring, but that fan-out is read-only, so a single instance is safe. If a second consumer of the `WorkerEvent` is ever introduced, restore the clone in `MapEvent`.
When the worker reply or event payload is missing, the mapper returns a synthetic public message with `ProtocolStatusCode.ProtocolViolation` (for replies) or a sentinel `MxEvent` with `MxEventFamily.Unspecified` (for events). The gateway never relays a partial frame to clients — anything missing is reported as a protocol violation against the worker, not a transport error against the client.
The mapper also exposes static factory methods for every `ProtocolStatusCode` (`Ok`, `InvalidRequest`, `SessionNotFound`, `SessionNotReady`, `WorkerUnavailable`, `Timeout`, `Canceled`, `ProtocolViolation`) so that handlers and tests can produce status payloads without duplicating the enum-to-string mapping.
+11 -10
View File
@@ -73,7 +73,7 @@ Observable gauges are pull-based; the `Meter` invokes the supplied callback when
| `mxgateway.sessions.open` | `_openSessions` | Currently open sessions tracked by `SessionManager`. |
| `mxgateway.workers.running` | `_workersRunning` | Worker clients in a running state. |
| `mxgateway.events.worker_queue.depth` | `_workerEventQueueDepth` | Last reported depth of the worker-side event queue. |
| `mxgateway.events.grpc_stream_queue.depth` | `_grpcEventStreamQueueDepth` | Backlog held by `EventStreamService` for the active gRPC stream consumer. |
| `mxgateway.events.grpc_stream_queue.depth` | `_eventStreamBacklogSources` (summed on demand) | Live backlog buffered across every active `EventStreamService` subscriber, summed from the subscribers' channel `Count` at collection time. |
## Snapshot Shape
@@ -173,14 +173,13 @@ This is the only fault path where the factory itself owns the kill decision; onc
### gRPC event stream service
`Grpc/EventStreamService.cs` records the dashboard/client event-stream backlog and disconnect counters:
`Grpc/EventStreamService.cs` reports the event-stream backlog through a live backlog source and records the disconnect/overflow counters:
```csharp
metrics.AdjustGrpcEventStreamQueueDepth(1);
IDisposable backlogRegistration = metrics.RegisterEventStreamBacklogSource(
() => subscriber.Reader.CanCount ? subscriber.Reader.Count : 0);
...
metrics.AdjustGrpcEventStreamQueueDepth(-1);
...
metrics.AdjustGrpcEventStreamQueueDepth(-remainingDepth);
backlogRegistration.Dispose();
metrics.StreamDisconnected("Detached");
...
metrics.QueueOverflow("grpc-event-stream");
@@ -189,14 +188,16 @@ metrics.Fault(SessionManagerErrorCode.EventQueueOverflow.ToString());
metrics.Fault(WorkerClientErrorCode.WorkerFaulted.ToString());
```
The service tracks per-message enqueues and dequeues, so `AdjustGrpcEventStreamQueueDepth` updates the aggregate stream backlog. The `Math.Max(0, ...)` clamp inside the adjuster prevents a negative depth if the bookkeeping ever drifts.
Rather than pushing a running total on every streamed event (which took the subscriber channel's `Count` lock and the metric lock per event), each subscriber registers its channel as a backlog source for the duration of the stream. The gauge callback and `GetSnapshot` sum those sources' `Count` on demand — only when the metric is scraped or projected — so the streaming hot path does no per-event gauge bookkeeping (GWC-15). Disposing the registration in the `finally` removes the subscriber's contribution, so the gauge reflects only the remaining subscribers and returns to zero once the last stream detaches. Negative source returns are clamped to zero when summed.
`Grpc/MxAccessGatewayService.cs` records gRPC event send latency around each response-stream write:
`Grpc/MxAccessGatewayService.cs` records gRPC event send latency around each response-stream write using the allocation-free timestamp API (no per-event `Stopwatch` object):
```csharp
Stopwatch stopwatch = Stopwatch.StartNew();
long sendStartTimestamp = Stopwatch.GetTimestamp();
await responseStream.WriteAsync(publicEvent).ConfigureAwait(false);
metrics.RecordEventStreamSend(publicEvent.Family.ToString(), stopwatch.Elapsed);
metrics.RecordEventStreamSend(
publicEvent.Family.ToString(),
Stopwatch.GetElapsedTime(sendStartTimestamp));
```
## Dashboard Consumption