test(loadharness)+docs: wire-path stream probes — R2 batching benchmark-proven at target scale

The WP-4 runs measured the stream path only up to the bounded send channel;
the probe deliberately substituted the socket hop with a controllable reader,
which is exactly where the R2 SiteStreamEventBatcher lives — so batching had
never been in any benchmark's hot path. The harness now also attaches per-site
wire probes: a real Kestrel h2c listener hosting the production
SiteStreamGrpcServer, dialed by the production SiteStreamGrpcClient with
batching negotiated, batch sizes read from the production batch_size histogram
via a MeterListener (--wire-probes-per-site, default 10; 0 restores the old
shape). Full-scale run (10x500x75, 20-min window, results doc §8 + raw JSON):
899,807 events in 12,462 frames (mean 72.2 ev/frame, p50=p99=75), wire P99
43.4 ms vs the 250 ms budget — every batch pays the full 25 ms linger since a
75-event burst never hits the 100 cap — zero drops, zero faults, offered load
100.0% of nominal with fan-out doubled. CI smoke Fact now asserts the wire
path carries traffic and observes frames.
This commit is contained in:
Joseph Doherty
2026-08-18 10:32:46 -04:00
parent d3be5f0527
commit e9e44a9ef0
14 changed files with 3400 additions and 8 deletions
@@ -47,6 +47,18 @@ public static class ResultsFormatter
builder.AppendLine(culture, $"events dropped {result.StreamProbeDropped:N0}");
builder.AppendLine();
if (result.WireStream is { } wire)
{
builder.AppendLine("-- wire stream subscribers (real gRPC, R2 batching in path) --");
builder.AppendLine(culture, $"probes {wire.ProbeCount} (Kestrel h2c + SiteStreamGrpcClient, batching negotiated)");
AppendLatency(builder, culture, wire.Latency);
builder.AppendLine(culture, $"events delivered {wire.EventsReceived:N0} in {wire.Batches.Frames:N0} frames");
builder.AppendLine(culture, $"batching mean {wire.Batches.MeanEventsPerFrame:F2} ev/frame p50 {wire.Batches.P50} p95 {wire.Batches.P95} p99 {wire.Batches.P99} max {wire.Batches.Max} ({wire.Batches.BatchedFrames:N0} frames >1 event)");
builder.AppendLine(culture, $"server channel drops {wire.ServerDroppedEvents:N0}");
builder.AppendLine(culture, $"faults / completions {wire.StreamFaults:N0} / {wire.GracefulCompletions:N0}");
builder.AppendLine();
}
builder.AppendLine("-- health report delivery --");
AppendLatency(builder, culture, result.HealthReportLatency);
builder.AppendLine(culture, $"reports ingested {result.HealthReportsDelivered:N0}");