test(loadharness)+docs: wire-path stream probes — R2 batching benchmark-proven at target scale

The WP-4 runs measured the stream path only up to the bounded send channel;
the probe deliberately substituted the socket hop with a controllable reader,
which is exactly where the R2 SiteStreamEventBatcher lives — so batching had
never been in any benchmark's hot path. The harness now also attaches per-site
wire probes: a real Kestrel h2c listener hosting the production
SiteStreamGrpcServer, dialed by the production SiteStreamGrpcClient with
batching negotiated, batch sizes read from the production batch_size histogram
via a MeterListener (--wire-probes-per-site, default 10; 0 restores the old
shape). Full-scale run (10x500x75, 20-min window, results doc §8 + raw JSON):
899,807 events in 12,462 frames (mean 72.2 ev/frame, p50=p99=75), wire P99
43.4 ms vs the 250 ms budget — every batch pays the full 25 ms linger since a
75-event burst never hits the 100 cap — zero drops, zero faults, offered load
100.0% of nominal with fan-out doubled. CI smoke Fact now asserts the wire
path carries traffic and observes frames.
This commit is contained in:
Joseph Doherty
2026-08-18 10:32:46 -04:00
parent d3be5f0527
commit e9e44a9ef0
14 changed files with 3400 additions and 8 deletions
@@ -49,6 +49,7 @@ public class TargetScaleHarnessSmokeTests
DebugProbeInterval = TimeSpan.FromSeconds(2),
SubscribeSettleDuration = TimeSpan.FromSeconds(5),
StreamProbesPerSite = 3,
WireProbesPerSite = 2,
StoreAndForwardDrainMessages = 200,
SlowSubscriberEvents = 2_000,
ResultsPath = Path.Combine(Path.GetTempPath(), $"loadharness-smoke-{Guid.NewGuid():N}.json"),
@@ -64,6 +65,18 @@ public class TargetScaleHarnessSmokeTests
Assert.True(result.StreamProbeReceived > 0, "Live stream subscribers received nothing.");
Assert.Equal(0, result.DriverSkippedNoCallback);
// The wire path (real Kestrel + real SiteStreamGrpcClient, R2 batching
// negotiated) carried traffic end to end: events crossed a real HTTP/2
// socket, the batch-size histogram observed real frames, and nothing faulted.
Assert.NotNull(result.WireStream);
Assert.True(result.WireStream!.EventsReceived > 0,
"Wire subscribers received nothing — the Kestrel/gRPC hop did not carry traffic.");
Assert.True(result.WireStream.Latency.Count > 0,
"No wire latency samples — the over-the-wire measurement path is broken.");
Assert.True(result.WireStream.Batches.Frames > 0,
"The batch-size MeterListener observed no frames — the R2 histogram path is broken.");
Assert.Equal(0, result.WireStream.StreamFaults);
// Observability held up.
Assert.Equal(config.Sites, result.SitesTrackedByAggregator);
Assert.True(result.DebugSnapshotsCompleted > 0, "No debug snapshot completed.");