test(loadharness)+docs: wire-path stream probes — R2 batching benchmark-proven at target scale

The WP-4 runs measured the stream path only up to the bounded send channel;
the probe deliberately substituted the socket hop with a controllable reader,
which is exactly where the R2 SiteStreamEventBatcher lives — so batching had
never been in any benchmark's hot path. The harness now also attaches per-site
wire probes: a real Kestrel h2c listener hosting the production
SiteStreamGrpcServer, dialed by the production SiteStreamGrpcClient with
batching negotiated, batch sizes read from the production batch_size histogram
via a MeterListener (--wire-probes-per-site, default 10; 0 restores the old
shape). Full-scale run (10x500x75, 20-min window, results doc §8 + raw JSON):
899,807 events in 12,462 frames (mean 72.2 ev/frame, p50=p99=75), wire P99
43.4 ms vs the 250 ms budget — every batch pays the full 25 ms linger since a
75-event burst never hits the 100 cap — zero drops, zero faults, offered load
100.0% of nominal with fan-out doubled. CI smoke Fact now asserts the wire
path carries traffic and observes frames.
This commit is contained in:
Joseph Doherty
2026-08-18 10:32:46 -04:00
parent d3be5f0527
commit e9e44a9ef0
14 changed files with 3400 additions and 8 deletions
@@ -68,6 +68,19 @@ public sealed record HarnessConfig
/// </summary>
public int StreamProbesPerSite { get; init; } = 10;
/// <summary>
/// Instances per site additionally watched over a REAL gRPC hop: a per-site
/// Kestrel h2c listener hosts the production <c>SiteStreamGrpcServer</c> and the
/// production <c>SiteStreamGrpcClient</c> dials in with batching negotiated, so
/// the R2 <c>SiteStreamEventBatcher</c>, protobuf serialization and HTTP/2 flow
/// control are inside the measured path (they are NOT for
/// <see cref="StreamProbesPerSite"/>, whose final hop is deliberately an
/// in-process reader). 0 disables the wire path entirely. Like the in-process
/// probes, each wire subscriber's stream graph sees the full site event flow and
/// filters it, so this adds to the fan-out multiplier.
/// </summary>
public int WireProbesPerSite { get; init; } = 10;
/// <summary>Store-and-forward messages enqueued for the drain-rate measurement (register row 50).</summary>
public int StoreAndForwardDrainMessages { get; init; } = 20_000;
@@ -131,6 +144,7 @@ public sealed record HarnessConfig
"health-interval-seconds" => config with { HealthReportInterval = TimeSpan.FromSeconds(double.Parse(value)) },
"debug-probe-interval-seconds" => config with { DebugProbeInterval = TimeSpan.FromSeconds(double.Parse(value)) },
"stream-probes-per-site" => config with { StreamProbesPerSite = int.Parse(value) },
"wire-probes-per-site" => config with { WireProbesPerSite = int.Parse(value) },
"sf-drain-messages" => config with { StoreAndForwardDrainMessages = int.Parse(value) },
"slow-subscriber-events" => config with { SlowSubscriberEvents = int.Parse(value) },
"data-dir" => config with { DataDirectory = value },