feat(comm): batch site→central stream events over gRPC (R2)

Every AttributeValueChanged/AlarmStateChanged rode its own gRPC message on
SiteStreamService; at target scale that is ~37.5k messages/s/site of pure
framing overhead. Coalesce them, additively, with no new RPC.

Wire (sitestream.proto, regenerated via docker/regen-proto.sh sitestream):
  InstanceStreamRequest.batching_supported = 3
  SiteStreamRequest.batching_supported     = 2
  SiteStreamEvent.batch                    = 4  (new oneof case)
  SiteStreamEventBatch { repeated SiteStreamEvent events = 1 }

The proto3 default of batching_supported IS the negotiation, and it is
load-bearing: a batch frame reaches a pre-R2 central as EventOneofCase.None,
whose ConvertToDomainEvent returns null — the whole batch would vanish with
no error anywhere. An old central cannot set the flag so it never receives
one; an old site ignores the unknown request field and keeps sending
per-event frames, which the new client's ForEachEvent handles as the
single-event case. Both skew directions are covered by tests that go through
a real proto serialize/parse round-trip.

Server: SiteStreamEventBatcher, a per-subscriber pump replacing the handler's
await-foreach/WriteAsync loop and byte-identical to it at a cap of 1. It
never delays a lone event — it drains the already-queued backlog for free and
lingers only once a backlog is proven — emits a single-event buffer as a
plain frame, preserves order and per-event Timestamps exactly, and flushes
what is buffered when the send channel's writer completes. It sits DOWNSTREAM
of StreamRelayActor's bounded DropOldest channel, so it changes framing only
and does not move the burst ceiling (deferred register row 31, which lives in
the shared publish stage upstream of the BroadcastHub).

Client: sets the flag on both subscriptions and unpacks in order into the
existing per-event pipeline, so SiteAlarmAggregatorActor,
DebugStreamBridgeActor, consumer-keepalive/orphan logic,
reconnect-on-graceful-completion, generation fencing, the (siteId, endpoint)
factory key and IsLive semantics are untouched.

Options (validated): GrpcStreamBatchMaxEvents 100 (1 disables),
GrpcStreamBatchWindow 25 ms — validated strictly under 250 ms, the load
test's end-to-end P99 threshold. Measured worst case (trickle-with-backlog,
window-bound rather than cap-bound): P50 13.8 ms, P99 25.4 ms, max 25.8 ms;
cap-bound case sent 600 queued events in 6 frames.

Telemetry: histogram scadabridge.site.stream.batch_size tagged by stream
kind, recorded only on negotiated streams (per-event otherwise). It rides
ScadaBridgeTelemetry.MeterName, already in the ObservedMeters allowlist.

Docs: Component-Communication.md gains an Event Batching section; CLAUDE.md's
gRPC streaming bullet records the wire shape and the negotiation rationale.
This commit is contained in:
Joseph Doherty
2026-08-15 03:39:59 -04:00
parent 2b74851f96
commit 9b5cb3dd9d
16 changed files with 2150 additions and 131 deletions
@@ -508,6 +508,125 @@ public class SiteStreamGrpcClientTests
}
}
// ── R2: batch unpacking on the client ───────────────────────────────────────
private static SiteStreamEvent Attr(string value, DateTimeOffset ts) => new()
{
AttributeChanged = new AttributeValueUpdate
{
InstanceUniqueName = "SiteA.Pump01",
AttributePath = "Modules.IO",
AttributeName = "Seq",
Value = value,
Quality = Quality.Good,
Timestamp = Timestamp.FromDateTimeOffset(ts)
}
};
[Fact]
public void ForEachEvent_PlainFrame_IsDeliveredAsIs()
{
// An OLD SITE (or any un-negotiated stream) sends one event per frame. The new
// client's unpack path must pass it straight through — this is the new-central ↔
// old-site skew direction.
var frame = Attr("1", DateTimeOffset.UnixEpoch);
var seen = new List<SiteStreamEvent>();
SiteStreamGrpcClient.ForEachEvent(frame, seen.Add);
Assert.Same(frame, Assert.Single(seen));
}
[Fact]
public void ForEachEvent_BatchFrame_UnpacksInOrderPreservingPerEventTimestamps()
{
// Order and per-event Timestamp fidelity are the two properties the downstream
// consumers (SiteAlarmAggregatorActor, DebugStreamBridgeActor) and the end-to-end
// latency measurement depend on.
var t0 = new DateTimeOffset(2026, 8, 15, 9, 0, 0, TimeSpan.Zero);
var frame = new SiteStreamEvent
{
CorrelationId = "corr-batch",
Batch = new SiteStreamEventBatch
{
Events =
{
Attr("0", t0),
Attr("1", t0.AddMilliseconds(3)),
Attr("2", t0.AddMilliseconds(11))
}
}
};
var seen = new List<SiteStreamEvent>();
SiteStreamGrpcClient.ForEachEvent(frame, seen.Add);
Assert.Equal(["0", "1", "2"], seen.Select(e => e.AttributeChanged.Value));
Assert.Equal(
[t0, t0.AddMilliseconds(3), t0.AddMilliseconds(11)],
seen.Select(e => e.AttributeChanged.Timestamp.ToDateTimeOffset()));
}
[Fact]
public void ForEachEvent_IgnoresNestedAndUnknownInnerCases()
{
// The server never nests a batch inside a batch. A nested (or empty) inner frame
// from a malformed or hostile peer must be skipped, not followed — unpacking is
// deliberately non-recursive so a crafted frame cannot drive unbounded recursion.
var frame = new SiteStreamEvent
{
CorrelationId = "corr-nested",
Batch = new SiteStreamEventBatch
{
Events =
{
Attr("0", DateTimeOffset.UnixEpoch),
new SiteStreamEvent { Batch = new SiteStreamEventBatch { Events = { Attr("hidden", DateTimeOffset.UnixEpoch) } } },
new SiteStreamEvent(),
Attr("1", DateTimeOffset.UnixEpoch)
}
}
};
var seen = new List<SiteStreamEvent>();
SiteStreamGrpcClient.ForEachEvent(frame, seen.Add);
Assert.Equal(["0", "1"], seen.Select(e => e.AttributeChanged.Value));
}
[Fact]
public async Task ConsumeStream_MixedBatchedAndPlainFrames_DeliverEveryEventInOrder()
{
// A reconnect can straddle a site upgrade, so one stream may legitimately carry
// both frame shapes. Driving the real ConsumeStreamAsync with the real unpack
// proves the combination is flat and ordered from the consumer's point of view.
var client = SiteStreamGrpcClient.CreateForTesting();
var cts = new CancellationTokenSource();
var delivered = new List<string>();
void Deliver(SiteStreamEvent e) => delivered.Add(e.AttributeChanged.Value);
await client.ConsumeStreamAsync(
"corr-mixed",
cts,
() => FakeCall(new StubStreamReader(
Attr("0", DateTimeOffset.UnixEpoch),
new SiteStreamEvent
{
CorrelationId = "corr-mixed",
Batch = new SiteStreamEventBatch
{
Events = { Attr("1", DateTimeOffset.UnixEpoch), Attr("2", DateTimeOffset.UnixEpoch) }
}
},
Attr("3", DateTimeOffset.UnixEpoch))),
frame => SiteStreamGrpcClient.ForEachEvent(frame, Deliver),
_ => { },
() => { });
Assert.Equal(["0", "1", "2", "3"], delivered);
}
private static AsyncServerStreamingCall<SiteStreamEvent> FakeCall(StubStreamReader reader) =>
FakeCall(reader, Task.FromResult(new Metadata()));