feat(comm): batch site→central stream events over gRPC (R2)
Every AttributeValueChanged/AlarmStateChanged rode its own gRPC message on
SiteStreamService; at target scale that is ~37.5k messages/s/site of pure
framing overhead. Coalesce them, additively, with no new RPC.
Wire (sitestream.proto, regenerated via docker/regen-proto.sh sitestream):
InstanceStreamRequest.batching_supported = 3
SiteStreamRequest.batching_supported = 2
SiteStreamEvent.batch = 4 (new oneof case)
SiteStreamEventBatch { repeated SiteStreamEvent events = 1 }
The proto3 default of batching_supported IS the negotiation, and it is
load-bearing: a batch frame reaches a pre-R2 central as EventOneofCase.None,
whose ConvertToDomainEvent returns null — the whole batch would vanish with
no error anywhere. An old central cannot set the flag so it never receives
one; an old site ignores the unknown request field and keeps sending
per-event frames, which the new client's ForEachEvent handles as the
single-event case. Both skew directions are covered by tests that go through
a real proto serialize/parse round-trip.
Server: SiteStreamEventBatcher, a per-subscriber pump replacing the handler's
await-foreach/WriteAsync loop and byte-identical to it at a cap of 1. It
never delays a lone event — it drains the already-queued backlog for free and
lingers only once a backlog is proven — emits a single-event buffer as a
plain frame, preserves order and per-event Timestamps exactly, and flushes
what is buffered when the send channel's writer completes. It sits DOWNSTREAM
of StreamRelayActor's bounded DropOldest channel, so it changes framing only
and does not move the burst ceiling (deferred register row 31, which lives in
the shared publish stage upstream of the BroadcastHub).
Client: sets the flag on both subscriptions and unpacks in order into the
existing per-event pipeline, so SiteAlarmAggregatorActor,
DebugStreamBridgeActor, consumer-keepalive/orphan logic,
reconnect-on-graceful-completion, generation fencing, the (siteId, endpoint)
factory key and IsLive semantics are untouched.
Options (validated): GrpcStreamBatchMaxEvents 100 (1 disables),
GrpcStreamBatchWindow 25 ms — validated strictly under 250 ms, the load
test's end-to-end P99 threshold. Measured worst case (trickle-with-backlog,
window-bound rather than cap-bound): P50 13.8 ms, P99 25.4 ms, max 25.8 ms;
cap-bound case sent 600 queued events in 6 frames.
Telemetry: histogram scadabridge.site.stream.batch_size tagged by stream
kind, recorded only on negotiated streams (per-event otherwise). It rides
ScadaBridgeTelemetry.MeterName, already in the ObservedMeters allowlist.
Docs: Component-Communication.md gains an Event Batching section; CLAUDE.md's
gRPC streaming bullet records the wire shape and the negotiation rationale.
This commit is contained in:
@@ -91,14 +91,14 @@ Delivered 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.m
|
||||
|
||||
#### Site-Side gRPC Streaming Components
|
||||
|
||||
- **SiteStreamGrpcServer**: gRPC service (`SiteStreamService.SiteStreamServiceBase`) hosted on each site node via Kestrel HTTP/2 on a dedicated port (default 8083). Implements the `SubscribeInstance` RPC. For each subscription, creates a `StreamRelayActor` that subscribes to `SiteStreamManager`, bridges events through a `Channel<SiteStreamEvent>` to the gRPC response stream. Tracks active subscriptions by `correlation_id` — duplicate IDs cancel the old stream. Enforces a max concurrent stream limit (default 100). Rejects streams with `StatusCode.Unavailable` before the actor system is ready.
|
||||
- **SiteStreamGrpcServer**: gRPC service (`SiteStreamService.SiteStreamServiceBase`) hosted on each site node via Kestrel HTTP/2 on a dedicated port (default 8083). Implements the `SubscribeInstance` RPC. For each subscription, creates a `StreamRelayActor` that subscribes to `SiteStreamManager`, bridges events through a `Channel<SiteStreamEvent>` to the gRPC response stream — via `SiteStreamEventBatcher`, the per-subscriber coalescing pump that optionally packs several events into one frame (see Event Batching below). Tracks active subscriptions by `correlation_id` — duplicate IDs cancel the old stream. Enforces a max concurrent stream limit (default 100). Rejects streams with `StatusCode.Unavailable` before the actor system is ready.
|
||||
- **StreamRelayActor**: Short-lived actor created per gRPC subscription. Receives domain events (`AttributeValueChanged`, `AlarmStateChanged`) from `SiteStreamManager`, converts them to protobuf `SiteStreamEvent` messages, and writes to the `Channel<SiteStreamEvent>` writer. Stopped when the gRPC stream is cancelled or the client disconnects.
|
||||
|
||||
#### Central-Side Debug Stream Components
|
||||
|
||||
- **DebugStreamService**: Singleton service that manages debug stream sessions. Resolves instance ID to unique name and site, creates and tears down `DebugStreamBridgeActor` instances, and provides a clean API for both Blazor components and the SignalR hub. Injects `SiteStreamGrpcClientFactory` for gRPC stream creation.
|
||||
- **DebugStreamBridgeActor**: One per active debug session. Opens a gRPC streaming subscription via `SiteStreamGrpcClient` and receives real-time events via callback. Also receives the initial `DebugViewSnapshot` over gRPC command/control (`SiteCommandService`). Forwards all events to the consumer via callbacks. Handles gRPC stream errors with reconnection logic: tries the other site node endpoint, retries with backoff (max 3 retries), terminates the session if all retries fail.
|
||||
- **SiteStreamGrpcClient**: Per-site gRPC client that manages `GrpcChannel` instances and streaming subscriptions. Reads from the gRPC response stream in a background task, converts protobuf messages to domain events, and invokes the `onEvent` callback.
|
||||
- **SiteStreamGrpcClient**: Per-site gRPC client that manages `GrpcChannel` instances and streaming subscriptions. Reads from the gRPC response stream in a background task, unpacks each frame via `ForEachEvent` (a plain frame passes through; a batch frame is flattened in order), converts protobuf messages to domain events, and invokes the `onEvent` callback.
|
||||
- **SiteStreamGrpcClientFactory**: Caches per-site `SiteStreamGrpcClient` instances. Reads `GrpcNodeAAddress` / `GrpcNodeBAddress` from the `Site` entity (loaded by `CentralCommunicationActor`). Falls back to NodeB if NodeA connection fails. Disposes clients on site removal or address change.
|
||||
- **DebugStreamHub**: SignalR hub at `/hubs/debug-stream` for external consumers (e.g., CLI). Authenticates via Basic Auth + LDAP and requires the **Deployment** role. Server-to-client methods: `OnSnapshot`, `OnAttributeChanged`, `OnAlarmChanged`, `OnStreamTerminated`.
|
||||
|
||||
@@ -113,9 +113,77 @@ The streaming protocol is defined in `sitestream.proto` (`src/ZB.MOM.WW.ScadaBri
|
||||
- `IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck)` — ingest receiving surface for the combined cached-call telemetry packet (audit row + `SiteCalls` operational upsert written in one transaction).
|
||||
- `PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse)` — central→site **reconciliation pull** for the Audit Log self-heal feed; the site serves `Pending`/`Forwarded` rows from its `ISiteAuditQueue`. **At-least-once (WP2.3):** rows are NOT retired when served. The site retires (`Reconciled`) everything at or before the cursor of the **next** pull, because that cursor is central's only proof of receipt — a fault between the response leaving the site and central committing it re-serves the rows instead of losing them (central dedups on `EventId`). The request carries an additive composite-keyset `after_id` (field 3) mirroring `PullSiteCallsRequest`; with it the read is a strict `(OccurredAtUtc, EventId)` keyset and the retirement is exact, without it the legacy inclusive `>=` read applies and only rows strictly older than the cursor instant are provably received.
|
||||
- `PullSiteCalls(PullSiteCallsRequest) returns (PullSiteCallsResponse)` — central→site reconciliation pull for the Site Call Audit (#22) self-heal feed; the site serves operation-tracking rows changed since a cursor from its `IOperationTrackingStore`. A separate RPC from `PullAuditEvents` because the tracking store is the operational source of truth, distinct from the site audit queue.
|
||||
- **Messages**: `InstanceStreamRequest` (correlation_id, instance_unique_name), `SiteStreamRequest` (correlation_id only — no instance name; drives the site-wide `SubscribeSite` stream), `SiteStreamEvent` (correlation_id, oneof event: `AttributeValueUpdate`, `AlarmStateUpdate`); `AuditEventDto`/`AuditEventBatch`/`IngestAck` for ingest; `CachedTelemetryPacket`/`CachedTelemetryBatch` (each packet pairing an `AuditEventDto` with a `SiteCallOperationalDto`); `PullAuditEventsRequest`/`PullAuditEventsResponse` and `PullSiteCallsRequest`/`PullSiteCallsResponse` (each request carries `since_utc` + `batch_size` + an optional composite-keyset `after_id`; each response carries `more_available` to signal a saturated batch).
|
||||
- **Messages**: `InstanceStreamRequest` (correlation_id, instance_unique_name, `batching_supported`), `SiteStreamRequest` (correlation_id + `batching_supported` — no instance name; drives the site-wide `SubscribeSite` stream), `SiteStreamEvent` (correlation_id, oneof event: `AttributeValueUpdate`, `AlarmStateUpdate`, `SiteStreamEventBatch`); `AuditEventDto`/`AuditEventBatch`/`IngestAck` for ingest; `CachedTelemetryPacket`/`CachedTelemetryBatch` (each packet pairing an `AuditEventDto` with a `SiteCallOperationalDto`); `PullAuditEventsRequest`/`PullAuditEventsResponse` and `PullSiteCallsRequest`/`PullSiteCallsResponse` (each request carries `since_utc` + `batch_size` + an optional composite-keyset `after_id`; each response carries `more_available` to signal a saturated batch).
|
||||
- The `oneof event` pattern is extensible — future event types (health metrics, connection state changes) are added as new fields without breaking existing consumers.
|
||||
- Proto field numbers are never reused; new RPCs and message fields are appended additively. Old clients ignore unknown `oneof` variants.
|
||||
- Proto field numbers are never reused; new RPCs and message fields are appended additively. Old clients ignore unknown `oneof` variants — **but "ignore" means the event vanishes silently**, which is why the batch frame below is gated behind an explicit negotiation flag rather than simply emitted.
|
||||
|
||||
##### Event Batching (R2, 2026-08-15)
|
||||
|
||||
At target scale a site emits ~37.5k stream events/second, and before R2 every one of them cost its
|
||||
own gRPC message. Batching amortises that framing overhead **additively**, with no new RPC:
|
||||
|
||||
| Element | Where | Field # |
|
||||
|---------|-------|---------|
|
||||
| `InstanceStreamRequest.batching_supported` | request | 3 |
|
||||
| `SiteStreamRequest.batching_supported` | request | 2 |
|
||||
| `SiteStreamEvent.batch` (`SiteStreamEventBatch`) | response `oneof event` | 4 |
|
||||
| `SiteStreamEventBatch.events` (`repeated SiteStreamEvent`) | new message | 1 |
|
||||
|
||||
**Negotiation is the proto3 default itself.** The server coalesces only when the subscription
|
||||
request set `batching_supported = true`; a central built before R2 cannot set it, so it keeps
|
||||
receiving exactly one event per frame. This is load-bearing rather than merely tidy: a batch frame
|
||||
rides field 4, which an older generated parser skips into unknown fields and reports as
|
||||
`EventOneofCase.None` — `ConvertToDomainEvent` then returns null and the **entire batch disappears
|
||||
with no error anywhere**. The two skew directions are therefore:
|
||||
|
||||
- **old central ↔ new site** — flag absent on the wire, site stays on the per-event path.
|
||||
- **new central ↔ old site** — the site ignores the unknown request field and keeps sending
|
||||
per-event frames, which the new client handles as the single-event case of the same unpack path.
|
||||
|
||||
**Server side.** `SiteStreamEventBatcher` is a per-subscriber coalescing pump that replaced the
|
||||
handler's `await foreach … WriteAsync(evt)` loop, and is byte-for-byte identical to it when the
|
||||
size cap is 1. Its latency contract: take the first event, drain whatever is **already queued**
|
||||
behind it (free), and linger for `GrpcStreamBatchWindow` only once a backlog has actually been
|
||||
observed — so a lone event on a quiet stream is never delayed, and a single-event frame is emitted
|
||||
as a plain `attribute_changed`/`alarm_changed` frame rather than a one-element batch. The batch's
|
||||
`correlation_id` is stamped once on the enclosing frame and blanked on the packed events. Ordering
|
||||
is preserved exactly; per-event `Timestamp`s are untouched (end-to-end latency measurement rides
|
||||
them). Buffered events are flushed when the send channel's writer completes; cancellation is not
|
||||
flushed (the client is already gone) and surfaces the same `OperationCanceledException` the old
|
||||
loop did.
|
||||
|
||||
**Client side.** `SiteStreamGrpcClient` sets `batching_supported = true` on both subscriptions and
|
||||
unpacks each frame through `ForEachEvent` into the existing per-event pipeline, so
|
||||
`SiteAlarmAggregatorActor`, `DebugStreamBridgeActor`, the consumer-keepalive/orphan logic,
|
||||
reconnect-on-graceful-completion, generation fencing, the `(siteId, endpoint)` factory key and
|
||||
`IsLive` semantics all see no difference. Unpacking is deliberately non-recursive: the server never
|
||||
nests batches, and a nested or unknown inner case is skipped rather than followed.
|
||||
|
||||
**This does NOT change the site's burst ceiling.** The pump sits strictly *downstream* of
|
||||
`StreamRelayActor`'s bounded `DropOldest` send channel and is per subscriber; events evicted by
|
||||
that channel are gone before the batcher sees them. The burst ceiling recorded in deferred-work
|
||||
register row 31 is set by the **shared publish stage upstream of the BroadcastHub** and by the
|
||||
per-subscriber channel capacity, neither of which batching touches.
|
||||
|
||||
**Options** (`ScadaBridge:Communication`, validated at startup):
|
||||
|
||||
- `GrpcStreamBatchMaxEvents`: 100 (default). Must be `> 0`; **1 disables batching** on this node
|
||||
with no wire change.
|
||||
- `GrpcStreamBatchWindow`: 25 ms (default). Must be non-negative and **strictly below 250 ms** —
|
||||
the end-to-end stream-latency budget the target-scale load test asserts a P99 against (measured
|
||||
P99 there: 4.57 ms). `TimeSpan.Zero` means "pack only what is already queued, never wait".
|
||||
|
||||
Measured cost of the defaults (`GrpcStreamBatchingIntegrationTests`, worst-case
|
||||
trickle-with-backlog workload where the window rather than the size cap closes every batch):
|
||||
P50 13.8 ms, P99 25.4 ms, max 25.8 ms — bounded by the window, ~10x inside the threshold. On a
|
||||
saturated stream the size cap binds instead: 600 queued events left the site in 6 frames.
|
||||
|
||||
**Telemetry**: `scadabridge.site.stream.batch_size` — a histogram of events per frame, tagged
|
||||
`stream=instance|site-alarms`, recorded **only** on batching-negotiated subscriptions (on an
|
||||
un-negotiated one it would degenerate into a per-event instrument on the hottest path in the
|
||||
product). It rides `ScadaBridgeTelemetry.MeterName`, which is already listed in
|
||||
`SiteServiceRegistration.ObservedMeters` — `ZbTelemetryOptions.Meters` is an allowlist and an
|
||||
unlisted meter exports nothing, silently.
|
||||
|
||||
##### Authentication (preshared key, 2026-07-22)
|
||||
|
||||
@@ -169,7 +237,7 @@ read the traffic". TLS is follow-on hardening and does not change this design.
|
||||
- **Placeholder rows are dropped at the relay**: `is_configured_placeholder` (field 23) is a **Debug View snapshot-only** concept emitted by `InstanceActor.BuildAlarmStatesSnapshot` for quiet bindings — it is never a real alarm transition (its timestamp may be `DateTimeOffset.MinValue`, the Protobuf `Timestamp` lower boundary). `StreamRelayActor.HandleAlarmStateChanged` therefore returns early — **never relaying a placeholder row to the live gRPC stream** — so field 23 is always `false` on the live stream and only ever carries `true` in the snapshot path.
|
||||
- **Client-side mapping (`SiteStreamGrpcClient.ConvertToDomainEvent`)**: reconstructs the domain `AlarmStateChanged` from the proto — `Kind` is parsed via `ParseAlarmKind`, the `Condition` is rebuilt with `severity` taken from the existing wire `priority`, and native metadata is repopulated from fields 8–23 (`native_source_canonical_name` → `NativeSourceCanonicalName`, `is_configured_placeholder` → `IsConfiguredPlaceholder`) — so central-side consumers receive the same domain event the site emitted.
|
||||
|
||||
> **Regeneration is manual (macOS-only).** `sitestream.proto` is **not** auto-compiled: the `<Protobuf>` include is commented out in the `.csproj`, and the generated C# is **vendored** under `SiteStreamGrpc/`. To regenerate after editing the proto: toggle the `<Protobuf>` include on, build so `Grpc.Tools` regenerates the C#, copy the generated files into `SiteStreamGrpc/`, then re-comment the include. Adding `AlarmStateUpdate` fields 8–23 and the four unary RPCs (`IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`, `PullSiteCalls`) plus their message types followed this process, as did WP2.3's two additive fields — `PullAuditEventsRequest.after_id` (field 3, `sitestream.proto`) and `DebugSnapshotRequestDto.alarms_only` (field 3, `site_command.proto`). The same applies to `site_command.proto`/`SiteCommandGrpc/` and `central_control.proto`/`CentralControlGrpc/`; `docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all]` automates the toggle-build-copy-untoggle and always restores the csproj.
|
||||
> **Regeneration is manual (macOS-only).** `sitestream.proto` is **not** auto-compiled: the `<Protobuf>` include is commented out in the `.csproj`, and the generated C# is **vendored** under `SiteStreamGrpc/`. To regenerate after editing the proto: toggle the `<Protobuf>` include on, build so `Grpc.Tools` regenerates the C#, copy the generated files into `SiteStreamGrpc/`, then re-comment the include. Adding `AlarmStateUpdate` fields 8–23 and the four unary RPCs (`IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`, `PullSiteCalls`) plus their message types followed this process, as did WP2.3's two additive fields — `PullAuditEventsRequest.after_id` (field 3, `sitestream.proto`) and `DebugSnapshotRequestDto.alarms_only` (field 3, `site_command.proto`) — and R2's batching additions (`InstanceStreamRequest.batching_supported` field 3, `SiteStreamRequest.batching_supported` field 2, `SiteStreamEvent.batch` field 4 and the new `SiteStreamEventBatch` message). The same applies to `site_command.proto`/`SiteCommandGrpc/` and `central_control.proto`/`CentralControlGrpc/`; `docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all]` automates the toggle-build-copy-untoggle and always restores the csproj.
|
||||
|
||||
#### gRPC Connection Keepalive
|
||||
|
||||
@@ -186,6 +254,8 @@ Keepalive settings are configurable via `CommunicationOptions`:
|
||||
- `GrpcKeepAlivePingTimeout`: 10 seconds (default)
|
||||
- `GrpcMaxStreamLifetime`: 4 hours (default)
|
||||
- `GrpcMaxConcurrentStreams`: 100 (default)
|
||||
- `GrpcStreamBatchMaxEvents`: 100 (default) — see Event Batching above
|
||||
- `GrpcStreamBatchWindow`: 25 ms (default) — see Event Batching above
|
||||
|
||||
### 6a. Debug Snapshot (Central → Site)
|
||||
- **Pattern**: Request/Response (one-shot, no subscription).
|
||||
|
||||
Reference in New Issue
Block a user