Merge branch 'grpc-event-batching' — additive site-stream event batching, negotiated, 100ev/25ms window (residual #3 / R2)

docs/requirements/Component-Communication.md

@@ -91,14 +91,14 @@ Delivered 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.m
 
 #### Site-Side gRPC Streaming Components
 
-- **SiteStreamGrpcServer**: gRPC service (`SiteStreamService.SiteStreamServiceBase`) hosted on each site node via Kestrel HTTP/2 on a dedicated port (default 8083). Implements the `SubscribeInstance` RPC. For each subscription, creates a `StreamRelayActor` that subscribes to `SiteStreamManager`, bridges events through a `Channel<SiteStreamEvent>` to the gRPC response stream. Tracks active subscriptions by `correlation_id` — duplicate IDs cancel the old stream. Enforces a max concurrent stream limit (default 100). Rejects streams with `StatusCode.Unavailable` before the actor system is ready.
+- **SiteStreamGrpcServer**: gRPC service (`SiteStreamService.SiteStreamServiceBase`) hosted on each site node via Kestrel HTTP/2 on a dedicated port (default 8083). Implements the `SubscribeInstance` RPC. For each subscription, creates a `StreamRelayActor` that subscribes to `SiteStreamManager`, bridges events through a `Channel<SiteStreamEvent>` to the gRPC response stream — via `SiteStreamEventBatcher`, the per-subscriber coalescing pump that optionally packs several events into one frame (see Event Batching below). Tracks active subscriptions by `correlation_id` — duplicate IDs cancel the old stream. Enforces a max concurrent stream limit (default 100). Rejects streams with `StatusCode.Unavailable` before the actor system is ready.
 - **StreamRelayActor**: Short-lived actor created per gRPC subscription. Receives domain events (`AttributeValueChanged`, `AlarmStateChanged`) from `SiteStreamManager`, converts them to protobuf `SiteStreamEvent` messages, and writes to the `Channel<SiteStreamEvent>` writer. Stopped when the gRPC stream is cancelled or the client disconnects.
 
 #### Central-Side Debug Stream Components
 
 - **DebugStreamService**: Singleton service that manages debug stream sessions. Resolves instance ID to unique name and site, creates and tears down `DebugStreamBridgeActor` instances, and provides a clean API for both Blazor components and the SignalR hub. Injects `SiteStreamGrpcClientFactory` for gRPC stream creation.
 - **DebugStreamBridgeActor**: One per active debug session. Opens a gRPC streaming subscription via `SiteStreamGrpcClient` and receives real-time events via callback. Also receives the initial `DebugViewSnapshot` over gRPC command/control (`SiteCommandService`). Forwards all events to the consumer via callbacks. Handles gRPC stream errors with reconnection logic: tries the other site node endpoint, retries with backoff (max 3 retries), terminates the session if all retries fail.
-- **SiteStreamGrpcClient**: Per-site gRPC client that manages `GrpcChannel` instances and streaming subscriptions. Reads from the gRPC response stream in a background task, converts protobuf messages to domain events, and invokes the `onEvent` callback.
+- **SiteStreamGrpcClient**: Per-site gRPC client that manages `GrpcChannel` instances and streaming subscriptions. Reads from the gRPC response stream in a background task, unpacks each frame via `ForEachEvent` (a plain frame passes through; a batch frame is flattened in order), converts protobuf messages to domain events, and invokes the `onEvent` callback.
 - **SiteStreamGrpcClientFactory**: Caches per-site `SiteStreamGrpcClient` instances. Reads `GrpcNodeAAddress` / `GrpcNodeBAddress` from the `Site` entity (loaded by `CentralCommunicationActor`). Falls back to NodeB if NodeA connection fails. Disposes clients on site removal or address change.
 - **DebugStreamHub**: SignalR hub at `/hubs/debug-stream` for external consumers (e.g., CLI). Authenticates via Basic Auth + LDAP and requires the **Deployment** role. Server-to-client methods: `OnSnapshot`, `OnAttributeChanged`, `OnAlarmChanged`, `OnStreamTerminated`.
 
@@ -113,9 +113,77 @@ The streaming protocol is defined in `sitestream.proto` (`src/ZB.MOM.WW.ScadaBri
   - `IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck)` — ingest receiving surface for the combined cached-call telemetry packet (audit row + `SiteCalls` operational upsert written in one transaction).
   - `PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse)` — central→site **reconciliation pull** for the Audit Log self-heal feed; the site serves `Pending`/`Forwarded` rows from its `ISiteAuditQueue`. **At-least-once (WP2.3):** rows are NOT retired when served. The site retires (`Reconciled`) everything at or before the cursor of the **next** pull, because that cursor is central's only proof of receipt — a fault between the response leaving the site and central committing it re-serves the rows instead of losing them (central dedups on `EventId`). The request carries an additive composite-keyset `after_id` (field 3) mirroring `PullSiteCallsRequest`; with it the read is a strict `(OccurredAtUtc, EventId)` keyset and the retirement is exact, without it the legacy inclusive `>=` read applies and only rows strictly older than the cursor instant are provably received.
   - `PullSiteCalls(PullSiteCallsRequest) returns (PullSiteCallsResponse)` — central→site reconciliation pull for the Site Call Audit (#22) self-heal feed; the site serves operation-tracking rows changed since a cursor from its `IOperationTrackingStore`. A separate RPC from `PullAuditEvents` because the tracking store is the operational source of truth, distinct from the site audit queue.
-- **Messages**: `InstanceStreamRequest` (correlation_id, instance_unique_name), `SiteStreamRequest` (correlation_id only — no instance name; drives the site-wide `SubscribeSite` stream), `SiteStreamEvent` (correlation_id, oneof event: `AttributeValueUpdate`, `AlarmStateUpdate`); `AuditEventDto`/`AuditEventBatch`/`IngestAck` for ingest; `CachedTelemetryPacket`/`CachedTelemetryBatch` (each packet pairing an `AuditEventDto` with a `SiteCallOperationalDto`); `PullAuditEventsRequest`/`PullAuditEventsResponse` and `PullSiteCallsRequest`/`PullSiteCallsResponse` (each request carries `since_utc` + `batch_size` + an optional composite-keyset `after_id`; each response carries `more_available` to signal a saturated batch).
+- **Messages**: `InstanceStreamRequest` (correlation_id, instance_unique_name, `batching_supported`), `SiteStreamRequest` (correlation_id + `batching_supported` — no instance name; drives the site-wide `SubscribeSite` stream), `SiteStreamEvent` (correlation_id, oneof event: `AttributeValueUpdate`, `AlarmStateUpdate`, `SiteStreamEventBatch`); `AuditEventDto`/`AuditEventBatch`/`IngestAck` for ingest; `CachedTelemetryPacket`/`CachedTelemetryBatch` (each packet pairing an `AuditEventDto` with a `SiteCallOperationalDto`); `PullAuditEventsRequest`/`PullAuditEventsResponse` and `PullSiteCallsRequest`/`PullSiteCallsResponse` (each request carries `since_utc` + `batch_size` + an optional composite-keyset `after_id`; each response carries `more_available` to signal a saturated batch).
 - The `oneof event` pattern is extensible — future event types (health metrics, connection state changes) are added as new fields without breaking existing consumers.
-- Proto field numbers are never reused; new RPCs and message fields are appended additively. Old clients ignore unknown `oneof` variants.
+- Proto field numbers are never reused; new RPCs and message fields are appended additively. Old clients ignore unknown `oneof` variants — **but "ignore" means the event vanishes silently**, which is why the batch frame below is gated behind an explicit negotiation flag rather than simply emitted.
+
+##### Event Batching (R2, 2026-08-15)
+
+At target scale a site emits ~37.5k stream events/second, and before R2 every one of them cost its
+own gRPC message. Batching amortises that framing overhead **additively**, with no new RPC:
+
+| Element | Where | Field # |
+|---------|-------|---------|
+| `InstanceStreamRequest.batching_supported` | request | 3 |
+| `SiteStreamRequest.batching_supported` | request | 2 |
+| `SiteStreamEvent.batch` (`SiteStreamEventBatch`) | response `oneof event` | 4 |
+| `SiteStreamEventBatch.events` (`repeated SiteStreamEvent`) | new message | 1 |
+
+**Negotiation is the proto3 default itself.** The server coalesces only when the subscription
+request set `batching_supported = true`; a central built before R2 cannot set it, so it keeps
+receiving exactly one event per frame. This is load-bearing rather than merely tidy: a batch frame
+rides field 4, which an older generated parser skips into unknown fields and reports as
+`EventOneofCase.None` — `ConvertToDomainEvent` then returns null and the **entire batch disappears
+with no error anywhere**. The two skew directions are therefore:
+
+- **old central ↔ new site** — flag absent on the wire, site stays on the per-event path.
+- **new central ↔ old site** — the site ignores the unknown request field and keeps sending
+  per-event frames, which the new client handles as the single-event case of the same unpack path.
+
+**Server side.** `SiteStreamEventBatcher` is a per-subscriber coalescing pump that replaced the
+handler's `await foreach … WriteAsync(evt)` loop, and is byte-for-byte identical to it when the
+size cap is 1. Its latency contract: take the first event, drain whatever is **already queued**
+behind it (free), and linger for `GrpcStreamBatchWindow` only once a backlog has actually been
+observed — so a lone event on a quiet stream is never delayed, and a single-event frame is emitted
+as a plain `attribute_changed`/`alarm_changed` frame rather than a one-element batch. The batch's
+`correlation_id` is stamped once on the enclosing frame and blanked on the packed events. Ordering
+is preserved exactly; per-event `Timestamp`s are untouched (end-to-end latency measurement rides
+them). Buffered events are flushed when the send channel's writer completes; cancellation is not
+flushed (the client is already gone) and surfaces the same `OperationCanceledException` the old
+loop did.
+
+**Client side.** `SiteStreamGrpcClient` sets `batching_supported = true` on both subscriptions and
+unpacks each frame through `ForEachEvent` into the existing per-event pipeline, so
+`SiteAlarmAggregatorActor`, `DebugStreamBridgeActor`, the consumer-keepalive/orphan logic,
+reconnect-on-graceful-completion, generation fencing, the `(siteId, endpoint)` factory key and
+`IsLive` semantics all see no difference. Unpacking is deliberately non-recursive: the server never
+nests batches, and a nested or unknown inner case is skipped rather than followed.
+
+**This does NOT change the site's burst ceiling.** The pump sits strictly *downstream* of
+`StreamRelayActor`'s bounded `DropOldest` send channel and is per subscriber; events evicted by
+that channel are gone before the batcher sees them. The burst ceiling recorded in deferred-work
+register row 31 is set by the **shared publish stage upstream of the BroadcastHub** and by the
+per-subscriber channel capacity, neither of which batching touches.
+
+**Options** (`ScadaBridge:Communication`, validated at startup):
+
+- `GrpcStreamBatchMaxEvents`: 100 (default). Must be `> 0`; **1 disables batching** on this node
+  with no wire change.
+- `GrpcStreamBatchWindow`: 25 ms (default). Must be non-negative and **strictly below 250 ms** —
+  the end-to-end stream-latency budget the target-scale load test asserts a P99 against (measured
+  P99 there: 4.57 ms). `TimeSpan.Zero` means "pack only what is already queued, never wait".
+
+Measured cost of the defaults (`GrpcStreamBatchingIntegrationTests`, worst-case
+trickle-with-backlog workload where the window rather than the size cap closes every batch):
+P50 13.8 ms, P99 25.4 ms, max 25.8 ms — bounded by the window, ~10x inside the threshold. On a
+saturated stream the size cap binds instead: 600 queued events left the site in 6 frames.
+
+**Telemetry**: `scadabridge.site.stream.batch_size` — a histogram of events per frame, tagged
+`stream=instance|site-alarms`, recorded **only** on batching-negotiated subscriptions (on an
+un-negotiated one it would degenerate into a per-event instrument on the hottest path in the
+product). It rides `ScadaBridgeTelemetry.MeterName`, which is already listed in
+`SiteServiceRegistration.ObservedMeters` — `ZbTelemetryOptions.Meters` is an allowlist and an
+unlisted meter exports nothing, silently.
 
 ##### Authentication (preshared key, 2026-07-22)
 
@@ -169,7 +237,7 @@ read the traffic". TLS is follow-on hardening and does not change this design.
 - **Placeholder rows are dropped at the relay**: `is_configured_placeholder` (field 23) is a **Debug View snapshot-only** concept emitted by `InstanceActor.BuildAlarmStatesSnapshot` for quiet bindings — it is never a real alarm transition (its timestamp may be `DateTimeOffset.MinValue`, the Protobuf `Timestamp` lower boundary). `StreamRelayActor.HandleAlarmStateChanged` therefore returns early — **never relaying a placeholder row to the live gRPC stream** — so field 23 is always `false` on the live stream and only ever carries `true` in the snapshot path.
 - **Client-side mapping (`SiteStreamGrpcClient.ConvertToDomainEvent`)**: reconstructs the domain `AlarmStateChanged` from the proto — `Kind` is parsed via `ParseAlarmKind`, the `Condition` is rebuilt with `severity` taken from the existing wire `priority`, and native metadata is repopulated from fields 8–23 (`native_source_canonical_name` → `NativeSourceCanonicalName`, `is_configured_placeholder` → `IsConfiguredPlaceholder`) — so central-side consumers receive the same domain event the site emitted.
 
-> **Regeneration is manual (macOS-only).** `sitestream.proto` is **not** auto-compiled: the `<Protobuf>` include is commented out in the `.csproj`, and the generated C# is **vendored** under `SiteStreamGrpc/`. To regenerate after editing the proto: toggle the `<Protobuf>` include on, build so `Grpc.Tools` regenerates the C#, copy the generated files into `SiteStreamGrpc/`, then re-comment the include. Adding `AlarmStateUpdate` fields 8–23 and the four unary RPCs (`IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`, `PullSiteCalls`) plus their message types followed this process, as did WP2.3's two additive fields — `PullAuditEventsRequest.after_id` (field 3, `sitestream.proto`) and `DebugSnapshotRequestDto.alarms_only` (field 3, `site_command.proto`). The same applies to `site_command.proto`/`SiteCommandGrpc/` and `central_control.proto`/`CentralControlGrpc/`; `docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all]` automates the toggle-build-copy-untoggle and always restores the csproj.
+> **Regeneration is manual (macOS-only).** `sitestream.proto` is **not** auto-compiled: the `<Protobuf>` include is commented out in the `.csproj`, and the generated C# is **vendored** under `SiteStreamGrpc/`. To regenerate after editing the proto: toggle the `<Protobuf>` include on, build so `Grpc.Tools` regenerates the C#, copy the generated files into `SiteStreamGrpc/`, then re-comment the include. Adding `AlarmStateUpdate` fields 8–23 and the four unary RPCs (`IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`, `PullSiteCalls`) plus their message types followed this process, as did WP2.3's two additive fields — `PullAuditEventsRequest.after_id` (field 3, `sitestream.proto`) and `DebugSnapshotRequestDto.alarms_only` (field 3, `site_command.proto`) — and R2's batching additions (`InstanceStreamRequest.batching_supported` field 3, `SiteStreamRequest.batching_supported` field 2, `SiteStreamEvent.batch` field 4 and the new `SiteStreamEventBatch` message). The same applies to `site_command.proto`/`SiteCommandGrpc/` and `central_control.proto`/`CentralControlGrpc/`; `docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all]` automates the toggle-build-copy-untoggle and always restores the csproj.
 
 #### gRPC Connection Keepalive
 
@@ -186,6 +254,8 @@ Keepalive settings are configurable via `CommunicationOptions`:
 - `GrpcKeepAlivePingTimeout`: 10 seconds (default)
 - `GrpcMaxStreamLifetime`: 4 hours (default)
 - `GrpcMaxConcurrentStreams`: 100 (default)
+- `GrpcStreamBatchMaxEvents`: 100 (default) — see Event Batching above
+- `GrpcStreamBatchWindow`: 25 ms (default) — see Event Batching above
 
 ### 6a. Debug Snapshot (Central → Site)
 - **Pattern**: Request/Response (one-shot, no subscription).