diff --git a/CLAUDE.md b/CLAUDE.md index 02b4edb8..1894dd60 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -99,7 +99,7 @@ spec for each is `docs/requirements/Component-.md`, and `README.md` carrie - **`ActiveNodeEvaluator.SelfIsOldestUp` is THE single definition of "active node"** (`Communication/ClusterState/ActiveNodeEvaluator.cs`) — the **oldest Up member** in a role scope, and explicitly **never `cluster.State.Leader`**: leadership (lowest address) is an Akka-internal concept that diverges from singleton placement permanently once the original first node restarts and rejoins, and both sides claim it during a partition. The equivalence *oldest-Up == where `ClusterSingletonManager` places singletons* **is** the design. `ClusterActivityEvaluator.SelfIsOldest`, the S&F delivery gate, `/health/active` and the heartbeat `IsActive` stamp all delegate here. - Site nodes carry **two Akka roles**: the base `Site` plus a site-specific `site-{SiteId}` (`AkkaHostedService.BuildRoles`). Singletons scope to the **site-specific** role. - **The gRPC boundary is authenticated (PSK) as of 2026-07-22; Akka remoting still is not, and nothing is encrypted.** Akka remoting sets no `enable-ssl`, no secure cookie, no `trusted-selection-paths` — so intra-cluster Akka remoting remains open to anyone who can reach the remoting port, and that boundary still assumes a trusted network. The gRPC listener stays **h2c**, but `SiteStreamService` is no longer open: `ControlPlaneAuthInterceptor` (`Host/ControlPlaneAuthInterceptor.cs`) gates `/sitestream.SiteStreamService/` — including the `PullAuditEvents`/`PullSiteCalls` RPCs that return audit rows — against a **per-site preshared key**, fail-closed, constant-time compared, alongside the separate `LocalDbSyncAuthInterceptor` on `/localdb_sync.v1.LocalDbSync/` with its own separate key. **Site side:** `ScadaBridge:Communication:GrpcPsk`, in production `${secret:SB-GRPC-PSK-}`, and **`StartupValidator` refuses to boot a site node without it** (an unset key would leave the node healthy-looking but serving nothing). **Central side:** `SitePskProvider` resolves `SB-GRPC-PSK-{siteId}` from the secrets store at channel-build time (sites are added at runtime, so no boot-time expansion is possible), with `ScadaBridge:Communication:SitePsks:{siteId}` as an override for hosts running without a master key — the docker rig uses the latter. One key per site, never fleet-wide. A bearer token over h2c is readable and replayable on-path; TLS is the follow-on hardening and needs no change to this design. Introduced by Phase 0 of the ClusterClient→gRPC migration (`docs/plans/2026-07-22-clusterclient-to-grpc-plan.md`). -- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Proto evolution is **additive only** and field numbers are never reused (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `` include commented out — regeneration is a manual toggle-build-copy-untoggle. **The client reconnects on graceful (OK-status) stream completion, not just on fault (arch-review remediation, commit `34a3f4bb`)** — Kestrel's `MaxStreamLifetime`/`Grpc.AspNetCore.Server` max-connection-age periodically ends a healthy stream with a normal completion, which the client used to treat as terminal (no reconnect attempt), silently killing a site's live feed until the next process restart (observed up to ~4h on the rig); live-probed at a 2-minute forced lifetime, reconnect lands within one reconcile tick and `IsLive` reflects the gap in between. +- gRPC streaming channel — **note the direction is inverted from the data flow**: data moves site→central, but each **site node hosts the gRPC server** (`SiteStreamGrpcServer`, Kestrel h2c, port 8083, mapped **only in the Site branch** of `Program.cs`) and **central is the client**, dialling in. Central creates per-site `SiteStreamGrpcClient` via `SiteStreamGrpcClientFactory`, keyed **`(siteId, endpoint)`** — the key was widened from site-only to fix an arch-review High where one session's NodeA→NodeB flip disposed a channel another session was still using. Proto evolution is **additive only** and field numbers are never reused (`AlarmStateUpdate` grew 7→23 fields for the native-alarm mirror, then 24 for `AckTime`). Generated C# is **vendored** under `Communication/SiteStreamGrpc/` with the `` include commented out — regeneration is a manual toggle-build-copy-untoggle, automated by `docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all]` (it always restores the csproj; an active `` item must never be committed — it breaks the Docker build). **Stream events are BATCHED as of R2 (2026-08-15)** — a site emitting ~37.5k events/s used to pay one gRPC message per event. Additive wire shape, no new RPC: `batching_supported` on the request (`InstanceStreamRequest` field 3 / `SiteStreamRequest` field 2) plus a `SiteStreamEvent.batch` oneof case (**field 4**) carrying a new `SiteStreamEventBatch { repeated SiteStreamEvent events = 1 }`. **The proto3 default of that flag IS the negotiation, and it is load-bearing** — a batch frame reaches a pre-R2 central as `EventOneofCase.None`, whose `ConvertToDomainEvent` returns null, so the whole batch would vanish with no error anywhere; an old central cannot set the flag, so it never gets one, and an old site ignores the unknown request field and keeps sending per-event frames the new client's `ForEachEvent` unpack handles as the single-event case. Server side is `SiteStreamEventBatcher`, a per-subscriber pump **downstream of `StreamRelayActor`'s bounded DropOldest channel** — so it changes framing only and **does NOT move the burst ceiling** (deferred register row 31; that ceiling lives in the shared publish stage upstream of the BroadcastHub). It never delays a lone event: it drains the already-queued backlog for free and lingers only once a backlog is proven, then emits a single-event buffer as a plain frame. `GrpcStreamBatchMaxEvents` (100, `1` disables) and `GrpcStreamBatchWindow` (25 ms, validated **strictly under 250 ms** = the load test's end-to-end P99 threshold); measured worst case P99 25.4 ms. Histogram `scadabridge.site.stream.batch_size`, recorded only on negotiated streams (`ScadaBridgeTelemetry.MeterName` is already in the `ObservedMeters` allowlist). **The client reconnects on graceful (OK-status) stream completion, not just on fault (arch-review remediation, commit `34a3f4bb`)** — Kestrel's `MaxStreamLifetime`/`Grpc.AspNetCore.Server` max-connection-age periodically ends a healthy stream with a normal completion, which the client used to treat as terminal (no reconnect attempt), silently killing a site's live feed until the next process restart (observed up to ~4h on the rig); live-probed at a 2-minute forced lifetime, reconnect lands within one reconcile tick and `IsLive` reflects the gap in between. - Native alarms are a **read-only** mirror of OPC UA Alarms & Conditions and MxAccess Gateway alarms — **no ack-back, no central tables**; state lives in the site's `native_alarm_state`, survives failover, and is cleared on redeploy/undeploy (mirrors static overrides). Central's per-site live alarm cache (`ISiteAlarmLiveCache`) is **transient in-memory only** — there is deliberately no persisted central alarm store, so the 15s poll remains the NotReporting authority behind the live stream. See `Component-DataConnectionLayer.md` / `Component-CentralUI.md` for the model and the authoring surface. - **`AckTime` mirror enrichment + the `Alarms` script accessor (MES alarm-status API Phase 1, 2026-08-01).** `AlarmStateChanged` carries an additive `AckTime` (`DateTimeOffset?`), mirrored on the vendored `AlarmStateUpdate` proto as **field 24** and persisted inside `native_alarm_state`'s `metadata_json` — deliberately NOT a new column, because that table is `RegisterReplicated` and LocalDb builds its CDC triggers from the column list at registration time. Set only while a condition is active AND acknowledged (so it is null while unacked and cleared on re-raise); the DCL stamps the source's own ack instant for OPC UA (new SelectClause **index 18** = `AckedState/TransitionTime`) and its observation time of the ack transition for MxGateway, which supplies none. Site `Call` scripts read alarms via the new **`Alarms.CurrentAsync()`** accessor (`ScriptRuntimeContext` + `ScriptGlobals`, local Ask on `GetAlarmSnapshotRequest`, returns `Commons.Types.Scripts.ScriptAlarm`), mirrored on `ScriptCompileSurface` AND the Central UI `SandboxScriptHost` editor surface. The trust model needed no change — it is a deny-list over API roots, not an allow-list of context members. Plan: `docs/plans/2026-06-30-mes-alarm-status-api.md` (Phases 2–4 are deployed config, not repo). - OPC UA cert trust is **site-local and not persisted centrally** (follow-up): the verify-endpoint probe captures an untrusted server cert but **NEVER trusts it**, and DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to **BOTH** site nodes — `CertStoreActor` runs on every site node, not as a singleton, so PKI stores stay consistent across failover. diff --git a/docs/requirements/Component-Communication.md b/docs/requirements/Component-Communication.md index 77147d52..45512df9 100644 --- a/docs/requirements/Component-Communication.md +++ b/docs/requirements/Component-Communication.md @@ -91,14 +91,14 @@ Delivered 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.m #### Site-Side gRPC Streaming Components -- **SiteStreamGrpcServer**: gRPC service (`SiteStreamService.SiteStreamServiceBase`) hosted on each site node via Kestrel HTTP/2 on a dedicated port (default 8083). Implements the `SubscribeInstance` RPC. For each subscription, creates a `StreamRelayActor` that subscribes to `SiteStreamManager`, bridges events through a `Channel` to the gRPC response stream. Tracks active subscriptions by `correlation_id` — duplicate IDs cancel the old stream. Enforces a max concurrent stream limit (default 100). Rejects streams with `StatusCode.Unavailable` before the actor system is ready. +- **SiteStreamGrpcServer**: gRPC service (`SiteStreamService.SiteStreamServiceBase`) hosted on each site node via Kestrel HTTP/2 on a dedicated port (default 8083). Implements the `SubscribeInstance` RPC. For each subscription, creates a `StreamRelayActor` that subscribes to `SiteStreamManager`, bridges events through a `Channel` to the gRPC response stream — via `SiteStreamEventBatcher`, the per-subscriber coalescing pump that optionally packs several events into one frame (see Event Batching below). Tracks active subscriptions by `correlation_id` — duplicate IDs cancel the old stream. Enforces a max concurrent stream limit (default 100). Rejects streams with `StatusCode.Unavailable` before the actor system is ready. - **StreamRelayActor**: Short-lived actor created per gRPC subscription. Receives domain events (`AttributeValueChanged`, `AlarmStateChanged`) from `SiteStreamManager`, converts them to protobuf `SiteStreamEvent` messages, and writes to the `Channel` writer. Stopped when the gRPC stream is cancelled or the client disconnects. #### Central-Side Debug Stream Components - **DebugStreamService**: Singleton service that manages debug stream sessions. Resolves instance ID to unique name and site, creates and tears down `DebugStreamBridgeActor` instances, and provides a clean API for both Blazor components and the SignalR hub. Injects `SiteStreamGrpcClientFactory` for gRPC stream creation. - **DebugStreamBridgeActor**: One per active debug session. Opens a gRPC streaming subscription via `SiteStreamGrpcClient` and receives real-time events via callback. Also receives the initial `DebugViewSnapshot` over gRPC command/control (`SiteCommandService`). Forwards all events to the consumer via callbacks. Handles gRPC stream errors with reconnection logic: tries the other site node endpoint, retries with backoff (max 3 retries), terminates the session if all retries fail. -- **SiteStreamGrpcClient**: Per-site gRPC client that manages `GrpcChannel` instances and streaming subscriptions. Reads from the gRPC response stream in a background task, converts protobuf messages to domain events, and invokes the `onEvent` callback. +- **SiteStreamGrpcClient**: Per-site gRPC client that manages `GrpcChannel` instances and streaming subscriptions. Reads from the gRPC response stream in a background task, unpacks each frame via `ForEachEvent` (a plain frame passes through; a batch frame is flattened in order), converts protobuf messages to domain events, and invokes the `onEvent` callback. - **SiteStreamGrpcClientFactory**: Caches per-site `SiteStreamGrpcClient` instances. Reads `GrpcNodeAAddress` / `GrpcNodeBAddress` from the `Site` entity (loaded by `CentralCommunicationActor`). Falls back to NodeB if NodeA connection fails. Disposes clients on site removal or address change. - **DebugStreamHub**: SignalR hub at `/hubs/debug-stream` for external consumers (e.g., CLI). Authenticates via Basic Auth + LDAP and requires the **Deployment** role. Server-to-client methods: `OnSnapshot`, `OnAttributeChanged`, `OnAlarmChanged`, `OnStreamTerminated`. @@ -113,9 +113,77 @@ The streaming protocol is defined in `sitestream.proto` (`src/ZB.MOM.WW.ScadaBri - `IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck)` — ingest receiving surface for the combined cached-call telemetry packet (audit row + `SiteCalls` operational upsert written in one transaction). - `PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse)` — central→site **reconciliation pull** for the Audit Log self-heal feed; the site serves `Pending`/`Forwarded` rows from its `ISiteAuditQueue`. **At-least-once (WP2.3):** rows are NOT retired when served. The site retires (`Reconciled`) everything at or before the cursor of the **next** pull, because that cursor is central's only proof of receipt — a fault between the response leaving the site and central committing it re-serves the rows instead of losing them (central dedups on `EventId`). The request carries an additive composite-keyset `after_id` (field 3) mirroring `PullSiteCallsRequest`; with it the read is a strict `(OccurredAtUtc, EventId)` keyset and the retirement is exact, without it the legacy inclusive `>=` read applies and only rows strictly older than the cursor instant are provably received. - `PullSiteCalls(PullSiteCallsRequest) returns (PullSiteCallsResponse)` — central→site reconciliation pull for the Site Call Audit (#22) self-heal feed; the site serves operation-tracking rows changed since a cursor from its `IOperationTrackingStore`. A separate RPC from `PullAuditEvents` because the tracking store is the operational source of truth, distinct from the site audit queue. -- **Messages**: `InstanceStreamRequest` (correlation_id, instance_unique_name), `SiteStreamRequest` (correlation_id only — no instance name; drives the site-wide `SubscribeSite` stream), `SiteStreamEvent` (correlation_id, oneof event: `AttributeValueUpdate`, `AlarmStateUpdate`); `AuditEventDto`/`AuditEventBatch`/`IngestAck` for ingest; `CachedTelemetryPacket`/`CachedTelemetryBatch` (each packet pairing an `AuditEventDto` with a `SiteCallOperationalDto`); `PullAuditEventsRequest`/`PullAuditEventsResponse` and `PullSiteCallsRequest`/`PullSiteCallsResponse` (each request carries `since_utc` + `batch_size` + an optional composite-keyset `after_id`; each response carries `more_available` to signal a saturated batch). +- **Messages**: `InstanceStreamRequest` (correlation_id, instance_unique_name, `batching_supported`), `SiteStreamRequest` (correlation_id + `batching_supported` — no instance name; drives the site-wide `SubscribeSite` stream), `SiteStreamEvent` (correlation_id, oneof event: `AttributeValueUpdate`, `AlarmStateUpdate`, `SiteStreamEventBatch`); `AuditEventDto`/`AuditEventBatch`/`IngestAck` for ingest; `CachedTelemetryPacket`/`CachedTelemetryBatch` (each packet pairing an `AuditEventDto` with a `SiteCallOperationalDto`); `PullAuditEventsRequest`/`PullAuditEventsResponse` and `PullSiteCallsRequest`/`PullSiteCallsResponse` (each request carries `since_utc` + `batch_size` + an optional composite-keyset `after_id`; each response carries `more_available` to signal a saturated batch). - The `oneof event` pattern is extensible — future event types (health metrics, connection state changes) are added as new fields without breaking existing consumers. -- Proto field numbers are never reused; new RPCs and message fields are appended additively. Old clients ignore unknown `oneof` variants. +- Proto field numbers are never reused; new RPCs and message fields are appended additively. Old clients ignore unknown `oneof` variants — **but "ignore" means the event vanishes silently**, which is why the batch frame below is gated behind an explicit negotiation flag rather than simply emitted. + +##### Event Batching (R2, 2026-08-15) + +At target scale a site emits ~37.5k stream events/second, and before R2 every one of them cost its +own gRPC message. Batching amortises that framing overhead **additively**, with no new RPC: + +| Element | Where | Field # | +|---------|-------|---------| +| `InstanceStreamRequest.batching_supported` | request | 3 | +| `SiteStreamRequest.batching_supported` | request | 2 | +| `SiteStreamEvent.batch` (`SiteStreamEventBatch`) | response `oneof event` | 4 | +| `SiteStreamEventBatch.events` (`repeated SiteStreamEvent`) | new message | 1 | + +**Negotiation is the proto3 default itself.** The server coalesces only when the subscription +request set `batching_supported = true`; a central built before R2 cannot set it, so it keeps +receiving exactly one event per frame. This is load-bearing rather than merely tidy: a batch frame +rides field 4, which an older generated parser skips into unknown fields and reports as +`EventOneofCase.None` — `ConvertToDomainEvent` then returns null and the **entire batch disappears +with no error anywhere**. The two skew directions are therefore: + +- **old central ↔ new site** — flag absent on the wire, site stays on the per-event path. +- **new central ↔ old site** — the site ignores the unknown request field and keeps sending + per-event frames, which the new client handles as the single-event case of the same unpack path. + +**Server side.** `SiteStreamEventBatcher` is a per-subscriber coalescing pump that replaced the +handler's `await foreach … WriteAsync(evt)` loop, and is byte-for-byte identical to it when the +size cap is 1. Its latency contract: take the first event, drain whatever is **already queued** +behind it (free), and linger for `GrpcStreamBatchWindow` only once a backlog has actually been +observed — so a lone event on a quiet stream is never delayed, and a single-event frame is emitted +as a plain `attribute_changed`/`alarm_changed` frame rather than a one-element batch. The batch's +`correlation_id` is stamped once on the enclosing frame and blanked on the packed events. Ordering +is preserved exactly; per-event `Timestamp`s are untouched (end-to-end latency measurement rides +them). Buffered events are flushed when the send channel's writer completes; cancellation is not +flushed (the client is already gone) and surfaces the same `OperationCanceledException` the old +loop did. + +**Client side.** `SiteStreamGrpcClient` sets `batching_supported = true` on both subscriptions and +unpacks each frame through `ForEachEvent` into the existing per-event pipeline, so +`SiteAlarmAggregatorActor`, `DebugStreamBridgeActor`, the consumer-keepalive/orphan logic, +reconnect-on-graceful-completion, generation fencing, the `(siteId, endpoint)` factory key and +`IsLive` semantics all see no difference. Unpacking is deliberately non-recursive: the server never +nests batches, and a nested or unknown inner case is skipped rather than followed. + +**This does NOT change the site's burst ceiling.** The pump sits strictly *downstream* of +`StreamRelayActor`'s bounded `DropOldest` send channel and is per subscriber; events evicted by +that channel are gone before the batcher sees them. The burst ceiling recorded in deferred-work +register row 31 is set by the **shared publish stage upstream of the BroadcastHub** and by the +per-subscriber channel capacity, neither of which batching touches. + +**Options** (`ScadaBridge:Communication`, validated at startup): + +- `GrpcStreamBatchMaxEvents`: 100 (default). Must be `> 0`; **1 disables batching** on this node + with no wire change. +- `GrpcStreamBatchWindow`: 25 ms (default). Must be non-negative and **strictly below 250 ms** — + the end-to-end stream-latency budget the target-scale load test asserts a P99 against (measured + P99 there: 4.57 ms). `TimeSpan.Zero` means "pack only what is already queued, never wait". + +Measured cost of the defaults (`GrpcStreamBatchingIntegrationTests`, worst-case +trickle-with-backlog workload where the window rather than the size cap closes every batch): +P50 13.8 ms, P99 25.4 ms, max 25.8 ms — bounded by the window, ~10x inside the threshold. On a +saturated stream the size cap binds instead: 600 queued events left the site in 6 frames. + +**Telemetry**: `scadabridge.site.stream.batch_size` — a histogram of events per frame, tagged +`stream=instance|site-alarms`, recorded **only** on batching-negotiated subscriptions (on an +un-negotiated one it would degenerate into a per-event instrument on the hottest path in the +product). It rides `ScadaBridgeTelemetry.MeterName`, which is already listed in +`SiteServiceRegistration.ObservedMeters` — `ZbTelemetryOptions.Meters` is an allowlist and an +unlisted meter exports nothing, silently. ##### Authentication (preshared key, 2026-07-22) @@ -169,7 +237,7 @@ read the traffic". TLS is follow-on hardening and does not change this design. - **Placeholder rows are dropped at the relay**: `is_configured_placeholder` (field 23) is a **Debug View snapshot-only** concept emitted by `InstanceActor.BuildAlarmStatesSnapshot` for quiet bindings — it is never a real alarm transition (its timestamp may be `DateTimeOffset.MinValue`, the Protobuf `Timestamp` lower boundary). `StreamRelayActor.HandleAlarmStateChanged` therefore returns early — **never relaying a placeholder row to the live gRPC stream** — so field 23 is always `false` on the live stream and only ever carries `true` in the snapshot path. - **Client-side mapping (`SiteStreamGrpcClient.ConvertToDomainEvent`)**: reconstructs the domain `AlarmStateChanged` from the proto — `Kind` is parsed via `ParseAlarmKind`, the `Condition` is rebuilt with `severity` taken from the existing wire `priority`, and native metadata is repopulated from fields 8–23 (`native_source_canonical_name` → `NativeSourceCanonicalName`, `is_configured_placeholder` → `IsConfiguredPlaceholder`) — so central-side consumers receive the same domain event the site emitted. -> **Regeneration is manual (macOS-only).** `sitestream.proto` is **not** auto-compiled: the `` include is commented out in the `.csproj`, and the generated C# is **vendored** under `SiteStreamGrpc/`. To regenerate after editing the proto: toggle the `` include on, build so `Grpc.Tools` regenerates the C#, copy the generated files into `SiteStreamGrpc/`, then re-comment the include. Adding `AlarmStateUpdate` fields 8–23 and the four unary RPCs (`IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`, `PullSiteCalls`) plus their message types followed this process, as did WP2.3's two additive fields — `PullAuditEventsRequest.after_id` (field 3, `sitestream.proto`) and `DebugSnapshotRequestDto.alarms_only` (field 3, `site_command.proto`). The same applies to `site_command.proto`/`SiteCommandGrpc/` and `central_control.proto`/`CentralControlGrpc/`; `docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all]` automates the toggle-build-copy-untoggle and always restores the csproj. +> **Regeneration is manual (macOS-only).** `sitestream.proto` is **not** auto-compiled: the `` include is commented out in the `.csproj`, and the generated C# is **vendored** under `SiteStreamGrpc/`. To regenerate after editing the proto: toggle the `` include on, build so `Grpc.Tools` regenerates the C#, copy the generated files into `SiteStreamGrpc/`, then re-comment the include. Adding `AlarmStateUpdate` fields 8–23 and the four unary RPCs (`IngestAuditEvents`, `IngestCachedTelemetry`, `PullAuditEvents`, `PullSiteCalls`) plus their message types followed this process, as did WP2.3's two additive fields — `PullAuditEventsRequest.after_id` (field 3, `sitestream.proto`) and `DebugSnapshotRequestDto.alarms_only` (field 3, `site_command.proto`) — and R2's batching additions (`InstanceStreamRequest.batching_supported` field 3, `SiteStreamRequest.batching_supported` field 2, `SiteStreamEvent.batch` field 4 and the new `SiteStreamEventBatch` message). The same applies to `site_command.proto`/`SiteCommandGrpc/` and `central_control.proto`/`CentralControlGrpc/`; `docker/regen-proto.sh [sitestream|centralcontrol|sitecommand|all]` automates the toggle-build-copy-untoggle and always restores the csproj. #### gRPC Connection Keepalive @@ -186,6 +254,8 @@ Keepalive settings are configurable via `CommunicationOptions`: - `GrpcKeepAlivePingTimeout`: 10 seconds (default) - `GrpcMaxStreamLifetime`: 4 hours (default) - `GrpcMaxConcurrentStreams`: 100 (default) +- `GrpcStreamBatchMaxEvents`: 100 (default) — see Event Batching above +- `GrpcStreamBatchWindow`: 25 ms (default) — see Event Batching above ### 6a. Debug Snapshot (Central → Site) - **Pattern**: Request/Response (one-shot, no subscription). diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs index aa5a60ae..0ffbf176 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs @@ -93,6 +93,20 @@ public static class ScadaBridgeTelemetry Meter.CreateCounter("scadabridge.site.stream.events_dropped", unit: "1", description: "Events evicted from a site gRPC stream's bounded send channel, tagged by stream kind."); + // ---------------- Histograms ---------------- + + /// + /// Distribution of how many events each site→central stream frame carried (R2 — gRPC + /// event batching), tagged by stream kind. Recorded ONLY for subscriptions that + /// negotiated batching, so the series' very existence says "this central speaks the + /// batched wire". A distribution pinned at 1 means the coalescing window never sees a + /// backlog (the site is quiet, or the window is too small to be earning anything); + /// mass at the size cap means the cap, not the window, is the binding constraint. + /// + private static readonly Histogram _siteStreamBatchSize = + Meter.CreateHistogram("scadabridge.site.stream.batch_size", unit: "1", + description: "Events per site gRPC stream frame (1 = unbatched frame), tagged by stream kind."); + // ---------------- Observable gauges ---------------- /// Current count of open site connections, mutated via . @@ -178,6 +192,16 @@ public static class ScadaBridgeTelemetry public static void RecordSiteStreamEventDropped(string streamKind) => _siteStreamEventDrops.Add(1, new KeyValuePair("stream", streamKind)); + /// + /// Records how many events one site gRPC stream frame carried. Called once per emitted + /// frame on a batching-negotiated subscription (never on an un-negotiated one, where it + /// would degenerate into a per-event instrument on the hottest path in the product). + /// + /// Stream kind tag (instance or site-alarms). + /// Events packed into the frame; 1 for a plain unbatched frame. + public static void RecordSiteStreamBatchSize(string streamKind, int events) => + _siteStreamBatchSize.Record(events, new KeyValuePair("stream", streamKind)); + /// /// Registers the provider the StoreAndForward queue-depth gauge reads on each observation. /// A later task supplies a provider that reads the real StoreAndForward depth. A null diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs index b8ff2a20..dfa3d01a 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs @@ -152,6 +152,26 @@ public class CommunicationOptions /// public int GrpcSiteAlarmStreamChannelCapacity { get; set; } = 20_000; + /// + /// Maximum events coalesced into one site→central stream frame (R2 — gRPC event + /// batching). At target scale a site emits ~37.5k events/s and every one of them used to + /// cost its own gRPC message; batching amortises that framing overhead. Set to 1 to + /// disable batching on this node without a wire change (every frame then carries exactly + /// one event, which is also what an un-negotiated subscription gets). + /// + public int GrpcStreamBatchMaxEvents { get; set; } = 100; + + /// + /// Maximum time the site lingers accumulating a stream batch once a backlog has been + /// observed. Deliberately small: the target-scale load test measures end-to-end event + /// latency against a 250 ms P99 threshold (measured P99 4.57 ms), and this window is the + /// only latency batching can add — so it is validated strictly below that threshold. + /// A lone event on a quiet stream is never delayed by it (see + /// SiteStreamEventBatcher); the window applies only after a backlog is proven. + /// means "pack only what is already queued, never wait". + /// + public TimeSpan GrpcStreamBatchWindow { get; set; } = TimeSpan.FromMilliseconds(25); + /// Akka.Remote transport heartbeat interval. public TimeSpan TransportHeartbeatInterval { get; set; } = TimeSpan.FromSeconds(5); diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptionsValidator.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptionsValidator.cs index 7bee6e27..35174942 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptionsValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptionsValidator.cs @@ -12,6 +12,14 @@ namespace ZB.MOM.WW.ScadaBridge.Communication; /// public sealed class CommunicationOptionsValidator : OptionsValidatorBase { + /// + /// Exclusive upper bound on — + /// the end-to-end site→central event latency budget the target-scale load test asserts + /// a P99 against. The coalescing window is the only latency batching introduces, so it + /// must stay strictly inside that budget rather than consuming it whole. + /// + internal static readonly TimeSpan StreamBatchWindowCeiling = TimeSpan.FromMilliseconds(250); + /// protected override void Validate(ValidationBuilder builder, CommunicationOptions options) { @@ -75,6 +83,23 @@ public sealed class CommunicationOptionsValidator : OptionsValidatorBase 0, $"ScadaBridge:Communication:GrpcMaxConcurrentStreams must be positive (was {options.GrpcMaxConcurrentStreams})."); + // ── Site→central stream event batching (R2) ───────────────────────────── + // 1 is legal and means "disabled" (one event per frame, the un-negotiated shape). + builder.RequireThat(options.GrpcStreamBatchMaxEvents > 0, + $"ScadaBridge:Communication:GrpcStreamBatchMaxEvents must be positive — 1 disables " + + $"batching (was {options.GrpcStreamBatchMaxEvents})."); + + // The coalescing window is the ONLY latency batching can add, and the target-scale + // load test holds end-to-end event latency to a 250 ms P99. Validate it strictly + // below that so a misconfigured window cannot silently spend the entire budget. + builder.RequireThat( + options.GrpcStreamBatchWindow >= TimeSpan.Zero + && options.GrpcStreamBatchWindow < StreamBatchWindowCeiling, + $"ScadaBridge:Communication:GrpcStreamBatchWindow must be non-negative and strictly " + + $"below {StreamBatchWindowCeiling.TotalMilliseconds:0} ms (the end-to-end stream " + + $"latency budget the coalescing window spends from); zero means \"pack only what is " + + $"already queued\" (was {options.GrpcStreamBatchWindow})."); + // The gRPC site→central transport needs at least one central endpoint to dial. gRPC is now // the only site→central transport (ClusterClient was removed in the migration's Phase 4), so // every site node must declare its central endpoints — there is no Akka fallback to ignore diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamEventBatcher.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamEventBatcher.cs new file mode 100644 index 00000000..5720ded6 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamEventBatcher.cs @@ -0,0 +1,225 @@ +using System.Diagnostics; +using System.Threading.Channels; + +namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc; + +/// +/// Per-subscriber coalescing pump that drains a stream's bounded send channel and writes +/// it to the gRPC response stream, optionally packing several consecutive events into one +/// frame (R2 — gRPC event batching). +/// +/// +/// Where this sits. Strictly DOWNSTREAM of StreamRelayActor's bounded +/// DropOldest channel, and one instance per subscriber. It therefore changes only +/// how many gRPC frames a given set of events costs — it does not change the +/// site's burst ceiling, which is set by the shared publish stage upstream of the +/// BroadcastHub (deferred-work register row 31) and by the per-subscriber channel capacity. +/// Events dropped by that channel are dropped before the batcher ever sees them. +/// +/// +/// +/// Latency contract. The pump never delays a lone event. It takes the first event, +/// drains whatever is already queued behind it (which costs no time at all), and +/// only then — having proven a backlog exists — lingers up to +/// for more. A quiet stream is therefore byte-identical and +/// latency-identical to the pre-batching wire: one plain +/// attribute_changed/alarm_changed frame, emitted immediately. A saturated +/// stream pays at most one window per batch, which is why the window is validated well +/// under the 250 ms end-to-end latency threshold the target-scale load test measures +/// against. +/// +/// +/// +/// Ordering. Events are emitted in the exact order they were read; a batch preserves +/// that order inside SiteStreamEventBatch.events, and the client unpacks in order. +/// Nothing is reordered or coalesced away — batching is purely a framing change. +/// +/// +internal static class SiteStreamEventBatcher +{ + /// + /// Drains until it completes or is + /// cancelled, writing frames through . + /// + /// + /// Pass = 1 to get the exact pre-batching behaviour + /// (one frame per event, no window, no metric) — that is what an un-negotiated + /// subscription uses, so an old central never sees a frame shape it cannot parse. + /// + /// + /// + /// When the channel writer completes with events still buffered, the buffered events + /// are flushed as a final frame before the pump returns. Cancellation is deliberately + /// NOT flushed: the token is cancelled precisely when the client is gone or the site is + /// shutting down, so the write would fail anyway; the + /// propagates to the caller exactly as the + /// pre-batching await foreach did. + /// + /// + /// The subscriber's bounded send-channel reader. + /// Correlation id stamped on an emitted batch frame. + /// Hard cap on events per frame; 1 disables batching entirely. + /// Maximum time to linger accumulating a batch once a backlog is observed. + /// Writes one frame to the gRPC response stream. + /// Optional observer of each emitted frame's event count (the batch-size histogram). + /// Cancels the pump (client disconnect, duplicate replacement, shutdown, stream lifetime). + /// A task that completes when the channel is drained and closed. + internal static async Task PumpAsync( + ChannelReader reader, + string correlationId, + int maxBatchEvents, + TimeSpan maxWindow, + Func writeAsync, + Action? onFrameEmitted, + CancellationToken ct) + { + var buffer = new List(Math.Max(1, Math.Min(maxBatchEvents, 256))); + + while (true) + { + if (!await reader.WaitToReadAsync(ct).ConfigureAwait(false)) + { + // Writer completed and the channel is empty — normal end of stream. + return; + } + + if (!reader.TryRead(out var first)) + { + // Raced another reader (there is only one, but WaitToReadAsync can also + // wake on completion); loop round and re-evaluate. + continue; + } + + buffer.Clear(); + buffer.Add(first); + + var writerCompleted = false; + + if (maxBatchEvents > 1) + { + // Free drain: everything already sitting in the channel costs no latency. + while (buffer.Count < maxBatchEvents && reader.TryRead(out var queued)) + { + buffer.Add(queued); + } + + // Linger ONLY when a real backlog was observed. A single event on an + // otherwise idle stream is emitted immediately — the window must never + // become a floor on latency for the quiet case. + if (buffer.Count > 1 && buffer.Count < maxBatchEvents && maxWindow > TimeSpan.Zero) + { + writerCompleted = await LingerAsync( + reader, buffer, maxBatchEvents, maxWindow, ct).ConfigureAwait(false); + } + } + + await EmitAsync(buffer, correlationId, writeAsync, onFrameEmitted, ct).ConfigureAwait(false); + + if (writerCompleted) + { + // Flush-on-close: the buffered events above were the tail of the stream. + return; + } + } + } + + /// + /// Accumulates further events into for at most + /// from the moment the backlog was observed, or until the + /// size cap is reached. + /// + /// The subscriber's send-channel reader. + /// Batch under construction; appended to in arrival order. + /// Hard cap on events per frame. + /// Maximum lingering time for this batch. + /// Cancels the pump. + /// + /// when the channel writer completed while lingering (the caller + /// must emit the buffer and then stop), otherwise . + /// + private static async Task LingerAsync( + ChannelReader reader, + List buffer, + int maxBatchEvents, + TimeSpan maxWindow, + CancellationToken ct) + { + var started = Stopwatch.GetTimestamp(); + + while (buffer.Count < maxBatchEvents) + { + var remaining = maxWindow - Stopwatch.GetElapsedTime(started); + if (remaining <= TimeSpan.Zero) + { + return false; + } + + using var lingerCts = CancellationTokenSource.CreateLinkedTokenSource(ct); + lingerCts.CancelAfter(remaining); + + bool more; + try + { + more = await reader.WaitToReadAsync(lingerCts.Token).ConfigureAwait(false); + } + catch (OperationCanceledException) when (!ct.IsCancellationRequested) + { + // The window elapsed — close the batch with what we have. + return false; + } + + if (!more) + { + // Writer completed with the buffer non-empty: flush it, then stop. + return true; + } + + while (buffer.Count < maxBatchEvents && reader.TryRead(out var queued)) + { + buffer.Add(queued); + } + } + + return false; + } + + /// + /// Writes as a single frame: a plain event frame when the + /// buffer holds exactly one event (identical to the pre-batching wire), otherwise a + /// frame carrying them in order. + /// + /// Events to emit, in arrival order. Never empty. + /// Correlation id for the enclosing batch frame. + /// Writes one frame to the gRPC response stream. + /// Optional observer of the emitted frame's event count. + /// Cancels the write. + /// A task that completes when the frame has been written. + private static async Task EmitAsync( + List buffer, + string correlationId, + Func writeAsync, + Action? onFrameEmitted, + CancellationToken ct) + { + if (buffer.Count == 1) + { + await writeAsync(buffer[0], ct).ConfigureAwait(false); + onFrameEmitted?.Invoke(1); + return; + } + + var batch = new SiteStreamEventBatch(); + foreach (var evt in buffer) + { + // The enclosing frame carries the correlation id once for the whole batch; + // clearing it on the inner events is the byte saving batching exists for. + // No consumer reads the inner value (see SiteStreamGrpcClient.ForEachEvent). + evt.CorrelationId = string.Empty; + batch.Events.Add(evt); + } + + await writeAsync( + new SiteStreamEvent { CorrelationId = correlationId, Batch = batch }, ct).ConfigureAwait(false); + onFrameEmitted?.Invoke(buffer.Count); + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs index 2789cabb..9cd78de9 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs @@ -199,19 +199,25 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable var request = new InstanceStreamRequest { CorrelationId = correlationId, - InstanceUniqueName = instanceUniqueName + InstanceUniqueName = instanceUniqueName, + // R2 batch negotiation. Declaring support is safe against ANY site: one that + // predates batching ignores the unknown field and keeps sending per-event + // frames, which ForEachEvent handles as the single-event case. + BatchingSupported = true }; + void Deliver(SiteStreamEvent single) + { + var domainEvent = ConvertToDomainEvent(single); + if (domainEvent != null) + onEvent(domainEvent); + } + await ConsumeStreamAsync( correlationId, cts, () => _client.SubscribeInstance(request, cancellationToken: cts.Token), - evt => - { - var domainEvent = ConvertToDomainEvent(evt); - if (domainEvent != null) - onEvent(domainEvent); - }, + frame => ForEachEvent(frame, Deliver), onError, onCompleted); } @@ -270,19 +276,23 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable var request = new SiteStreamRequest { - CorrelationId = correlationId + CorrelationId = correlationId, + // R2 batch negotiation — see SubscribeAsync. + BatchingSupported = true }; + void Deliver(SiteStreamEvent single) + { + // Site-wide stream is alarm-only by contract; defensively ignore anything else. + if (ConvertToAlarmEvent(single) is { } alarm) + onAlarmEvent(alarm); + } + await ConsumeStreamAsync( correlationId, cts, () => _client.SubscribeSite(request, cancellationToken: cts.Token), - evt => - { - // Site-wide stream is alarm-only by contract; defensively ignore anything else. - if (ConvertToAlarmEvent(evt) is { } alarm) - onAlarmEvent(alarm); - }, + frame => ForEachEvent(frame, Deliver), onError, onCompleted, onConnected); @@ -451,6 +461,39 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable } } + /// + /// Unpacks one wire frame into the per-event pipeline (R2 — gRPC event batching). + /// A plain attribute_changed/alarm_changed frame is delivered as-is; a + /// frame is unpacked in order into the same + /// callback, so every downstream consumer (SiteAlarmAggregatorActor, + /// DebugStreamBridgeActor, the consumer-keepalive/orphan logic, per-event + /// Timestamp fidelity) sees no difference between a batched and an unbatched site. + /// + /// Unpacking is deliberately NON-RECURSIVE: the server never nests a batch inside a + /// batch, and a nested or unknown inner case from a malformed peer is skipped rather + /// than followed. Internal for testability. + /// + /// + /// The wire frame received from the site. + /// Invoked once per contained event, in arrival order. + internal static void ForEachEvent(SiteStreamEvent frame, Action handler) + { + if (frame.EventCase != SiteStreamEvent.EventOneofCase.Batch) + { + handler(frame); + return; + } + + foreach (var inner in frame.Batch.Events) + { + if (inner.EventCase is SiteStreamEvent.EventOneofCase.AttributeChanged + or SiteStreamEvent.EventOneofCase.AlarmChanged) + { + handler(inner); + } + } + } + /// /// Converts a proto SiteStreamEvent to the corresponding domain message. /// Internal for testability. diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs index 48178262..2acb2a91 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs @@ -29,6 +29,8 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase private readonly TimeSpan _maxStreamLifetime; private readonly int _instanceChannelCapacity; private readonly int _siteAlarmChannelCapacity; + private readonly int _streamBatchMaxEvents; + private readonly TimeSpan _streamBatchWindow; private volatile bool _ready; // Flipped by CancelAllStreams() when the host enters // CoordinatedShutdown so SubscribeInstance refuses new streams with @@ -75,7 +77,8 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase ILogger logger, int maxConcurrentStreams = 100) : this(streamSubscriber, logger, maxConcurrentStreams, TimeSpan.FromHours(4), - DefaultInstanceChannelCapacity, DefaultSiteAlarmChannelCapacity) + DefaultInstanceChannelCapacity, DefaultSiteAlarmChannelCapacity, + DefaultStreamBatchMaxEvents, DefaultStreamBatchWindow) { } @@ -85,6 +88,12 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase /// Fallback site-wide alarm send-channel capacity when no options are bound. internal const int DefaultSiteAlarmChannelCapacity = 20_000; + /// Fallback stream-batch size cap when no options are bound (R2). + internal const int DefaultStreamBatchMaxEvents = 100; + + /// Fallback stream-batch coalescing window when no options are bound (R2). + internal static readonly TimeSpan DefaultStreamBatchWindow = TimeSpan.FromMilliseconds(25); + /// /// DI constructor — binds /// and so the documented @@ -102,7 +111,9 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase options.Value.GrpcMaxConcurrentStreams, options.Value.GrpcMaxStreamLifetime, options.Value.GrpcInstanceStreamChannelCapacity, - options.Value.GrpcSiteAlarmStreamChannelCapacity) + options.Value.GrpcSiteAlarmStreamChannelCapacity, + options.Value.GrpcStreamBatchMaxEvents, + options.Value.GrpcStreamBatchWindow) { } @@ -112,7 +123,9 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase int maxConcurrentStreams, TimeSpan maxStreamLifetime, int instanceChannelCapacity, - int siteAlarmChannelCapacity) + int siteAlarmChannelCapacity, + int streamBatchMaxEvents, + TimeSpan streamBatchWindow) { _streamSubscriber = streamSubscriber; _logger = logger; @@ -120,6 +133,11 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase _maxStreamLifetime = maxStreamLifetime; _instanceChannelCapacity = Math.Max(1, instanceChannelCapacity); _siteAlarmChannelCapacity = Math.Max(1, siteAlarmChannelCapacity); + // Floored/clamped rather than thrown on: CommunicationOptionsValidator already + // fails the boot on a bad value, and a degenerate one must not blow up deep + // inside a live RPC on a host composed without validation (tests, embedded use). + _streamBatchMaxEvents = Math.Max(1, streamBatchMaxEvents); + _streamBatchWindow = streamBatchWindow < TimeSpan.Zero ? TimeSpan.Zero : streamBatchWindow; } /// @@ -230,6 +248,12 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase /// Effective site-wide alarm send-channel capacity. Exposed for tests. internal int SiteAlarmChannelCapacity => _siteAlarmChannelCapacity; + /// Effective stream-batch size cap (R2). Exposed for tests. + internal int StreamBatchMaxEvents => _streamBatchMaxEvents; + + /// Effective stream-batch coalescing window (R2). Exposed for tests. + internal TimeSpan StreamBatchWindow => _streamBatchWindow; + /// /// Total events evicted from stream send channels on this node since start (both stream /// kinds). Exported as scadabridge.site.stream.events_dropped; exposed here so a @@ -251,7 +275,10 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase relay => _streamSubscriber.Subscribe(request.InstanceUniqueName, relay), request.InstanceUniqueName, _instanceChannelCapacity, - streamKind: "instance"); + streamKind: "instance", + // R2 batch negotiation: proto3 defaults this to false, so a central built + // before batching existed keeps getting one frame per event. + batchingSupported: request.BatchingSupported); /// public override Task SubscribeSite( @@ -271,7 +298,8 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase // DropOldest meant an alarm burst during a WAN stall silently evicted operator- // visible transitions to make room for diagnostics traffic. _siteAlarmChannelCapacity, - streamKind: "site-alarms"); + streamKind: "site-alarms", + batchingSupported: request.BatchingSupported); /// /// Shared streaming pipeline behind and @@ -289,6 +317,12 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase /// Human-readable subscription description for logging. /// Send-channel capacity for this stream kind. /// Telemetry tag for this stream kind (instance/site-alarms). + /// + /// Whether the SUBSCRIBING CLIENT declared it understands the SiteStreamEventBatch + /// oneof case (R2). False — the proto3 default an older central necessarily sends — + /// pins this stream to one event per frame, so a peer that predates batching can never + /// receive a frame case its generated code drops on the floor. + /// private async Task RunSubscriptionStreamAsync( string correlationId, IServerStreamWriter responseStream, @@ -296,7 +330,8 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase Func subscribe, string description, int channelCapacity, - string streamKind) + string streamKind, + bool batchingSupported) { if (!_ready) throw new RpcException(new GrpcStatus(StatusCode.Unavailable, "Server not ready")); @@ -318,11 +353,29 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase StatusCode.InvalidArgument, "correlation_id is missing or not a valid identifier")); } - // Duplicate prevention -- cancel existing stream for this correlationId + // Duplicate prevention -- cancel existing stream for this correlationId. + // + // CANCEL ONLY, never Dispose. The replaced stream's CTS belongs to its own + // handler's `using var streamCts`, which is still running and still reads + // `streamCts.Token` (at the pump call below, and previously at the + // `ReadAllAsync(streamCts.Token)` it replaced). Disposing it from here raced that + // read and surfaced as an unhandled ObjectDisposedException escaping the RPC — + // observed as a full-suite-load-only failure of + // GrpcStreamIntegrationTests.Pipeline_DuplicateCorrelationId_ReplacesStream, and a + // pre-existing hazard (the same Dispose + the same first-token-read relationship + // exist unchanged before R2). Cancellation alone is what replacement needs; the + // owning handler's `using` still disposes it exactly once on every exit path. if (_activeStreams.TryRemove(correlationId, out var existingEntry)) { - existingEntry.Cts.Cancel(); - existingEntry.Cts.Dispose(); + try + { + existingEntry.Cts.Cancel(); + } + catch (ObjectDisposedException) + { + // Its owner finished and disposed it between the TryRemove and here — + // already terminal, nothing to cancel. Mirrors CancelAllStreams(). + } } // Check max concurrent streams after duplicate removal. @@ -422,10 +475,24 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase ScadaBridgeTelemetry.SiteConnectionOpened(); try { - await foreach (var evt in channel.Reader.ReadAllAsync(streamCts.Token)) - { - await responseStream.WriteAsync(evt, streamCts.Token); - } + // R2 — event batching. The pump replaces the old per-event + // `await foreach (…) WriteAsync(evt)` loop and is byte-for-byte identical to + // it when maxBatchEvents == 1, which is exactly what an un-negotiated + // subscription gets. It sits DOWNSTREAM of the bounded DropOldest channel + // above, so it changes framing only — never the site's burst ceiling. + await SiteStreamEventBatcher.PumpAsync( + channel.Reader, + correlationId, + maxBatchEvents: batchingSupported ? _streamBatchMaxEvents : 1, + maxWindow: _streamBatchWindow, + (evt, token) => responseStream.WriteAsync(evt, token), + // Recorded only on a negotiated stream: on an un-negotiated one every + // frame carries exactly one event, so the histogram would degenerate + // into a per-event instrument on the product's hottest path. + batchingSupported + ? size => ScadaBridgeTelemetry.RecordSiteStreamBatchSize(streamKind, size) + : null, + streamCts.Token); } catch (OperationCanceledException) { diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto b/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto index 850816da..831f9e89 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto @@ -20,6 +20,14 @@ service SiteStreamService { message InstanceStreamRequest { string correlation_id = 1; string instance_unique_name = 2; + // Client-declared BATCH NEGOTIATION (R2, event batching). When true the client + // understands the SiteStreamEventBatch oneof case and the server may coalesce + // consecutive events into one frame. proto3 defaults this to false, so an OLD + // central that never sets it keeps receiving one frame per event — that default + // IS the negotiation, and it is what makes new-site↔old-central safe. A NEW + // central sets it against an OLD site, which ignores the unknown field and + // keeps sending per-event frames the new client also accepts. Additive-only. + bool batching_supported = 3; } // Request for the site-wide, alarm-only SubscribeSite stream. Unlike @@ -27,6 +35,8 @@ message InstanceStreamRequest { // transitions for every instance on the site. message SiteStreamRequest { string correlation_id = 1; + // See InstanceStreamRequest.batching_supported. Additive-only. + bool batching_supported = 2; } message SiteStreamEvent { @@ -34,9 +44,26 @@ message SiteStreamEvent { oneof event { AttributeValueUpdate attribute_changed = 2; AlarmStateUpdate alarm_changed = 3; + // Coalesced frame (R2). Emitted ONLY when the subscription request set + // batching_supported = true. A batch is never nested inside a batch, and a + // single event is always sent as a plain attribute_changed/alarm_changed + // frame — so a quiet stream's wire shape is byte-identical to before. + SiteStreamEventBatch batch = 4; } } +// Coalesced carrier for several consecutive stream events (R2). Ordering is +// significant: events appear in the exact order the site produced them, and the +// client unpacks them in order into the same per-event pipeline, so per-event +// Timestamp fidelity and downstream sequencing are unchanged. +// +// The inner events deliberately leave correlation_id EMPTY — the enclosing +// SiteStreamEvent carries it once for the whole frame, which is the byte saving +// batching exists for. No consumer reads the inner correlation_id. +message SiteStreamEventBatch { + repeated SiteStreamEvent events = 1; +} + enum Quality { QUALITY_UNSPECIFIED = 0; QUALITY_GOOD = 1; diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/Sitestream.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/Sitestream.cs index 40173172..ab2d8dfc 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/Sitestream.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/Sitestream.cs @@ -26,97 +26,102 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { string.Concat( "ChdQcm90b3Mvc2l0ZXN0cmVhbS5wcm90bxIKc2l0ZXN0cmVhbRofZ29vZ2xl", "L3Byb3RvYnVmL3RpbWVzdGFtcC5wcm90bxoeZ29vZ2xlL3Byb3RvYnVmL3dy", - "YXBwZXJzLnByb3RvIk0KFUluc3RhbmNlU3RyZWFtUmVxdWVzdBIWCg5jb3Jy", + "YXBwZXJzLnByb3RvImkKFUluc3RhbmNlU3RyZWFtUmVxdWVzdBIWCg5jb3Jy", "ZWxhdGlvbl9pZBgBIAEoCRIcChRpbnN0YW5jZV91bmlxdWVfbmFtZRgCIAEo", - "CSIrChFTaXRlU3RyZWFtUmVxdWVzdBIWCg5jb3JyZWxhdGlvbl9pZBgBIAEo", - "CSKoAQoPU2l0ZVN0cmVhbUV2ZW50EhYKDmNvcnJlbGF0aW9uX2lkGAEgASgJ", - "Ej0KEWF0dHJpYnV0ZV9jaGFuZ2VkGAIgASgLMiAuc2l0ZXN0cmVhbS5BdHRy", - "aWJ1dGVWYWx1ZVVwZGF0ZUgAEjUKDWFsYXJtX2NoYW5nZWQYAyABKAsyHC5z", - "aXRlc3RyZWFtLkFsYXJtU3RhdGVVcGRhdGVIAEIHCgVldmVudCLIAQoUQXR0", - "cmlidXRlVmFsdWVVcGRhdGUSHAoUaW5zdGFuY2VfdW5pcXVlX25hbWUYASAB", - "KAkSFgoOYXR0cmlidXRlX3BhdGgYAiABKAkSFgoOYXR0cmlidXRlX25hbWUY", - "AyABKAkSDQoFdmFsdWUYBCABKAkSJAoHcXVhbGl0eRgFIAEoDjITLnNpdGVz", - "dHJlYW0uUXVhbGl0eRItCgl0aW1lc3RhbXAYBiABKAsyGi5nb29nbGUucHJv", - "dG9idWYuVGltZXN0YW1wIq8FChBBbGFybVN0YXRlVXBkYXRlEhwKFGluc3Rh", - "bmNlX3VuaXF1ZV9uYW1lGAEgASgJEhIKCmFsYXJtX25hbWUYAiABKAkSKQoF", - "c3RhdGUYAyABKA4yGi5zaXRlc3RyZWFtLkFsYXJtU3RhdGVFbnVtEhAKCHBy", - "aW9yaXR5GAQgASgFEi0KCXRpbWVzdGFtcBgFIAEoCzIaLmdvb2dsZS5wcm90", - "b2J1Zi5UaW1lc3RhbXASKQoFbGV2ZWwYBiABKA4yGi5zaXRlc3RyZWFtLkFs", - "YXJtTGV2ZWxFbnVtEg8KB21lc3NhZ2UYByABKAkSDAoEa2luZBgIIAEoCRIO", - "CgZhY3RpdmUYCSABKAgSFAoMYWNrbm93bGVkZ2VkGAogASgIEhEKCWNvbmZp", - "cm1lZBgLIAEoCBIUCgxzaGVsdmVfc3RhdGUYDCABKAkSEgoKc3VwcHJlc3Nl", - "ZBgNIAEoCBIYChBzb3VyY2VfcmVmZXJlbmNlGA4gASgJEhcKD2FsYXJtX3R5", - "cGVfbmFtZRgPIAEoCRIQCghjYXRlZ29yeRgQIAEoCRIVCg1vcGVyYXRvcl91", - "c2VyGBEgASgJEhgKEG9wZXJhdG9yX2NvbW1lbnQYEiABKAkSNwoTb3JpZ2lu", - "YWxfcmFpc2VfdGltZRgTIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3Rh", - "bXASFQoNY3VycmVudF92YWx1ZRgUIAEoCRITCgtsaW1pdF92YWx1ZRgVIAEo", - "CRIkChxuYXRpdmVfc291cmNlX2Nhbm9uaWNhbF9uYW1lGBYgASgJEiEKGWlz", - "X2NvbmZpZ3VyZWRfcGxhY2Vob2xkZXIYFyABKAgSLAoIYWNrX3RpbWUYGCAB", - "KAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wIr0ECg1BdWRpdEV2ZW50", - "RHRvEhAKCGV2ZW50X2lkGAEgASgJEjMKD29jY3VycmVkX2F0X3V0YxgCIAEo", - "CzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASDwoHY2hhbm5lbBgDIAEo", - "CRIMCgRraW5kGAQgASgJEhYKDmNvcnJlbGF0aW9uX2lkGAUgASgJEhYKDnNv", - "dXJjZV9zaXRlX2lkGAYgASgJEhoKEnNvdXJjZV9pbnN0YW5jZV9pZBgHIAEo", - "CRIVCg1zb3VyY2Vfc2NyaXB0GAggASgJEg0KBWFjdG9yGAkgASgJEg4KBnRh", - "cmdldBgKIAEoCRIOCgZzdGF0dXMYCyABKAkSMAoLaHR0cF9zdGF0dXMYDCAB", - "KAsyGy5nb29nbGUucHJvdG9idWYuSW50MzJWYWx1ZRIwCgtkdXJhdGlvbl9t", - "cxgNIAEoCzIbLmdvb2dsZS5wcm90b2J1Zi5JbnQzMlZhbHVlEhUKDWVycm9y", - "X21lc3NhZ2UYDiABKAkSFAoMZXJyb3JfZGV0YWlsGA8gASgJEhcKD3JlcXVl", - "c3Rfc3VtbWFyeRgQIAEoCRIYChByZXNwb25zZV9zdW1tYXJ5GBEgASgJEhkK", - "EXBheWxvYWRfdHJ1bmNhdGVkGBIgASgIEg0KBWV4dHJhGBMgASgJEhQKDGV4", - "ZWN1dGlvbl9pZBgUIAEoCRIbChNwYXJlbnRfZXhlY3V0aW9uX2lkGBUgASgJ", - "EhMKC3NvdXJjZV9ub2RlGBYgASgJIjwKD0F1ZGl0RXZlbnRCYXRjaBIpCgZl", - "dmVudHMYASADKAsyGS5zaXRlc3RyZWFtLkF1ZGl0RXZlbnREdG8iJwoJSW5n", - "ZXN0QWNrEhoKEmFjY2VwdGVkX2V2ZW50X2lkcxgBIAMoCSKJAwoWU2l0ZUNh", - "bGxPcGVyYXRpb25hbER0bxIcChR0cmFja2VkX29wZXJhdGlvbl9pZBgBIAEo", - "CRIPCgdjaGFubmVsGAIgASgJEg4KBnRhcmdldBgDIAEoCRITCgtzb3VyY2Vf", - "c2l0ZRgEIAEoCRIOCgZzdGF0dXMYBSABKAkSEwoLcmV0cnlfY291bnQYBiAB", - "KAUSEgoKbGFzdF9lcnJvchgHIAEoCRIwCgtodHRwX3N0YXR1cxgIIAEoCzIb", - "Lmdvb2dsZS5wcm90b2J1Zi5JbnQzMlZhbHVlEjIKDmNyZWF0ZWRfYXRfdXRj", - "GAkgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIyCg51cGRhdGVk", - "X2F0X3V0YxgKIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASMwoP", - "dGVybWluYWxfYXRfdXRjGAsgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVz", - "dGFtcBITCgtzb3VyY2Vfbm9kZRgMIAEoCSKAAQoVQ2FjaGVkVGVsZW1ldHJ5", - "UGFja2V0Ei4KC2F1ZGl0X2V2ZW50GAEgASgLMhkuc2l0ZXN0cmVhbS5BdWRp", - "dEV2ZW50RHRvEjcKC29wZXJhdGlvbmFsGAIgASgLMiIuc2l0ZXN0cmVhbS5T", - "aXRlQ2FsbE9wZXJhdGlvbmFsRHRvIkoKFENhY2hlZFRlbGVtZXRyeUJhdGNo", - "EjIKB3BhY2tldHMYASADKAsyIS5zaXRlc3RyZWFtLkNhY2hlZFRlbGVtZXRy", - "eVBhY2tldCJtChZQdWxsQXVkaXRFdmVudHNSZXF1ZXN0Ei0KCXNpbmNlX3V0", - "YxgBIAEoCzIaLmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXASEgoKYmF0Y2hf", - "c2l6ZRgCIAEoBRIQCghhZnRlcl9pZBgDIAEoCSJcChdQdWxsQXVkaXRFdmVu", - "dHNSZXNwb25zZRIpCgZldmVudHMYASADKAsyGS5zaXRlc3RyZWFtLkF1ZGl0", - "RXZlbnREdG8SFgoObW9yZV9hdmFpbGFibGUYAiABKAgiawoUUHVsbFNpdGVD", - "YWxsc1JlcXVlc3QSLQoJc2luY2VfdXRjGAEgASgLMhouZ29vZ2xlLnByb3Rv", - "YnVmLlRpbWVzdGFtcBISCgpiYXRjaF9zaXplGAIgASgFEhAKCGFmdGVyX2lk", - "GAMgASgJImkKFVB1bGxTaXRlQ2FsbHNSZXNwb25zZRI4CgxvcGVyYXRpb25h", - "bHMYASADKAsyIi5zaXRlc3RyZWFtLlNpdGVDYWxsT3BlcmF0aW9uYWxEdG8S", - "FgoObW9yZV9hdmFpbGFibGUYAiABKAgqXAoHUXVhbGl0eRIXChNRVUFMSVRZ", - "X1VOU1BFQ0lGSUVEEAASEAoMUVVBTElUWV9HT09EEAESFQoRUVVBTElUWV9V", - "TkNFUlRBSU4QAhIPCgtRVUFMSVRZX0JBRBADKl0KDkFsYXJtU3RhdGVFbnVt", - "EhsKF0FMQVJNX1NUQVRFX1VOU1BFQ0lGSUVEEAASFgoSQUxBUk1fU1RBVEVf", - "Tk9STUFMEAESFgoSQUxBUk1fU1RBVEVfQUNUSVZFEAIqhQEKDkFsYXJtTGV2", - "ZWxFbnVtEhQKEEFMQVJNX0xFVkVMX05PTkUQABITCg9BTEFSTV9MRVZFTF9M", - "T1cQARIXChNBTEFSTV9MRVZFTF9MT1dfTE9XEAISFAoQQUxBUk1fTEVWRUxf", - "SElHSBADEhkKFUFMQVJNX0xFVkVMX0hJR0hfSElHSBAEMoYEChFTaXRlU3Ry", - "ZWFtU2VydmljZRJVChFTdWJzY3JpYmVJbnN0YW5jZRIhLnNpdGVzdHJlYW0u", - "SW5zdGFuY2VTdHJlYW1SZXF1ZXN0Ghsuc2l0ZXN0cmVhbS5TaXRlU3RyZWFt", - "RXZlbnQwARJNCg1TdWJzY3JpYmVTaXRlEh0uc2l0ZXN0cmVhbS5TaXRlU3Ry", - "ZWFtUmVxdWVzdBobLnNpdGVzdHJlYW0uU2l0ZVN0cmVhbUV2ZW50MAESRwoR", - "SW5nZXN0QXVkaXRFdmVudHMSGy5zaXRlc3RyZWFtLkF1ZGl0RXZlbnRCYXRj", - "aBoVLnNpdGVzdHJlYW0uSW5nZXN0QWNrElAKFUluZ2VzdENhY2hlZFRlbGVt", - "ZXRyeRIgLnNpdGVzdHJlYW0uQ2FjaGVkVGVsZW1ldHJ5QmF0Y2gaFS5zaXRl", - "c3RyZWFtLkluZ2VzdEFjaxJaCg9QdWxsQXVkaXRFdmVudHMSIi5zaXRlc3Ry", - "ZWFtLlB1bGxBdWRpdEV2ZW50c1JlcXVlc3QaIy5zaXRlc3RyZWFtLlB1bGxB", - "dWRpdEV2ZW50c1Jlc3BvbnNlElQKDVB1bGxTaXRlQ2FsbHMSIC5zaXRlc3Ry", - "ZWFtLlB1bGxTaXRlQ2FsbHNSZXF1ZXN0GiEuc2l0ZXN0cmVhbS5QdWxsU2l0", - "ZUNhbGxzUmVzcG9uc2VCK6oCKFpCLk1PTS5XVy5TY2FkYUJyaWRnZS5Db21t", - "dW5pY2F0aW9uLkdycGNiBnByb3RvMw==")); + "CRIaChJiYXRjaGluZ19zdXBwb3J0ZWQYAyABKAgiRwoRU2l0ZVN0cmVhbVJl", + "cXVlc3QSFgoOY29ycmVsYXRpb25faWQYASABKAkSGgoSYmF0Y2hpbmdfc3Vw", + "cG9ydGVkGAIgASgIItsBCg9TaXRlU3RyZWFtRXZlbnQSFgoOY29ycmVsYXRp", + "b25faWQYASABKAkSPQoRYXR0cmlidXRlX2NoYW5nZWQYAiABKAsyIC5zaXRl", + "c3RyZWFtLkF0dHJpYnV0ZVZhbHVlVXBkYXRlSAASNQoNYWxhcm1fY2hhbmdl", + "ZBgDIAEoCzIcLnNpdGVzdHJlYW0uQWxhcm1TdGF0ZVVwZGF0ZUgAEjEKBWJh", + "dGNoGAQgASgLMiAuc2l0ZXN0cmVhbS5TaXRlU3RyZWFtRXZlbnRCYXRjaEgA", + "QgcKBWV2ZW50IkMKFFNpdGVTdHJlYW1FdmVudEJhdGNoEisKBmV2ZW50cxgB", + "IAMoCzIbLnNpdGVzdHJlYW0uU2l0ZVN0cmVhbUV2ZW50IsgBChRBdHRyaWJ1", + "dGVWYWx1ZVVwZGF0ZRIcChRpbnN0YW5jZV91bmlxdWVfbmFtZRgBIAEoCRIW", + "Cg5hdHRyaWJ1dGVfcGF0aBgCIAEoCRIWCg5hdHRyaWJ1dGVfbmFtZRgDIAEo", + "CRINCgV2YWx1ZRgEIAEoCRIkCgdxdWFsaXR5GAUgASgOMhMuc2l0ZXN0cmVh", + "bS5RdWFsaXR5Ei0KCXRpbWVzdGFtcBgGIAEoCzIaLmdvb2dsZS5wcm90b2J1", + "Zi5UaW1lc3RhbXAirwUKEEFsYXJtU3RhdGVVcGRhdGUSHAoUaW5zdGFuY2Vf", + "dW5pcXVlX25hbWUYASABKAkSEgoKYWxhcm1fbmFtZRgCIAEoCRIpCgVzdGF0", + "ZRgDIAEoDjIaLnNpdGVzdHJlYW0uQWxhcm1TdGF0ZUVudW0SEAoIcHJpb3Jp", + "dHkYBCABKAUSLQoJdGltZXN0YW1wGAUgASgLMhouZ29vZ2xlLnByb3RvYnVm", + "LlRpbWVzdGFtcBIpCgVsZXZlbBgGIAEoDjIaLnNpdGVzdHJlYW0uQWxhcm1M", + "ZXZlbEVudW0SDwoHbWVzc2FnZRgHIAEoCRIMCgRraW5kGAggASgJEg4KBmFj", + "dGl2ZRgJIAEoCBIUCgxhY2tub3dsZWRnZWQYCiABKAgSEQoJY29uZmlybWVk", + "GAsgASgIEhQKDHNoZWx2ZV9zdGF0ZRgMIAEoCRISCgpzdXBwcmVzc2VkGA0g", + "ASgIEhgKEHNvdXJjZV9yZWZlcmVuY2UYDiABKAkSFwoPYWxhcm1fdHlwZV9u", + "YW1lGA8gASgJEhAKCGNhdGVnb3J5GBAgASgJEhUKDW9wZXJhdG9yX3VzZXIY", + "ESABKAkSGAoQb3BlcmF0b3JfY29tbWVudBgSIAEoCRI3ChNvcmlnaW5hbF9y", + "YWlzZV90aW1lGBMgASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIV", + "Cg1jdXJyZW50X3ZhbHVlGBQgASgJEhMKC2xpbWl0X3ZhbHVlGBUgASgJEiQK", + "HG5hdGl2ZV9zb3VyY2VfY2Fub25pY2FsX25hbWUYFiABKAkSIQoZaXNfY29u", + "ZmlndXJlZF9wbGFjZWhvbGRlchgXIAEoCBIsCghhY2tfdGltZRgYIAEoCzIa", + "Lmdvb2dsZS5wcm90b2J1Zi5UaW1lc3RhbXAivQQKDUF1ZGl0RXZlbnREdG8S", + "EAoIZXZlbnRfaWQYASABKAkSMwoPb2NjdXJyZWRfYXRfdXRjGAIgASgLMhou", + "Z29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIPCgdjaGFubmVsGAMgASgJEgwK", + "BGtpbmQYBCABKAkSFgoOY29ycmVsYXRpb25faWQYBSABKAkSFgoOc291cmNl", + "X3NpdGVfaWQYBiABKAkSGgoSc291cmNlX2luc3RhbmNlX2lkGAcgASgJEhUK", + "DXNvdXJjZV9zY3JpcHQYCCABKAkSDQoFYWN0b3IYCSABKAkSDgoGdGFyZ2V0", + "GAogASgJEg4KBnN0YXR1cxgLIAEoCRIwCgtodHRwX3N0YXR1cxgMIAEoCzIb", + "Lmdvb2dsZS5wcm90b2J1Zi5JbnQzMlZhbHVlEjAKC2R1cmF0aW9uX21zGA0g", + "ASgLMhsuZ29vZ2xlLnByb3RvYnVmLkludDMyVmFsdWUSFQoNZXJyb3JfbWVz", + "c2FnZRgOIAEoCRIUCgxlcnJvcl9kZXRhaWwYDyABKAkSFwoPcmVxdWVzdF9z", + "dW1tYXJ5GBAgASgJEhgKEHJlc3BvbnNlX3N1bW1hcnkYESABKAkSGQoRcGF5", + "bG9hZF90cnVuY2F0ZWQYEiABKAgSDQoFZXh0cmEYEyABKAkSFAoMZXhlY3V0", + "aW9uX2lkGBQgASgJEhsKE3BhcmVudF9leGVjdXRpb25faWQYFSABKAkSEwoL", + "c291cmNlX25vZGUYFiABKAkiPAoPQXVkaXRFdmVudEJhdGNoEikKBmV2ZW50", + "cxgBIAMoCzIZLnNpdGVzdHJlYW0uQXVkaXRFdmVudER0byInCglJbmdlc3RB", + "Y2sSGgoSYWNjZXB0ZWRfZXZlbnRfaWRzGAEgAygJIokDChZTaXRlQ2FsbE9w", + "ZXJhdGlvbmFsRHRvEhwKFHRyYWNrZWRfb3BlcmF0aW9uX2lkGAEgASgJEg8K", + "B2NoYW5uZWwYAiABKAkSDgoGdGFyZ2V0GAMgASgJEhMKC3NvdXJjZV9zaXRl", + "GAQgASgJEg4KBnN0YXR1cxgFIAEoCRITCgtyZXRyeV9jb3VudBgGIAEoBRIS", + "CgpsYXN0X2Vycm9yGAcgASgJEjAKC2h0dHBfc3RhdHVzGAggASgLMhsuZ29v", + "Z2xlLnByb3RvYnVmLkludDMyVmFsdWUSMgoOY3JlYXRlZF9hdF91dGMYCSAB", + "KAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1wEjIKDnVwZGF0ZWRfYXRf", + "dXRjGAogASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBIzCg90ZXJt", + "aW5hbF9hdF91dGMYCyABKAsyGi5nb29nbGUucHJvdG9idWYuVGltZXN0YW1w", + "EhMKC3NvdXJjZV9ub2RlGAwgASgJIoABChVDYWNoZWRUZWxlbWV0cnlQYWNr", + "ZXQSLgoLYXVkaXRfZXZlbnQYASABKAsyGS5zaXRlc3RyZWFtLkF1ZGl0RXZl", + "bnREdG8SNwoLb3BlcmF0aW9uYWwYAiABKAsyIi5zaXRlc3RyZWFtLlNpdGVD", + "YWxsT3BlcmF0aW9uYWxEdG8iSgoUQ2FjaGVkVGVsZW1ldHJ5QmF0Y2gSMgoH", + "cGFja2V0cxgBIAMoCzIhLnNpdGVzdHJlYW0uQ2FjaGVkVGVsZW1ldHJ5UGFj", + "a2V0Im0KFlB1bGxBdWRpdEV2ZW50c1JlcXVlc3QSLQoJc2luY2VfdXRjGAEg", + "ASgLMhouZ29vZ2xlLnByb3RvYnVmLlRpbWVzdGFtcBISCgpiYXRjaF9zaXpl", + "GAIgASgFEhAKCGFmdGVyX2lkGAMgASgJIlwKF1B1bGxBdWRpdEV2ZW50c1Jl", + "c3BvbnNlEikKBmV2ZW50cxgBIAMoCzIZLnNpdGVzdHJlYW0uQXVkaXRFdmVu", + "dER0bxIWCg5tb3JlX2F2YWlsYWJsZRgCIAEoCCJrChRQdWxsU2l0ZUNhbGxz", + "UmVxdWVzdBItCglzaW5jZV91dGMYASABKAsyGi5nb29nbGUucHJvdG9idWYu", + "VGltZXN0YW1wEhIKCmJhdGNoX3NpemUYAiABKAUSEAoIYWZ0ZXJfaWQYAyAB", + "KAkiaQoVUHVsbFNpdGVDYWxsc1Jlc3BvbnNlEjgKDG9wZXJhdGlvbmFscxgB", + "IAMoCzIiLnNpdGVzdHJlYW0uU2l0ZUNhbGxPcGVyYXRpb25hbER0bxIWCg5t", + "b3JlX2F2YWlsYWJsZRgCIAEoCCpcCgdRdWFsaXR5EhcKE1FVQUxJVFlfVU5T", + "UEVDSUZJRUQQABIQCgxRVUFMSVRZX0dPT0QQARIVChFRVUFMSVRZX1VOQ0VS", + "VEFJThACEg8KC1FVQUxJVFlfQkFEEAMqXQoOQWxhcm1TdGF0ZUVudW0SGwoX", + "QUxBUk1fU1RBVEVfVU5TUEVDSUZJRUQQABIWChJBTEFSTV9TVEFURV9OT1JN", + "QUwQARIWChJBTEFSTV9TVEFURV9BQ1RJVkUQAiqFAQoOQWxhcm1MZXZlbEVu", + "dW0SFAoQQUxBUk1fTEVWRUxfTk9ORRAAEhMKD0FMQVJNX0xFVkVMX0xPVxAB", + "EhcKE0FMQVJNX0xFVkVMX0xPV19MT1cQAhIUChBBTEFSTV9MRVZFTF9ISUdI", + "EAMSGQoVQUxBUk1fTEVWRUxfSElHSF9ISUdIEAQyhgQKEVNpdGVTdHJlYW1T", + "ZXJ2aWNlElUKEVN1YnNjcmliZUluc3RhbmNlEiEuc2l0ZXN0cmVhbS5JbnN0", + "YW5jZVN0cmVhbVJlcXVlc3QaGy5zaXRlc3RyZWFtLlNpdGVTdHJlYW1FdmVu", + "dDABEk0KDVN1YnNjcmliZVNpdGUSHS5zaXRlc3RyZWFtLlNpdGVTdHJlYW1S", + "ZXF1ZXN0Ghsuc2l0ZXN0cmVhbS5TaXRlU3RyZWFtRXZlbnQwARJHChFJbmdl", + "c3RBdWRpdEV2ZW50cxIbLnNpdGVzdHJlYW0uQXVkaXRFdmVudEJhdGNoGhUu", + "c2l0ZXN0cmVhbS5Jbmdlc3RBY2sSUAoVSW5nZXN0Q2FjaGVkVGVsZW1ldHJ5", + "EiAuc2l0ZXN0cmVhbS5DYWNoZWRUZWxlbWV0cnlCYXRjaBoVLnNpdGVzdHJl", + "YW0uSW5nZXN0QWNrEloKD1B1bGxBdWRpdEV2ZW50cxIiLnNpdGVzdHJlYW0u", + "UHVsbEF1ZGl0RXZlbnRzUmVxdWVzdBojLnNpdGVzdHJlYW0uUHVsbEF1ZGl0", + "RXZlbnRzUmVzcG9uc2USVAoNUHVsbFNpdGVDYWxscxIgLnNpdGVzdHJlYW0u", + "UHVsbFNpdGVDYWxsc1JlcXVlc3QaIS5zaXRlc3RyZWFtLlB1bGxTaXRlQ2Fs", + "bHNSZXNwb25zZUIrqgIoWkIuTU9NLldXLlNjYWRhQnJpZGdlLkNvbW11bmlj", + "YXRpb24uR3JwY2IGcHJvdG8z")); descriptor = pbr::FileDescriptor.FromGeneratedCode(descriptorData, new pbr::FileDescriptor[] { global::Google.Protobuf.WellKnownTypes.TimestampReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.WrappersReflection.Descriptor, }, new pbr::GeneratedClrTypeInfo(new[] {typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.Quality), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmStateEnum), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmLevelEnum), }, null, new pbr::GeneratedClrTypeInfo[] { - new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.InstanceStreamRequest), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.InstanceStreamRequest.Parser, new[]{ "CorrelationId", "InstanceUniqueName" }, null, null, null, null), - new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest.Parser, new[]{ "CorrelationId" }, null, null, null, null), - new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEvent), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEvent.Parser, new[]{ "CorrelationId", "AttributeChanged", "AlarmChanged" }, new[]{ "Event" }, null, null, null), + new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.InstanceStreamRequest), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.InstanceStreamRequest.Parser, new[]{ "CorrelationId", "InstanceUniqueName", "BatchingSupported" }, null, null, null, null), + new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest.Parser, new[]{ "CorrelationId", "BatchingSupported" }, null, null, null, null), + new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEvent), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEvent.Parser, new[]{ "CorrelationId", "AttributeChanged", "AlarmChanged", "Batch" }, new[]{ "Event" }, null, null, null), + new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEventBatch), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEventBatch.Parser, new[]{ "Events" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AttributeValueUpdate), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AttributeValueUpdate.Parser, new[]{ "InstanceUniqueName", "AttributePath", "AttributeName", "Value", "Quality", "Timestamp" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmStateUpdate), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmStateUpdate.Parser, new[]{ "InstanceUniqueName", "AlarmName", "State", "Priority", "Timestamp", "Level", "Message", "Kind", "Active", "Acknowledged", "Confirmed", "ShelveState", "Suppressed", "SourceReference", "AlarmTypeName", "Category", "OperatorUser", "OperatorComment", "OriginalRaiseTime", "CurrentValue", "LimitValue", "NativeSourceCanonicalName", "IsConfiguredPlaceholder", "AckTime" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventDto), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventDto.Parser, new[]{ "EventId", "OccurredAtUtc", "Channel", "Kind", "CorrelationId", "SourceSiteId", "SourceInstanceId", "SourceScript", "Actor", "Target", "Status", "HttpStatus", "DurationMs", "ErrorMessage", "ErrorDetail", "RequestSummary", "ResponseSummary", "PayloadTruncated", "Extra", "ExecutionId", "ParentExecutionId", "SourceNode" }, null, null, null, null), @@ -201,6 +206,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { public InstanceStreamRequest(InstanceStreamRequest other) : this() { correlationId_ = other.correlationId_; instanceUniqueName_ = other.instanceUniqueName_; + batchingSupported_ = other.batchingSupported_; _unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields); } @@ -234,6 +240,27 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { } } + /// Field number for the "batching_supported" field. + public const int BatchingSupportedFieldNumber = 3; + private bool batchingSupported_; + /// + /// Client-declared BATCH NEGOTIATION (R2, event batching). When true the client + /// understands the SiteStreamEventBatch oneof case and the server may coalesce + /// consecutive events into one frame. proto3 defaults this to false, so an OLD + /// central that never sets it keeps receiving one frame per event — that default + /// IS the negotiation, and it is what makes new-site↔old-central safe. A NEW + /// central sets it against an OLD site, which ignores the unknown field and + /// keeps sending per-event frames the new client also accepts. Additive-only. + /// + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public bool BatchingSupported { + get { return batchingSupported_; } + set { + batchingSupported_ = value; + } + } + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public override bool Equals(object other) { @@ -251,6 +278,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { } if (CorrelationId != other.CorrelationId) return false; if (InstanceUniqueName != other.InstanceUniqueName) return false; + if (BatchingSupported != other.BatchingSupported) return false; return Equals(_unknownFields, other._unknownFields); } @@ -260,6 +288,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { int hash = 1; if (CorrelationId.Length != 0) hash ^= CorrelationId.GetHashCode(); if (InstanceUniqueName.Length != 0) hash ^= InstanceUniqueName.GetHashCode(); + if (BatchingSupported != false) hash ^= BatchingSupported.GetHashCode(); if (_unknownFields != null) { hash ^= _unknownFields.GetHashCode(); } @@ -286,6 +315,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { output.WriteRawTag(18); output.WriteString(InstanceUniqueName); } + if (BatchingSupported != false) { + output.WriteRawTag(24); + output.WriteBool(BatchingSupported); + } if (_unknownFields != null) { _unknownFields.WriteTo(output); } @@ -304,6 +337,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { output.WriteRawTag(18); output.WriteString(InstanceUniqueName); } + if (BatchingSupported != false) { + output.WriteRawTag(24); + output.WriteBool(BatchingSupported); + } if (_unknownFields != null) { _unknownFields.WriteTo(ref output); } @@ -320,6 +357,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (InstanceUniqueName.Length != 0) { size += 1 + pb::CodedOutputStream.ComputeStringSize(InstanceUniqueName); } + if (BatchingSupported != false) { + size += 1 + 1; + } if (_unknownFields != null) { size += _unknownFields.CalculateSize(); } @@ -338,6 +378,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (other.InstanceUniqueName.Length != 0) { InstanceUniqueName = other.InstanceUniqueName; } + if (other.BatchingSupported != false) { + BatchingSupported = other.BatchingSupported; + } _unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields); } @@ -365,6 +408,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { InstanceUniqueName = input.ReadString(); break; } + case 24: { + BatchingSupported = input.ReadBool(); + break; + } } } #endif @@ -392,6 +439,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { InstanceUniqueName = input.ReadString(); break; } + case 24: { + BatchingSupported = input.ReadBool(); + break; + } } } } @@ -440,6 +491,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public SiteStreamRequest(SiteStreamRequest other) : this() { correlationId_ = other.correlationId_; + batchingSupported_ = other.batchingSupported_; _unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields); } @@ -461,6 +513,21 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { } } + /// Field number for the "batching_supported" field. + public const int BatchingSupportedFieldNumber = 2; + private bool batchingSupported_; + /// + /// See InstanceStreamRequest.batching_supported. Additive-only. + /// + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public bool BatchingSupported { + get { return batchingSupported_; } + set { + batchingSupported_ = value; + } + } + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public override bool Equals(object other) { @@ -477,6 +544,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { return true; } if (CorrelationId != other.CorrelationId) return false; + if (BatchingSupported != other.BatchingSupported) return false; return Equals(_unknownFields, other._unknownFields); } @@ -485,6 +553,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { public override int GetHashCode() { int hash = 1; if (CorrelationId.Length != 0) hash ^= CorrelationId.GetHashCode(); + if (BatchingSupported != false) hash ^= BatchingSupported.GetHashCode(); if (_unknownFields != null) { hash ^= _unknownFields.GetHashCode(); } @@ -507,6 +576,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { output.WriteRawTag(10); output.WriteString(CorrelationId); } + if (BatchingSupported != false) { + output.WriteRawTag(16); + output.WriteBool(BatchingSupported); + } if (_unknownFields != null) { _unknownFields.WriteTo(output); } @@ -521,6 +594,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { output.WriteRawTag(10); output.WriteString(CorrelationId); } + if (BatchingSupported != false) { + output.WriteRawTag(16); + output.WriteBool(BatchingSupported); + } if (_unknownFields != null) { _unknownFields.WriteTo(ref output); } @@ -534,6 +611,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (CorrelationId.Length != 0) { size += 1 + pb::CodedOutputStream.ComputeStringSize(CorrelationId); } + if (BatchingSupported != false) { + size += 1 + 1; + } if (_unknownFields != null) { size += _unknownFields.CalculateSize(); } @@ -549,6 +629,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (other.CorrelationId.Length != 0) { CorrelationId = other.CorrelationId; } + if (other.BatchingSupported != false) { + BatchingSupported = other.BatchingSupported; + } _unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields); } @@ -572,6 +655,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { CorrelationId = input.ReadString(); break; } + case 16: { + BatchingSupported = input.ReadBool(); + break; + } } } #endif @@ -595,6 +682,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { CorrelationId = input.ReadString(); break; } + case 16: { + BatchingSupported = input.ReadBool(); + break; + } } } } @@ -645,6 +736,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { case EventOneofCase.AlarmChanged: AlarmChanged = other.AlarmChanged.Clone(); break; + case EventOneofCase.Batch: + Batch = other.Batch.Clone(); + break; } _unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields); @@ -692,12 +786,31 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { } } + /// Field number for the "batch" field. + public const int BatchFieldNumber = 4; + /// + /// Coalesced frame (R2). Emitted ONLY when the subscription request set + /// batching_supported = true. A batch is never nested inside a batch, and a + /// single event is always sent as a plain attribute_changed/alarm_changed + /// frame — so a quiet stream's wire shape is byte-identical to before. + /// + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEventBatch Batch { + get { return eventCase_ == EventOneofCase.Batch ? (global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEventBatch) event_ : null; } + set { + event_ = value; + eventCase_ = value == null ? EventOneofCase.None : EventOneofCase.Batch; + } + } + private object event_; /// Enum of possible cases for the "event" oneof. public enum EventOneofCase { None = 0, AttributeChanged = 2, AlarmChanged = 3, + Batch = 4, } private EventOneofCase eventCase_ = EventOneofCase.None; [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -731,6 +844,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (CorrelationId != other.CorrelationId) return false; if (!object.Equals(AttributeChanged, other.AttributeChanged)) return false; if (!object.Equals(AlarmChanged, other.AlarmChanged)) return false; + if (!object.Equals(Batch, other.Batch)) return false; if (EventCase != other.EventCase) return false; return Equals(_unknownFields, other._unknownFields); } @@ -742,6 +856,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (CorrelationId.Length != 0) hash ^= CorrelationId.GetHashCode(); if (eventCase_ == EventOneofCase.AttributeChanged) hash ^= AttributeChanged.GetHashCode(); if (eventCase_ == EventOneofCase.AlarmChanged) hash ^= AlarmChanged.GetHashCode(); + if (eventCase_ == EventOneofCase.Batch) hash ^= Batch.GetHashCode(); hash ^= (int) eventCase_; if (_unknownFields != null) { hash ^= _unknownFields.GetHashCode(); @@ -773,6 +888,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { output.WriteRawTag(26); output.WriteMessage(AlarmChanged); } + if (eventCase_ == EventOneofCase.Batch) { + output.WriteRawTag(34); + output.WriteMessage(Batch); + } if (_unknownFields != null) { _unknownFields.WriteTo(output); } @@ -795,6 +914,10 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { output.WriteRawTag(26); output.WriteMessage(AlarmChanged); } + if (eventCase_ == EventOneofCase.Batch) { + output.WriteRawTag(34); + output.WriteMessage(Batch); + } if (_unknownFields != null) { _unknownFields.WriteTo(ref output); } @@ -814,6 +937,9 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { if (eventCase_ == EventOneofCase.AlarmChanged) { size += 1 + pb::CodedOutputStream.ComputeMessageSize(AlarmChanged); } + if (eventCase_ == EventOneofCase.Batch) { + size += 1 + pb::CodedOutputStream.ComputeMessageSize(Batch); + } if (_unknownFields != null) { size += _unknownFields.CalculateSize(); } @@ -842,6 +968,12 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { } AlarmChanged.MergeFrom(other.AlarmChanged); break; + case EventOneofCase.Batch: + if (Batch == null) { + Batch = new global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEventBatch(); + } + Batch.MergeFrom(other.Batch); + break; } _unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields); @@ -885,6 +1017,15 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { AlarmChanged = subBuilder; break; } + case 34: { + global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEventBatch subBuilder = new global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEventBatch(); + if (eventCase_ == EventOneofCase.Batch) { + subBuilder.MergeFrom(Batch); + } + input.ReadMessage(subBuilder); + Batch = subBuilder; + break; + } } } #endif @@ -926,6 +1067,212 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { AlarmChanged = subBuilder; break; } + case 34: { + global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEventBatch subBuilder = new global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEventBatch(); + if (eventCase_ == EventOneofCase.Batch) { + subBuilder.MergeFrom(Batch); + } + input.ReadMessage(subBuilder); + Batch = subBuilder; + break; + } + } + } + } + #endif + + } + + /// + /// Coalesced carrier for several consecutive stream events (R2). Ordering is + /// significant: events appear in the exact order the site produced them, and the + /// client unpacks them in order into the same per-event pipeline, so per-event + /// Timestamp fidelity and downstream sequencing are unchanged. + /// + /// The inner events deliberately leave correlation_id EMPTY — the enclosing + /// SiteStreamEvent carries it once for the whole frame, which is the byte saving + /// batching exists for. No consumer reads the inner correlation_id. + /// + [global::System.Diagnostics.DebuggerDisplayAttribute("{ToString(),nq}")] + public sealed partial class SiteStreamEventBatch : pb::IMessage + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + , pb::IBufferMessage + #endif + { + private static readonly pb::MessageParser _parser = new pb::MessageParser(() => new SiteStreamEventBatch()); + private pb::UnknownFieldSet _unknownFields; + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public static pb::MessageParser Parser { get { return _parser; } } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public static pbr::MessageDescriptor Descriptor { + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[3]; } + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + pbr::MessageDescriptor pb::IMessage.Descriptor { + get { return Descriptor; } + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public SiteStreamEventBatch() { + OnConstruction(); + } + + partial void OnConstruction(); + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public SiteStreamEventBatch(SiteStreamEventBatch other) : this() { + events_ = other.events_.Clone(); + _unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public SiteStreamEventBatch Clone() { + return new SiteStreamEventBatch(this); + } + + /// Field number for the "events" field. + public const int EventsFieldNumber = 1; + private static readonly pb::FieldCodec _repeated_events_codec + = pb::FieldCodec.ForMessage(10, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEvent.Parser); + private readonly pbc::RepeatedField events_ = new pbc::RepeatedField(); + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public pbc::RepeatedField Events { + get { return events_; } + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public override bool Equals(object other) { + return Equals(other as SiteStreamEventBatch); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public bool Equals(SiteStreamEventBatch other) { + if (ReferenceEquals(other, null)) { + return false; + } + if (ReferenceEquals(other, this)) { + return true; + } + if(!events_.Equals(other.events_)) return false; + return Equals(_unknownFields, other._unknownFields); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public override int GetHashCode() { + int hash = 1; + hash ^= events_.GetHashCode(); + if (_unknownFields != null) { + hash ^= _unknownFields.GetHashCode(); + } + return hash; + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public override string ToString() { + return pb::JsonFormatter.ToDiagnosticString(this); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public void WriteTo(pb::CodedOutputStream output) { + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + output.WriteRawMessage(this); + #else + events_.WriteTo(output, _repeated_events_codec); + if (_unknownFields != null) { + _unknownFields.WriteTo(output); + } + #endif + } + + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + void pb::IBufferMessage.InternalWriteTo(ref pb::WriteContext output) { + events_.WriteTo(ref output, _repeated_events_codec); + if (_unknownFields != null) { + _unknownFields.WriteTo(ref output); + } + } + #endif + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public int CalculateSize() { + int size = 0; + size += events_.CalculateSize(_repeated_events_codec); + if (_unknownFields != null) { + size += _unknownFields.CalculateSize(); + } + return size; + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public void MergeFrom(SiteStreamEventBatch other) { + if (other == null) { + return; + } + events_.Add(other.events_); + _unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public void MergeFrom(pb::CodedInputStream input) { + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + input.ReadRawMessage(this); + #else + uint tag; + while ((tag = input.ReadTag()) != 0) { + if ((tag & 7) == 4) { + // Abort on any end group tag. + return; + } + switch(tag) { + default: + _unknownFields = pb::UnknownFieldSet.MergeFieldFrom(_unknownFields, input); + break; + case 10: { + events_.AddEntriesFrom(input, _repeated_events_codec); + break; + } + } + } + #endif + } + + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + void pb::IBufferMessage.InternalMergeFrom(ref pb::ParseContext input) { + uint tag; + while ((tag = input.ReadTag()) != 0) { + if ((tag & 7) == 4) { + // Abort on any end group tag. + return; + } + switch(tag) { + default: + _unknownFields = pb::UnknownFieldSet.MergeFieldFrom(_unknownFields, ref input); + break; + case 10: { + events_.AddEntriesFrom(ref input, _repeated_events_codec); + break; + } } } } @@ -948,7 +1295,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[3]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[4]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -1340,7 +1687,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[4]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[5]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -2461,7 +2808,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[5]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[6]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -3476,7 +3823,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[6]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[7]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -3663,7 +4010,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[7]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[8]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -3856,7 +4203,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[8]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[9]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -4514,7 +4861,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[9]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[10]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -4767,7 +5114,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[10]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[11]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -4965,7 +5312,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[11]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[12]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -5257,7 +5604,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[12]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[13]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -5490,7 +5837,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[13]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[14]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -5780,7 +6127,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[14]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[15]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CommunicationOptionsValidatorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CommunicationOptionsValidatorTests.cs index 8df4e45b..832c8f2c 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CommunicationOptionsValidatorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CommunicationOptionsValidatorTests.cs @@ -90,6 +90,73 @@ public class CommunicationOptionsValidatorTests Assert.Contains("GrpcMaxConcurrentStreams", result.FailureMessage); } + // ── R2: site→central stream event batching ────────────────────────────────── + + [Fact] + public void DefaultStreamBatchOptions_AreValid() + { + var options = new CommunicationOptions(); + Assert.Equal(100, options.GrpcStreamBatchMaxEvents); + Assert.Equal(TimeSpan.FromMilliseconds(25), options.GrpcStreamBatchWindow); + Assert.True(Validate(options).Succeeded); + } + + [Fact] + public void StreamBatchMaxEventsOfOne_IsValid_AndMeansBatchingDisabled() + { + var result = Validate(new CommunicationOptions { GrpcStreamBatchMaxEvents = 1 }); + Assert.True(result.Succeeded, result.FailureMessage); + } + + [Fact] + public void NonPositiveStreamBatchMaxEvents_IsRejected() + { + var result = Validate(new CommunicationOptions { GrpcStreamBatchMaxEvents = 0 }); + Assert.True(result.Failed); + Assert.Contains("GrpcStreamBatchMaxEvents", result.FailureMessage); + } + + [Fact] + public void ZeroStreamBatchWindow_IsValid() + { + // Zero = "pack only what is already queued, never wait" — a legitimate posture for + // a latency-critical deployment that still wants the framing saving. + var result = Validate(new CommunicationOptions { GrpcStreamBatchWindow = TimeSpan.Zero }); + Assert.True(result.Succeeded, result.FailureMessage); + } + + [Fact] + public void NegativeStreamBatchWindow_IsRejected() + { + var result = Validate(new CommunicationOptions + { + GrpcStreamBatchWindow = TimeSpan.FromMilliseconds(-1) + }); + Assert.True(result.Failed); + Assert.Contains("GrpcStreamBatchWindow", result.FailureMessage); + } + + [Fact] + public void StreamBatchWindowAtOrAboveTheLatencyBudget_IsRejected() + { + // The coalescing window is the only latency batching adds and the target-scale + // load test holds end-to-end stream latency to a 250 ms P99 — a window that could + // spend the whole budget must not boot. + foreach (var window in new[] { TimeSpan.FromMilliseconds(250), TimeSpan.FromSeconds(1) }) + { + var result = Validate(new CommunicationOptions { GrpcStreamBatchWindow = window }); + Assert.True(result.Failed, $"{window} was accepted"); + Assert.Contains("GrpcStreamBatchWindow", result.FailureMessage); + } + + // Just inside the ceiling is accepted — the bound is exclusive, not a round-down. + Assert.True(Validate(new CommunicationOptions + { + GrpcStreamBatchWindow = CommunicationOptionsValidator.StreamBatchWindowCeiling + - TimeSpan.FromMilliseconds(1) + }).Succeeded); + } + // ── Aggregated live alarm cache options (plan #10, Task 6) ─────────────────── [Fact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/ProtoContractTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/ProtoContractTests.cs index 5dca8360..b24a243e 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/ProtoContractTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/ProtoContractTests.cs @@ -20,6 +20,17 @@ public class ProtoContractTests SiteStreamEvent.EventOneofCase.AlarmChanged ]; + /// + /// Oneof variants that are NOT per-event payloads and so are deliberately absent from + /// . Batch (R2) is a framing envelope: it is unpacked + /// by into the per-event cases above + /// BEFORE conversion, and never reaches ConvertToDomainEvent as a whole frame. + /// + private static readonly SiteStreamEvent.EventOneofCase[] FramingCases = + [ + SiteStreamEvent.EventOneofCase.Batch + ]; + [Fact] public void AllOneofVariants_HaveConversionHandlers() { @@ -27,9 +38,37 @@ public class ProtoContractTests .Where(c => c != SiteStreamEvent.EventOneofCase.None) .ToArray(); - Assert.Equal(allCases.Length, HandledCases.Length); + var accountedFor = HandledCases.Concat(FramingCases).ToArray(); + Assert.Equal(allCases.Length, accountedFor.Length); foreach (var c in allCases) - Assert.Contains(c, HandledCases); + Assert.Contains(c, accountedFor); + } + + [Fact] + public void BatchFrame_IsUnpackedIntoPerEventCases_NotConvertedWhole() + { + // The framing case's contract: ForEachEvent hands the per-event cases to the + // handler in order, and ConvertToDomainEvent is never asked to make sense of the + // envelope itself (it would return null, silently dropping the whole batch). + var inner = new[] + { + CreateTestEvent(SiteStreamEvent.EventOneofCase.AttributeChanged), + CreateTestEvent(SiteStreamEvent.EventOneofCase.AlarmChanged) + }; + var frame = new SiteStreamEvent + { + CorrelationId = "test", + Batch = new SiteStreamEventBatch { Events = { inner } } + }; + + Assert.Null(SiteStreamGrpcClient.ConvertToDomainEvent(frame)); + + var seen = new List(); + SiteStreamGrpcClient.ForEachEvent(frame, e => seen.Add(e.EventCase)); + + Assert.Equal( + [SiteStreamEvent.EventOneofCase.AttributeChanged, SiteStreamEvent.EventOneofCase.AlarmChanged], + seen); } [Theory] diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamEventBatcherTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamEventBatcherTests.cs new file mode 100644 index 00000000..18c25a39 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamEventBatcherTests.cs @@ -0,0 +1,316 @@ +using System.Diagnostics; +using System.Threading.Channels; +using Google.Protobuf.WellKnownTypes; +using ZB.MOM.WW.ScadaBridge.Communication.Grpc; + +namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Grpc; + +/// +/// Unit tests for the per-subscriber coalescing pump behind R2 (gRPC event batching). +/// The pump is the only new behaviour on the site→central hot path, so its contract is +/// pinned directly rather than only through the server: the size cap, the time cap, the +/// flush when the channel writer completes, the "never reorder" guarantee, and the +/// un-negotiated (maxBatchEvents == 1) shape that keeps an older central working. +/// +public class SiteStreamEventBatcherTests +{ + private const string Corr = "corr-batch"; + + private static SiteStreamEvent Event(int seq) => new() + { + CorrelationId = Corr, + AttributeChanged = new AttributeValueUpdate + { + InstanceUniqueName = "SiteA.Pump01", + AttributePath = "Modules.IO", + AttributeName = "Seq", + Value = seq.ToString(), + Quality = Quality.Good, + Timestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UnixEpoch.AddSeconds(seq)) + } + }; + + /// Flattens an emitted frame into the sequence numbers it carried, in order. + private static IEnumerable Seqs(SiteStreamEvent frame) + { + if (frame.EventCase == SiteStreamEvent.EventOneofCase.Batch) + { + foreach (var inner in frame.Batch.Events) + yield return int.Parse(inner.AttributeChanged.Value); + yield break; + } + + yield return int.Parse(frame.AttributeChanged.Value); + } + + private sealed record PumpRun(List Frames, List FrameSizes, Task Completion); + + private static PumpRun StartPump( + ChannelReader reader, + int maxBatchEvents, + TimeSpan window, + CancellationToken ct = default) + { + var frames = new List(); + var sizes = new List(); + var task = SiteStreamEventBatcher.PumpAsync( + reader, + Corr, + maxBatchEvents, + window, + (evt, _) => + { + lock (frames) { frames.Add(evt); } + return Task.CompletedTask; + }, + size => { lock (frames) { sizes.Add(size); } }, + ct); + return new PumpRun(frames, sizes, task); + } + + // ── Size cap ──────────────────────────────────────────────────────────────── + + [Fact] + public async Task SizeCap_SplitsABacklogIntoFramesOfAtMostMaxEvents() + { + // A backlog already sitting in the channel is drained without waiting, but never + // beyond the size cap — 250 queued events at a cap of 100 must come out as + // 100 + 100 + 50, in order, with nothing lost or duplicated. + var channel = Channel.CreateUnbounded(); + for (var i = 0; i < 250; i++) + Assert.True(channel.Writer.TryWrite(Event(i))); + channel.Writer.Complete(); + + var run = StartPump(channel.Reader, maxBatchEvents: 100, window: TimeSpan.FromMilliseconds(25)); + await run.Completion; + + Assert.All(run.FrameSizes, s => Assert.True(s <= 100, $"frame carried {s} events (cap 100)")); + Assert.Equal([100, 100, 50], run.FrameSizes); + Assert.Equal(Enumerable.Range(0, 250), run.Frames.SelectMany(Seqs)); + } + + [Fact] + public async Task SizeCapOfOne_EmitsPlainPerEventFrames_TheUnnegotiatedShape() + { + // maxBatchEvents == 1 is what an un-negotiated subscription (an older central) + // gets. Every event must ride its own plain frame — never a Batch case, which + // that central's generated code cannot parse. + var channel = Channel.CreateUnbounded(); + for (var i = 0; i < 5; i++) + channel.Writer.TryWrite(Event(i)); + channel.Writer.Complete(); + + var run = StartPump(channel.Reader, maxBatchEvents: 1, window: TimeSpan.FromMilliseconds(25)); + await run.Completion; + + Assert.Equal(5, run.Frames.Count); + Assert.All(run.Frames, f => + Assert.Equal(SiteStreamEvent.EventOneofCase.AttributeChanged, f.EventCase)); + Assert.All(run.Frames, f => Assert.Equal(Corr, f.CorrelationId)); + Assert.Equal(Enumerable.Range(0, 5), run.Frames.SelectMany(Seqs)); + } + + // ── Time cap ──────────────────────────────────────────────────────────────── + + [Fact] + public async Task TimeCap_ClosesAnUnderfullBatchWhenTheWindowElapses() + { + // Two events arrive (a backlog, so the pump lingers), then the source goes quiet + // well short of the size cap. The window — not the cap — must close the batch, + // and it must do so within a bounded time rather than waiting for a 100th event + // that never comes. + var window = TimeSpan.FromMilliseconds(120); + var channel = Channel.CreateUnbounded(); + channel.Writer.TryWrite(Event(0)); + channel.Writer.TryWrite(Event(1)); + + var started = Stopwatch.GetTimestamp(); + var run = StartPump(channel.Reader, maxBatchEvents: 100, window); + + SiteStreamEvent frame; + while (true) + { + lock (run.Frames) + { + if (run.Frames.Count > 0) { frame = run.Frames[0]; break; } + } + Assert.True(Stopwatch.GetElapsedTime(started) < TimeSpan.FromSeconds(5), + "the window never closed the underfull batch"); + await Task.Delay(5); + } + + var elapsed = Stopwatch.GetElapsedTime(started); + + channel.Writer.Complete(); + await run.Completion; + + Assert.Equal(SiteStreamEvent.EventOneofCase.Batch, frame.EventCase); + Assert.Equal([0, 1], Seqs(frame)); + // The batch waited (it did not close instantly on the two queued events) and it + // closed on the window, not on a cap it never reached. + Assert.True(elapsed >= window - TimeSpan.FromMilliseconds(20), + $"batch closed after {elapsed.TotalMilliseconds:0.0} ms, before the {window.TotalMilliseconds:0} ms window"); + } + + [Fact] + public async Task LoneEventOnAQuietStream_IsNeverDelayedByTheWindow() + { + // The latency contract: the window applies only AFTER a backlog has been observed. + // A single event on an idle stream must be emitted immediately as a plain frame, + // so per-event latency on a quiet site is unchanged by batching. + var window = TimeSpan.FromSeconds(5); + var channel = Channel.CreateUnbounded(); + var run = StartPump(channel.Reader, maxBatchEvents: 100, window); + + var started = Stopwatch.GetTimestamp(); + channel.Writer.TryWrite(Event(7)); + + while (true) + { + lock (run.Frames) + { + if (run.Frames.Count > 0) break; + } + Assert.True(Stopwatch.GetElapsedTime(started) < TimeSpan.FromSeconds(3), + "a lone event was held by the coalescing window"); + await Task.Delay(2); + } + + var elapsed = Stopwatch.GetElapsedTime(started); + channel.Writer.Complete(); + await run.Completion; + + Assert.Equal(SiteStreamEvent.EventOneofCase.AttributeChanged, run.Frames[0].EventCase); + Assert.True(elapsed < TimeSpan.FromSeconds(1), + $"lone event took {elapsed.TotalMilliseconds:0.0} ms against a {window.TotalSeconds:0} s window"); + } + + // ── Flush on stream close ─────────────────────────────────────────────────── + + [Fact] + public async Task WriterCompletion_FlushesTheInFlightBatchBeforeReturning() + { + // The channel writer completing mid-window (the site stopping the relay actor and + // calling channel.Writer.TryComplete()) must flush what is already buffered rather + // than silently discarding it while waiting out the window. + var channel = Channel.CreateUnbounded(); + channel.Writer.TryWrite(Event(0)); + channel.Writer.TryWrite(Event(1)); + + // A long window guarantees the pump is lingering, not already past the emit. + var run = StartPump(channel.Reader, maxBatchEvents: 100, window: TimeSpan.FromSeconds(30)); + + await Task.Delay(100); + lock (run.Frames) + { + Assert.Empty(run.Frames); // still lingering + } + + channel.Writer.Complete(); + await run.Completion.WaitAsync(TimeSpan.FromSeconds(5)); + + Assert.Single(run.Frames); + Assert.Equal([0, 1], Seqs(run.Frames[0])); + } + + [Fact] + public async Task WriterCompletionWithNothingBuffered_ReturnsWithoutEmitting() + { + var channel = Channel.CreateUnbounded(); + var run = StartPump(channel.Reader, maxBatchEvents: 100, window: TimeSpan.FromMilliseconds(25)); + + channel.Writer.Complete(); + await run.Completion.WaitAsync(TimeSpan.FromSeconds(5)); + + Assert.Empty(run.Frames); + } + + [Fact] + public async Task Cancellation_EndsThePumpWithOperationCanceled() + { + // Client disconnect / duplicate replacement / site shutdown. The pump must + // surface OperationCanceledException exactly as the pre-batching await-foreach + // loop did, so SiteStreamGrpcServer's existing catch and finally are unchanged. + var channel = Channel.CreateUnbounded(); + using var cts = new CancellationTokenSource(); + var run = StartPump(channel.Reader, maxBatchEvents: 100, window: TimeSpan.FromMilliseconds(25), cts.Token); + + await cts.CancelAsync(); + + await Assert.ThrowsAnyAsync( + () => run.Completion.WaitAsync(TimeSpan.FromSeconds(5))); + } + + // ── Ordering ──────────────────────────────────────────────────────────────── + + [Fact] + public async Task Ordering_IsPreservedAcrossManyBatchesUnderAProducerRace() + { + // Batching is a framing change and nothing else: with a producer writing + // concurrently with the pump, the flattened output must be the exact input + // sequence — no reordering, no loss, no duplication, across many frames. + const int total = 5_000; + var channel = Channel.CreateUnbounded(); + var run = StartPump(channel.Reader, maxBatchEvents: 32, window: TimeSpan.FromMilliseconds(5)); + + var producer = Task.Run(async () => + { + for (var i = 0; i < total; i++) + { + channel.Writer.TryWrite(Event(i)); + if (i % 250 == 0) await Task.Yield(); + } + channel.Writer.Complete(); + }); + + await producer; + await run.Completion.WaitAsync(TimeSpan.FromSeconds(30)); + + Assert.Equal(Enumerable.Range(0, total), run.Frames.SelectMany(Seqs)); + Assert.All(run.FrameSizes, s => Assert.InRange(s, 1, 32)); + Assert.Equal(total, run.FrameSizes.Sum()); + } + + // ── Frame shape ───────────────────────────────────────────────────────────── + + [Fact] + public async Task BatchFrame_CarriesTheCorrelationIdOnceAndBlanksItOnInnerEvents() + { + // The byte saving batching exists for: the correlation id is stamped once on the + // enclosing frame, not repeated on every packed event. No consumer reads the + // inner value (SiteStreamGrpcClient.ForEachEvent ignores it). + var channel = Channel.CreateUnbounded(); + for (var i = 0; i < 4; i++) channel.Writer.TryWrite(Event(i)); + channel.Writer.Complete(); + + var run = StartPump(channel.Reader, maxBatchEvents: 100, window: TimeSpan.FromMilliseconds(25)); + await run.Completion; + + var frame = Assert.Single(run.Frames); + Assert.Equal(SiteStreamEvent.EventOneofCase.Batch, frame.EventCase); + Assert.Equal(Corr, frame.CorrelationId); + Assert.All(frame.Batch.Events, e => Assert.Equal(string.Empty, e.CorrelationId)); + } + + [Fact] + public async Task PerEventTimestampsSurviveBatching() + { + // End-to-end latency measurement rides the per-event Timestamp; coalescing must + // not rewrite it to a single frame-level stamp. + var channel = Channel.CreateUnbounded(); + for (var i = 0; i < 3; i++) channel.Writer.TryWrite(Event(i)); + channel.Writer.Complete(); + + var run = StartPump(channel.Reader, maxBatchEvents: 100, window: TimeSpan.FromMilliseconds(25)); + await run.Completion; + + var frame = Assert.Single(run.Frames); + Assert.Equal( + [ + DateTimeOffset.UnixEpoch, + DateTimeOffset.UnixEpoch.AddSeconds(1), + DateTimeOffset.UnixEpoch.AddSeconds(2) + ], + frame.Batch.Events.Select(e => e.AttributeChanged.Timestamp.ToDateTimeOffset())); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcClientTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcClientTests.cs index 331ce023..84978f08 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcClientTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcClientTests.cs @@ -508,6 +508,125 @@ public class SiteStreamGrpcClientTests } } + // ── R2: batch unpacking on the client ─────────────────────────────────────── + + private static SiteStreamEvent Attr(string value, DateTimeOffset ts) => new() + { + AttributeChanged = new AttributeValueUpdate + { + InstanceUniqueName = "SiteA.Pump01", + AttributePath = "Modules.IO", + AttributeName = "Seq", + Value = value, + Quality = Quality.Good, + Timestamp = Timestamp.FromDateTimeOffset(ts) + } + }; + + [Fact] + public void ForEachEvent_PlainFrame_IsDeliveredAsIs() + { + // An OLD SITE (or any un-negotiated stream) sends one event per frame. The new + // client's unpack path must pass it straight through — this is the new-central ↔ + // old-site skew direction. + var frame = Attr("1", DateTimeOffset.UnixEpoch); + var seen = new List(); + + SiteStreamGrpcClient.ForEachEvent(frame, seen.Add); + + Assert.Same(frame, Assert.Single(seen)); + } + + [Fact] + public void ForEachEvent_BatchFrame_UnpacksInOrderPreservingPerEventTimestamps() + { + // Order and per-event Timestamp fidelity are the two properties the downstream + // consumers (SiteAlarmAggregatorActor, DebugStreamBridgeActor) and the end-to-end + // latency measurement depend on. + var t0 = new DateTimeOffset(2026, 8, 15, 9, 0, 0, TimeSpan.Zero); + var frame = new SiteStreamEvent + { + CorrelationId = "corr-batch", + Batch = new SiteStreamEventBatch + { + Events = + { + Attr("0", t0), + Attr("1", t0.AddMilliseconds(3)), + Attr("2", t0.AddMilliseconds(11)) + } + } + }; + + var seen = new List(); + SiteStreamGrpcClient.ForEachEvent(frame, seen.Add); + + Assert.Equal(["0", "1", "2"], seen.Select(e => e.AttributeChanged.Value)); + Assert.Equal( + [t0, t0.AddMilliseconds(3), t0.AddMilliseconds(11)], + seen.Select(e => e.AttributeChanged.Timestamp.ToDateTimeOffset())); + } + + [Fact] + public void ForEachEvent_IgnoresNestedAndUnknownInnerCases() + { + // The server never nests a batch inside a batch. A nested (or empty) inner frame + // from a malformed or hostile peer must be skipped, not followed — unpacking is + // deliberately non-recursive so a crafted frame cannot drive unbounded recursion. + var frame = new SiteStreamEvent + { + CorrelationId = "corr-nested", + Batch = new SiteStreamEventBatch + { + Events = + { + Attr("0", DateTimeOffset.UnixEpoch), + new SiteStreamEvent { Batch = new SiteStreamEventBatch { Events = { Attr("hidden", DateTimeOffset.UnixEpoch) } } }, + new SiteStreamEvent(), + Attr("1", DateTimeOffset.UnixEpoch) + } + } + }; + + var seen = new List(); + SiteStreamGrpcClient.ForEachEvent(frame, seen.Add); + + Assert.Equal(["0", "1"], seen.Select(e => e.AttributeChanged.Value)); + } + + [Fact] + public async Task ConsumeStream_MixedBatchedAndPlainFrames_DeliverEveryEventInOrder() + { + // A reconnect can straddle a site upgrade, so one stream may legitimately carry + // both frame shapes. Driving the real ConsumeStreamAsync with the real unpack + // proves the combination is flat and ordered from the consumer's point of view. + var client = SiteStreamGrpcClient.CreateForTesting(); + var cts = new CancellationTokenSource(); + var delivered = new List(); + + void Deliver(SiteStreamEvent e) => delivered.Add(e.AttributeChanged.Value); + + await client.ConsumeStreamAsync( + "corr-mixed", + cts, + () => FakeCall(new StubStreamReader( + Attr("0", DateTimeOffset.UnixEpoch), + new SiteStreamEvent + { + CorrelationId = "corr-mixed", + Batch = new SiteStreamEventBatch + { + Events = { Attr("1", DateTimeOffset.UnixEpoch), Attr("2", DateTimeOffset.UnixEpoch) } + } + }, + Attr("3", DateTimeOffset.UnixEpoch))), + frame => SiteStreamGrpcClient.ForEachEvent(frame, Deliver), + _ => { }, + () => { }); + + Assert.Equal(["0", "1", "2", "3"], delivered); + } + private static AsyncServerStreamingCall FakeCall(StubStreamReader reader) => FakeCall(reader, Task.FromResult(new Metadata())); diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcServerTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcServerTests.cs index 7ab9bc72..43a3c331 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcServerTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcServerTests.cs @@ -580,4 +580,277 @@ public class SiteStreamGrpcServerTests : TestKit var server = CreateServer(); Assert.Equal(0, server.DroppedStreamEventCount); } + + [Fact] + public async Task DuplicateReplacement_CancelsTheReplacedStream_WithoutDisposingItsCts() + { + // Regression: the duplicate-replacement path used to Cancel AND Dispose the + // replaced stream's CancellationTokenSource. That CTS belongs to the replaced + // handler's own `using var streamCts`, which is still running and still has to + // read `streamCts.Token` — so the Dispose raced that read and escaped the RPC as + // an unhandled ObjectDisposedException. It surfaced only under full-suite load + // (GrpcStreamIntegrationTests.Pipeline_DuplicateCorrelationId_ReplacesStream) and + // predates R2: the same Dispose and the same first-token-read relationship existed + // when the handler still used `ReadAllAsync(streamCts.Token)`. + // + // The race is made DETERMINISTIC here by gating the first stream inside its setup + // window (its _activeStreams entry is registered before Subscribe is called), so + // the replacement always lands before the first stream reads its token. + using var gate = new ManualResetEventSlim(false); + var calls = 0; + var subscriber = Substitute.For(); + subscriber.Subscribe(Arg.Any(), Arg.Any()) + .Returns(ci => + { + var n = Interlocked.Increment(ref calls); + if (n == 1) + gate.Wait(TimeSpan.FromSeconds(15)); + return $"sub-dup-race-{n}"; + }); + + var server = new SiteStreamGrpcServer(subscriber, _logger); + server.SetReady(Sys); + + using var cts1 = new CancellationTokenSource(); + var stream1 = Task.Run(() => server.SubscribeInstance( + MakeRequest("corr-dup-race"), + Substitute.For>(), + CreateMockContext(cts1.Token))); + + await WaitForConditionAsync(() => server.ActiveStreamCount == 1); + await WaitForConditionAsync(() => Volatile.Read(ref calls) == 1); + + using var cts2 = new CancellationTokenSource(); + var stream2 = Task.Run(() => server.SubscribeInstance( + MakeRequest("corr-dup-race"), + Substitute.For>(), + CreateMockContext(cts2.Token))); + + // The replacement has taken the slot (and cancelled stream 1's CTS) by the time + // its own Subscribe has been called. + await WaitForConditionAsync(() => Volatile.Read(ref calls) == 2); + + gate.Set(); + + // Pre-fix this threw ObjectDisposedException out of the RPC. Post-fix the replaced + // stream observes a plain cancellation and unwinds through its normal finally. + await stream1; + + cts2.Cancel(); + await stream2; + + Assert.Equal(0, server.ActiveStreamCount); + } + + // ── R2: gRPC event batching, and its negotiation ──────────────────────────── + + [Fact] + public void BatchOptions_AreBoundFromOptions_AndClampDegenerateValues() + { + var options = Microsoft.Extensions.Options.Options.Create(new CommunicationOptions()); + var server = new SiteStreamGrpcServer(_subscriber, _logger, options); + + Assert.Equal(100, server.StreamBatchMaxEvents); + Assert.Equal(TimeSpan.FromMilliseconds(25), server.StreamBatchWindow); + + // CommunicationOptionsValidator fails the boot on these, but a host composed + // without validation must not blow up deep inside a live RPC. + var degenerate = new SiteStreamGrpcServer(_subscriber, _logger, + Microsoft.Extensions.Options.Options.Create(new CommunicationOptions + { + GrpcStreamBatchMaxEvents = 0, + GrpcStreamBatchWindow = TimeSpan.FromMilliseconds(-5), + })); + Assert.Equal(1, degenerate.StreamBatchMaxEvents); + Assert.Equal(TimeSpan.Zero, degenerate.StreamBatchWindow); + } + + [Fact] + public async Task UnnegotiatedSubscription_NeverEmitsABatchFrame() + { + // OLD-CENTRAL ↔ NEW-SITE skew. proto3 defaults batching_supported to false, which + // is exactly what a central built before R2 sends. The site must then keep to one + // event per frame — a Batch frame would arrive at that central as + // EventOneofCase.None and be silently dropped by its ConvertToDomainEvent. + var (server, capture, cts, streamTask, relay) = + await StartCapturingStreamAsync(batchingSupported: false); + + for (var i = 0; i < 50; i++) + { + relay.Tell(new Commons.Messages.Streaming.AttributeValueChanged( + "Site1.Pump01", "Path", "Attr", i, "Good", DateTimeOffset.UtcNow)); + } + + await WaitForConditionAsync(() => CountEvents(capture) >= 50, 10_000); + + cts.Cancel(); + await streamTask; + + lock (capture) + { + Assert.All(capture, f => Assert.NotEqual(SiteStreamEvent.EventOneofCase.Batch, f.EventCase)); + Assert.Equal(50, capture.Count); + } + + GC.KeepAlive(server); + } + + [Fact] + public async Task NegotiatedSubscription_CoalescesABacklogIntoFewerFramesThanEvents() + { + // NEW-CENTRAL ↔ NEW-SITE. A burst pushed at the relay faster than the pump drains + // it must come out in strictly fewer frames than events, with every event + // preserved in order. + const int burst = 400; + var (server, capture, cts, streamTask, relay) = + await StartCapturingStreamAsync(batchingSupported: true); + + for (var i = 0; i < burst; i++) + { + relay.Tell(new Commons.Messages.Streaming.AttributeValueChanged( + "Site1.Pump01", "Path", "Attr", i, "Good", DateTimeOffset.UtcNow)); + } + + await WaitForConditionAsync(() => CountEvents(capture) >= burst, 15_000); + + cts.Cancel(); + await streamTask; + + List frames; + lock (capture) { frames = [.. capture]; } + + Assert.Equal(burst, frames.Sum(CountFrameEvents)); + Assert.True(frames.Count < burst, + $"batching produced {frames.Count} frames for {burst} events — no coalescing happened"); + Assert.Contains(frames, f => f.EventCase == SiteStreamEvent.EventOneofCase.Batch); + + // Order is preserved end to end: the values arrive 0..burst-1 exactly once each. + var values = frames.SelectMany(FlattenAttributeValues).ToArray(); + Assert.Equal(Enumerable.Range(0, burst).Select(i => i.ToString()), values); + + GC.KeepAlive(server); + } + + [Fact] + public async Task BatchSizeHistogram_IsRecordedOnlyForNegotiatedStreams() + { + // scadabridge.site.stream.batch_size rides ScadaBridgeTelemetry.MeterName, which is + // already in SiteServiceRegistration.ObservedMeters — an unlisted meter exports + // nothing, silently. Assert the instrument actually fires, and that it does NOT + // fire on an un-negotiated stream (where it would degenerate into a per-event + // instrument on the hottest path in the product). + var measurements = new List(); + using var listener = new MeterListener(); + listener.InstrumentPublished = (instrument, l) => + { + if (instrument.Meter.Name == ScadaBridgeTelemetry.MeterName && + instrument.Name == "scadabridge.site.stream.batch_size") + { + l.EnableMeasurementEvents(instrument); + } + }; + listener.SetMeasurementEventCallback((_, m, _, _) => + { + lock (measurements) { measurements.Add(m); } + }); + listener.Start(); + + // Un-negotiated: no measurements at all. + var (_, plainCapture, plainCts, plainTask, plainRelay) = + await StartCapturingStreamAsync(batchingSupported: false, correlationId: "corr-hist-off"); + plainRelay.Tell(new Commons.Messages.Streaming.AttributeValueChanged( + "Site1.Pump01", "Path", "Attr", 1, "Good", DateTimeOffset.UtcNow)); + await WaitForConditionAsync(() => CountEvents(plainCapture) >= 1); + plainCts.Cancel(); + await plainTask; + + lock (measurements) { Assert.Empty(measurements); } + + // Negotiated: one measurement per emitted frame, each within the size cap. + var (_, capture, cts, streamTask, relay) = + await StartCapturingStreamAsync(batchingSupported: true, correlationId: "corr-hist-on"); + for (var i = 0; i < 20; i++) + { + relay.Tell(new Commons.Messages.Streaming.AttributeValueChanged( + "Site1.Pump01", "Path", "Attr", i, "Good", DateTimeOffset.UtcNow)); + } + await WaitForConditionAsync(() => CountEvents(capture) >= 20, 10_000); + cts.Cancel(); + await streamTask; + + lock (measurements) + { + Assert.NotEmpty(measurements); + Assert.Equal(20, measurements.Sum()); + Assert.All(measurements, m => Assert.InRange(m, 1, SiteStreamGrpcServer.DefaultStreamBatchMaxEvents)); + } + } + + /// Total events carried across all captured frames (unpacking batch frames). + private static int CountEvents(List capture) + { + lock (capture) { return capture.Sum(CountFrameEvents); } + } + + private static int CountFrameEvents(SiteStreamEvent frame) => + frame.EventCase == SiteStreamEvent.EventOneofCase.Batch ? frame.Batch.Events.Count : 1; + + private static IEnumerable FlattenAttributeValues(SiteStreamEvent frame) + { + if (frame.EventCase == SiteStreamEvent.EventOneofCase.Batch) + { + foreach (var inner in frame.Batch.Events) + yield return inner.AttributeChanged.Value; + yield break; + } + + yield return frame.AttributeChanged.Value; + } + + /// + /// Starts a SubscribeInstance stream with the given batch negotiation, capturing every + /// written frame and handing back the relay actor so the test can drive domain events. + /// + private async Task<(SiteStreamGrpcServer Server, List Capture, + CancellationTokenSource Cts, Task StreamTask, IActorRef Relay)> + StartCapturingStreamAsync(bool batchingSupported, string correlationId = "corr-batch") + { + IActorRef? capturedActor = null; + var subscriber = Substitute.For(); + subscriber.Subscribe(Arg.Any(), Arg.Any()) + .Returns(ci => + { + capturedActor = ci.Arg(); + return "sub-batch"; + }); + + var server = new SiteStreamGrpcServer(subscriber, _logger, + Microsoft.Extensions.Options.Options.Create(new CommunicationOptions())); + server.SetReady(Sys); + + var capture = new List(); + var writer = Substitute.For>(); + writer.WriteAsync(Arg.Any(), Arg.Any()) + .Returns(Task.CompletedTask) + .AndDoes(ci => + { + var frame = ci.Arg(); + lock (capture) { capture.Add(frame); } + }); + + var cts = new CancellationTokenSource(); + var context = CreateMockContext(cts.Token); + + var request = new InstanceStreamRequest + { + CorrelationId = correlationId, + InstanceUniqueName = "Site1.Pump01", + BatchingSupported = batchingSupported + }; + + var streamTask = Task.Run(() => server.SubscribeInstance(request, writer, context)); + await WaitForConditionAsync(() => capturedActor != null); + + return (server, capture, cts, streamTask, capturedActor!); + } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Grpc/GrpcStreamBatchingIntegrationTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Grpc/GrpcStreamBatchingIntegrationTests.cs new file mode 100644 index 00000000..650b777d --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Grpc/GrpcStreamBatchingIntegrationTests.cs @@ -0,0 +1,436 @@ +using System.Diagnostics; +using Akka.Actor; +using Akka.TestKit.Xunit2; +using Google.Protobuf; +using Grpc.Core; +using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Options; +using NSubstitute; +using Xunit.Abstractions; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; +using ZB.MOM.WW.ScadaBridge.Communication; +using ZB.MOM.WW.ScadaBridge.Communication.Grpc; +using ZB.MOM.WW.ScadaBridge.SiteRuntime; +using ZB.MOM.WW.ScadaBridge.SiteRuntime.Streaming; + +namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Grpc; + +/// +/// End-to-end coverage for R2 — gRPC event batching on the site→central +/// SiteStreamService stream. +/// +/// +/// The chain assembled here is the real one, mocking only the HTTP/2 transport: +/// domain event → real broadcast → real +/// handler → real StreamRelayActor → real +/// coalescing pump → proto serialize/parse round-trip (what the wire actually +/// carries) → real unpack → real +/// ConvertToDomainEvent. The serialize/parse step is what makes these +/// version-skew claims real rather than in-memory object graph assertions. +/// +/// +/// +/// Version skew is covered in both directions. Negotiation is a single additive +/// request field (batching_supported), whose proto3 default of false IS the +/// compatibility mechanism: an old central cannot set it, so a new site never sends it a +/// frame case its generated code would drop; a new central always sets it, and an old +/// site ignores the unknown field and keeps sending per-event frames the new client +/// accepts unchanged. +/// +/// +public class GrpcStreamBatchingIntegrationTests(ITestOutputHelper output) : TestKit +{ + private const string Instance = "SiteA.Pump01"; + + /// + /// End-to-end latency threshold the target-scale load test asserts a P99 against + /// (measured P99 there: 4.57 ms). The coalescing window is the only latency batching + /// can add, so the batched pipe must stay comfortably inside the same budget. + /// + private static readonly TimeSpan LatencyThreshold = TimeSpan.FromMilliseconds(250); + + // ── Round trip: batched frames deliver every event, in order ──────────────── + + [Fact] + public async Task NegotiatedStream_RoundTripsEveryEventThroughTheWire_InOrder() + { + var (server, manager, frames, cts, streamTask) = await StartAsync(batchingSupported: true); + + const int total = 600; + var t0 = new DateTimeOffset(2026, 8, 15, 12, 0, 0, TimeSpan.Zero); + for (var i = 0; i < total; i++) + { + manager.PublishAttributeValueChanged(new AttributeValueChanged( + Instance, "Modules.IO", "Seq", i, "Good", t0.AddMilliseconds(i))); + } + + await WaitForConditionAsync(() => TotalEvents(frames) >= total, 30_000); + cts.Cancel(); + await streamTask; + + var wire = SnapshotThroughTheWire(frames); + + // Batching actually happened — otherwise this test proves nothing about batching. + Assert.Contains(wire, f => f.EventCase == SiteStreamEvent.EventOneofCase.Batch); + Assert.True(wire.Count < total, + $"{wire.Count} frames for {total} events — no coalescing happened"); + + var delivered = Unpack(wire); + Assert.Equal(total, delivered.Count); + + // Every event, exactly once, in the order the site produced it — and with its OWN + // timestamp, not a frame-level one (end-to-end latency measurement rides it). + Assert.Equal( + Enumerable.Range(0, total).Select(i => i.ToString()), + delivered.Select(e => e.Value)); + Assert.Equal( + Enumerable.Range(0, total).Select(i => t0.AddMilliseconds(i)), + delivered.Select(e => e.Timestamp)); + + output.WriteLine($"round-trip: {total} events in {wire.Count} frames " + + $"(mean {(double)total / wire.Count:0.0} events/frame)"); + + GC.KeepAlive(server); + } + + // ── Latency cost of the default window ───────────────────────────────────── + + [Fact] + public async Task DefaultWindow_KeepsPerEventLatencyFarBelowTheThreshold() + { + // A trickle-with-backlog workload is the case the coalescing window actually + // bites on: each burst is far short of the 100-event size cap, so the batch is + // closed by the 25 ms window rather than by the cap. That makes this the WORST + // case for added latency, not the best. + var options = new CommunicationOptions(); + var (server, manager, frames, cts, streamTask) = await StartAsync( + batchingSupported: true, options: options); + + const int bursts = 150; + const int perBurst = 8; + var stamps = new Dictionary(); + + var seq = 0; + for (var b = 0; b < bursts; b++) + { + for (var i = 0; i < perBurst; i++) + { + var ts = DateTimeOffset.UtcNow; + stamps[seq] = ts; + manager.PublishAttributeValueChanged(new AttributeValueChanged( + Instance, "Modules.IO", "Seq", seq, "Good", ts)); + seq++; + } + await Task.Delay(5); + } + + var total = seq; + await WaitForConditionAsync(() => TotalEvents(frames) >= total, 60_000); + cts.Cancel(); + await streamTask; + + // Latency = the event's own site-side timestamp → the instant the frame carrying + // it was handed to the response stream. That interval contains the coalescing + // window and nothing else the pre-batching pipe did not already have. + var latencies = new List(); + lock (frames) + { + foreach (var (frame, writtenAt) in frames) + { + foreach (var evt in Flatten(frame)) + { + var s = int.Parse(evt.AttributeChanged.Value); + latencies.Add((writtenAt - stamps[s]).TotalMilliseconds); + } + } + } + + latencies.Sort(); + var p50 = latencies[(int)(latencies.Count * 0.50)]; + var p99 = latencies[(int)(latencies.Count * 0.99)]; + var max = latencies[^1]; + + output.WriteLine( + $"window={options.GrpcStreamBatchWindow.TotalMilliseconds:0} ms cap={options.GrpcStreamBatchMaxEvents} " + + $"events={latencies.Count} P50={p50:0.00} ms P99={p99:0.00} ms max={max:0.00} ms"); + + Assert.Equal(total, latencies.Count); + Assert.True(p99 < LatencyThreshold.TotalMilliseconds, + $"P99 {p99:0.00} ms exceeded the {LatencyThreshold.TotalMilliseconds:0} ms end-to-end threshold"); + + GC.KeepAlive(server); + } + + // ── Version skew: OLD central ↔ NEW site ─────────────────────────────────── + + [Fact] + public async Task OldCentral_AgainstNewSite_NeverReceivesABatchFrame() + { + // An old central's InstanceStreamRequest bytes simply have no field 3 — build + // exactly those bytes and let the NEW site parse them, so the negotiation default + // is exercised off the wire rather than asserted on an object. + var oldCentralBytes = BuildLegacyInstanceRequest("corr-old-central", Instance); + var request = InstanceStreamRequest.Parser.ParseFrom(oldCentralBytes); + Assert.False(request.BatchingSupported); + + var (server, manager, frames, cts, streamTask) = await StartAsync(request); + + const int total = 300; + for (var i = 0; i < total; i++) + { + manager.PublishAttributeValueChanged(new AttributeValueChanged( + Instance, "Modules.IO", "Seq", i, "Good", DateTimeOffset.UtcNow)); + } + + await WaitForConditionAsync(() => TotalEvents(frames) >= total, 30_000); + cts.Cancel(); + await streamTask; + + var wire = SnapshotThroughTheWire(frames); + + // One event per frame, and — checked at the byte level, since that is what the + // old peer's parser sees — never the field-4 batch tag. + Assert.Equal(total, wire.Count); + Assert.All(wire, f => + Assert.Equal(SiteStreamEvent.EventOneofCase.AttributeChanged, f.EventCase)); + Assert.All(wire, f => Assert.DoesNotContain(4, FieldNumbers(f))); + + GC.KeepAlive(server); + } + + [Fact] + public void BatchFrameRidesFieldFour_WhichAPreBatchingParserWouldDropSilently() + { + // WHY negotiation is mandatory rather than "just send batches". A batch frame is a + // length-delimited field 4: an older generated parser skips it into unknown fields + // and reports EventOneofCase.None, whose ConvertToDomainEvent returns null — the + // whole batch would vanish with no error anywhere. The proto3 default on + // batching_supported is what guarantees such a peer never receives one. + var batch = new SiteStreamEvent + { + CorrelationId = "corr-shape", + Batch = new SiteStreamEventBatch + { + Events = { MakeAttributeEvent(1), MakeAttributeEvent(2) } + } + }; + + var fields = FieldNumbers(batch); + Assert.Contains(4, fields); + Assert.DoesNotContain(2, fields); + Assert.DoesNotContain(3, fields); + + // Field 4 is length-delimited (wire type 2) — the shape an unknown-field-tolerant + // parser can skip without corrupting the rest of the message. + Assert.Equal(2u, WireTypeOfField(batch, 4)); + + // And the per-event frames a pre-batching site emits still parse and convert on the + // NEW client (the other skew direction, at the same byte level). + var plain = SiteStreamEvent.Parser.ParseFrom(MakeAttributeEvent(7).ToByteArray()); + Assert.NotNull(SiteStreamGrpcClient.ConvertToDomainEvent(plain)); + } + + // ── Version skew: NEW central ↔ OLD site ─────────────────────────────────── + + [Fact] + public async Task NewCentral_AgainstOldSite_StillReceivesEveryEvent() + { + // An old site ignores batching_supported and emits per-event frames. That emission + // shape is exactly what the current server produces with batching off, so drive + // the real server that way and feed the result through the NEW client's unpack — + // which must handle the single-event case identically to before R2. + var (server, manager, frames, cts, streamTask) = await StartAsync(batchingSupported: false); + + const int total = 200; + var t0 = new DateTimeOffset(2026, 8, 15, 13, 0, 0, TimeSpan.Zero); + for (var i = 0; i < total; i++) + { + manager.PublishAttributeValueChanged(new AttributeValueChanged( + Instance, "Modules.IO", "Seq", i, "Good", t0.AddMilliseconds(i))); + } + + await WaitForConditionAsync(() => TotalEvents(frames) >= total, 30_000); + cts.Cancel(); + await streamTask; + + var wire = SnapshotThroughTheWire(frames); + Assert.Equal(total, wire.Count); + + var delivered = Unpack(wire); + Assert.Equal( + Enumerable.Range(0, total).Select(i => i.ToString()), + delivered.Select(e => e.Value)); + Assert.Equal( + Enumerable.Range(0, total).Select(i => t0.AddMilliseconds(i)), + delivered.Select(e => e.Timestamp)); + + GC.KeepAlive(server); + } + + // ── Helpers ──────────────────────────────────────────────────────────────── + + private static SiteStreamEvent MakeAttributeEvent(int seq) => new() + { + CorrelationId = "corr-shape", + AttributeChanged = new AttributeValueUpdate + { + InstanceUniqueName = Instance, + AttributePath = "Modules.IO", + AttributeName = "Seq", + Value = seq.ToString(), + Quality = Quality.Good, + Timestamp = Google.Protobuf.WellKnownTypes.Timestamp.FromDateTimeOffset(DateTimeOffset.UnixEpoch) + } + }; + + /// + /// Serializes an InstanceStreamRequest the way a central built BEFORE R2 would: + /// fields 1 and 2 only, with no batching_supported on the wire at all. + /// + private static byte[] BuildLegacyInstanceRequest(string correlationId, string instance) + { + using var ms = new MemoryStream(); + var output = new CodedOutputStream(ms); + output.WriteTag(1, WireFormat.WireType.LengthDelimited); + output.WriteString(correlationId); + output.WriteTag(2, WireFormat.WireType.LengthDelimited); + output.WriteString(instance); + output.Flush(); + return ms.ToArray(); + } + + /// Top-level field numbers present in a serialized message. + private static HashSet FieldNumbers(IMessage message) + { + var fields = new HashSet(); + var input = new CodedInputStream(message.ToByteArray()); + uint tag; + while ((tag = input.ReadTag()) != 0) + { + fields.Add(WireFormat.GetTagFieldNumber(tag)); + input.SkipLastField(); + } + return fields; + } + + /// Wire type of the given top-level field number in a serialized message. + private static uint WireTypeOfField(IMessage message, int fieldNumber) + { + var input = new CodedInputStream(message.ToByteArray()); + uint tag; + while ((tag = input.ReadTag()) != 0) + { + if (WireFormat.GetTagFieldNumber(tag) == fieldNumber) + return (uint)WireFormat.GetTagWireType(tag); + input.SkipLastField(); + } + throw new InvalidOperationException($"field {fieldNumber} not present"); + } + + private static IEnumerable Flatten(SiteStreamEvent frame) + { + if (frame.EventCase == SiteStreamEvent.EventOneofCase.Batch) + { + foreach (var inner in frame.Batch.Events) yield return inner; + yield break; + } + yield return frame; + } + + private static int TotalEvents(List<(SiteStreamEvent Frame, DateTimeOffset WrittenAt)> frames) + { + lock (frames) { return frames.Sum(f => Flatten(f.Frame).Count()); } + } + + /// + /// Takes the captured frames through a real protobuf serialize/parse round-trip — the + /// step that makes every claim in this file about wire compatibility a wire claim. + /// + private static List SnapshotThroughTheWire( + List<(SiteStreamEvent Frame, DateTimeOffset WrittenAt)> frames) + { + lock (frames) + { + return [.. frames.Select(f => SiteStreamEvent.Parser.ParseFrom(f.Frame.ToByteArray()))]; + } + } + + /// Unpacks wire frames through the REAL client path into domain events. + private static List Unpack(IEnumerable wire) + { + var delivered = new List(); + foreach (var frame in wire) + { + SiteStreamGrpcClient.ForEachEvent(frame, e => + { + if (SiteStreamGrpcClient.ConvertToDomainEvent(e) is AttributeValueChanged a) + delivered.Add(a); + }); + } + return delivered; + } + + private Task<(SiteStreamGrpcServer Server, SiteStreamManager Manager, + List<(SiteStreamEvent Frame, DateTimeOffset WrittenAt)> Frames, + CancellationTokenSource Cts, Task StreamTask)> + StartAsync(bool batchingSupported, CommunicationOptions? options = null) + => StartAsync(new InstanceStreamRequest + { + CorrelationId = "corr-batching", + InstanceUniqueName = Instance, + BatchingSupported = batchingSupported + }, options); + + /// + /// Brings up a real site broadcast hub + real gRPC server handler for the supplied + /// subscription request, capturing every written frame with the instant it was written. + /// + private async Task<(SiteStreamGrpcServer Server, SiteStreamManager Manager, + List<(SiteStreamEvent Frame, DateTimeOffset WrittenAt)> Frames, + CancellationTokenSource Cts, Task StreamTask)> + StartAsync(InstanceStreamRequest request, CommunicationOptions? options = null) + { + var manager = new SiteStreamManager( + new SiteRuntimeOptions { StreamBufferSize = 4096 }, + NullLogger.Instance); + manager.Initialize(Sys); + + var server = new SiteStreamGrpcServer( + manager, + NullLogger.Instance, + Options.Create(options ?? new CommunicationOptions())); + server.SetReady(Sys); + + var frames = new List<(SiteStreamEvent, DateTimeOffset)>(); + var writer = Substitute.For>(); + writer.WriteAsync(Arg.Any(), Arg.Any()) + .Returns(Task.CompletedTask) + .AndDoes(ci => + { + var frame = ci.Arg(); + var at = DateTimeOffset.UtcNow; + lock (frames) { frames.Add((frame, at)); } + }); + + var cts = new CancellationTokenSource(); + var context = Substitute.For(); + context.CancellationToken.Returns(cts.Token); + + var streamTask = Task.Run(() => server.SubscribeInstance(request, writer, context)); + + // The publish must not race the materialized subscription. + await WaitForConditionAsync(() => manager.SubscriptionCount == 1); + + return (server, manager, frames, cts, streamTask); + } + + private static async Task WaitForConditionAsync(Func condition, int timeoutMs = 5000) + { + var started = Stopwatch.GetTimestamp(); + while (!condition() && Stopwatch.GetElapsedTime(started) < TimeSpan.FromMilliseconds(timeoutMs)) + { + await Task.Delay(10); + } + Assert.True(condition(), $"Condition not met within {timeoutMs}ms"); + } +}