diff --git a/CLAUDE.md b/CLAUDE.md index e6748306..b8ce6646 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -124,7 +124,7 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): ` - Inter-cluster communication uses two transports: ClusterClient for command/control (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots) and gRPC server-streaming for real-time data (attribute values, alarm states). Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist. Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only. - gRPC streaming channel: SiteStreamGrpcServer on each site node (Kestrel HTTP/2, port 8083); central creates per-site SiteStreamGrpcClient via SiteStreamGrpcClientFactory. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: sitestream.proto with SiteStreamService, SiteStreamEvent (oneof: AttributeValueUpdate, AlarmStateUpdate). DebugStreamEvent message removed (no longer flows through ClusterClient). - Native alarms: a read-only mirror of native alarms from OPC UA Alarms & Conditions servers and the MxAccess Gateway, unified onto an A&C-style condition model (`AlarmConditionState`: orthogonal Active/Acked/Confirmed/Shelved/Suppressed + 0–1000 severity) plus an `AlarmKind` discriminator (Computed/NativeOpcUa/NativeMxAccess). New DCL capability seam `IAlarmSubscribableConnection` (implemented by the OPC UA and MxGateway adapters); the `DataConnectionActor` opens ONE alarm feed per connection and routes transitions to instances by source-object reference. A `NativeAlarmActor` (peer to the computed `AlarmActor` under `InstanceActor`) mirrors one source binding: snapshot atomic-swap on (re)subscribe, retention (drops once inactive+acked), per-source cap, and site SQLite persistence (`native_alarm_state`, survives failover, cleared on redeploy/undeploy — mirrors static overrides). State streams to central over the additively-enriched gRPC `AlarmStateUpdate` (the existing computed `AlarmStateChanged` was enriched additively) and seeds via the DebugView snapshot. Authoring: `TemplateNativeAlarmSource` / `InstanceNativeAlarmSourceOverride` entities flatten to `ResolvedNativeAlarmSource` (inherit/compose/override); management commands + ManagementActor handlers + CLI (`template/instance native-alarm-source`) + Central UI (template editor tab + instance override panel) + enriched DebugView alarm table. Read-only — no ack-back; no central tables. -- OPC UA / MxGateway UX (M7): operator **Alarm Summary** page (`/monitoring/alarms`, RequireDeployment, read-only) fans out the existing per-instance `DebugViewSnapshot` Ask (SemaphoreSlim-capped, partial-results tolerant) and aggregates client-side — no central alarm store, no aggregated live stream yet (follow-up); shared `AlarmStateBadges` component. OPC UA node browser gains `BrowseNext` continuation paging ("Load more"), a bounded recursive address-space **search** (`IAddressSpaceSearchable` seam; depth + result caps; substring on DisplayName/path), and **type-info** (DataType/ValueRank/Writable on `BrowseNode` for Variables). Attribute-override **CSV bulk import** (`OverrideCsvParser`, all-or-nothing) via InstanceConfigure `InputFile` + CLI `instance import-overrides --file` (native-alarm-source-override CSV deferred). **Verify-endpoint** probe (temporary `RealOpcUaClient`, short timeout, captures an untrusted server cert but NEVER trusts it) + **site-local cert trust**: per-node `CertStoreActor` (runs on every site node, not a singleton) writing the `.der` into the node's OPC UA trusted-peer PKI store; DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to BOTH site nodes so PKI stores stay consistent across failover; Admin-gated cert-management UI (`/design/connections/{id}/certificates`). No central persistence of cert trust (follow-up). +- OPC UA / MxGateway UX (M7): operator **Alarm Summary** page (`/monitoring/alarms`, RequireDeployment, read-only) fans out the existing per-instance `DebugViewSnapshot` Ask (SemaphoreSlim-capped, partial-results tolerant) and aggregates client-side — no central alarm store; shared `AlarmStateBadges` component. **Aggregated live stream shipped 2026-07-10** (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`): a transient in-memory per-site central live cache (`ISiteAlarmLiveCache`) fed by a site-wide, alarm-only `SubscribeSite` gRPC stream (seed-then-stream), pushing near-real-time deltas to the page over the Blazor circuit with the 15s poll kept as fallback + NotReporting authority — still no persisted central alarm store. OPC UA node browser gains `BrowseNext` continuation paging ("Load more"), a bounded recursive address-space **search** (`IAddressSpaceSearchable` seam; depth + result caps; substring on DisplayName/path), and **type-info** (DataType/ValueRank/Writable on `BrowseNode` for Variables). Attribute-override **CSV bulk import** (`OverrideCsvParser`, all-or-nothing) via InstanceConfigure `InputFile` + CLI `instance import-overrides --file` (native-alarm-source-override CSV deferred). **Verify-endpoint** probe (temporary `RealOpcUaClient`, short timeout, captures an untrusted server cert but NEVER trusts it) + **site-local cert trust**: per-node `CertStoreActor` (runs on every site node, not a singleton) writing the `.der` into the node's OPC UA trusted-peer PKI store; DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to BOTH site nodes so PKI stores stay consistent across failover; Admin-gated cert-management UI (`/design/connections/{id}/certificates`). No central persistence of cert trust (follow-up). ### External Integrations - External System Gateway: HTTP/REST only, JSON serialization, API key + Basic Auth. diff --git a/docs/plans/2026-05-29-native-alarms-design.md b/docs/plans/2026-05-29-native-alarms-design.md index 674e6b18..bacc34e0 100644 --- a/docs/plans/2026-05-29-native-alarms-design.md +++ b/docs/plans/2026-05-29-native-alarms-design.md @@ -21,7 +21,7 @@ These are **mirrored** alarms — the source is the source of truth — which is | State model scope | **Unified** A&C-style state model for *all* alarms (computed + native) | | Interactivity | **Read-only mirror** — display source-reported state; no acking/shelving from ScadaBridge, no command relay, no operator identity captured by ScadaBridge (source-supplied operator user/comment *are* displayed for native alarms) | | Binding | Instance declares a `NativeAlarmSource` (connection + source ref); conditions under it are **discovered at runtime**, keyed by source reference | -| State location | **Site-only**, persisted to SQLite (survives restart/failover); central **queries live** (snapshot + live stream); **no central tables, no central history** | +| State location | **Site-only**, persisted to SQLite (survives restart/failover); central **queries live** (snapshot + live stream); **no central tables, no central history** (the M9 aggregated live alarm stream — `2026-07-10-aggregated-live-alarm-stream-plan.md` — honors this: a **transient, in-memory** central cache only, nothing persisted) | | MxGateway transport | Gateway session-less `StreamAlarms` feed | | OPC UA transport | Alarms & Conditions events + `ConditionRefresh` snapshot | | Site actor structure | New `NativeAlarmActor` child of `InstanceActor`, peer to computed `AlarmActor`s (Approach 1) | diff --git a/docs/plans/2026-06-15-stillpending-completion-design.md b/docs/plans/2026-06-15-stillpending-completion-design.md index bc6c6770..7f661dbf 100644 --- a/docs/plans/2026-06-15-stillpending-completion-design.md +++ b/docs/plans/2026-06-15-stillpending-completion-design.md @@ -92,7 +92,7 @@ Delivered per `docs/plans/2026-06-18-m7-opcua-mxgateway-ux-design.md` (full scop - **T16** — browse type-info (DataType/ValueRank/Writable) + **attribute**-override CSV import (InstanceConfigure InputFile + CLI `instance import-overrides --file`). Native-alarm-source-override CSV import was **deferred** (attribute overrides only). - **T17** — Verify-endpoint probe (captures-but-never-trusts an untrusted server cert) + **site-local** cert trust (per-node `CertStoreActor`, DeploymentManager broadcast to **both** site nodes; D6) + Admin-gated cert-management UI. -Small follow-ups logged (not blocking): stamp `SourceNode` on the `SecuredWrite` audit rows (currently NULL); an aggregated **live** alarm stream for the summary page (snapshot + poll today); central-persisted, auditable cert trust (site-local today). +Small follow-ups logged (not blocking): stamp `SourceNode` on the `SecuredWrite` audit rows (currently NULL); an aggregated **live** alarm stream for the summary page (snapshot + poll today) — *delivered 2026-07-10, `docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`*; central-persisted, auditable cert trust (site-local today). #### M8 — Transport (T18, T20) — **DELIVERED** Delivered both features, with the silent gap #16 folded in: diff --git a/docs/plans/2026-06-18-m7-opcua-mxgateway-ux-design.md b/docs/plans/2026-06-18-m7-opcua-mxgateway-ux-design.md index 423bc464..ce4ca94d 100644 --- a/docs/plans/2026-06-18-m7-opcua-mxgateway-ux-design.md +++ b/docs/plans/2026-06-18-m7-opcua-mxgateway-ux-design.md @@ -32,6 +32,10 @@ from the merged `fix/mxgw-supervisory-write`). M7 does **not** reinvent those - **Central alarm store / history / journal** — T13 reads live snapshots; no central alarm tables (remains `[PERM]` per `2026-05-29-native-alarms-design.md`). No aggregated live gRPC stream for the summary page in v1 (snapshot + poll only). + **Delivered 2026-07-10** (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`): the + aggregated live gRPC stream shipped as a **transient, in-memory** per-site central live cache + (site-wide alarm-only `SubscribeSite` stream, seed-then-stream). The `[PERM]` no-central-store + rule is unchanged — the cache persists nothing. - **Native-alarm ack / shelve / suppress write-back** — read-only by design, unchanged. - **CSV bulk import of native-alarm-source overrides** (`InstanceNativeAlarmSourceOverride`) — the T16 importer targets *attribute* overrides only; native-alarm-source CSV is a follow-up. @@ -102,7 +106,9 @@ M7 adds these verbs on that path (all additive, mirroring `BrowseCommands.cs` + - **Reuse:** extract DebugView's inline alarm badge/formatter markup into a shared `AlarmStateBadges` component consumed by both DebugView and the summary page. - **Refresh:** manual button + optional poll timer (mirrors Health dashboard 10 s). No - aggregated live stream in v1. + aggregated live stream in v1. *(Aggregated live stream delivered 2026-07-10 — + `docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`; the poll is now a fallback + + `NotReporting` authority behind the transient per-site live cache.)* - **Files (indicative):** `Components/Pages/Monitoring/AlarmSummary.razor(.cs)`, a CentralUI `IAlarmSummaryService`/impl (fan-out + aggregate), `Components/Shared/AlarmStateBadges.razor`, `NavMenu.razor`, Playwright test. diff --git a/docs/plans/2026-07-08-deferred-work-register.md b/docs/plans/2026-07-08-deferred-work-register.md index f688acbc..dc2c361d 100644 --- a/docs/plans/2026-07-08-deferred-work-register.md +++ b/docs/plans/2026-07-08-deferred-work-register.md @@ -20,7 +20,6 @@ them and are removed from this table when that plan's task lands. |---|------|-------------|------------------------|-----------------| | 8 | Hash-chain tamper evidence (T1); CLI verify-chain is a no-op stub | audit-log roadmap :12 | v1.x by locked decision; append-only DB roles are the control | Compliance requirement for cryptographic tamper evidence | | 9 | Parquet audit archival (T2); endpoint returns 501 | AuditEndpoints.cs:204 | v1.x; 501 + CLI messaging are honest | AuditLog partition volume nears retention ceiling | -| 10 | Aggregated live alarm stream for Alarm Summary | m7 design :252 | Snapshot fan-out acceptable at current instance counts | Alarm Summary latency complaints or >~50 instances/site | | 11 | Central-persisted OPC UA cert-trust audit | m7 follow-ups | Broadcast-to-both-nodes covers HA | Governance/audit requirement for trust decisions | | 12 | Native-alarm-source-override CSV import — **Central UI `InstanceConfigure` upload affordance only** (CLI + Management API + parser shipped 2026-07-10, see Resolved) | m7 follow-ups | CLI/API path closes the operator parity gap; the Blazor upload button is polish | First request to bulk-import native sources from the UI rather than the CLI | | 17 | Unified notifications+site-calls outbox page | stillpending :118 | Explicit M9 decision to keep two pages | Operator confusion reports | @@ -32,6 +31,7 @@ Rows removed from the Deferred table above once confirmed shipped. Kept here for | # | Item | Resolution | |---|------|-----------| +| 10 | Aggregated live alarm stream for Alarm Summary | Shipped 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`): a **transient, in-memory** per-site central live alarm cache (`ISiteAlarmLiveCache`/`SiteAlarmLiveCacheService` + per-site `SiteAlarmAggregatorActor`) fed by a new site-wide, alarm-only `SubscribeSite` gRPC stream (`SiteStreamManager.SubscribeSiteAlarms`), seed-then-stream with dedup + NodeA↔NodeB re-seed + periodic reconcile. Alarm Summary is now live-cache-driven (`AlarmSummaryService.BuildFromLiveAlarms`) with the 15s poll retained as fallback + `NotReporting` authority. Honors the `[PERM]` no-central-store rule — nothing persisted (no EF table/migration). Options on `CommunicationOptions` (eagerly validated) + two `ScadaBridgeTelemetry` signals. | | 7 | SecuredWrite audit rows leave SourceNode NULL | Resolved (PLAN-07): `ManagementActor.EmitSecuredWriteAuditAsync` routes through `ICentralAuditWriter`, which stamps `SourceNode` (`central-a`/`central-b`) from `INodeIdentityProvider`. | | 12 (CLI/API) | Native-alarm-source-override CSV import | Shipped 2026-07-10: shared `CsvLineSplitter`, `NativeAlarmSourceOverrideCsvParser`, bulk all-or-nothing `SetInstanceNativeAlarmSourceOverridesCommand` + ManagementActor handler (Deployer-gated), CLI `instance native-alarm-source import --file`, parser/CLI/handler tests. **UI upload affordance still pending — see row 12 above.** | | 13 | WaitForAttribute quality-gated ("Good"-only) mode | Already implemented (Commons `WaitForAttribute.RequireGoodQuality`, enforced in `InstanceActor`, threaded through `ScriptRuntimeContext`, tested in `InstanceActorWaitForAttributeTests`). Stale "planned enhancement" doc line corrected 2026-07-10. | diff --git a/docs/requirements/Component-CentralUI.md b/docs/requirements/Component-CentralUI.md index 64ed9251..5853d690 100644 --- a/docs/requirements/Component-CentralUI.md +++ b/docs/requirements/Component-CentralUI.md @@ -187,10 +187,11 @@ Per-leaf alarm rendering (leaf nodes are individual conditions for native alarms ### Alarm Summary (Deployment Role) - A dedicated operator **Alarm Summary** page (`/monitoring/alarms`, `RequireDeployment`) gives a **cross-instance, read-only** roll-up of live alarm state at a site — the operator-facing complement to the per-instance Debug View. -- **Data path** — no new site-side code and no central alarm store. The page selects a site, queries its deployed instances, then fans out the existing per-instance `DebugViewSnapshot` Ask **concurrently** (capped with a `SemaphoreSlim`) and aggregates the returned `AlarmStates` client-side. The fan-out is **partial-results tolerant**: instances that time out are listed as "not reporting" while the rest still render. +- **Data path** — no new site-side code and no central alarm store. The page selects a site, queries its deployed instances, then fans out the existing per-instance `DebugViewSnapshot` Ask **concurrently** (capped with a `SemaphoreSlim`) and aggregates the returned `AlarmStates` client-side. The fan-out is **partial-results tolerant**: instances that time out are listed as "not reporting" while the rest still render. This snapshot fan-out now doubles as the **seed** for a near-real-time live feed (see **Live updates** below) rather than the sole refresh mechanism. +- **Live updates** — the page is driven by a **transient, per-site central live alarm cache** (`ISiteAlarmLiveCache`, owned by the Communication component; see [Component-Communication](Component-Communication.md)). On site select the page subscribes to the cache; the cache runs one shared, reference-counted per-site aggregator that **seeds** from the snapshot fan-out and then stays warm on a single **site-wide, alarm-only** `SubscribeSite` gRPC stream (seed-then-stream, dedup by `(InstanceUniqueName, AlarmName, SourceReference)`). Applied deltas raise an in-process change event (mirroring `IDeploymentStatusNotifier`) that the Blazor circuit pushes to the browser via `StateHasChanged()` — no new SignalR hub. `AlarmSummaryService.BuildFromLiveAlarms` rebuilds the roll-up + rows from the cache's current alarm set. The cache is **purely in-memory on the active central node** — there is still **no persisted central alarm store**; on a NodeA↔NodeB failover the new active node re-seeds from scratch. - **View** — roll-up tiles (total active, worst severity, unacked count, per-`AlarmKind` counts) plus a flat, sortable, filterable table. Filters cover instance, `AlarmKind` (Computed / NativeOpcUa / NativeMxAccess), state, acked/unacked, severity threshold, and name search. - **Read-only** — there are no ack / shelve / suppress controls (native alarms remain read-only by design). -- **Refresh** — manual refresh button plus an optional poll timer (mirroring the Health dashboard); there is no aggregated live alarm stream in this release (snapshot + poll only — logged as a follow-up). +- **Refresh** — manual refresh button plus the 15s poll timer (mirroring the Health dashboard), now retained as a **fallback + `NotReporting` authority** behind the live cache: when the cache reports `IsLive`, the page renders live-cache state; when a stream is unhealthy or a site has not yet seeded, the poll keeps the page fresh so a stream failure never blanks it. (Aggregated live stream **delivered 2026-07-10** — see `docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`.) - **Reuse** — the alarm badge/formatter markup is factored out of Debug View into a shared `AlarmStateBadges` component consumed by both Debug View and this page. ### Parked Message Management (Deployment Role) diff --git a/docs/requirements/Component-Communication.md b/docs/requirements/Component-Communication.md index 83daa298..bcf92152 100644 --- a/docs/requirements/Component-Communication.md +++ b/docs/requirements/Component-Communication.md @@ -67,6 +67,17 @@ Both central and site clusters. Each side has communication actors that handle m - **Accepted limitation (central-side session locality):** debug sessions are process-local state on the central node hosting the `DebugStreamBridgeActor`. A central **restart or failover drops all active sessions** with no `OnStreamTerminated` signal to the operator — the engineer simply re-establishes the debug session from the UI. Debug streaming is an interactive, transient diagnostic aid, so this is an accepted trade-off rather than a durability gap (arch review 02, U7). - **Backpressure is lossy-by-design and now observable:** the per-session `Channel` (bounded 1000, `DropOldest`) silently evicts the oldest event when a slow consumer falls behind. The eviction is counted via the channel's `itemDropped` callback and logged at Warning (first eviction, then every 500th) so real event loss on a debug stream is visible instead of silent. +### 6.1 Aggregated Live Alarm Stream (Site → Central) + +Delivered 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`) for the operator Alarm Summary page (Central UI #9). Where §6 streams **one instance** for the interactive Debug View, this feeds a **transient, per-site central live alarm cache** so the Alarm Summary reflects alarm transitions in near-real-time instead of re-polling the whole site every 15s. + +- **Site-wide, alarm-only stream**: a single `SubscribeSite` gRPC stream per site carries `AlarmStateChanged` events for **all** instances (attribute events are dropped — the summary never shows them, and attributes are far higher-volume). On the site this reuses the existing `SiteStreamManager` broadcast hub via `SiteStreamManager.SubscribeSiteAlarms(...)` (same wiring as the per-instance `Subscribe`, minus the `InstanceUniqueName` filter and filtered to alarms). `StreamRelayActor` is reused unchanged, so placeholder rows (`is_configured_placeholder`) are still dropped at the relay. +- **Central live cache** (`ISiteAlarmLiveCache`, singleton `SiteAlarmLiveCacheService`): a DI singleton on the active central node. For each site with ≥1 active viewer it runs ONE shared, **reference-counted** per-site aggregator (`SiteAlarmAggregatorActor`); the first `Subscribe(siteId, onChanged)` starts it, the last subscriber leaving stops it after a short **linger** to avoid re-seed thrash. `GetCurrentAlarms(siteId)` returns the current immutable snapshot; `IsLive(siteId)` reports whether the aggregator has seeded and published at least once. +- **Seed-then-stream** (copied from `DebugStreamBridgeActor` ordering): open the `SubscribeSite` stream first (buffer live deltas), run the snapshot fan-out once via the existing `DebugViewSnapshot` path (bounded by `LiveAlarmCacheSeedConcurrency`), flush the buffer with **dedup by `(InstanceUniqueName, AlarmName, SourceReference)`**, then live pass-through. Placeholders are seeded from the snapshot and never expected on the live stream. +- **Failover & drift**: NodeA↔NodeB reconnect (client via `SiteStreamGrpcClient.SubscribeSiteAsync`, factory endpoint failover) re-seeds rather than serving stale; a periodic **reconcile snapshot** (default 60s) corrects any missed enable/disable so the cache can't drift indefinitely. `[PERM]` (`docs/plans/2026-05-29-native-alarms-design.md`): the cache is **purely in-memory** — no EF entity/table/migration, no persisted central alarm store — so a new active node simply re-seeds from scratch. +- **Options** (`Communication` section, `CommunicationOptions`; eagerly validated by `CommunicationOptionsValidator` / `ValidateOnStart`): `LiveAlarmCacheLinger` (default 30s), `LiveAlarmCacheReconcileInterval` (default 60s), `LiveAlarmCacheSeedConcurrency` (default 8), `LiveAlarmCacheMaxSubscribersPerSite` (default 200). +- **Telemetry** (`ScadaBridgeTelemetry` meter): observable gauge `scadabridge.site.alarm_cache.aggregators.active` (running per-site aggregators) and counter `scadabridge.site.alarm_cache.reconnects` (site-wide stream reconnects — a NodeA↔NodeB flip or reconcile-driven reopen; a sustained climb signals a flapping site link). + #### Site-Side gRPC Streaming Components - **SiteStreamGrpcServer**: gRPC service (`SiteStreamService.SiteStreamServiceBase`) hosted on each site node via Kestrel HTTP/2 on a dedicated port (default 8083). Implements the `SubscribeInstance` RPC. For each subscription, creates a `StreamRelayActor` that subscribes to `SiteStreamManager`, bridges events through a `Channel` to the gRPC response stream. Tracks active subscriptions by `correlation_id` — duplicate IDs cancel the old stream. Enforces a max concurrent stream limit (default 100). Rejects streams with `StatusCode.Unavailable` before the actor system is ready. @@ -84,13 +95,14 @@ Both central and site clusters. Each side has communication actors that handle m The streaming protocol is defined in `sitestream.proto` (`src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto`): -- **Service**: `SiteStreamService` — hosted on each site node by `SiteStreamGrpcServer` — exposes five RPCs. One is the original real-time **server-streaming** subscription; the other four are **unary request/response** calls added by the Audit Log (#23) and Site Call Audit (#22) components. A unary call is request/response and is distinct from the command/control ClusterClient channel — gRPC on this service is no longer real-time-stream-only: - - `SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent)` — the real-time debug stream (§6); the only server-streaming RPC. +- **Service**: `SiteStreamService` — hosted on each site node by `SiteStreamGrpcServer` — exposes six RPCs. Two are real-time **server-streaming** subscriptions (`SubscribeInstance`, `SubscribeSite`); the other four are **unary request/response** calls added by the Audit Log (#23) and Site Call Audit (#22) components. A unary call is request/response and is distinct from the command/control ClusterClient channel — gRPC on this service is no longer real-time-stream-only: + - `SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent)` — the per-instance real-time debug stream (§6). + - `SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent)` — the **site-wide, alarm-only** aggregated stream (§6.1) added for the operator Alarm Summary live cache. Additive (new RPC + new `SiteStreamRequest { correlation_id }` message; no field renumbering). The server handler mirrors `SubscribeInstance` — same bounded `Channel(1000, DropOldest)`, `GrpcMaxConcurrentStreams`/`GrpcMaxStreamLifetime` limits, `SiteConnectionOpened/Closed` telemetry, and `StreamRelayActor` mapping — but subscribes via `SiteStreamManager.SubscribeSiteAlarms` (all instances, `AlarmStateChanged` only; attribute events dropped, no per-instance filter). - `IngestAuditEvents(AuditEventBatch) returns (IngestAck)` — central-side **ingest** receiving surface for Audit Log (#23) telemetry; routes the batch to the central `AuditLogIngestActor` proxy and returns the accepted `EventId`s. (The production *push* path is still ClusterClient via `ClusterClientSiteAuditClient`; this RPC is the gRPC-receiving counterpart.) - `IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck)` — ingest receiving surface for the combined cached-call telemetry packet (audit row + `SiteCalls` operational upsert written in one transaction). - `PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse)` — central→site **reconciliation pull** for the Audit Log self-heal feed; the site serves `Pending`/`Forwarded` rows from its `ISiteAuditQueue`. - `PullSiteCalls(PullSiteCallsRequest) returns (PullSiteCallsResponse)` — central→site reconciliation pull for the Site Call Audit (#22) self-heal feed; the site serves operation-tracking rows changed since a cursor from its `IOperationTrackingStore`. A separate RPC from `PullAuditEvents` because the tracking store is the operational source of truth, distinct from the site audit queue. -- **Messages**: `InstanceStreamRequest` (correlation_id, instance_unique_name), `SiteStreamEvent` (correlation_id, oneof event: `AttributeValueUpdate`, `AlarmStateUpdate`); `AuditEventDto`/`AuditEventBatch`/`IngestAck` for ingest; `CachedTelemetryPacket`/`CachedTelemetryBatch` (each packet pairing an `AuditEventDto` with a `SiteCallOperationalDto`); `PullAuditEventsRequest`/`PullAuditEventsResponse` and `PullSiteCallsRequest`/`PullSiteCallsResponse` (each request carries `since_utc` + `batch_size`; each response carries `more_available` to signal a saturated batch). +- **Messages**: `InstanceStreamRequest` (correlation_id, instance_unique_name), `SiteStreamRequest` (correlation_id only — no instance name; drives the site-wide `SubscribeSite` stream), `SiteStreamEvent` (correlation_id, oneof event: `AttributeValueUpdate`, `AlarmStateUpdate`); `AuditEventDto`/`AuditEventBatch`/`IngestAck` for ingest; `CachedTelemetryPacket`/`CachedTelemetryBatch` (each packet pairing an `AuditEventDto` with a `SiteCallOperationalDto`); `PullAuditEventsRequest`/`PullAuditEventsResponse` and `PullSiteCallsRequest`/`PullSiteCallsResponse` (each request carries `since_utc` + `batch_size`; each response carries `more_available` to signal a saturated batch). - The `oneof event` pattern is extensible — future event types (health metrics, connection state changes) are added as new fields without breaking existing consumers. - Proto field numbers are never reused; new RPCs and message fields are appended additively. Old clients ignore unknown `oneof` variants. diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor index 1ab17c31..536bdf1c 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Components/Pages/Monitoring/AlarmSummary.razor @@ -5,9 +5,11 @@ @using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites @using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories @using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums +@using ZB.MOM.WW.ScadaBridge.Communication @implements IDisposable @inject IAlarmSummaryService AlarmSummaryService @inject ISiteRepository SiteRepository +@inject ISiteAlarmLiveCache LiveAlarmCache
@@ -259,13 +261,23 @@ _notReporting = Array.Empty(); _rollup = new AlarmRollup(0, 0, 0, new Dictionary()); StopTimer(); + DisposeLiveSubscription(); return; } _selectedSiteId = siteId; ClearFilters(); + // Subscribe to the shared live cache FIRST so the aggregator (and its + // seed-then-stream) starts warming while the initial poll runs. + SubscribeLive(siteId); await RefreshAsync(); StartTimer(); + // If another circuit already warmed the cache for this site, the initial + // poll may be staler than what's live — apply the live snapshot on top. + if (LiveAlarmCache.IsLive(siteId)) + { + ApplyLiveSnapshot(siteId); + } } private async Task RefreshAsync() @@ -322,6 +334,64 @@ _refreshTimer = null; } + // ── Live cache (plan #10, Task 5) ────────────────────────────────────────── + // The page is live-cache-first with the 15s poll kept as a fallback/safety net. + // Reconciliation model: + // • The live cache pushes onChanged deltas (near-real-time) whenever the site's + // aggregated alarm set changes. We rebuild _rows/_rollup/_visibleRows from the + // immutable live snapshot — but deliberately DO NOT touch _notReporting, since + // the alarm-only live cache can't compute it. + // • The 15s poll (RefreshAsync) still runs untouched: it is the authority for + // _notReporting and the safety net when the cache is not live (pre-seed or a + // degraded/failed stream — IsLive == false). When live, the poll simply + // re-affirms the same snapshot; the live path just makes updates arrive sooner. + // • Both paths mutate shared state only via the Blazor dispatcher (the poll via + // its InvokeAsync callback, the live delta via OnLiveChanged's InvokeAsync), so + // they are serialized and never race. Each rebuild is an idempotent snapshot, so + // a live rebuild immediately followed by a poll rebuild (or vice-versa) is safe. + private IDisposable? _liveSubscription; + + private void SubscribeLive(int siteId) + { + DisposeLiveSubscription(); + _liveSubscription = LiveAlarmCache.Subscribe(siteId, () => OnLiveAlarmsChanged(siteId)); + } + + private void DisposeLiveSubscription() + { + _liveSubscription?.Dispose(); + _liveSubscription = null; + } + + // Raised on the aggregator's thread — marshal onto the circuit before touching state. + private void OnLiveAlarmsChanged(int siteId) + { + _ = InvokeAsync(() => + { + // Drop stale callbacks for a site we've since navigated away from, and + // let the poll drive until the aggregator has actually seeded (so we never + // clobber a good poll snapshot with an empty pre-seed list). + if (_selectedSiteId != siteId || !LiveAlarmCache.IsLive(siteId)) + { + return; + } + + ApplyLiveSnapshot(siteId); + StateHasChanged(); + }); + } + + // Rebuilds rows/rollup/visible-rows from the live snapshot. Leaves _notReporting + // as the last poll computed it (the alarm-only cache can't know it). Idempotent. + private void ApplyLiveSnapshot(int siteId) + { + var current = LiveAlarmCache.GetCurrentAlarms(siteId); + var result = AlarmSummaryService.BuildFromLiveAlarms(current); + _rows = result.Alarms; + _rollup = AlarmSummaryService.ComputeRollup(_rows); + RecomputeVisibleRows(); + } + private IEnumerable DistinctInstances => _rows.Select(r => r.InstanceUniqueName).Distinct().OrderBy(n => n, StringComparer.OrdinalIgnoreCase); @@ -434,5 +504,9 @@ _ => "Computed" }; - public void Dispose() => StopTimer(); + public void Dispose() + { + StopTimer(); + DisposeLiveSubscription(); + } } diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/AlarmSummaryService.cs b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/AlarmSummaryService.cs index 9d2af872..0e2bcc9d 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/AlarmSummaryService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/AlarmSummaryService.cs @@ -127,6 +127,26 @@ public sealed class AlarmSummaryService : IAlarmSummaryService } } + /// + public AlarmSummaryResult BuildFromLiveAlarms(IReadOnlyList alarms) + { + ArgumentNullException.ThrowIfNull(alarms); + + // Flatten one row per alarm (instance from the alarm itself, mirroring + // FetchInstanceAsync) and apply the SAME deterministic instance-then-name + // sort as GetSiteAlarmsAsync so the live and poll paths render identically. + var orderedRows = alarms + .Select(alarm => new AlarmSummaryRow(alarm.InstanceUniqueName, alarm)) + .OrderBy(r => r.InstanceUniqueName, StringComparer.OrdinalIgnoreCase) + .ThenBy(r => r.Alarm.AlarmName, StringComparer.OrdinalIgnoreCase) + .ToList(); + + // The live cache is alarm-only, so it cannot enumerate "not reporting" + // instances — left empty here; the periodic poll (GetSiteAlarmsAsync) + // remains the authority for that list. + return new AlarmSummaryResult(orderedRows, Array.Empty()); + } + /// public AlarmRollup ComputeRollup(IReadOnlyList rows) { diff --git a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IAlarmSummaryService.cs b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IAlarmSummaryService.cs index 4719a88c..9ffe92ea 100644 --- a/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IAlarmSummaryService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/IAlarmSummaryService.cs @@ -42,6 +42,28 @@ public interface IAlarmSummaryService /// Task GetSiteAlarmsAsync(int siteId, CancellationToken cancellationToken = default); + /// + /// Builds an directly from an in-memory live + /// snapshot (plan #10, Task 5) — the current alarms served by + /// + /// — instead of fanning out per-instance snapshot Asks. Each + /// is flattened to one + /// (instance taken from + /// ) and ordered with the SAME + /// deterministic instance-then-alarm-name sort as + /// , so the two paths are interchangeable. + /// + /// + /// The live cache carries only alarm state, so it cannot know which Enabled + /// instances are silent-but-reporting versus not reporting at all; + /// is therefore always + /// empty on this path. The page keeps its periodic + /// poll as the authority for "not reporting" (and as a live-stream safety net). + /// + /// The current live alarm snapshot for a site. + /// An whose is empty. + AlarmSummaryResult BuildFromLiveAlarms(IReadOnlyList alarms); + /// /// Pure roll-up over a set of s. Exposed so the /// page (and tests) can recompute the headline tiles without re-querying. diff --git a/src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs b/src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs index d6b5bc54..1f8cc2ad 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Commons/Observability/ScadaBridgeTelemetry.cs @@ -47,11 +47,23 @@ public static class ScadaBridgeTelemetry Meter.CreateCounter("scadabridge.store_and_forward.replication.failures", unit: "1", description: "S&F buffer replication operations that failed to dispatch/deliver to the peer node"); + /// + /// Incremented each time a per-site live-alarm aggregator re-establishes its site-wide + /// gRPC stream — a NodeA↔NodeB failover flip or a reconcile-driven reopen after the + /// stream was given up (plan #10, Task 6). A sustained climb signals a flapping site link. + /// + private static readonly Counter _liveAlarmStreamReconnects = + Meter.CreateCounter("scadabridge.site.alarm_cache.reconnects", unit: "1", + description: "Live-alarm aggregator site-wide gRPC stream reconnects (NodeA↔NodeB flip or reconcile-driven reopen)."); + // ---------------- Observable gauges ---------------- /// Current count of open site connections, mutated via . private static long _siteConnectionsUp; + /// Current count of running per-site live-alarm aggregators, mutated via . + private static long _liveAlarmAggregatorsActive; + /// Provider that yields the live StoreAndForward queue depth; set by a later task. private static Func? _queueDepthProvider; @@ -68,6 +80,13 @@ public static class ScadaBridgeTelemetry () => Volatile.Read(ref _queueDepthProvider) is { } p ? p() : 0L, unit: "items", description: "Current StoreAndForward queue depth."); + + /// Gauge reporting the number of currently running per-site live-alarm aggregators. + private static readonly ObservableGauge _liveAlarmAggregatorsActiveGauge = + Meter.CreateObservableGauge("scadabridge.site.alarm_cache.aggregators.active", + () => Interlocked.Read(ref _liveAlarmAggregatorsActive), + unit: "1", + description: "Number of per-site live-alarm aggregators currently running on the active central node."); #pragma warning restore IDE0052 // ---------------- Emit helpers ---------------- @@ -94,6 +113,15 @@ public static class ScadaBridgeTelemetry /// Records that a site connection closed (decrements the up-count gauge). public static void SiteConnectionClosed() => Interlocked.Decrement(ref _siteConnectionsUp); + /// Records that a per-site live-alarm aggregator started (increments the active-aggregator gauge). + public static void LiveAlarmAggregatorStarted() => Interlocked.Increment(ref _liveAlarmAggregatorsActive); + + /// Records that a per-site live-alarm aggregator stopped (decrements the active-aggregator gauge). + public static void LiveAlarmAggregatorStopped() => Interlocked.Decrement(ref _liveAlarmAggregatorsActive); + + /// Records that a per-site live-alarm aggregator re-established its site-wide gRPC stream. + public static void RecordLiveAlarmStreamReconnect() => _liveAlarmStreamReconnects.Add(1); + /// /// Registers the provider the StoreAndForward queue-depth gauge reads on each observation. /// A later task supplies a provider that reads the real StoreAndForward depth. A null diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs new file mode 100644 index 00000000..7d19e0cb --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs @@ -0,0 +1,530 @@ +using Akka.Actor; +using Akka.Event; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; +using ZB.MOM.WW.ScadaBridge.Commons.Observability; +using ZB.MOM.WW.ScadaBridge.Communication.Grpc; + +namespace ZB.MOM.WW.ScadaBridge.Communication.Actors; + +/// +/// One-per-site aggregator on the active central node backing the operator Alarm +/// Summary live cache (plan #10, Task 4). Holds a transient in-memory +/// Dictionary<AlarmKey, AlarmStateChanged> of the whole site's current +/// alarm state — NO persistence (locked [PERM]: no central alarm store). Created +/// and torn down by under viewer reference-count. +/// +/// Seed-then-stream ordering (copied from ): +/// the site-wide, alarm-only gRPC stream (SubscribeSite) is opened FIRST in +/// so live deltas start flowing during the seed's snapshot-build +/// + transit window; deltas arriving while a seed/reconcile fan-out is in flight are +/// buffered in arrival order. When the fan-out completes the cache is rebuilt +/// authoritatively from the fresh snapshot, then the buffer is flushed with per-key +/// dedup against the seed, then the actor passes live deltas straight into the cache. +/// +/// +/// Placeholder reconciliation: the snapshot fan-out carries +/// rows (a configured native +/// source binding with no active conditions); the live stream drops them server-side. +/// Placeholders are seeded and never expected on the live stream — a live delta for a +/// real alarm has a distinct AlarmKey so it can never wipe a placeholder row and +/// vice-versa. Placeholder-vs-real coherence is refreshed by the periodic reconcile. +/// +/// +/// Failover + drift: a gRPC error flips NodeA↔NodeB with the same retry budget + +/// stability window as , and each reconnect triggers +/// a RE-SEED (never silently serve stale). A periodic reconcile snapshot +/// (, default 60s) corrects instance-set drift and any +/// missed delta. +/// +/// All state is mutated only on the actor thread: gRPC callbacks and fan-out results are +/// marshalled back via Self.Tell, so the cache needs no internal lock. The +/// published snapshot handed to the service is a fresh immutable list (reference swap), +/// so Blazor render threads never observe a partially-mutated cache. +/// +public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers +{ + private readonly ILoggingAdapter _log = Context.GetLogger(); + private readonly string _siteIdentifier; + private readonly string _correlationId; + private readonly Func>> _seedFn; + private readonly Action> _publish; + private readonly SiteStreamGrpcClientFactory _grpcFactory; + private readonly string _grpcNodeAAddress; + private readonly string _grpcNodeBAddress; + private readonly TimeSpan _reconcileInterval; + + private const int MaxRetries = 3; + private const string ReconnectTimerKey = "alarm-grpc-reconnect"; + private const string StabilityTimerKey = "alarm-grpc-stability"; + private const string ReconcileTimerKey = "alarm-reconcile"; + + /// Delay between gRPC reconnection attempts. Settable for tests. + internal static TimeSpan ReconnectDelay { get; set; } = TimeSpan.FromSeconds(5); + + /// + /// How long a freshly-opened gRPC stream must stay up before its retry budget is + /// considered recovered (mirrors ). + /// Settable for tests. + /// + internal static TimeSpan StabilityWindow { get; set; } = TimeSpan.FromSeconds(60); + + private int _retryCount; + private bool _useNodeA = true; + private bool _stopped; + + /// + /// True once the live gRPC stream has been given up (retry budget exhausted). Reconcile + /// snapshots keep serving in the meantime; the next reconcile tick self-heals the stream + /// by resetting the retry budget and reopening it, so a sustained site outage does not + /// permanently drop the live feed. Actor-thread only. + /// + private bool _streamDown; + private CancellationTokenSource? _grpcCts; + private CancellationTokenSource? _lifetimeCts; + + /// Current whole-site alarm state, keyed by . Actor-thread only. + private readonly Dictionary _cache = new(); + + /// True once the first seed has completed and been published. Actor-thread only. + private bool _seeded; + + /// True while a seed/reconcile snapshot fan-out is in flight (deltas buffer). Actor-thread only. + private bool _fanoutInFlight; + + /// Ordered buffer of live deltas that arrived while a fan-out was in flight. Actor-thread only. + private readonly List _buffer = new(); + + private const int BufferWarnThreshold = 10_000; + private bool _bufferWarned; + + /// + public ITimerScheduler Timers { get; set; } = null!; + + /// + /// Creates a per-site alarm aggregator. + /// + /// Site identifier (for logging / gRPC client keying). + /// Correlation id for the site-wide gRPC subscription. + /// + /// Snapshot fan-out that returns the whole site's current alarm rows (including + /// placeholders), best-effort and tolerant of per-instance failure. Re-run on every + /// seed and reconcile, so it re-enumerates the site's Enabled instances each call. + /// + /// + /// Publishes a fresh immutable snapshot of the cache to the owning service, which + /// stores it and raises viewer onChanged callbacks. Invoked on the actor thread. + /// + /// Factory caching one gRPC client per (site, endpoint). + /// gRPC address of the site's node A. + /// gRPC address of the site's node B. + /// Periodic reconcile snapshot cadence. + public SiteAlarmAggregatorActor( + string siteIdentifier, + string correlationId, + Func>> seedFn, + Action> publish, + SiteStreamGrpcClientFactory grpcFactory, + string grpcNodeAAddress, + string grpcNodeBAddress, + TimeSpan reconcileInterval) + { + _siteIdentifier = siteIdentifier; + _correlationId = correlationId; + _seedFn = seedFn; + _publish = publish; + _grpcFactory = grpcFactory; + _grpcNodeAAddress = grpcNodeAAddress; + _grpcNodeBAddress = grpcNodeBAddress; + _reconcileInterval = reconcileInterval; + + // Live delta from the site-wide alarm stream (marshalled in via Self.Tell). + // A received delta must NOT reset the retry budget (a flapping stream that + // delivers one delta between failures would otherwise never trip MaxRetries). + Receive(HandleLiveDelta); + + // A seed/reconcile fan-out completed. + Receive(OnSeedCompleted); + + // A seed/reconcile fan-out threw as a whole (individual per-instance faults are + // swallowed inside seedFn and degrade to fewer rows, not a whole-fan-out failure). + Receive(OnSeedFailed); + + // Periodic reconcile tick (and the re-seed kicked after a reconnect). + Receive(_ => OnReconcileTick()); + + // Stream stayed up for StabilityWindow — recover the retry budget. + Receive(_ => + { + if (_stopped) return; + _retryCount = 0; + _log.Debug("Site-alarm gRPC stream for {0} stable; retry count reset", _siteIdentifier); + }); + + // gRPC stream error — flip node + reconnect + re-seed. + Receive(msg => + { + _log.Warning("Site-alarm gRPC stream error for {0}: {1}", _siteIdentifier, msg.Exception.Message); + HandleGrpcError(); + }); + + Receive(_ => OpenGrpcStream()); + + // Owning service asks us to stop (last viewer left + linger elapsed). + Receive(_ => + { + _log.Info("Stopping site-alarm aggregator for {0}", _siteIdentifier); + CleanupGrpc(); + _stopped = true; + Context.Stop(Self); + }); + } + + /// + protected override void PreStart() + { + _log.Info("Starting site-alarm aggregator for site {0}", _siteIdentifier); + // Telemetry: this aggregator is now a running per-site live cache (gauge +1). Balanced + // in PostStop, which Akka always runs on termination for any reason. + ScadaBridgeTelemetry.LiveAlarmAggregatorStarted(); + _lifetimeCts = new CancellationTokenSource(); + + // Stream-first: open the site-wide alarm stream BEFORE the first seed so deltas + // in the seed window are captured (buffered) rather than lost. + OpenGrpcStream(); + + // Kick the initial seed fan-out. + StartFanout(isInitial: true); + + // Periodic reconcile backstop. + Timers.StartPeriodicTimer(ReconcileTimerKey, new RunReconcile(), _reconcileInterval, _reconcileInterval); + } + + /// + protected override void PostStop() + { + _grpcCts?.Cancel(); + _grpcCts?.Dispose(); + _grpcCts = null; + _lifetimeCts?.Cancel(); + _lifetimeCts?.Dispose(); + _lifetimeCts = null; + // Telemetry: this aggregator is no longer running (gauge -1). Balances PreStart. + ScadaBridgeTelemetry.LiveAlarmAggregatorStopped(); + base.PostStop(); + } + + // ── Reconcile tick ────────────────────────────────────────────────────────── + + /// + /// Periodic reconcile: always re-run the snapshot fan-out (corrects drift + missed + /// deltas), and if the live stream was previously given up, self-heal it by resetting + /// the retry budget and reopening — so a sustained outage never permanently kills the + /// live feed. + /// + private void OnReconcileTick() + { + if (_stopped) return; + StartFanout(isInitial: false); + if (_streamDown) + { + _log.Info("Site-alarm gRPC stream for {0} was down; reopening on reconcile tick", _siteIdentifier); + _retryCount = 0; + // Telemetry: a reconcile-driven reopen after the stream was given up is a reconnect. + ScadaBridgeTelemetry.RecordLiveAlarmStreamReconnect(); + OpenGrpcStream(); + } + } + + // ── Seed / reconcile fan-out ──────────────────────────────────────────────── + + /// + /// Kicks a snapshot fan-out as a background task, marshalling the result back via + /// Self.Tell. While in flight, live deltas buffer. A reconcile that arrives + /// while a fan-out is already running is skipped (no stacking). + /// + private void StartFanout(bool isInitial) + { + if (_stopped) return; + if (_fanoutInFlight) + { + // Already seeding/reconciling — don't stack a second fan-out. + return; + } + + _fanoutInFlight = true; + var self = Self; + var ct = _lifetimeCts?.Token ?? CancellationToken.None; + + Task.Run(async () => + { + try + { + var alarms = await _seedFn(ct); + self.Tell(new SeedCompleted(alarms, isInitial)); + } + catch (OperationCanceledException) when (ct.IsCancellationRequested) + { + // Actor stopping — drop silently. + } + catch (Exception ex) + { + self.Tell(new SeedFailed(ex, isInitial)); + } + }, ct); + } + + private void OnSeedCompleted(SeedCompleted msg) + { + if (_stopped) return; + + // Rebuild the cache authoritatively from the fresh snapshot (this is what makes + // reconcile able to DROP rows for instances/alarms that disappeared — a merge + // could never remove a stale row since the live stream sends no "removed" event). + _cache.Clear(); + foreach (var alarm in msg.Alarms) + { + var key = AlarmKey(alarm); + if (!_cache.TryGetValue(key, out var existing) || alarm.Timestamp >= existing.Timestamp) + _cache[key] = alarm; + } + + // Flush the deltas buffered during the fan-out, deduped against the seed: a + // buffered delta whose key is in the fresh snapshot with an equal-or-newer + // timestamp is already reflected → drop; a strictly-newer (or new-key) delta is + // applied. Inclusive-on-snapshot boundary matches DebugStreamBridgeActor. + FlushBuffer(); + + _fanoutInFlight = false; + _seeded = true; + + _log.Debug("Site-alarm {0} {1} complete: {2} alarm row(s)", + _siteIdentifier, msg.IsInitial ? "seed" : "reconcile", _cache.Count); + + Publish(); + } + + private void OnSeedFailed(SeedFailed msg) + { + if (_stopped) return; + + _log.Warning(msg.Exception, + "Site-alarm {0} {1} fan-out failed; keeping current cache and relying on the next reconcile", + _siteIdentifier, msg.IsInitial ? "seed" : "reconcile"); + + // Don't lose deltas captured during the failed window — apply them pass-through + // into the (possibly stale/empty) cache. The next reconcile re-seeds authoritatively. + _fanoutInFlight = false; + FlushBuffer(dedupAgainstSeed: false); + + // Only publish if we already had a seed (so IsLive doesn't flip true on a + // failed initial seed — the page keeps its poll fallback until we truly seed). + if (_seeded) + Publish(); + } + + /// + /// Flushes the pre-fan-out buffer in arrival order. When + /// is true (normal completion) a buffered delta already reflected in the just-installed + /// snapshot (same key, timestamp <= cache entry) is dropped; otherwise every buffered + /// delta is applied pass-through. + /// + private void FlushBuffer(bool dedupAgainstSeed = true) + { + if (_buffer.Count == 0) return; + + foreach (var delta in _buffer) + { + if (dedupAgainstSeed) + ApplyDelta(delta, requireStrictlyNewer: true); + else + ApplyDelta(delta, requireStrictlyNewer: false); + } + _buffer.Clear(); + } + + // ── Live delta handling ───────────────────────────────────────────────────── + + private void HandleLiveDelta(AlarmStateChanged delta) + { + if (_stopped) return; + + if (_fanoutInFlight) + { + _buffer.Add(delta); + if (!_bufferWarned && _buffer.Count > BufferWarnThreshold) + { + _bufferWarned = true; + _log.Warning( + "Site-alarm pre-seed buffer for {0} exceeded {1} deltas while a fan-out was in flight " + + "(deltas retained, not dropped).", + _siteIdentifier, BufferWarnThreshold); + } + return; + } + + // Pass-through: apply and publish only if the cache actually changed. + if (ApplyDelta(delta, requireStrictlyNewer: false)) + Publish(); + } + + /// + /// Applies one alarm delta to the cache keyed by . Returns + /// true if the cache changed. A stale (older) delta for an existing key is + /// ignored. Never carries a placeholder (the live stream drops those), so a live + /// delta for a real alarm can only add/replace its own key — it never touches a + /// placeholder row under a different key. + /// + private bool ApplyDelta(AlarmStateChanged delta, bool requireStrictlyNewer) + { + var key = AlarmKey(delta); + if (_cache.TryGetValue(key, out var existing)) + { + var newer = requireStrictlyNewer + ? delta.Timestamp > existing.Timestamp + : delta.Timestamp >= existing.Timestamp; + if (!newer) return false; + } + _cache[key] = delta; + return true; + } + + private void Publish() + { + // Fresh immutable list — reference swap; readers never see a partial mutation. + var snapshot = _cache.Values.ToList(); + try + { + _publish(snapshot); + } + catch (Exception ex) + { + _log.Warning(ex, "Site-alarm publish callback threw for {0}; ignoring", _siteIdentifier); + } + } + + // ── gRPC stream lifecycle (mirrors DebugStreamBridgeActor) ────────────────── + + private void OpenGrpcStream() + { + if (_stopped) return; + + _streamDown = false; + var endpoint = _useNodeA ? _grpcNodeAAddress : _grpcNodeBAddress; + _log.Info("Opening site-alarm gRPC stream for {0} to {1}", _siteIdentifier, endpoint); + + _grpcCts?.Cancel(); + _grpcCts?.Dispose(); + _grpcCts = new CancellationTokenSource(); + + Timers.StartSingleTimer(StabilityTimerKey, new GrpcAlarmStreamStable(), StabilityWindow); + + var client = _grpcFactory.GetOrCreate(_siteIdentifier, endpoint); + var self = Self; + var ct = _grpcCts.Token; + + Task.Run(async () => + { + await client.SubscribeSiteAsync( + _correlationId, + alarm => self.Tell(alarm), + ex => self.Tell(new GrpcAlarmStreamError(ex)), + ct); + }, ct); + } + + private void HandleGrpcError() + { + if (_stopped) return; + + // Stream failed before the stability window — retry budget NOT recovered. + Timers.Cancel(StabilityTimerKey); + + _retryCount++; + + if (_retryCount > MaxRetries) + { + // Give up the stream, but do NOT stop the aggregator: the periodic reconcile + // still refreshes the cache from ClusterClient snapshots, so the page keeps a + // (slower) live-ish view rather than going dark. A later reconcile-triggered + // reconnect is not attempted here; the stream is simply left down. + _log.Error("Site-alarm gRPC stream for {0} exceeded max retries ({1}); leaving stream down, " + + "reconcile snapshots continue and the next reconcile tick will retry the stream", + _siteIdentifier, MaxRetries); + _streamDown = true; + CleanupGrpc(); + return; + } + + // Unsubscribe the failed stream on the previous endpoint (TryGet, never + // GetOrCreate) so the site releases its relay actor instead of leaving a zombie. + var previousEndpoint = _useNodeA ? _grpcNodeAAddress : _grpcNodeBAddress; + _grpcFactory.TryGet(_siteIdentifier, previousEndpoint)?.Unsubscribe(_correlationId); + + // Flip to the other node. + _useNodeA = !_useNodeA; + + // Telemetry: a NodeA↔NodeB failover flip is a reconnect + re-seed. + ScadaBridgeTelemetry.RecordLiveAlarmStreamReconnect(); + + // A failover flip must RE-SEED (never silently serve stale) — kick a reconcile + // fan-out alongside the reconnect. Buffering during the fan-out keeps the new + // stream's deltas coherent with the fresh snapshot. + StartFanout(isInitial: false); + + if (_retryCount == 1) + Self.Tell(new ReconnectAlarmStream()); + else + Timers.StartSingleTimer(ReconnectTimerKey, new ReconnectAlarmStream(), ReconnectDelay); + } + + private void CleanupGrpc() + { + _grpcCts?.Cancel(); + _grpcCts?.Dispose(); + _grpcCts = null; + + var endpoint = _useNodeA ? _grpcNodeAAddress : _grpcNodeBAddress; + _grpcFactory.TryGet(_siteIdentifier, endpoint)?.Unsubscribe(_correlationId); + } + + // ── Dedup key (copied from DebugStreamBridgeActor.AlarmKey with attribution) ── + + /// + /// NUL delimiter so distinct identities never collide on a shared boundary. Cannot + /// appear in an instance/alarm name. Mirrors . + /// + private const char KeyDelimiter = '\u0000'; + + /// + /// Per-alarm dedup identity = (InstanceUniqueName, AlarmName, SourceReference) — + /// identical to DebugStreamBridgeActor.AlarmKey so native per-condition alarms + /// sharing an AlarmName but differing by source reference are not conflated. Each + /// nullable component is guarded to prevent silent null/empty key collisions. + /// + private static string AlarmKey(AlarmStateChanged a) => + string.Concat( + a.InstanceUniqueName ?? string.Empty, KeyDelimiter, + a.AlarmName ?? string.Empty, KeyDelimiter, + a.SourceReference ?? string.Empty); +} + +/// Message asking a to stop (last viewer left). +public sealed record StopSiteAlarmAggregator; + +/// Internal: a seed/reconcile snapshot fan-out completed with the whole-site alarm rows. +internal sealed record SeedCompleted(IReadOnlyList Alarms, bool IsInitial); + +/// Internal: a seed/reconcile snapshot fan-out threw as a whole. +internal sealed record SeedFailed(Exception Exception, bool IsInitial); + +/// Internal: periodic reconcile tick (and the re-seed kicked after a reconnect). +internal sealed record RunReconcile; + +/// Internal: site-alarm gRPC stream error occurred. +internal sealed record GrpcAlarmStreamError(Exception Exception); + +/// Internal: reconnect the site-alarm gRPC stream (flip node). +internal sealed record ReconnectAlarmStream; + +/// Internal: the current site-alarm gRPC stream has stayed up long enough to recover the retry budget. +internal sealed record GrpcAlarmStreamStable; diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs index 5ae541b9..c5957947 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptions.cs @@ -88,4 +88,38 @@ public class CommunicationOptions /// that are deployed once and never re-deployed. Default 1 hour ≫ . /// public TimeSpan PendingDeploymentPurgeInterval { get; set; } = TimeSpan.FromHours(1); + + // ── Aggregated live alarm cache (plan #10, Task 4) ─────────────────────────── + // Introduced by Task 4 with sane defaults; Task 6 formalizes eager validation + // (positive interval/linger, positive concurrency/subscriber cap) and telemetry. + + /// + /// How long the shared per-site live alarm aggregator keeps its gRPC stream + + /// in-memory cache warm after its LAST Alarm Summary viewer leaves, before it is + /// torn down. A short linger avoids re-seed thrash when an operator navigates away + /// and straight back. Default 30s. + /// + public TimeSpan LiveAlarmCacheLinger { get; set; } = TimeSpan.FromSeconds(30); + + /// + /// Cadence of the per-site aggregator's periodic reconcile snapshot fan-out. The + /// backstop that corrects instance-set drift (enable/disable/deploy/delete) and any + /// alarm delta missed by the live stream, since the aggregated stream never carries + /// an explicit "removed" event. Default 60s. + /// + public TimeSpan LiveAlarmCacheReconcileInterval { get; set; } = TimeSpan.FromSeconds(60); + + /// + /// Max concurrent per-instance snapshot fetches during a live-cache seed/reconcile + /// fan-out — the aggregated analogue of the Alarm Summary poll's + /// MaxConcurrentFetches. Default 8. + /// + public int LiveAlarmCacheSeedConcurrency { get; set; } = 8; + + /// + /// Upper bound on concurrent Alarm Summary viewers sharing one site's aggregator. + /// A defensive cap only — one gRPC stream per site is shared regardless of viewer + /// count; this just bounds the subscriber list. Default 200. + /// + public int LiveAlarmCacheMaxSubscribersPerSite { get; set; } = 200; } diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptionsValidator.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptionsValidator.cs index d101741c..19236555 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptionsValidator.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/CommunicationOptionsValidator.cs @@ -65,5 +65,25 @@ public sealed class CommunicationOptionsValidator : OptionsValidatorBase 0, $"Communication:GrpcMaxConcurrentStreams must be positive (was {options.GrpcMaxConcurrentStreams})."); + + // ── Aggregated live alarm cache (plan #10, Task 6) ─────────────────────── + // Linger drives a Timer dueTime; TimeSpan.Zero is valid (stop the aggregator + // immediately when the last viewer leaves), only a negative value is invalid. + builder.RequireThat(options.LiveAlarmCacheLinger >= TimeSpan.Zero, + $"Communication:LiveAlarmCacheLinger must be zero or a positive duration (was {options.LiveAlarmCacheLinger})."); + + // Reconcile interval is a periodic timer cadence AND the start-retry cadence; a + // zero/negative value would spin or never fire. + builder.RequireThat(options.LiveAlarmCacheReconcileInterval > TimeSpan.Zero, + $"Communication:LiveAlarmCacheReconcileInterval must be a positive duration (was {options.LiveAlarmCacheReconcileInterval})."); + + // Seed fan-out concurrency gates a SemaphoreSlim; at least 1, capped so a + // fat-fingered config can't open an unbounded burst of cross-cluster Asks. + builder.RequireThat(options.LiveAlarmCacheSeedConcurrency is >= 1 and <= 64, + $"Communication:LiveAlarmCacheSeedConcurrency must be between 1 and 64 (was {options.LiveAlarmCacheSeedConcurrency})."); + + // Per-site viewer cap must admit at least one viewer, else the page could never go live. + builder.RequireThat(options.LiveAlarmCacheMaxSubscribersPerSite >= 1, + $"Communication:LiveAlarmCacheMaxSubscribersPerSite must be at least 1 (was {options.LiveAlarmCacheMaxSubscribersPerSite})."); } } diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/ISiteStreamSubscriber.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/ISiteStreamSubscriber.cs index 2995d7d2..398cbf3e 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/ISiteStreamSubscriber.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/ISiteStreamSubscriber.cs @@ -17,6 +17,16 @@ public interface ISiteStreamSubscriber /// A subscription ID that can be used for unsubscription. string Subscribe(string instanceName, IActorRef subscriber); + /// + /// Subscribes an actor to receive ALARM events for ALL instances on the site + /// (no per-instance filter). Only AlarmStateChanged events are + /// forwarded; attribute-value events are dropped. Backs the site-wide + /// SubscribeSite gRPC stream used by the aggregated Alarm Summary. + /// + /// The actor reference that will receive alarm stream event messages. + /// A subscription ID that can be used for unsubscription. + string SubscribeSiteAlarms(IActorRef subscriber); + /// /// Removes all subscriptions for the given actor. /// diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs index 8e57e851..65afa9ee 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs @@ -200,6 +200,74 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable } } + /// + /// Opens the site-wide, alarm-only server-streaming subscription + /// (SubscribeSite) for a whole site rather than a single instance. This is + /// the central per-site feed that backs the aggregated Alarm Summary live cache + /// (plan #10): the site runtime's SubscribeSiteAlarms hub drops the + /// per-instance filter and carries only events for + /// all instances on the site (attributes are deliberately excluded — the + /// summary never shows them and they are far higher-volume). + /// + /// The callback is deliberately typed as + /// rather than the generic Action<object> used by + /// : this stream is alarm-only by contract, so Task 4's + /// per-site cache consumes an alarm delta directly with no downstream type test. + /// The mapping is still shared via ; a non-alarm + /// event (which should never appear on this stream) is defensively ignored rather + /// than delivered or thrown. + /// + /// This is a long-running async method; the caller launches it as a background task. + /// Error handling, cancellation and cleanup mirror . + /// + /// Unique identifier for this subscription. + /// Callback invoked for each alarm delta received from the site-wide stream. + /// Callback invoked when the subscription encounters an error. + /// Cancellation token to stop the subscription. + /// A task that represents the asynchronous operation. + public virtual async Task SubscribeSiteAsync( + string correlationId, + Action onAlarmEvent, + Action onError, + CancellationToken ct) + { + if (_client is null) + throw new InvalidOperationException("Cannot subscribe on a test-only client."); + + var cts = CancellationTokenSource.CreateLinkedTokenSource(ct); + RegisterSubscription(correlationId, cts); + + var request = new SiteStreamRequest + { + CorrelationId = correlationId + }; + + try + { + using var call = _client.SubscribeSite(request, cancellationToken: cts.Token); + + await foreach (var evt in call.ResponseStream.ReadAllAsync(cts.Token)) + { + // Site-wide stream is alarm-only by contract; defensively ignore anything else. + if (ConvertToAlarmEvent(evt) is { } alarm) + onAlarmEvent(alarm); + } + } + catch (RpcException ex) when (ex.StatusCode == StatusCode.Cancelled) + { + // Normal cancellation — not an error + } + catch (Exception ex) + { + onError(ex); + } + finally + { + // Remove only our own entry -- a racing reconnect may already own the slot. + RemoveSubscription(correlationId, cts); + } + } + /// /// Cancels an active subscription by correlation ID. /// @@ -260,6 +328,17 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable _ => null }; + /// + /// Maps a proto to a domain , + /// returning null for any non-alarm event. Used by + /// to enforce the alarm-only contract of the site-wide stream without duplicating the + /// enrichment mapping in . Internal for testability. + /// + /// The protobuf site stream event to convert. + /// The mapped , or null if the event is not an alarm. + internal static AlarmStateChanged? ConvertToAlarmEvent(SiteStreamEvent evt) => + ConvertToDomainEvent(evt) as AlarmStateChanged; + /// Parses the wire "kind" string back to ; defaults to Computed. /// The wire "kind" string from the gRPC payload; null or unrecognised defaults to . /// The parsed , or when the value is null or unrecognised. diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs index a8e2c465..0fb89c4a 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs @@ -210,10 +210,52 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase internal TimeSpan MaxStreamLifetime => _maxStreamLifetime; /// - public override async Task SubscribeInstance( + public override Task SubscribeInstance( InstanceStreamRequest request, IServerStreamWriter responseStream, ServerCallContext context) + => RunSubscriptionStreamAsync( + request.CorrelationId, + responseStream, + context, + relay => _streamSubscriber.Subscribe(request.InstanceUniqueName, relay), + request.InstanceUniqueName); + + /// + public override Task SubscribeSite( + SiteStreamRequest request, + IServerStreamWriter responseStream, + ServerCallContext context) + => RunSubscriptionStreamAsync( + request.CorrelationId, + responseStream, + context, + // Site-wide, alarm-only: no per-instance filter. StreamRelayActor + // already drops IsConfiguredPlaceholder rows and maps only the + // enriched AlarmStateUpdate, so it is reused unchanged. + _streamSubscriber.SubscribeSiteAlarms, + "site-wide alarms"); + + /// + /// Shared streaming pipeline behind and + /// : readiness/shutdown guards, correlation-id + /// validation, duplicate replacement, the concurrency cap, the bounded + /// DropOldest channel, the relay actor, connection telemetry, and the + /// guaranteed cleanup. The only variation between the two RPCs is how the + /// relay actor is subscribed to the site broadcast hub — supplied by + /// — and the log/telemetry description. + /// + /// The client-supplied correlation id (also the actor-name/dedup key). + /// The gRPC response stream to pump events into. + /// The server call context (carries the client cancellation token). + /// Subscribes the relay actor to the hub, returning a subscription id. + /// Human-readable subscription description for logging. + private async Task RunSubscriptionStreamAsync( + string correlationId, + IServerStreamWriter responseStream, + ServerCallContext context, + Func subscribe, + string description) { if (!_ready) throw new RpcException(new GrpcStatus(StatusCode.Unavailable, "Server not ready")); @@ -228,15 +270,15 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase // have a restricted character set — a id containing '/', whitespace, or other // disallowed characters would make ActorOf throw InvalidActorNameException, // escaping as an unhandled RPC fault. Reject unsafe ids cleanly up front. - if (string.IsNullOrEmpty(request.CorrelationId) || - !ActorPath.IsValidPathElement(request.CorrelationId)) + if (string.IsNullOrEmpty(correlationId) || + !ActorPath.IsValidPathElement(correlationId)) { throw new RpcException(new GrpcStatus( StatusCode.InvalidArgument, "correlation_id is missing or not a valid identifier")); } // Duplicate prevention -- cancel existing stream for this correlationId - if (_activeStreams.TryRemove(request.CorrelationId, out var existingEntry)) + if (_activeStreams.TryRemove(correlationId, out var existingEntry)) { existingEntry.Cts.Cancel(); existingEntry.Cts.Dispose(); @@ -256,10 +298,9 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase if (_maxStreamLifetime > TimeSpan.Zero && _maxStreamLifetime != Timeout.InfiniteTimeSpan) streamCts.CancelAfter(_maxStreamLifetime); var entry = new StreamEntry(streamCts); - _activeStreams[request.CorrelationId] = entry; + _activeStreams[correlationId] = entry; long dropped = 0; - var correlationId = request.CorrelationId; var channel = Channel.CreateBounded( new BoundedChannelOptions(1000) { FullMode = BoundedChannelFullMode.DropOldest }, _ => @@ -275,8 +316,8 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase var actorSeq = Interlocked.Increment(ref _actorCounter); var relayActor = _actorSystem!.ActorOf( - Props.Create(typeof(Actors.StreamRelayActor), request.CorrelationId, channel.Writer), - $"stream-relay-{request.CorrelationId}-{actorSeq}"); + Props.Create(typeof(Actors.StreamRelayActor), correlationId, channel.Writer), + $"stream-relay-{correlationId}-{actorSeq}"); // The previous code called _streamSubscriber.Subscribe // OUTSIDE the try block that owns relay-actor cleanup. If Subscribe threw @@ -289,23 +330,23 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase string subscriptionId; try { - subscriptionId = _streamSubscriber.Subscribe(request.InstanceUniqueName, relayActor); + subscriptionId = subscribe(relayActor); } catch (Exception ex) { _logger.LogWarning(ex, - "Subscribe failed for {Instance} (correlation {CorrelationId}); cleaning up relay actor.", - request.InstanceUniqueName, request.CorrelationId); + "Subscribe failed for {Description} (correlation {CorrelationId}); cleaning up relay actor.", + description, correlationId); _actorSystem!.Stop(relayActor); channel.Writer.TryComplete(); _activeStreams.TryRemove( - new KeyValuePair(request.CorrelationId, entry)); + new KeyValuePair(correlationId, entry)); throw; } _logger.LogInformation( - "Stream {CorrelationId} started for {Instance} (subscription {SubscriptionId})", - request.CorrelationId, request.InstanceUniqueName, subscriptionId); + "Stream {CorrelationId} started for {Description} (subscription {SubscriptionId})", + correlationId, description, subscriptionId); // Telemetry follow-on: the connection is now fully established (Subscribe // succeeded, so no leak via the catch above). Count it up here and balance @@ -335,11 +376,11 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase // Only remove our own entry -- a replacement stream may have already taken the slot _activeStreams.TryRemove( - new KeyValuePair(request.CorrelationId, entry)); + new KeyValuePair(correlationId, entry)); _logger.LogInformation( - "Stream {CorrelationId} for {Instance} ended", - request.CorrelationId, request.InstanceUniqueName); + "Stream {CorrelationId} for {Description} ended", + correlationId, description); } } diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/ISiteAlarmLiveCache.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/ISiteAlarmLiveCache.cs new file mode 100644 index 00000000..44a13efe --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/ISiteAlarmLiveCache.cs @@ -0,0 +1,62 @@ +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; + +namespace ZB.MOM.WW.ScadaBridge.Communication; + +/// +/// Central, transient, in-memory per-site live alarm cache (plan #10, Task 4). +/// A DI singleton shared by every Alarm Summary Blazor circuit on the active central +/// node. For each site with at least one active viewer it runs ONE shared per-site +/// aggregator that seeds from the existing debug-snapshot fan-out and stays warm on a +/// single site-wide, alarm-only gRPC stream (SubscribeSite), so the page +/// reflects alarm transitions in near-real-time instead of re-polling every 15s. +/// +/// Hard constraint (locked [PERM]): there is NO persisted central alarm +/// store. This cache is purely in-memory and lives only on the active central node — +/// no EF entity/table/migration backs it. On a NodeA↔NodeB failover the new active +/// node re-seeds from scratch. +/// +/// +/// Reference-counted lifecycle: the first for a site +/// starts its aggregator (open stream + snapshot seed); the last subscriber leaving +/// stops it after a short linger (see +/// ) to avoid re-seed thrash. +/// One gRPC stream per site, not per browser circuit. +/// +/// +public interface ISiteAlarmLiveCache +{ + /// + /// Registers a viewer for a site's live alarm feed and returns an + /// that unregisters it. The first subscriber for a site + /// starts the shared aggregator; the last subscriber's + /// schedules a linger-delayed stop. is raised (on the + /// aggregator's thread) after each applied change — the handler must not block and + /// should marshal any UI work onto its own dispatcher (mirrors + /// IDeploymentStatusNotifier). Disposing the returned handle is idempotent. + /// + /// The numeric site id whose alarms the viewer wants. + /// Callback invoked whenever the site's cached alarm set changes. + /// A disposable that unregisters this viewer (idempotent). + IDisposable Subscribe(int siteId, Action onChanged); + + /// + /// Returns the current in-memory alarm snapshot for a site — exactly what the page + /// should render. Empty when the site has no aggregator or has not yet completed its + /// first seed. Safe to call from Blazor render threads: it returns an immutable + /// point-in-time list (the aggregator publishes fresh immutable lists; readers never + /// see a partially-mutated collection). + /// + /// The numeric site id. + /// The current cached alarms (possibly empty), never null. + IReadOnlyList GetCurrentAlarms(int siteId); + + /// + /// Whether the site's aggregator has completed its initial seed and is serving live + /// state. The page can use this to decide whether to keep its 15s poll fallback + /// (Task 5): false means "not live yet — keep polling"; true means the + /// cache is authoritative. + /// + /// The numeric site id. + /// true once the site's aggregator has seeded and published at least once. + bool IsLive(int siteId); +} diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto b/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto index a7c9c478..5876dcbf 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto @@ -7,6 +7,10 @@ import "google/protobuf/wrappers.proto"; // Int32Value service SiteStreamService { rpc SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent); + // Site-wide, alarm-only live stream (aggregated Alarm Summary): every + // AlarmStateChanged for ALL instances on the site, no per-instance filter. + // Attribute updates are never carried on this stream. + rpc SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent); rpc IngestAuditEvents(AuditEventBatch) returns (IngestAck); rpc IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck); rpc PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse); @@ -18,6 +22,13 @@ message InstanceStreamRequest { string instance_unique_name = 2; } +// Request for the site-wide, alarm-only SubscribeSite stream. Unlike +// InstanceStreamRequest there is NO instance filter — the stream carries alarm +// transitions for every instance on the site. +message SiteStreamRequest { + string correlation_id = 1; +} + message SiteStreamEvent { string correlation_id = 1; oneof event { diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/ServiceCollectionExtensions.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/ServiceCollectionExtensions.cs index dc9577f2..0fb9ccdd 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/ServiceCollectionExtensions.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/ServiceCollectionExtensions.cs @@ -22,6 +22,13 @@ public static class ServiceCollectionExtensions services.AddSingleton(); services.AddSingleton(); + // Aggregated live alarm cache (plan #10, Task 4): transient, in-memory, shared + // per-site aggregator backing the operator Alarm Summary page. Singleton so every + // Blazor circuit shares one aggregator per site; started/stopped by viewer + // reference-count. No persistence (locked [PERM]: no central alarm store). + services.AddSingleton(); + services.AddSingleton(sp => sp.GetRequiredService()); + // Startup reconciliation handler — scoped (holds scoped repositories), resolved // per-request by CentralCommunicationActor inside a DI scope. Harmless on site // hosts: only the central actor ever resolves it. diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs new file mode 100644 index 00000000..ea0c5036 --- /dev/null +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs @@ -0,0 +1,489 @@ +using System.Collections.Concurrent; +using Akka.Actor; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.Communication.Actors; +using ZB.MOM.WW.ScadaBridge.Communication.Grpc; + +namespace ZB.MOM.WW.ScadaBridge.Communication; + +/// +/// Default — the DI singleton that owns the shared, +/// reference-counted, transient per-site live alarm cache (plan #10, Task 4). For each +/// site with active Alarm Summary viewers it runs ONE +/// , seeded by the existing debug-snapshot fan-out +/// () and kept warm on a +/// single site-wide, alarm-only gRPC stream. The actor holds the state and enforces +/// seed-then-stream ordering, dedup, placeholder coherence, failover re-seed and periodic +/// reconcile; this service is the DI façade: viewer reference-counting, a linger-delayed +/// last-out stop, the seed fan-out closure, and the published-snapshot store the page reads. +/// +/// No persistence (locked [PERM]): nothing here writes to the database — the +/// cache lives only in the actor's memory on the active central node. +/// +/// +public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache +{ + private static readonly IReadOnlyList Empty = Array.Empty(); + + private readonly IServiceProvider _serviceProvider; + private readonly CommunicationService _communicationService; + private readonly SiteStreamGrpcClientFactory _grpcFactory; + private readonly CommunicationOptions _options; + private readonly ILogger _logger; + + private readonly object _lock = new(); + private readonly Dictionary _sites = new(); + private ActorSystem? _actorSystem; + + /// + /// Initializes the live alarm cache service. + /// + /// Root provider; scopes are created per snapshot fan-out to resolve repositories. + /// Central-side comms used to Ask each instance for a debug snapshot (the seed). + /// Factory for the per-site site-wide alarm gRPC client. + /// Communication options (linger, reconcile interval, seed concurrency, subscriber cap). + /// Logger. + public SiteAlarmLiveCacheService( + IServiceProvider serviceProvider, + CommunicationService communicationService, + SiteStreamGrpcClientFactory grpcFactory, + IOptions options, + ILogger logger) + { + _serviceProvider = serviceProvider; + _communicationService = communicationService; + _grpcFactory = grpcFactory; + _options = options.Value; + _logger = logger; + } + + /// + /// Supplies the once it exists (called from + /// AkkaHostedService on the active central node, mirroring + /// ). Until set, + /// registers viewers but cannot start an aggregator, so the page keeps its poll fallback. + /// + /// The actor system to host per-site aggregators. + public void SetActorSystem(ActorSystem actorSystem) => _actorSystem = actorSystem; + + /// + public IDisposable Subscribe(int siteId, Action onChanged) + { + ArgumentNullException.ThrowIfNull(onChanged); + + var subscription = new Subscription(this, siteId, onChanged); + bool startNeeded; + SiteEntry entry; + + lock (_lock) + { + if (!_sites.TryGetValue(siteId, out entry!)) + { + entry = new SiteEntry(siteId); + _sites[siteId] = entry; + } + + // A pending linger stop is now moot — a viewer arrived. Invalidate it. + entry.CancelLinger(); + + // Enforce the per-site viewer cap — fail SAFE: reject the excess viewer with a + // no-op disposable rather than throwing (this runs on a Blazor render path) or + // growing the subscriber list unbounded. The shared aggregator keeps serving the + // already-registered viewers; the rejected circuit just keeps its 15s poll + // fallback. Reaching the cap almost always means a Dispose leak or a runaway page. + if (entry.Subscribers.Count >= _options.LiveAlarmCacheMaxSubscribersPerSite) + { + _logger.LogWarning( + "Site {SiteId} live alarm cache is at its viewer cap ({Cap}); rejecting the new " + + "viewer (it will keep polling). This indicates a subscription leak or a very busy page.", + siteId, _options.LiveAlarmCacheMaxSubscribersPerSite); + return NoOpSubscription.Instance; + } + + entry.Subscribers.Add(subscription); + + // Start the aggregator on the first viewer (or if a prior start failed and left + // no actor). Guarded by Starting so concurrent first-viewers don't double-start. + startNeeded = entry.Actor is null && !entry.Starting; + if (startNeeded) + entry.Starting = true; + } + + if (startNeeded) + _ = StartAggregatorAsync(entry); + + return subscription; + } + + /// + public IReadOnlyList GetCurrentAlarms(int siteId) + { + lock (_lock) + return _sites.TryGetValue(siteId, out var entry) ? entry.Current : Empty; + } + + /// + public bool IsLive(int siteId) + { + lock (_lock) + return _sites.TryGetValue(siteId, out var entry) && entry.HasPublished; + } + + // ── Subscriber teardown ───────────────────────────────────────────────────── + + private void Unsubscribe(Subscription subscription) + { + lock (_lock) + { + if (!_sites.TryGetValue(subscription.SiteId, out var entry)) + return; + + if (!entry.Subscribers.Remove(subscription)) + return; // already removed (idempotent Dispose) + + if (entry.Subscribers.Count == 0) + ScheduleLingerStop(entry); + } + } + + /// + /// Schedules a linger-delayed stop for a site whose last viewer just left. A + /// re-subscribe within the linger window cancels it (). + /// Must be called under . + /// + private void ScheduleLingerStop(SiteEntry entry) + { + var version = ++entry.LingerVersion; + entry.LingerTimer?.Dispose(); + entry.LingerTimer = new Timer( + _ => LingerStop(entry.SiteId, version), + state: null, + dueTime: _options.LiveAlarmCacheLinger, + period: Timeout.InfiniteTimeSpan); + } + + private void LingerStop(int siteId, int version) + { + IActorRef? actorToStop = null; + lock (_lock) + { + if (!_sites.TryGetValue(siteId, out var entry)) + return; + // A viewer may have returned (version bumped) or re-subscribed (subscribers > 0) + // after this timer was armed — only stop if this is still the latest, empty state. + if (entry.LingerVersion != version || entry.Subscribers.Count != 0) + return; + + actorToStop = entry.Actor; + entry.DisposeTimers(); + _sites.Remove(siteId); + } + + actorToStop?.Tell(new StopSiteAlarmAggregator()); + _logger.LogDebug("Stopped live alarm aggregator for site {SiteId} (last viewer left)", siteId); + } + + // ── Aggregator start ──────────────────────────────────────────────────────── + + private async Task StartAggregatorAsync(SiteEntry entry) + { + try + { + var system = _actorSystem; + if (system is null) + { + _logger.LogWarning( + "Live alarm cache asked to start site {SiteId} before the ActorSystem was set; " + + "the page will keep its poll fallback", entry.SiteId); + return; + } + + var resolved = await ResolveSiteAsync(entry.SiteId); + if (resolved is null) + return; // logged inside ResolveSiteAsync + + var (siteIdentifier, grpcA, grpcB) = resolved.Value; + var correlationId = $"site-alarm-{entry.SiteId}-{Guid.NewGuid():N}"; + + Func>> seedFn = + ct => FanOutSnapshotsAsync(entry.SiteId, ct); + Action> publish = + snapshot => OnPublish(entry, snapshot); + + lock (_lock) + { + // Bail if the entry was torn down (last viewer left + linger fired) while we + // resolved the site, or if a racing start already created the actor. + if (!_sites.TryGetValue(entry.SiteId, out var current) || !ReferenceEquals(current, entry)) + return; + if (entry.Actor is not null) + return; + + var props = Props.Create(() => new SiteAlarmAggregatorActor( + siteIdentifier, + correlationId, + seedFn, + publish, + _grpcFactory, + grpcA, + grpcB, + _options.LiveAlarmCacheReconcileInterval)); + + entry.Actor = system.ActorOf(props, $"site-alarm-aggregator-{entry.SiteId}-{Guid.NewGuid():N}"); + entry.StartRetryTimer?.Dispose(); + entry.StartRetryTimer = null; + _logger.LogInformation("Started live alarm aggregator for site {SiteId} ({SiteIdentifier})", + entry.SiteId, siteIdentifier); + } + } + catch (Exception ex) + { + _logger.LogError(ex, "Failed to start live alarm aggregator for site {SiteId}", entry.SiteId); + } + finally + { + lock (_lock) + { + entry.Starting = false; + + // A transient start failure (ActorSystem not yet set, a DB blip resolving + // the site, ActorOf throwing) would otherwise leave this viewer cohort + // polling forever — nothing re-attempts until the NEXT Subscribe. Arm a + // bounded retry so the feature self-heals once the condition clears, as + // long as viewers remain and the entry is still live. + var stillWanted = _sites.TryGetValue(entry.SiteId, out var current) + && ReferenceEquals(current, entry) + && entry.Actor is null + && entry.Subscribers.Count > 0; + if (stillWanted) + { + entry.Starting = true; // keep the guard closed so a Subscribe won't double-start + entry.StartRetryTimer?.Dispose(); + entry.StartRetryTimer = new Timer( + _ => { _ = StartAggregatorAsync(entry); }, + state: null, + dueTime: _options.LiveAlarmCacheReconcileInterval, + period: Timeout.InfiniteTimeSpan); + } + } + } + } + + /// + /// Resolves a site's identifier + both gRPC node endpoints. Returns null (and + /// logs) when the site is unknown or has no gRPC addresses configured, in which case + /// no aggregator is started and the page keeps polling. + /// + private async Task<(string SiteIdentifier, string GrpcA, string GrpcB)?> ResolveSiteAsync(int siteId) + { + using var scope = _serviceProvider.CreateScope(); + var siteRepo = scope.ServiceProvider.GetRequiredService(); + var site = await siteRepo.GetSiteByIdAsync(siteId); + if (site is null) + { + _logger.LogWarning("Live alarm cache: site {SiteId} not found; not starting an aggregator", siteId); + return null; + } + + if (string.IsNullOrWhiteSpace(site.GrpcNodeAAddress) || string.IsNullOrWhiteSpace(site.GrpcNodeBAddress)) + { + _logger.LogWarning( + "Live alarm cache: site {SiteId} ({SiteIdentifier}) has no gRPC node addresses; " + + "not starting an aggregator", siteId, site.SiteIdentifier); + return null; + } + + return (site.SiteIdentifier, site.GrpcNodeAAddress!, site.GrpcNodeBAddress!); + } + + // ── Snapshot fan-out (the seed / reconcile) ───────────────────────────────── + + /// + /// Fans out one debug-snapshot Ask per Enabled instance on the site (capped at + /// ) and returns every + /// snapshot's alarm rows — including + /// placeholders. Best-effort: an instance that faults, times out, or reports + /// contributes no rows rather than + /// failing the whole seed (mirrors AlarmSummaryService). Re-enumerates instances + /// each call so a reconcile picks up enable/disable/deploy/delete drift. + /// + private async Task> FanOutSnapshotsAsync(int siteId, CancellationToken ct) + { + string siteIdentifier; + List enabledInstanceNames; + + using (var scope = _serviceProvider.CreateScope()) + { + var siteRepo = scope.ServiceProvider.GetRequiredService(); + var site = await siteRepo.GetSiteByIdAsync(siteId, ct); + if (site is null) + return Empty; + siteIdentifier = site.SiteIdentifier; + + var instanceRepo = scope.ServiceProvider.GetRequiredService(); + var instances = await instanceRepo.GetInstancesBySiteIdAsync(siteId, ct); + enabledInstanceNames = instances + .Where(i => i.State == InstanceState.Enabled) + .Select(i => i.UniqueName) + .ToList(); + } + + if (enabledInstanceNames.Count == 0) + return Empty; + + var rows = new ConcurrentBag(); + using var gate = new SemaphoreSlim( + Math.Max(1, _options.LiveAlarmCacheSeedConcurrency), + Math.Max(1, _options.LiveAlarmCacheSeedConcurrency)); + + var fetches = enabledInstanceNames.Select(name => + FetchInstanceSnapshotAsync(siteIdentifier, name, gate, rows, ct)); + await Task.WhenAll(fetches); + + return rows.ToList(); + } + + private async Task FetchInstanceSnapshotAsync( + string siteIdentifier, + string instanceUniqueName, + SemaphoreSlim gate, + ConcurrentBag rows, + CancellationToken ct) + { + await gate.WaitAsync(ct); + try + { + var request = new DebugSnapshotRequest(instanceUniqueName, Guid.NewGuid().ToString("N")); + var snapshot = await _communicationService.RequestDebugSnapshotAsync(siteIdentifier, request, ct); + if (snapshot.InstanceNotFound) + return; + + foreach (var alarm in snapshot.AlarmStates) + rows.Add(alarm); + } + catch (OperationCanceledException) when (ct.IsCancellationRequested) + { + throw; // seed cancelled (actor stopping) — propagate + } + catch + { + // Degrade this one instance to "no rows" rather than failing the whole seed. + } + finally + { + gate.Release(); + } + } + + // ── Publish (actor thread → viewers) ──────────────────────────────────────── + + private void OnPublish(SiteEntry entry, IReadOnlyList snapshot) + { + Subscription[] subscribers; + lock (_lock) + { + // Store the fresh immutable snapshot (readers get it lock-free-ish via GetCurrentAlarms). + entry.Current = snapshot; + entry.HasPublished = true; + subscribers = entry.Subscribers.ToArray(); + } + + // Invoke viewer callbacks OUTSIDE the lock — a Blazor InvokeAsync(StateHasChanged) + // must never re-enter our lock, and one faulting viewer must not stop the others. + foreach (var subscriber in subscribers) + { + try + { + subscriber.OnChanged(); + } + catch (Exception ex) + { + _logger.LogWarning(ex, + "A live alarm viewer callback threw for site {SiteId}; continuing", entry.SiteId); + } + } + } + + // ── Nested types ──────────────────────────────────────────────────────────── + + /// Per-site bookkeeping. All mutable fields are guarded by the service's _lock. + private sealed class SiteEntry + { + public SiteEntry(int siteId) => SiteId = siteId; + + public int SiteId { get; } + public List Subscribers { get; } = new(); + public IActorRef? Actor { get; set; } + public bool Starting { get; set; } + + /// Current published snapshot the page renders. Swapped as a whole immutable list. + public IReadOnlyList Current { get; set; } = Empty; + + /// True once the aggregator has seeded and published at least once. + public bool HasPublished { get; set; } + + public Timer? LingerTimer { get; set; } + public int LingerVersion { get; set; } + + /// Bounded retry timer armed when a start attempt failed to create the actor. + public Timer? StartRetryTimer { get; set; } + + /// Cancels any pending linger stop (a viewer arrived). + public void CancelLinger() + { + LingerVersion++; + LingerTimer?.Dispose(); + LingerTimer = null; + } + + /// Disposes both timers — called when the entry is torn down. + public void DisposeTimers() + { + LingerTimer?.Dispose(); + LingerTimer = null; + StartRetryTimer?.Dispose(); + StartRetryTimer = null; + } + } + + /// + /// The handle returned when a Subscribe is rejected at the per-site viewer cap. It was + /// never registered, so is a genuine no-op — safe to dispose any + /// number of times and it never touches the site's subscriber list. + /// + private sealed class NoOpSubscription : IDisposable + { + public static readonly NoOpSubscription Instance = new(); + private NoOpSubscription() { } + public void Dispose() { } + } + + /// The disposable handed to a viewer; idempotent unregisters it. + private sealed class Subscription : IDisposable + { + private readonly SiteAlarmLiveCacheService _owner; + private int _disposed; + + public Subscription(SiteAlarmLiveCacheService owner, int siteId, Action onChanged) + { + _owner = owner; + SiteId = siteId; + OnChanged = onChanged; + } + + public int SiteId { get; } + public Action OnChanged { get; } + + public void Dispose() + { + if (Interlocked.Exchange(ref _disposed, 1) == 0) + _owner.Unsubscribe(this); + } + } +} diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/Sitestream.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/Sitestream.cs index ca300fa5..6cd9eb49 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/Sitestream.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/Sitestream.cs @@ -28,6 +28,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { "L3Byb3RvYnVmL3RpbWVzdGFtcC5wcm90bxoeZ29vZ2xlL3Byb3RvYnVmL3dy", "YXBwZXJzLnByb3RvIk0KFUluc3RhbmNlU3RyZWFtUmVxdWVzdBIWCg5jb3Jy", "ZWxhdGlvbl9pZBgBIAEoCRIcChRpbnN0YW5jZV91bmlxdWVfbmFtZRgCIAEo", + "CSIrChFTaXRlU3RyZWFtUmVxdWVzdBIWCg5jb3JyZWxhdGlvbl9pZBgBIAEo", "CSKoAQoPU2l0ZVN0cmVhbUV2ZW50EhYKDmNvcnJlbGF0aW9uX2lkGAEgASgJ", "Ej0KEWF0dHJpYnV0ZV9jaGFuZ2VkGAIgASgLMiAuc2l0ZXN0cmVhbS5BdHRy", "aWJ1dGVWYWx1ZVVwZGF0ZUgAEjUKDWFsYXJtX2NoYW5nZWQYAyABKAsyHC5z", @@ -95,22 +96,25 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { "VEFURV9BQ1RJVkUQAiqFAQoOQWxhcm1MZXZlbEVudW0SFAoQQUxBUk1fTEVW", "RUxfTk9ORRAAEhMKD0FMQVJNX0xFVkVMX0xPVxABEhcKE0FMQVJNX0xFVkVM", "X0xPV19MT1cQAhIUChBBTEFSTV9MRVZFTF9ISUdIEAMSGQoVQUxBUk1fTEVW", - "RUxfSElHSF9ISUdIEAQytwMKEVNpdGVTdHJlYW1TZXJ2aWNlElUKEVN1YnNj", + "RUxfSElHSF9ISUdIEAQyhgQKEVNpdGVTdHJlYW1TZXJ2aWNlElUKEVN1YnNj", "cmliZUluc3RhbmNlEiEuc2l0ZXN0cmVhbS5JbnN0YW5jZVN0cmVhbVJlcXVl", - "c3QaGy5zaXRlc3RyZWFtLlNpdGVTdHJlYW1FdmVudDABEkcKEUluZ2VzdEF1", - "ZGl0RXZlbnRzEhsuc2l0ZXN0cmVhbS5BdWRpdEV2ZW50QmF0Y2gaFS5zaXRl", - "c3RyZWFtLkluZ2VzdEFjaxJQChVJbmdlc3RDYWNoZWRUZWxlbWV0cnkSIC5z", - "aXRlc3RyZWFtLkNhY2hlZFRlbGVtZXRyeUJhdGNoGhUuc2l0ZXN0cmVhbS5J", - "bmdlc3RBY2sSWgoPUHVsbEF1ZGl0RXZlbnRzEiIuc2l0ZXN0cmVhbS5QdWxs", - "QXVkaXRFdmVudHNSZXF1ZXN0GiMuc2l0ZXN0cmVhbS5QdWxsQXVkaXRFdmVu", - "dHNSZXNwb25zZRJUCg1QdWxsU2l0ZUNhbGxzEiAuc2l0ZXN0cmVhbS5QdWxs", - "U2l0ZUNhbGxzUmVxdWVzdBohLnNpdGVzdHJlYW0uUHVsbFNpdGVDYWxsc1Jl", - "c3BvbnNlQiuqAihaQi5NT00uV1cuU2NhZGFCcmlkZ2UuQ29tbXVuaWNhdGlv", - "bi5HcnBjYgZwcm90bzM=")); + "c3QaGy5zaXRlc3RyZWFtLlNpdGVTdHJlYW1FdmVudDABEk0KDVN1YnNjcmli", + "ZVNpdGUSHS5zaXRlc3RyZWFtLlNpdGVTdHJlYW1SZXF1ZXN0Ghsuc2l0ZXN0", + "cmVhbS5TaXRlU3RyZWFtRXZlbnQwARJHChFJbmdlc3RBdWRpdEV2ZW50cxIb", + "LnNpdGVzdHJlYW0uQXVkaXRFdmVudEJhdGNoGhUuc2l0ZXN0cmVhbS5Jbmdl", + "c3RBY2sSUAoVSW5nZXN0Q2FjaGVkVGVsZW1ldHJ5EiAuc2l0ZXN0cmVhbS5D", + "YWNoZWRUZWxlbWV0cnlCYXRjaBoVLnNpdGVzdHJlYW0uSW5nZXN0QWNrEloK", + "D1B1bGxBdWRpdEV2ZW50cxIiLnNpdGVzdHJlYW0uUHVsbEF1ZGl0RXZlbnRz", + "UmVxdWVzdBojLnNpdGVzdHJlYW0uUHVsbEF1ZGl0RXZlbnRzUmVzcG9uc2US", + "VAoNUHVsbFNpdGVDYWxscxIgLnNpdGVzdHJlYW0uUHVsbFNpdGVDYWxsc1Jl", + "cXVlc3QaIS5zaXRlc3RyZWFtLlB1bGxTaXRlQ2FsbHNSZXNwb25zZUIrqgIo", + "WkIuTU9NLldXLlNjYWRhQnJpZGdlLkNvbW11bmljYXRpb24uR3JwY2IGcHJv", + "dG8z")); descriptor = pbr::FileDescriptor.FromGeneratedCode(descriptorData, new pbr::FileDescriptor[] { global::Google.Protobuf.WellKnownTypes.TimestampReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.WrappersReflection.Descriptor, }, new pbr::GeneratedClrTypeInfo(new[] {typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.Quality), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmStateEnum), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmLevelEnum), }, null, new pbr::GeneratedClrTypeInfo[] { new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.InstanceStreamRequest), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.InstanceStreamRequest.Parser, new[]{ "CorrelationId", "InstanceUniqueName" }, null, null, null, null), + new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest.Parser, new[]{ "CorrelationId" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEvent), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEvent.Parser, new[]{ "CorrelationId", "AttributeChanged", "AlarmChanged" }, new[]{ "Event" }, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AttributeValueUpdate), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AttributeValueUpdate.Parser, new[]{ "InstanceUniqueName", "AttributePath", "AttributeName", "Value", "Quality", "Timestamp" }, null, null, null, null), new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmStateUpdate), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmStateUpdate.Parser, new[]{ "InstanceUniqueName", "AlarmName", "State", "Priority", "Timestamp", "Level", "Message", "Kind", "Active", "Acknowledged", "Confirmed", "ShelveState", "Suppressed", "SourceReference", "AlarmTypeName", "Category", "OperatorUser", "OperatorComment", "OriginalRaiseTime", "CurrentValue", "LimitValue", "NativeSourceCanonicalName", "IsConfiguredPlaceholder" }, null, null, null, null), @@ -394,6 +398,209 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { } + /// + /// Request for the site-wide, alarm-only SubscribeSite stream. Unlike + /// InstanceStreamRequest there is NO instance filter — the stream carries alarm + /// transitions for every instance on the site. + /// + [global::System.Diagnostics.DebuggerDisplayAttribute("{ToString(),nq}")] + public sealed partial class SiteStreamRequest : pb::IMessage + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + , pb::IBufferMessage + #endif + { + private static readonly pb::MessageParser _parser = new pb::MessageParser(() => new SiteStreamRequest()); + private pb::UnknownFieldSet _unknownFields; + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public static pb::MessageParser Parser { get { return _parser; } } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public static pbr::MessageDescriptor Descriptor { + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[1]; } + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + pbr::MessageDescriptor pb::IMessage.Descriptor { + get { return Descriptor; } + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public SiteStreamRequest() { + OnConstruction(); + } + + partial void OnConstruction(); + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public SiteStreamRequest(SiteStreamRequest other) : this() { + correlationId_ = other.correlationId_; + _unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public SiteStreamRequest Clone() { + return new SiteStreamRequest(this); + } + + /// Field number for the "correlation_id" field. + public const int CorrelationIdFieldNumber = 1; + private string correlationId_ = ""; + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public string CorrelationId { + get { return correlationId_; } + set { + correlationId_ = pb::ProtoPreconditions.CheckNotNull(value, "value"); + } + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public override bool Equals(object other) { + return Equals(other as SiteStreamRequest); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public bool Equals(SiteStreamRequest other) { + if (ReferenceEquals(other, null)) { + return false; + } + if (ReferenceEquals(other, this)) { + return true; + } + if (CorrelationId != other.CorrelationId) return false; + return Equals(_unknownFields, other._unknownFields); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public override int GetHashCode() { + int hash = 1; + if (CorrelationId.Length != 0) hash ^= CorrelationId.GetHashCode(); + if (_unknownFields != null) { + hash ^= _unknownFields.GetHashCode(); + } + return hash; + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public override string ToString() { + return pb::JsonFormatter.ToDiagnosticString(this); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public void WriteTo(pb::CodedOutputStream output) { + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + output.WriteRawMessage(this); + #else + if (CorrelationId.Length != 0) { + output.WriteRawTag(10); + output.WriteString(CorrelationId); + } + if (_unknownFields != null) { + _unknownFields.WriteTo(output); + } + #endif + } + + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + void pb::IBufferMessage.InternalWriteTo(ref pb::WriteContext output) { + if (CorrelationId.Length != 0) { + output.WriteRawTag(10); + output.WriteString(CorrelationId); + } + if (_unknownFields != null) { + _unknownFields.WriteTo(ref output); + } + } + #endif + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public int CalculateSize() { + int size = 0; + if (CorrelationId.Length != 0) { + size += 1 + pb::CodedOutputStream.ComputeStringSize(CorrelationId); + } + if (_unknownFields != null) { + size += _unknownFields.CalculateSize(); + } + return size; + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public void MergeFrom(SiteStreamRequest other) { + if (other == null) { + return; + } + if (other.CorrelationId.Length != 0) { + CorrelationId = other.CorrelationId; + } + _unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields); + } + + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + public void MergeFrom(pb::CodedInputStream input) { + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + input.ReadRawMessage(this); + #else + uint tag; + while ((tag = input.ReadTag()) != 0) { + if ((tag & 7) == 4) { + // Abort on any end group tag. + return; + } + switch(tag) { + default: + _unknownFields = pb::UnknownFieldSet.MergeFieldFrom(_unknownFields, input); + break; + case 10: { + CorrelationId = input.ReadString(); + break; + } + } + } + #endif + } + + #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE + [global::System.Diagnostics.DebuggerNonUserCodeAttribute] + [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] + void pb::IBufferMessage.InternalMergeFrom(ref pb::ParseContext input) { + uint tag; + while ((tag = input.ReadTag()) != 0) { + if ((tag & 7) == 4) { + // Abort on any end group tag. + return; + } + switch(tag) { + default: + _unknownFields = pb::UnknownFieldSet.MergeFieldFrom(_unknownFields, ref input); + break; + case 10: { + CorrelationId = input.ReadString(); + break; + } + } + } + } + #endif + + } + [global::System.Diagnostics.DebuggerDisplayAttribute("{ToString(),nq}")] public sealed partial class SiteStreamEvent : pb::IMessage #if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE @@ -409,7 +616,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[1]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[2]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -740,7 +947,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[2]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[3]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -1132,7 +1339,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[3]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[4]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -2201,7 +2408,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[4]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[5]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -3216,7 +3423,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[5]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[6]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -3403,7 +3610,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[6]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[7]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -3596,7 +3803,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[7]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[8]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -4254,7 +4461,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[8]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[9]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -4507,7 +4714,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[9]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[10]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -4702,7 +4909,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[10]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[11]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -4946,7 +5153,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[11]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[12]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -5179,7 +5386,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[12]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[13]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] @@ -5469,7 +5676,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.Diagnostics.DebuggerNonUserCodeAttribute] [global::System.CodeDom.Compiler.GeneratedCode("protoc", null)] public static pbr::MessageDescriptor Descriptor { - get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[13]; } + get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[14]; } } [global::System.Diagnostics.DebuggerNonUserCodeAttribute] diff --git a/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/SitestreamGrpc.cs b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/SitestreamGrpc.cs index b57de38e..6db5da2a 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/SitestreamGrpc.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Communication/SiteStreamGrpc/SitestreamGrpc.cs @@ -50,6 +50,8 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { [global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)] static readonly grpc::Marshaller __Marshaller_sitestream_SiteStreamEvent = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEvent.Parser)); [global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)] + static readonly grpc::Marshaller __Marshaller_sitestream_SiteStreamRequest = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest.Parser)); + [global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)] static readonly grpc::Marshaller __Marshaller_sitestream_AuditEventBatch = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch.Parser)); [global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)] static readonly grpc::Marshaller __Marshaller_sitestream_IngestAck = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck.Parser)); @@ -72,6 +74,14 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { __Marshaller_sitestream_InstanceStreamRequest, __Marshaller_sitestream_SiteStreamEvent); + [global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)] + static readonly grpc::Method __Method_SubscribeSite = new grpc::Method( + grpc::MethodType.ServerStreaming, + __ServiceName, + "SubscribeSite", + __Marshaller_sitestream_SiteStreamRequest, + __Marshaller_sitestream_SiteStreamEvent); + [global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)] static readonly grpc::Method __Method_IngestAuditEvents = new grpc::Method( grpc::MethodType.Unary, @@ -120,6 +130,21 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, "")); } + /// + /// Site-wide, alarm-only live stream (aggregated Alarm Summary): every + /// AlarmStateChanged for ALL instances on the site, no per-instance filter. + /// Attribute updates are never carried on this stream. + /// + /// The request received from the client. + /// Used for sending responses back to the client. + /// The context of the server-side call handler being invoked. + /// A task indicating completion of the handler. + [global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)] + public virtual global::System.Threading.Tasks.Task SubscribeSite(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest request, grpc::IServerStreamWriter responseStream, grpc::ServerCallContext context) + { + throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, "")); + } + [global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)] public virtual global::System.Threading.Tasks.Task IngestAuditEvents(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch request, grpc::ServerCallContext context) { @@ -183,6 +208,34 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { { return CallInvoker.AsyncServerStreamingCall(__Method_SubscribeInstance, null, options, request); } + /// + /// Site-wide, alarm-only live stream (aggregated Alarm Summary): every + /// AlarmStateChanged for ALL instances on the site, no per-instance filter. + /// Attribute updates are never carried on this stream. + /// + /// The request to send to the server. + /// The initial metadata to send with the call. This parameter is optional. + /// An optional deadline for the call. The call will be cancelled if deadline is hit. + /// An optional token for canceling the call. + /// The call object. + [global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)] + public virtual grpc::AsyncServerStreamingCall SubscribeSite(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken)) + { + return SubscribeSite(request, new grpc::CallOptions(headers, deadline, cancellationToken)); + } + /// + /// Site-wide, alarm-only live stream (aggregated Alarm Summary): every + /// AlarmStateChanged for ALL instances on the site, no per-instance filter. + /// Attribute updates are never carried on this stream. + /// + /// The request to send to the server. + /// The options for the call. + /// The call object. + [global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)] + public virtual grpc::AsyncServerStreamingCall SubscribeSite(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest request, grpc::CallOptions options) + { + return CallInvoker.AsyncServerStreamingCall(__Method_SubscribeSite, null, options, request); + } [global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)] public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck IngestAuditEvents(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken)) { @@ -278,6 +331,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { { return grpc::ServerServiceDefinition.CreateBuilder() .AddMethod(__Method_SubscribeInstance, serviceImpl.SubscribeInstance) + .AddMethod(__Method_SubscribeSite, serviceImpl.SubscribeSite) .AddMethod(__Method_IngestAuditEvents, serviceImpl.IngestAuditEvents) .AddMethod(__Method_IngestCachedTelemetry, serviceImpl.IngestCachedTelemetry) .AddMethod(__Method_PullAuditEvents, serviceImpl.PullAuditEvents) @@ -292,6 +346,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc { public static void BindService(grpc::ServiceBinderBase serviceBinder, SiteStreamServiceBase serviceImpl) { serviceBinder.AddMethod(__Method_SubscribeInstance, serviceImpl == null ? null : new grpc::ServerStreamingServerMethod(serviceImpl.SubscribeInstance)); + serviceBinder.AddMethod(__Method_SubscribeSite, serviceImpl == null ? null : new grpc::ServerStreamingServerMethod(serviceImpl.SubscribeSite)); serviceBinder.AddMethod(__Method_IngestAuditEvents, serviceImpl == null ? null : new grpc::UnaryServerMethod(serviceImpl.IngestAuditEvents)); serviceBinder.AddMethod(__Method_IngestCachedTelemetry, serviceImpl == null ? null : new grpc::UnaryServerMethod(serviceImpl.IngestCachedTelemetry)); serviceBinder.AddMethod(__Method_PullAuditEvents, serviceImpl == null ? null : new grpc::UnaryServerMethod(serviceImpl.PullAuditEvents)); diff --git a/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs b/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs index 99141f60..7f060921 100644 --- a/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs +++ b/src/ZB.MOM.WW.ScadaBridge.Host/Actors/AkkaHostedService.cs @@ -424,6 +424,11 @@ akka {{ var debugStreamService = _serviceProvider.GetService(); debugStreamService?.SetActorSystem(_actorSystem!); + // Wire up the aggregated live alarm cache with the ActorSystem so it can host + // per-site aggregator actors on the active central node (plan #10, Task 4). + var siteAlarmLiveCache = _serviceProvider.GetService(); + siteAlarmLiveCache?.SetActorSystem(_actorSystem!); + // Management Service — accessible via ClusterClient var mgmtLogger = _serviceProvider.GetRequiredService() diff --git a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Streaming/SiteStreamManager.cs b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Streaming/SiteStreamManager.cs index 7d9a0da8..a49fedb4 100644 --- a/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Streaming/SiteStreamManager.cs +++ b/src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Streaming/SiteStreamManager.cs @@ -20,6 +20,9 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Streaming; /// public class SiteStreamManager : ISiteStreamSubscriber { + /// Sentinel instance name recorded for site-wide (non-instance-scoped) subscriptions. + private const string SiteWideInstanceName = "*"; + private ActorSystem? _system; private IMaterializer? _materializer; private readonly int _bufferSize; @@ -117,6 +120,44 @@ public class SiteStreamManager : ISiteStreamSubscriber return subscriptionId; } + /// + /// Subscribe to ALARM events for ALL instances on the site (no per-instance + /// filter). Only events are forwarded; + /// events are dropped (attributes are far + /// higher-volume and the aggregated Alarm Summary never shows them). Same + /// broadcast-hub wiring as , and the returned + /// subscription id is torn down via exactly like the + /// per-instance variant. + /// + /// The actor that receives forwarded events. + /// A subscription id to pass to . + public string SubscribeSiteAlarms(IActorRef subscriber) + { + if (_hubSource is null || _materializer is null) + throw new InvalidOperationException("SiteStreamManager.Initialize must be called before SubscribeSiteAlarms"); + + var subscriptionId = Guid.NewGuid().ToString(); + var capturedSubscriber = subscriber; + + var killSwitch = _hubSource + .Where(ev => ev is AlarmStateChanged) + .Buffer(_bufferSize, OverflowStrategy.DropHead) + .ViaMaterialized(KillSwitches.Single(), Keep.Right) + .To(Sink.ForEach(ev => capturedSubscriber.Tell(ev))) + .Run(_materializer); + + lock (_lock) + { + _subscriptions[subscriptionId] = new SubscriptionInfo( + SiteWideInstanceName, subscriber, killSwitch, DateTimeOffset.UtcNow); + } + + _logger.LogDebug( + "Subscriber {SubscriptionId} registered for site-wide alarm events", subscriptionId); + + return subscriptionId; + } + /// /// Unsubscribe from instance events. Shuts down the per-subscriber /// stream graph via its KillSwitch. diff --git a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryRenderTests.cs b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryRenderTests.cs index 3a93a100..51cfefc6 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryRenderTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Monitoring/AlarmSummaryRenderTests.cs @@ -9,6 +9,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites; using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.Communication; using ZB.MOM.WW.ScadaBridge.Security; using AlarmSummaryPage = ZB.MOM.WW.ScadaBridge.CentralUI.Components.Pages.Monitoring.AlarmSummary; @@ -26,6 +27,7 @@ public class AlarmSummaryRenderTests : BunitContext { private readonly IAlarmSummaryService _summary = Substitute.For(); private readonly ISiteRepository _siteRepo = Substitute.For(); + private readonly FakeSiteAlarmLiveCache _liveCache = new(); public AlarmSummaryRenderTests() { @@ -39,14 +41,33 @@ public class AlarmSummaryRenderTests : BunitContext _summary.GetSiteAlarmsAsync(Arg.Any(), Arg.Any()) .Returns(Task.FromResult(new AlarmSummaryResult(rows, Array.Empty()))); + // BuildFromLiveAlarms (Task 5): flatten the pushed live snapshot the same way + // the real service does, so the live-path render tests exercise real behavior. + _summary.BuildFromLiveAlarms(Arg.Any>()) + .Returns(ci => + { + var alarms = ci.Arg>(); + var built = alarms + .Select(a => new AlarmSummaryRow(a.InstanceUniqueName, a)) + .OrderBy(r => r.InstanceUniqueName, StringComparer.OrdinalIgnoreCase) + .ThenBy(r => r.Alarm.AlarmName, StringComparer.OrdinalIgnoreCase) + .ToList(); + return new AlarmSummaryResult(built, Array.Empty()); + }); _summary.ComputeRollup(Arg.Any>()) .Returns(new AlarmRollup(2, 900, 0, new Dictionary())); Services.AddSingleton(_summary); _siteRepo.GetAllSitesAsync(Arg.Any()) - .Returns(Task.FromResult>(new List { new("Site 1", "site1") { Id = 1 } })); + .Returns(Task.FromResult>(new List + { + new("Site 1", "site1") { Id = 1 }, + new("Site 2", "site2") { Id = 2 }, + })); Services.AddSingleton(_siteRepo); + Services.AddSingleton(_liveCache); + var claims = new[] { new Claim(JwtTokenService.UsernameClaimType, "tester"), @@ -109,4 +130,133 @@ public class AlarmSummaryRenderTests : BunitContext private static string FirstRowInstance(IRenderedComponent cut) => cut.FindAll("tr[data-test='alarm-summary-row']")[0] .QuerySelector("td")!.TextContent.Trim(); + + // ── plan #10 Task 5: live-cache-first with poll fallback ─────────────────── + + [Fact] + public void SelectingSite_SubscribesToTheLiveCache() + { + RenderWithSiteSelected(); + + Assert.Equal(1, _liveCache.SubscribeCount); + Assert.Equal(1, _liveCache.LastSubscribedSite); + } + + [Fact] + public void PollFallbackRenders_WhenCacheNotLiveYet() + { + // Cache starts not-live, so the site select falls back to the poll's + // GetSiteAlarmsAsync snapshot (Zeta + Alpha). + Assert.False(_liveCache.IsLive(1)); + + var cut = RenderWithSiteSelected(); + + Assert.Equal(2, cut.FindAll("tr[data-test='alarm-summary-row']").Count); + Assert.Equal("Zeta", FirstRowInstance(cut)); // severity-desc default + } + + [Fact] + public void OnChangedDelta_RebuildsRenderedRowsFromLiveSnapshot() + { + var cut = RenderWithSiteSelected(); + + // Push a fresh live snapshot: a single new alarm on a new instance. + _liveCache.PushAlarms(new List + { + new("Gamma", "G-alarm", AlarmState.Active, 300, DateTimeOffset.UtcNow), + }); + + cut.WaitForAssertion(() => + { + var rows = cut.FindAll("tr[data-test='alarm-summary-row']"); + Assert.Single(rows); + Assert.Equal("Gamma", FirstRowInstance(cut)); + }); + } + + [Fact] + public void LeavingSite_UnsubscribesFromTheLiveCache() + { + var cut = RenderWithSiteSelected(); + Assert.Equal(0, _liveCache.DisposeCount); + + // Clearing the site picker tears the subscription down. + cut.Find("[data-test='alarm-summary-site']").Change(""); + + Assert.Equal(1, _liveCache.DisposeCount); + } + + [Fact] + public void ChangingSite_ResubscribesToTheNewSite() + { + var cut = RenderWithSiteSelected(); // site 1 + + cut.Find("[data-test='alarm-summary-site']").Change("2"); + + Assert.Equal(2, _liveCache.SubscribeCount); // one per site + Assert.Equal(1, _liveCache.DisposeCount); // old subscription disposed + Assert.Equal(2, _liveCache.LastSubscribedSite); + } + + [Fact] + public void DisposingComponent_UnsubscribesFromTheLiveCache() + { + var cut = RenderWithSiteSelected(); + + cut.Instance.Dispose(); + + Assert.True(_liveCache.DisposeCount >= 1); + } + + /// + /// Controllable in-memory fake of : records + /// subscribe/dispose counts and lets a test push a live snapshot (which flips + /// true and fires the stored onChanged), so the page's + /// live-delta path can be driven deterministically. + /// + private sealed class FakeSiteAlarmLiveCache : ISiteAlarmLiveCache + { + private Action? _onChanged; + private IReadOnlyList _current = Array.Empty(); + private bool _live; + + public int SubscribeCount { get; private set; } + public int DisposeCount { get; private set; } + public int? LastSubscribedSite { get; private set; } + + public IDisposable Subscribe(int siteId, Action onChanged) + { + SubscribeCount++; + LastSubscribedSite = siteId; + _onChanged = onChanged; + return new Subscription(this); + } + + public IReadOnlyList GetCurrentAlarms(int siteId) => _current; + + public bool IsLive(int siteId) => _live; + + public void PushAlarms(IReadOnlyList alarms) + { + _current = alarms; + _live = true; + _onChanged?.Invoke(); + } + + private sealed class Subscription : IDisposable + { + private readonly FakeSiteAlarmLiveCache _owner; + private bool _disposed; + + public Subscription(FakeSiteAlarmLiveCache owner) => _owner = owner; + + public void Dispose() + { + if (_disposed) return; + _disposed = true; + _owner.DisposeCount++; + _owner._onChanged = null; + } + } + } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/AlarmSummaryServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/AlarmSummaryServiceTests.cs index 6afe3a32..56dd6f6d 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/AlarmSummaryServiceTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.CentralUI.Tests/Services/AlarmSummaryServiceTests.cs @@ -167,4 +167,40 @@ public class AlarmSummaryServiceTests Assert.Equal(0, rollup.WorstSeverity); Assert.Equal(0, rollup.UnackedCount); } + + // ── BuildFromLiveAlarms (plan #10, Task 5) ──────────────────────────────── + + [Fact] + public void BuildFromLiveAlarms_FlattensAndSorts_SameAsPollPath_WithEmptyNotReporting() + { + // Intentionally out-of-order input to prove the deterministic + // instance-then-alarm-name sort matches GetSiteAlarmsAsync. + var alarms = new List + { + NativeAlarm("zeta", "Zulu", AlarmState.Active, 900, active: true, acked: false), + NativeAlarm("alpha", "Bravo", AlarmState.Active, 500, active: true, acked: true), + NativeAlarm("alpha", "Alpha", AlarmState.Active, 700, active: true, acked: false), + }; + + var result = CreateSut().BuildFromLiveAlarms(alarms); + + // One row per alarm, instance taken from the alarm itself. + Assert.Equal(3, result.Alarms.Count); + Assert.Collection(result.Alarms, + r => Assert.Equal(("alpha", "Alpha"), (r.InstanceUniqueName, r.Alarm.AlarmName)), + r => Assert.Equal(("alpha", "Bravo"), (r.InstanceUniqueName, r.Alarm.AlarmName)), + r => Assert.Equal(("zeta", "Zulu"), (r.InstanceUniqueName, r.Alarm.AlarmName))); + + // The live path can't compute "not reporting" — always empty. + Assert.Empty(result.NotReportingInstances); + } + + [Fact] + public void BuildFromLiveAlarms_EmptySnapshot_YieldsNoRows() + { + var result = CreateSut().BuildFromLiveAlarms(Array.Empty()); + + Assert.Empty(result.Alarms); + Assert.Empty(result.NotReportingInstances); + } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CommunicationOptionsValidatorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CommunicationOptionsValidatorTests.cs index 2ce5f0ff..c29d677c 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CommunicationOptionsValidatorTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/CommunicationOptionsValidatorTests.cs @@ -36,4 +36,54 @@ public class CommunicationOptionsValidatorTests Assert.True(result.Failed); Assert.Contains("GrpcMaxConcurrentStreams", result.FailureMessage); } + + // ── Aggregated live alarm cache options (plan #10, Task 6) ─────────────────── + + [Fact] + public void ZeroLiveAlarmCacheLinger_IsValid() + { + // Zero linger = stop the aggregator immediately when the last viewer leaves. + var result = Validate(new CommunicationOptions { LiveAlarmCacheLinger = TimeSpan.Zero }); + Assert.True(result.Succeeded, result.FailureMessage); + } + + [Fact] + public void NegativeLiveAlarmCacheLinger_IsRejected() + { + var result = Validate(new CommunicationOptions { LiveAlarmCacheLinger = TimeSpan.FromSeconds(-1) }); + Assert.True(result.Failed); + Assert.Contains("LiveAlarmCacheLinger", result.FailureMessage); + } + + [Fact] + public void ZeroLiveAlarmCacheReconcileInterval_IsRejected() + { + var result = Validate(new CommunicationOptions { LiveAlarmCacheReconcileInterval = TimeSpan.Zero }); + Assert.True(result.Failed); + Assert.Contains("LiveAlarmCacheReconcileInterval", result.FailureMessage); + } + + [Fact] + public void ZeroLiveAlarmCacheSeedConcurrency_IsRejected() + { + var result = Validate(new CommunicationOptions { LiveAlarmCacheSeedConcurrency = 0 }); + Assert.True(result.Failed); + Assert.Contains("LiveAlarmCacheSeedConcurrency", result.FailureMessage); + } + + [Fact] + public void ExcessiveLiveAlarmCacheSeedConcurrency_IsRejected() + { + var result = Validate(new CommunicationOptions { LiveAlarmCacheSeedConcurrency = 65 }); + Assert.True(result.Failed); + Assert.Contains("LiveAlarmCacheSeedConcurrency", result.FailureMessage); + } + + [Fact] + public void ZeroLiveAlarmCacheMaxSubscribersPerSite_IsRejected() + { + var result = Validate(new CommunicationOptions { LiveAlarmCacheMaxSubscribersPerSite = 0 }); + Assert.True(result.Failed); + Assert.Contains("LiveAlarmCacheMaxSubscribersPerSite", result.FailureMessage); + } } diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteAlarmAggregatorActorTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteAlarmAggregatorActorTests.cs new file mode 100644 index 00000000..32846846 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteAlarmAggregatorActorTests.cs @@ -0,0 +1,414 @@ +using System.Collections.Concurrent; +using Akka.Actor; +using Akka.TestKit.Xunit2; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.Communication.Actors; +using ZB.MOM.WW.ScadaBridge.Communication.Grpc; + +namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Grpc; + +/// +/// Tests for (plan #10, Task 4) — the transient, +/// in-memory, per-site live alarm aggregator. Covers the tricky bits: seed-then-stream +/// ordering (a delta arriving before the seed completes is neither lost nor double-applied), +/// dedup by AlarmKey, placeholder seeded-but-not-on-stream coherence, NodeA↔NodeB reconnect +/// re-seed, and periodic reconcile authoritative-replace (drift correction). +/// +public class SiteAlarmAggregatorActorTests : TestKit +{ + private const string SiteId = "site-alpha"; + private const string GrpcNodeA = "http://localhost:5100"; + private const string GrpcNodeB = "http://localhost:5200"; + private const string Instance = "Site1.Pump01"; + + public SiteAlarmAggregatorActorTests() : base(@"akka.loglevel = WARNING") + { + SiteAlarmAggregatorActor.ReconnectDelay = TimeSpan.FromMilliseconds(50); + SiteAlarmAggregatorActor.StabilityWindow = TimeSpan.FromSeconds(30); + } + + // ── Test doubles ──────────────────────────────────────────────────────────── + + /// + /// Controllable seed fan-out: each invocation returns a fresh Task the test completes + /// on demand (so seed/reconcile timing relative to buffered deltas is deterministic). + /// + private sealed class SeedStub + { + private readonly ConcurrentQueue>> _pending = new(); + private int _callCount; + + public int CallCount => Volatile.Read(ref _callCount); + + public Task> Seed(CancellationToken ct) + { + Interlocked.Increment(ref _callCount); + var tcs = new TaskCompletionSource>( + TaskCreationOptions.RunContinuationsAsynchronously); + ct.Register(() => tcs.TrySetCanceled()); + _pending.Enqueue(tcs); + return tcs.Task; + } + + /// Completes the oldest not-yet-completed seed call with the given rows. + public void CompleteNext(params AlarmStateChanged[] rows) + { + SpinWait.SpinUntil(() => _pending.TryPeek(out _), TimeSpan.FromSeconds(3)); + if (_pending.TryDequeue(out var tcs)) + tcs.TrySetResult(rows); + else + throw new InvalidOperationException("No pending seed to complete."); + } + + /// Faults the oldest not-yet-completed seed call (simulates a whole-fan-out failure). + public void FaultNext() + { + SpinWait.SpinUntil(() => _pending.TryPeek(out _), TimeSpan.FromSeconds(3)); + if (_pending.TryDequeue(out var tcs)) + tcs.TrySetException(new InvalidOperationException("seed boom")); + else + throw new InvalidOperationException("No pending seed to fault."); + } + } + + private sealed class PublishSink + { + private readonly object _lock = new(); + public List> Snapshots { get; } = new(); + + public void Publish(IReadOnlyList snapshot) + { + lock (_lock) { Snapshots.Add(snapshot); } + } + + public IReadOnlyList? Latest + { + get { lock (_lock) { return Snapshots.Count == 0 ? null : Snapshots[^1]; } } + } + + public int Count { get { lock (_lock) { return Snapshots.Count; } } } + } + + private sealed record SiteSub( + string CorrelationId, Action OnAlarm, Action OnError, CancellationToken Ct); + + private sealed class MockSiteAlarmStreamClient : SiteStreamGrpcClient + { + private readonly object _lock = new(); + private readonly List _subs = new(); + private readonly List _unsubscribed = new(); + + public List Subs { get { lock (_lock) { return _subs.ToList(); } } } + public List Unsubscribed { get { lock (_lock) { return _unsubscribed.ToList(); } } } + + public MockSiteAlarmStreamClient() : base() { } + + public override Task SubscribeSiteAsync( + string correlationId, Action onAlarmEvent, Action onError, CancellationToken ct) + { + lock (_lock) { _subs.Add(new SiteSub(correlationId, onAlarmEvent, onError, ct)); } + var tcs = new TaskCompletionSource(); + ct.Register(() => tcs.TrySetResult()); + return tcs.Task; // never completes until cancelled (simulates a live stream) + } + + public override void Unsubscribe(string correlationId) + { + lock (_lock) { _unsubscribed.Add(correlationId); } + } + } + + private sealed class MockSiteAlarmStreamClientFactory : SiteStreamGrpcClientFactory + { + private readonly ConcurrentDictionary _byEndpoint = new(); + + public MockSiteAlarmStreamClientFactory() + : base(Microsoft.Extensions.Logging.Abstractions.NullLoggerFactory.Instance) { } + + public MockSiteAlarmStreamClient ClientFor(string endpoint) => + _byEndpoint.GetOrAdd(endpoint, _ => new MockSiteAlarmStreamClient()); + + public override SiteStreamGrpcClient GetOrCreate(string siteIdentifier, string grpcEndpoint) + => ClientFor(grpcEndpoint); + + public override SiteStreamGrpcClient? TryGet(string siteIdentifier, string grpcEndpoint) + => _byEndpoint.TryGetValue(grpcEndpoint, out var c) ? c : null; + } + + private (IActorRef Actor, SeedStub Seed, PublishSink Sink, MockSiteAlarmStreamClientFactory Factory) CreateActor( + TimeSpan? reconcileInterval = null) + { + var seed = new SeedStub(); + var sink = new PublishSink(); + var factory = new MockSiteAlarmStreamClientFactory(); + + var props = Props.Create(() => new SiteAlarmAggregatorActor( + SiteId, "corr-1", seed.Seed, sink.Publish, factory, GrpcNodeA, GrpcNodeB, + reconcileInterval ?? TimeSpan.FromMinutes(10))); + + var actor = Sys.ActorOf(props); + return (actor, seed, sink, factory); + } + + private static AlarmStateChanged Alarm( + string alarmName, string sourceRef, int priority, DateTimeOffset ts, + bool placeholder = false, string instance = Instance) => + new(instance, alarmName, AlarmState.Active, priority, ts) + { + SourceReference = sourceRef, + IsConfiguredPlaceholder = placeholder, + Kind = placeholder ? AlarmKind.NativeOpcUa : AlarmKind.Computed + }; + + private static AlarmStateChanged? Find(IReadOnlyList? rows, string alarmName, string sourceRef) => + rows?.FirstOrDefault(r => r.AlarmName == alarmName && r.SourceReference == sourceRef); + + // ── Tests ─────────────────────────────────────────────────────────────────── + + [Fact] + public void Delta_Arriving_Before_Seed_Completes_Is_Buffered_And_Applied_Exactly_Once() + { + var (actor, seed, sink, _) = CreateActor(); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + + // Live delta arrives DURING the seed window (new key, not in the seed). + var t1 = DateTimeOffset.UtcNow; + actor.Tell(Alarm("Overheat", "", 900, t1)); + + // Still buffering — nothing published yet. + Assert.Equal(0, sink.Count); + + // Seed completes with a DIFFERENT alarm. + seed.CompleteNext(Alarm("PumpFault", "", 500, t1.AddSeconds(-1))); + + AwaitCondition(() => sink.Latest is { } l && l.Count == 2, TimeSpan.FromSeconds(3)); + var latest = sink.Latest!; + Assert.NotNull(Find(latest, "PumpFault", "")); // seeded + var overheat = Find(latest, "Overheat", ""); // buffered gap-window delta survived + Assert.NotNull(overheat); + Assert.Single(latest, r => r.AlarmName == "Overheat"); // exactly once + } + + [Fact] + public void Buffered_Delta_Older_Than_Seed_Entry_Is_Dropped() + { + var (actor, seed, sink, _) = CreateActor(); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + + var tSeed = DateTimeOffset.UtcNow; + // Buffered delta for the SAME key but OLDER than the seed entry. + actor.Tell(Alarm("PumpFault", "", 100, tSeed.AddSeconds(-1))); + seed.CompleteNext(Alarm("PumpFault", "", 500, tSeed)); + + AwaitCondition(() => sink.Count >= 1, TimeSpan.FromSeconds(3)); + // Give a beat to ensure no late (dropped) delta sneaks in. + Thread.Sleep(150); + var latest = sink.Latest!; + var pumpFault = Find(latest, "PumpFault", ""); + Assert.NotNull(pumpFault); + Assert.Equal(500, pumpFault!.Priority); // seed value kept; older buffered delta dropped + } + + [Fact] + public void Buffered_Delta_Newer_Than_Seed_Entry_Is_Applied() + { + var (actor, seed, sink, _) = CreateActor(); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + + var tSeed = DateTimeOffset.UtcNow; + actor.Tell(Alarm("PumpFault", "", 900, tSeed.AddSeconds(1))); // newer than seed + seed.CompleteNext(Alarm("PumpFault", "", 500, tSeed)); + + AwaitCondition(() => sink.Latest is { } l && Find(l, "PumpFault", "")?.Priority == 900, + TimeSpan.FromSeconds(3)); + } + + [Fact] + public void Dedup_By_AlarmKey_Distinguishes_SourceReference() + { + var (actor, seed, sink, _) = CreateActor(); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + seed.CompleteNext(); // empty seed + AwaitCondition(() => sink.Count >= 1, TimeSpan.FromSeconds(3)); + + var t = DateTimeOffset.UtcNow; + // Same instance + alarm name, different source reference → two distinct rows. + actor.Tell(Alarm("LevelAlarm", "Tank01.Level.HiHi", 700, t)); + actor.Tell(Alarm("LevelAlarm", "Tank01.Level.LoLo", 300, t)); + + AwaitCondition(() => sink.Latest is { } l && l.Count == 2, TimeSpan.FromSeconds(3)); + var latest = sink.Latest!; + Assert.NotNull(Find(latest, "LevelAlarm", "Tank01.Level.HiHi")); + Assert.NotNull(Find(latest, "LevelAlarm", "Tank01.Level.LoLo")); + } + + [Fact] + public void Same_Key_Live_Update_Replaces_In_Place() + { + var (actor, seed, sink, _) = CreateActor(); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + seed.CompleteNext(); + AwaitCondition(() => sink.Count >= 1, TimeSpan.FromSeconds(3)); + + var t = DateTimeOffset.UtcNow; + actor.Tell(Alarm("PumpFault", "", 500, t)); + AwaitCondition(() => Find(sink.Latest, "PumpFault", "")?.Priority == 500, TimeSpan.FromSeconds(3)); + + // Newer update for the same key replaces (not duplicates). + actor.Tell(Alarm("PumpFault", "", 800, t.AddSeconds(1))); + AwaitCondition(() => Find(sink.Latest, "PumpFault", "")?.Priority == 800, TimeSpan.FromSeconds(3)); + Assert.Single(sink.Latest!); // still exactly one row for the key + } + + [Fact] + public void Seeded_Native_Alarm_And_Its_Live_Delta_Collapse_Onto_One_Key() + { + // Identity parity across the two paths (snapshot seed vs live stream): a native + // alarm seeded with a non-empty SourceReference and a later live delta for the + // SAME (instance, alarmName, sourceRef) must collapse to ONE in-place-updated row, + // not a ghost duplicate. Both paths key on SourceReference, so they must agree. + var (actor, seed, sink, _) = CreateActor(); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + + var t = DateTimeOffset.UtcNow; + seed.CompleteNext(Alarm("Motor1.MotorAlarms", "Motor1.MotorAlarms.HiHi", 300, t)); + AwaitCondition(() => Find(sink.Latest, "Motor1.MotorAlarms", "Motor1.MotorAlarms.HiHi") is not null, + TimeSpan.FromSeconds(3)); + + // Live delta for the exact same identity, newer → updates in place. + actor.Tell(Alarm("Motor1.MotorAlarms", "Motor1.MotorAlarms.HiHi", 800, t.AddSeconds(1))); + + AwaitCondition( + () => Find(sink.Latest, "Motor1.MotorAlarms", "Motor1.MotorAlarms.HiHi")?.Priority == 800, + TimeSpan.FromSeconds(3)); + Assert.Single(sink.Latest!); // exactly one row — no seed/live ghost duplicate + } + + [Fact] + public void Placeholder_Seeded_Coexists_With_Live_Real_Alarm_And_Is_Not_Wiped() + { + var (actor, seed, sink, _) = CreateActor(); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + + var t = DateTimeOffset.UtcNow; + // Seed carries a configured-placeholder row (binding with no active conditions). + var placeholder = Alarm("Motor1.MotorAlarms", "", 0, t, placeholder: true); + seed.CompleteNext(placeholder); + AwaitCondition(() => sink.Count >= 1, TimeSpan.FromSeconds(3)); + + // A live REAL condition for the same instance arrives (distinct AlarmKey). + actor.Tell(Alarm("Motor1.MotorAlarms", "Motor1.MotorAlarms.HiHi", 800, t.AddSeconds(1))); + + AwaitCondition(() => sink.Latest is { } l && l.Count == 2, TimeSpan.FromSeconds(3)); + var latest = sink.Latest!; + // Placeholder still present (not wiped by the real alarm under a different key). + var ph = Find(latest, "Motor1.MotorAlarms", ""); + Assert.NotNull(ph); + Assert.True(ph!.IsConfiguredPlaceholder); + Assert.NotNull(Find(latest, "Motor1.MotorAlarms", "Motor1.MotorAlarms.HiHi")); + } + + [Fact] + public void GrpcError_Flips_Node_Reconnects_And_ReSeeds() + { + var (_, seed, sink, factory) = CreateActor(); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3)); + seed.CompleteNext(); // initial seed done + // Wait for the seed to actually be APPLIED (published) so the re-seed on error is + // not skipped as "fan-out already in flight". + AwaitCondition(() => sink.Count >= 1, TimeSpan.FromSeconds(3)); + + // Fail the NodeA stream. + factory.ClientFor(GrpcNodeA).Subs[0].OnError(new Exception("NodeA down")); + + // Reconnect must reach NodeB, AND a re-seed fan-out must have been triggered. + AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 1, TimeSpan.FromSeconds(5)); + AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(5)); + // The failed NodeA stream was unsubscribed (relay released, not left zombie). + Assert.Contains("corr-1", factory.ClientFor(GrpcNodeA).Unsubscribed); + } + + [Fact] + public void Periodic_Reconcile_Authoritatively_Replaces_Stale_Rows() + { + var (_, seed, sink, _) = CreateActor(reconcileInterval: TimeSpan.FromMilliseconds(250)); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + + var t = DateTimeOffset.UtcNow; + seed.CompleteNext(Alarm("PumpFault", "", 500, t)); // seed has a row + AwaitCondition(() => Find(sink.Latest, "PumpFault", "") is not null, TimeSpan.FromSeconds(3)); + + // Reconcile tick fires → seed #2 (e.g. instance was disabled) returns EMPTY. + AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3)); + seed.CompleteNext(); // empty + + // The stale PumpFault row must disappear (authoritative replace, not merge). + AwaitCondition(() => sink.Latest is { } l && l.Count == 0, TimeSpan.FromSeconds(3)); + } + + [Fact] + public void Failed_Initial_Seed_Does_Not_Publish_But_A_Later_Reconcile_Recovers() + { + var (_, seed, sink, _) = CreateActor(reconcileInterval: TimeSpan.FromMilliseconds(250)); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + + // The initial seed fan-out throws as a whole. + seed.FaultNext(); + + // A failed INITIAL seed must not publish (page keeps its poll fallback — not live yet). + Thread.Sleep(200); + Assert.Equal(0, sink.Count); + + // The periodic reconcile is the backstop: it re-runs the fan-out (flag was cleared + // on failure), completes, and the actor publishes → now live. + AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3)); + seed.CompleteNext(Alarm("PumpFault", "", 500, DateTimeOffset.UtcNow)); + AwaitCondition(() => Find(sink.Latest, "PumpFault", "") is not null, TimeSpan.FromSeconds(3)); + } + + [Fact] + public void Stream_GivenUp_After_MaxRetries_Reopens_On_Reconcile_Tick() + { + // After the retry budget is exhausted the live stream is left down, but the + // aggregator is NOT stopped — the next reconcile tick self-heals it (resets the + // budget + reopens) so a sustained site outage never permanently kills the feed. + var (_, seed, _, factory) = CreateActor(reconcileInterval: TimeSpan.FromMilliseconds(300)); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3)); + + // Drive initial + 3 retries (each flips node) to exhaust the budget. + factory.ClientFor(GrpcNodeA).Subs[0].OnError(new Exception("1")); + AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 1, TimeSpan.FromSeconds(5)); + factory.ClientFor(GrpcNodeB).Subs[0].OnError(new Exception("2")); + AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 2, TimeSpan.FromSeconds(5)); + factory.ClientFor(GrpcNodeA).Subs[1].OnError(new Exception("3")); + AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 2, TimeSpan.FromSeconds(5)); + + int TotalSubs() => factory.ClientFor(GrpcNodeA).Subs.Count + factory.ClientFor(GrpcNodeB).Subs.Count; + var before = TotalSubs(); // 4 + + // Fourth error exceeds MaxRetries → stream given up (no immediate reopen). + factory.ClientFor(GrpcNodeB).Subs[1].OnError(new Exception("4")); + + // The reconcile tick reopens the stream (self-heal). + AwaitCondition(() => TotalSubs() > before, TimeSpan.FromSeconds(3)); + } + + [Fact] + public void Stop_Message_TearsDown_Grpc_And_Stops_Actor() + { + var (actor, seed, _, factory) = CreateActor(); + AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3)); + AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3)); + seed.CompleteNext(); + + Watch(actor); + actor.Tell(new StopSiteAlarmAggregator()); + + ExpectTerminated(actor, TimeSpan.FromSeconds(3)); + AwaitCondition(() => factory.ClientFor(GrpcNodeA).Unsubscribed.Contains("corr-1"), + TimeSpan.FromSeconds(3)); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcClientTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcClientTests.cs index 82d4addb..34b8563b 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcClientTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcClientTests.cs @@ -208,6 +208,78 @@ public class SiteStreamGrpcClientTests Assert.True(cts2.IsCancellationRequested); } + // --- Site-wide (SubscribeSite) alarm-only stream (plan #10 T3) --- + + [Fact] + public void ConvertToAlarmEvent_AlarmChanged_ReturnsMappedAlarm() + { + // An alarm event on the site-wide stream is delivered to onAlarmEvent with full enrichment. + var ts = DateTimeOffset.UtcNow; + var evt = new SiteStreamEvent + { + CorrelationId = "site-corr", + AlarmChanged = new AlarmStateUpdate + { + InstanceUniqueName = "Site1.Motor01", + AlarmName = "T01.Hi", + State = AlarmStateEnum.AlarmStateActive, + Priority = 700, + Timestamp = Timestamp.FromDateTimeOffset(ts), + Kind = "NativeOpcUa", + Active = true, + SourceReference = "T01.Hi" + } + }; + + var alarm = SiteStreamGrpcClient.ConvertToAlarmEvent(evt); + + Assert.NotNull(alarm); + Assert.Equal("Site1.Motor01", alarm!.InstanceUniqueName); + Assert.Equal("T01.Hi", alarm.AlarmName); + Assert.Equal(AlarmState.Active, alarm.State); + Assert.Equal(AlarmKind.NativeOpcUa, alarm.Kind); + Assert.Equal("T01.Hi", alarm.SourceReference); + } + + [Fact] + public void ConvertToAlarmEvent_AttributeChanged_ReturnsNull() + { + // Attribute events must never appear on the alarm-only site-wide stream; if one + // somehow arrives it is defensively filtered out rather than delivered or thrown. + var evt = new SiteStreamEvent + { + CorrelationId = "site-corr", + AttributeChanged = new AttributeValueUpdate + { + InstanceUniqueName = "Site1.Pump01", + AttributePath = "Modules.IO", + AttributeName = "Temperature", + Value = "42.5", + Quality = Quality.Good, + Timestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow) + } + }; + + Assert.Null(SiteStreamGrpcClient.ConvertToAlarmEvent(evt)); + } + + [Fact] + public void ConvertToAlarmEvent_UnknownEvent_ReturnsNull() + { + var evt = new SiteStreamEvent { CorrelationId = "site-corr" }; + Assert.Null(SiteStreamGrpcClient.ConvertToAlarmEvent(evt)); + } + + [Fact] + public async Task SubscribeSiteAsync_OnTestOnlyClient_Throws() + { + // Guards against subscribing on a channel-less test double, mirroring SubscribeAsync. + var client = SiteStreamGrpcClient.CreateForTesting(); + + await Assert.ThrowsAsync(() => + client.SubscribeSiteAsync("corr", _ => { }, _ => { }, CancellationToken.None)); + } + // --- Communication-003 regression tests --- [Fact] diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcServerTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcServerTests.cs index 1e3b01c7..a44e570e 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcServerTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/Grpc/SiteStreamGrpcServerTests.cs @@ -21,6 +21,8 @@ public class SiteStreamGrpcServerTests : TestKit _subscriber = Substitute.For(); _subscriber.Subscribe(Arg.Any(), Arg.Any()) .Returns("sub-1"); + _subscriber.SubscribeSiteAlarms(Arg.Any()) + .Returns("site-sub-1"); _logger = NullLogger.Instance; } @@ -266,6 +268,102 @@ public class SiteStreamGrpcServerTests : TestKit await streamTask; } + // --- SubscribeSite (site-wide, alarm-only aggregated stream, plan #10 T2) --- + + private static SiteStreamRequest MakeSiteRequest(string correlationId = "site-corr-1") + => new() { CorrelationId = correlationId }; + + [Fact] + public async Task SubscribeSite_SubscribesSiteAlarmsAndRemovesOnCancel() + { + var server = CreateServer(); + server.SetReady(Sys); + + var cts = new CancellationTokenSource(); + var context = CreateMockContext(cts.Token); + var writer = Substitute.For>(); + + var streamTask = Task.Run(() => server.SubscribeSite( + MakeSiteRequest("site-corr-sub"), writer, context)); + + await WaitForConditionAsync(() => server.ActiveStreamCount == 1); + + // Site-wide handler must call SubscribeSiteAlarms (no instance filter), + // never the per-instance Subscribe. + _subscriber.Received(1).SubscribeSiteAlarms(Arg.Any()); + _subscriber.DidNotReceive().Subscribe(Arg.Any(), Arg.Any()); + + cts.Cancel(); + await streamTask; + + _subscriber.Received(1).RemoveSubscriber(Arg.Any()); + Assert.Equal(0, server.ActiveStreamCount); + } + + [Fact] + public async Task SubscribeSite_RejectsUnsafeCorrelationId() + { + var server = CreateServer(); + server.SetReady(Sys); + + var writer = Substitute.For>(); + var context = CreateMockContext(); + + var ex = await Assert.ThrowsAsync( + () => server.SubscribeSite(MakeSiteRequest("bad/id"), writer, context)); + + Assert.Equal(StatusCode.InvalidArgument, ex.StatusCode); + Assert.Equal(0, server.ActiveStreamCount); + } + + [Fact] + public async Task SubscribeSite_RelaysAlarmStateChangedAsAlarmStateUpdate() + { + var server = CreateServer(); + server.SetReady(Sys); + + // Capture the relay actor spawned for the site-wide subscription. + IActorRef? capturedActor = null; + _subscriber.SubscribeSiteAlarms(Arg.Any()) + .Returns(ci => + { + capturedActor = ci.Arg(); + return "site-sub-relay"; + }); + + var cts = new CancellationTokenSource(); + var context = CreateMockContext(cts.Token); + var writer = Substitute.For>(); + var writtenEvents = new List(); + writer.WriteAsync(Arg.Any(), Arg.Any()) + .Returns(Task.CompletedTask) + .AndDoes(ci => writtenEvents.Add(ci.Arg())); + + var streamTask = Task.Run(() => server.SubscribeSite( + MakeSiteRequest("site-corr-write"), writer, context)); + + await WaitForConditionAsync(() => capturedActor != null); + + // A real alarm transition for ANY instance must arrive as an AlarmStateUpdate. + capturedActor!.Tell(new Commons.Messages.Streaming.AlarmStateChanged( + "Site1.Pump01", + "HighPressure", + Commons.Types.Enums.AlarmState.Active, + 700, + DateTimeOffset.UtcNow)); + + await WaitForConditionAsync(() => writtenEvents.Count >= 1); + + Assert.Single(writtenEvents); + Assert.Equal("site-corr-write", writtenEvents[0].CorrelationId); + Assert.Equal(SiteStreamEvent.EventOneofCase.AlarmChanged, writtenEvents[0].EventCase); + Assert.Equal("Site1.Pump01", writtenEvents[0].AlarmChanged.InstanceUniqueName); + Assert.Equal("HighPressure", writtenEvents[0].AlarmChanged.AlarmName); + + cts.Cancel(); + await streamTask; + } + [Theory] [InlineData("corr/with/slash")] [InlineData("corr with space")] diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheServiceTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheServiceTests.cs new file mode 100644 index 00000000..3d7e07ad --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheServiceTests.cs @@ -0,0 +1,261 @@ +using Akka.TestKit.Xunit2; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging.Abstractions; +using Microsoft.Extensions.Options; +using NSubstitute; +using ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances; +using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites; +using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.Communication; +using ZB.MOM.WW.ScadaBridge.Communication.Grpc; + +namespace ZB.MOM.WW.ScadaBridge.Communication.Tests; + +/// +/// Tests for (plan #10, Task 4) — the DI singleton +/// that reference-counts Alarm Summary viewers over one shared per-site aggregator. Covers +/// shared start (one aggregator for many viewers), last-out linger stop, and re-subscribe +/// cancelling a pending stop. Uses a real (in-memory) ActorSystem and a mock gRPC factory +/// whose site-wide stream never faults, so no reconnect noise. +/// +public class SiteAlarmLiveCacheServiceTests : TestKit +{ + private const int SiteId = 7; + + // A mock site-wide alarm stream client whose subscription hangs until cancelled. + private sealed class HangingClient : SiteStreamGrpcClient + { + public HangingClient() : base() { } + public override Task SubscribeSiteAsync( + string correlationId, Action onAlarmEvent, + Action onError, CancellationToken ct) + { + var tcs = new TaskCompletionSource(); + ct.Register(() => tcs.TrySetResult()); + return tcs.Task; + } + public override void Unsubscribe(string correlationId) { } + } + + private sealed class CountingFactory : SiteStreamGrpcClientFactory + { + private readonly HangingClient _client = new(); + public int GetOrCreateCount; + + public CountingFactory() : base(NullLoggerFactory.Instance) { } + + public override SiteStreamGrpcClient GetOrCreate(string siteIdentifier, string grpcEndpoint) + { + Interlocked.Increment(ref GetOrCreateCount); + return _client; + } + + public override SiteStreamGrpcClient? TryGet(string siteIdentifier, string grpcEndpoint) => _client; + } + + private SiteAlarmLiveCacheService CreateService(TimeSpan linger, out CountingFactory factory, + int maxSubscribersPerSite = 200) + { + // Site with gRPC addresses, and NO enabled instances → the seed fan-out returns + // empty immediately (so IsLive flips true fast without any snapshot Asks). + var site = new Site("Alpha", "site-alpha") + { + Id = SiteId, + GrpcNodeAAddress = "http://localhost:5100", + GrpcNodeBAddress = "http://localhost:5200" + }; + + var siteRepo = Substitute.For(); + siteRepo.GetSiteByIdAsync(SiteId, Arg.Any()).Returns(site); + + var instanceRepo = Substitute.For(); + instanceRepo.GetInstancesBySiteIdAsync(SiteId, Arg.Any()) + .Returns(new List()); + + var services = new ServiceCollection(); + services.AddScoped(_ => siteRepo); + services.AddScoped(_ => instanceRepo); + var provider = services.BuildServiceProvider(); + + var options = Options.Create(new CommunicationOptions + { + LiveAlarmCacheLinger = linger, + LiveAlarmCacheReconcileInterval = TimeSpan.FromMinutes(10), + LiveAlarmCacheMaxSubscribersPerSite = maxSubscribersPerSite + }); + + var comm = new CommunicationService(Options.Create(new CommunicationOptions()), + NullLogger.Instance); + + factory = new CountingFactory(); + var service = new SiteAlarmLiveCacheService( + provider, comm, factory, options, NullLogger.Instance); + service.SetActorSystem(Sys); + return service; + } + + [Fact] + public void First_Subscriber_Starts_One_Aggregator_Shared_By_Multiple_Viewers() + { + var service = CreateService(TimeSpan.FromMilliseconds(200), out var factory); + + var changes1 = 0; + var changes2 = 0; + using var sub1 = service.Subscribe(SiteId, () => Interlocked.Increment(ref changes1)); + using var sub2 = service.Subscribe(SiteId, () => Interlocked.Increment(ref changes2)); + + // The (empty) seed completes → aggregator publishes → live. + AwaitCondition(() => service.IsLive(SiteId), TimeSpan.FromSeconds(5)); + + // Exactly ONE gRPC stream opened for the site regardless of viewer count. + Assert.Equal(1, factory.GetOrCreateCount); + // Empty seed → empty current snapshot. + Assert.Empty(service.GetCurrentAlarms(SiteId)); + // Both viewers were notified of the initial publish. + AwaitCondition(() => changes1 >= 1 && changes2 >= 1, TimeSpan.FromSeconds(3)); + } + + [Fact] + public void Last_Viewer_Leaving_Stops_Aggregator_After_Linger() + { + var service = CreateService(TimeSpan.FromMilliseconds(200), out _); + + var sub1 = service.Subscribe(SiteId, () => { }); + var sub2 = service.Subscribe(SiteId, () => { }); + AwaitCondition(() => service.IsLive(SiteId), TimeSpan.FromSeconds(5)); + + sub1.Dispose(); + // One viewer remains — still live, not stopped. + Thread.Sleep(300); + Assert.True(service.IsLive(SiteId)); + + sub2.Dispose(); + // Last viewer left → after the linger the aggregator is torn down and the entry cleared. + AwaitCondition(() => !service.IsLive(SiteId), TimeSpan.FromSeconds(3)); + Assert.Empty(service.GetCurrentAlarms(SiteId)); + } + + [Fact] + public void Resubscribe_Within_Linger_Cancels_The_Pending_Stop() + { + var service = CreateService(TimeSpan.FromMilliseconds(400), out var factory); + + var sub1 = service.Subscribe(SiteId, () => { }); + AwaitCondition(() => service.IsLive(SiteId), TimeSpan.FromSeconds(5)); + + sub1.Dispose(); // schedules linger stop + var sub2 = service.Subscribe(SiteId, () => { }); // returns within the linger window → cancels it + + // Wait comfortably past the original linger; the aggregator must still be alive. + Thread.Sleep(700); + Assert.True(service.IsLive(SiteId)); + // No second aggregator/stream was opened — the same one was kept warm. + Assert.Equal(1, factory.GetOrCreateCount); + + sub2.Dispose(); + } + + [Fact] + public void Dispose_Is_Idempotent() + { + var service = CreateService(TimeSpan.FromMilliseconds(200), out _); + var sub = service.Subscribe(SiteId, () => { }); + AwaitCondition(() => service.IsLive(SiteId), TimeSpan.FromSeconds(5)); + + sub.Dispose(); + sub.Dispose(); // must not throw or double-decrement + + AwaitCondition(() => !service.IsLive(SiteId), TimeSpan.FromSeconds(3)); + } + + [Fact] + public void Transient_Start_Failure_Self_Heals_On_Retry_For_A_Stable_Viewer_Cohort() + { + // A transient failure resolving the site at first-subscribe must NOT leave the + // viewer cohort polling forever — a bounded retry re-attempts and self-heals. + var site = new Site("Alpha", "site-alpha") + { + Id = SiteId, + GrpcNodeAAddress = "http://localhost:5100", + GrpcNodeBAddress = "http://localhost:5200" + }; + + var siteRepo = Substitute.For(); + // First resolve returns null (transient blip); every resolve thereafter succeeds. + siteRepo.GetSiteByIdAsync(SiteId, Arg.Any()).Returns((Site?)null, site); + + var instanceRepo = Substitute.For(); + instanceRepo.GetInstancesBySiteIdAsync(SiteId, Arg.Any()) + .Returns(new List()); + + var services = new ServiceCollection(); + services.AddScoped(_ => siteRepo); + services.AddScoped(_ => instanceRepo); + var provider = services.BuildServiceProvider(); + + var options = Options.Create(new CommunicationOptions + { + LiveAlarmCacheLinger = TimeSpan.FromSeconds(30), + // Retry cadence reuses the reconcile interval — keep it short for the test. + LiveAlarmCacheReconcileInterval = TimeSpan.FromMilliseconds(300) + }); + var comm = new CommunicationService(Options.Create(new CommunicationOptions()), + NullLogger.Instance); + var factory = new CountingFactory(); + + var service = new SiteAlarmLiveCacheService( + provider, comm, factory, options, NullLogger.Instance); + service.SetActorSystem(Sys); + + using var sub = service.Subscribe(SiteId, () => { }); + + // First start failed (site not found); a stable single viewer stays subscribed. + // The retry eventually resolves the site and the aggregator goes live — without + // any further Subscribe call. + AwaitCondition(() => service.IsLive(SiteId), TimeSpan.FromSeconds(5)); + } + + [Fact] + public void Viewer_Over_The_Per_Site_Cap_Is_Rejected_Safely_And_Not_Counted() + { + // Cap of 2: the first two viewers register; a third is rejected fail-safe with a + // no-op handle (never throws into the render path, never grows the list). Proof it + // was NOT counted: disposing only the two real viewers tears the aggregator down + // after the linger — if the rejected viewer had been registered, one subscriber + // would remain and the site would stay live forever. + var service = CreateService(TimeSpan.FromMilliseconds(200), out _, maxSubscribersPerSite: 2); + + var sub1 = service.Subscribe(SiteId, () => { }); + var sub2 = service.Subscribe(SiteId, () => { }); + AwaitCondition(() => service.IsLive(SiteId), TimeSpan.FromSeconds(5)); + + // Third viewer is over the cap. + var overflow = service.Subscribe(SiteId, () => { }); + Assert.NotNull(overflow); + // Disposing the rejected handle is a genuine no-op — safe, idempotent, touches nothing. + overflow.Dispose(); + overflow.Dispose(); + + // Still live with the two real viewers present. + Assert.True(service.IsLive(SiteId)); + + sub1.Dispose(); + sub2.Dispose(); + + // Both real viewers gone → after the linger the aggregator stops. This only holds + // if the overflow viewer was never added to the subscriber list. + AwaitCondition(() => !service.IsLive(SiteId), TimeSpan.FromSeconds(3)); + } + + [Fact] + public void Unknown_Site_Registers_Viewer_But_Never_Goes_Live() + { + var service = CreateService(TimeSpan.FromMilliseconds(200), out _); + + using var sub = service.Subscribe(999, () => { }); // no such site in the repo + Thread.Sleep(300); + Assert.False(service.IsLive(999)); + Assert.Empty(service.GetCurrentAlarms(999)); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheTelemetryTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheTelemetryTests.cs new file mode 100644 index 00000000..87549372 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.Communication.Tests/SiteAlarmLiveCacheTelemetryTests.cs @@ -0,0 +1,94 @@ +using System.Diagnostics.Metrics; +using ZB.MOM.WW.ScadaBridge.Commons.Observability; + +namespace ZB.MOM.WW.ScadaBridge.Communication.Tests; + +/// +/// Telemetry guards for the aggregated live alarm cache (plan #10, Task 6): the +/// active-aggregator observable gauge reflects Started/Stopped, and the reconnect +/// counter increments on each recorded reconnect. Mirrors the MeterListener read +/// pattern used by SiteStreamGrpcServerTests.SiteConnectionUpGauge_*; reads are +/// relative to a baseline so the process-wide static instruments are robust to any +/// parallel test interleaving. +/// +public class SiteAlarmLiveCacheTelemetryTests +{ + private static long ReadGauge(string instrumentName) + { + long observed = 0; + using var listener = new MeterListener(); + listener.InstrumentPublished = (instrument, l) => + { + if (instrument.Meter.Name == ScadaBridgeTelemetry.MeterName && + instrument.Name == instrumentName) + { + l.EnableMeasurementEvents(instrument); + } + }; + listener.SetMeasurementEventCallback((_, measurement, _, _) => observed = measurement); + listener.Start(); + listener.RecordObservableInstruments(); + return observed; + } + + private static long ReadCounter(string instrumentName, Action act) + { + long delta = 0; + using var listener = new MeterListener(); + listener.InstrumentPublished = (instrument, l) => + { + if (instrument.Meter.Name == ScadaBridgeTelemetry.MeterName && + instrument.Name == instrumentName) + { + l.EnableMeasurementEvents(instrument); + } + }; + listener.SetMeasurementEventCallback((_, measurement, _, _) => delta += measurement); + listener.Start(); + act(); + return delta; + } + + [Fact] + public void ActiveAggregatorGauge_ReflectsStartAndStop() + { + const string gauge = "scadabridge.site.alarm_cache.aggregators.active"; + var baseline = ReadGauge(gauge); + + ScadaBridgeTelemetry.LiveAlarmAggregatorStarted(); + try + { + Assert.Equal(baseline + 1, ReadGauge(gauge)); + + ScadaBridgeTelemetry.LiveAlarmAggregatorStarted(); + try + { + Assert.Equal(baseline + 2, ReadGauge(gauge)); + } + finally + { + ScadaBridgeTelemetry.LiveAlarmAggregatorStopped(); + } + + Assert.Equal(baseline + 1, ReadGauge(gauge)); + } + finally + { + ScadaBridgeTelemetry.LiveAlarmAggregatorStopped(); + } + + Assert.Equal(baseline, ReadGauge(gauge)); + } + + [Fact] + public void ReconnectCounter_IncrementsOnEachRecordedReconnect() + { + var delta = ReadCounter("scadabridge.site.alarm_cache.reconnects", () => + { + ScadaBridgeTelemetry.RecordLiveAlarmStreamReconnect(); + ScadaBridgeTelemetry.RecordLiveAlarmStreamReconnect(); + }); + + Assert.Equal(2, delta); + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ScadaBridgeTelemetryTests.cs b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ScadaBridgeTelemetryTests.cs index c081386e..6c89f559 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ScadaBridgeTelemetryTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.Host.Tests/ScadaBridgeTelemetryTests.cs @@ -24,6 +24,9 @@ public class ScadaBridgeTelemetryTests ScadaBridgeTelemetry.RecordInboundApiRequest("X"); ScadaBridgeTelemetry.SiteConnectionOpened(); ScadaBridgeTelemetry.SiteConnectionClosed(); + ScadaBridgeTelemetry.LiveAlarmAggregatorStarted(); + ScadaBridgeTelemetry.LiveAlarmAggregatorStopped(); + ScadaBridgeTelemetry.RecordLiveAlarmStreamReconnect(); ScadaBridgeTelemetry.SetQueueDepthProvider(() => 5); }); diff --git a/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Grpc/SiteAlarmStreamEndToEndTests.cs b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Grpc/SiteAlarmStreamEndToEndTests.cs new file mode 100644 index 00000000..9694b975 --- /dev/null +++ b/tests/ZB.MOM.WW.ScadaBridge.IntegrationTests/Grpc/SiteAlarmStreamEndToEndTests.cs @@ -0,0 +1,298 @@ +using System.Collections.Concurrent; +using Akka.Actor; +using Akka.TestKit.Xunit2; +using Google.Protobuf; +using Grpc.Core; +using Microsoft.Extensions.Logging.Abstractions; +using NSubstitute; +using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Alarms; +using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums; +using ZB.MOM.WW.ScadaBridge.Communication.Actors; +using ZB.MOM.WW.ScadaBridge.Communication.Grpc; +using ZB.MOM.WW.ScadaBridge.SiteRuntime; +using ZB.MOM.WW.ScadaBridge.SiteRuntime.Streaming; + +namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Grpc; + +/// +/// End-to-end trace for the aggregated live alarm stream (plan #10 deferred item, Task 7). +/// +/// Every per-task unit test (T1–T6) exercises exactly one layer with its neighbour mocked: +/// T1 the site broadcast filter (subscriber = TestKit probe, domain events), T2/the existing +/// the server+relay+channel with a MOCKED +/// stopping at the proto event, T3 the proto→domain client +/// mapping in isolation, and T4 the aggregator fed hand-built domain events. Nothing wires the +/// whole pipe together. +/// +/// This test assembles the REAL chain end to end — mocking only the gRPC HTTP/2 transport: +/// +/// domain AlarmStateChanged +/// → SiteStreamManager.SubscribeSiteAlarms (real site-wide broadcast + alarm-only filter) +/// → SiteStreamGrpcServer.SubscribeSite (real server handler) +/// → StreamRelayActor (real domain→proto mapping + placeholder drop) +/// → SiteStreamEvent proto → wire round-trip (ToByteArray/ParseFrom, simulates HTTP/2) +/// → SiteStreamGrpcClient.ConvertToAlarmEvent (real proto→domain mapping) +/// → SiteAlarmAggregatorActor cache (real dedup/seed/live) +/// +/// and asserts the invariants the seams can't prove alone: AlarmKey identity +/// (instance, name, sourceRef) + native enrichment survive every boundary; attributes and +/// placeholder rows never reach the live cache; the single site-wide stream carries alarms +/// for MULTIPLE instances (no per-instance filter); and a snapshot-seed row and a live delta +/// for the same native alarm — both mapped through the real pipe — collapse onto ONE cache row. +/// +/// Full gRPC-over-HTTP/2 remains a manual docker-cluster smoke (plan §4 Task 7). +/// +public class SiteAlarmStreamEndToEndTests : TestKit +{ + private const string InstanceA = "SiteA.Pump01"; + private const string InstanceB = "SiteA.Motor07"; + + // ── The full site→proto→client trace ──────────────────────────────────────── + + [Fact] + public async Task SiteWideAlarm_TraversesManagerToRelayToClient_PreservingIdentityAndEnrichment_DroppingAttributesAndPlaceholders() + { + // Site side: a REAL broadcast hub (no mock subscriber) wired straight into the + // REAL SubscribeSite server handler. + var manager = new SiteStreamManager( + new SiteRuntimeOptions { StreamBufferSize = 256 }, + NullLogger.Instance); + manager.Initialize(Sys); + + var server = new SiteStreamGrpcServer(manager, NullLogger.Instance); + server.SetReady(Sys); + + var written = new ConcurrentQueue(); + var writer = Substitute.For>(); + writer.WriteAsync(Arg.Any(), Arg.Any()) + .Returns(Task.CompletedTask) + .AndDoes(ci => written.Enqueue(ci.Arg())); + + using var cts = new CancellationTokenSource(); + var context = CreateMockContext(cts.Token); + + var streamTask = Task.Run(() => server.SubscribeSite( + new SiteStreamRequest { CorrelationId = "e2e-site" }, writer, context)); + + // Wait until the server has actually subscribed to the site-wide hub, otherwise a + // publish can race ahead of the materialized subscription and be missed. + await WaitForConditionAsync(() => manager.SubscriptionCount == 1); + + var raise = new DateTimeOffset(2026, 4, 2, 8, 30, 0, TimeSpan.Zero); + var ts = raise.AddSeconds(5); + + // A fully-enriched NATIVE alarm on instance A (the payload whose fidelity we trace). + var native = new AlarmStateChanged(InstanceA, "Tank01.LevelAlarm", AlarmState.Active, 725, ts) + { + Level = AlarmLevel.HighHigh, + Message = "Tank 01 level critically high", + Kind = AlarmKind.NativeOpcUa, + Condition = new AlarmConditionState( + Active: true, Acknowledged: true, Confirmed: false, + Shelve: AlarmShelveState.OneShotShelved, Suppressed: false, Severity: 725), + SourceReference = "Tank01.Level.HiHi", + AlarmTypeName = "AnalogLimitAlarm.HiHi", + Category = "Process", + OperatorUser = "op.jane", + OperatorComment = "ack — investigating", + OriginalRaiseTime = raise, + CurrentValue = "98.4", + LimitValue = "95.0", + NativeSourceCanonicalName = "Tank01.LevelAlarm", + IsConfiguredPlaceholder = false + }; + + // A COMPUTED alarm on a DIFFERENT instance — must arrive over the SAME site-wide + // stream (proves the per-instance filter is gone). + var computedOtherInstance = new AlarmStateChanged(InstanceB, "OverSpeed", AlarmState.Active, 400, ts); + + // Noise that MUST be dropped somewhere in the pipe: + var attribute = new AttributeValueChanged(InstanceA, "Modules.Flow", "GPM", 12.3, "Good", ts); + var placeholder = new AlarmStateChanged(InstanceA, "Motor1.MotorAlarms", AlarmState.Normal, 0, ts) + { + Kind = AlarmKind.NativeOpcUa, + IsConfiguredPlaceholder = true + }; + + // Interleave so a leak of the dropped rows would change the observed count/order. + manager.PublishAttributeValueChanged(attribute); // filtered at the manager (alarm-only) + manager.PublishAlarmStateChanged(native); // → 1 proto + manager.PublishAlarmStateChanged(placeholder); // dropped by StreamRelayActor + manager.PublishAlarmStateChanged(computedOtherInstance); // → 1 proto + + await WaitForConditionAsync(() => written.Count >= 2); + // Give any leaked (attribute/placeholder) event a chance to show up before asserting. + await Task.Delay(150); + + cts.Cancel(); + await streamTask; + + var protos = written.ToArray(); + Assert.Equal(2, protos.Length); + Assert.All(protos, p => + Assert.Equal(SiteStreamEvent.EventOneofCase.AlarmChanged, p.EventCase)); // no attribute leaked + Assert.DoesNotContain(protos, p => p.AlarmChanged.IsConfiguredPlaceholder); // no placeholder leaked + + // Wire round-trip (serialize→deserialize) then the REAL client mapping back to domain. + var mapped = protos + .Select(p => SiteStreamGrpcClient.ConvertToAlarmEvent(RoundTripOverWire(p))) + .ToList(); + Assert.All(mapped, m => Assert.NotNull(m)); + + // Both instances present — one site-wide stream, no per-instance filter. + var backA = mapped.Single(m => m!.InstanceUniqueName == InstanceA)!; + var backB = mapped.Single(m => m!.InstanceUniqueName == InstanceB)!; + Assert.Equal("OverSpeed", backB.AlarmName); + + // AlarmKey identity survived end to end. + Assert.Equal("Tank01.LevelAlarm", backA.AlarmName); + Assert.Equal("Tank01.Level.HiHi", backA.SourceReference); + + // Native enrichment survived every boundary (manager→relay→proto→wire→client). + Assert.Equal(AlarmKind.NativeOpcUa, backA.Kind); + Assert.Equal(AlarmState.Active, backA.State); + Assert.Equal(725, backA.Priority); + Assert.Equal(AlarmLevel.HighHigh, backA.Level); + Assert.Equal("Tank 01 level critically high", backA.Message); + Assert.True(backA.Condition.Active); + Assert.True(backA.Condition.Acknowledged); + Assert.Equal(AlarmShelveState.OneShotShelved, backA.Condition.Shelve); + Assert.Equal(725, backA.Condition.Severity); + Assert.Equal("AnalogLimitAlarm.HiHi", backA.AlarmTypeName); + Assert.Equal("Process", backA.Category); + Assert.Equal("op.jane", backA.OperatorUser); + Assert.Equal("ack — investigating", backA.OperatorComment); + Assert.Equal(raise, backA.OriginalRaiseTime); + Assert.Equal("98.4", backA.CurrentValue); + Assert.Equal("95.0", backA.LimitValue); + Assert.Equal("Tank01.LevelAlarm", backA.NativeSourceCanonicalName); + Assert.Equal(ts, backA.Timestamp); + Assert.False(backA.IsConfiguredPlaceholder); + } + + // ── Snapshot-seed vs live-delta identity parity, through the real pipe ──────── + + [Fact] + public async Task RealPipeMapped_SeedRow_And_LiveDelta_ForSameNativeAlarm_CollapseOntoOneCacheRow() + { + // Produce TWO proto events for the SAME native alarm identity through the real + // manager→relay→proto pipe: an older one (used as the snapshot-seed row) and a + // newer one (fed as the live delta). Because both are mapped by the SAME real + // ConvertToAlarmEvent, their AlarmKeys are identical by construction — so the real + // aggregator must collapse them onto ONE row (no seed/live ghost duplicate). + var manager = new SiteStreamManager( + new SiteRuntimeOptions { StreamBufferSize = 256 }, + NullLogger.Instance); + manager.Initialize(Sys); + var server = new SiteStreamGrpcServer(manager, NullLogger.Instance); + server.SetReady(Sys); + + var written = new ConcurrentQueue(); + var writer = Substitute.For>(); + writer.WriteAsync(Arg.Any(), Arg.Any()) + .Returns(Task.CompletedTask) + .AndDoes(ci => written.Enqueue(ci.Arg())); + using var cts = new CancellationTokenSource(); + var streamTask = Task.Run(() => server.SubscribeSite( + new SiteStreamRequest { CorrelationId = "e2e-parity" }, writer, CreateMockContext(cts.Token))); + await WaitForConditionAsync(() => manager.SubscriptionCount == 1); + + AlarmStateChanged Native(int priority, DateTimeOffset t) => + new(InstanceA, "Tank01.LevelAlarm", AlarmState.Active, priority, t) + { + Kind = AlarmKind.NativeOpcUa, + SourceReference = "Tank01.Level.HiHi", + NativeSourceCanonicalName = "Tank01.LevelAlarm" + }; + + var t0 = new DateTimeOffset(2026, 4, 2, 9, 0, 0, TimeSpan.Zero); + manager.PublishAlarmStateChanged(Native(300, t0)); // → seed row (older) + manager.PublishAlarmStateChanged(Native(800, t0.AddSeconds(5))); // → live delta (newer) + await WaitForConditionAsync(() => written.Count >= 2); + cts.Cancel(); + await streamTask; + + var protos = written.ToArray(); + var seedRow = SiteStreamGrpcClient.ConvertToAlarmEvent(RoundTripOverWire(protos[0]))!; + var liveRow = SiteStreamGrpcClient.ConvertToAlarmEvent(RoundTripOverWire(protos[1]))!; + + // Drive a REAL aggregator: seed returns the older mapped row; then Tell it the newer + // mapped delta on the live path. + var factory = new NoopSiteStreamClientFactory(); + var sink = new PublishSink(); + var aggregator = Sys.ActorOf(Props.Create(() => new SiteAlarmAggregatorActor( + "site-alpha", "e2e-parity", _ => Task.FromResult>(new[] { seedRow }), + sink.Publish, factory, "http://a:5100", "http://b:5100", TimeSpan.FromMinutes(10)))); + + await WaitForConditionAsync(() => sink.Latest is { Count: 1 }); // seed applied + aggregator.Tell(liveRow); + await WaitForConditionAsync(() => + sink.Latest is { Count: 1 } l && l[0].Priority == 800); // live delta collapsed in place + + var final = sink.Latest!; + Assert.Single(final); // exactly ONE row for the (instance, name, sourceRef) identity + Assert.Equal("Tank01.Level.HiHi", final[0].SourceReference); + Assert.Equal(AlarmKind.NativeOpcUa, final[0].Kind); + } + + // ── helpers ────────────────────────────────────────────────────────────────── + + /// + /// Serialize→deserialize a proto event to mimic the gRPC HTTP/2 hop, proving the + /// enriched AlarmStateUpdate is fully wire-representable (no field lost in codec). + /// + private static SiteStreamEvent RoundTripOverWire(SiteStreamEvent evt) => + SiteStreamEvent.Parser.ParseFrom(evt.ToByteArray()); + + private static ServerCallContext CreateMockContext(CancellationToken cancellationToken) + { + var context = Substitute.For(); + context.CancellationToken.Returns(cancellationToken); + return context; + } + + private static async Task WaitForConditionAsync(Func condition, int timeoutMs = 5000) + { + var deadline = DateTime.UtcNow.AddMilliseconds(timeoutMs); + while (!condition() && DateTime.UtcNow < deadline) + await Task.Delay(25); + Assert.True(condition(), $"Condition not met within {timeoutMs}ms"); + } + + private sealed class PublishSink + { + private readonly object _lock = new(); + private IReadOnlyList? _latest; + public void Publish(IReadOnlyList snapshot) + { + lock (_lock) { _latest = snapshot; } + } + public IReadOnlyList? Latest + { + get { lock (_lock) { return _latest; } } + } + } + + /// Aggregator transport double: its live stream simply stays open until cancelled. + private sealed class NoopSiteStreamClient : SiteStreamGrpcClient + { + public override Task SubscribeSiteAsync( + string correlationId, Action onAlarmEvent, Action onError, CancellationToken ct) + { + var tcs = new TaskCompletionSource(); + ct.Register(() => tcs.TrySetResult()); + return tcs.Task; + } + + public override void Unsubscribe(string correlationId) { } + } + + private sealed class NoopSiteStreamClientFactory : SiteStreamGrpcClientFactory + { + private readonly NoopSiteStreamClient _client = new(); + public NoopSiteStreamClientFactory() : base(NullLoggerFactory.Instance) { } + public override SiteStreamGrpcClient GetOrCreate(string siteIdentifier, string grpcEndpoint) => _client; + public override SiteStreamGrpcClient? TryGet(string siteIdentifier, string grpcEndpoint) => _client; + } +} diff --git a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Streaming/SiteStreamManagerTests.cs b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Streaming/SiteStreamManagerTests.cs index 64316ffb..e4dbfbe7 100644 --- a/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Streaming/SiteStreamManagerTests.cs +++ b/tests/ZB.MOM.WW.ScadaBridge.SiteRuntime.Tests/Streaming/SiteStreamManagerTests.cs @@ -103,6 +103,50 @@ public class SiteStreamManagerTests : TestKit, IDisposable probe2.ExpectNoMsg(TimeSpan.FromMilliseconds(500)); } + [Fact] + public void SubscribeSiteAlarms_ForwardsAlarmsForAllInstances_ButNotAttributes() + { + var probe = CreateTestProbe(); + _streamManager.SubscribeSiteAlarms(probe.Ref); + + // Alarm events from two different instances — both should arrive. + _streamManager.PublishAlarmStateChanged(new AlarmStateChanged( + "Pump1", "HighTemp", AlarmState.Active, 1, DateTimeOffset.UtcNow)); + _streamManager.PublishAlarmStateChanged(new AlarmStateChanged( + "Pump2", "LowFlow", AlarmState.Active, 2, DateTimeOffset.UtcNow)); + + // Attribute events must be filtered out entirely. + _streamManager.PublishAttributeValueChanged(new AttributeValueChanged( + "Pump1", "Temperature", "Temperature", "100", "Good", DateTimeOffset.UtcNow)); + _streamManager.PublishAttributeValueChanged(new AttributeValueChanged( + "Pump3", "Flow", "Flow", "42", "Good", DateTimeOffset.UtcNow)); + + // Collect the two alarms (order across instances is not guaranteed). + var received = new[] + { + probe.ExpectMsg(TimeSpan.FromSeconds(3)), + probe.ExpectMsg(TimeSpan.FromSeconds(3)), + }; + + var instances = received.Select(a => a.InstanceUniqueName).OrderBy(n => n).ToArray(); + Assert.Equal(new[] { "Pump1", "Pump2" }, instances); + + // No attribute events (nor any further alarm) should be delivered. + probe.ExpectNoMsg(TimeSpan.FromMilliseconds(500)); + } + + [Fact] + public void SubscribeSiteAlarms_SubscriptionRemovableViaUnsubscribe() + { + var probe = CreateTestProbe(); + var id = _streamManager.SubscribeSiteAlarms(probe.Ref); + + Assert.NotNull(id); + Assert.Equal(1, _streamManager.SubscriptionCount); + Assert.True(_streamManager.Unsubscribe(id)); + Assert.Equal(0, _streamManager.SubscriptionCount); + } + [Fact] public void RemoveSubscriber_RemovesAllSubscriptionsForActor() {