merge(deferred-10): aggregated live alarm stream for Alarm Summary
Merges plan #10 (feat/live-alarm-stream) into main alongside plan #22. Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
This commit is contained in:
Joseph Doherty
@@ -124,7 +124,7 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): `
|
||||
- Inter-cluster communication uses two transports: ClusterClient for command/control (deployments, lifecycle, subscribe/unsubscribe handshake, snapshots) and gRPC server-streaming for real-time data (attribute values, alarm states). Both CentralCommunicationActor and SiteCommunicationActor registered with receptionist. Central creates one ClusterClient per site using NodeA/NodeB as contact points. Sites configure multiple central contact points for failover. Addresses cached in CentralCommunicationActor, refreshed periodically (60s) and on admin changes. Heartbeats serve health monitoring only.
|
||||
- gRPC streaming channel: SiteStreamGrpcServer on each site node (Kestrel HTTP/2, port 8083); central creates per-site SiteStreamGrpcClient via SiteStreamGrpcClientFactory. Site entity has GrpcNodeAAddress/GrpcNodeBAddress fields. Proto: sitestream.proto with SiteStreamService, SiteStreamEvent (oneof: AttributeValueUpdate, AlarmStateUpdate). DebugStreamEvent message removed (no longer flows through ClusterClient).
|
||||
- Native alarms: a read-only mirror of native alarms from OPC UA Alarms & Conditions servers and the MxAccess Gateway, unified onto an A&C-style condition model (`AlarmConditionState`: orthogonal Active/Acked/Confirmed/Shelved/Suppressed + 0–1000 severity) plus an `AlarmKind` discriminator (Computed/NativeOpcUa/NativeMxAccess). New DCL capability seam `IAlarmSubscribableConnection` (implemented by the OPC UA and MxGateway adapters); the `DataConnectionActor` opens ONE alarm feed per connection and routes transitions to instances by source-object reference. A `NativeAlarmActor` (peer to the computed `AlarmActor` under `InstanceActor`) mirrors one source binding: snapshot atomic-swap on (re)subscribe, retention (drops once inactive+acked), per-source cap, and site SQLite persistence (`native_alarm_state`, survives failover, cleared on redeploy/undeploy — mirrors static overrides). State streams to central over the additively-enriched gRPC `AlarmStateUpdate` (the existing computed `AlarmStateChanged` was enriched additively) and seeds via the DebugView snapshot. Authoring: `TemplateNativeAlarmSource` / `InstanceNativeAlarmSourceOverride` entities flatten to `ResolvedNativeAlarmSource` (inherit/compose/override); management commands + ManagementActor handlers + CLI (`template/instance native-alarm-source`) + Central UI (template editor tab + instance override panel) + enriched DebugView alarm table. Read-only — no ack-back; no central tables.
|
||||
- OPC UA / MxGateway UX (M7): operator **Alarm Summary** page (`/monitoring/alarms`, RequireDeployment, read-only) fans out the existing per-instance `DebugViewSnapshot` Ask (SemaphoreSlim-capped, partial-results tolerant) and aggregates client-side — no central alarm store, no aggregated live stream yet (follow-up); shared `AlarmStateBadges` component. OPC UA node browser gains `BrowseNext` continuation paging ("Load more"), a bounded recursive address-space **search** (`IAddressSpaceSearchable` seam; depth + result caps; substring on DisplayName/path), and **type-info** (DataType/ValueRank/Writable on `BrowseNode` for Variables). Attribute-override **CSV bulk import** (`OverrideCsvParser`, all-or-nothing) via InstanceConfigure `InputFile` + CLI `instance import-overrides --file` (native-alarm-source-override CSV deferred). **Verify-endpoint** probe (temporary `RealOpcUaClient`, short timeout, captures an untrusted server cert but NEVER trusts it) + **site-local cert trust**: per-node `CertStoreActor` (runs on every site node, not a singleton) writing the `.der` into the node's OPC UA trusted-peer PKI store; DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to BOTH site nodes so PKI stores stay consistent across failover; Admin-gated cert-management UI (`/design/connections/{id}/certificates`). No central persistence of cert trust (follow-up).
|
||||
- OPC UA / MxGateway UX (M7): operator **Alarm Summary** page (`/monitoring/alarms`, RequireDeployment, read-only) fans out the existing per-instance `DebugViewSnapshot` Ask (SemaphoreSlim-capped, partial-results tolerant) and aggregates client-side — no central alarm store; shared `AlarmStateBadges` component. **Aggregated live stream shipped 2026-07-10** (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`): a transient in-memory per-site central live cache (`ISiteAlarmLiveCache`) fed by a site-wide, alarm-only `SubscribeSite` gRPC stream (seed-then-stream), pushing near-real-time deltas to the page over the Blazor circuit with the 15s poll kept as fallback + NotReporting authority — still no persisted central alarm store. OPC UA node browser gains `BrowseNext` continuation paging ("Load more"), a bounded recursive address-space **search** (`IAddressSpaceSearchable` seam; depth + result caps; substring on DisplayName/path), and **type-info** (DataType/ValueRank/Writable on `BrowseNode` for Variables). Attribute-override **CSV bulk import** (`OverrideCsvParser`, all-or-nothing) via InstanceConfigure `InputFile` + CLI `instance import-overrides --file` (native-alarm-source-override CSV deferred). **Verify-endpoint** probe (temporary `RealOpcUaClient`, short timeout, captures an untrusted server cert but NEVER trusts it) + **site-local cert trust**: per-node `CertStoreActor` (runs on every site node, not a singleton) writing the `.der` into the node's OPC UA trusted-peer PKI store; DeploymentManager broadcasts `TrustServerCertCommand`/`RemoveServerCertCommand` to BOTH site nodes so PKI stores stay consistent across failover; Admin-gated cert-management UI (`/design/connections/{id}/certificates`). No central persistence of cert trust (follow-up).
|
||||
|
||||
### External Integrations
|
||||
- External System Gateway: HTTP/REST only, JSON serialization, API key + Basic Auth.
|
||||
|
||||
@@ -21,7 +21,7 @@ These are **mirrored** alarms — the source is the source of truth — which is
|
||||
| State model scope | **Unified** A&C-style state model for *all* alarms (computed + native) |
|
||||
| Interactivity | **Read-only mirror** — display source-reported state; no acking/shelving from ScadaBridge, no command relay, no operator identity captured by ScadaBridge (source-supplied operator user/comment *are* displayed for native alarms) |
|
||||
| Binding | Instance declares a `NativeAlarmSource` (connection + source ref); conditions under it are **discovered at runtime**, keyed by source reference |
|
||||
| State location | **Site-only**, persisted to SQLite (survives restart/failover); central **queries live** (snapshot + live stream); **no central tables, no central history** |
|
||||
| State location | **Site-only**, persisted to SQLite (survives restart/failover); central **queries live** (snapshot + live stream); **no central tables, no central history** (the M9 aggregated live alarm stream — `2026-07-10-aggregated-live-alarm-stream-plan.md` — honors this: a **transient, in-memory** central cache only, nothing persisted) |
|
||||
| MxGateway transport | Gateway session-less `StreamAlarms` feed |
|
||||
| OPC UA transport | Alarms & Conditions events + `ConditionRefresh` snapshot |
|
||||
| Site actor structure | New `NativeAlarmActor` child of `InstanceActor`, peer to computed `AlarmActor`s (Approach 1) |
|
||||
|
||||
@@ -92,7 +92,7 @@ Delivered per `docs/plans/2026-06-18-m7-opcua-mxgateway-ux-design.md` (full scop
|
||||
- **T16** — browse type-info (DataType/ValueRank/Writable) + **attribute**-override CSV import (InstanceConfigure InputFile + CLI `instance import-overrides --file`). Native-alarm-source-override CSV import was **deferred** (attribute overrides only).
|
||||
- **T17** — Verify-endpoint probe (captures-but-never-trusts an untrusted server cert) + **site-local** cert trust (per-node `CertStoreActor`, DeploymentManager broadcast to **both** site nodes; D6) + Admin-gated cert-management UI.
|
||||
|
||||
Small follow-ups logged (not blocking): stamp `SourceNode` on the `SecuredWrite` audit rows (currently NULL); an aggregated **live** alarm stream for the summary page (snapshot + poll today); central-persisted, auditable cert trust (site-local today).
|
||||
Small follow-ups logged (not blocking): stamp `SourceNode` on the `SecuredWrite` audit rows (currently NULL); an aggregated **live** alarm stream for the summary page (snapshot + poll today) — *delivered 2026-07-10, `docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`*; central-persisted, auditable cert trust (site-local today).
|
||||
|
||||
#### M8 — Transport (T18, T20) — **DELIVERED**
|
||||
Delivered both features, with the silent gap #16 folded in:
|
||||
|
||||
@@ -32,6 +32,10 @@ from the merged `fix/mxgw-supervisory-write`). M7 does **not** reinvent those
|
||||
- **Central alarm store / history / journal** — T13 reads live snapshots; no central alarm
|
||||
tables (remains `[PERM]` per `2026-05-29-native-alarms-design.md`). No aggregated live gRPC
|
||||
stream for the summary page in v1 (snapshot + poll only).
|
||||
**Delivered 2026-07-10** (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`): the
|
||||
aggregated live gRPC stream shipped as a **transient, in-memory** per-site central live cache
|
||||
(site-wide alarm-only `SubscribeSite` stream, seed-then-stream). The `[PERM]` no-central-store
|
||||
rule is unchanged — the cache persists nothing.
|
||||
- **Native-alarm ack / shelve / suppress write-back** — read-only by design, unchanged.
|
||||
- **CSV bulk import of native-alarm-source overrides** (`InstanceNativeAlarmSourceOverride`) —
|
||||
the T16 importer targets *attribute* overrides only; native-alarm-source CSV is a follow-up.
|
||||
@@ -102,7 +106,9 @@ M7 adds these verbs on that path (all additive, mirroring `BrowseCommands.cs` +
|
||||
- **Reuse:** extract DebugView's inline alarm badge/formatter markup into a shared
|
||||
`AlarmStateBadges` component consumed by both DebugView and the summary page.
|
||||
- **Refresh:** manual button + optional poll timer (mirrors Health dashboard 10 s). No
|
||||
aggregated live stream in v1.
|
||||
aggregated live stream in v1. *(Aggregated live stream delivered 2026-07-10 —
|
||||
`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`; the poll is now a fallback +
|
||||
`NotReporting` authority behind the transient per-site live cache.)*
|
||||
- **Files (indicative):** `Components/Pages/Monitoring/AlarmSummary.razor(.cs)`, a CentralUI
|
||||
`IAlarmSummaryService`/impl (fan-out + aggregate), `Components/Shared/AlarmStateBadges.razor`,
|
||||
`NavMenu.razor`, Playwright test.
|
||||
|
||||
@@ -20,7 +20,6 @@ them and are removed from this table when that plan's task lands.
|
||||
|---|------|-------------|------------------------|-----------------|
|
||||
| 8 | Hash-chain tamper evidence (T1); CLI verify-chain is a no-op stub | audit-log roadmap :12 | v1.x by locked decision; append-only DB roles are the control | Compliance requirement for cryptographic tamper evidence |
|
||||
| 9 | Parquet audit archival (T2); endpoint returns 501 | AuditEndpoints.cs:204 | v1.x; 501 + CLI messaging are honest | AuditLog partition volume nears retention ceiling |
|
||||
| 10 | Aggregated live alarm stream for Alarm Summary | m7 design :252 | Snapshot fan-out acceptable at current instance counts | Alarm Summary latency complaints or >~50 instances/site |
|
||||
| 11 | Central-persisted OPC UA cert-trust audit | m7 follow-ups | Broadcast-to-both-nodes covers HA | Governance/audit requirement for trust decisions |
|
||||
| 12 | Native-alarm-source-override CSV import — **Central UI `InstanceConfigure` upload affordance only** (CLI + Management API + parser shipped 2026-07-10, see Resolved) | m7 follow-ups | CLI/API path closes the operator parity gap; the Blazor upload button is polish | First request to bulk-import native sources from the UI rather than the CLI |
|
||||
| 17 | Unified notifications+site-calls outbox page | stillpending :118 | Explicit M9 decision to keep two pages | Operator confusion reports |
|
||||
@@ -32,6 +31,7 @@ Rows removed from the Deferred table above once confirmed shipped. Kept here for
|
||||
|
||||
| # | Item | Resolution |
|
||||
|---|------|-----------|
|
||||
| 10 | Aggregated live alarm stream for Alarm Summary | Shipped 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`): a **transient, in-memory** per-site central live alarm cache (`ISiteAlarmLiveCache`/`SiteAlarmLiveCacheService` + per-site `SiteAlarmAggregatorActor`) fed by a new site-wide, alarm-only `SubscribeSite` gRPC stream (`SiteStreamManager.SubscribeSiteAlarms`), seed-then-stream with dedup + NodeA↔NodeB re-seed + periodic reconcile. Alarm Summary is now live-cache-driven (`AlarmSummaryService.BuildFromLiveAlarms`) with the 15s poll retained as fallback + `NotReporting` authority. Honors the `[PERM]` no-central-store rule — nothing persisted (no EF table/migration). Options on `CommunicationOptions` (eagerly validated) + two `ScadaBridgeTelemetry` signals. |
|
||||
| 7 | SecuredWrite audit rows leave SourceNode NULL | Resolved (PLAN-07): `ManagementActor.EmitSecuredWriteAuditAsync` routes through `ICentralAuditWriter`, which stamps `SourceNode` (`central-a`/`central-b`) from `INodeIdentityProvider`. |
|
||||
| 12 (CLI/API) | Native-alarm-source-override CSV import | Shipped 2026-07-10: shared `CsvLineSplitter`, `NativeAlarmSourceOverrideCsvParser`, bulk all-or-nothing `SetInstanceNativeAlarmSourceOverridesCommand` + ManagementActor handler (Deployer-gated), CLI `instance native-alarm-source import --file`, parser/CLI/handler tests. **UI upload affordance still pending — see row 12 above.** |
|
||||
| 13 | WaitForAttribute quality-gated ("Good"-only) mode | Already implemented (Commons `WaitForAttribute.RequireGoodQuality`, enforced in `InstanceActor`, threaded through `ScriptRuntimeContext`, tested in `InstanceActorWaitForAttributeTests`). Stale "planned enhancement" doc line corrected 2026-07-10. |
|
||||
|
||||
@@ -187,10 +187,11 @@ Per-leaf alarm rendering (leaf nodes are individual conditions for native alarms
|
||||
|
||||
### Alarm Summary (Deployment Role)
|
||||
- A dedicated operator **Alarm Summary** page (`/monitoring/alarms`, `RequireDeployment`) gives a **cross-instance, read-only** roll-up of live alarm state at a site — the operator-facing complement to the per-instance Debug View.
|
||||
- **Data path** — no new site-side code and no central alarm store. The page selects a site, queries its deployed instances, then fans out the existing per-instance `DebugViewSnapshot` Ask **concurrently** (capped with a `SemaphoreSlim`) and aggregates the returned `AlarmStates` client-side. The fan-out is **partial-results tolerant**: instances that time out are listed as "not reporting" while the rest still render.
|
||||
- **Data path** — no new site-side code and no central alarm store. The page selects a site, queries its deployed instances, then fans out the existing per-instance `DebugViewSnapshot` Ask **concurrently** (capped with a `SemaphoreSlim`) and aggregates the returned `AlarmStates` client-side. The fan-out is **partial-results tolerant**: instances that time out are listed as "not reporting" while the rest still render. This snapshot fan-out now doubles as the **seed** for a near-real-time live feed (see **Live updates** below) rather than the sole refresh mechanism.
|
||||
- **Live updates** — the page is driven by a **transient, per-site central live alarm cache** (`ISiteAlarmLiveCache`, owned by the Communication component; see [Component-Communication](Component-Communication.md)). On site select the page subscribes to the cache; the cache runs one shared, reference-counted per-site aggregator that **seeds** from the snapshot fan-out and then stays warm on a single **site-wide, alarm-only** `SubscribeSite` gRPC stream (seed-then-stream, dedup by `(InstanceUniqueName, AlarmName, SourceReference)`). Applied deltas raise an in-process change event (mirroring `IDeploymentStatusNotifier`) that the Blazor circuit pushes to the browser via `StateHasChanged()` — no new SignalR hub. `AlarmSummaryService.BuildFromLiveAlarms` rebuilds the roll-up + rows from the cache's current alarm set. The cache is **purely in-memory on the active central node** — there is still **no persisted central alarm store**; on a NodeA↔NodeB failover the new active node re-seeds from scratch.
|
||||
- **View** — roll-up tiles (total active, worst severity, unacked count, per-`AlarmKind` counts) plus a flat, sortable, filterable table. Filters cover instance, `AlarmKind` (Computed / NativeOpcUa / NativeMxAccess), state, acked/unacked, severity threshold, and name search.
|
||||
- **Read-only** — there are no ack / shelve / suppress controls (native alarms remain read-only by design).
|
||||
- **Refresh** — manual refresh button plus an optional poll timer (mirroring the Health dashboard); there is no aggregated live alarm stream in this release (snapshot + poll only — logged as a follow-up).
|
||||
- **Refresh** — manual refresh button plus the 15s poll timer (mirroring the Health dashboard), now retained as a **fallback + `NotReporting` authority** behind the live cache: when the cache reports `IsLive`, the page renders live-cache state; when a stream is unhealthy or a site has not yet seeded, the poll keeps the page fresh so a stream failure never blanks it. (Aggregated live stream **delivered 2026-07-10** — see `docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`.)
|
||||
- **Reuse** — the alarm badge/formatter markup is factored out of Debug View into a shared `AlarmStateBadges` component consumed by both Debug View and this page.
|
||||
|
||||
### Parked Message Management (Deployment Role)
|
||||
|
||||
@@ -67,6 +67,17 @@ Both central and site clusters. Each side has communication actors that handle m
|
||||
- **Accepted limitation (central-side session locality):** debug sessions are process-local state on the central node hosting the `DebugStreamBridgeActor`. A central **restart or failover drops all active sessions** with no `OnStreamTerminated` signal to the operator — the engineer simply re-establishes the debug session from the UI. Debug streaming is an interactive, transient diagnostic aid, so this is an accepted trade-off rather than a durability gap (arch review 02, U7).
|
||||
- **Backpressure is lossy-by-design and now observable:** the per-session `Channel<SiteStreamEvent>` (bounded 1000, `DropOldest`) silently evicts the oldest event when a slow consumer falls behind. The eviction is counted via the channel's `itemDropped` callback and logged at Warning (first eviction, then every 500th) so real event loss on a debug stream is visible instead of silent.
|
||||
|
||||
### 6.1 Aggregated Live Alarm Stream (Site → Central)
|
||||
|
||||
Delivered 2026-07-10 (`docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md`) for the operator Alarm Summary page (Central UI #9). Where §6 streams **one instance** for the interactive Debug View, this feeds a **transient, per-site central live alarm cache** so the Alarm Summary reflects alarm transitions in near-real-time instead of re-polling the whole site every 15s.
|
||||
|
||||
- **Site-wide, alarm-only stream**: a single `SubscribeSite` gRPC stream per site carries `AlarmStateChanged` events for **all** instances (attribute events are dropped — the summary never shows them, and attributes are far higher-volume). On the site this reuses the existing `SiteStreamManager` broadcast hub via `SiteStreamManager.SubscribeSiteAlarms(...)` (same wiring as the per-instance `Subscribe`, minus the `InstanceUniqueName` filter and filtered to alarms). `StreamRelayActor` is reused unchanged, so placeholder rows (`is_configured_placeholder`) are still dropped at the relay.
|
||||
- **Central live cache** (`ISiteAlarmLiveCache`, singleton `SiteAlarmLiveCacheService`): a DI singleton on the active central node. For each site with ≥1 active viewer it runs ONE shared, **reference-counted** per-site aggregator (`SiteAlarmAggregatorActor`); the first `Subscribe(siteId, onChanged)` starts it, the last subscriber leaving stops it after a short **linger** to avoid re-seed thrash. `GetCurrentAlarms(siteId)` returns the current immutable snapshot; `IsLive(siteId)` reports whether the aggregator has seeded and published at least once.
|
||||
- **Seed-then-stream** (copied from `DebugStreamBridgeActor` ordering): open the `SubscribeSite` stream first (buffer live deltas), run the snapshot fan-out once via the existing `DebugViewSnapshot` path (bounded by `LiveAlarmCacheSeedConcurrency`), flush the buffer with **dedup by `(InstanceUniqueName, AlarmName, SourceReference)`**, then live pass-through. Placeholders are seeded from the snapshot and never expected on the live stream.
|
||||
- **Failover & drift**: NodeA↔NodeB reconnect (client via `SiteStreamGrpcClient.SubscribeSiteAsync`, factory endpoint failover) re-seeds rather than serving stale; a periodic **reconcile snapshot** (default 60s) corrects any missed enable/disable so the cache can't drift indefinitely. `[PERM]` (`docs/plans/2026-05-29-native-alarms-design.md`): the cache is **purely in-memory** — no EF entity/table/migration, no persisted central alarm store — so a new active node simply re-seeds from scratch.
|
||||
- **Options** (`Communication` section, `CommunicationOptions`; eagerly validated by `CommunicationOptionsValidator` / `ValidateOnStart`): `LiveAlarmCacheLinger` (default 30s), `LiveAlarmCacheReconcileInterval` (default 60s), `LiveAlarmCacheSeedConcurrency` (default 8), `LiveAlarmCacheMaxSubscribersPerSite` (default 200).
|
||||
- **Telemetry** (`ScadaBridgeTelemetry` meter): observable gauge `scadabridge.site.alarm_cache.aggregators.active` (running per-site aggregators) and counter `scadabridge.site.alarm_cache.reconnects` (site-wide stream reconnects — a NodeA↔NodeB flip or reconcile-driven reopen; a sustained climb signals a flapping site link).
|
||||
|
||||
#### Site-Side gRPC Streaming Components
|
||||
|
||||
- **SiteStreamGrpcServer**: gRPC service (`SiteStreamService.SiteStreamServiceBase`) hosted on each site node via Kestrel HTTP/2 on a dedicated port (default 8083). Implements the `SubscribeInstance` RPC. For each subscription, creates a `StreamRelayActor` that subscribes to `SiteStreamManager`, bridges events through a `Channel<SiteStreamEvent>` to the gRPC response stream. Tracks active subscriptions by `correlation_id` — duplicate IDs cancel the old stream. Enforces a max concurrent stream limit (default 100). Rejects streams with `StatusCode.Unavailable` before the actor system is ready.
|
||||
@@ -84,13 +95,14 @@ Both central and site clusters. Each side has communication actors that handle m
|
||||
|
||||
The streaming protocol is defined in `sitestream.proto` (`src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto`):
|
||||
|
||||
- **Service**: `SiteStreamService` — hosted on each site node by `SiteStreamGrpcServer` — exposes five RPCs. One is the original real-time **server-streaming** subscription; the other four are **unary request/response** calls added by the Audit Log (#23) and Site Call Audit (#22) components. A unary call is request/response and is distinct from the command/control ClusterClient channel — gRPC on this service is no longer real-time-stream-only:
|
||||
- `SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent)` — the real-time debug stream (§6); the only server-streaming RPC.
|
||||
- **Service**: `SiteStreamService` — hosted on each site node by `SiteStreamGrpcServer` — exposes six RPCs. Two are real-time **server-streaming** subscriptions (`SubscribeInstance`, `SubscribeSite`); the other four are **unary request/response** calls added by the Audit Log (#23) and Site Call Audit (#22) components. A unary call is request/response and is distinct from the command/control ClusterClient channel — gRPC on this service is no longer real-time-stream-only:
|
||||
- `SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent)` — the per-instance real-time debug stream (§6).
|
||||
- `SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent)` — the **site-wide, alarm-only** aggregated stream (§6.1) added for the operator Alarm Summary live cache. Additive (new RPC + new `SiteStreamRequest { correlation_id }` message; no field renumbering). The server handler mirrors `SubscribeInstance` — same bounded `Channel(1000, DropOldest)`, `GrpcMaxConcurrentStreams`/`GrpcMaxStreamLifetime` limits, `SiteConnectionOpened/Closed` telemetry, and `StreamRelayActor` mapping — but subscribes via `SiteStreamManager.SubscribeSiteAlarms` (all instances, `AlarmStateChanged` only; attribute events dropped, no per-instance filter).
|
||||
- `IngestAuditEvents(AuditEventBatch) returns (IngestAck)` — central-side **ingest** receiving surface for Audit Log (#23) telemetry; routes the batch to the central `AuditLogIngestActor` proxy and returns the accepted `EventId`s. (The production *push* path is still ClusterClient via `ClusterClientSiteAuditClient`; this RPC is the gRPC-receiving counterpart.)
|
||||
- `IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck)` — ingest receiving surface for the combined cached-call telemetry packet (audit row + `SiteCalls` operational upsert written in one transaction).
|
||||
- `PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse)` — central→site **reconciliation pull** for the Audit Log self-heal feed; the site serves `Pending`/`Forwarded` rows from its `ISiteAuditQueue`.
|
||||
- `PullSiteCalls(PullSiteCallsRequest) returns (PullSiteCallsResponse)` — central→site reconciliation pull for the Site Call Audit (#22) self-heal feed; the site serves operation-tracking rows changed since a cursor from its `IOperationTrackingStore`. A separate RPC from `PullAuditEvents` because the tracking store is the operational source of truth, distinct from the site audit queue.
|
||||
- **Messages**: `InstanceStreamRequest` (correlation_id, instance_unique_name), `SiteStreamEvent` (correlation_id, oneof event: `AttributeValueUpdate`, `AlarmStateUpdate`); `AuditEventDto`/`AuditEventBatch`/`IngestAck` for ingest; `CachedTelemetryPacket`/`CachedTelemetryBatch` (each packet pairing an `AuditEventDto` with a `SiteCallOperationalDto`); `PullAuditEventsRequest`/`PullAuditEventsResponse` and `PullSiteCallsRequest`/`PullSiteCallsResponse` (each request carries `since_utc` + `batch_size`; each response carries `more_available` to signal a saturated batch).
|
||||
- **Messages**: `InstanceStreamRequest` (correlation_id, instance_unique_name), `SiteStreamRequest` (correlation_id only — no instance name; drives the site-wide `SubscribeSite` stream), `SiteStreamEvent` (correlation_id, oneof event: `AttributeValueUpdate`, `AlarmStateUpdate`); `AuditEventDto`/`AuditEventBatch`/`IngestAck` for ingest; `CachedTelemetryPacket`/`CachedTelemetryBatch` (each packet pairing an `AuditEventDto` with a `SiteCallOperationalDto`); `PullAuditEventsRequest`/`PullAuditEventsResponse` and `PullSiteCallsRequest`/`PullSiteCallsResponse` (each request carries `since_utc` + `batch_size`; each response carries `more_available` to signal a saturated batch).
|
||||
- The `oneof event` pattern is extensible — future event types (health metrics, connection state changes) are added as new fields without breaking existing consumers.
|
||||
- Proto field numbers are never reused; new RPCs and message fields are appended additively. Old clients ignore unknown `oneof` variants.
|
||||
|
||||
|
||||
@@ -5,9 +5,11 @@
|
||||
@using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites
|
||||
@using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories
|
||||
@using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums
|
||||
@using ZB.MOM.WW.ScadaBridge.Communication
|
||||
@implements IDisposable
|
||||
@inject IAlarmSummaryService AlarmSummaryService
|
||||
@inject ISiteRepository SiteRepository
|
||||
@inject ISiteAlarmLiveCache LiveAlarmCache
|
||||
|
||||
<div class="container-fluid mt-3" data-test="alarm-summary">
|
||||
<div class="d-flex justify-content-between align-items-center mb-3">
|
||||
@@ -259,13 +261,23 @@
|
||||
_notReporting = Array.Empty<string>();
|
||||
_rollup = new AlarmRollup(0, 0, 0, new Dictionary<AlarmKind, int>());
|
||||
StopTimer();
|
||||
DisposeLiveSubscription();
|
||||
return;
|
||||
}
|
||||
|
||||
_selectedSiteId = siteId;
|
||||
ClearFilters();
|
||||
// Subscribe to the shared live cache FIRST so the aggregator (and its
|
||||
// seed-then-stream) starts warming while the initial poll runs.
|
||||
SubscribeLive(siteId);
|
||||
await RefreshAsync();
|
||||
StartTimer();
|
||||
// If another circuit already warmed the cache for this site, the initial
|
||||
// poll may be staler than what's live — apply the live snapshot on top.
|
||||
if (LiveAlarmCache.IsLive(siteId))
|
||||
{
|
||||
ApplyLiveSnapshot(siteId);
|
||||
}
|
||||
}
|
||||
|
||||
private async Task RefreshAsync()
|
||||
@@ -322,6 +334,64 @@
|
||||
_refreshTimer = null;
|
||||
}
|
||||
|
||||
// ── Live cache (plan #10, Task 5) ──────────────────────────────────────────
|
||||
// The page is live-cache-first with the 15s poll kept as a fallback/safety net.
|
||||
// Reconciliation model:
|
||||
// • The live cache pushes onChanged deltas (near-real-time) whenever the site's
|
||||
// aggregated alarm set changes. We rebuild _rows/_rollup/_visibleRows from the
|
||||
// immutable live snapshot — but deliberately DO NOT touch _notReporting, since
|
||||
// the alarm-only live cache can't compute it.
|
||||
// • The 15s poll (RefreshAsync) still runs untouched: it is the authority for
|
||||
// _notReporting and the safety net when the cache is not live (pre-seed or a
|
||||
// degraded/failed stream — IsLive == false). When live, the poll simply
|
||||
// re-affirms the same snapshot; the live path just makes updates arrive sooner.
|
||||
// • Both paths mutate shared state only via the Blazor dispatcher (the poll via
|
||||
// its InvokeAsync callback, the live delta via OnLiveChanged's InvokeAsync), so
|
||||
// they are serialized and never race. Each rebuild is an idempotent snapshot, so
|
||||
// a live rebuild immediately followed by a poll rebuild (or vice-versa) is safe.
|
||||
private IDisposable? _liveSubscription;
|
||||
|
||||
private void SubscribeLive(int siteId)
|
||||
{
|
||||
DisposeLiveSubscription();
|
||||
_liveSubscription = LiveAlarmCache.Subscribe(siteId, () => OnLiveAlarmsChanged(siteId));
|
||||
}
|
||||
|
||||
private void DisposeLiveSubscription()
|
||||
{
|
||||
_liveSubscription?.Dispose();
|
||||
_liveSubscription = null;
|
||||
}
|
||||
|
||||
// Raised on the aggregator's thread — marshal onto the circuit before touching state.
|
||||
private void OnLiveAlarmsChanged(int siteId)
|
||||
{
|
||||
_ = InvokeAsync(() =>
|
||||
{
|
||||
// Drop stale callbacks for a site we've since navigated away from, and
|
||||
// let the poll drive until the aggregator has actually seeded (so we never
|
||||
// clobber a good poll snapshot with an empty pre-seed list).
|
||||
if (_selectedSiteId != siteId || !LiveAlarmCache.IsLive(siteId))
|
||||
{
|
||||
return;
|
||||
}
|
||||
|
||||
ApplyLiveSnapshot(siteId);
|
||||
StateHasChanged();
|
||||
});
|
||||
}
|
||||
|
||||
// Rebuilds rows/rollup/visible-rows from the live snapshot. Leaves _notReporting
|
||||
// as the last poll computed it (the alarm-only cache can't know it). Idempotent.
|
||||
private void ApplyLiveSnapshot(int siteId)
|
||||
{
|
||||
var current = LiveAlarmCache.GetCurrentAlarms(siteId);
|
||||
var result = AlarmSummaryService.BuildFromLiveAlarms(current);
|
||||
_rows = result.Alarms;
|
||||
_rollup = AlarmSummaryService.ComputeRollup(_rows);
|
||||
RecomputeVisibleRows();
|
||||
}
|
||||
|
||||
private IEnumerable<string> DistinctInstances =>
|
||||
_rows.Select(r => r.InstanceUniqueName).Distinct().OrderBy(n => n, StringComparer.OrdinalIgnoreCase);
|
||||
|
||||
@@ -434,5 +504,9 @@
|
||||
_ => "Computed"
|
||||
};
|
||||
|
||||
public void Dispose() => StopTimer();
|
||||
public void Dispose()
|
||||
{
|
||||
StopTimer();
|
||||
DisposeLiveSubscription();
|
||||
}
|
||||
}
|
||||
|
||||
@@ -127,6 +127,26 @@ public sealed class AlarmSummaryService : IAlarmSummaryService
|
||||
}
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
public AlarmSummaryResult BuildFromLiveAlarms(IReadOnlyList<AlarmStateChanged> alarms)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(alarms);
|
||||
|
||||
// Flatten one row per alarm (instance from the alarm itself, mirroring
|
||||
// FetchInstanceAsync) and apply the SAME deterministic instance-then-name
|
||||
// sort as GetSiteAlarmsAsync so the live and poll paths render identically.
|
||||
var orderedRows = alarms
|
||||
.Select(alarm => new AlarmSummaryRow(alarm.InstanceUniqueName, alarm))
|
||||
.OrderBy(r => r.InstanceUniqueName, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenBy(r => r.Alarm.AlarmName, StringComparer.OrdinalIgnoreCase)
|
||||
.ToList();
|
||||
|
||||
// The live cache is alarm-only, so it cannot enumerate "not reporting"
|
||||
// instances — left empty here; the periodic poll (GetSiteAlarmsAsync)
|
||||
// remains the authority for that list.
|
||||
return new AlarmSummaryResult(orderedRows, Array.Empty<string>());
|
||||
}
|
||||
|
||||
/// <inheritdoc/>
|
||||
public AlarmRollup ComputeRollup(IReadOnlyList<AlarmSummaryRow> rows)
|
||||
{
|
||||
|
||||
@@ -42,6 +42,28 @@ public interface IAlarmSummaryService
|
||||
/// </returns>
|
||||
Task<AlarmSummaryResult> GetSiteAlarmsAsync(int siteId, CancellationToken cancellationToken = default);
|
||||
|
||||
/// <summary>
|
||||
/// Builds an <see cref="AlarmSummaryResult"/> directly from an in-memory live
|
||||
/// snapshot (plan #10, Task 5) — the current alarms served by
|
||||
/// <see cref="ZB.MOM.WW.ScadaBridge.Communication.ISiteAlarmLiveCache.GetCurrentAlarms"/>
|
||||
/// — instead of fanning out per-instance snapshot Asks. Each
|
||||
/// <see cref="AlarmStateChanged"/> is flattened to one
|
||||
/// <see cref="AlarmSummaryRow"/> (instance taken from
|
||||
/// <see cref="AlarmStateChanged.InstanceUniqueName"/>) and ordered with the SAME
|
||||
/// deterministic instance-then-alarm-name sort as
|
||||
/// <see cref="GetSiteAlarmsAsync"/>, so the two paths are interchangeable.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// The live cache carries only alarm state, so it cannot know which Enabled
|
||||
/// instances are silent-but-reporting versus not reporting at all;
|
||||
/// <see cref="AlarmSummaryResult.NotReportingInstances"/> is therefore always
|
||||
/// empty on this path. The page keeps its periodic <see cref="GetSiteAlarmsAsync"/>
|
||||
/// poll as the authority for "not reporting" (and as a live-stream safety net).
|
||||
/// </remarks>
|
||||
/// <param name="alarms">The current live alarm snapshot for a site.</param>
|
||||
/// <returns>An <see cref="AlarmSummaryResult"/> whose <see cref="AlarmSummaryResult.NotReportingInstances"/> is empty.</returns>
|
||||
AlarmSummaryResult BuildFromLiveAlarms(IReadOnlyList<AlarmStateChanged> alarms);
|
||||
|
||||
/// <summary>
|
||||
/// Pure roll-up over a set of <see cref="AlarmSummaryRow"/>s. Exposed so the
|
||||
/// page (and tests) can recompute the headline tiles without re-querying.
|
||||
|
||||
@@ -47,11 +47,23 @@ public static class ScadaBridgeTelemetry
|
||||
Meter.CreateCounter<long>("scadabridge.store_and_forward.replication.failures", unit: "1",
|
||||
description: "S&F buffer replication operations that failed to dispatch/deliver to the peer node");
|
||||
|
||||
/// <summary>
|
||||
/// Incremented each time a per-site live-alarm aggregator re-establishes its site-wide
|
||||
/// gRPC stream — a NodeA↔NodeB failover flip or a reconcile-driven reopen after the
|
||||
/// stream was given up (plan #10, Task 6). A sustained climb signals a flapping site link.
|
||||
/// </summary>
|
||||
private static readonly Counter<long> _liveAlarmStreamReconnects =
|
||||
Meter.CreateCounter<long>("scadabridge.site.alarm_cache.reconnects", unit: "1",
|
||||
description: "Live-alarm aggregator site-wide gRPC stream reconnects (NodeA↔NodeB flip or reconcile-driven reopen).");
|
||||
|
||||
// ---------------- Observable gauges ----------------
|
||||
|
||||
/// <summary>Current count of open site connections, mutated via <see cref="Interlocked"/>.</summary>
|
||||
private static long _siteConnectionsUp;
|
||||
|
||||
/// <summary>Current count of running per-site live-alarm aggregators, mutated via <see cref="Interlocked"/>.</summary>
|
||||
private static long _liveAlarmAggregatorsActive;
|
||||
|
||||
/// <summary>Provider that yields the live StoreAndForward queue depth; set by a later task.</summary>
|
||||
private static Func<long>? _queueDepthProvider;
|
||||
|
||||
@@ -68,6 +80,13 @@ public static class ScadaBridgeTelemetry
|
||||
() => Volatile.Read(ref _queueDepthProvider) is { } p ? p() : 0L,
|
||||
unit: "items",
|
||||
description: "Current StoreAndForward queue depth.");
|
||||
|
||||
/// <summary>Gauge reporting the number of currently running per-site live-alarm aggregators.</summary>
|
||||
private static readonly ObservableGauge<long> _liveAlarmAggregatorsActiveGauge =
|
||||
Meter.CreateObservableGauge<long>("scadabridge.site.alarm_cache.aggregators.active",
|
||||
() => Interlocked.Read(ref _liveAlarmAggregatorsActive),
|
||||
unit: "1",
|
||||
description: "Number of per-site live-alarm aggregators currently running on the active central node.");
|
||||
#pragma warning restore IDE0052
|
||||
|
||||
// ---------------- Emit helpers ----------------
|
||||
@@ -94,6 +113,15 @@ public static class ScadaBridgeTelemetry
|
||||
/// <summary>Records that a site connection closed (decrements the up-count gauge).</summary>
|
||||
public static void SiteConnectionClosed() => Interlocked.Decrement(ref _siteConnectionsUp);
|
||||
|
||||
/// <summary>Records that a per-site live-alarm aggregator started (increments the active-aggregator gauge).</summary>
|
||||
public static void LiveAlarmAggregatorStarted() => Interlocked.Increment(ref _liveAlarmAggregatorsActive);
|
||||
|
||||
/// <summary>Records that a per-site live-alarm aggregator stopped (decrements the active-aggregator gauge).</summary>
|
||||
public static void LiveAlarmAggregatorStopped() => Interlocked.Decrement(ref _liveAlarmAggregatorsActive);
|
||||
|
||||
/// <summary>Records that a per-site live-alarm aggregator re-established its site-wide gRPC stream.</summary>
|
||||
public static void RecordLiveAlarmStreamReconnect() => _liveAlarmStreamReconnects.Add(1);
|
||||
|
||||
/// <summary>
|
||||
/// Registers the provider the StoreAndForward queue-depth gauge reads on each observation.
|
||||
/// A later task supplies a provider that reads the real StoreAndForward depth. A null
|
||||
|
||||
@@ -0,0 +1,530 @@
|
||||
using Akka.Actor;
|
||||
using Akka.Event;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Communication.Actors;
|
||||
|
||||
/// <summary>
|
||||
/// One-per-site aggregator on the active central node backing the operator Alarm
|
||||
/// Summary live cache (plan #10, Task 4). Holds a transient in-memory
|
||||
/// <c>Dictionary<AlarmKey, AlarmStateChanged></c> of the whole site's current
|
||||
/// alarm state — NO persistence (locked <c>[PERM]</c>: no central alarm store). Created
|
||||
/// and torn down by <see cref="SiteAlarmLiveCacheService"/> under viewer reference-count.
|
||||
/// <para>
|
||||
/// <b>Seed-then-stream ordering</b> (copied from <see cref="DebugStreamBridgeActor"/>):
|
||||
/// the site-wide, alarm-only gRPC stream (<c>SubscribeSite</c>) is opened FIRST in
|
||||
/// <see cref="PreStart"/> so live deltas start flowing during the seed's snapshot-build
|
||||
/// + transit window; deltas arriving while a seed/reconcile fan-out is in flight are
|
||||
/// <em>buffered in arrival order</em>. When the fan-out completes the cache is rebuilt
|
||||
/// authoritatively from the fresh snapshot, then the buffer is flushed with per-key
|
||||
/// dedup against the seed, then the actor passes live deltas straight into the cache.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Placeholder reconciliation:</b> the snapshot fan-out carries
|
||||
/// <see cref="AlarmStateChanged.IsConfiguredPlaceholder"/> rows (a configured native
|
||||
/// source binding with no active conditions); the live stream drops them server-side.
|
||||
/// Placeholders are seeded and never expected on the live stream — a live delta for a
|
||||
/// real alarm has a distinct <c>AlarmKey</c> so it can never wipe a placeholder row and
|
||||
/// vice-versa. Placeholder-vs-real coherence is refreshed by the periodic reconcile.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Failover + drift:</b> a gRPC error flips NodeA↔NodeB with the same retry budget +
|
||||
/// stability window as <see cref="DebugStreamBridgeActor"/>, and each reconnect triggers
|
||||
/// a RE-SEED (never silently serve stale). A periodic reconcile snapshot
|
||||
/// (<see cref="_reconcileInterval"/>, default 60s) corrects instance-set drift and any
|
||||
/// missed delta.
|
||||
/// </para>
|
||||
/// All state is mutated only on the actor thread: gRPC callbacks and fan-out results are
|
||||
/// marshalled back via <c>Self.Tell</c>, so the cache needs no internal lock. The
|
||||
/// published snapshot handed to the service is a fresh immutable list (reference swap),
|
||||
/// so Blazor render threads never observe a partially-mutated cache.
|
||||
/// </summary>
|
||||
public sealed class SiteAlarmAggregatorActor : ReceiveActor, IWithTimers
|
||||
{
|
||||
private readonly ILoggingAdapter _log = Context.GetLogger();
|
||||
private readonly string _siteIdentifier;
|
||||
private readonly string _correlationId;
|
||||
private readonly Func<CancellationToken, Task<IReadOnlyList<AlarmStateChanged>>> _seedFn;
|
||||
private readonly Action<IReadOnlyList<AlarmStateChanged>> _publish;
|
||||
private readonly SiteStreamGrpcClientFactory _grpcFactory;
|
||||
private readonly string _grpcNodeAAddress;
|
||||
private readonly string _grpcNodeBAddress;
|
||||
private readonly TimeSpan _reconcileInterval;
|
||||
|
||||
private const int MaxRetries = 3;
|
||||
private const string ReconnectTimerKey = "alarm-grpc-reconnect";
|
||||
private const string StabilityTimerKey = "alarm-grpc-stability";
|
||||
private const string ReconcileTimerKey = "alarm-reconcile";
|
||||
|
||||
/// <summary>Delay between gRPC reconnection attempts. Settable for tests.</summary>
|
||||
internal static TimeSpan ReconnectDelay { get; set; } = TimeSpan.FromSeconds(5);
|
||||
|
||||
/// <summary>
|
||||
/// How long a freshly-opened gRPC stream must stay up before its retry budget is
|
||||
/// considered recovered (mirrors <see cref="DebugStreamBridgeActor.StabilityWindow"/>).
|
||||
/// Settable for tests.
|
||||
/// </summary>
|
||||
internal static TimeSpan StabilityWindow { get; set; } = TimeSpan.FromSeconds(60);
|
||||
|
||||
private int _retryCount;
|
||||
private bool _useNodeA = true;
|
||||
private bool _stopped;
|
||||
|
||||
/// <summary>
|
||||
/// True once the live gRPC stream has been given up (retry budget exhausted). Reconcile
|
||||
/// snapshots keep serving in the meantime; the next reconcile tick self-heals the stream
|
||||
/// by resetting the retry budget and reopening it, so a sustained site outage does not
|
||||
/// permanently drop the live feed. Actor-thread only.
|
||||
/// </summary>
|
||||
private bool _streamDown;
|
||||
private CancellationTokenSource? _grpcCts;
|
||||
private CancellationTokenSource? _lifetimeCts;
|
||||
|
||||
/// <summary>Current whole-site alarm state, keyed by <see cref="AlarmKey"/>. Actor-thread only.</summary>
|
||||
private readonly Dictionary<string, AlarmStateChanged> _cache = new();
|
||||
|
||||
/// <summary>True once the first seed has completed and been published. Actor-thread only.</summary>
|
||||
private bool _seeded;
|
||||
|
||||
/// <summary>True while a seed/reconcile snapshot fan-out is in flight (deltas buffer). Actor-thread only.</summary>
|
||||
private bool _fanoutInFlight;
|
||||
|
||||
/// <summary>Ordered buffer of live deltas that arrived while a fan-out was in flight. Actor-thread only.</summary>
|
||||
private readonly List<AlarmStateChanged> _buffer = new();
|
||||
|
||||
private const int BufferWarnThreshold = 10_000;
|
||||
private bool _bufferWarned;
|
||||
|
||||
/// <inheritdoc />
|
||||
public ITimerScheduler Timers { get; set; } = null!;
|
||||
|
||||
/// <summary>
|
||||
/// Creates a per-site alarm aggregator.
|
||||
/// </summary>
|
||||
/// <param name="siteIdentifier">Site identifier (for logging / gRPC client keying).</param>
|
||||
/// <param name="correlationId">Correlation id for the site-wide gRPC subscription.</param>
|
||||
/// <param name="seedFn">
|
||||
/// Snapshot fan-out that returns the whole site's current alarm rows (including
|
||||
/// placeholders), best-effort and tolerant of per-instance failure. Re-run on every
|
||||
/// seed and reconcile, so it re-enumerates the site's Enabled instances each call.
|
||||
/// </param>
|
||||
/// <param name="publish">
|
||||
/// Publishes a fresh immutable snapshot of the cache to the owning service, which
|
||||
/// stores it and raises viewer <c>onChanged</c> callbacks. Invoked on the actor thread.
|
||||
/// </param>
|
||||
/// <param name="grpcFactory">Factory caching one gRPC client per (site, endpoint).</param>
|
||||
/// <param name="grpcNodeAAddress">gRPC address of the site's node A.</param>
|
||||
/// <param name="grpcNodeBAddress">gRPC address of the site's node B.</param>
|
||||
/// <param name="reconcileInterval">Periodic reconcile snapshot cadence.</param>
|
||||
public SiteAlarmAggregatorActor(
|
||||
string siteIdentifier,
|
||||
string correlationId,
|
||||
Func<CancellationToken, Task<IReadOnlyList<AlarmStateChanged>>> seedFn,
|
||||
Action<IReadOnlyList<AlarmStateChanged>> publish,
|
||||
SiteStreamGrpcClientFactory grpcFactory,
|
||||
string grpcNodeAAddress,
|
||||
string grpcNodeBAddress,
|
||||
TimeSpan reconcileInterval)
|
||||
{
|
||||
_siteIdentifier = siteIdentifier;
|
||||
_correlationId = correlationId;
|
||||
_seedFn = seedFn;
|
||||
_publish = publish;
|
||||
_grpcFactory = grpcFactory;
|
||||
_grpcNodeAAddress = grpcNodeAAddress;
|
||||
_grpcNodeBAddress = grpcNodeBAddress;
|
||||
_reconcileInterval = reconcileInterval;
|
||||
|
||||
// Live delta from the site-wide alarm stream (marshalled in via Self.Tell).
|
||||
// A received delta must NOT reset the retry budget (a flapping stream that
|
||||
// delivers one delta between failures would otherwise never trip MaxRetries).
|
||||
Receive<AlarmStateChanged>(HandleLiveDelta);
|
||||
|
||||
// A seed/reconcile fan-out completed.
|
||||
Receive<SeedCompleted>(OnSeedCompleted);
|
||||
|
||||
// A seed/reconcile fan-out threw as a whole (individual per-instance faults are
|
||||
// swallowed inside seedFn and degrade to fewer rows, not a whole-fan-out failure).
|
||||
Receive<SeedFailed>(OnSeedFailed);
|
||||
|
||||
// Periodic reconcile tick (and the re-seed kicked after a reconnect).
|
||||
Receive<RunReconcile>(_ => OnReconcileTick());
|
||||
|
||||
// Stream stayed up for StabilityWindow — recover the retry budget.
|
||||
Receive<GrpcAlarmStreamStable>(_ =>
|
||||
{
|
||||
if (_stopped) return;
|
||||
_retryCount = 0;
|
||||
_log.Debug("Site-alarm gRPC stream for {0} stable; retry count reset", _siteIdentifier);
|
||||
});
|
||||
|
||||
// gRPC stream error — flip node + reconnect + re-seed.
|
||||
Receive<GrpcAlarmStreamError>(msg =>
|
||||
{
|
||||
_log.Warning("Site-alarm gRPC stream error for {0}: {1}", _siteIdentifier, msg.Exception.Message);
|
||||
HandleGrpcError();
|
||||
});
|
||||
|
||||
Receive<ReconnectAlarmStream>(_ => OpenGrpcStream());
|
||||
|
||||
// Owning service asks us to stop (last viewer left + linger elapsed).
|
||||
Receive<StopSiteAlarmAggregator>(_ =>
|
||||
{
|
||||
_log.Info("Stopping site-alarm aggregator for {0}", _siteIdentifier);
|
||||
CleanupGrpc();
|
||||
_stopped = true;
|
||||
Context.Stop(Self);
|
||||
});
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PreStart()
|
||||
{
|
||||
_log.Info("Starting site-alarm aggregator for site {0}", _siteIdentifier);
|
||||
// Telemetry: this aggregator is now a running per-site live cache (gauge +1). Balanced
|
||||
// in PostStop, which Akka always runs on termination for any reason.
|
||||
ScadaBridgeTelemetry.LiveAlarmAggregatorStarted();
|
||||
_lifetimeCts = new CancellationTokenSource();
|
||||
|
||||
// Stream-first: open the site-wide alarm stream BEFORE the first seed so deltas
|
||||
// in the seed window are captured (buffered) rather than lost.
|
||||
OpenGrpcStream();
|
||||
|
||||
// Kick the initial seed fan-out.
|
||||
StartFanout(isInitial: true);
|
||||
|
||||
// Periodic reconcile backstop.
|
||||
Timers.StartPeriodicTimer(ReconcileTimerKey, new RunReconcile(), _reconcileInterval, _reconcileInterval);
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
protected override void PostStop()
|
||||
{
|
||||
_grpcCts?.Cancel();
|
||||
_grpcCts?.Dispose();
|
||||
_grpcCts = null;
|
||||
_lifetimeCts?.Cancel();
|
||||
_lifetimeCts?.Dispose();
|
||||
_lifetimeCts = null;
|
||||
// Telemetry: this aggregator is no longer running (gauge -1). Balances PreStart.
|
||||
ScadaBridgeTelemetry.LiveAlarmAggregatorStopped();
|
||||
base.PostStop();
|
||||
}
|
||||
|
||||
// ── Reconcile tick ──────────────────────────────────────────────────────────
|
||||
|
||||
/// <summary>
|
||||
/// Periodic reconcile: always re-run the snapshot fan-out (corrects drift + missed
|
||||
/// deltas), and if the live stream was previously given up, self-heal it by resetting
|
||||
/// the retry budget and reopening — so a sustained outage never permanently kills the
|
||||
/// live feed.
|
||||
/// </summary>
|
||||
private void OnReconcileTick()
|
||||
{
|
||||
if (_stopped) return;
|
||||
StartFanout(isInitial: false);
|
||||
if (_streamDown)
|
||||
{
|
||||
_log.Info("Site-alarm gRPC stream for {0} was down; reopening on reconcile tick", _siteIdentifier);
|
||||
_retryCount = 0;
|
||||
// Telemetry: a reconcile-driven reopen after the stream was given up is a reconnect.
|
||||
ScadaBridgeTelemetry.RecordLiveAlarmStreamReconnect();
|
||||
OpenGrpcStream();
|
||||
}
|
||||
}
|
||||
|
||||
// ── Seed / reconcile fan-out ────────────────────────────────────────────────
|
||||
|
||||
/// <summary>
|
||||
/// Kicks a snapshot fan-out as a background task, marshalling the result back via
|
||||
/// <c>Self.Tell</c>. While in flight, live deltas buffer. A reconcile that arrives
|
||||
/// while a fan-out is already running is skipped (no stacking).
|
||||
/// </summary>
|
||||
private void StartFanout(bool isInitial)
|
||||
{
|
||||
if (_stopped) return;
|
||||
if (_fanoutInFlight)
|
||||
{
|
||||
// Already seeding/reconciling — don't stack a second fan-out.
|
||||
return;
|
||||
}
|
||||
|
||||
_fanoutInFlight = true;
|
||||
var self = Self;
|
||||
var ct = _lifetimeCts?.Token ?? CancellationToken.None;
|
||||
|
||||
Task.Run(async () =>
|
||||
{
|
||||
try
|
||||
{
|
||||
var alarms = await _seedFn(ct);
|
||||
self.Tell(new SeedCompleted(alarms, isInitial));
|
||||
}
|
||||
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
||||
{
|
||||
// Actor stopping — drop silently.
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
self.Tell(new SeedFailed(ex, isInitial));
|
||||
}
|
||||
}, ct);
|
||||
}
|
||||
|
||||
private void OnSeedCompleted(SeedCompleted msg)
|
||||
{
|
||||
if (_stopped) return;
|
||||
|
||||
// Rebuild the cache authoritatively from the fresh snapshot (this is what makes
|
||||
// reconcile able to DROP rows for instances/alarms that disappeared — a merge
|
||||
// could never remove a stale row since the live stream sends no "removed" event).
|
||||
_cache.Clear();
|
||||
foreach (var alarm in msg.Alarms)
|
||||
{
|
||||
var key = AlarmKey(alarm);
|
||||
if (!_cache.TryGetValue(key, out var existing) || alarm.Timestamp >= existing.Timestamp)
|
||||
_cache[key] = alarm;
|
||||
}
|
||||
|
||||
// Flush the deltas buffered during the fan-out, deduped against the seed: a
|
||||
// buffered delta whose key is in the fresh snapshot with an equal-or-newer
|
||||
// timestamp is already reflected → drop; a strictly-newer (or new-key) delta is
|
||||
// applied. Inclusive-on-snapshot boundary matches DebugStreamBridgeActor.
|
||||
FlushBuffer();
|
||||
|
||||
_fanoutInFlight = false;
|
||||
_seeded = true;
|
||||
|
||||
_log.Debug("Site-alarm {0} {1} complete: {2} alarm row(s)",
|
||||
_siteIdentifier, msg.IsInitial ? "seed" : "reconcile", _cache.Count);
|
||||
|
||||
Publish();
|
||||
}
|
||||
|
||||
private void OnSeedFailed(SeedFailed msg)
|
||||
{
|
||||
if (_stopped) return;
|
||||
|
||||
_log.Warning(msg.Exception,
|
||||
"Site-alarm {0} {1} fan-out failed; keeping current cache and relying on the next reconcile",
|
||||
_siteIdentifier, msg.IsInitial ? "seed" : "reconcile");
|
||||
|
||||
// Don't lose deltas captured during the failed window — apply them pass-through
|
||||
// into the (possibly stale/empty) cache. The next reconcile re-seeds authoritatively.
|
||||
_fanoutInFlight = false;
|
||||
FlushBuffer(dedupAgainstSeed: false);
|
||||
|
||||
// Only publish if we already had a seed (so IsLive doesn't flip true on a
|
||||
// failed initial seed — the page keeps its poll fallback until we truly seed).
|
||||
if (_seeded)
|
||||
Publish();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Flushes the pre-fan-out buffer in arrival order. When <paramref name="dedupAgainstSeed"/>
|
||||
/// is true (normal completion) a buffered delta already reflected in the just-installed
|
||||
/// snapshot (same key, timestamp <= cache entry) is dropped; otherwise every buffered
|
||||
/// delta is applied pass-through.
|
||||
/// </summary>
|
||||
private void FlushBuffer(bool dedupAgainstSeed = true)
|
||||
{
|
||||
if (_buffer.Count == 0) return;
|
||||
|
||||
foreach (var delta in _buffer)
|
||||
{
|
||||
if (dedupAgainstSeed)
|
||||
ApplyDelta(delta, requireStrictlyNewer: true);
|
||||
else
|
||||
ApplyDelta(delta, requireStrictlyNewer: false);
|
||||
}
|
||||
_buffer.Clear();
|
||||
}
|
||||
|
||||
// ── Live delta handling ─────────────────────────────────────────────────────
|
||||
|
||||
private void HandleLiveDelta(AlarmStateChanged delta)
|
||||
{
|
||||
if (_stopped) return;
|
||||
|
||||
if (_fanoutInFlight)
|
||||
{
|
||||
_buffer.Add(delta);
|
||||
if (!_bufferWarned && _buffer.Count > BufferWarnThreshold)
|
||||
{
|
||||
_bufferWarned = true;
|
||||
_log.Warning(
|
||||
"Site-alarm pre-seed buffer for {0} exceeded {1} deltas while a fan-out was in flight " +
|
||||
"(deltas retained, not dropped).",
|
||||
_siteIdentifier, BufferWarnThreshold);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
// Pass-through: apply and publish only if the cache actually changed.
|
||||
if (ApplyDelta(delta, requireStrictlyNewer: false))
|
||||
Publish();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Applies one alarm delta to the cache keyed by <see cref="AlarmKey"/>. Returns
|
||||
/// <c>true</c> if the cache changed. A stale (older) delta for an existing key is
|
||||
/// ignored. Never carries a placeholder (the live stream drops those), so a live
|
||||
/// delta for a real alarm can only add/replace its own key — it never touches a
|
||||
/// placeholder row under a different key.
|
||||
/// </summary>
|
||||
private bool ApplyDelta(AlarmStateChanged delta, bool requireStrictlyNewer)
|
||||
{
|
||||
var key = AlarmKey(delta);
|
||||
if (_cache.TryGetValue(key, out var existing))
|
||||
{
|
||||
var newer = requireStrictlyNewer
|
||||
? delta.Timestamp > existing.Timestamp
|
||||
: delta.Timestamp >= existing.Timestamp;
|
||||
if (!newer) return false;
|
||||
}
|
||||
_cache[key] = delta;
|
||||
return true;
|
||||
}
|
||||
|
||||
private void Publish()
|
||||
{
|
||||
// Fresh immutable list — reference swap; readers never see a partial mutation.
|
||||
var snapshot = _cache.Values.ToList();
|
||||
try
|
||||
{
|
||||
_publish(snapshot);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_log.Warning(ex, "Site-alarm publish callback threw for {0}; ignoring", _siteIdentifier);
|
||||
}
|
||||
}
|
||||
|
||||
// ── gRPC stream lifecycle (mirrors DebugStreamBridgeActor) ──────────────────
|
||||
|
||||
private void OpenGrpcStream()
|
||||
{
|
||||
if (_stopped) return;
|
||||
|
||||
_streamDown = false;
|
||||
var endpoint = _useNodeA ? _grpcNodeAAddress : _grpcNodeBAddress;
|
||||
_log.Info("Opening site-alarm gRPC stream for {0} to {1}", _siteIdentifier, endpoint);
|
||||
|
||||
_grpcCts?.Cancel();
|
||||
_grpcCts?.Dispose();
|
||||
_grpcCts = new CancellationTokenSource();
|
||||
|
||||
Timers.StartSingleTimer(StabilityTimerKey, new GrpcAlarmStreamStable(), StabilityWindow);
|
||||
|
||||
var client = _grpcFactory.GetOrCreate(_siteIdentifier, endpoint);
|
||||
var self = Self;
|
||||
var ct = _grpcCts.Token;
|
||||
|
||||
Task.Run(async () =>
|
||||
{
|
||||
await client.SubscribeSiteAsync(
|
||||
_correlationId,
|
||||
alarm => self.Tell(alarm),
|
||||
ex => self.Tell(new GrpcAlarmStreamError(ex)),
|
||||
ct);
|
||||
}, ct);
|
||||
}
|
||||
|
||||
private void HandleGrpcError()
|
||||
{
|
||||
if (_stopped) return;
|
||||
|
||||
// Stream failed before the stability window — retry budget NOT recovered.
|
||||
Timers.Cancel(StabilityTimerKey);
|
||||
|
||||
_retryCount++;
|
||||
|
||||
if (_retryCount > MaxRetries)
|
||||
{
|
||||
// Give up the stream, but do NOT stop the aggregator: the periodic reconcile
|
||||
// still refreshes the cache from ClusterClient snapshots, so the page keeps a
|
||||
// (slower) live-ish view rather than going dark. A later reconcile-triggered
|
||||
// reconnect is not attempted here; the stream is simply left down.
|
||||
_log.Error("Site-alarm gRPC stream for {0} exceeded max retries ({1}); leaving stream down, " +
|
||||
"reconcile snapshots continue and the next reconcile tick will retry the stream",
|
||||
_siteIdentifier, MaxRetries);
|
||||
_streamDown = true;
|
||||
CleanupGrpc();
|
||||
return;
|
||||
}
|
||||
|
||||
// Unsubscribe the failed stream on the previous endpoint (TryGet, never
|
||||
// GetOrCreate) so the site releases its relay actor instead of leaving a zombie.
|
||||
var previousEndpoint = _useNodeA ? _grpcNodeAAddress : _grpcNodeBAddress;
|
||||
_grpcFactory.TryGet(_siteIdentifier, previousEndpoint)?.Unsubscribe(_correlationId);
|
||||
|
||||
// Flip to the other node.
|
||||
_useNodeA = !_useNodeA;
|
||||
|
||||
// Telemetry: a NodeA↔NodeB failover flip is a reconnect + re-seed.
|
||||
ScadaBridgeTelemetry.RecordLiveAlarmStreamReconnect();
|
||||
|
||||
// A failover flip must RE-SEED (never silently serve stale) — kick a reconcile
|
||||
// fan-out alongside the reconnect. Buffering during the fan-out keeps the new
|
||||
// stream's deltas coherent with the fresh snapshot.
|
||||
StartFanout(isInitial: false);
|
||||
|
||||
if (_retryCount == 1)
|
||||
Self.Tell(new ReconnectAlarmStream());
|
||||
else
|
||||
Timers.StartSingleTimer(ReconnectTimerKey, new ReconnectAlarmStream(), ReconnectDelay);
|
||||
}
|
||||
|
||||
private void CleanupGrpc()
|
||||
{
|
||||
_grpcCts?.Cancel();
|
||||
_grpcCts?.Dispose();
|
||||
_grpcCts = null;
|
||||
|
||||
var endpoint = _useNodeA ? _grpcNodeAAddress : _grpcNodeBAddress;
|
||||
_grpcFactory.TryGet(_siteIdentifier, endpoint)?.Unsubscribe(_correlationId);
|
||||
}
|
||||
|
||||
// ── Dedup key (copied from DebugStreamBridgeActor.AlarmKey with attribution) ──
|
||||
|
||||
/// <summary>
|
||||
/// NUL delimiter so distinct identities never collide on a shared boundary. Cannot
|
||||
/// appear in an instance/alarm name. Mirrors <see cref="DebugStreamBridgeActor"/>.
|
||||
/// </summary>
|
||||
private const char KeyDelimiter = '\u0000';
|
||||
|
||||
/// <summary>
|
||||
/// Per-alarm dedup identity = (InstanceUniqueName, AlarmName, SourceReference) —
|
||||
/// identical to <c>DebugStreamBridgeActor.AlarmKey</c> so native per-condition alarms
|
||||
/// sharing an AlarmName but differing by source reference are not conflated. Each
|
||||
/// nullable component is guarded to prevent silent null/empty key collisions.
|
||||
/// </summary>
|
||||
private static string AlarmKey(AlarmStateChanged a) =>
|
||||
string.Concat(
|
||||
a.InstanceUniqueName ?? string.Empty, KeyDelimiter,
|
||||
a.AlarmName ?? string.Empty, KeyDelimiter,
|
||||
a.SourceReference ?? string.Empty);
|
||||
}
|
||||
|
||||
/// <summary>Message asking a <see cref="SiteAlarmAggregatorActor"/> to stop (last viewer left).</summary>
|
||||
public sealed record StopSiteAlarmAggregator;
|
||||
|
||||
/// <summary>Internal: a seed/reconcile snapshot fan-out completed with the whole-site alarm rows.</summary>
|
||||
internal sealed record SeedCompleted(IReadOnlyList<AlarmStateChanged> Alarms, bool IsInitial);
|
||||
|
||||
/// <summary>Internal: a seed/reconcile snapshot fan-out threw as a whole.</summary>
|
||||
internal sealed record SeedFailed(Exception Exception, bool IsInitial);
|
||||
|
||||
/// <summary>Internal: periodic reconcile tick (and the re-seed kicked after a reconnect).</summary>
|
||||
internal sealed record RunReconcile;
|
||||
|
||||
/// <summary>Internal: site-alarm gRPC stream error occurred.</summary>
|
||||
internal sealed record GrpcAlarmStreamError(Exception Exception);
|
||||
|
||||
/// <summary>Internal: reconnect the site-alarm gRPC stream (flip node).</summary>
|
||||
internal sealed record ReconnectAlarmStream;
|
||||
|
||||
/// <summary>Internal: the current site-alarm gRPC stream has stayed up long enough to recover the retry budget.</summary>
|
||||
internal sealed record GrpcAlarmStreamStable;
|
||||
@@ -88,4 +88,38 @@ public class CommunicationOptions
|
||||
/// that are deployed once and never re-deployed. Default 1 hour ≫ <see cref="PendingDeploymentTtl"/>.
|
||||
/// </summary>
|
||||
public TimeSpan PendingDeploymentPurgeInterval { get; set; } = TimeSpan.FromHours(1);
|
||||
|
||||
// ── Aggregated live alarm cache (plan #10, Task 4) ───────────────────────────
|
||||
// Introduced by Task 4 with sane defaults; Task 6 formalizes eager validation
|
||||
// (positive interval/linger, positive concurrency/subscriber cap) and telemetry.
|
||||
|
||||
/// <summary>
|
||||
/// How long the shared per-site live alarm aggregator keeps its gRPC stream +
|
||||
/// in-memory cache warm after its LAST Alarm Summary viewer leaves, before it is
|
||||
/// torn down. A short linger avoids re-seed thrash when an operator navigates away
|
||||
/// and straight back. Default 30s.
|
||||
/// </summary>
|
||||
public TimeSpan LiveAlarmCacheLinger { get; set; } = TimeSpan.FromSeconds(30);
|
||||
|
||||
/// <summary>
|
||||
/// Cadence of the per-site aggregator's periodic reconcile snapshot fan-out. The
|
||||
/// backstop that corrects instance-set drift (enable/disable/deploy/delete) and any
|
||||
/// alarm delta missed by the live stream, since the aggregated stream never carries
|
||||
/// an explicit "removed" event. Default 60s.
|
||||
/// </summary>
|
||||
public TimeSpan LiveAlarmCacheReconcileInterval { get; set; } = TimeSpan.FromSeconds(60);
|
||||
|
||||
/// <summary>
|
||||
/// Max concurrent per-instance snapshot fetches during a live-cache seed/reconcile
|
||||
/// fan-out — the aggregated analogue of the Alarm Summary poll's
|
||||
/// <c>MaxConcurrentFetches</c>. Default 8.
|
||||
/// </summary>
|
||||
public int LiveAlarmCacheSeedConcurrency { get; set; } = 8;
|
||||
|
||||
/// <summary>
|
||||
/// Upper bound on concurrent Alarm Summary viewers sharing one site's aggregator.
|
||||
/// A defensive cap only — one gRPC stream per site is shared regardless of viewer
|
||||
/// count; this just bounds the subscriber list. Default 200.
|
||||
/// </summary>
|
||||
public int LiveAlarmCacheMaxSubscribersPerSite { get; set; } = 200;
|
||||
}
|
||||
|
||||
@@ -65,5 +65,25 @@ public sealed class CommunicationOptionsValidator : OptionsValidatorBase<Communi
|
||||
|
||||
builder.RequireThat(options.GrpcMaxConcurrentStreams > 0,
|
||||
$"Communication:GrpcMaxConcurrentStreams must be positive (was {options.GrpcMaxConcurrentStreams}).");
|
||||
|
||||
// ── Aggregated live alarm cache (plan #10, Task 6) ───────────────────────
|
||||
// Linger drives a Timer dueTime; TimeSpan.Zero is valid (stop the aggregator
|
||||
// immediately when the last viewer leaves), only a negative value is invalid.
|
||||
builder.RequireThat(options.LiveAlarmCacheLinger >= TimeSpan.Zero,
|
||||
$"Communication:LiveAlarmCacheLinger must be zero or a positive duration (was {options.LiveAlarmCacheLinger}).");
|
||||
|
||||
// Reconcile interval is a periodic timer cadence AND the start-retry cadence; a
|
||||
// zero/negative value would spin or never fire.
|
||||
builder.RequireThat(options.LiveAlarmCacheReconcileInterval > TimeSpan.Zero,
|
||||
$"Communication:LiveAlarmCacheReconcileInterval must be a positive duration (was {options.LiveAlarmCacheReconcileInterval}).");
|
||||
|
||||
// Seed fan-out concurrency gates a SemaphoreSlim; at least 1, capped so a
|
||||
// fat-fingered config can't open an unbounded burst of cross-cluster Asks.
|
||||
builder.RequireThat(options.LiveAlarmCacheSeedConcurrency is >= 1 and <= 64,
|
||||
$"Communication:LiveAlarmCacheSeedConcurrency must be between 1 and 64 (was {options.LiveAlarmCacheSeedConcurrency}).");
|
||||
|
||||
// Per-site viewer cap must admit at least one viewer, else the page could never go live.
|
||||
builder.RequireThat(options.LiveAlarmCacheMaxSubscribersPerSite >= 1,
|
||||
$"Communication:LiveAlarmCacheMaxSubscribersPerSite must be at least 1 (was {options.LiveAlarmCacheMaxSubscribersPerSite}).");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -17,6 +17,16 @@ public interface ISiteStreamSubscriber
|
||||
/// <returns>A subscription ID that can be used for unsubscription.</returns>
|
||||
string Subscribe(string instanceName, IActorRef subscriber);
|
||||
|
||||
/// <summary>
|
||||
/// Subscribes an actor to receive ALARM events for ALL instances on the site
|
||||
/// (no per-instance filter). Only <c>AlarmStateChanged</c> events are
|
||||
/// forwarded; attribute-value events are dropped. Backs the site-wide
|
||||
/// <c>SubscribeSite</c> gRPC stream used by the aggregated Alarm Summary.
|
||||
/// </summary>
|
||||
/// <param name="subscriber">The actor reference that will receive alarm stream event messages.</param>
|
||||
/// <returns>A subscription ID that can be used for unsubscription.</returns>
|
||||
string SubscribeSiteAlarms(IActorRef subscriber);
|
||||
|
||||
/// <summary>
|
||||
/// Removes all subscriptions for the given actor.
|
||||
/// </summary>
|
||||
|
||||
@@ -200,6 +200,74 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Opens the site-wide, <b>alarm-only</b> server-streaming subscription
|
||||
/// (<c>SubscribeSite</c>) for a whole site rather than a single instance. This is
|
||||
/// the central per-site feed that backs the aggregated Alarm Summary live cache
|
||||
/// (plan #10): the site runtime's <c>SubscribeSiteAlarms</c> hub drops the
|
||||
/// per-instance filter and carries only <see cref="AlarmStateChanged"/> events for
|
||||
/// <em>all</em> instances on the site (attributes are deliberately excluded — the
|
||||
/// summary never shows them and they are far higher-volume).
|
||||
/// <para>
|
||||
/// The callback is deliberately <b>typed</b> as <see cref="AlarmStateChanged"/>
|
||||
/// rather than the generic <c>Action<object></c> used by
|
||||
/// <see cref="SubscribeAsync"/>: this stream is alarm-only by contract, so Task 4's
|
||||
/// per-site cache consumes an alarm delta directly with no downstream type test.
|
||||
/// The mapping is still shared via <see cref="ConvertToDomainEvent"/>; a non-alarm
|
||||
/// event (which should never appear on this stream) is defensively ignored rather
|
||||
/// than delivered or thrown.
|
||||
/// </para>
|
||||
/// This is a long-running async method; the caller launches it as a background task.
|
||||
/// Error handling, cancellation and cleanup mirror <see cref="SubscribeAsync"/>.
|
||||
/// </summary>
|
||||
/// <param name="correlationId">Unique identifier for this subscription.</param>
|
||||
/// <param name="onAlarmEvent">Callback invoked for each alarm delta received from the site-wide stream.</param>
|
||||
/// <param name="onError">Callback invoked when the subscription encounters an error.</param>
|
||||
/// <param name="ct">Cancellation token to stop the subscription.</param>
|
||||
/// <returns>A task that represents the asynchronous operation.</returns>
|
||||
public virtual async Task SubscribeSiteAsync(
|
||||
string correlationId,
|
||||
Action<AlarmStateChanged> onAlarmEvent,
|
||||
Action<Exception> onError,
|
||||
CancellationToken ct)
|
||||
{
|
||||
if (_client is null)
|
||||
throw new InvalidOperationException("Cannot subscribe on a test-only client.");
|
||||
|
||||
var cts = CancellationTokenSource.CreateLinkedTokenSource(ct);
|
||||
RegisterSubscription(correlationId, cts);
|
||||
|
||||
var request = new SiteStreamRequest
|
||||
{
|
||||
CorrelationId = correlationId
|
||||
};
|
||||
|
||||
try
|
||||
{
|
||||
using var call = _client.SubscribeSite(request, cancellationToken: cts.Token);
|
||||
|
||||
await foreach (var evt in call.ResponseStream.ReadAllAsync(cts.Token))
|
||||
{
|
||||
// Site-wide stream is alarm-only by contract; defensively ignore anything else.
|
||||
if (ConvertToAlarmEvent(evt) is { } alarm)
|
||||
onAlarmEvent(alarm);
|
||||
}
|
||||
}
|
||||
catch (RpcException ex) when (ex.StatusCode == StatusCode.Cancelled)
|
||||
{
|
||||
// Normal cancellation — not an error
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
onError(ex);
|
||||
}
|
||||
finally
|
||||
{
|
||||
// Remove only our own entry -- a racing reconnect may already own the slot.
|
||||
RemoveSubscription(correlationId, cts);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Cancels an active subscription by correlation ID.
|
||||
/// </summary>
|
||||
@@ -260,6 +328,17 @@ public class SiteStreamGrpcClient : IAsyncDisposable, IDisposable
|
||||
_ => null
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// Maps a proto <see cref="SiteStreamEvent"/> to a domain <see cref="AlarmStateChanged"/>,
|
||||
/// returning <c>null</c> for any non-alarm event. Used by <see cref="SubscribeSiteAsync"/>
|
||||
/// to enforce the alarm-only contract of the site-wide stream without duplicating the
|
||||
/// enrichment mapping in <see cref="ConvertToDomainEvent"/>. Internal for testability.
|
||||
/// </summary>
|
||||
/// <param name="evt">The protobuf site stream event to convert.</param>
|
||||
/// <returns>The mapped <see cref="AlarmStateChanged"/>, or <c>null</c> if the event is not an alarm.</returns>
|
||||
internal static AlarmStateChanged? ConvertToAlarmEvent(SiteStreamEvent evt) =>
|
||||
ConvertToDomainEvent(evt) as AlarmStateChanged;
|
||||
|
||||
/// <summary>Parses the wire "kind" string back to <see cref="AlarmKind"/>; defaults to Computed.</summary>
|
||||
/// <param name="kind">The wire "kind" string from the gRPC payload; null or unrecognised defaults to <see cref="AlarmKind.Computed"/>.</param>
|
||||
/// <returns>The parsed <see cref="AlarmKind"/>, or <see cref="AlarmKind.Computed"/> when the value is null or unrecognised.</returns>
|
||||
|
||||
@@ -210,10 +210,52 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
internal TimeSpan MaxStreamLifetime => _maxStreamLifetime;
|
||||
|
||||
/// <inheritdoc />
|
||||
public override async Task SubscribeInstance(
|
||||
public override Task SubscribeInstance(
|
||||
InstanceStreamRequest request,
|
||||
IServerStreamWriter<SiteStreamEvent> responseStream,
|
||||
ServerCallContext context)
|
||||
=> RunSubscriptionStreamAsync(
|
||||
request.CorrelationId,
|
||||
responseStream,
|
||||
context,
|
||||
relay => _streamSubscriber.Subscribe(request.InstanceUniqueName, relay),
|
||||
request.InstanceUniqueName);
|
||||
|
||||
/// <inheritdoc />
|
||||
public override Task SubscribeSite(
|
||||
SiteStreamRequest request,
|
||||
IServerStreamWriter<SiteStreamEvent> responseStream,
|
||||
ServerCallContext context)
|
||||
=> RunSubscriptionStreamAsync(
|
||||
request.CorrelationId,
|
||||
responseStream,
|
||||
context,
|
||||
// Site-wide, alarm-only: no per-instance filter. StreamRelayActor
|
||||
// already drops IsConfiguredPlaceholder rows and maps only the
|
||||
// enriched AlarmStateUpdate, so it is reused unchanged.
|
||||
_streamSubscriber.SubscribeSiteAlarms,
|
||||
"site-wide alarms");
|
||||
|
||||
/// <summary>
|
||||
/// Shared streaming pipeline behind <see cref="SubscribeInstance"/> and
|
||||
/// <see cref="SubscribeSite"/>: readiness/shutdown guards, correlation-id
|
||||
/// validation, duplicate replacement, the concurrency cap, the bounded
|
||||
/// <c>DropOldest</c> channel, the relay actor, connection telemetry, and the
|
||||
/// guaranteed cleanup. The only variation between the two RPCs is how the
|
||||
/// relay actor is subscribed to the site broadcast hub — supplied by
|
||||
/// <paramref name="subscribe"/> — and the log/telemetry description.
|
||||
/// </summary>
|
||||
/// <param name="correlationId">The client-supplied correlation id (also the actor-name/dedup key).</param>
|
||||
/// <param name="responseStream">The gRPC response stream to pump events into.</param>
|
||||
/// <param name="context">The server call context (carries the client cancellation token).</param>
|
||||
/// <param name="subscribe">Subscribes the relay actor to the hub, returning a subscription id.</param>
|
||||
/// <param name="description">Human-readable subscription description for logging.</param>
|
||||
private async Task RunSubscriptionStreamAsync(
|
||||
string correlationId,
|
||||
IServerStreamWriter<SiteStreamEvent> responseStream,
|
||||
ServerCallContext context,
|
||||
Func<IActorRef, string> subscribe,
|
||||
string description)
|
||||
{
|
||||
if (!_ready)
|
||||
throw new RpcException(new GrpcStatus(StatusCode.Unavailable, "Server not ready"));
|
||||
@@ -228,15 +270,15 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
// have a restricted character set — a id containing '/', whitespace, or other
|
||||
// disallowed characters would make ActorOf throw InvalidActorNameException,
|
||||
// escaping as an unhandled RPC fault. Reject unsafe ids cleanly up front.
|
||||
if (string.IsNullOrEmpty(request.CorrelationId) ||
|
||||
!ActorPath.IsValidPathElement(request.CorrelationId))
|
||||
if (string.IsNullOrEmpty(correlationId) ||
|
||||
!ActorPath.IsValidPathElement(correlationId))
|
||||
{
|
||||
throw new RpcException(new GrpcStatus(
|
||||
StatusCode.InvalidArgument, "correlation_id is missing or not a valid identifier"));
|
||||
}
|
||||
|
||||
// Duplicate prevention -- cancel existing stream for this correlationId
|
||||
if (_activeStreams.TryRemove(request.CorrelationId, out var existingEntry))
|
||||
if (_activeStreams.TryRemove(correlationId, out var existingEntry))
|
||||
{
|
||||
existingEntry.Cts.Cancel();
|
||||
existingEntry.Cts.Dispose();
|
||||
@@ -256,10 +298,9 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
if (_maxStreamLifetime > TimeSpan.Zero && _maxStreamLifetime != Timeout.InfiniteTimeSpan)
|
||||
streamCts.CancelAfter(_maxStreamLifetime);
|
||||
var entry = new StreamEntry(streamCts);
|
||||
_activeStreams[request.CorrelationId] = entry;
|
||||
_activeStreams[correlationId] = entry;
|
||||
|
||||
long dropped = 0;
|
||||
var correlationId = request.CorrelationId;
|
||||
var channel = Channel.CreateBounded<SiteStreamEvent>(
|
||||
new BoundedChannelOptions(1000) { FullMode = BoundedChannelFullMode.DropOldest },
|
||||
_ =>
|
||||
@@ -275,8 +316,8 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
|
||||
var actorSeq = Interlocked.Increment(ref _actorCounter);
|
||||
var relayActor = _actorSystem!.ActorOf(
|
||||
Props.Create(typeof(Actors.StreamRelayActor), request.CorrelationId, channel.Writer),
|
||||
$"stream-relay-{request.CorrelationId}-{actorSeq}");
|
||||
Props.Create(typeof(Actors.StreamRelayActor), correlationId, channel.Writer),
|
||||
$"stream-relay-{correlationId}-{actorSeq}");
|
||||
|
||||
// The previous code called _streamSubscriber.Subscribe
|
||||
// OUTSIDE the try block that owns relay-actor cleanup. If Subscribe threw
|
||||
@@ -289,23 +330,23 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
string subscriptionId;
|
||||
try
|
||||
{
|
||||
subscriptionId = _streamSubscriber.Subscribe(request.InstanceUniqueName, relayActor);
|
||||
subscriptionId = subscribe(relayActor);
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex,
|
||||
"Subscribe failed for {Instance} (correlation {CorrelationId}); cleaning up relay actor.",
|
||||
request.InstanceUniqueName, request.CorrelationId);
|
||||
"Subscribe failed for {Description} (correlation {CorrelationId}); cleaning up relay actor.",
|
||||
description, correlationId);
|
||||
_actorSystem!.Stop(relayActor);
|
||||
channel.Writer.TryComplete();
|
||||
_activeStreams.TryRemove(
|
||||
new KeyValuePair<string, StreamEntry>(request.CorrelationId, entry));
|
||||
new KeyValuePair<string, StreamEntry>(correlationId, entry));
|
||||
throw;
|
||||
}
|
||||
|
||||
_logger.LogInformation(
|
||||
"Stream {CorrelationId} started for {Instance} (subscription {SubscriptionId})",
|
||||
request.CorrelationId, request.InstanceUniqueName, subscriptionId);
|
||||
"Stream {CorrelationId} started for {Description} (subscription {SubscriptionId})",
|
||||
correlationId, description, subscriptionId);
|
||||
|
||||
// Telemetry follow-on: the connection is now fully established (Subscribe
|
||||
// succeeded, so no leak via the catch above). Count it up here and balance
|
||||
@@ -335,11 +376,11 @@ public class SiteStreamGrpcServer : SiteStreamService.SiteStreamServiceBase
|
||||
|
||||
// Only remove our own entry -- a replacement stream may have already taken the slot
|
||||
_activeStreams.TryRemove(
|
||||
new KeyValuePair<string, StreamEntry>(request.CorrelationId, entry));
|
||||
new KeyValuePair<string, StreamEntry>(correlationId, entry));
|
||||
|
||||
_logger.LogInformation(
|
||||
"Stream {CorrelationId} for {Instance} ended",
|
||||
request.CorrelationId, request.InstanceUniqueName);
|
||||
"Stream {CorrelationId} for {Description} ended",
|
||||
correlationId, description);
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -0,0 +1,62 @@
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Communication;
|
||||
|
||||
/// <summary>
|
||||
/// Central, transient, in-memory per-site live alarm cache (plan #10, Task 4).
|
||||
/// A DI singleton shared by every Alarm Summary Blazor circuit on the active central
|
||||
/// node. For each site with at least one active viewer it runs ONE shared per-site
|
||||
/// aggregator that seeds from the existing debug-snapshot fan-out and stays warm on a
|
||||
/// single site-wide, alarm-only gRPC stream (<c>SubscribeSite</c>), so the page
|
||||
/// reflects alarm transitions in near-real-time instead of re-polling every 15s.
|
||||
/// <para>
|
||||
/// <b>Hard constraint (locked <c>[PERM]</c>):</b> there is NO persisted central alarm
|
||||
/// store. This cache is purely in-memory and lives only on the active central node —
|
||||
/// no EF entity/table/migration backs it. On a NodeA↔NodeB failover the new active
|
||||
/// node re-seeds from scratch.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Reference-counted lifecycle:</b> the first <see cref="Subscribe"/> for a site
|
||||
/// starts its aggregator (open stream + snapshot seed); the last subscriber leaving
|
||||
/// stops it after a short linger (see
|
||||
/// <see cref="CommunicationOptions.LiveAlarmCacheLinger"/>) to avoid re-seed thrash.
|
||||
/// One gRPC stream per <em>site</em>, not per browser circuit.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public interface ISiteAlarmLiveCache
|
||||
{
|
||||
/// <summary>
|
||||
/// Registers a viewer for a site's live alarm feed and returns an
|
||||
/// <see cref="IDisposable"/> that unregisters it. The first subscriber for a site
|
||||
/// starts the shared aggregator; the last subscriber's <see cref="IDisposable.Dispose"/>
|
||||
/// schedules a linger-delayed stop. <paramref name="onChanged"/> is raised (on the
|
||||
/// aggregator's thread) after each applied change — the handler must not block and
|
||||
/// should marshal any UI work onto its own dispatcher (mirrors
|
||||
/// <c>IDeploymentStatusNotifier</c>). Disposing the returned handle is idempotent.
|
||||
/// </summary>
|
||||
/// <param name="siteId">The numeric site id whose alarms the viewer wants.</param>
|
||||
/// <param name="onChanged">Callback invoked whenever the site's cached alarm set changes.</param>
|
||||
/// <returns>A disposable that unregisters this viewer (idempotent).</returns>
|
||||
IDisposable Subscribe(int siteId, Action onChanged);
|
||||
|
||||
/// <summary>
|
||||
/// Returns the current in-memory alarm snapshot for a site — exactly what the page
|
||||
/// should render. Empty when the site has no aggregator or has not yet completed its
|
||||
/// first seed. Safe to call from Blazor render threads: it returns an immutable
|
||||
/// point-in-time list (the aggregator publishes fresh immutable lists; readers never
|
||||
/// see a partially-mutated collection).
|
||||
/// </summary>
|
||||
/// <param name="siteId">The numeric site id.</param>
|
||||
/// <returns>The current cached alarms (possibly empty), never <c>null</c>.</returns>
|
||||
IReadOnlyList<AlarmStateChanged> GetCurrentAlarms(int siteId);
|
||||
|
||||
/// <summary>
|
||||
/// Whether the site's aggregator has completed its initial seed and is serving live
|
||||
/// state. The page can use this to decide whether to keep its 15s poll fallback
|
||||
/// (Task 5): <c>false</c> means "not live yet — keep polling"; <c>true</c> means the
|
||||
/// cache is authoritative.
|
||||
/// </summary>
|
||||
/// <param name="siteId">The numeric site id.</param>
|
||||
/// <returns><c>true</c> once the site's aggregator has seeded and published at least once.</returns>
|
||||
bool IsLive(int siteId);
|
||||
}
|
||||
@@ -7,6 +7,10 @@ import "google/protobuf/wrappers.proto"; // Int32Value
|
||||
|
||||
service SiteStreamService {
|
||||
rpc SubscribeInstance(InstanceStreamRequest) returns (stream SiteStreamEvent);
|
||||
// Site-wide, alarm-only live stream (aggregated Alarm Summary): every
|
||||
// AlarmStateChanged for ALL instances on the site, no per-instance filter.
|
||||
// Attribute updates are never carried on this stream.
|
||||
rpc SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent);
|
||||
rpc IngestAuditEvents(AuditEventBatch) returns (IngestAck);
|
||||
rpc IngestCachedTelemetry(CachedTelemetryBatch) returns (IngestAck);
|
||||
rpc PullAuditEvents(PullAuditEventsRequest) returns (PullAuditEventsResponse);
|
||||
@@ -18,6 +22,13 @@ message InstanceStreamRequest {
|
||||
string instance_unique_name = 2;
|
||||
}
|
||||
|
||||
// Request for the site-wide, alarm-only SubscribeSite stream. Unlike
|
||||
// InstanceStreamRequest there is NO instance filter — the stream carries alarm
|
||||
// transitions for every instance on the site.
|
||||
message SiteStreamRequest {
|
||||
string correlation_id = 1;
|
||||
}
|
||||
|
||||
message SiteStreamEvent {
|
||||
string correlation_id = 1;
|
||||
oneof event {
|
||||
|
||||
@@ -22,6 +22,13 @@ public static class ServiceCollectionExtensions
|
||||
services.AddSingleton<SiteStreamGrpcClientFactory>();
|
||||
services.AddSingleton<DebugStreamService>();
|
||||
|
||||
// Aggregated live alarm cache (plan #10, Task 4): transient, in-memory, shared
|
||||
// per-site aggregator backing the operator Alarm Summary page. Singleton so every
|
||||
// Blazor circuit shares one aggregator per site; started/stopped by viewer
|
||||
// reference-count. No persistence (locked [PERM]: no central alarm store).
|
||||
services.AddSingleton<SiteAlarmLiveCacheService>();
|
||||
services.AddSingleton<ISiteAlarmLiveCache>(sp => sp.GetRequiredService<SiteAlarmLiveCacheService>());
|
||||
|
||||
// Startup reconciliation handler — scoped (holds scoped repositories), resolved
|
||||
// per-request by CentralCommunicationActor inside a DI scope. Harmless on site
|
||||
// hosts: only the central actor ever resolves it.
|
||||
|
||||
@@ -0,0 +1,489 @@
|
||||
using System.Collections.Concurrent;
|
||||
using Akka.Actor;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Options;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.DebugView;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Communication;
|
||||
|
||||
/// <summary>
|
||||
/// Default <see cref="ISiteAlarmLiveCache"/> — the DI singleton that owns the shared,
|
||||
/// reference-counted, transient per-site live alarm cache (plan #10, Task 4). For each
|
||||
/// site with active Alarm Summary viewers it runs ONE
|
||||
/// <see cref="SiteAlarmAggregatorActor"/>, seeded by the existing debug-snapshot fan-out
|
||||
/// (<see cref="CommunicationService.RequestDebugSnapshotAsync"/>) and kept warm on a
|
||||
/// single site-wide, alarm-only gRPC stream. The actor holds the state and enforces
|
||||
/// seed-then-stream ordering, dedup, placeholder coherence, failover re-seed and periodic
|
||||
/// reconcile; this service is the DI façade: viewer reference-counting, a linger-delayed
|
||||
/// last-out stop, the seed fan-out closure, and the published-snapshot store the page reads.
|
||||
/// <para>
|
||||
/// <b>No persistence</b> (locked <c>[PERM]</c>): nothing here writes to the database — the
|
||||
/// cache lives only in the actor's memory on the active central node.
|
||||
/// </para>
|
||||
/// </summary>
|
||||
public sealed class SiteAlarmLiveCacheService : ISiteAlarmLiveCache
|
||||
{
|
||||
private static readonly IReadOnlyList<AlarmStateChanged> Empty = Array.Empty<AlarmStateChanged>();
|
||||
|
||||
private readonly IServiceProvider _serviceProvider;
|
||||
private readonly CommunicationService _communicationService;
|
||||
private readonly SiteStreamGrpcClientFactory _grpcFactory;
|
||||
private readonly CommunicationOptions _options;
|
||||
private readonly ILogger<SiteAlarmLiveCacheService> _logger;
|
||||
|
||||
private readonly object _lock = new();
|
||||
private readonly Dictionary<int, SiteEntry> _sites = new();
|
||||
private ActorSystem? _actorSystem;
|
||||
|
||||
/// <summary>
|
||||
/// Initializes the live alarm cache service.
|
||||
/// </summary>
|
||||
/// <param name="serviceProvider">Root provider; scopes are created per snapshot fan-out to resolve repositories.</param>
|
||||
/// <param name="communicationService">Central-side comms used to Ask each instance for a debug snapshot (the seed).</param>
|
||||
/// <param name="grpcFactory">Factory for the per-site site-wide alarm gRPC client.</param>
|
||||
/// <param name="options">Communication options (linger, reconcile interval, seed concurrency, subscriber cap).</param>
|
||||
/// <param name="logger">Logger.</param>
|
||||
public SiteAlarmLiveCacheService(
|
||||
IServiceProvider serviceProvider,
|
||||
CommunicationService communicationService,
|
||||
SiteStreamGrpcClientFactory grpcFactory,
|
||||
IOptions<CommunicationOptions> options,
|
||||
ILogger<SiteAlarmLiveCacheService> logger)
|
||||
{
|
||||
_serviceProvider = serviceProvider;
|
||||
_communicationService = communicationService;
|
||||
_grpcFactory = grpcFactory;
|
||||
_options = options.Value;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Supplies the <see cref="ActorSystem"/> once it exists (called from
|
||||
/// <c>AkkaHostedService</c> on the active central node, mirroring
|
||||
/// <see cref="DebugStreamService.SetActorSystem"/>). Until set, <see cref="Subscribe"/>
|
||||
/// registers viewers but cannot start an aggregator, so the page keeps its poll fallback.
|
||||
/// </summary>
|
||||
/// <param name="actorSystem">The actor system to host per-site aggregators.</param>
|
||||
public void SetActorSystem(ActorSystem actorSystem) => _actorSystem = actorSystem;
|
||||
|
||||
/// <inheritdoc />
|
||||
public IDisposable Subscribe(int siteId, Action onChanged)
|
||||
{
|
||||
ArgumentNullException.ThrowIfNull(onChanged);
|
||||
|
||||
var subscription = new Subscription(this, siteId, onChanged);
|
||||
bool startNeeded;
|
||||
SiteEntry entry;
|
||||
|
||||
lock (_lock)
|
||||
{
|
||||
if (!_sites.TryGetValue(siteId, out entry!))
|
||||
{
|
||||
entry = new SiteEntry(siteId);
|
||||
_sites[siteId] = entry;
|
||||
}
|
||||
|
||||
// A pending linger stop is now moot — a viewer arrived. Invalidate it.
|
||||
entry.CancelLinger();
|
||||
|
||||
// Enforce the per-site viewer cap — fail SAFE: reject the excess viewer with a
|
||||
// no-op disposable rather than throwing (this runs on a Blazor render path) or
|
||||
// growing the subscriber list unbounded. The shared aggregator keeps serving the
|
||||
// already-registered viewers; the rejected circuit just keeps its 15s poll
|
||||
// fallback. Reaching the cap almost always means a Dispose leak or a runaway page.
|
||||
if (entry.Subscribers.Count >= _options.LiveAlarmCacheMaxSubscribersPerSite)
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Site {SiteId} live alarm cache is at its viewer cap ({Cap}); rejecting the new " +
|
||||
"viewer (it will keep polling). This indicates a subscription leak or a very busy page.",
|
||||
siteId, _options.LiveAlarmCacheMaxSubscribersPerSite);
|
||||
return NoOpSubscription.Instance;
|
||||
}
|
||||
|
||||
entry.Subscribers.Add(subscription);
|
||||
|
||||
// Start the aggregator on the first viewer (or if a prior start failed and left
|
||||
// no actor). Guarded by Starting so concurrent first-viewers don't double-start.
|
||||
startNeeded = entry.Actor is null && !entry.Starting;
|
||||
if (startNeeded)
|
||||
entry.Starting = true;
|
||||
}
|
||||
|
||||
if (startNeeded)
|
||||
_ = StartAggregatorAsync(entry);
|
||||
|
||||
return subscription;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public IReadOnlyList<AlarmStateChanged> GetCurrentAlarms(int siteId)
|
||||
{
|
||||
lock (_lock)
|
||||
return _sites.TryGetValue(siteId, out var entry) ? entry.Current : Empty;
|
||||
}
|
||||
|
||||
/// <inheritdoc />
|
||||
public bool IsLive(int siteId)
|
||||
{
|
||||
lock (_lock)
|
||||
return _sites.TryGetValue(siteId, out var entry) && entry.HasPublished;
|
||||
}
|
||||
|
||||
// ── Subscriber teardown ─────────────────────────────────────────────────────
|
||||
|
||||
private void Unsubscribe(Subscription subscription)
|
||||
{
|
||||
lock (_lock)
|
||||
{
|
||||
if (!_sites.TryGetValue(subscription.SiteId, out var entry))
|
||||
return;
|
||||
|
||||
if (!entry.Subscribers.Remove(subscription))
|
||||
return; // already removed (idempotent Dispose)
|
||||
|
||||
if (entry.Subscribers.Count == 0)
|
||||
ScheduleLingerStop(entry);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Schedules a linger-delayed stop for a site whose last viewer just left. A
|
||||
/// re-subscribe within the linger window cancels it (<see cref="SiteEntry.CancelLinger"/>).
|
||||
/// Must be called under <see cref="_lock"/>.
|
||||
/// </summary>
|
||||
private void ScheduleLingerStop(SiteEntry entry)
|
||||
{
|
||||
var version = ++entry.LingerVersion;
|
||||
entry.LingerTimer?.Dispose();
|
||||
entry.LingerTimer = new Timer(
|
||||
_ => LingerStop(entry.SiteId, version),
|
||||
state: null,
|
||||
dueTime: _options.LiveAlarmCacheLinger,
|
||||
period: Timeout.InfiniteTimeSpan);
|
||||
}
|
||||
|
||||
private void LingerStop(int siteId, int version)
|
||||
{
|
||||
IActorRef? actorToStop = null;
|
||||
lock (_lock)
|
||||
{
|
||||
if (!_sites.TryGetValue(siteId, out var entry))
|
||||
return;
|
||||
// A viewer may have returned (version bumped) or re-subscribed (subscribers > 0)
|
||||
// after this timer was armed — only stop if this is still the latest, empty state.
|
||||
if (entry.LingerVersion != version || entry.Subscribers.Count != 0)
|
||||
return;
|
||||
|
||||
actorToStop = entry.Actor;
|
||||
entry.DisposeTimers();
|
||||
_sites.Remove(siteId);
|
||||
}
|
||||
|
||||
actorToStop?.Tell(new StopSiteAlarmAggregator());
|
||||
_logger.LogDebug("Stopped live alarm aggregator for site {SiteId} (last viewer left)", siteId);
|
||||
}
|
||||
|
||||
// ── Aggregator start ────────────────────────────────────────────────────────
|
||||
|
||||
private async Task StartAggregatorAsync(SiteEntry entry)
|
||||
{
|
||||
try
|
||||
{
|
||||
var system = _actorSystem;
|
||||
if (system is null)
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Live alarm cache asked to start site {SiteId} before the ActorSystem was set; " +
|
||||
"the page will keep its poll fallback", entry.SiteId);
|
||||
return;
|
||||
}
|
||||
|
||||
var resolved = await ResolveSiteAsync(entry.SiteId);
|
||||
if (resolved is null)
|
||||
return; // logged inside ResolveSiteAsync
|
||||
|
||||
var (siteIdentifier, grpcA, grpcB) = resolved.Value;
|
||||
var correlationId = $"site-alarm-{entry.SiteId}-{Guid.NewGuid():N}";
|
||||
|
||||
Func<CancellationToken, Task<IReadOnlyList<AlarmStateChanged>>> seedFn =
|
||||
ct => FanOutSnapshotsAsync(entry.SiteId, ct);
|
||||
Action<IReadOnlyList<AlarmStateChanged>> publish =
|
||||
snapshot => OnPublish(entry, snapshot);
|
||||
|
||||
lock (_lock)
|
||||
{
|
||||
// Bail if the entry was torn down (last viewer left + linger fired) while we
|
||||
// resolved the site, or if a racing start already created the actor.
|
||||
if (!_sites.TryGetValue(entry.SiteId, out var current) || !ReferenceEquals(current, entry))
|
||||
return;
|
||||
if (entry.Actor is not null)
|
||||
return;
|
||||
|
||||
var props = Props.Create(() => new SiteAlarmAggregatorActor(
|
||||
siteIdentifier,
|
||||
correlationId,
|
||||
seedFn,
|
||||
publish,
|
||||
_grpcFactory,
|
||||
grpcA,
|
||||
grpcB,
|
||||
_options.LiveAlarmCacheReconcileInterval));
|
||||
|
||||
entry.Actor = system.ActorOf(props, $"site-alarm-aggregator-{entry.SiteId}-{Guid.NewGuid():N}");
|
||||
entry.StartRetryTimer?.Dispose();
|
||||
entry.StartRetryTimer = null;
|
||||
_logger.LogInformation("Started live alarm aggregator for site {SiteId} ({SiteIdentifier})",
|
||||
entry.SiteId, siteIdentifier);
|
||||
}
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogError(ex, "Failed to start live alarm aggregator for site {SiteId}", entry.SiteId);
|
||||
}
|
||||
finally
|
||||
{
|
||||
lock (_lock)
|
||||
{
|
||||
entry.Starting = false;
|
||||
|
||||
// A transient start failure (ActorSystem not yet set, a DB blip resolving
|
||||
// the site, ActorOf throwing) would otherwise leave this viewer cohort
|
||||
// polling forever — nothing re-attempts until the NEXT Subscribe. Arm a
|
||||
// bounded retry so the feature self-heals once the condition clears, as
|
||||
// long as viewers remain and the entry is still live.
|
||||
var stillWanted = _sites.TryGetValue(entry.SiteId, out var current)
|
||||
&& ReferenceEquals(current, entry)
|
||||
&& entry.Actor is null
|
||||
&& entry.Subscribers.Count > 0;
|
||||
if (stillWanted)
|
||||
{
|
||||
entry.Starting = true; // keep the guard closed so a Subscribe won't double-start
|
||||
entry.StartRetryTimer?.Dispose();
|
||||
entry.StartRetryTimer = new Timer(
|
||||
_ => { _ = StartAggregatorAsync(entry); },
|
||||
state: null,
|
||||
dueTime: _options.LiveAlarmCacheReconcileInterval,
|
||||
period: Timeout.InfiniteTimeSpan);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Resolves a site's identifier + both gRPC node endpoints. Returns <c>null</c> (and
|
||||
/// logs) when the site is unknown or has no gRPC addresses configured, in which case
|
||||
/// no aggregator is started and the page keeps polling.
|
||||
/// </summary>
|
||||
private async Task<(string SiteIdentifier, string GrpcA, string GrpcB)?> ResolveSiteAsync(int siteId)
|
||||
{
|
||||
using var scope = _serviceProvider.CreateScope();
|
||||
var siteRepo = scope.ServiceProvider.GetRequiredService<ISiteRepository>();
|
||||
var site = await siteRepo.GetSiteByIdAsync(siteId);
|
||||
if (site is null)
|
||||
{
|
||||
_logger.LogWarning("Live alarm cache: site {SiteId} not found; not starting an aggregator", siteId);
|
||||
return null;
|
||||
}
|
||||
|
||||
if (string.IsNullOrWhiteSpace(site.GrpcNodeAAddress) || string.IsNullOrWhiteSpace(site.GrpcNodeBAddress))
|
||||
{
|
||||
_logger.LogWarning(
|
||||
"Live alarm cache: site {SiteId} ({SiteIdentifier}) has no gRPC node addresses; " +
|
||||
"not starting an aggregator", siteId, site.SiteIdentifier);
|
||||
return null;
|
||||
}
|
||||
|
||||
return (site.SiteIdentifier, site.GrpcNodeAAddress!, site.GrpcNodeBAddress!);
|
||||
}
|
||||
|
||||
// ── Snapshot fan-out (the seed / reconcile) ─────────────────────────────────
|
||||
|
||||
/// <summary>
|
||||
/// Fans out one debug-snapshot Ask per Enabled instance on the site (capped at
|
||||
/// <see cref="CommunicationOptions.LiveAlarmCacheSeedConcurrency"/>) and returns every
|
||||
/// snapshot's alarm rows — <b>including</b> <see cref="AlarmStateChanged.IsConfiguredPlaceholder"/>
|
||||
/// placeholders. Best-effort: an instance that faults, times out, or reports
|
||||
/// <see cref="DebugViewSnapshot.InstanceNotFound"/> contributes no rows rather than
|
||||
/// failing the whole seed (mirrors <c>AlarmSummaryService</c>). Re-enumerates instances
|
||||
/// each call so a reconcile picks up enable/disable/deploy/delete drift.
|
||||
/// </summary>
|
||||
private async Task<IReadOnlyList<AlarmStateChanged>> FanOutSnapshotsAsync(int siteId, CancellationToken ct)
|
||||
{
|
||||
string siteIdentifier;
|
||||
List<string> enabledInstanceNames;
|
||||
|
||||
using (var scope = _serviceProvider.CreateScope())
|
||||
{
|
||||
var siteRepo = scope.ServiceProvider.GetRequiredService<ISiteRepository>();
|
||||
var site = await siteRepo.GetSiteByIdAsync(siteId, ct);
|
||||
if (site is null)
|
||||
return Empty;
|
||||
siteIdentifier = site.SiteIdentifier;
|
||||
|
||||
var instanceRepo = scope.ServiceProvider.GetRequiredService<ITemplateEngineRepository>();
|
||||
var instances = await instanceRepo.GetInstancesBySiteIdAsync(siteId, ct);
|
||||
enabledInstanceNames = instances
|
||||
.Where(i => i.State == InstanceState.Enabled)
|
||||
.Select(i => i.UniqueName)
|
||||
.ToList();
|
||||
}
|
||||
|
||||
if (enabledInstanceNames.Count == 0)
|
||||
return Empty;
|
||||
|
||||
var rows = new ConcurrentBag<AlarmStateChanged>();
|
||||
using var gate = new SemaphoreSlim(
|
||||
Math.Max(1, _options.LiveAlarmCacheSeedConcurrency),
|
||||
Math.Max(1, _options.LiveAlarmCacheSeedConcurrency));
|
||||
|
||||
var fetches = enabledInstanceNames.Select(name =>
|
||||
FetchInstanceSnapshotAsync(siteIdentifier, name, gate, rows, ct));
|
||||
await Task.WhenAll(fetches);
|
||||
|
||||
return rows.ToList();
|
||||
}
|
||||
|
||||
private async Task FetchInstanceSnapshotAsync(
|
||||
string siteIdentifier,
|
||||
string instanceUniqueName,
|
||||
SemaphoreSlim gate,
|
||||
ConcurrentBag<AlarmStateChanged> rows,
|
||||
CancellationToken ct)
|
||||
{
|
||||
await gate.WaitAsync(ct);
|
||||
try
|
||||
{
|
||||
var request = new DebugSnapshotRequest(instanceUniqueName, Guid.NewGuid().ToString("N"));
|
||||
var snapshot = await _communicationService.RequestDebugSnapshotAsync(siteIdentifier, request, ct);
|
||||
if (snapshot.InstanceNotFound)
|
||||
return;
|
||||
|
||||
foreach (var alarm in snapshot.AlarmStates)
|
||||
rows.Add(alarm);
|
||||
}
|
||||
catch (OperationCanceledException) when (ct.IsCancellationRequested)
|
||||
{
|
||||
throw; // seed cancelled (actor stopping) — propagate
|
||||
}
|
||||
catch
|
||||
{
|
||||
// Degrade this one instance to "no rows" rather than failing the whole seed.
|
||||
}
|
||||
finally
|
||||
{
|
||||
gate.Release();
|
||||
}
|
||||
}
|
||||
|
||||
// ── Publish (actor thread → viewers) ────────────────────────────────────────
|
||||
|
||||
private void OnPublish(SiteEntry entry, IReadOnlyList<AlarmStateChanged> snapshot)
|
||||
{
|
||||
Subscription[] subscribers;
|
||||
lock (_lock)
|
||||
{
|
||||
// Store the fresh immutable snapshot (readers get it lock-free-ish via GetCurrentAlarms).
|
||||
entry.Current = snapshot;
|
||||
entry.HasPublished = true;
|
||||
subscribers = entry.Subscribers.ToArray();
|
||||
}
|
||||
|
||||
// Invoke viewer callbacks OUTSIDE the lock — a Blazor InvokeAsync(StateHasChanged)
|
||||
// must never re-enter our lock, and one faulting viewer must not stop the others.
|
||||
foreach (var subscriber in subscribers)
|
||||
{
|
||||
try
|
||||
{
|
||||
subscriber.OnChanged();
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
_logger.LogWarning(ex,
|
||||
"A live alarm viewer callback threw for site {SiteId}; continuing", entry.SiteId);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// ── Nested types ────────────────────────────────────────────────────────────
|
||||
|
||||
/// <summary>Per-site bookkeeping. All mutable fields are guarded by the service's <c>_lock</c>.</summary>
|
||||
private sealed class SiteEntry
|
||||
{
|
||||
public SiteEntry(int siteId) => SiteId = siteId;
|
||||
|
||||
public int SiteId { get; }
|
||||
public List<Subscription> Subscribers { get; } = new();
|
||||
public IActorRef? Actor { get; set; }
|
||||
public bool Starting { get; set; }
|
||||
|
||||
/// <summary>Current published snapshot the page renders. Swapped as a whole immutable list.</summary>
|
||||
public IReadOnlyList<AlarmStateChanged> Current { get; set; } = Empty;
|
||||
|
||||
/// <summary>True once the aggregator has seeded and published at least once.</summary>
|
||||
public bool HasPublished { get; set; }
|
||||
|
||||
public Timer? LingerTimer { get; set; }
|
||||
public int LingerVersion { get; set; }
|
||||
|
||||
/// <summary>Bounded retry timer armed when a start attempt failed to create the actor.</summary>
|
||||
public Timer? StartRetryTimer { get; set; }
|
||||
|
||||
/// <summary>Cancels any pending linger stop (a viewer arrived).</summary>
|
||||
public void CancelLinger()
|
||||
{
|
||||
LingerVersion++;
|
||||
LingerTimer?.Dispose();
|
||||
LingerTimer = null;
|
||||
}
|
||||
|
||||
/// <summary>Disposes both timers — called when the entry is torn down.</summary>
|
||||
public void DisposeTimers()
|
||||
{
|
||||
LingerTimer?.Dispose();
|
||||
LingerTimer = null;
|
||||
StartRetryTimer?.Dispose();
|
||||
StartRetryTimer = null;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The handle returned when a Subscribe is rejected at the per-site viewer cap. It was
|
||||
/// never registered, so <see cref="Dispose"/> is a genuine no-op — safe to dispose any
|
||||
/// number of times and it never touches the site's subscriber list.
|
||||
/// </summary>
|
||||
private sealed class NoOpSubscription : IDisposable
|
||||
{
|
||||
public static readonly NoOpSubscription Instance = new();
|
||||
private NoOpSubscription() { }
|
||||
public void Dispose() { }
|
||||
}
|
||||
|
||||
/// <summary>The disposable handed to a viewer; idempotent <see cref="Dispose"/> unregisters it.</summary>
|
||||
private sealed class Subscription : IDisposable
|
||||
{
|
||||
private readonly SiteAlarmLiveCacheService _owner;
|
||||
private int _disposed;
|
||||
|
||||
public Subscription(SiteAlarmLiveCacheService owner, int siteId, Action onChanged)
|
||||
{
|
||||
_owner = owner;
|
||||
SiteId = siteId;
|
||||
OnChanged = onChanged;
|
||||
}
|
||||
|
||||
public int SiteId { get; }
|
||||
public Action OnChanged { get; }
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
if (Interlocked.Exchange(ref _disposed, 1) == 0)
|
||||
_owner.Unsubscribe(this);
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -28,6 +28,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
"L3Byb3RvYnVmL3RpbWVzdGFtcC5wcm90bxoeZ29vZ2xlL3Byb3RvYnVmL3dy",
|
||||
"YXBwZXJzLnByb3RvIk0KFUluc3RhbmNlU3RyZWFtUmVxdWVzdBIWCg5jb3Jy",
|
||||
"ZWxhdGlvbl9pZBgBIAEoCRIcChRpbnN0YW5jZV91bmlxdWVfbmFtZRgCIAEo",
|
||||
"CSIrChFTaXRlU3RyZWFtUmVxdWVzdBIWCg5jb3JyZWxhdGlvbl9pZBgBIAEo",
|
||||
"CSKoAQoPU2l0ZVN0cmVhbUV2ZW50EhYKDmNvcnJlbGF0aW9uX2lkGAEgASgJ",
|
||||
"Ej0KEWF0dHJpYnV0ZV9jaGFuZ2VkGAIgASgLMiAuc2l0ZXN0cmVhbS5BdHRy",
|
||||
"aWJ1dGVWYWx1ZVVwZGF0ZUgAEjUKDWFsYXJtX2NoYW5nZWQYAyABKAsyHC5z",
|
||||
@@ -95,22 +96,25 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
"VEFURV9BQ1RJVkUQAiqFAQoOQWxhcm1MZXZlbEVudW0SFAoQQUxBUk1fTEVW",
|
||||
"RUxfTk9ORRAAEhMKD0FMQVJNX0xFVkVMX0xPVxABEhcKE0FMQVJNX0xFVkVM",
|
||||
"X0xPV19MT1cQAhIUChBBTEFSTV9MRVZFTF9ISUdIEAMSGQoVQUxBUk1fTEVW",
|
||||
"RUxfSElHSF9ISUdIEAQytwMKEVNpdGVTdHJlYW1TZXJ2aWNlElUKEVN1YnNj",
|
||||
"RUxfSElHSF9ISUdIEAQyhgQKEVNpdGVTdHJlYW1TZXJ2aWNlElUKEVN1YnNj",
|
||||
"cmliZUluc3RhbmNlEiEuc2l0ZXN0cmVhbS5JbnN0YW5jZVN0cmVhbVJlcXVl",
|
||||
"c3QaGy5zaXRlc3RyZWFtLlNpdGVTdHJlYW1FdmVudDABEkcKEUluZ2VzdEF1",
|
||||
"ZGl0RXZlbnRzEhsuc2l0ZXN0cmVhbS5BdWRpdEV2ZW50QmF0Y2gaFS5zaXRl",
|
||||
"c3RyZWFtLkluZ2VzdEFjaxJQChVJbmdlc3RDYWNoZWRUZWxlbWV0cnkSIC5z",
|
||||
"aXRlc3RyZWFtLkNhY2hlZFRlbGVtZXRyeUJhdGNoGhUuc2l0ZXN0cmVhbS5J",
|
||||
"bmdlc3RBY2sSWgoPUHVsbEF1ZGl0RXZlbnRzEiIuc2l0ZXN0cmVhbS5QdWxs",
|
||||
"QXVkaXRFdmVudHNSZXF1ZXN0GiMuc2l0ZXN0cmVhbS5QdWxsQXVkaXRFdmVu",
|
||||
"dHNSZXNwb25zZRJUCg1QdWxsU2l0ZUNhbGxzEiAuc2l0ZXN0cmVhbS5QdWxs",
|
||||
"U2l0ZUNhbGxzUmVxdWVzdBohLnNpdGVzdHJlYW0uUHVsbFNpdGVDYWxsc1Jl",
|
||||
"c3BvbnNlQiuqAihaQi5NT00uV1cuU2NhZGFCcmlkZ2UuQ29tbXVuaWNhdGlv",
|
||||
"bi5HcnBjYgZwcm90bzM="));
|
||||
"c3QaGy5zaXRlc3RyZWFtLlNpdGVTdHJlYW1FdmVudDABEk0KDVN1YnNjcmli",
|
||||
"ZVNpdGUSHS5zaXRlc3RyZWFtLlNpdGVTdHJlYW1SZXF1ZXN0Ghsuc2l0ZXN0",
|
||||
"cmVhbS5TaXRlU3RyZWFtRXZlbnQwARJHChFJbmdlc3RBdWRpdEV2ZW50cxIb",
|
||||
"LnNpdGVzdHJlYW0uQXVkaXRFdmVudEJhdGNoGhUuc2l0ZXN0cmVhbS5Jbmdl",
|
||||
"c3RBY2sSUAoVSW5nZXN0Q2FjaGVkVGVsZW1ldHJ5EiAuc2l0ZXN0cmVhbS5D",
|
||||
"YWNoZWRUZWxlbWV0cnlCYXRjaBoVLnNpdGVzdHJlYW0uSW5nZXN0QWNrEloK",
|
||||
"D1B1bGxBdWRpdEV2ZW50cxIiLnNpdGVzdHJlYW0uUHVsbEF1ZGl0RXZlbnRz",
|
||||
"UmVxdWVzdBojLnNpdGVzdHJlYW0uUHVsbEF1ZGl0RXZlbnRzUmVzcG9uc2US",
|
||||
"VAoNUHVsbFNpdGVDYWxscxIgLnNpdGVzdHJlYW0uUHVsbFNpdGVDYWxsc1Jl",
|
||||
"cXVlc3QaIS5zaXRlc3RyZWFtLlB1bGxTaXRlQ2FsbHNSZXNwb25zZUIrqgIo",
|
||||
"WkIuTU9NLldXLlNjYWRhQnJpZGdlLkNvbW11bmljYXRpb24uR3JwY2IGcHJv",
|
||||
"dG8z"));
|
||||
descriptor = pbr::FileDescriptor.FromGeneratedCode(descriptorData,
|
||||
new pbr::FileDescriptor[] { global::Google.Protobuf.WellKnownTypes.TimestampReflection.Descriptor, global::Google.Protobuf.WellKnownTypes.WrappersReflection.Descriptor, },
|
||||
new pbr::GeneratedClrTypeInfo(new[] {typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.Quality), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmStateEnum), typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmLevelEnum), }, null, new pbr::GeneratedClrTypeInfo[] {
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.InstanceStreamRequest), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.InstanceStreamRequest.Parser, new[]{ "CorrelationId", "InstanceUniqueName" }, null, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest.Parser, new[]{ "CorrelationId" }, null, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEvent), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEvent.Parser, new[]{ "CorrelationId", "AttributeChanged", "AlarmChanged" }, new[]{ "Event" }, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AttributeValueUpdate), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AttributeValueUpdate.Parser, new[]{ "InstanceUniqueName", "AttributePath", "AttributeName", "Value", "Quality", "Timestamp" }, null, null, null, null),
|
||||
new pbr::GeneratedClrTypeInfo(typeof(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmStateUpdate), global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AlarmStateUpdate.Parser, new[]{ "InstanceUniqueName", "AlarmName", "State", "Priority", "Timestamp", "Level", "Message", "Kind", "Active", "Acknowledged", "Confirmed", "ShelveState", "Suppressed", "SourceReference", "AlarmTypeName", "Category", "OperatorUser", "OperatorComment", "OriginalRaiseTime", "CurrentValue", "LimitValue", "NativeSourceCanonicalName", "IsConfiguredPlaceholder" }, null, null, null, null),
|
||||
@@ -394,6 +398,209 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Request for the site-wide, alarm-only SubscribeSite stream. Unlike
|
||||
/// InstanceStreamRequest there is NO instance filter — the stream carries alarm
|
||||
/// transitions for every instance on the site.
|
||||
/// </summary>
|
||||
[global::System.Diagnostics.DebuggerDisplayAttribute("{ToString(),nq}")]
|
||||
public sealed partial class SiteStreamRequest : pb::IMessage<SiteStreamRequest>
|
||||
#if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE
|
||||
, pb::IBufferMessage
|
||||
#endif
|
||||
{
|
||||
private static readonly pb::MessageParser<SiteStreamRequest> _parser = new pb::MessageParser<SiteStreamRequest>(() => new SiteStreamRequest());
|
||||
private pb::UnknownFieldSet _unknownFields;
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public static pb::MessageParser<SiteStreamRequest> Parser { get { return _parser; } }
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public static pbr::MessageDescriptor Descriptor {
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[1]; }
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
pbr::MessageDescriptor pb::IMessage.Descriptor {
|
||||
get { return Descriptor; }
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public SiteStreamRequest() {
|
||||
OnConstruction();
|
||||
}
|
||||
|
||||
partial void OnConstruction();
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public SiteStreamRequest(SiteStreamRequest other) : this() {
|
||||
correlationId_ = other.correlationId_;
|
||||
_unknownFields = pb::UnknownFieldSet.Clone(other._unknownFields);
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public SiteStreamRequest Clone() {
|
||||
return new SiteStreamRequest(this);
|
||||
}
|
||||
|
||||
/// <summary>Field number for the "correlation_id" field.</summary>
|
||||
public const int CorrelationIdFieldNumber = 1;
|
||||
private string correlationId_ = "";
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public string CorrelationId {
|
||||
get { return correlationId_; }
|
||||
set {
|
||||
correlationId_ = pb::ProtoPreconditions.CheckNotNull(value, "value");
|
||||
}
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public override bool Equals(object other) {
|
||||
return Equals(other as SiteStreamRequest);
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public bool Equals(SiteStreamRequest other) {
|
||||
if (ReferenceEquals(other, null)) {
|
||||
return false;
|
||||
}
|
||||
if (ReferenceEquals(other, this)) {
|
||||
return true;
|
||||
}
|
||||
if (CorrelationId != other.CorrelationId) return false;
|
||||
return Equals(_unknownFields, other._unknownFields);
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public override int GetHashCode() {
|
||||
int hash = 1;
|
||||
if (CorrelationId.Length != 0) hash ^= CorrelationId.GetHashCode();
|
||||
if (_unknownFields != null) {
|
||||
hash ^= _unknownFields.GetHashCode();
|
||||
}
|
||||
return hash;
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public override string ToString() {
|
||||
return pb::JsonFormatter.ToDiagnosticString(this);
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public void WriteTo(pb::CodedOutputStream output) {
|
||||
#if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE
|
||||
output.WriteRawMessage(this);
|
||||
#else
|
||||
if (CorrelationId.Length != 0) {
|
||||
output.WriteRawTag(10);
|
||||
output.WriteString(CorrelationId);
|
||||
}
|
||||
if (_unknownFields != null) {
|
||||
_unknownFields.WriteTo(output);
|
||||
}
|
||||
#endif
|
||||
}
|
||||
|
||||
#if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
void pb::IBufferMessage.InternalWriteTo(ref pb::WriteContext output) {
|
||||
if (CorrelationId.Length != 0) {
|
||||
output.WriteRawTag(10);
|
||||
output.WriteString(CorrelationId);
|
||||
}
|
||||
if (_unknownFields != null) {
|
||||
_unknownFields.WriteTo(ref output);
|
||||
}
|
||||
}
|
||||
#endif
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public int CalculateSize() {
|
||||
int size = 0;
|
||||
if (CorrelationId.Length != 0) {
|
||||
size += 1 + pb::CodedOutputStream.ComputeStringSize(CorrelationId);
|
||||
}
|
||||
if (_unknownFields != null) {
|
||||
size += _unknownFields.CalculateSize();
|
||||
}
|
||||
return size;
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public void MergeFrom(SiteStreamRequest other) {
|
||||
if (other == null) {
|
||||
return;
|
||||
}
|
||||
if (other.CorrelationId.Length != 0) {
|
||||
CorrelationId = other.CorrelationId;
|
||||
}
|
||||
_unknownFields = pb::UnknownFieldSet.MergeFrom(_unknownFields, other._unknownFields);
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public void MergeFrom(pb::CodedInputStream input) {
|
||||
#if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE
|
||||
input.ReadRawMessage(this);
|
||||
#else
|
||||
uint tag;
|
||||
while ((tag = input.ReadTag()) != 0) {
|
||||
if ((tag & 7) == 4) {
|
||||
// Abort on any end group tag.
|
||||
return;
|
||||
}
|
||||
switch(tag) {
|
||||
default:
|
||||
_unknownFields = pb::UnknownFieldSet.MergeFieldFrom(_unknownFields, input);
|
||||
break;
|
||||
case 10: {
|
||||
CorrelationId = input.ReadString();
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
#endif
|
||||
}
|
||||
|
||||
#if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
void pb::IBufferMessage.InternalMergeFrom(ref pb::ParseContext input) {
|
||||
uint tag;
|
||||
while ((tag = input.ReadTag()) != 0) {
|
||||
if ((tag & 7) == 4) {
|
||||
// Abort on any end group tag.
|
||||
return;
|
||||
}
|
||||
switch(tag) {
|
||||
default:
|
||||
_unknownFields = pb::UnknownFieldSet.MergeFieldFrom(_unknownFields, ref input);
|
||||
break;
|
||||
case 10: {
|
||||
CorrelationId = input.ReadString();
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
#endif
|
||||
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerDisplayAttribute("{ToString(),nq}")]
|
||||
public sealed partial class SiteStreamEvent : pb::IMessage<SiteStreamEvent>
|
||||
#if !GOOGLE_PROTOBUF_REFSTRUCT_COMPATIBILITY_MODE
|
||||
@@ -409,7 +616,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public static pbr::MessageDescriptor Descriptor {
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[1]; }
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[2]; }
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
@@ -740,7 +947,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public static pbr::MessageDescriptor Descriptor {
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[2]; }
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[3]; }
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
@@ -1132,7 +1339,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public static pbr::MessageDescriptor Descriptor {
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[3]; }
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[4]; }
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
@@ -2201,7 +2408,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public static pbr::MessageDescriptor Descriptor {
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[4]; }
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[5]; }
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
@@ -3216,7 +3423,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public static pbr::MessageDescriptor Descriptor {
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[5]; }
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[6]; }
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
@@ -3403,7 +3610,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public static pbr::MessageDescriptor Descriptor {
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[6]; }
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[7]; }
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
@@ -3596,7 +3803,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public static pbr::MessageDescriptor Descriptor {
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[7]; }
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[8]; }
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
@@ -4254,7 +4461,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public static pbr::MessageDescriptor Descriptor {
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[8]; }
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[9]; }
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
@@ -4507,7 +4714,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public static pbr::MessageDescriptor Descriptor {
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[9]; }
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[10]; }
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
@@ -4702,7 +4909,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public static pbr::MessageDescriptor Descriptor {
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[10]; }
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[11]; }
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
@@ -4946,7 +5153,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public static pbr::MessageDescriptor Descriptor {
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[11]; }
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[12]; }
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
@@ -5179,7 +5386,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public static pbr::MessageDescriptor Descriptor {
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[12]; }
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[13]; }
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
@@ -5469,7 +5676,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("protoc", null)]
|
||||
public static pbr::MessageDescriptor Descriptor {
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[13]; }
|
||||
get { return global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SitestreamReflection.Descriptor.MessageTypes[14]; }
|
||||
}
|
||||
|
||||
[global::System.Diagnostics.DebuggerNonUserCodeAttribute]
|
||||
|
||||
@@ -50,6 +50,8 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
|
||||
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEvent> __Marshaller_sitestream_SiteStreamEvent = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEvent.Parser));
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
|
||||
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest> __Marshaller_sitestream_SiteStreamRequest = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest.Parser));
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
|
||||
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch> __Marshaller_sitestream_AuditEventBatch = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch.Parser));
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
|
||||
static readonly grpc::Marshaller<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> __Marshaller_sitestream_IngestAck = grpc::Marshallers.Create(__Helper_SerializeMessage, context => __Helper_DeserializeMessage(context, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck.Parser));
|
||||
@@ -72,6 +74,14 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
__Marshaller_sitestream_InstanceStreamRequest,
|
||||
__Marshaller_sitestream_SiteStreamEvent);
|
||||
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
|
||||
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEvent> __Method_SubscribeSite = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEvent>(
|
||||
grpc::MethodType.ServerStreaming,
|
||||
__ServiceName,
|
||||
"SubscribeSite",
|
||||
__Marshaller_sitestream_SiteStreamRequest,
|
||||
__Marshaller_sitestream_SiteStreamEvent);
|
||||
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
|
||||
static readonly grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> __Method_IngestAuditEvents = new grpc::Method<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck>(
|
||||
grpc::MethodType.Unary,
|
||||
@@ -120,6 +130,21 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Site-wide, alarm-only live stream (aggregated Alarm Summary): every
|
||||
/// AlarmStateChanged for ALL instances on the site, no per-instance filter.
|
||||
/// Attribute updates are never carried on this stream.
|
||||
/// </summary>
|
||||
/// <param name="request">The request received from the client.</param>
|
||||
/// <param name="responseStream">Used for sending responses back to the client.</param>
|
||||
/// <param name="context">The context of the server-side call handler being invoked.</param>
|
||||
/// <returns>A task indicating completion of the handler.</returns>
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
|
||||
public virtual global::System.Threading.Tasks.Task SubscribeSite(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest request, grpc::IServerStreamWriter<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEvent> responseStream, grpc::ServerCallContext context)
|
||||
{
|
||||
throw new grpc::RpcException(new grpc::Status(grpc::StatusCode.Unimplemented, ""));
|
||||
}
|
||||
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
|
||||
public virtual global::System.Threading.Tasks.Task<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck> IngestAuditEvents(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch request, grpc::ServerCallContext context)
|
||||
{
|
||||
@@ -183,6 +208,34 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
{
|
||||
return CallInvoker.AsyncServerStreamingCall(__Method_SubscribeInstance, null, options, request);
|
||||
}
|
||||
/// <summary>
|
||||
/// Site-wide, alarm-only live stream (aggregated Alarm Summary): every
|
||||
/// AlarmStateChanged for ALL instances on the site, no per-instance filter.
|
||||
/// Attribute updates are never carried on this stream.
|
||||
/// </summary>
|
||||
/// <param name="request">The request to send to the server.</param>
|
||||
/// <param name="headers">The initial metadata to send with the call. This parameter is optional.</param>
|
||||
/// <param name="deadline">An optional deadline for the call. The call will be cancelled if deadline is hit.</param>
|
||||
/// <param name="cancellationToken">An optional token for canceling the call.</param>
|
||||
/// <returns>The call object.</returns>
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
|
||||
public virtual grpc::AsyncServerStreamingCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEvent> SubscribeSite(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
|
||||
{
|
||||
return SubscribeSite(request, new grpc::CallOptions(headers, deadline, cancellationToken));
|
||||
}
|
||||
/// <summary>
|
||||
/// Site-wide, alarm-only live stream (aggregated Alarm Summary): every
|
||||
/// AlarmStateChanged for ALL instances on the site, no per-instance filter.
|
||||
/// Attribute updates are never carried on this stream.
|
||||
/// </summary>
|
||||
/// <param name="request">The request to send to the server.</param>
|
||||
/// <param name="options">The options for the call.</param>
|
||||
/// <returns>The call object.</returns>
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
|
||||
public virtual grpc::AsyncServerStreamingCall<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEvent> SubscribeSite(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest request, grpc::CallOptions options)
|
||||
{
|
||||
return CallInvoker.AsyncServerStreamingCall(__Method_SubscribeSite, null, options, request);
|
||||
}
|
||||
[global::System.CodeDom.Compiler.GeneratedCode("grpc_csharp_plugin", null)]
|
||||
public virtual global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck IngestAuditEvents(global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch request, grpc::Metadata headers = null, global::System.DateTime? deadline = null, global::System.Threading.CancellationToken cancellationToken = default(global::System.Threading.CancellationToken))
|
||||
{
|
||||
@@ -278,6 +331,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
{
|
||||
return grpc::ServerServiceDefinition.CreateBuilder()
|
||||
.AddMethod(__Method_SubscribeInstance, serviceImpl.SubscribeInstance)
|
||||
.AddMethod(__Method_SubscribeSite, serviceImpl.SubscribeSite)
|
||||
.AddMethod(__Method_IngestAuditEvents, serviceImpl.IngestAuditEvents)
|
||||
.AddMethod(__Method_IngestCachedTelemetry, serviceImpl.IngestCachedTelemetry)
|
||||
.AddMethod(__Method_PullAuditEvents, serviceImpl.PullAuditEvents)
|
||||
@@ -292,6 +346,7 @@ namespace ZB.MOM.WW.ScadaBridge.Communication.Grpc {
|
||||
public static void BindService(grpc::ServiceBinderBase serviceBinder, SiteStreamServiceBase serviceImpl)
|
||||
{
|
||||
serviceBinder.AddMethod(__Method_SubscribeInstance, serviceImpl == null ? null : new grpc::ServerStreamingServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.InstanceStreamRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEvent>(serviceImpl.SubscribeInstance));
|
||||
serviceBinder.AddMethod(__Method_SubscribeSite, serviceImpl == null ? null : new grpc::ServerStreamingServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.SiteStreamEvent>(serviceImpl.SubscribeSite));
|
||||
serviceBinder.AddMethod(__Method_IngestAuditEvents, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.AuditEventBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck>(serviceImpl.IngestAuditEvents));
|
||||
serviceBinder.AddMethod(__Method_IngestCachedTelemetry, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.CachedTelemetryBatch, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.IngestAck>(serviceImpl.IngestCachedTelemetry));
|
||||
serviceBinder.AddMethod(__Method_PullAuditEvents, serviceImpl == null ? null : new grpc::UnaryServerMethod<global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullAuditEventsRequest, global::ZB.MOM.WW.ScadaBridge.Communication.Grpc.PullAuditEventsResponse>(serviceImpl.PullAuditEvents));
|
||||
|
||||
@@ -424,6 +424,11 @@ akka {{
|
||||
var debugStreamService = _serviceProvider.GetService<DebugStreamService>();
|
||||
debugStreamService?.SetActorSystem(_actorSystem!);
|
||||
|
||||
// Wire up the aggregated live alarm cache with the ActorSystem so it can host
|
||||
// per-site aggregator actors on the active central node (plan #10, Task 4).
|
||||
var siteAlarmLiveCache = _serviceProvider.GetService<ZB.MOM.WW.ScadaBridge.Communication.SiteAlarmLiveCacheService>();
|
||||
siteAlarmLiveCache?.SetActorSystem(_actorSystem!);
|
||||
|
||||
|
||||
// Management Service — accessible via ClusterClient
|
||||
var mgmtLogger = _serviceProvider.GetRequiredService<ILoggerFactory>()
|
||||
|
||||
@@ -20,6 +20,9 @@ namespace ZB.MOM.WW.ScadaBridge.SiteRuntime.Streaming;
|
||||
/// </summary>
|
||||
public class SiteStreamManager : ISiteStreamSubscriber
|
||||
{
|
||||
/// <summary>Sentinel instance name recorded for site-wide (non-instance-scoped) subscriptions.</summary>
|
||||
private const string SiteWideInstanceName = "*";
|
||||
|
||||
private ActorSystem? _system;
|
||||
private IMaterializer? _materializer;
|
||||
private readonly int _bufferSize;
|
||||
@@ -117,6 +120,44 @@ public class SiteStreamManager : ISiteStreamSubscriber
|
||||
return subscriptionId;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Subscribe to ALARM events for ALL instances on the site (no per-instance
|
||||
/// filter). Only <see cref="AlarmStateChanged"/> events are forwarded;
|
||||
/// <see cref="AttributeValueChanged"/> events are dropped (attributes are far
|
||||
/// higher-volume and the aggregated Alarm Summary never shows them). Same
|
||||
/// broadcast-hub wiring as <see cref="Subscribe"/>, and the returned
|
||||
/// subscription id is torn down via <see cref="Unsubscribe"/> exactly like the
|
||||
/// per-instance variant.
|
||||
/// </summary>
|
||||
/// <param name="subscriber">The actor that receives forwarded <see cref="AlarmStateChanged"/> events.</param>
|
||||
/// <returns>A subscription id to pass to <see cref="Unsubscribe"/>.</returns>
|
||||
public string SubscribeSiteAlarms(IActorRef subscriber)
|
||||
{
|
||||
if (_hubSource is null || _materializer is null)
|
||||
throw new InvalidOperationException("SiteStreamManager.Initialize must be called before SubscribeSiteAlarms");
|
||||
|
||||
var subscriptionId = Guid.NewGuid().ToString();
|
||||
var capturedSubscriber = subscriber;
|
||||
|
||||
var killSwitch = _hubSource
|
||||
.Where(ev => ev is AlarmStateChanged)
|
||||
.Buffer(_bufferSize, OverflowStrategy.DropHead)
|
||||
.ViaMaterialized(KillSwitches.Single<ISiteStreamEvent>(), Keep.Right)
|
||||
.To(Sink.ForEach<ISiteStreamEvent>(ev => capturedSubscriber.Tell(ev)))
|
||||
.Run(_materializer);
|
||||
|
||||
lock (_lock)
|
||||
{
|
||||
_subscriptions[subscriptionId] = new SubscriptionInfo(
|
||||
SiteWideInstanceName, subscriber, killSwitch, DateTimeOffset.UtcNow);
|
||||
}
|
||||
|
||||
_logger.LogDebug(
|
||||
"Subscriber {SubscriptionId} registered for site-wide alarm events", subscriptionId);
|
||||
|
||||
return subscriptionId;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Unsubscribe from instance events. Shuts down the per-subscriber
|
||||
/// stream graph via its KillSwitch.
|
||||
|
||||
@@ -9,6 +9,7 @@ using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication;
|
||||
using ZB.MOM.WW.ScadaBridge.Security;
|
||||
using AlarmSummaryPage = ZB.MOM.WW.ScadaBridge.CentralUI.Components.Pages.Monitoring.AlarmSummary;
|
||||
|
||||
@@ -26,6 +27,7 @@ public class AlarmSummaryRenderTests : BunitContext
|
||||
{
|
||||
private readonly IAlarmSummaryService _summary = Substitute.For<IAlarmSummaryService>();
|
||||
private readonly ISiteRepository _siteRepo = Substitute.For<ISiteRepository>();
|
||||
private readonly FakeSiteAlarmLiveCache _liveCache = new();
|
||||
|
||||
public AlarmSummaryRenderTests()
|
||||
{
|
||||
@@ -39,14 +41,33 @@ public class AlarmSummaryRenderTests : BunitContext
|
||||
|
||||
_summary.GetSiteAlarmsAsync(Arg.Any<int>(), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.FromResult(new AlarmSummaryResult(rows, Array.Empty<string>())));
|
||||
// BuildFromLiveAlarms (Task 5): flatten the pushed live snapshot the same way
|
||||
// the real service does, so the live-path render tests exercise real behavior.
|
||||
_summary.BuildFromLiveAlarms(Arg.Any<IReadOnlyList<AlarmStateChanged>>())
|
||||
.Returns(ci =>
|
||||
{
|
||||
var alarms = ci.Arg<IReadOnlyList<AlarmStateChanged>>();
|
||||
var built = alarms
|
||||
.Select(a => new AlarmSummaryRow(a.InstanceUniqueName, a))
|
||||
.OrderBy(r => r.InstanceUniqueName, StringComparer.OrdinalIgnoreCase)
|
||||
.ThenBy(r => r.Alarm.AlarmName, StringComparer.OrdinalIgnoreCase)
|
||||
.ToList();
|
||||
return new AlarmSummaryResult(built, Array.Empty<string>());
|
||||
});
|
||||
_summary.ComputeRollup(Arg.Any<IReadOnlyList<AlarmSummaryRow>>())
|
||||
.Returns(new AlarmRollup(2, 900, 0, new Dictionary<AlarmKind, int>()));
|
||||
Services.AddSingleton(_summary);
|
||||
|
||||
_siteRepo.GetAllSitesAsync(Arg.Any<CancellationToken>())
|
||||
.Returns(Task.FromResult<IReadOnlyList<Site>>(new List<Site> { new("Site 1", "site1") { Id = 1 } }));
|
||||
.Returns(Task.FromResult<IReadOnlyList<Site>>(new List<Site>
|
||||
{
|
||||
new("Site 1", "site1") { Id = 1 },
|
||||
new("Site 2", "site2") { Id = 2 },
|
||||
}));
|
||||
Services.AddSingleton(_siteRepo);
|
||||
|
||||
Services.AddSingleton<ISiteAlarmLiveCache>(_liveCache);
|
||||
|
||||
var claims = new[]
|
||||
{
|
||||
new Claim(JwtTokenService.UsernameClaimType, "tester"),
|
||||
@@ -109,4 +130,133 @@ public class AlarmSummaryRenderTests : BunitContext
|
||||
private static string FirstRowInstance(IRenderedComponent<AlarmSummaryPage> cut) =>
|
||||
cut.FindAll("tr[data-test='alarm-summary-row']")[0]
|
||||
.QuerySelector("td")!.TextContent.Trim();
|
||||
|
||||
// ── plan #10 Task 5: live-cache-first with poll fallback ───────────────────
|
||||
|
||||
[Fact]
|
||||
public void SelectingSite_SubscribesToTheLiveCache()
|
||||
{
|
||||
RenderWithSiteSelected();
|
||||
|
||||
Assert.Equal(1, _liveCache.SubscribeCount);
|
||||
Assert.Equal(1, _liveCache.LastSubscribedSite);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void PollFallbackRenders_WhenCacheNotLiveYet()
|
||||
{
|
||||
// Cache starts not-live, so the site select falls back to the poll's
|
||||
// GetSiteAlarmsAsync snapshot (Zeta + Alpha).
|
||||
Assert.False(_liveCache.IsLive(1));
|
||||
|
||||
var cut = RenderWithSiteSelected();
|
||||
|
||||
Assert.Equal(2, cut.FindAll("tr[data-test='alarm-summary-row']").Count);
|
||||
Assert.Equal("Zeta", FirstRowInstance(cut)); // severity-desc default
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void OnChangedDelta_RebuildsRenderedRowsFromLiveSnapshot()
|
||||
{
|
||||
var cut = RenderWithSiteSelected();
|
||||
|
||||
// Push a fresh live snapshot: a single new alarm on a new instance.
|
||||
_liveCache.PushAlarms(new List<AlarmStateChanged>
|
||||
{
|
||||
new("Gamma", "G-alarm", AlarmState.Active, 300, DateTimeOffset.UtcNow),
|
||||
});
|
||||
|
||||
cut.WaitForAssertion(() =>
|
||||
{
|
||||
var rows = cut.FindAll("tr[data-test='alarm-summary-row']");
|
||||
Assert.Single(rows);
|
||||
Assert.Equal("Gamma", FirstRowInstance(cut));
|
||||
});
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void LeavingSite_UnsubscribesFromTheLiveCache()
|
||||
{
|
||||
var cut = RenderWithSiteSelected();
|
||||
Assert.Equal(0, _liveCache.DisposeCount);
|
||||
|
||||
// Clearing the site picker tears the subscription down.
|
||||
cut.Find("[data-test='alarm-summary-site']").Change("");
|
||||
|
||||
Assert.Equal(1, _liveCache.DisposeCount);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ChangingSite_ResubscribesToTheNewSite()
|
||||
{
|
||||
var cut = RenderWithSiteSelected(); // site 1
|
||||
|
||||
cut.Find("[data-test='alarm-summary-site']").Change("2");
|
||||
|
||||
Assert.Equal(2, _liveCache.SubscribeCount); // one per site
|
||||
Assert.Equal(1, _liveCache.DisposeCount); // old subscription disposed
|
||||
Assert.Equal(2, _liveCache.LastSubscribedSite);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void DisposingComponent_UnsubscribesFromTheLiveCache()
|
||||
{
|
||||
var cut = RenderWithSiteSelected();
|
||||
|
||||
cut.Instance.Dispose();
|
||||
|
||||
Assert.True(_liveCache.DisposeCount >= 1);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Controllable in-memory fake of <see cref="ISiteAlarmLiveCache"/>: records
|
||||
/// subscribe/dispose counts and lets a test push a live snapshot (which flips
|
||||
/// <see cref="IsLive"/> true and fires the stored onChanged), so the page's
|
||||
/// live-delta path can be driven deterministically.
|
||||
/// </summary>
|
||||
private sealed class FakeSiteAlarmLiveCache : ISiteAlarmLiveCache
|
||||
{
|
||||
private Action? _onChanged;
|
||||
private IReadOnlyList<AlarmStateChanged> _current = Array.Empty<AlarmStateChanged>();
|
||||
private bool _live;
|
||||
|
||||
public int SubscribeCount { get; private set; }
|
||||
public int DisposeCount { get; private set; }
|
||||
public int? LastSubscribedSite { get; private set; }
|
||||
|
||||
public IDisposable Subscribe(int siteId, Action onChanged)
|
||||
{
|
||||
SubscribeCount++;
|
||||
LastSubscribedSite = siteId;
|
||||
_onChanged = onChanged;
|
||||
return new Subscription(this);
|
||||
}
|
||||
|
||||
public IReadOnlyList<AlarmStateChanged> GetCurrentAlarms(int siteId) => _current;
|
||||
|
||||
public bool IsLive(int siteId) => _live;
|
||||
|
||||
public void PushAlarms(IReadOnlyList<AlarmStateChanged> alarms)
|
||||
{
|
||||
_current = alarms;
|
||||
_live = true;
|
||||
_onChanged?.Invoke();
|
||||
}
|
||||
|
||||
private sealed class Subscription : IDisposable
|
||||
{
|
||||
private readonly FakeSiteAlarmLiveCache _owner;
|
||||
private bool _disposed;
|
||||
|
||||
public Subscription(FakeSiteAlarmLiveCache owner) => _owner = owner;
|
||||
|
||||
public void Dispose()
|
||||
{
|
||||
if (_disposed) return;
|
||||
_disposed = true;
|
||||
_owner.DisposeCount++;
|
||||
_owner._onChanged = null;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -167,4 +167,40 @@ public class AlarmSummaryServiceTests
|
||||
Assert.Equal(0, rollup.WorstSeverity);
|
||||
Assert.Equal(0, rollup.UnackedCount);
|
||||
}
|
||||
|
||||
// ── BuildFromLiveAlarms (plan #10, Task 5) ────────────────────────────────
|
||||
|
||||
[Fact]
|
||||
public void BuildFromLiveAlarms_FlattensAndSorts_SameAsPollPath_WithEmptyNotReporting()
|
||||
{
|
||||
// Intentionally out-of-order input to prove the deterministic
|
||||
// instance-then-alarm-name sort matches GetSiteAlarmsAsync.
|
||||
var alarms = new List<AlarmStateChanged>
|
||||
{
|
||||
NativeAlarm("zeta", "Zulu", AlarmState.Active, 900, active: true, acked: false),
|
||||
NativeAlarm("alpha", "Bravo", AlarmState.Active, 500, active: true, acked: true),
|
||||
NativeAlarm("alpha", "Alpha", AlarmState.Active, 700, active: true, acked: false),
|
||||
};
|
||||
|
||||
var result = CreateSut().BuildFromLiveAlarms(alarms);
|
||||
|
||||
// One row per alarm, instance taken from the alarm itself.
|
||||
Assert.Equal(3, result.Alarms.Count);
|
||||
Assert.Collection(result.Alarms,
|
||||
r => Assert.Equal(("alpha", "Alpha"), (r.InstanceUniqueName, r.Alarm.AlarmName)),
|
||||
r => Assert.Equal(("alpha", "Bravo"), (r.InstanceUniqueName, r.Alarm.AlarmName)),
|
||||
r => Assert.Equal(("zeta", "Zulu"), (r.InstanceUniqueName, r.Alarm.AlarmName)));
|
||||
|
||||
// The live path can't compute "not reporting" — always empty.
|
||||
Assert.Empty(result.NotReportingInstances);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void BuildFromLiveAlarms_EmptySnapshot_YieldsNoRows()
|
||||
{
|
||||
var result = CreateSut().BuildFromLiveAlarms(Array.Empty<AlarmStateChanged>());
|
||||
|
||||
Assert.Empty(result.Alarms);
|
||||
Assert.Empty(result.NotReportingInstances);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -36,4 +36,54 @@ public class CommunicationOptionsValidatorTests
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("GrpcMaxConcurrentStreams", result.FailureMessage);
|
||||
}
|
||||
|
||||
// ── Aggregated live alarm cache options (plan #10, Task 6) ───────────────────
|
||||
|
||||
[Fact]
|
||||
public void ZeroLiveAlarmCacheLinger_IsValid()
|
||||
{
|
||||
// Zero linger = stop the aggregator immediately when the last viewer leaves.
|
||||
var result = Validate(new CommunicationOptions { LiveAlarmCacheLinger = TimeSpan.Zero });
|
||||
Assert.True(result.Succeeded, result.FailureMessage);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void NegativeLiveAlarmCacheLinger_IsRejected()
|
||||
{
|
||||
var result = Validate(new CommunicationOptions { LiveAlarmCacheLinger = TimeSpan.FromSeconds(-1) });
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("LiveAlarmCacheLinger", result.FailureMessage);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ZeroLiveAlarmCacheReconcileInterval_IsRejected()
|
||||
{
|
||||
var result = Validate(new CommunicationOptions { LiveAlarmCacheReconcileInterval = TimeSpan.Zero });
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("LiveAlarmCacheReconcileInterval", result.FailureMessage);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ZeroLiveAlarmCacheSeedConcurrency_IsRejected()
|
||||
{
|
||||
var result = Validate(new CommunicationOptions { LiveAlarmCacheSeedConcurrency = 0 });
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("LiveAlarmCacheSeedConcurrency", result.FailureMessage);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ExcessiveLiveAlarmCacheSeedConcurrency_IsRejected()
|
||||
{
|
||||
var result = Validate(new CommunicationOptions { LiveAlarmCacheSeedConcurrency = 65 });
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("LiveAlarmCacheSeedConcurrency", result.FailureMessage);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ZeroLiveAlarmCacheMaxSubscribersPerSite_IsRejected()
|
||||
{
|
||||
var result = Validate(new CommunicationOptions { LiveAlarmCacheMaxSubscribersPerSite = 0 });
|
||||
Assert.True(result.Failed);
|
||||
Assert.Contains("LiveAlarmCacheMaxSubscribersPerSite", result.FailureMessage);
|
||||
}
|
||||
}
|
||||
|
||||
+414
@@ -0,0 +1,414 @@
|
||||
using System.Collections.Concurrent;
|
||||
using Akka.Actor;
|
||||
using Akka.TestKit.Xunit2;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests.Grpc;
|
||||
|
||||
/// <summary>
|
||||
/// Tests for <see cref="SiteAlarmAggregatorActor"/> (plan #10, Task 4) — the transient,
|
||||
/// in-memory, per-site live alarm aggregator. Covers the tricky bits: seed-then-stream
|
||||
/// ordering (a delta arriving before the seed completes is neither lost nor double-applied),
|
||||
/// dedup by AlarmKey, placeholder seeded-but-not-on-stream coherence, NodeA↔NodeB reconnect
|
||||
/// re-seed, and periodic reconcile authoritative-replace (drift correction).
|
||||
/// </summary>
|
||||
public class SiteAlarmAggregatorActorTests : TestKit
|
||||
{
|
||||
private const string SiteId = "site-alpha";
|
||||
private const string GrpcNodeA = "http://localhost:5100";
|
||||
private const string GrpcNodeB = "http://localhost:5200";
|
||||
private const string Instance = "Site1.Pump01";
|
||||
|
||||
public SiteAlarmAggregatorActorTests() : base(@"akka.loglevel = WARNING")
|
||||
{
|
||||
SiteAlarmAggregatorActor.ReconnectDelay = TimeSpan.FromMilliseconds(50);
|
||||
SiteAlarmAggregatorActor.StabilityWindow = TimeSpan.FromSeconds(30);
|
||||
}
|
||||
|
||||
// ── Test doubles ────────────────────────────────────────────────────────────
|
||||
|
||||
/// <summary>
|
||||
/// Controllable seed fan-out: each invocation returns a fresh Task the test completes
|
||||
/// on demand (so seed/reconcile timing relative to buffered deltas is deterministic).
|
||||
/// </summary>
|
||||
private sealed class SeedStub
|
||||
{
|
||||
private readonly ConcurrentQueue<TaskCompletionSource<IReadOnlyList<AlarmStateChanged>>> _pending = new();
|
||||
private int _callCount;
|
||||
|
||||
public int CallCount => Volatile.Read(ref _callCount);
|
||||
|
||||
public Task<IReadOnlyList<AlarmStateChanged>> Seed(CancellationToken ct)
|
||||
{
|
||||
Interlocked.Increment(ref _callCount);
|
||||
var tcs = new TaskCompletionSource<IReadOnlyList<AlarmStateChanged>>(
|
||||
TaskCreationOptions.RunContinuationsAsynchronously);
|
||||
ct.Register(() => tcs.TrySetCanceled());
|
||||
_pending.Enqueue(tcs);
|
||||
return tcs.Task;
|
||||
}
|
||||
|
||||
/// <summary>Completes the oldest not-yet-completed seed call with the given rows.</summary>
|
||||
public void CompleteNext(params AlarmStateChanged[] rows)
|
||||
{
|
||||
SpinWait.SpinUntil(() => _pending.TryPeek(out _), TimeSpan.FromSeconds(3));
|
||||
if (_pending.TryDequeue(out var tcs))
|
||||
tcs.TrySetResult(rows);
|
||||
else
|
||||
throw new InvalidOperationException("No pending seed to complete.");
|
||||
}
|
||||
|
||||
/// <summary>Faults the oldest not-yet-completed seed call (simulates a whole-fan-out failure).</summary>
|
||||
public void FaultNext()
|
||||
{
|
||||
SpinWait.SpinUntil(() => _pending.TryPeek(out _), TimeSpan.FromSeconds(3));
|
||||
if (_pending.TryDequeue(out var tcs))
|
||||
tcs.TrySetException(new InvalidOperationException("seed boom"));
|
||||
else
|
||||
throw new InvalidOperationException("No pending seed to fault.");
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class PublishSink
|
||||
{
|
||||
private readonly object _lock = new();
|
||||
public List<IReadOnlyList<AlarmStateChanged>> Snapshots { get; } = new();
|
||||
|
||||
public void Publish(IReadOnlyList<AlarmStateChanged> snapshot)
|
||||
{
|
||||
lock (_lock) { Snapshots.Add(snapshot); }
|
||||
}
|
||||
|
||||
public IReadOnlyList<AlarmStateChanged>? Latest
|
||||
{
|
||||
get { lock (_lock) { return Snapshots.Count == 0 ? null : Snapshots[^1]; } }
|
||||
}
|
||||
|
||||
public int Count { get { lock (_lock) { return Snapshots.Count; } } }
|
||||
}
|
||||
|
||||
private sealed record SiteSub(
|
||||
string CorrelationId, Action<AlarmStateChanged> OnAlarm, Action<Exception> OnError, CancellationToken Ct);
|
||||
|
||||
private sealed class MockSiteAlarmStreamClient : SiteStreamGrpcClient
|
||||
{
|
||||
private readonly object _lock = new();
|
||||
private readonly List<SiteSub> _subs = new();
|
||||
private readonly List<string> _unsubscribed = new();
|
||||
|
||||
public List<SiteSub> Subs { get { lock (_lock) { return _subs.ToList(); } } }
|
||||
public List<string> Unsubscribed { get { lock (_lock) { return _unsubscribed.ToList(); } } }
|
||||
|
||||
public MockSiteAlarmStreamClient() : base() { }
|
||||
|
||||
public override Task SubscribeSiteAsync(
|
||||
string correlationId, Action<AlarmStateChanged> onAlarmEvent, Action<Exception> onError, CancellationToken ct)
|
||||
{
|
||||
lock (_lock) { _subs.Add(new SiteSub(correlationId, onAlarmEvent, onError, ct)); }
|
||||
var tcs = new TaskCompletionSource();
|
||||
ct.Register(() => tcs.TrySetResult());
|
||||
return tcs.Task; // never completes until cancelled (simulates a live stream)
|
||||
}
|
||||
|
||||
public override void Unsubscribe(string correlationId)
|
||||
{
|
||||
lock (_lock) { _unsubscribed.Add(correlationId); }
|
||||
}
|
||||
}
|
||||
|
||||
private sealed class MockSiteAlarmStreamClientFactory : SiteStreamGrpcClientFactory
|
||||
{
|
||||
private readonly ConcurrentDictionary<string, MockSiteAlarmStreamClient> _byEndpoint = new();
|
||||
|
||||
public MockSiteAlarmStreamClientFactory()
|
||||
: base(Microsoft.Extensions.Logging.Abstractions.NullLoggerFactory.Instance) { }
|
||||
|
||||
public MockSiteAlarmStreamClient ClientFor(string endpoint) =>
|
||||
_byEndpoint.GetOrAdd(endpoint, _ => new MockSiteAlarmStreamClient());
|
||||
|
||||
public override SiteStreamGrpcClient GetOrCreate(string siteIdentifier, string grpcEndpoint)
|
||||
=> ClientFor(grpcEndpoint);
|
||||
|
||||
public override SiteStreamGrpcClient? TryGet(string siteIdentifier, string grpcEndpoint)
|
||||
=> _byEndpoint.TryGetValue(grpcEndpoint, out var c) ? c : null;
|
||||
}
|
||||
|
||||
private (IActorRef Actor, SeedStub Seed, PublishSink Sink, MockSiteAlarmStreamClientFactory Factory) CreateActor(
|
||||
TimeSpan? reconcileInterval = null)
|
||||
{
|
||||
var seed = new SeedStub();
|
||||
var sink = new PublishSink();
|
||||
var factory = new MockSiteAlarmStreamClientFactory();
|
||||
|
||||
var props = Props.Create(() => new SiteAlarmAggregatorActor(
|
||||
SiteId, "corr-1", seed.Seed, sink.Publish, factory, GrpcNodeA, GrpcNodeB,
|
||||
reconcileInterval ?? TimeSpan.FromMinutes(10)));
|
||||
|
||||
var actor = Sys.ActorOf(props);
|
||||
return (actor, seed, sink, factory);
|
||||
}
|
||||
|
||||
private static AlarmStateChanged Alarm(
|
||||
string alarmName, string sourceRef, int priority, DateTimeOffset ts,
|
||||
bool placeholder = false, string instance = Instance) =>
|
||||
new(instance, alarmName, AlarmState.Active, priority, ts)
|
||||
{
|
||||
SourceReference = sourceRef,
|
||||
IsConfiguredPlaceholder = placeholder,
|
||||
Kind = placeholder ? AlarmKind.NativeOpcUa : AlarmKind.Computed
|
||||
};
|
||||
|
||||
private static AlarmStateChanged? Find(IReadOnlyList<AlarmStateChanged>? rows, string alarmName, string sourceRef) =>
|
||||
rows?.FirstOrDefault(r => r.AlarmName == alarmName && r.SourceReference == sourceRef);
|
||||
|
||||
// ── Tests ───────────────────────────────────────────────────────────────────
|
||||
|
||||
[Fact]
|
||||
public void Delta_Arriving_Before_Seed_Completes_Is_Buffered_And_Applied_Exactly_Once()
|
||||
{
|
||||
var (actor, seed, sink, _) = CreateActor();
|
||||
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
||||
|
||||
// Live delta arrives DURING the seed window (new key, not in the seed).
|
||||
var t1 = DateTimeOffset.UtcNow;
|
||||
actor.Tell(Alarm("Overheat", "", 900, t1));
|
||||
|
||||
// Still buffering — nothing published yet.
|
||||
Assert.Equal(0, sink.Count);
|
||||
|
||||
// Seed completes with a DIFFERENT alarm.
|
||||
seed.CompleteNext(Alarm("PumpFault", "", 500, t1.AddSeconds(-1)));
|
||||
|
||||
AwaitCondition(() => sink.Latest is { } l && l.Count == 2, TimeSpan.FromSeconds(3));
|
||||
var latest = sink.Latest!;
|
||||
Assert.NotNull(Find(latest, "PumpFault", "")); // seeded
|
||||
var overheat = Find(latest, "Overheat", ""); // buffered gap-window delta survived
|
||||
Assert.NotNull(overheat);
|
||||
Assert.Single(latest, r => r.AlarmName == "Overheat"); // exactly once
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Buffered_Delta_Older_Than_Seed_Entry_Is_Dropped()
|
||||
{
|
||||
var (actor, seed, sink, _) = CreateActor();
|
||||
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
||||
|
||||
var tSeed = DateTimeOffset.UtcNow;
|
||||
// Buffered delta for the SAME key but OLDER than the seed entry.
|
||||
actor.Tell(Alarm("PumpFault", "", 100, tSeed.AddSeconds(-1)));
|
||||
seed.CompleteNext(Alarm("PumpFault", "", 500, tSeed));
|
||||
|
||||
AwaitCondition(() => sink.Count >= 1, TimeSpan.FromSeconds(3));
|
||||
// Give a beat to ensure no late (dropped) delta sneaks in.
|
||||
Thread.Sleep(150);
|
||||
var latest = sink.Latest!;
|
||||
var pumpFault = Find(latest, "PumpFault", "");
|
||||
Assert.NotNull(pumpFault);
|
||||
Assert.Equal(500, pumpFault!.Priority); // seed value kept; older buffered delta dropped
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Buffered_Delta_Newer_Than_Seed_Entry_Is_Applied()
|
||||
{
|
||||
var (actor, seed, sink, _) = CreateActor();
|
||||
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
||||
|
||||
var tSeed = DateTimeOffset.UtcNow;
|
||||
actor.Tell(Alarm("PumpFault", "", 900, tSeed.AddSeconds(1))); // newer than seed
|
||||
seed.CompleteNext(Alarm("PumpFault", "", 500, tSeed));
|
||||
|
||||
AwaitCondition(() => sink.Latest is { } l && Find(l, "PumpFault", "")?.Priority == 900,
|
||||
TimeSpan.FromSeconds(3));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Dedup_By_AlarmKey_Distinguishes_SourceReference()
|
||||
{
|
||||
var (actor, seed, sink, _) = CreateActor();
|
||||
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
||||
seed.CompleteNext(); // empty seed
|
||||
AwaitCondition(() => sink.Count >= 1, TimeSpan.FromSeconds(3));
|
||||
|
||||
var t = DateTimeOffset.UtcNow;
|
||||
// Same instance + alarm name, different source reference → two distinct rows.
|
||||
actor.Tell(Alarm("LevelAlarm", "Tank01.Level.HiHi", 700, t));
|
||||
actor.Tell(Alarm("LevelAlarm", "Tank01.Level.LoLo", 300, t));
|
||||
|
||||
AwaitCondition(() => sink.Latest is { } l && l.Count == 2, TimeSpan.FromSeconds(3));
|
||||
var latest = sink.Latest!;
|
||||
Assert.NotNull(Find(latest, "LevelAlarm", "Tank01.Level.HiHi"));
|
||||
Assert.NotNull(Find(latest, "LevelAlarm", "Tank01.Level.LoLo"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Same_Key_Live_Update_Replaces_In_Place()
|
||||
{
|
||||
var (actor, seed, sink, _) = CreateActor();
|
||||
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
||||
seed.CompleteNext();
|
||||
AwaitCondition(() => sink.Count >= 1, TimeSpan.FromSeconds(3));
|
||||
|
||||
var t = DateTimeOffset.UtcNow;
|
||||
actor.Tell(Alarm("PumpFault", "", 500, t));
|
||||
AwaitCondition(() => Find(sink.Latest, "PumpFault", "")?.Priority == 500, TimeSpan.FromSeconds(3));
|
||||
|
||||
// Newer update for the same key replaces (not duplicates).
|
||||
actor.Tell(Alarm("PumpFault", "", 800, t.AddSeconds(1)));
|
||||
AwaitCondition(() => Find(sink.Latest, "PumpFault", "")?.Priority == 800, TimeSpan.FromSeconds(3));
|
||||
Assert.Single(sink.Latest!); // still exactly one row for the key
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Seeded_Native_Alarm_And_Its_Live_Delta_Collapse_Onto_One_Key()
|
||||
{
|
||||
// Identity parity across the two paths (snapshot seed vs live stream): a native
|
||||
// alarm seeded with a non-empty SourceReference and a later live delta for the
|
||||
// SAME (instance, alarmName, sourceRef) must collapse to ONE in-place-updated row,
|
||||
// not a ghost duplicate. Both paths key on SourceReference, so they must agree.
|
||||
var (actor, seed, sink, _) = CreateActor();
|
||||
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
||||
|
||||
var t = DateTimeOffset.UtcNow;
|
||||
seed.CompleteNext(Alarm("Motor1.MotorAlarms", "Motor1.MotorAlarms.HiHi", 300, t));
|
||||
AwaitCondition(() => Find(sink.Latest, "Motor1.MotorAlarms", "Motor1.MotorAlarms.HiHi") is not null,
|
||||
TimeSpan.FromSeconds(3));
|
||||
|
||||
// Live delta for the exact same identity, newer → updates in place.
|
||||
actor.Tell(Alarm("Motor1.MotorAlarms", "Motor1.MotorAlarms.HiHi", 800, t.AddSeconds(1)));
|
||||
|
||||
AwaitCondition(
|
||||
() => Find(sink.Latest, "Motor1.MotorAlarms", "Motor1.MotorAlarms.HiHi")?.Priority == 800,
|
||||
TimeSpan.FromSeconds(3));
|
||||
Assert.Single(sink.Latest!); // exactly one row — no seed/live ghost duplicate
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Placeholder_Seeded_Coexists_With_Live_Real_Alarm_And_Is_Not_Wiped()
|
||||
{
|
||||
var (actor, seed, sink, _) = CreateActor();
|
||||
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
||||
|
||||
var t = DateTimeOffset.UtcNow;
|
||||
// Seed carries a configured-placeholder row (binding with no active conditions).
|
||||
var placeholder = Alarm("Motor1.MotorAlarms", "", 0, t, placeholder: true);
|
||||
seed.CompleteNext(placeholder);
|
||||
AwaitCondition(() => sink.Count >= 1, TimeSpan.FromSeconds(3));
|
||||
|
||||
// A live REAL condition for the same instance arrives (distinct AlarmKey).
|
||||
actor.Tell(Alarm("Motor1.MotorAlarms", "Motor1.MotorAlarms.HiHi", 800, t.AddSeconds(1)));
|
||||
|
||||
AwaitCondition(() => sink.Latest is { } l && l.Count == 2, TimeSpan.FromSeconds(3));
|
||||
var latest = sink.Latest!;
|
||||
// Placeholder still present (not wiped by the real alarm under a different key).
|
||||
var ph = Find(latest, "Motor1.MotorAlarms", "");
|
||||
Assert.NotNull(ph);
|
||||
Assert.True(ph!.IsConfiguredPlaceholder);
|
||||
Assert.NotNull(Find(latest, "Motor1.MotorAlarms", "Motor1.MotorAlarms.HiHi"));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void GrpcError_Flips_Node_Reconnects_And_ReSeeds()
|
||||
{
|
||||
var (_, seed, sink, factory) = CreateActor();
|
||||
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
||||
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
||||
seed.CompleteNext(); // initial seed done
|
||||
// Wait for the seed to actually be APPLIED (published) so the re-seed on error is
|
||||
// not skipped as "fan-out already in flight".
|
||||
AwaitCondition(() => sink.Count >= 1, TimeSpan.FromSeconds(3));
|
||||
|
||||
// Fail the NodeA stream.
|
||||
factory.ClientFor(GrpcNodeA).Subs[0].OnError(new Exception("NodeA down"));
|
||||
|
||||
// Reconnect must reach NodeB, AND a re-seed fan-out must have been triggered.
|
||||
AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 1, TimeSpan.FromSeconds(5));
|
||||
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(5));
|
||||
// The failed NodeA stream was unsubscribed (relay released, not left zombie).
|
||||
Assert.Contains("corr-1", factory.ClientFor(GrpcNodeA).Unsubscribed);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Periodic_Reconcile_Authoritatively_Replaces_Stale_Rows()
|
||||
{
|
||||
var (_, seed, sink, _) = CreateActor(reconcileInterval: TimeSpan.FromMilliseconds(250));
|
||||
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
||||
|
||||
var t = DateTimeOffset.UtcNow;
|
||||
seed.CompleteNext(Alarm("PumpFault", "", 500, t)); // seed has a row
|
||||
AwaitCondition(() => Find(sink.Latest, "PumpFault", "") is not null, TimeSpan.FromSeconds(3));
|
||||
|
||||
// Reconcile tick fires → seed #2 (e.g. instance was disabled) returns EMPTY.
|
||||
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3));
|
||||
seed.CompleteNext(); // empty
|
||||
|
||||
// The stale PumpFault row must disappear (authoritative replace, not merge).
|
||||
AwaitCondition(() => sink.Latest is { } l && l.Count == 0, TimeSpan.FromSeconds(3));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Failed_Initial_Seed_Does_Not_Publish_But_A_Later_Reconcile_Recovers()
|
||||
{
|
||||
var (_, seed, sink, _) = CreateActor(reconcileInterval: TimeSpan.FromMilliseconds(250));
|
||||
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
||||
|
||||
// The initial seed fan-out throws as a whole.
|
||||
seed.FaultNext();
|
||||
|
||||
// A failed INITIAL seed must not publish (page keeps its poll fallback — not live yet).
|
||||
Thread.Sleep(200);
|
||||
Assert.Equal(0, sink.Count);
|
||||
|
||||
// The periodic reconcile is the backstop: it re-runs the fan-out (flag was cleared
|
||||
// on failure), completes, and the actor publishes → now live.
|
||||
AwaitCondition(() => seed.CallCount == 2, TimeSpan.FromSeconds(3));
|
||||
seed.CompleteNext(Alarm("PumpFault", "", 500, DateTimeOffset.UtcNow));
|
||||
AwaitCondition(() => Find(sink.Latest, "PumpFault", "") is not null, TimeSpan.FromSeconds(3));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Stream_GivenUp_After_MaxRetries_Reopens_On_Reconcile_Tick()
|
||||
{
|
||||
// After the retry budget is exhausted the live stream is left down, but the
|
||||
// aggregator is NOT stopped — the next reconcile tick self-heals it (resets the
|
||||
// budget + reopens) so a sustained site outage never permanently kills the feed.
|
||||
var (_, seed, _, factory) = CreateActor(reconcileInterval: TimeSpan.FromMilliseconds(300));
|
||||
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
||||
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
||||
|
||||
// Drive initial + 3 retries (each flips node) to exhaust the budget.
|
||||
factory.ClientFor(GrpcNodeA).Subs[0].OnError(new Exception("1"));
|
||||
AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 1, TimeSpan.FromSeconds(5));
|
||||
factory.ClientFor(GrpcNodeB).Subs[0].OnError(new Exception("2"));
|
||||
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 2, TimeSpan.FromSeconds(5));
|
||||
factory.ClientFor(GrpcNodeA).Subs[1].OnError(new Exception("3"));
|
||||
AwaitCondition(() => factory.ClientFor(GrpcNodeB).Subs.Count == 2, TimeSpan.FromSeconds(5));
|
||||
|
||||
int TotalSubs() => factory.ClientFor(GrpcNodeA).Subs.Count + factory.ClientFor(GrpcNodeB).Subs.Count;
|
||||
var before = TotalSubs(); // 4
|
||||
|
||||
// Fourth error exceeds MaxRetries → stream given up (no immediate reopen).
|
||||
factory.ClientFor(GrpcNodeB).Subs[1].OnError(new Exception("4"));
|
||||
|
||||
// The reconcile tick reopens the stream (self-heal).
|
||||
AwaitCondition(() => TotalSubs() > before, TimeSpan.FromSeconds(3));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Stop_Message_TearsDown_Grpc_And_Stops_Actor()
|
||||
{
|
||||
var (actor, seed, _, factory) = CreateActor();
|
||||
AwaitCondition(() => seed.CallCount == 1, TimeSpan.FromSeconds(3));
|
||||
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Subs.Count == 1, TimeSpan.FromSeconds(3));
|
||||
seed.CompleteNext();
|
||||
|
||||
Watch(actor);
|
||||
actor.Tell(new StopSiteAlarmAggregator());
|
||||
|
||||
ExpectTerminated(actor, TimeSpan.FromSeconds(3));
|
||||
AwaitCondition(() => factory.ClientFor(GrpcNodeA).Unsubscribed.Contains("corr-1"),
|
||||
TimeSpan.FromSeconds(3));
|
||||
}
|
||||
}
|
||||
@@ -208,6 +208,78 @@ public class SiteStreamGrpcClientTests
|
||||
Assert.True(cts2.IsCancellationRequested);
|
||||
}
|
||||
|
||||
// --- Site-wide (SubscribeSite) alarm-only stream (plan #10 T3) ---
|
||||
|
||||
[Fact]
|
||||
public void ConvertToAlarmEvent_AlarmChanged_ReturnsMappedAlarm()
|
||||
{
|
||||
// An alarm event on the site-wide stream is delivered to onAlarmEvent with full enrichment.
|
||||
var ts = DateTimeOffset.UtcNow;
|
||||
var evt = new SiteStreamEvent
|
||||
{
|
||||
CorrelationId = "site-corr",
|
||||
AlarmChanged = new AlarmStateUpdate
|
||||
{
|
||||
InstanceUniqueName = "Site1.Motor01",
|
||||
AlarmName = "T01.Hi",
|
||||
State = AlarmStateEnum.AlarmStateActive,
|
||||
Priority = 700,
|
||||
Timestamp = Timestamp.FromDateTimeOffset(ts),
|
||||
Kind = "NativeOpcUa",
|
||||
Active = true,
|
||||
SourceReference = "T01.Hi"
|
||||
}
|
||||
};
|
||||
|
||||
var alarm = SiteStreamGrpcClient.ConvertToAlarmEvent(evt);
|
||||
|
||||
Assert.NotNull(alarm);
|
||||
Assert.Equal("Site1.Motor01", alarm!.InstanceUniqueName);
|
||||
Assert.Equal("T01.Hi", alarm.AlarmName);
|
||||
Assert.Equal(AlarmState.Active, alarm.State);
|
||||
Assert.Equal(AlarmKind.NativeOpcUa, alarm.Kind);
|
||||
Assert.Equal("T01.Hi", alarm.SourceReference);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ConvertToAlarmEvent_AttributeChanged_ReturnsNull()
|
||||
{
|
||||
// Attribute events must never appear on the alarm-only site-wide stream; if one
|
||||
// somehow arrives it is defensively filtered out rather than delivered or thrown.
|
||||
var evt = new SiteStreamEvent
|
||||
{
|
||||
CorrelationId = "site-corr",
|
||||
AttributeChanged = new AttributeValueUpdate
|
||||
{
|
||||
InstanceUniqueName = "Site1.Pump01",
|
||||
AttributePath = "Modules.IO",
|
||||
AttributeName = "Temperature",
|
||||
Value = "42.5",
|
||||
Quality = Quality.Good,
|
||||
Timestamp = Timestamp.FromDateTimeOffset(DateTimeOffset.UtcNow)
|
||||
}
|
||||
};
|
||||
|
||||
Assert.Null(SiteStreamGrpcClient.ConvertToAlarmEvent(evt));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ConvertToAlarmEvent_UnknownEvent_ReturnsNull()
|
||||
{
|
||||
var evt = new SiteStreamEvent { CorrelationId = "site-corr" };
|
||||
Assert.Null(SiteStreamGrpcClient.ConvertToAlarmEvent(evt));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SubscribeSiteAsync_OnTestOnlyClient_Throws()
|
||||
{
|
||||
// Guards against subscribing on a channel-less test double, mirroring SubscribeAsync.
|
||||
var client = SiteStreamGrpcClient.CreateForTesting();
|
||||
|
||||
await Assert.ThrowsAsync<InvalidOperationException>(() =>
|
||||
client.SubscribeSiteAsync("corr", _ => { }, _ => { }, CancellationToken.None));
|
||||
}
|
||||
|
||||
// --- Communication-003 regression tests ---
|
||||
|
||||
[Fact]
|
||||
|
||||
@@ -21,6 +21,8 @@ public class SiteStreamGrpcServerTests : TestKit
|
||||
_subscriber = Substitute.For<ISiteStreamSubscriber>();
|
||||
_subscriber.Subscribe(Arg.Any<string>(), Arg.Any<IActorRef>())
|
||||
.Returns("sub-1");
|
||||
_subscriber.SubscribeSiteAlarms(Arg.Any<IActorRef>())
|
||||
.Returns("site-sub-1");
|
||||
_logger = NullLogger<SiteStreamGrpcServer>.Instance;
|
||||
}
|
||||
|
||||
@@ -266,6 +268,102 @@ public class SiteStreamGrpcServerTests : TestKit
|
||||
await streamTask;
|
||||
}
|
||||
|
||||
// --- SubscribeSite (site-wide, alarm-only aggregated stream, plan #10 T2) ---
|
||||
|
||||
private static SiteStreamRequest MakeSiteRequest(string correlationId = "site-corr-1")
|
||||
=> new() { CorrelationId = correlationId };
|
||||
|
||||
[Fact]
|
||||
public async Task SubscribeSite_SubscribesSiteAlarmsAndRemovesOnCancel()
|
||||
{
|
||||
var server = CreateServer();
|
||||
server.SetReady(Sys);
|
||||
|
||||
var cts = new CancellationTokenSource();
|
||||
var context = CreateMockContext(cts.Token);
|
||||
var writer = Substitute.For<IServerStreamWriter<SiteStreamEvent>>();
|
||||
|
||||
var streamTask = Task.Run(() => server.SubscribeSite(
|
||||
MakeSiteRequest("site-corr-sub"), writer, context));
|
||||
|
||||
await WaitForConditionAsync(() => server.ActiveStreamCount == 1);
|
||||
|
||||
// Site-wide handler must call SubscribeSiteAlarms (no instance filter),
|
||||
// never the per-instance Subscribe.
|
||||
_subscriber.Received(1).SubscribeSiteAlarms(Arg.Any<IActorRef>());
|
||||
_subscriber.DidNotReceive().Subscribe(Arg.Any<string>(), Arg.Any<IActorRef>());
|
||||
|
||||
cts.Cancel();
|
||||
await streamTask;
|
||||
|
||||
_subscriber.Received(1).RemoveSubscriber(Arg.Any<IActorRef>());
|
||||
Assert.Equal(0, server.ActiveStreamCount);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SubscribeSite_RejectsUnsafeCorrelationId()
|
||||
{
|
||||
var server = CreateServer();
|
||||
server.SetReady(Sys);
|
||||
|
||||
var writer = Substitute.For<IServerStreamWriter<SiteStreamEvent>>();
|
||||
var context = CreateMockContext();
|
||||
|
||||
var ex = await Assert.ThrowsAsync<RpcException>(
|
||||
() => server.SubscribeSite(MakeSiteRequest("bad/id"), writer, context));
|
||||
|
||||
Assert.Equal(StatusCode.InvalidArgument, ex.StatusCode);
|
||||
Assert.Equal(0, server.ActiveStreamCount);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task SubscribeSite_RelaysAlarmStateChangedAsAlarmStateUpdate()
|
||||
{
|
||||
var server = CreateServer();
|
||||
server.SetReady(Sys);
|
||||
|
||||
// Capture the relay actor spawned for the site-wide subscription.
|
||||
IActorRef? capturedActor = null;
|
||||
_subscriber.SubscribeSiteAlarms(Arg.Any<IActorRef>())
|
||||
.Returns(ci =>
|
||||
{
|
||||
capturedActor = ci.Arg<IActorRef>();
|
||||
return "site-sub-relay";
|
||||
});
|
||||
|
||||
var cts = new CancellationTokenSource();
|
||||
var context = CreateMockContext(cts.Token);
|
||||
var writer = Substitute.For<IServerStreamWriter<SiteStreamEvent>>();
|
||||
var writtenEvents = new List<SiteStreamEvent>();
|
||||
writer.WriteAsync(Arg.Any<SiteStreamEvent>(), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.CompletedTask)
|
||||
.AndDoes(ci => writtenEvents.Add(ci.Arg<SiteStreamEvent>()));
|
||||
|
||||
var streamTask = Task.Run(() => server.SubscribeSite(
|
||||
MakeSiteRequest("site-corr-write"), writer, context));
|
||||
|
||||
await WaitForConditionAsync(() => capturedActor != null);
|
||||
|
||||
// A real alarm transition for ANY instance must arrive as an AlarmStateUpdate.
|
||||
capturedActor!.Tell(new Commons.Messages.Streaming.AlarmStateChanged(
|
||||
"Site1.Pump01",
|
||||
"HighPressure",
|
||||
Commons.Types.Enums.AlarmState.Active,
|
||||
700,
|
||||
DateTimeOffset.UtcNow));
|
||||
|
||||
await WaitForConditionAsync(() => writtenEvents.Count >= 1);
|
||||
|
||||
Assert.Single(writtenEvents);
|
||||
Assert.Equal("site-corr-write", writtenEvents[0].CorrelationId);
|
||||
Assert.Equal(SiteStreamEvent.EventOneofCase.AlarmChanged, writtenEvents[0].EventCase);
|
||||
Assert.Equal("Site1.Pump01", writtenEvents[0].AlarmChanged.InstanceUniqueName);
|
||||
Assert.Equal("HighPressure", writtenEvents[0].AlarmChanged.AlarmName);
|
||||
|
||||
cts.Cancel();
|
||||
await streamTask;
|
||||
}
|
||||
|
||||
[Theory]
|
||||
[InlineData("corr/with/slash")]
|
||||
[InlineData("corr with space")]
|
||||
|
||||
@@ -0,0 +1,261 @@
|
||||
using Akka.TestKit.Xunit2;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Microsoft.Extensions.Options;
|
||||
using NSubstitute;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Instances;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Entities.Sites;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Interfaces.Repositories;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Tests for <see cref="SiteAlarmLiveCacheService"/> (plan #10, Task 4) — the DI singleton
|
||||
/// that reference-counts Alarm Summary viewers over one shared per-site aggregator. Covers
|
||||
/// shared start (one aggregator for many viewers), last-out linger stop, and re-subscribe
|
||||
/// cancelling a pending stop. Uses a real (in-memory) ActorSystem and a mock gRPC factory
|
||||
/// whose site-wide stream never faults, so no reconnect noise.
|
||||
/// </summary>
|
||||
public class SiteAlarmLiveCacheServiceTests : TestKit
|
||||
{
|
||||
private const int SiteId = 7;
|
||||
|
||||
// A mock site-wide alarm stream client whose subscription hangs until cancelled.
|
||||
private sealed class HangingClient : SiteStreamGrpcClient
|
||||
{
|
||||
public HangingClient() : base() { }
|
||||
public override Task SubscribeSiteAsync(
|
||||
string correlationId, Action<Commons.Messages.Streaming.AlarmStateChanged> onAlarmEvent,
|
||||
Action<Exception> onError, CancellationToken ct)
|
||||
{
|
||||
var tcs = new TaskCompletionSource();
|
||||
ct.Register(() => tcs.TrySetResult());
|
||||
return tcs.Task;
|
||||
}
|
||||
public override void Unsubscribe(string correlationId) { }
|
||||
}
|
||||
|
||||
private sealed class CountingFactory : SiteStreamGrpcClientFactory
|
||||
{
|
||||
private readonly HangingClient _client = new();
|
||||
public int GetOrCreateCount;
|
||||
|
||||
public CountingFactory() : base(NullLoggerFactory.Instance) { }
|
||||
|
||||
public override SiteStreamGrpcClient GetOrCreate(string siteIdentifier, string grpcEndpoint)
|
||||
{
|
||||
Interlocked.Increment(ref GetOrCreateCount);
|
||||
return _client;
|
||||
}
|
||||
|
||||
public override SiteStreamGrpcClient? TryGet(string siteIdentifier, string grpcEndpoint) => _client;
|
||||
}
|
||||
|
||||
private SiteAlarmLiveCacheService CreateService(TimeSpan linger, out CountingFactory factory,
|
||||
int maxSubscribersPerSite = 200)
|
||||
{
|
||||
// Site with gRPC addresses, and NO enabled instances → the seed fan-out returns
|
||||
// empty immediately (so IsLive flips true fast without any snapshot Asks).
|
||||
var site = new Site("Alpha", "site-alpha")
|
||||
{
|
||||
Id = SiteId,
|
||||
GrpcNodeAAddress = "http://localhost:5100",
|
||||
GrpcNodeBAddress = "http://localhost:5200"
|
||||
};
|
||||
|
||||
var siteRepo = Substitute.For<ISiteRepository>();
|
||||
siteRepo.GetSiteByIdAsync(SiteId, Arg.Any<CancellationToken>()).Returns(site);
|
||||
|
||||
var instanceRepo = Substitute.For<ITemplateEngineRepository>();
|
||||
instanceRepo.GetInstancesBySiteIdAsync(SiteId, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<Instance>());
|
||||
|
||||
var services = new ServiceCollection();
|
||||
services.AddScoped(_ => siteRepo);
|
||||
services.AddScoped(_ => instanceRepo);
|
||||
var provider = services.BuildServiceProvider();
|
||||
|
||||
var options = Options.Create(new CommunicationOptions
|
||||
{
|
||||
LiveAlarmCacheLinger = linger,
|
||||
LiveAlarmCacheReconcileInterval = TimeSpan.FromMinutes(10),
|
||||
LiveAlarmCacheMaxSubscribersPerSite = maxSubscribersPerSite
|
||||
});
|
||||
|
||||
var comm = new CommunicationService(Options.Create(new CommunicationOptions()),
|
||||
NullLogger<CommunicationService>.Instance);
|
||||
|
||||
factory = new CountingFactory();
|
||||
var service = new SiteAlarmLiveCacheService(
|
||||
provider, comm, factory, options, NullLogger<SiteAlarmLiveCacheService>.Instance);
|
||||
service.SetActorSystem(Sys);
|
||||
return service;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void First_Subscriber_Starts_One_Aggregator_Shared_By_Multiple_Viewers()
|
||||
{
|
||||
var service = CreateService(TimeSpan.FromMilliseconds(200), out var factory);
|
||||
|
||||
var changes1 = 0;
|
||||
var changes2 = 0;
|
||||
using var sub1 = service.Subscribe(SiteId, () => Interlocked.Increment(ref changes1));
|
||||
using var sub2 = service.Subscribe(SiteId, () => Interlocked.Increment(ref changes2));
|
||||
|
||||
// The (empty) seed completes → aggregator publishes → live.
|
||||
AwaitCondition(() => service.IsLive(SiteId), TimeSpan.FromSeconds(5));
|
||||
|
||||
// Exactly ONE gRPC stream opened for the site regardless of viewer count.
|
||||
Assert.Equal(1, factory.GetOrCreateCount);
|
||||
// Empty seed → empty current snapshot.
|
||||
Assert.Empty(service.GetCurrentAlarms(SiteId));
|
||||
// Both viewers were notified of the initial publish.
|
||||
AwaitCondition(() => changes1 >= 1 && changes2 >= 1, TimeSpan.FromSeconds(3));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Last_Viewer_Leaving_Stops_Aggregator_After_Linger()
|
||||
{
|
||||
var service = CreateService(TimeSpan.FromMilliseconds(200), out _);
|
||||
|
||||
var sub1 = service.Subscribe(SiteId, () => { });
|
||||
var sub2 = service.Subscribe(SiteId, () => { });
|
||||
AwaitCondition(() => service.IsLive(SiteId), TimeSpan.FromSeconds(5));
|
||||
|
||||
sub1.Dispose();
|
||||
// One viewer remains — still live, not stopped.
|
||||
Thread.Sleep(300);
|
||||
Assert.True(service.IsLive(SiteId));
|
||||
|
||||
sub2.Dispose();
|
||||
// Last viewer left → after the linger the aggregator is torn down and the entry cleared.
|
||||
AwaitCondition(() => !service.IsLive(SiteId), TimeSpan.FromSeconds(3));
|
||||
Assert.Empty(service.GetCurrentAlarms(SiteId));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Resubscribe_Within_Linger_Cancels_The_Pending_Stop()
|
||||
{
|
||||
var service = CreateService(TimeSpan.FromMilliseconds(400), out var factory);
|
||||
|
||||
var sub1 = service.Subscribe(SiteId, () => { });
|
||||
AwaitCondition(() => service.IsLive(SiteId), TimeSpan.FromSeconds(5));
|
||||
|
||||
sub1.Dispose(); // schedules linger stop
|
||||
var sub2 = service.Subscribe(SiteId, () => { }); // returns within the linger window → cancels it
|
||||
|
||||
// Wait comfortably past the original linger; the aggregator must still be alive.
|
||||
Thread.Sleep(700);
|
||||
Assert.True(service.IsLive(SiteId));
|
||||
// No second aggregator/stream was opened — the same one was kept warm.
|
||||
Assert.Equal(1, factory.GetOrCreateCount);
|
||||
|
||||
sub2.Dispose();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Dispose_Is_Idempotent()
|
||||
{
|
||||
var service = CreateService(TimeSpan.FromMilliseconds(200), out _);
|
||||
var sub = service.Subscribe(SiteId, () => { });
|
||||
AwaitCondition(() => service.IsLive(SiteId), TimeSpan.FromSeconds(5));
|
||||
|
||||
sub.Dispose();
|
||||
sub.Dispose(); // must not throw or double-decrement
|
||||
|
||||
AwaitCondition(() => !service.IsLive(SiteId), TimeSpan.FromSeconds(3));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Transient_Start_Failure_Self_Heals_On_Retry_For_A_Stable_Viewer_Cohort()
|
||||
{
|
||||
// A transient failure resolving the site at first-subscribe must NOT leave the
|
||||
// viewer cohort polling forever — a bounded retry re-attempts and self-heals.
|
||||
var site = new Site("Alpha", "site-alpha")
|
||||
{
|
||||
Id = SiteId,
|
||||
GrpcNodeAAddress = "http://localhost:5100",
|
||||
GrpcNodeBAddress = "http://localhost:5200"
|
||||
};
|
||||
|
||||
var siteRepo = Substitute.For<ISiteRepository>();
|
||||
// First resolve returns null (transient blip); every resolve thereafter succeeds.
|
||||
siteRepo.GetSiteByIdAsync(SiteId, Arg.Any<CancellationToken>()).Returns((Site?)null, site);
|
||||
|
||||
var instanceRepo = Substitute.For<ITemplateEngineRepository>();
|
||||
instanceRepo.GetInstancesBySiteIdAsync(SiteId, Arg.Any<CancellationToken>())
|
||||
.Returns(new List<Instance>());
|
||||
|
||||
var services = new ServiceCollection();
|
||||
services.AddScoped(_ => siteRepo);
|
||||
services.AddScoped(_ => instanceRepo);
|
||||
var provider = services.BuildServiceProvider();
|
||||
|
||||
var options = Options.Create(new CommunicationOptions
|
||||
{
|
||||
LiveAlarmCacheLinger = TimeSpan.FromSeconds(30),
|
||||
// Retry cadence reuses the reconcile interval — keep it short for the test.
|
||||
LiveAlarmCacheReconcileInterval = TimeSpan.FromMilliseconds(300)
|
||||
});
|
||||
var comm = new CommunicationService(Options.Create(new CommunicationOptions()),
|
||||
NullLogger<CommunicationService>.Instance);
|
||||
var factory = new CountingFactory();
|
||||
|
||||
var service = new SiteAlarmLiveCacheService(
|
||||
provider, comm, factory, options, NullLogger<SiteAlarmLiveCacheService>.Instance);
|
||||
service.SetActorSystem(Sys);
|
||||
|
||||
using var sub = service.Subscribe(SiteId, () => { });
|
||||
|
||||
// First start failed (site not found); a stable single viewer stays subscribed.
|
||||
// The retry eventually resolves the site and the aggregator goes live — without
|
||||
// any further Subscribe call.
|
||||
AwaitCondition(() => service.IsLive(SiteId), TimeSpan.FromSeconds(5));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Viewer_Over_The_Per_Site_Cap_Is_Rejected_Safely_And_Not_Counted()
|
||||
{
|
||||
// Cap of 2: the first two viewers register; a third is rejected fail-safe with a
|
||||
// no-op handle (never throws into the render path, never grows the list). Proof it
|
||||
// was NOT counted: disposing only the two real viewers tears the aggregator down
|
||||
// after the linger — if the rejected viewer had been registered, one subscriber
|
||||
// would remain and the site would stay live forever.
|
||||
var service = CreateService(TimeSpan.FromMilliseconds(200), out _, maxSubscribersPerSite: 2);
|
||||
|
||||
var sub1 = service.Subscribe(SiteId, () => { });
|
||||
var sub2 = service.Subscribe(SiteId, () => { });
|
||||
AwaitCondition(() => service.IsLive(SiteId), TimeSpan.FromSeconds(5));
|
||||
|
||||
// Third viewer is over the cap.
|
||||
var overflow = service.Subscribe(SiteId, () => { });
|
||||
Assert.NotNull(overflow);
|
||||
// Disposing the rejected handle is a genuine no-op — safe, idempotent, touches nothing.
|
||||
overflow.Dispose();
|
||||
overflow.Dispose();
|
||||
|
||||
// Still live with the two real viewers present.
|
||||
Assert.True(service.IsLive(SiteId));
|
||||
|
||||
sub1.Dispose();
|
||||
sub2.Dispose();
|
||||
|
||||
// Both real viewers gone → after the linger the aggregator stops. This only holds
|
||||
// if the overflow viewer was never added to the subscriber list.
|
||||
AwaitCondition(() => !service.IsLive(SiteId), TimeSpan.FromSeconds(3));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Unknown_Site_Registers_Viewer_But_Never_Goes_Live()
|
||||
{
|
||||
var service = CreateService(TimeSpan.FromMilliseconds(200), out _);
|
||||
|
||||
using var sub = service.Subscribe(999, () => { }); // no such site in the repo
|
||||
Thread.Sleep(300);
|
||||
Assert.False(service.IsLive(999));
|
||||
Assert.Empty(service.GetCurrentAlarms(999));
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,94 @@
|
||||
using System.Diagnostics.Metrics;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Observability;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.Communication.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Telemetry guards for the aggregated live alarm cache (plan #10, Task 6): the
|
||||
/// active-aggregator observable gauge reflects Started/Stopped, and the reconnect
|
||||
/// counter increments on each recorded reconnect. Mirrors the MeterListener read
|
||||
/// pattern used by <c>SiteStreamGrpcServerTests.SiteConnectionUpGauge_*</c>; reads are
|
||||
/// relative to a baseline so the process-wide static instruments are robust to any
|
||||
/// parallel test interleaving.
|
||||
/// </summary>
|
||||
public class SiteAlarmLiveCacheTelemetryTests
|
||||
{
|
||||
private static long ReadGauge(string instrumentName)
|
||||
{
|
||||
long observed = 0;
|
||||
using var listener = new MeterListener();
|
||||
listener.InstrumentPublished = (instrument, l) =>
|
||||
{
|
||||
if (instrument.Meter.Name == ScadaBridgeTelemetry.MeterName &&
|
||||
instrument.Name == instrumentName)
|
||||
{
|
||||
l.EnableMeasurementEvents(instrument);
|
||||
}
|
||||
};
|
||||
listener.SetMeasurementEventCallback<long>((_, measurement, _, _) => observed = measurement);
|
||||
listener.Start();
|
||||
listener.RecordObservableInstruments();
|
||||
return observed;
|
||||
}
|
||||
|
||||
private static long ReadCounter(string instrumentName, Action act)
|
||||
{
|
||||
long delta = 0;
|
||||
using var listener = new MeterListener();
|
||||
listener.InstrumentPublished = (instrument, l) =>
|
||||
{
|
||||
if (instrument.Meter.Name == ScadaBridgeTelemetry.MeterName &&
|
||||
instrument.Name == instrumentName)
|
||||
{
|
||||
l.EnableMeasurementEvents(instrument);
|
||||
}
|
||||
};
|
||||
listener.SetMeasurementEventCallback<long>((_, measurement, _, _) => delta += measurement);
|
||||
listener.Start();
|
||||
act();
|
||||
return delta;
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ActiveAggregatorGauge_ReflectsStartAndStop()
|
||||
{
|
||||
const string gauge = "scadabridge.site.alarm_cache.aggregators.active";
|
||||
var baseline = ReadGauge(gauge);
|
||||
|
||||
ScadaBridgeTelemetry.LiveAlarmAggregatorStarted();
|
||||
try
|
||||
{
|
||||
Assert.Equal(baseline + 1, ReadGauge(gauge));
|
||||
|
||||
ScadaBridgeTelemetry.LiveAlarmAggregatorStarted();
|
||||
try
|
||||
{
|
||||
Assert.Equal(baseline + 2, ReadGauge(gauge));
|
||||
}
|
||||
finally
|
||||
{
|
||||
ScadaBridgeTelemetry.LiveAlarmAggregatorStopped();
|
||||
}
|
||||
|
||||
Assert.Equal(baseline + 1, ReadGauge(gauge));
|
||||
}
|
||||
finally
|
||||
{
|
||||
ScadaBridgeTelemetry.LiveAlarmAggregatorStopped();
|
||||
}
|
||||
|
||||
Assert.Equal(baseline, ReadGauge(gauge));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void ReconnectCounter_IncrementsOnEachRecordedReconnect()
|
||||
{
|
||||
var delta = ReadCounter("scadabridge.site.alarm_cache.reconnects", () =>
|
||||
{
|
||||
ScadaBridgeTelemetry.RecordLiveAlarmStreamReconnect();
|
||||
ScadaBridgeTelemetry.RecordLiveAlarmStreamReconnect();
|
||||
});
|
||||
|
||||
Assert.Equal(2, delta);
|
||||
}
|
||||
}
|
||||
@@ -24,6 +24,9 @@ public class ScadaBridgeTelemetryTests
|
||||
ScadaBridgeTelemetry.RecordInboundApiRequest("X");
|
||||
ScadaBridgeTelemetry.SiteConnectionOpened();
|
||||
ScadaBridgeTelemetry.SiteConnectionClosed();
|
||||
ScadaBridgeTelemetry.LiveAlarmAggregatorStarted();
|
||||
ScadaBridgeTelemetry.LiveAlarmAggregatorStopped();
|
||||
ScadaBridgeTelemetry.RecordLiveAlarmStreamReconnect();
|
||||
ScadaBridgeTelemetry.SetQueueDepthProvider(() => 5);
|
||||
});
|
||||
|
||||
|
||||
@@ -0,0 +1,298 @@
|
||||
using System.Collections.Concurrent;
|
||||
using Akka.Actor;
|
||||
using Akka.TestKit.Xunit2;
|
||||
using Google.Protobuf;
|
||||
using Grpc.Core;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using NSubstitute;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Messages.Streaming;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Alarms;
|
||||
using ZB.MOM.WW.ScadaBridge.Commons.Types.Enums;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Actors;
|
||||
using ZB.MOM.WW.ScadaBridge.Communication.Grpc;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime;
|
||||
using ZB.MOM.WW.ScadaBridge.SiteRuntime.Streaming;
|
||||
|
||||
namespace ZB.MOM.WW.ScadaBridge.IntegrationTests.Grpc;
|
||||
|
||||
/// <summary>
|
||||
/// End-to-end trace for the aggregated live alarm stream (plan #10 deferred item, Task 7).
|
||||
///
|
||||
/// Every per-task unit test (T1–T6) exercises exactly one layer with its neighbour mocked:
|
||||
/// T1 the site broadcast filter (subscriber = TestKit probe, domain events), T2/the existing
|
||||
/// <see cref="GrpcStreamIntegrationTests"/> the server+relay+channel with a MOCKED
|
||||
/// <see cref="ISiteStreamSubscriber"/> stopping at the proto event, T3 the proto→domain client
|
||||
/// mapping in isolation, and T4 the aggregator fed hand-built domain events. Nothing wires the
|
||||
/// whole pipe together.
|
||||
///
|
||||
/// This test assembles the REAL chain end to end — mocking only the gRPC HTTP/2 transport:
|
||||
///
|
||||
/// domain AlarmStateChanged
|
||||
/// → SiteStreamManager.SubscribeSiteAlarms (real site-wide broadcast + alarm-only filter)
|
||||
/// → SiteStreamGrpcServer.SubscribeSite (real server handler)
|
||||
/// → StreamRelayActor (real domain→proto mapping + placeholder drop)
|
||||
/// → SiteStreamEvent proto → wire round-trip (ToByteArray/ParseFrom, simulates HTTP/2)
|
||||
/// → SiteStreamGrpcClient.ConvertToAlarmEvent (real proto→domain mapping)
|
||||
/// → SiteAlarmAggregatorActor cache (real dedup/seed/live)
|
||||
///
|
||||
/// and asserts the invariants the seams can't prove alone: AlarmKey identity
|
||||
/// (instance, name, sourceRef) + native enrichment survive every boundary; attributes and
|
||||
/// placeholder rows never reach the live cache; the single site-wide stream carries alarms
|
||||
/// for MULTIPLE instances (no per-instance filter); and a snapshot-seed row and a live delta
|
||||
/// for the same native alarm — both mapped through the real pipe — collapse onto ONE cache row.
|
||||
///
|
||||
/// Full gRPC-over-HTTP/2 remains a manual docker-cluster smoke (plan §4 Task 7).
|
||||
/// </summary>
|
||||
public class SiteAlarmStreamEndToEndTests : TestKit
|
||||
{
|
||||
private const string InstanceA = "SiteA.Pump01";
|
||||
private const string InstanceB = "SiteA.Motor07";
|
||||
|
||||
// ── The full site→proto→client trace ────────────────────────────────────────
|
||||
|
||||
[Fact]
|
||||
public async Task SiteWideAlarm_TraversesManagerToRelayToClient_PreservingIdentityAndEnrichment_DroppingAttributesAndPlaceholders()
|
||||
{
|
||||
// Site side: a REAL broadcast hub (no mock subscriber) wired straight into the
|
||||
// REAL SubscribeSite server handler.
|
||||
var manager = new SiteStreamManager(
|
||||
new SiteRuntimeOptions { StreamBufferSize = 256 },
|
||||
NullLogger<SiteStreamManager>.Instance);
|
||||
manager.Initialize(Sys);
|
||||
|
||||
var server = new SiteStreamGrpcServer(manager, NullLogger<SiteStreamGrpcServer>.Instance);
|
||||
server.SetReady(Sys);
|
||||
|
||||
var written = new ConcurrentQueue<SiteStreamEvent>();
|
||||
var writer = Substitute.For<IServerStreamWriter<SiteStreamEvent>>();
|
||||
writer.WriteAsync(Arg.Any<SiteStreamEvent>(), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.CompletedTask)
|
||||
.AndDoes(ci => written.Enqueue(ci.Arg<SiteStreamEvent>()));
|
||||
|
||||
using var cts = new CancellationTokenSource();
|
||||
var context = CreateMockContext(cts.Token);
|
||||
|
||||
var streamTask = Task.Run(() => server.SubscribeSite(
|
||||
new SiteStreamRequest { CorrelationId = "e2e-site" }, writer, context));
|
||||
|
||||
// Wait until the server has actually subscribed to the site-wide hub, otherwise a
|
||||
// publish can race ahead of the materialized subscription and be missed.
|
||||
await WaitForConditionAsync(() => manager.SubscriptionCount == 1);
|
||||
|
||||
var raise = new DateTimeOffset(2026, 4, 2, 8, 30, 0, TimeSpan.Zero);
|
||||
var ts = raise.AddSeconds(5);
|
||||
|
||||
// A fully-enriched NATIVE alarm on instance A (the payload whose fidelity we trace).
|
||||
var native = new AlarmStateChanged(InstanceA, "Tank01.LevelAlarm", AlarmState.Active, 725, ts)
|
||||
{
|
||||
Level = AlarmLevel.HighHigh,
|
||||
Message = "Tank 01 level critically high",
|
||||
Kind = AlarmKind.NativeOpcUa,
|
||||
Condition = new AlarmConditionState(
|
||||
Active: true, Acknowledged: true, Confirmed: false,
|
||||
Shelve: AlarmShelveState.OneShotShelved, Suppressed: false, Severity: 725),
|
||||
SourceReference = "Tank01.Level.HiHi",
|
||||
AlarmTypeName = "AnalogLimitAlarm.HiHi",
|
||||
Category = "Process",
|
||||
OperatorUser = "op.jane",
|
||||
OperatorComment = "ack — investigating",
|
||||
OriginalRaiseTime = raise,
|
||||
CurrentValue = "98.4",
|
||||
LimitValue = "95.0",
|
||||
NativeSourceCanonicalName = "Tank01.LevelAlarm",
|
||||
IsConfiguredPlaceholder = false
|
||||
};
|
||||
|
||||
// A COMPUTED alarm on a DIFFERENT instance — must arrive over the SAME site-wide
|
||||
// stream (proves the per-instance filter is gone).
|
||||
var computedOtherInstance = new AlarmStateChanged(InstanceB, "OverSpeed", AlarmState.Active, 400, ts);
|
||||
|
||||
// Noise that MUST be dropped somewhere in the pipe:
|
||||
var attribute = new AttributeValueChanged(InstanceA, "Modules.Flow", "GPM", 12.3, "Good", ts);
|
||||
var placeholder = new AlarmStateChanged(InstanceA, "Motor1.MotorAlarms", AlarmState.Normal, 0, ts)
|
||||
{
|
||||
Kind = AlarmKind.NativeOpcUa,
|
||||
IsConfiguredPlaceholder = true
|
||||
};
|
||||
|
||||
// Interleave so a leak of the dropped rows would change the observed count/order.
|
||||
manager.PublishAttributeValueChanged(attribute); // filtered at the manager (alarm-only)
|
||||
manager.PublishAlarmStateChanged(native); // → 1 proto
|
||||
manager.PublishAlarmStateChanged(placeholder); // dropped by StreamRelayActor
|
||||
manager.PublishAlarmStateChanged(computedOtherInstance); // → 1 proto
|
||||
|
||||
await WaitForConditionAsync(() => written.Count >= 2);
|
||||
// Give any leaked (attribute/placeholder) event a chance to show up before asserting.
|
||||
await Task.Delay(150);
|
||||
|
||||
cts.Cancel();
|
||||
await streamTask;
|
||||
|
||||
var protos = written.ToArray();
|
||||
Assert.Equal(2, protos.Length);
|
||||
Assert.All(protos, p =>
|
||||
Assert.Equal(SiteStreamEvent.EventOneofCase.AlarmChanged, p.EventCase)); // no attribute leaked
|
||||
Assert.DoesNotContain(protos, p => p.AlarmChanged.IsConfiguredPlaceholder); // no placeholder leaked
|
||||
|
||||
// Wire round-trip (serialize→deserialize) then the REAL client mapping back to domain.
|
||||
var mapped = protos
|
||||
.Select(p => SiteStreamGrpcClient.ConvertToAlarmEvent(RoundTripOverWire(p)))
|
||||
.ToList();
|
||||
Assert.All(mapped, m => Assert.NotNull(m));
|
||||
|
||||
// Both instances present — one site-wide stream, no per-instance filter.
|
||||
var backA = mapped.Single(m => m!.InstanceUniqueName == InstanceA)!;
|
||||
var backB = mapped.Single(m => m!.InstanceUniqueName == InstanceB)!;
|
||||
Assert.Equal("OverSpeed", backB.AlarmName);
|
||||
|
||||
// AlarmKey identity survived end to end.
|
||||
Assert.Equal("Tank01.LevelAlarm", backA.AlarmName);
|
||||
Assert.Equal("Tank01.Level.HiHi", backA.SourceReference);
|
||||
|
||||
// Native enrichment survived every boundary (manager→relay→proto→wire→client).
|
||||
Assert.Equal(AlarmKind.NativeOpcUa, backA.Kind);
|
||||
Assert.Equal(AlarmState.Active, backA.State);
|
||||
Assert.Equal(725, backA.Priority);
|
||||
Assert.Equal(AlarmLevel.HighHigh, backA.Level);
|
||||
Assert.Equal("Tank 01 level critically high", backA.Message);
|
||||
Assert.True(backA.Condition.Active);
|
||||
Assert.True(backA.Condition.Acknowledged);
|
||||
Assert.Equal(AlarmShelveState.OneShotShelved, backA.Condition.Shelve);
|
||||
Assert.Equal(725, backA.Condition.Severity);
|
||||
Assert.Equal("AnalogLimitAlarm.HiHi", backA.AlarmTypeName);
|
||||
Assert.Equal("Process", backA.Category);
|
||||
Assert.Equal("op.jane", backA.OperatorUser);
|
||||
Assert.Equal("ack — investigating", backA.OperatorComment);
|
||||
Assert.Equal(raise, backA.OriginalRaiseTime);
|
||||
Assert.Equal("98.4", backA.CurrentValue);
|
||||
Assert.Equal("95.0", backA.LimitValue);
|
||||
Assert.Equal("Tank01.LevelAlarm", backA.NativeSourceCanonicalName);
|
||||
Assert.Equal(ts, backA.Timestamp);
|
||||
Assert.False(backA.IsConfiguredPlaceholder);
|
||||
}
|
||||
|
||||
// ── Snapshot-seed vs live-delta identity parity, through the real pipe ────────
|
||||
|
||||
[Fact]
|
||||
public async Task RealPipeMapped_SeedRow_And_LiveDelta_ForSameNativeAlarm_CollapseOntoOneCacheRow()
|
||||
{
|
||||
// Produce TWO proto events for the SAME native alarm identity through the real
|
||||
// manager→relay→proto pipe: an older one (used as the snapshot-seed row) and a
|
||||
// newer one (fed as the live delta). Because both are mapped by the SAME real
|
||||
// ConvertToAlarmEvent, their AlarmKeys are identical by construction — so the real
|
||||
// aggregator must collapse them onto ONE row (no seed/live ghost duplicate).
|
||||
var manager = new SiteStreamManager(
|
||||
new SiteRuntimeOptions { StreamBufferSize = 256 },
|
||||
NullLogger<SiteStreamManager>.Instance);
|
||||
manager.Initialize(Sys);
|
||||
var server = new SiteStreamGrpcServer(manager, NullLogger<SiteStreamGrpcServer>.Instance);
|
||||
server.SetReady(Sys);
|
||||
|
||||
var written = new ConcurrentQueue<SiteStreamEvent>();
|
||||
var writer = Substitute.For<IServerStreamWriter<SiteStreamEvent>>();
|
||||
writer.WriteAsync(Arg.Any<SiteStreamEvent>(), Arg.Any<CancellationToken>())
|
||||
.Returns(Task.CompletedTask)
|
||||
.AndDoes(ci => written.Enqueue(ci.Arg<SiteStreamEvent>()));
|
||||
using var cts = new CancellationTokenSource();
|
||||
var streamTask = Task.Run(() => server.SubscribeSite(
|
||||
new SiteStreamRequest { CorrelationId = "e2e-parity" }, writer, CreateMockContext(cts.Token)));
|
||||
await WaitForConditionAsync(() => manager.SubscriptionCount == 1);
|
||||
|
||||
AlarmStateChanged Native(int priority, DateTimeOffset t) =>
|
||||
new(InstanceA, "Tank01.LevelAlarm", AlarmState.Active, priority, t)
|
||||
{
|
||||
Kind = AlarmKind.NativeOpcUa,
|
||||
SourceReference = "Tank01.Level.HiHi",
|
||||
NativeSourceCanonicalName = "Tank01.LevelAlarm"
|
||||
};
|
||||
|
||||
var t0 = new DateTimeOffset(2026, 4, 2, 9, 0, 0, TimeSpan.Zero);
|
||||
manager.PublishAlarmStateChanged(Native(300, t0)); // → seed row (older)
|
||||
manager.PublishAlarmStateChanged(Native(800, t0.AddSeconds(5))); // → live delta (newer)
|
||||
await WaitForConditionAsync(() => written.Count >= 2);
|
||||
cts.Cancel();
|
||||
await streamTask;
|
||||
|
||||
var protos = written.ToArray();
|
||||
var seedRow = SiteStreamGrpcClient.ConvertToAlarmEvent(RoundTripOverWire(protos[0]))!;
|
||||
var liveRow = SiteStreamGrpcClient.ConvertToAlarmEvent(RoundTripOverWire(protos[1]))!;
|
||||
|
||||
// Drive a REAL aggregator: seed returns the older mapped row; then Tell it the newer
|
||||
// mapped delta on the live path.
|
||||
var factory = new NoopSiteStreamClientFactory();
|
||||
var sink = new PublishSink();
|
||||
var aggregator = Sys.ActorOf(Props.Create(() => new SiteAlarmAggregatorActor(
|
||||
"site-alpha", "e2e-parity", _ => Task.FromResult<IReadOnlyList<AlarmStateChanged>>(new[] { seedRow }),
|
||||
sink.Publish, factory, "http://a:5100", "http://b:5100", TimeSpan.FromMinutes(10))));
|
||||
|
||||
await WaitForConditionAsync(() => sink.Latest is { Count: 1 }); // seed applied
|
||||
aggregator.Tell(liveRow);
|
||||
await WaitForConditionAsync(() =>
|
||||
sink.Latest is { Count: 1 } l && l[0].Priority == 800); // live delta collapsed in place
|
||||
|
||||
var final = sink.Latest!;
|
||||
Assert.Single(final); // exactly ONE row for the (instance, name, sourceRef) identity
|
||||
Assert.Equal("Tank01.Level.HiHi", final[0].SourceReference);
|
||||
Assert.Equal(AlarmKind.NativeOpcUa, final[0].Kind);
|
||||
}
|
||||
|
||||
// ── helpers ──────────────────────────────────────────────────────────────────
|
||||
|
||||
/// <summary>
|
||||
/// Serialize→deserialize a proto event to mimic the gRPC HTTP/2 hop, proving the
|
||||
/// enriched <c>AlarmStateUpdate</c> is fully wire-representable (no field lost in codec).
|
||||
/// </summary>
|
||||
private static SiteStreamEvent RoundTripOverWire(SiteStreamEvent evt) =>
|
||||
SiteStreamEvent.Parser.ParseFrom(evt.ToByteArray());
|
||||
|
||||
private static ServerCallContext CreateMockContext(CancellationToken cancellationToken)
|
||||
{
|
||||
var context = Substitute.For<ServerCallContext>();
|
||||
context.CancellationToken.Returns(cancellationToken);
|
||||
return context;
|
||||
}
|
||||
|
||||
private static async Task WaitForConditionAsync(Func<bool> condition, int timeoutMs = 5000)
|
||||
{
|
||||
var deadline = DateTime.UtcNow.AddMilliseconds(timeoutMs);
|
||||
while (!condition() && DateTime.UtcNow < deadline)
|
||||
await Task.Delay(25);
|
||||
Assert.True(condition(), $"Condition not met within {timeoutMs}ms");
|
||||
}
|
||||
|
||||
private sealed class PublishSink
|
||||
{
|
||||
private readonly object _lock = new();
|
||||
private IReadOnlyList<AlarmStateChanged>? _latest;
|
||||
public void Publish(IReadOnlyList<AlarmStateChanged> snapshot)
|
||||
{
|
||||
lock (_lock) { _latest = snapshot; }
|
||||
}
|
||||
public IReadOnlyList<AlarmStateChanged>? Latest
|
||||
{
|
||||
get { lock (_lock) { return _latest; } }
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>Aggregator transport double: its live stream simply stays open until cancelled.</summary>
|
||||
private sealed class NoopSiteStreamClient : SiteStreamGrpcClient
|
||||
{
|
||||
public override Task SubscribeSiteAsync(
|
||||
string correlationId, Action<AlarmStateChanged> onAlarmEvent, Action<Exception> onError, CancellationToken ct)
|
||||
{
|
||||
var tcs = new TaskCompletionSource();
|
||||
ct.Register(() => tcs.TrySetResult());
|
||||
return tcs.Task;
|
||||
}
|
||||
|
||||
public override void Unsubscribe(string correlationId) { }
|
||||
}
|
||||
|
||||
private sealed class NoopSiteStreamClientFactory : SiteStreamGrpcClientFactory
|
||||
{
|
||||
private readonly NoopSiteStreamClient _client = new();
|
||||
public NoopSiteStreamClientFactory() : base(NullLoggerFactory.Instance) { }
|
||||
public override SiteStreamGrpcClient GetOrCreate(string siteIdentifier, string grpcEndpoint) => _client;
|
||||
public override SiteStreamGrpcClient? TryGet(string siteIdentifier, string grpcEndpoint) => _client;
|
||||
}
|
||||
}
|
||||
@@ -103,6 +103,50 @@ public class SiteStreamManagerTests : TestKit, IDisposable
|
||||
probe2.ExpectNoMsg(TimeSpan.FromMilliseconds(500));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SubscribeSiteAlarms_ForwardsAlarmsForAllInstances_ButNotAttributes()
|
||||
{
|
||||
var probe = CreateTestProbe();
|
||||
_streamManager.SubscribeSiteAlarms(probe.Ref);
|
||||
|
||||
// Alarm events from two different instances — both should arrive.
|
||||
_streamManager.PublishAlarmStateChanged(new AlarmStateChanged(
|
||||
"Pump1", "HighTemp", AlarmState.Active, 1, DateTimeOffset.UtcNow));
|
||||
_streamManager.PublishAlarmStateChanged(new AlarmStateChanged(
|
||||
"Pump2", "LowFlow", AlarmState.Active, 2, DateTimeOffset.UtcNow));
|
||||
|
||||
// Attribute events must be filtered out entirely.
|
||||
_streamManager.PublishAttributeValueChanged(new AttributeValueChanged(
|
||||
"Pump1", "Temperature", "Temperature", "100", "Good", DateTimeOffset.UtcNow));
|
||||
_streamManager.PublishAttributeValueChanged(new AttributeValueChanged(
|
||||
"Pump3", "Flow", "Flow", "42", "Good", DateTimeOffset.UtcNow));
|
||||
|
||||
// Collect the two alarms (order across instances is not guaranteed).
|
||||
var received = new[]
|
||||
{
|
||||
probe.ExpectMsg<AlarmStateChanged>(TimeSpan.FromSeconds(3)),
|
||||
probe.ExpectMsg<AlarmStateChanged>(TimeSpan.FromSeconds(3)),
|
||||
};
|
||||
|
||||
var instances = received.Select(a => a.InstanceUniqueName).OrderBy(n => n).ToArray();
|
||||
Assert.Equal(new[] { "Pump1", "Pump2" }, instances);
|
||||
|
||||
// No attribute events (nor any further alarm) should be delivered.
|
||||
probe.ExpectNoMsg(TimeSpan.FromMilliseconds(500));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void SubscribeSiteAlarms_SubscriptionRemovableViaUnsubscribe()
|
||||
{
|
||||
var probe = CreateTestProbe();
|
||||
var id = _streamManager.SubscribeSiteAlarms(probe.Ref);
|
||||
|
||||
Assert.NotNull(id);
|
||||
Assert.Equal(1, _streamManager.SubscriptionCount);
|
||||
Assert.True(_streamManager.Unsubscribe(id));
|
||||
Assert.Equal(0, _streamManager.SubscriptionCount);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void RemoveSubscriber_RemovesAllSubscriptionsForActor()
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user