ScadaBridge/docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md

Plan: Aggregated Live Alarm Stream for Alarm Summary (deferred #10)

Status: Draft plan (not yet executed) — 2026-07-10 Register row: docs/plans/2026-07-08-deferred-work-register.md #10 Revisit trigger that fired this plan: "Alarm Summary latency complaints or >~50 instances/site." Owning component: Central UI (#9) + Central–Site Communication (#5); no new component.

---

1. Problem

The operator Alarm Summary page (/monitoring/alarms, RequireDeployment, read-only) currently rebuilds its whole view every 15 s by fanning out a per-instance DebugViewSnapshot Ask to every Enabled instance on the selected site and aggregating client-side (AlarmSummaryService, MaxConcurrentFetches = 8). At current instance counts this is acceptable; past ~50 instances/site the poll does 50 cross-cluster Asks every 15 s (worst case 50 × ClusterClient round-trips gated 8-at-a-time), so freshness degrades and the active central node carries a periodic burst. We want an aggregated live feed so the page reflects alarm transitions in near-real-time without re-polling the whole site.

2. Hard constraint (locked decision — do not violate)

docs/plans/2026-05-29-native-alarms-design.md:24 marks "no central alarm tables, no central history" as [PERM]. Therefore this plan introduces no persisted central alarm store — only a transient in-memory per-site live cache on the active central node, seeded from the existing snapshot fan-out and kept warm by a live gRPC delta stream. If a persisted store is ever wanted, that is a separate re-decision, not part of this plan.

3. Chosen approach

A shared, reference-counted, per-site central live alarm cache, seeded by the existing snapshot fan-out and updated by a new site-wide, alarm-only gRPC stream, pushed to the Blazor page via an in-process singleton event (the IDeploymentStatusNotifier pattern — no new SignalR hub; the Blazor circuit already pushes).

Why site-wide RPC (Option B) over N per-instance streams (Option A)

The revisit trigger is >~50 instances/site — exactly where opening one gRPC stream per instance (Option A) strains GrpcMaxConcurrentStreams = 100 and duplicates the fan-out problem in streaming form. A single SubscribeSite stream per site (alarm events only; attributes are far higher-volume and the summary never shows them) is the scalable answer and is an additive proto change. It reuses the existing SiteStreamManager broadcast hub by adding a subscribe variant that drops the per-instance InstanceUniqueName filter (SiteStreamManager.cs:101) and keeps only AlarmStateChanged events.

Lifecycle & failover

• Shared per-site cache, reference-counted by active viewers: first viewer of a site starts the subscription + snapshot seed; last viewer leaving stops it (with a short linger to avoid thrash). One gRPC stream per site, not per browser circuit.
• Seed-then-stream ordering copied from DebugStreamBridgeActor: open the stream first, buffer live deltas, run the snapshot fan-out once, flush buffer with dedup, then live pass-through. Dedup identity = (InstanceUniqueName, AlarmName, SourceReference) (matches DebugStreamBridgeActor.AlarmKey).
• NodeA↔NodeB reconnect with retry budget + stability window, reusing the reconnect logic already proven in DebugStreamBridgeActor.
• Placeholder reconciliation: live stream drops IsConfiguredPlaceholder rows (StreamRelayActor.cs:63); snapshots include them. Cache seeds placeholders from the snapshot and never expects them on the live stream.
• Instance-set drift: on deploy/enable/disable/delete the site-wide stream naturally starts/stops carrying that instance's events; a periodic (e.g. 60 s) reconcile snapshot corrects any missed enable/disable so the cache can't drift indefinitely. This replaces the old "re-query everything every 15 s" with "stream + occasional reconcile."

4. Task breakdown

Classification legend: trivial / small (code review) / standard (spec+code review) / high-risk (serial reviews + integration). Times are single-task estimates.

Task 1 — Site-wide alarm-only stream on the site runtime [standard]

• Files: src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Streaming/SiteStreamManager.cs
• Add SubscribeSiteAlarms(ISiteStreamSubscriber subscriber) — same broadcast-hub wiring as Subscribe(...) but without the InstanceUniqueName filter and filtering to AlarmStateChanged only (skip AttributeValueChanged).
• Return an IDisposable/kill-switch symmetric with the existing per-instance subscribe. Unit-test that alarm events for all instances arrive and attribute events do not.

Task 2 — SubscribeSite gRPC contract + server handler [standard]

• Files: src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto, src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs, src/ZB.MOM.WW.ScadaBridge.Communication/Actors/StreamRelayActor.cs
• Add additive rpc SubscribeSite(SiteStreamRequest) returns (stream SiteStreamEvent) + SiteStreamRequest { correlation_id } (no instance name). Keep bundleFormatVersion/message numbering additive — do not renumber existing fields.
• Server handler mirrors SubscribeInstance (bounded Channel(1000, DropOldest), GrpcMaxConcurrentStreams/GrpcMaxStreamLifetime limits, the StreamRelayActor mapping, SiteConnectionOpened/Closed telemetry) but calls SubscribeSiteAlarms. StreamRelayActor already drops placeholder rows and emits enriched AlarmStateUpdate — reuse unchanged.

Task 3 — Central per-site subscription client [standard]

• Files: src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs, .../Grpc/SiteStreamGrpcClientFactory.cs
• Add SubscribeSiteAsync(correlationId, onAlarmEvent, onError, ct) alongside SubscribeAsync, reusing ConvertToDomainEvent (maps proto → domain AlarmStateChanged). Factory already caches channels per (site, endpoint) for failover — no change needed beyond the new call.

Task 4 — Central live alarm cache actor/service [high-risk]

• Files (new): src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs, src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs, src/ZB.MOM.WW.ScadaBridge.Communication/ISiteAlarmLiveCache.cs
• Files (edit): src/ZB.MOM.WW.ScadaBridge.Communication/ServiceCollectionExtensions.cs
• Per-site aggregator: seed-then-stream (buffer → snapshot fan-out via existing CommunicationService.RequestDebugSnapshotAsync → flush+dedup → live), holds a Dictionary<AlarmKey, AlarmStateChanged> snapshot of current alarm state. Reference-counted start/stop keyed by site. NodeA↔NodeB reconnect + periodic reconcile snapshot (60 s). Reuse DebugStreamBridgeActor reconnect logic (extract shared helper or copy with attribution).
• Expose ISiteAlarmLiveCache (singleton) with Subscribe(siteId, onChange) → IDisposable, raising an IDeploymentStatusNotifier-style event on each applied delta. This is the highest-risk task (concurrency, failover, seed/stream ordering) — spawn a code-reviewer subagent and add integration coverage.

Task 5 — Wire Alarm Summary page to the live cache [standard]

• Files: src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/AlarmSummaryService.cs, .../Services/IAlarmSummaryService.cs, .../Components/Pages/Monitoring/AlarmSummary.razor, .../ServiceCollectionExtensions.cs
• Add SubscribeSiteAlarmsAsync(siteId, onChange) to the service, delegating to ISiteAlarmLiveCache. Page: on site select, subscribe; onChange updates _rows + recomputes _rollup/_visibleRows and StateHasChanged() (over the Blazor circuit). Keep the 15 s poll as a fallback but gate it behind a feature flag / degrade path when the live cache is healthy, so a stream failure never blanks the page. Unsubscribe in Dispose(). AlarmStateBadges and all client-side filters/sort are unchanged (they already consume AlarmStateChanged).

Task 6 — Options, limits, and telemetry [small]

• Files: communication options class (grep CommunicationOptions / GrpcMaxConcurrentStreams), SiteAlarmLiveCacheService
• Add options: live-cache linger, reconcile interval, per-site subscriber cap. Validate (eager options validation is the house rule). Surface a gauge for active per-site aggregators + reconnect count on the health snapshot, mirroring SiteConnectionUp gauges.

Task 7 — Tests + integration trace [standard]

• Files: tests/.../Communication.Tests/*, tests/.../CentralUI.Tests/*, tests/.../SiteRuntime.Tests/Streaming/SiteStreamManagerTests.cs
• Unit: SubscribeSiteAlarms filtering; SubscribeSite server/client mapping; aggregator seed/stream/dedup/placeholder/reconnect; page subscribe/unsubscribe.
• Integration: end-to-end site→central alarm delta reaches the cache; failover flip re-seeds; last-viewer stop tears the stream down. Live-smoke on the docker cluster (bash docker/deploy.sh) with >1 instance raising alarms.

Task 8 — Docs [trivial]

• Files: docs/requirements/Component-CentralUI.md (Alarm Summary section — replace "no aggregated live stream" with the shipped design), docs/requirements/Component-Communication.md (new SubscribeSite RPC), docs/plans/2026-06-18-m7-opcua-mxgateway-ux-design.md (mark T13 follow-up delivered), and move register #10 to Resolved.

5. Risks / call-outs

• [PERM] no-central-store — the cache is explicitly transient/in-memory. Reviewer must confirm no EF entity/table/migration is introduced.
• Proto evolution — SubscribeSite must be purely additive (new RPC + new message; no field renumbering). PLAN-08's ClusterClient/gRPC contract-lock tests apply.
• Stream-vs-snapshot race — the seed-then-stream dedup is the classic bug surface; reuse the DebugStreamBridgeActor ordering exactly.
• Failover gap — a NodeA↔NodeB flip must re-seed (not silently serve stale); the 60 s reconcile snapshot is the backstop.
• Effort estimate: ~6–9 focused tasks; Task 4 is the only genuinely hard one. Everything else reuses existing, proven machinery (SiteStreamManager/SiteStreamGrpcClientFactory/DebugStreamBridgeActor/ IDeploymentStatusNotifier/AlarmSummaryService).

6. Explicitly out of scope

• Persisted central alarm store or history (locked [PERM]).
• Aggregated attribute stream (summary needs alarms only).
• CLI/SignalR surface for the aggregated feed (Blazor-only; revisit if a CLI consumer appears).
• Ack-back / write path (native alarms remain read-only mirrors).