ScadaBridge/docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md

# Plan: Aggregated Live Alarm Stream for Alarm Summary (deferred #10)

**Status:** Draft plan (not yet executed) — 2026-07-10
**Register row:** `docs/plans/2026-07-08-deferred-work-register.md` #10
**Revisit trigger that fired this plan:** *"Alarm Summary latency complaints or >~50 instances/site."*
**Owning component:** Central UI (#9) + Central–Site Communication (#5); no new component.

---

## 1. Problem

The operator Alarm Summary page (`/monitoring/alarms`, `RequireDeployment`, read-only)
currently rebuilds its whole view every 15 s by fanning out a per-instance
`DebugViewSnapshot` Ask to **every Enabled instance on the selected site** and
aggregating client-side (`AlarmSummaryService`, `MaxConcurrentFetches = 8`). At
current instance counts this is acceptable; past **~50 instances/site** the poll
does 50 cross-cluster Asks every 15 s (worst case 50 × ClusterClient round-trips
gated 8-at-a-time), so freshness degrades and the active central node carries a
periodic burst. We want an **aggregated live feed** so the page reflects alarm
transitions in near-real-time without re-polling the whole site.

## 2. Hard constraint (locked decision — do not violate)

`docs/plans/2026-05-29-native-alarms-design.md:24` marks **"no central alarm
tables, no central history"** as `[PERM]`. Therefore this plan introduces **no
persisted central alarm store** — only a **transient in-memory per-site live
cache** on the active central node, seeded from the existing snapshot fan-out and
kept warm by a live gRPC delta stream. If a persisted store is ever wanted, that
is a separate re-decision, not part of this plan.

## 3. Chosen approach

**A shared, reference-counted, per-site central live alarm cache**, seeded by the
existing snapshot fan-out and updated by a new **site-wide, alarm-only** gRPC
stream, pushed to the Blazor page via an in-process singleton event (the
`IDeploymentStatusNotifier` pattern — no new SignalR hub; the Blazor circuit
already pushes).

### Why site-wide RPC (Option B) over N per-instance streams (Option A)

The revisit trigger is *>~50 instances/site* — exactly where opening one gRPC
stream per instance (Option A) strains `GrpcMaxConcurrentStreams = 100` and
duplicates the fan-out problem in streaming form. A single **`SubscribeSite`**
stream per site (alarm events only; attributes are far higher-volume and the
summary never shows them) is the scalable answer and is an *additive* proto
change. It reuses the existing `SiteStreamManager` broadcast hub by adding a
subscribe variant that drops the per-instance `InstanceUniqueName` filter
(`SiteStreamManager.cs:101`) and keeps only `AlarmStateChanged` events.

### Lifecycle & failover

- **Shared per-site cache**, reference-counted by active viewers: first viewer of
  a site starts the subscription + snapshot seed; last viewer leaving stops it
  (with a short linger to avoid thrash). One gRPC stream per *site*, not per
  browser circuit.
- **Seed-then-stream** ordering copied from `DebugStreamBridgeActor`: open the
  stream first, buffer live deltas, run the snapshot fan-out once, flush buffer
  with dedup, then live pass-through. Dedup identity = `(InstanceUniqueName,
  AlarmName, SourceReference)` (matches `DebugStreamBridgeActor.AlarmKey`).
- **NodeA↔NodeB reconnect** with retry budget + stability window, reusing the
  reconnect logic already proven in `DebugStreamBridgeActor`.
- **Placeholder reconciliation:** live stream drops `IsConfiguredPlaceholder`
  rows (`StreamRelayActor.cs:63`); snapshots include them. Cache seeds
  placeholders from the snapshot and never expects them on the live stream.
- **Instance-set drift:** on deploy/enable/disable/delete the site-wide stream
  naturally starts/stops carrying that instance's events; a periodic (e.g. 60 s)
  reconcile snapshot corrects any missed enable/disable so the cache can't drift
  indefinitely. This replaces the old "re-query everything every 15 s" with
  "stream + occasional reconcile."

## 4. Task breakdown

Classification legend: `trivial` / `small` (code review) / `standard` (spec+code
review) / `high-risk` (serial reviews + integration). Times are single-task
estimates.

### Task 1 — Site-wide alarm-only stream on the site runtime `[standard]`
- **Files:** `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Streaming/SiteStreamManager.cs`
- Add `SubscribeSiteAlarms(ISiteStreamSubscriber subscriber)` — same broadcast-hub
  wiring as `Subscribe(...)` but **without** the `InstanceUniqueName` filter and
  **filtering to `AlarmStateChanged` only** (skip `AttributeValueChanged`).
- Return an `IDisposable`/kill-switch symmetric with the existing per-instance
  subscribe. Unit-test that alarm events for *all* instances arrive and attribute
  events do not.

### Task 2 — `SubscribeSite` gRPC contract + server handler `[standard]`
- **Files:** `src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto`,
  `src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs`,
  `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/StreamRelayActor.cs`
- Add **additive** `rpc SubscribeSite(SiteStreamRequest) returns (stream
  SiteStreamEvent)` + `SiteStreamRequest { correlation_id }` (no instance name).
  Keep `bundleFormatVersion`/message numbering additive — do not renumber
  existing fields.
- Server handler mirrors `SubscribeInstance` (bounded `Channel(1000,
  DropOldest)`, `GrpcMaxConcurrentStreams`/`GrpcMaxStreamLifetime` limits, the
  `StreamRelayActor` mapping, `SiteConnectionOpened/Closed` telemetry) but calls
  `SubscribeSiteAlarms`. `StreamRelayActor` already drops placeholder rows and
  emits enriched `AlarmStateUpdate` — reuse unchanged.

### Task 3 — Central per-site subscription client `[standard]`
- **Files:** `src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs`,
  `.../Grpc/SiteStreamGrpcClientFactory.cs`
- Add `SubscribeSiteAsync(correlationId, onAlarmEvent, onError, ct)` alongside
  `SubscribeAsync`, reusing `ConvertToDomainEvent` (maps proto → domain
  `AlarmStateChanged`). Factory already caches channels per `(site, endpoint)` for
  failover — no change needed beyond the new call.

### Task 4 — Central live alarm cache actor/service `[high-risk]`
- **Files (new):** `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs`,
  `src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs`,
  `src/ZB.MOM.WW.ScadaBridge.Communication/ISiteAlarmLiveCache.cs`
- **Files (edit):** `src/ZB.MOM.WW.ScadaBridge.Communication/ServiceCollectionExtensions.cs`
- Per-site aggregator: seed-then-stream (buffer → snapshot fan-out via existing
  `CommunicationService.RequestDebugSnapshotAsync` → flush+dedup → live), holds a
  `Dictionary<AlarmKey, AlarmStateChanged>` snapshot of current alarm state.
  Reference-counted start/stop keyed by site. NodeA↔NodeB reconnect + periodic
  reconcile snapshot (60 s). Reuse `DebugStreamBridgeActor` reconnect logic
  (extract shared helper or copy with attribution).
- Expose `ISiteAlarmLiveCache` (singleton) with `Subscribe(siteId, onChange)` →
  `IDisposable`, raising an `IDeploymentStatusNotifier`-style event on each
  applied delta. This is the **highest-risk** task (concurrency, failover,
  seed/stream ordering) — spawn a code-reviewer subagent and add integration
  coverage.

### Task 5 — Wire Alarm Summary page to the live cache `[standard]`
- **Files:** `src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/AlarmSummaryService.cs`,
  `.../Services/IAlarmSummaryService.cs`,
  `.../Components/Pages/Monitoring/AlarmSummary.razor`,
  `.../ServiceCollectionExtensions.cs`
- Add `SubscribeSiteAlarmsAsync(siteId, onChange)` to the service, delegating to
  `ISiteAlarmLiveCache`. Page: on site select, subscribe; `onChange` updates
  `_rows` + recomputes `_rollup`/`_visibleRows` and `StateHasChanged()` (over the
  Blazor circuit). **Keep the 15 s poll as a fallback** but gate it behind a
  feature flag / degrade path when the live cache is healthy, so a stream failure
  never blanks the page. Unsubscribe in `Dispose()`. `AlarmStateBadges` and all
  client-side filters/sort are unchanged (they already consume `AlarmStateChanged`).

### Task 6 — Options, limits, and telemetry `[small]`
- **Files:** communication options class (grep `CommunicationOptions` /
  `GrpcMaxConcurrentStreams`), `SiteAlarmLiveCacheService`
- Add options: live-cache linger, reconcile interval, per-site subscriber cap.
  Validate (eager options validation is the house rule). Surface a gauge for
  active per-site aggregators + reconnect count on the health snapshot, mirroring
  `SiteConnectionUp` gauges.

### Task 7 — Tests + integration trace `[standard]`
- **Files:** `tests/.../Communication.Tests/*`, `tests/.../CentralUI.Tests/*`,
  `tests/.../SiteRuntime.Tests/Streaming/SiteStreamManagerTests.cs`
- Unit: `SubscribeSiteAlarms` filtering; `SubscribeSite` server/client mapping;
  aggregator seed/stream/dedup/placeholder/reconnect; page subscribe/unsubscribe.
- Integration: end-to-end site→central alarm delta reaches the cache; failover
  flip re-seeds; last-viewer stop tears the stream down. Live-smoke on the docker
  cluster (`bash docker/deploy.sh`) with >1 instance raising alarms.

### Task 8 — Docs `[trivial]`
- **Files:** `docs/requirements/Component-CentralUI.md` (Alarm Summary section —
  replace "no aggregated live stream" with the shipped design),
  `docs/requirements/Component-Communication.md` (new `SubscribeSite` RPC),
  `docs/plans/2026-06-18-m7-opcua-mxgateway-ux-design.md` (mark T13 follow-up
  delivered), and move register #10 to **Resolved**.

## 5. Risks / call-outs

- **`[PERM]` no-central-store** — the cache is explicitly transient/in-memory.
  Reviewer must confirm no EF entity/table/migration is introduced.
- **Proto evolution** — `SubscribeSite` must be purely additive (new RPC + new
  message; no field renumbering). PLAN-08's ClusterClient/gRPC contract-lock
  tests apply.
- **Stream-vs-snapshot race** — the seed-then-stream dedup is the classic bug
  surface; reuse the `DebugStreamBridgeActor` ordering exactly.
- **Failover gap** — a NodeA↔NodeB flip must re-seed (not silently serve stale);
  the 60 s reconcile snapshot is the backstop.
- **Effort estimate:** ~6–9 focused tasks; Task 4 is the only genuinely hard one.
  Everything else reuses existing, proven machinery
  (`SiteStreamManager`/`SiteStreamGrpcClientFactory`/`DebugStreamBridgeActor`/
  `IDeploymentStatusNotifier`/`AlarmSummaryService`).

## 6. Explicitly out of scope
- Persisted central alarm store or history (locked `[PERM]`).
- Aggregated attribute stream (summary needs alarms only).
- CLI/SignalR surface for the aggregated feed (Blazor-only; revisit if a CLI
  consumer appears).
- Ack-back / write path (native alarms remain read-only mirrors).