docs(plans): add implementation plans for deferred #10 (aggregated live alarm stream) and #22 (KPI hourly rollups)

Two draft task-by-task plans for the two Scale/YAGNI deferrals whose revisit
triggers are now in view:

- #10 aggregated live alarm stream: transient per-site in-memory central live
  cache seeded by the snapshot fan-out + additive SubscribeSite alarm-only gRPC
  stream, pushed via the IDeploymentStatusNotifier pattern; honors the [PERM]
  no-central-alarm-store rule. 8 tasks.
- #22 KPI hourly rollups: separate KpiRollupHourly table, recorder hourly fold
  with failover-safe lookback re-fold, per-metric gauge-vs-rate aggregation,
  range-threshold routing in the query service, longer rollup retention, one-shot
  backfill, plus 30d/90d window buttons so rollups have a caller. 8 tasks.

Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj
This commit is contained in:
Joseph Doherty
2026-07-10 11:36:35 -04:00
parent 22136d36d9
commit 84112db344
2 changed files with 393 additions and 0 deletions
@@ -0,0 +1,183 @@
# Plan: Aggregated Live Alarm Stream for Alarm Summary (deferred #10)
**Status:** Draft plan (not yet executed) — 2026-07-10
**Register row:** `docs/plans/2026-07-08-deferred-work-register.md` #10
**Revisit trigger that fired this plan:** *"Alarm Summary latency complaints or >~50 instances/site."*
**Owning component:** Central UI (#9) + CentralSite Communication (#5); no new component.
---
## 1. Problem
The operator Alarm Summary page (`/monitoring/alarms`, `RequireDeployment`, read-only)
currently rebuilds its whole view every 15 s by fanning out a per-instance
`DebugViewSnapshot` Ask to **every Enabled instance on the selected site** and
aggregating client-side (`AlarmSummaryService`, `MaxConcurrentFetches = 8`). At
current instance counts this is acceptable; past **~50 instances/site** the poll
does 50 cross-cluster Asks every 15 s (worst case 50 × ClusterClient round-trips
gated 8-at-a-time), so freshness degrades and the active central node carries a
periodic burst. We want an **aggregated live feed** so the page reflects alarm
transitions in near-real-time without re-polling the whole site.
## 2. Hard constraint (locked decision — do not violate)
`docs/plans/2026-05-29-native-alarms-design.md:24` marks **"no central alarm
tables, no central history"** as `[PERM]`. Therefore this plan introduces **no
persisted central alarm store** — only a **transient in-memory per-site live
cache** on the active central node, seeded from the existing snapshot fan-out and
kept warm by a live gRPC delta stream. If a persisted store is ever wanted, that
is a separate re-decision, not part of this plan.
## 3. Chosen approach
**A shared, reference-counted, per-site central live alarm cache**, seeded by the
existing snapshot fan-out and updated by a new **site-wide, alarm-only** gRPC
stream, pushed to the Blazor page via an in-process singleton event (the
`IDeploymentStatusNotifier` pattern — no new SignalR hub; the Blazor circuit
already pushes).
### Why site-wide RPC (Option B) over N per-instance streams (Option A)
The revisit trigger is *>~50 instances/site* — exactly where opening one gRPC
stream per instance (Option A) strains `GrpcMaxConcurrentStreams = 100` and
duplicates the fan-out problem in streaming form. A single **`SubscribeSite`**
stream per site (alarm events only; attributes are far higher-volume and the
summary never shows them) is the scalable answer and is an *additive* proto
change. It reuses the existing `SiteStreamManager` broadcast hub by adding a
subscribe variant that drops the per-instance `InstanceUniqueName` filter
(`SiteStreamManager.cs:101`) and keeps only `AlarmStateChanged` events.
### Lifecycle & failover
- **Shared per-site cache**, reference-counted by active viewers: first viewer of
a site starts the subscription + snapshot seed; last viewer leaving stops it
(with a short linger to avoid thrash). One gRPC stream per *site*, not per
browser circuit.
- **Seed-then-stream** ordering copied from `DebugStreamBridgeActor`: open the
stream first, buffer live deltas, run the snapshot fan-out once, flush buffer
with dedup, then live pass-through. Dedup identity = `(InstanceUniqueName,
AlarmName, SourceReference)` (matches `DebugStreamBridgeActor.AlarmKey`).
- **NodeA↔NodeB reconnect** with retry budget + stability window, reusing the
reconnect logic already proven in `DebugStreamBridgeActor`.
- **Placeholder reconciliation:** live stream drops `IsConfiguredPlaceholder`
rows (`StreamRelayActor.cs:63`); snapshots include them. Cache seeds
placeholders from the snapshot and never expects them on the live stream.
- **Instance-set drift:** on deploy/enable/disable/delete the site-wide stream
naturally starts/stops carrying that instance's events; a periodic (e.g. 60 s)
reconcile snapshot corrects any missed enable/disable so the cache can't drift
indefinitely. This replaces the old "re-query everything every 15 s" with
"stream + occasional reconcile."
## 4. Task breakdown
Classification legend: `trivial` / `small` (code review) / `standard` (spec+code
review) / `high-risk` (serial reviews + integration). Times are single-task
estimates.
### Task 1 — Site-wide alarm-only stream on the site runtime `[standard]`
- **Files:** `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Streaming/SiteStreamManager.cs`
- Add `SubscribeSiteAlarms(ISiteStreamSubscriber subscriber)` — same broadcast-hub
wiring as `Subscribe(...)` but **without** the `InstanceUniqueName` filter and
**filtering to `AlarmStateChanged` only** (skip `AttributeValueChanged`).
- Return an `IDisposable`/kill-switch symmetric with the existing per-instance
subscribe. Unit-test that alarm events for *all* instances arrive and attribute
events do not.
### Task 2 — `SubscribeSite` gRPC contract + server handler `[standard]`
- **Files:** `src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto`,
`src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs`,
`src/ZB.MOM.WW.ScadaBridge.Communication/Actors/StreamRelayActor.cs`
- Add **additive** `rpc SubscribeSite(SiteStreamRequest) returns (stream
SiteStreamEvent)` + `SiteStreamRequest { correlation_id }` (no instance name).
Keep `bundleFormatVersion`/message numbering additive — do not renumber
existing fields.
- Server handler mirrors `SubscribeInstance` (bounded `Channel(1000,
DropOldest)`, `GrpcMaxConcurrentStreams`/`GrpcMaxStreamLifetime` limits, the
`StreamRelayActor` mapping, `SiteConnectionOpened/Closed` telemetry) but calls
`SubscribeSiteAlarms`. `StreamRelayActor` already drops placeholder rows and
emits enriched `AlarmStateUpdate` — reuse unchanged.
### Task 3 — Central per-site subscription client `[standard]`
- **Files:** `src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs`,
`.../Grpc/SiteStreamGrpcClientFactory.cs`
- Add `SubscribeSiteAsync(correlationId, onAlarmEvent, onError, ct)` alongside
`SubscribeAsync`, reusing `ConvertToDomainEvent` (maps proto → domain
`AlarmStateChanged`). Factory already caches channels per `(site, endpoint)` for
failover — no change needed beyond the new call.
### Task 4 — Central live alarm cache actor/service `[high-risk]`
- **Files (new):** `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs`,
`src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs`,
`src/ZB.MOM.WW.ScadaBridge.Communication/ISiteAlarmLiveCache.cs`
- **Files (edit):** `src/ZB.MOM.WW.ScadaBridge.Communication/ServiceCollectionExtensions.cs`
- Per-site aggregator: seed-then-stream (buffer → snapshot fan-out via existing
`CommunicationService.RequestDebugSnapshotAsync` → flush+dedup → live), holds a
`Dictionary<AlarmKey, AlarmStateChanged>` snapshot of current alarm state.
Reference-counted start/stop keyed by site. NodeA↔NodeB reconnect + periodic
reconcile snapshot (60 s). Reuse `DebugStreamBridgeActor` reconnect logic
(extract shared helper or copy with attribution).
- Expose `ISiteAlarmLiveCache` (singleton) with `Subscribe(siteId, onChange)` →
`IDisposable`, raising an `IDeploymentStatusNotifier`-style event on each
applied delta. This is the **highest-risk** task (concurrency, failover,
seed/stream ordering) — spawn a code-reviewer subagent and add integration
coverage.
### Task 5 — Wire Alarm Summary page to the live cache `[standard]`
- **Files:** `src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/AlarmSummaryService.cs`,
`.../Services/IAlarmSummaryService.cs`,
`.../Components/Pages/Monitoring/AlarmSummary.razor`,
`.../ServiceCollectionExtensions.cs`
- Add `SubscribeSiteAlarmsAsync(siteId, onChange)` to the service, delegating to
`ISiteAlarmLiveCache`. Page: on site select, subscribe; `onChange` updates
`_rows` + recomputes `_rollup`/`_visibleRows` and `StateHasChanged()` (over the
Blazor circuit). **Keep the 15 s poll as a fallback** but gate it behind a
feature flag / degrade path when the live cache is healthy, so a stream failure
never blanks the page. Unsubscribe in `Dispose()`. `AlarmStateBadges` and all
client-side filters/sort are unchanged (they already consume `AlarmStateChanged`).
### Task 6 — Options, limits, and telemetry `[small]`
- **Files:** communication options class (grep `CommunicationOptions` /
`GrpcMaxConcurrentStreams`), `SiteAlarmLiveCacheService`
- Add options: live-cache linger, reconcile interval, per-site subscriber cap.
Validate (eager options validation is the house rule). Surface a gauge for
active per-site aggregators + reconnect count on the health snapshot, mirroring
`SiteConnectionUp` gauges.
### Task 7 — Tests + integration trace `[standard]`
- **Files:** `tests/.../Communication.Tests/*`, `tests/.../CentralUI.Tests/*`,
`tests/.../SiteRuntime.Tests/Streaming/SiteStreamManagerTests.cs`
- Unit: `SubscribeSiteAlarms` filtering; `SubscribeSite` server/client mapping;
aggregator seed/stream/dedup/placeholder/reconnect; page subscribe/unsubscribe.
- Integration: end-to-end site→central alarm delta reaches the cache; failover
flip re-seeds; last-viewer stop tears the stream down. Live-smoke on the docker
cluster (`bash docker/deploy.sh`) with >1 instance raising alarms.
### Task 8 — Docs `[trivial]`
- **Files:** `docs/requirements/Component-CentralUI.md` (Alarm Summary section —
replace "no aggregated live stream" with the shipped design),
`docs/requirements/Component-Communication.md` (new `SubscribeSite` RPC),
`docs/plans/2026-06-18-m7-opcua-mxgateway-ux-design.md` (mark T13 follow-up
delivered), and move register #10 to **Resolved**.
## 5. Risks / call-outs
- **`[PERM]` no-central-store** — the cache is explicitly transient/in-memory.
Reviewer must confirm no EF entity/table/migration is introduced.
- **Proto evolution** — `SubscribeSite` must be purely additive (new RPC + new
message; no field renumbering). PLAN-08's ClusterClient/gRPC contract-lock
tests apply.
- **Stream-vs-snapshot race** — the seed-then-stream dedup is the classic bug
surface; reuse the `DebugStreamBridgeActor` ordering exactly.
- **Failover gap** — a NodeA↔NodeB flip must re-seed (not silently serve stale);
the 60 s reconcile snapshot is the backstop.
- **Effort estimate:** ~69 focused tasks; Task 4 is the only genuinely hard one.
Everything else reuses existing, proven machinery
(`SiteStreamManager`/`SiteStreamGrpcClientFactory`/`DebugStreamBridgeActor`/
`IDeploymentStatusNotifier`/`AlarmSummaryService`).
## 6. Explicitly out of scope
- Persisted central alarm store or history (locked `[PERM]`).
- Aggregated attribute stream (summary needs alarms only).
- CLI/SignalR surface for the aggregated feed (Blazor-only; revisit if a CLI
consumer appears).
- Ack-back / write path (native alarms remain read-only mirrors).