Joseph Doherty
d1ade5653b
A verified audit of all ~90 plan documents (2026-08-01) found ~30 .tasks.json
trackers and several plan headers still reporting 'pending'/'draft' for work
fully merged to main. Sync them so future audits don't re-litigate closed work:
- Flip ~380 stale task statuses to completed across March/May/June/July
trackers (audit-log series, milestones M5-M10, playwright waves, stillpending,
LocalDb, ClusterClient->gRPC DoD rows, and more), each verified against
code/git evidence before flipping.
- Annotate obsolete-not-done rows: ClusterClient CLI transport (never built,
HTTP shipped), TreeView Areas/Instances pages (replaced by Topology),
template-tree drag-drop (dropped for M9 menu reorder), otopcua item C
(premise superseded by #17).
- Flip stale headers: aggregated-live-alarm + kpi-rollups 'Draft not executed'
-> Delivered 2026-07-10; otopcua cutover-scope SCOPING -> DECIDED;
scadabridge-rename -> Implemented; LocalDb phase1/2 status strings ->
merged 28ca04d7.
- Fix doc drift: T9/T10 'deferred' -> shipped as SMS (Teams dropped); waitfor
sandbox follow-up shipped; followups #52/#53/#54/#162/#207 resolved; purge
TODO closed by PendingDeploymentPurgeActor; live-gate pre-existing failures
#28/#29/#31 fixed; auto-down boot-alone residual superseded by self-first
seeds; supersession banners on keep-oldest SBR + ClusterClient-era designs;
requirements-traceability 'Pending' clarified as frozen plan-generation
status.
Deliberately left pending (genuinely open, tracked in the pending-work list):
opcua-tag-browser task 19 (live smoke), ipsen tasks 7-8 (vd03 verification),
selfform task 7 (vd03 overlay, user-held), live-gate observation 1
(external-system delete orphan bug), otopcua item A + maxDepth calibration.
187 lines
11 KiB
Markdown
187 lines
11 KiB
Markdown
# Plan: Aggregated Live Alarm Stream for Alarm Summary (deferred #10)
|
||
|
||
**Status:** **Delivered 2026-07-10** — shipped as designed (`SiteAlarmAggregatorActor`,
|
||
`ISiteAlarmLiveCache`, the `SubscribeSite` streaming RPC, and
|
||
`AlarmSummaryService.BuildFromLiveAlarms` are all in `src/`). The 15 s poll is retained as the
|
||
NotReporting authority behind the live stream, per the design below.
|
||
**Register row:** `docs/plans/2026-07-08-deferred-work-register.md` #10
|
||
**Revisit trigger that fired this plan:** *"Alarm Summary latency complaints or >~50 instances/site."*
|
||
**Owning component:** Central UI (#9) + Central–Site Communication (#5); no new component.
|
||
|
||
---
|
||
|
||
## 1. Problem
|
||
|
||
The operator Alarm Summary page (`/monitoring/alarms`, `RequireDeployment`, read-only)
|
||
currently rebuilds its whole view every 15 s by fanning out a per-instance
|
||
`DebugViewSnapshot` Ask to **every Enabled instance on the selected site** and
|
||
aggregating client-side (`AlarmSummaryService`, `MaxConcurrentFetches = 8`). At
|
||
current instance counts this is acceptable; past **~50 instances/site** the poll
|
||
does 50 cross-cluster Asks every 15 s (worst case 50 × ClusterClient round-trips
|
||
gated 8-at-a-time), so freshness degrades and the active central node carries a
|
||
periodic burst. We want an **aggregated live feed** so the page reflects alarm
|
||
transitions in near-real-time without re-polling the whole site.
|
||
|
||
## 2. Hard constraint (locked decision — do not violate)
|
||
|
||
`docs/plans/2026-05-29-native-alarms-design.md:24` marks **"no central alarm
|
||
tables, no central history"** as `[PERM]`. Therefore this plan introduces **no
|
||
persisted central alarm store** — only a **transient in-memory per-site live
|
||
cache** on the active central node, seeded from the existing snapshot fan-out and
|
||
kept warm by a live gRPC delta stream. If a persisted store is ever wanted, that
|
||
is a separate re-decision, not part of this plan.
|
||
|
||
## 3. Chosen approach
|
||
|
||
**A shared, reference-counted, per-site central live alarm cache**, seeded by the
|
||
existing snapshot fan-out and updated by a new **site-wide, alarm-only** gRPC
|
||
stream, pushed to the Blazor page via an in-process singleton event (the
|
||
`IDeploymentStatusNotifier` pattern — no new SignalR hub; the Blazor circuit
|
||
already pushes).
|
||
|
||
### Why site-wide RPC (Option B) over N per-instance streams (Option A)
|
||
|
||
The revisit trigger is *>~50 instances/site* — exactly where opening one gRPC
|
||
stream per instance (Option A) strains `GrpcMaxConcurrentStreams = 100` and
|
||
duplicates the fan-out problem in streaming form. A single **`SubscribeSite`**
|
||
stream per site (alarm events only; attributes are far higher-volume and the
|
||
summary never shows them) is the scalable answer and is an *additive* proto
|
||
change. It reuses the existing `SiteStreamManager` broadcast hub by adding a
|
||
subscribe variant that drops the per-instance `InstanceUniqueName` filter
|
||
(`SiteStreamManager.cs:101`) and keeps only `AlarmStateChanged` events.
|
||
|
||
### Lifecycle & failover
|
||
|
||
- **Shared per-site cache**, reference-counted by active viewers: first viewer of
|
||
a site starts the subscription + snapshot seed; last viewer leaving stops it
|
||
(with a short linger to avoid thrash). One gRPC stream per *site*, not per
|
||
browser circuit.
|
||
- **Seed-then-stream** ordering copied from `DebugStreamBridgeActor`: open the
|
||
stream first, buffer live deltas, run the snapshot fan-out once, flush buffer
|
||
with dedup, then live pass-through. Dedup identity = `(InstanceUniqueName,
|
||
AlarmName, SourceReference)` (matches `DebugStreamBridgeActor.AlarmKey`).
|
||
- **NodeA↔NodeB reconnect** with retry budget + stability window, reusing the
|
||
reconnect logic already proven in `DebugStreamBridgeActor`.
|
||
- **Placeholder reconciliation:** live stream drops `IsConfiguredPlaceholder`
|
||
rows (`StreamRelayActor.cs:63`); snapshots include them. Cache seeds
|
||
placeholders from the snapshot and never expects them on the live stream.
|
||
- **Instance-set drift:** on deploy/enable/disable/delete the site-wide stream
|
||
naturally starts/stops carrying that instance's events; a periodic (e.g. 60 s)
|
||
reconcile snapshot corrects any missed enable/disable so the cache can't drift
|
||
indefinitely. This replaces the old "re-query everything every 15 s" with
|
||
"stream + occasional reconcile."
|
||
|
||
## 4. Task breakdown
|
||
|
||
Classification legend: `trivial` / `small` (code review) / `standard` (spec+code
|
||
review) / `high-risk` (serial reviews + integration). Times are single-task
|
||
estimates.
|
||
|
||
### Task 1 — Site-wide alarm-only stream on the site runtime `[standard]`
|
||
- **Files:** `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Streaming/SiteStreamManager.cs`
|
||
- Add `SubscribeSiteAlarms(ISiteStreamSubscriber subscriber)` — same broadcast-hub
|
||
wiring as `Subscribe(...)` but **without** the `InstanceUniqueName` filter and
|
||
**filtering to `AlarmStateChanged` only** (skip `AttributeValueChanged`).
|
||
- Return an `IDisposable`/kill-switch symmetric with the existing per-instance
|
||
subscribe. Unit-test that alarm events for *all* instances arrive and attribute
|
||
events do not.
|
||
|
||
### Task 2 — `SubscribeSite` gRPC contract + server handler `[standard]`
|
||
- **Files:** `src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto`,
|
||
`src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs`,
|
||
`src/ZB.MOM.WW.ScadaBridge.Communication/Actors/StreamRelayActor.cs`
|
||
- Add **additive** `rpc SubscribeSite(SiteStreamRequest) returns (stream
|
||
SiteStreamEvent)` + `SiteStreamRequest { correlation_id }` (no instance name).
|
||
Keep `bundleFormatVersion`/message numbering additive — do not renumber
|
||
existing fields.
|
||
- Server handler mirrors `SubscribeInstance` (bounded `Channel(1000,
|
||
DropOldest)`, `GrpcMaxConcurrentStreams`/`GrpcMaxStreamLifetime` limits, the
|
||
`StreamRelayActor` mapping, `SiteConnectionOpened/Closed` telemetry) but calls
|
||
`SubscribeSiteAlarms`. `StreamRelayActor` already drops placeholder rows and
|
||
emits enriched `AlarmStateUpdate` — reuse unchanged.
|
||
|
||
### Task 3 — Central per-site subscription client `[standard]`
|
||
- **Files:** `src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs`,
|
||
`.../Grpc/SiteStreamGrpcClientFactory.cs`
|
||
- Add `SubscribeSiteAsync(correlationId, onAlarmEvent, onError, ct)` alongside
|
||
`SubscribeAsync`, reusing `ConvertToDomainEvent` (maps proto → domain
|
||
`AlarmStateChanged`). Factory already caches channels per `(site, endpoint)` for
|
||
failover — no change needed beyond the new call.
|
||
|
||
### Task 4 — Central live alarm cache actor/service `[high-risk]`
|
||
- **Files (new):** `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs`,
|
||
`src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs`,
|
||
`src/ZB.MOM.WW.ScadaBridge.Communication/ISiteAlarmLiveCache.cs`
|
||
- **Files (edit):** `src/ZB.MOM.WW.ScadaBridge.Communication/ServiceCollectionExtensions.cs`
|
||
- Per-site aggregator: seed-then-stream (buffer → snapshot fan-out via existing
|
||
`CommunicationService.RequestDebugSnapshotAsync` → flush+dedup → live), holds a
|
||
`Dictionary<AlarmKey, AlarmStateChanged>` snapshot of current alarm state.
|
||
Reference-counted start/stop keyed by site. NodeA↔NodeB reconnect + periodic
|
||
reconcile snapshot (60 s). Reuse `DebugStreamBridgeActor` reconnect logic
|
||
(extract shared helper or copy with attribution).
|
||
- Expose `ISiteAlarmLiveCache` (singleton) with `Subscribe(siteId, onChange)` →
|
||
`IDisposable`, raising an `IDeploymentStatusNotifier`-style event on each
|
||
applied delta. This is the **highest-risk** task (concurrency, failover,
|
||
seed/stream ordering) — spawn a code-reviewer subagent and add integration
|
||
coverage.
|
||
|
||
### Task 5 — Wire Alarm Summary page to the live cache `[standard]`
|
||
- **Files:** `src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/AlarmSummaryService.cs`,
|
||
`.../Services/IAlarmSummaryService.cs`,
|
||
`.../Components/Pages/Monitoring/AlarmSummary.razor`,
|
||
`.../ServiceCollectionExtensions.cs`
|
||
- Add `SubscribeSiteAlarmsAsync(siteId, onChange)` to the service, delegating to
|
||
`ISiteAlarmLiveCache`. Page: on site select, subscribe; `onChange` updates
|
||
`_rows` + recomputes `_rollup`/`_visibleRows` and `StateHasChanged()` (over the
|
||
Blazor circuit). **Keep the 15 s poll as a fallback** but gate it behind a
|
||
feature flag / degrade path when the live cache is healthy, so a stream failure
|
||
never blanks the page. Unsubscribe in `Dispose()`. `AlarmStateBadges` and all
|
||
client-side filters/sort are unchanged (they already consume `AlarmStateChanged`).
|
||
|
||
### Task 6 — Options, limits, and telemetry `[small]`
|
||
- **Files:** communication options class (grep `CommunicationOptions` /
|
||
`GrpcMaxConcurrentStreams`), `SiteAlarmLiveCacheService`
|
||
- Add options: live-cache linger, reconcile interval, per-site subscriber cap.
|
||
Validate (eager options validation is the house rule). Surface a gauge for
|
||
active per-site aggregators + reconnect count on the health snapshot, mirroring
|
||
`SiteConnectionUp` gauges.
|
||
|
||
### Task 7 — Tests + integration trace `[standard]`
|
||
- **Files:** `tests/.../Communication.Tests/*`, `tests/.../CentralUI.Tests/*`,
|
||
`tests/.../SiteRuntime.Tests/Streaming/SiteStreamManagerTests.cs`
|
||
- Unit: `SubscribeSiteAlarms` filtering; `SubscribeSite` server/client mapping;
|
||
aggregator seed/stream/dedup/placeholder/reconnect; page subscribe/unsubscribe.
|
||
- Integration: end-to-end site→central alarm delta reaches the cache; failover
|
||
flip re-seeds; last-viewer stop tears the stream down. Live-smoke on the docker
|
||
cluster (`bash docker/deploy.sh`) with >1 instance raising alarms.
|
||
|
||
### Task 8 — Docs `[trivial]`
|
||
- **Files:** `docs/requirements/Component-CentralUI.md` (Alarm Summary section —
|
||
replace "no aggregated live stream" with the shipped design),
|
||
`docs/requirements/Component-Communication.md` (new `SubscribeSite` RPC),
|
||
`docs/plans/2026-06-18-m7-opcua-mxgateway-ux-design.md` (mark T13 follow-up
|
||
delivered), and move register #10 to **Resolved**.
|
||
|
||
## 5. Risks / call-outs
|
||
|
||
- **`[PERM]` no-central-store** — the cache is explicitly transient/in-memory.
|
||
Reviewer must confirm no EF entity/table/migration is introduced.
|
||
- **Proto evolution** — `SubscribeSite` must be purely additive (new RPC + new
|
||
message; no field renumbering). PLAN-08's ClusterClient/gRPC contract-lock
|
||
tests apply.
|
||
- **Stream-vs-snapshot race** — the seed-then-stream dedup is the classic bug
|
||
surface; reuse the `DebugStreamBridgeActor` ordering exactly.
|
||
- **Failover gap** — a NodeA↔NodeB flip must re-seed (not silently serve stale);
|
||
the 60 s reconcile snapshot is the backstop.
|
||
- **Effort estimate:** ~6–9 focused tasks; Task 4 is the only genuinely hard one.
|
||
Everything else reuses existing, proven machinery
|
||
(`SiteStreamManager`/`SiteStreamGrpcClientFactory`/`DebugStreamBridgeActor`/
|
||
`IDeploymentStatusNotifier`/`AlarmSummaryService`).
|
||
|
||
## 6. Explicitly out of scope
|
||
- Persisted central alarm store or history (locked `[PERM]`).
|
||
- Aggregated attribute stream (summary needs alarms only).
|
||
- CLI/SignalR surface for the aggregated feed (Blazor-only; revisit if a CLI
|
||
consumer appears).
|
||
- Ack-back / write path (native alarms remain read-only mirrors).
|