docs(plans): add implementation plans for deferred #10 (aggregated live alarm stream) and #22 (KPI hourly rollups)

Two draft task-by-task plans for the two Scale/YAGNI deferrals whose revisit
triggers are now in view:

- #10 aggregated live alarm stream: transient per-site in-memory central live
  cache seeded by the snapshot fan-out + additive SubscribeSite alarm-only gRPC
  stream, pushed via the IDeploymentStatusNotifier pattern; honors the [PERM]
  no-central-alarm-store rule. 8 tasks.
- #22 KPI hourly rollups: separate KpiRollupHourly table, recorder hourly fold
  with failover-safe lookback re-fold, per-metric gauge-vs-rate aggregation,
  range-threshold routing in the query service, longer rollup retention, one-shot
  backfill, plus 30d/90d window buttons so rollups have a caller. 8 tasks.

Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj

docs/plans/2026-07-10-kpi-history-hourly-rollups-plan.md

@@ -0,0 +1,210 @@
+# Plan: KPI History Hourly Rollups (deferred #22)
+
+**Status:** Draft plan (not yet executed) — 2026-07-10
+**Register row:** `docs/plans/2026-07-08-deferred-work-register.md` #22
+**Revisit trigger that fired this plan:** *"KpiSample query latency on dashboards."*
+**Owning component:** KPI History (#26); touches Configuration Database (#17),
+Commons (#16), Central UI (#9), and every `IKpiSampleSource` owner.
+
+---
+
+## 1. Problem
+
+`KpiSample` is a tall/EAV table sampled once per minute per series
+(`(Source, Metric, Scope, ScopeKey)` 4-tuple). Long-range trend queries fetch and
+sort every raw row in the window before `KpiSeriesBucketer` downsamples to
+`DefaultMaxSeriesPoints = 200`:
+
+| Window | Raw rows/series @ 1/min |
+|--------|-------------------------|
+| 24 h   | 1,440 |
+| 7 d    | 10,080 |
+| 30 d   | 43,200 |
+| 90 d   | 129,600 |
+
+The four trend surfaces (Notification Outbox, Site Calls, Audit Log, Health) today
+offer only **24 h** and **7 d** windows, so the worst case is ~10k rows/series.
+The latency trigger only truly bites once **30 d / 90 d** windows are added — so
+this plan must decide whether to add those windows *and* pre-aggregate them, or
+rollups have no caller.
+
+## 2. Key facts that shape the design (from the code)
+
+- **The bucketer is not the bottleneck.** `KpiSeriesBucketer.Bucket` derives bucket
+  width purely from `(to − from) / maxPoints` *after* rows are in memory. The cost
+  is entirely in `GetRawSeriesAsync` fetching/sorting rows. Hourly rollups plug in
+  by feeding the bucketer a **pre-thinned ≤1-row/hour series** (≈60× fewer rows).
+- **`KpiHistoryQueryService.GetSeriesAsync` already owns fetch→bucket + options** —
+  the natural place to branch raw-vs-rollup by range.
+- **`KpiHistoryRecorderActor` takes the root `IServiceProvider` and opens a
+  per-tick scope** — a new hourly rollup timer needs **no constructor change**,
+  just a new timer + repo method (same shape as the existing sample/purge timers).
+- **Metrics are not all gauges.** Most are instantaneous gauges (`queueDepth`,
+  `buffered`, `connectionsDown`, `oldestPendingAgeSeconds`) where last-value-per-
+  hour is correct. But several are **already-windowed rate counters**
+  (`deliveredLastInterval`, `failedLastInterval`, `*LastHour`, `scriptErrors`,
+  `deadLetters`) where last-value-per-hour **undercounts** (keeps one minute's
+  delta, discards the other 59). This is the single biggest correctness decision.
+- Some metrics are **conditionally emitted** (`oldestPendingAgeSeconds` skipped
+  when null) — a rollup must not assume a row exists every minute.
+- **This is greenfield** — no `rollup`/`KpiRollup` table/column/code exists.
+
+## 3. Chosen approach
+
+**A separate `KpiRollupHourly` table, populated by an hourly recorder tick, with
+per-metric aggregation intent, queried via a range threshold, retained longer than
+raw samples, and backfilled once at migration.** Also add **30 d / 90 d** windows
+to the four surfaces so the rollups have a caller.
+
+Rationale for each decision (the research surfaced these as the required choices):
+
+**(a) Separate table vs. computed-on-read → separate `KpiRollupHourly` table.**
+Computed-on-read (`GROUP BY DATEADD(hour…)`) needs no migration/backfill but still
+**scans all raw rows** in SQL every query — it shrinks rows *returned*, not rows
+*scanned*, so it only partially addresses the latency trigger. A pre-aggregated
+table (design-consistent with the non-partitioned `KpiSample` model) makes a 90 d
+read a small indexed scan (~2,160 rows vs ~130k).
+
+**(b) Who computes → recorder-side hourly tick.** Add a third Akka timer
+(`RollupTick`) to `KpiHistoryRecorderActor`. Fold the **trailing N hours**
+(lookback window, not just "last hour") so a singleton-failover handover that
+misses a tick re-folds on recovery. Idempotent **upsert** keyed on
+`(Source, Metric, Scope, ScopeKey, HourStartUtc)`. Query-time fold was rejected
+because it doesn't reduce SQL scan (see (a)).
+
+**(c) Range routing → in `KpiHistoryQueryService.GetSeriesAsync`.** New option
+`RollupThresholdHours` (default 168 = 7 d): windows ≤ threshold → `GetRawSeriesAsync`
+(preserves intra-minute detail on 24 h/7 d); windows > threshold →
+`GetHourlySeriesAsync`. `KpiSeriesBucketer.Bucket` then runs unchanged on whichever
+series (note 90 d rollup = 2,160 rows > 200, so bucketing still applies on top).
+
+**(d) Retention → rollups outlive raw.** Keep raw `KpiSample` at 90 d (existing
+purge). Retain `KpiRollupHourly` much longer (new `RollupRetentionDays`, default
+365) via a **second purge pass** mirroring the 1-hour-sliced batch DELETE in
+`PurgeOlderThanAsync`. Validator must enforce `RollupRetentionDays ≥ RetentionDays`.
+This delivers *both* read-speed and history depth; without it the feature only
+buys speed.
+
+**(e) Backfill → one-shot fold at rollout.** Fold existing ≤90 d of `KpiSample`
+into `KpiRollupHourly` once (data migration or idempotent startup fold), so
+30 d/90 d charts aren't blank until enough wall-clock passes. The idempotent
+upsert key makes a re-run safe.
+
+**Cross-cutting correctness — per-metric aggregation.** Classify each charted
+metric as **gauge** (store last-value-per-hour) or **rate** (store sum-per-hour);
+store `Value` plus `Min`/`Max`/`Count` so the fold is faithful and future avg/min/
+max charts are unblocked. A metric→kind classification lives next to the
+`KpiMetrics` catalog. Rate metrics folded with `last-value` would silently
+undercount long-range totals — this must be explicit, not incidental.
+
+## 4. Task breakdown
+
+### Task 1 — `KpiRollupHourly` entity + EF config + migration `[standard]`
+- **Files (new):** `src/ZB.MOM.WW.ScadaBridge.Commons/Entities/Kpi/KpiRollupHourly.cs`,
+  `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/KpiRollupHourlyEntityTypeConfiguration.cs`,
+  a new `Migrations/*AddKpiRollupHourlyTable.cs` (append after `20260710…`).
+- **Files (edit):** `.../ConfigurationDatabase/ScadaBridgeDbContext.cs` (DbSet).
+- Columns: 4-tuple series key (same varchar sizes as `KpiSample`) + `HourStartUtc`
+  (`datetime2`, hour-truncated UTC) + `Value` (folded aggregate) + `MinValue` +
+  `MaxValue` + `SampleCount`. Unique index on
+  `(Source, Metric, Scope, ScopeKey, HourStartUtc)` (upsert key + covers range
+  reads); secondary index on `(HourStartUtc)` for purge. Non-partitioned,
+  `[PRIMARY]`, no DB-role restriction (mirror `KpiSample`). **Build before
+  scaffolding the migration** (repo gotcha: `--no-build` emits an empty migration).
+
+### Task 2 — Metric aggregation classification `[small]`
+- **Files:** `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiMetrics.cs` (or a new
+  `KpiMetricAggregation.cs` beside it).
+- Add a `Gauge`/`Rate` classification and a lookup by `(Source, Metric)` covering
+  all charted metrics across the four sources. Rate: `deliveredLastInterval`,
+  `failedLastInterval`, `totalEventsLastHour`, `errorEventsLastHour`,
+  `scriptErrors`, `alarmEvalErrors`, `deadLetters`, `eventLogWriteFailures`.
+  Gauge: everything else (`queueDepth`, `buffered`, `stuck`, `parked*`,
+  `oldestPendingAgeSeconds`, `connectionsUp/Down`, `deployedInstances`, …).
+  Unknown metric → default `Gauge` (last-value) + log, so a new metric can't crash
+  the fold. Unit-test the classification is total over the current catalog.
+
+### Task 3 — Rollup fold repository method `[standard]`
+- **Files:** `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IKpiHistoryRepository.cs`,
+  `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs`
+- `FoldHourlyRollupsAsync(DateTime fromHourUtc, DateTime toHourUtc, ct)`:
+  group `KpiSample` in the window by `(series, hour)`, apply per-metric
+  aggregation (Task 2) → `Value`, plus `MinValue`/`MaxValue`/`SampleCount`;
+  **idempotent upsert** into `KpiRollupHourly` on the unique key. Skip the current
+  (incomplete) hour. Handle the conditionally-emitted metric (missing rows in some
+  minutes) gracefully.
+- `GetHourlySeriesAsync(source, metric, scope, scopeKey, fromUtc, toUtc, ct)` →
+  `IReadOnlyList<KpiSeriesPoint>` (project `HourStartUtc`→`CapturedAtUtc`,
+  `Value`), ordered ascending — same contract as `GetRawSeriesAsync` so the
+  bucketer is agnostic.
+- `PurgeRollupsOlderThanAsync(before, ct)` — 1-hour-sliced batch DELETE (mirror
+  `PurgeOlderThanAsync`).
+
+### Task 4 — Recorder hourly tick + rollup purge `[standard]`
+- **Files:** `src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs`,
+  `.../KpiHistoryOptions.cs`, `.../KpiHistoryOptionsValidator.cs`
+- New Akka timer `RollupTick` at `RollupInterval` (default 1 h): per-tick scope →
+  `FoldHourlyRollupsAsync(now − RollupLookbackHours, now)`. Re-fold lookback
+  (default 3 h) makes failover-missed hours self-heal via the idempotent upsert.
+  `_rollupInFlight` guard (mirror `_sampleInFlight`). Extend the daily purge tick
+  to also call `PurgeRollupsOlderThanAsync(now − RollupRetentionDays)`.
+- Options: `RollupInterval` (1 h), `RollupLookbackHours` (3), `RollupRetentionDays`
+  (365), `RollupThresholdHours` (168). Validator: all `> 0`; `RollupRetentionDays`
+  bound `[1, 3650]` **and ≥ `RetentionDays`**; `RollupThresholdHours ≥ 24`.
+
+### Task 5 — Query service range routing `[small]`
+- **Files:** `src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/KpiHistoryQueryService.cs`
+- In `GetSeriesAsync`: if `(toUtc − fromUtc).TotalHours > RollupThresholdHours`,
+  call `GetHourlySeriesAsync`, else `GetRawSeriesAsync`; then `Bucket` as today.
+  Both ctors (production `IServiceScopeFactory`, test `IKpiHistoryRepository`)
+  keep working. Unit-test the routing boundary (168 h raw vs 169 h rollup).
+
+### Task 6 — Backfill existing samples once `[standard]`
+- **Files:** startup fold in Host (grep `AkkaHostedService` KPI wiring) OR a data
+  migration.
+- One idempotent `FoldHourlyRollupsAsync(now − RetentionDays, now)` at first
+  startup after deploy (guarded so it runs once / is cheap to re-run). Log rows
+  folded. Prefer the startup fold over a data-migration so it reuses Task 3 logic.
+
+### Task 7 — Add 30 d / 90 d windows to the four surfaces `[standard]`
+- **Files:** `.../Pages/Audit/AuditLogPage.razor.cs`,
+  `.../Pages/SiteCalls/SiteCallsReport.razor.cs`,
+  `.../Pages/Notifications/NotificationKpis.razor`,
+  `.../Pages/Monitoring/Health.razor`
+- Add 30 d (720 h) and 90 d (2160 h) toggle buttons alongside the existing 24 h /
+  7 d. No fetch-call changes — they already pass `fromUtc/toUtc` to the query
+  service, which now transparently routes to rollups for the long windows. This is
+  what actually *exercises* the rollup path (without it, rollups have no caller).
+
+### Task 8 — Tests + docs `[standard]`
+- **Files:** `tests/.../Commons.Tests/*` (bucketer unchanged; classification;
+  fold aggregation math — especially rate-sum vs gauge-last),
+  `tests/.../ConfigurationDatabase.Tests/*` (fold upsert idempotency, purge,
+  `GetHourlySeriesAsync`), `tests/.../KpiHistory.Tests/*` (rollup tick + purge +
+  in-flight guard + failover re-fold), `tests/.../CentralUI.Tests/*` (query
+  routing; new window buttons).
+- Docs: `docs/requirements/Component-KpiHistory.md` (schema section for the rollup
+  table, retention section, query section: raw≤threshold / rollup>threshold, new
+  windows), `docs/plans/2026-06-17-m6-kpi-history-design.md` (mark downsampling
+  deferral delivered), and move register #22 to **Resolved**.
+
+## 5. Risks / call-outs
+- **Rate-metric undercount** — the plan's biggest correctness risk; Task 2's
+  classification + Task 3's per-metric fold must be reviewed together. A rate
+  metric folded as last-value silently misreports long-range totals.
+- **Failover-missed hour** — the lookback re-fold + idempotent upsert is the
+  backstop; verify a simulated missed tick self-heals.
+- **Retention coherence** — `RollupRetentionDays ≥ RetentionDays` must be
+  validator-enforced or long-range charts get holes where raw is purged but
+  rollups weren't yet written.
+- **Migration hygiene** — build before `migrations add`; delete empty scaffolds
+  (repo gotcha).
+- **No new infra** — like `KpiSample`, the rollup table lives in central MS SQL,
+  non-partitioned; no new services.
+
+## 6. Explicitly out of scope
+- Multi-resolution rollups (daily/weekly) — hourly only; revisit if 1-year+
+  windows appear.
+- Charting `Min`/`Max`/`Avg` bands — the columns are stored to unblock it, but the
+  `KpiTrendChart` still plots a single series in this plan.
+- Changing live-sample cadence or the point-in-time KPI reads (unaffected).