ScadaBridge/docs/plans/2026-07-10-kpi-history-hourly-rollups-plan.md

# Plan: KPI History Hourly Rollups (deferred #22)

**Status:** Draft plan (not yet executed) — 2026-07-10
**Register row:** `docs/plans/2026-07-08-deferred-work-register.md` #22
**Revisit trigger that fired this plan:** *"KpiSample query latency on dashboards."*
**Owning component:** KPI History (#26); touches Configuration Database (#17),
Commons (#16), Central UI (#9), and every `IKpiSampleSource` owner.

---

## 1. Problem

`KpiSample` is a tall/EAV table sampled once per minute per series
(`(Source, Metric, Scope, ScopeKey)` 4-tuple). Long-range trend queries fetch and
sort every raw row in the window before `KpiSeriesBucketer` downsamples to
`DefaultMaxSeriesPoints = 200`:

| Window | Raw rows/series @ 1/min |
|--------|-------------------------|
| 24 h   | 1,440 |
| 7 d    | 10,080 |
| 30 d   | 43,200 |
| 90 d   | 129,600 |

The four trend surfaces (Notification Outbox, Site Calls, Audit Log, Health) today
offer only **24 h** and **7 d** windows, so the worst case is ~10k rows/series.
The latency trigger only truly bites once **30 d / 90 d** windows are added — so
this plan must decide whether to add those windows *and* pre-aggregate them, or
rollups have no caller.

## 2. Key facts that shape the design (from the code)

- **The bucketer is not the bottleneck.** `KpiSeriesBucketer.Bucket` derives bucket
  width purely from `(to − from) / maxPoints` *after* rows are in memory. The cost
  is entirely in `GetRawSeriesAsync` fetching/sorting rows. Hourly rollups plug in
  by feeding the bucketer a **pre-thinned ≤1-row/hour series** (≈60× fewer rows).
- **`KpiHistoryQueryService.GetSeriesAsync` already owns fetch→bucket + options** —
  the natural place to branch raw-vs-rollup by range.
- **`KpiHistoryRecorderActor` takes the root `IServiceProvider` and opens a
  per-tick scope** — a new hourly rollup timer needs **no constructor change**,
  just a new timer + repo method (same shape as the existing sample/purge timers).
- **Metrics are not all gauges.** Most are instantaneous gauges (`queueDepth`,
  `buffered`, `connectionsDown`, `oldestPendingAgeSeconds`) where last-value-per-
  hour is correct. But several are **already-windowed rate counters**
  (`deliveredLastInterval`, `failedLastInterval`, `*LastHour`, `scriptErrors`,
  `deadLetters`) where last-value-per-hour **undercounts** (keeps one minute's
  delta, discards the other 59). This is the single biggest correctness decision.
- Some metrics are **conditionally emitted** (`oldestPendingAgeSeconds` skipped
  when null) — a rollup must not assume a row exists every minute.
- **This is greenfield** — no `rollup`/`KpiRollup` table/column/code exists.

## 3. Chosen approach

**A separate `KpiRollupHourly` table, populated by an hourly recorder tick, with
per-metric aggregation intent, queried via a range threshold, retained longer than
raw samples, and backfilled once at migration.** Also add **30 d / 90 d** windows
to the four surfaces so the rollups have a caller.

Rationale for each decision (the research surfaced these as the required choices):

**(a) Separate table vs. computed-on-read → separate `KpiRollupHourly` table.**
Computed-on-read (`GROUP BY DATEADD(hour…)`) needs no migration/backfill but still
**scans all raw rows** in SQL every query — it shrinks rows *returned*, not rows
*scanned*, so it only partially addresses the latency trigger. A pre-aggregated
table (design-consistent with the non-partitioned `KpiSample` model) makes a 90 d
read a small indexed scan (~2,160 rows vs ~130k).

**(b) Who computes → recorder-side hourly tick.** Add a third Akka timer
(`RollupTick`) to `KpiHistoryRecorderActor`. Fold the **trailing N hours**
(lookback window, not just "last hour") so a singleton-failover handover that
misses a tick re-folds on recovery. Idempotent **upsert** keyed on
`(Source, Metric, Scope, ScopeKey, HourStartUtc)`. Query-time fold was rejected
because it doesn't reduce SQL scan (see (a)).

**(c) Range routing → in `KpiHistoryQueryService.GetSeriesAsync`.** New option
`RollupThresholdHours` (default 168 = 7 d): windows ≤ threshold → `GetRawSeriesAsync`
(preserves intra-minute detail on 24 h/7 d); windows > threshold →
`GetHourlySeriesAsync`. `KpiSeriesBucketer.Bucket` then runs unchanged on whichever
series (note 90 d rollup = 2,160 rows > 200, so bucketing still applies on top).

**(d) Retention → rollups outlive raw.** Keep raw `KpiSample` at 90 d (existing
purge). Retain `KpiRollupHourly` much longer (new `RollupRetentionDays`, default
365) via a **second purge pass** mirroring the 1-hour-sliced batch DELETE in
`PurgeOlderThanAsync`. Validator must enforce `RollupRetentionDays ≥ RetentionDays`.
This delivers *both* read-speed and history depth; without it the feature only
buys speed.

**(e) Backfill → one-shot fold at rollout.** Fold existing ≤90 d of `KpiSample`
into `KpiRollupHourly` once (data migration or idempotent startup fold), so
30 d/90 d charts aren't blank until enough wall-clock passes. The idempotent
upsert key makes a re-run safe.

**Cross-cutting correctness — per-metric aggregation.** Classify each charted
metric as **gauge** (store last-value-per-hour) or **rate** (store sum-per-hour);
store `Value` plus `Min`/`Max`/`Count` so the fold is faithful and future avg/min/
max charts are unblocked. A metric→kind classification lives next to the
`KpiMetrics` catalog. Rate metrics folded with `last-value` would silently
undercount long-range totals — this must be explicit, not incidental.

## 4. Task breakdown

### Task 1 — `KpiRollupHourly` entity + EF config + migration `[standard]`
- **Files (new):** `src/ZB.MOM.WW.ScadaBridge.Commons/Entities/Kpi/KpiRollupHourly.cs`,
  `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/KpiRollupHourlyEntityTypeConfiguration.cs`,
  a new `Migrations/*AddKpiRollupHourlyTable.cs` (append after `20260710…`).
- **Files (edit):** `.../ConfigurationDatabase/ScadaBridgeDbContext.cs` (DbSet).
- Columns: 4-tuple series key (same varchar sizes as `KpiSample`) + `HourStartUtc`
  (`datetime2`, hour-truncated UTC) + `Value` (folded aggregate) + `MinValue` +
  `MaxValue` + `SampleCount`. Unique index on
  `(Source, Metric, Scope, ScopeKey, HourStartUtc)` (upsert key + covers range
  reads); secondary index on `(HourStartUtc)` for purge. Non-partitioned,
  `[PRIMARY]`, no DB-role restriction (mirror `KpiSample`). **Build before
  scaffolding the migration** (repo gotcha: `--no-build` emits an empty migration).

### Task 2 — Metric aggregation classification `[small]`
- **Files:** `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiMetrics.cs` (or a new
  `KpiMetricAggregation.cs` beside it).
- Add a `Gauge`/`Rate` classification and a lookup by `(Source, Metric)` covering
  all charted metrics across the four sources. Rate: `deliveredLastInterval`,
  `failedLastInterval`, `totalEventsLastHour`, `errorEventsLastHour`,
  `scriptErrors`, `alarmEvalErrors`, `deadLetters`, `eventLogWriteFailures`.
  Gauge: everything else (`queueDepth`, `buffered`, `stuck`, `parked*`,
  `oldestPendingAgeSeconds`, `connectionsUp/Down`, `deployedInstances`, …).
  Unknown metric → default `Gauge` (last-value) + log, so a new metric can't crash
  the fold. Unit-test the classification is total over the current catalog.

### Task 3 — Rollup fold repository method `[standard]`
- **Files:** `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IKpiHistoryRepository.cs`,
  `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs`
- `FoldHourlyRollupsAsync(DateTime fromHourUtc, DateTime toHourUtc, ct)`:
  group `KpiSample` in the window by `(series, hour)`, apply per-metric
  aggregation (Task 2) → `Value`, plus `MinValue`/`MaxValue`/`SampleCount`;
  **idempotent upsert** into `KpiRollupHourly` on the unique key. Skip the current
  (incomplete) hour. Handle the conditionally-emitted metric (missing rows in some
  minutes) gracefully.
- `GetHourlySeriesAsync(source, metric, scope, scopeKey, fromUtc, toUtc, ct)` →
  `IReadOnlyList<KpiSeriesPoint>` (project `HourStartUtc`→`CapturedAtUtc`,
  `Value`), ordered ascending — same contract as `GetRawSeriesAsync` so the
  bucketer is agnostic.
- `PurgeRollupsOlderThanAsync(before, ct)` — 1-hour-sliced batch DELETE (mirror
  `PurgeOlderThanAsync`).

### Task 4 — Recorder hourly tick + rollup purge `[standard]`
- **Files:** `src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs`,
  `.../KpiHistoryOptions.cs`, `.../KpiHistoryOptionsValidator.cs`
- New Akka timer `RollupTick` at `RollupInterval` (default 1 h): per-tick scope →
  `FoldHourlyRollupsAsync(now − RollupLookbackHours, now)`. Re-fold lookback
  (default 3 h) makes failover-missed hours self-heal via the idempotent upsert.
  `_rollupInFlight` guard (mirror `_sampleInFlight`). Extend the daily purge tick
  to also call `PurgeRollupsOlderThanAsync(now − RollupRetentionDays)`.
- Options: `RollupInterval` (1 h), `RollupLookbackHours` (3), `RollupRetentionDays`
  (365), `RollupThresholdHours` (168). Validator: all `> 0`; `RollupRetentionDays`
  bound `[1, 3650]` **and ≥ `RetentionDays`**; `RollupThresholdHours ≥ 24`.

### Task 5 — Query service range routing `[small]`
- **Files:** `src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/KpiHistoryQueryService.cs`
- In `GetSeriesAsync`: if `(toUtc − fromUtc).TotalHours > RollupThresholdHours`,
  call `GetHourlySeriesAsync`, else `GetRawSeriesAsync`; then `Bucket` as today.
  Both ctors (production `IServiceScopeFactory`, test `IKpiHistoryRepository`)
  keep working. Unit-test the routing boundary (168 h raw vs 169 h rollup).

### Task 6 — Backfill existing samples once `[standard]`
- **Files:** startup fold in Host (grep `AkkaHostedService` KPI wiring) OR a data
  migration.
- One idempotent `FoldHourlyRollupsAsync(now − RetentionDays, now)` at first
  startup after deploy (guarded so it runs once / is cheap to re-run). Log rows
  folded. Prefer the startup fold over a data-migration so it reuses Task 3 logic.

### Task 7 — Add 30 d / 90 d windows to the four surfaces `[standard]`
- **Files:** `.../Pages/Audit/AuditLogPage.razor.cs`,
  `.../Pages/SiteCalls/SiteCallsReport.razor.cs`,
  `.../Pages/Notifications/NotificationKpis.razor`,
  `.../Pages/Monitoring/Health.razor`
- Add 30 d (720 h) and 90 d (2160 h) toggle buttons alongside the existing 24 h /
  7 d. No fetch-call changes — they already pass `fromUtc/toUtc` to the query
  service, which now transparently routes to rollups for the long windows. This is
  what actually *exercises* the rollup path (without it, rollups have no caller).

### Task 8 — Tests + docs `[standard]`
- **Files:** `tests/.../Commons.Tests/*` (bucketer unchanged; classification;
  fold aggregation math — especially rate-sum vs gauge-last),
  `tests/.../ConfigurationDatabase.Tests/*` (fold upsert idempotency, purge,
  `GetHourlySeriesAsync`), `tests/.../KpiHistory.Tests/*` (rollup tick + purge +
  in-flight guard + failover re-fold), `tests/.../CentralUI.Tests/*` (query
  routing; new window buttons).
- Docs: `docs/requirements/Component-KpiHistory.md` (schema section for the rollup
  table, retention section, query section: raw≤threshold / rollup>threshold, new
  windows), `docs/plans/2026-06-17-m6-kpi-history-design.md` (mark downsampling
  deferral delivered), and move register #22 to **Resolved**.

## 5. Risks / call-outs
- **Rate-metric undercount** — the plan's biggest correctness risk; Task 2's
  classification + Task 3's per-metric fold must be reviewed together. A rate
  metric folded as last-value silently misreports long-range totals.
- **Failover-missed hour** — the lookback re-fold + idempotent upsert is the
  backstop; verify a simulated missed tick self-heals.
- **Retention coherence** — `RollupRetentionDays ≥ RetentionDays` must be
  validator-enforced or long-range charts get holes where raw is purged but
  rollups weren't yet written.
- **Migration hygiene** — build before `migrations add`; delete empty scaffolds
  (repo gotcha).
- **No new infra** — like `KpiSample`, the rollup table lives in central MS SQL,
  non-partitioned; no new services.

## 6. Explicitly out of scope
- Multi-resolution rollups (daily/weekly) — hourly only; revisit if 1-year+
  windows appear.
- Charting `Min`/`Max`/`Avg` bands — the columns are stored to unblock it, but the
  `KpiTrendChart` still plots a single series in this plan.
- Changing live-sample cadence or the point-in-time KPI reads (unaffected).