ScadaBridge/docs/plans/2026-07-10-kpi-history-hourly-rollups-plan.md

Plan: KPI History Hourly Rollups (deferred #22)

Status: Draft plan (not yet executed) — 2026-07-10 Register row: docs/plans/2026-07-08-deferred-work-register.md #22 Revisit trigger that fired this plan: "KpiSample query latency on dashboards." Owning component: KPI History (#26); touches Configuration Database (#17), Commons (#16), Central UI (#9), and every IKpiSampleSource owner.

---

1. Problem

KpiSample is a tall/EAV table sampled once per minute per series ((Source, Metric, Scope, ScopeKey) 4-tuple). Long-range trend queries fetch and sort every raw row in the window before KpiSeriesBucketer downsamples to DefaultMaxSeriesPoints = 200:

Window	Raw rows/series @ 1/min
24 h	1,440
7 d	10,080
30 d	43,200
90 d	129,600

The four trend surfaces (Notification Outbox, Site Calls, Audit Log, Health) today offer only 24 h and 7 d windows, so the worst case is ~10k rows/series. The latency trigger only truly bites once 30 d / 90 d windows are added — so this plan must decide whether to add those windows and pre-aggregate them, or rollups have no caller.

2. Key facts that shape the design (from the code)

• The bucketer is not the bottleneck. KpiSeriesBucketer.Bucket derives bucket width purely from (to − from) / maxPoints after rows are in memory. The cost is entirely in GetRawSeriesAsync fetching/sorting rows. Hourly rollups plug in by feeding the bucketer a pre-thinned ≤1-row/hour series (≈60× fewer rows).
• KpiHistoryQueryService.GetSeriesAsync already owns fetch→bucket + options — the natural place to branch raw-vs-rollup by range.
• KpiHistoryRecorderActor takes the root IServiceProvider and opens a per-tick scope — a new hourly rollup timer needs no constructor change, just a new timer + repo method (same shape as the existing sample/purge timers).
• Metrics are not all gauges. Most are instantaneous gauges (queueDepth, buffered, connectionsDown, oldestPendingAgeSeconds) where last-value-per- hour is correct. But several are already-windowed rate counters (deliveredLastInterval, failedLastInterval, *LastHour, scriptErrors, deadLetters) where last-value-per-hour undercounts (keeps one minute's delta, discards the other 59). This is the single biggest correctness decision.
• Some metrics are conditionally emitted (oldestPendingAgeSeconds skipped when null) — a rollup must not assume a row exists every minute.
• This is greenfield — no rollup/KpiRollup table/column/code exists.

3. Chosen approach

A separate KpiRollupHourly table, populated by an hourly recorder tick, with per-metric aggregation intent, queried via a range threshold, retained longer than raw samples, and backfilled once at migration. Also add 30 d / 90 d windows to the four surfaces so the rollups have a caller.

Rationale for each decision (the research surfaced these as the required choices):

(a) Separate table vs. computed-on-read → separate KpiRollupHourly table. Computed-on-read (GROUP BY DATEADD(hour…)) needs no migration/backfill but still scans all raw rows in SQL every query — it shrinks rows returned, not rows scanned, so it only partially addresses the latency trigger. A pre-aggregated table (design-consistent with the non-partitioned KpiSample model) makes a 90 d read a small indexed scan (~2,160 rows vs ~130k).

(b) Who computes → recorder-side hourly tick. Add a third Akka timer (RollupTick) to KpiHistoryRecorderActor. Fold the trailing N hours (lookback window, not just "last hour") so a singleton-failover handover that misses a tick re-folds on recovery. Idempotent upsert keyed on (Source, Metric, Scope, ScopeKey, HourStartUtc). Query-time fold was rejected because it doesn't reduce SQL scan (see (a)).

(c) Range routing → in KpiHistoryQueryService.GetSeriesAsync. New option RollupThresholdHours (default 168 = 7 d): windows ≤ threshold → GetRawSeriesAsync (preserves intra-minute detail on 24 h/7 d); windows > threshold → GetHourlySeriesAsync. KpiSeriesBucketer.Bucket then runs unchanged on whichever series (note 90 d rollup = 2,160 rows > 200, so bucketing still applies on top).

(d) Retention → rollups outlive raw. Keep raw KpiSample at 90 d (existing purge). Retain KpiRollupHourly much longer (new RollupRetentionDays, default 365) via a second purge pass mirroring the 1-hour-sliced batch DELETE in PurgeOlderThanAsync. Validator must enforce RollupRetentionDays ≥ RetentionDays. This delivers both read-speed and history depth; without it the feature only buys speed.

(e) Backfill → one-shot fold at rollout. Fold existing ≤90 d of KpiSample into KpiRollupHourly once (data migration or idempotent startup fold), so 30 d/90 d charts aren't blank until enough wall-clock passes. The idempotent upsert key makes a re-run safe.

Cross-cutting correctness — per-metric aggregation. Classify each charted metric as gauge (store last-value-per-hour) or rate (store sum-per-hour); store Value plus Min/Max/Count so the fold is faithful and future avg/min/ max charts are unblocked. A metric→kind classification lives next to the KpiMetrics catalog. Rate metrics folded with last-value would silently undercount long-range totals — this must be explicit, not incidental.

4. Task breakdown

Task 1 — KpiRollupHourly entity + EF config + migration [standard]

• Files (new): src/ZB.MOM.WW.ScadaBridge.Commons/Entities/Kpi/KpiRollupHourly.cs, src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/KpiRollupHourlyEntityTypeConfiguration.cs, a new Migrations/*AddKpiRollupHourlyTable.cs (append after 20260710…).
• Files (edit): .../ConfigurationDatabase/ScadaBridgeDbContext.cs (DbSet).
• Columns: 4-tuple series key (same varchar sizes as KpiSample) + HourStartUtc (datetime2, hour-truncated UTC) + Value (folded aggregate) + MinValue + MaxValue + SampleCount. Unique index on (Source, Metric, Scope, ScopeKey, HourStartUtc) (upsert key + covers range reads); secondary index on (HourStartUtc) for purge. Non-partitioned, [PRIMARY], no DB-role restriction (mirror KpiSample). Build before scaffolding the migration (repo gotcha: --no-build emits an empty migration).

Task 2 — Metric aggregation classification [small]

• Files: src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiMetrics.cs (or a new KpiMetricAggregation.cs beside it).
• Add a Gauge/Rate classification and a lookup by (Source, Metric) covering all charted metrics across the four sources. Rate: deliveredLastInterval, failedLastInterval, totalEventsLastHour, errorEventsLastHour, scriptErrors, alarmEvalErrors, deadLetters, eventLogWriteFailures. Gauge: everything else (queueDepth, buffered, stuck, parked*, oldestPendingAgeSeconds, connectionsUp/Down, deployedInstances, …). Unknown metric → default Gauge (last-value) + log, so a new metric can't crash the fold. Unit-test the classification is total over the current catalog.

Task 3 — Rollup fold repository method [standard]

• Files: src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IKpiHistoryRepository.cs, src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs
• FoldHourlyRollupsAsync(DateTime fromHourUtc, DateTime toHourUtc, ct): group KpiSample in the window by (series, hour), apply per-metric aggregation (Task 2) → Value, plus MinValue/MaxValue/SampleCount; idempotent upsert into KpiRollupHourly on the unique key. Skip the current (incomplete) hour. Handle the conditionally-emitted metric (missing rows in some minutes) gracefully.
• GetHourlySeriesAsync(source, metric, scope, scopeKey, fromUtc, toUtc, ct) → IReadOnlyList<KpiSeriesPoint> (project HourStartUtc→CapturedAtUtc, Value), ordered ascending — same contract as GetRawSeriesAsync so the bucketer is agnostic.
• PurgeRollupsOlderThanAsync(before, ct) — 1-hour-sliced batch DELETE (mirror PurgeOlderThanAsync).

Task 4 — Recorder hourly tick + rollup purge [standard]

• Files: src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs, .../KpiHistoryOptions.cs, .../KpiHistoryOptionsValidator.cs
• New Akka timer RollupTick at RollupInterval (default 1 h): per-tick scope → FoldHourlyRollupsAsync(now − RollupLookbackHours, now). Re-fold lookback (default 3 h) makes failover-missed hours self-heal via the idempotent upsert. _rollupInFlight guard (mirror _sampleInFlight). Extend the daily purge tick to also call PurgeRollupsOlderThanAsync(now − RollupRetentionDays).
• Options: RollupInterval (1 h), RollupLookbackHours (3), RollupRetentionDays (365), RollupThresholdHours (168). Validator: all > 0; RollupRetentionDays bound [1, 3650] and ≥ RetentionDays; RollupThresholdHours ≥ 24.

Task 5 — Query service range routing [small]

• Files: src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/KpiHistoryQueryService.cs
• In GetSeriesAsync: if (toUtc − fromUtc).TotalHours > RollupThresholdHours, call GetHourlySeriesAsync, else GetRawSeriesAsync; then Bucket as today. Both ctors (production IServiceScopeFactory, test IKpiHistoryRepository) keep working. Unit-test the routing boundary (168 h raw vs 169 h rollup).

Task 6 — Backfill existing samples once [standard]

• Files: startup fold in Host (grep AkkaHostedService KPI wiring) OR a data migration.
• One idempotent FoldHourlyRollupsAsync(now − RetentionDays, now) at first startup after deploy (guarded so it runs once / is cheap to re-run). Log rows folded. Prefer the startup fold over a data-migration so it reuses Task 3 logic.

Task 7 — Add 30 d / 90 d windows to the four surfaces [standard]

• Files: .../Pages/Audit/AuditLogPage.razor.cs, .../Pages/SiteCalls/SiteCallsReport.razor.cs, .../Pages/Notifications/NotificationKpis.razor, .../Pages/Monitoring/Health.razor
• Add 30 d (720 h) and 90 d (2160 h) toggle buttons alongside the existing 24 h / 7 d. No fetch-call changes — they already pass fromUtc/toUtc to the query service, which now transparently routes to rollups for the long windows. This is what actually exercises the rollup path (without it, rollups have no caller).

Task 8 — Tests + docs [standard]

• Files: tests/.../Commons.Tests/* (bucketer unchanged; classification; fold aggregation math — especially rate-sum vs gauge-last), tests/.../ConfigurationDatabase.Tests/* (fold upsert idempotency, purge, GetHourlySeriesAsync), tests/.../KpiHistory.Tests/* (rollup tick + purge + in-flight guard + failover re-fold), tests/.../CentralUI.Tests/* (query routing; new window buttons).
• Docs: docs/requirements/Component-KpiHistory.md (schema section for the rollup table, retention section, query section: raw≤threshold / rollup>threshold, new windows), docs/plans/2026-06-17-m6-kpi-history-design.md (mark downsampling deferral delivered), and move register #22 to Resolved.

5. Risks / call-outs

• Rate-metric undercount — the plan's biggest correctness risk; Task 2's classification + Task 3's per-metric fold must be reviewed together. A rate metric folded as last-value silently misreports long-range totals.
• Failover-missed hour — the lookback re-fold + idempotent upsert is the backstop; verify a simulated missed tick self-heals.
• Retention coherence — RollupRetentionDays ≥ RetentionDays must be validator-enforced or long-range charts get holes where raw is purged but rollups weren't yet written.
• Migration hygiene — build before migrations add; delete empty scaffolds (repo gotcha).
• No new infra — like KpiSample, the rollup table lives in central MS SQL, non-partitioned; no new services.

6. Explicitly out of scope

• Multi-resolution rollups (daily/weekly) — hourly only; revisit if 1-year+ windows appear.
• Charting Min/Max/Avg bands — the columns are stored to unblock it, but the KpiTrendChart still plots a single series in this plan.
• Changing live-sample cadence or the point-in-time KPI reads (unaffected).