From c9fa3b07f4028f101e3df75ef36e967d0ba8c493 Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Fri, 10 Jul 2026 12:18:46 -0400 Subject: [PATCH] docs(kpihistory): document hourly rollups + resolve deferred #22 (plan #22 T8) Update Component-KpiHistory.md with the KpiRollupHourly schema, the third recorder rollup tick + one-shot backfill, dual raw/rollup purge retention, raw-vs-rollup query routing by RollupThresholdHours, per-metric gauge-vs-rate aggregation (KpiMetricAggregationCatalog), the four new options + bounds, and 30d/90d trend windows. Append a Delivered-2026-07-10 note to the m6 design plan, move register row #22 from Deferred to Resolved, and note rollups in the CLAUDE.md KPI History bullet. Docs-only; no code changed. Claude-Session: https://claude.ai/code/session_01MtdgwpEeCUn6cUA5f1LMPj --- CLAUDE.md | 2 +- .../plans/2026-06-17-m6-kpi-history-design.md | 7 ++ .../2026-07-08-deferred-work-register.md | 2 +- docs/requirements/Component-KpiHistory.md | 77 +++++++++++++++---- 4 files changed, 72 insertions(+), 16 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index d6c45deb..e6748306 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -212,7 +212,7 @@ Other peers in the `scadaproj` family (see `scadaproj/CLAUDE.md` for details): ` - Stuck = `Pending`/`Retrying` older than a configurable age threshold (default 10 min) — display-only (KPI count + row badge), no escalation/alerting. - Headline KPI tiles surface on the Health dashboard; a new Central UI Notification Outbox page offers a queryable list with Retry/Discard actions on parked notifications. - Site Call Audit KPIs are central-computed point-in-time from the `SiteCalls` table (global + per-site), mirroring the Notification Outbox KPI shape; tiles surface on the Health dashboard alongside a queryable Central UI Site Calls page with Retry/Discard on parked rows. -- KPI History & Trends (#26, M6): a reusable central KPI-history backbone — supersedes the prior "point-in-time only, no time-series store" stance — backed by a tall/EAV `KpiSample` table in central MS SQL (no new infra). A `KpiHistoryRecorderActor` cluster singleton (`kpi-history-recorder`, **not readiness-gated**, best-effort with per-source isolation) samples every minute by enumerating DI-registered `IKpiSampleSource`s (each lives with its owner, registered via `TryAddEnumerable`, reusing existing KPI/aggregator reads); daily purge after `RetentionDays` (default 90). Querying is `IKpiHistoryRepository.GetRawSeriesAsync` → `KpiSeriesBucketer` (last-value-per-bucket) → scoped dual-ctor `KpiHistoryQueryService` → a reusable **custom-SVG** `KpiTrendChart` (no third-party charting lib). Trends ship on four surfaces: Notification Outbox, Site Calls, Audit Log pages + a per-site Health-dashboard panel. `KpiHistoryOptions` (`ScadaBridge:KpiHistory`): SampleInterval 60s, RetentionDays 90, PurgeInterval 1d, DefaultMaxSeriesPoints 200; validated. M6's T9 and T10 were originally deferred. **T9/T10 delivered 2026-06-19** as an SMS (Twilio) adapter — Teams was evaluated and dropped (outbound-only backends cannot send 1:1 chat bodies via Graph without a Bot Framework inbound endpoint; SMS is inherently per-person and outbound-only). `NotificationType.Sms` added; `SmsNotificationDeliveryAdapter` (Twilio REST, no SDK) registered alongside Email; Central UI Type selector (adapter-gated) + SMS recipient input; CLI `--type sms --phones` on list create/update; `notification sms list|update` commands. +- KPI History & Trends (#26, M6): a reusable central KPI-history backbone — supersedes the prior "point-in-time only, no time-series store" stance — backed by a tall/EAV `KpiSample` table in central MS SQL (no new infra). A `KpiHistoryRecorderActor` cluster singleton (`kpi-history-recorder`, **not readiness-gated**, best-effort with per-source isolation) samples every minute by enumerating DI-registered `IKpiSampleSource`s (each lives with its owner, registered via `TryAddEnumerable`, reusing existing KPI/aggregator reads); daily purge after `RetentionDays` (default 90). Querying is `IKpiHistoryRepository.GetRawSeriesAsync` → `KpiSeriesBucketer` (last-value-per-bucket) → scoped dual-ctor `KpiHistoryQueryService` → a reusable **custom-SVG** `KpiTrendChart` (no third-party charting lib). Trends ship on four surfaces: Notification Outbox, Site Calls, Audit Log pages + a per-site Health-dashboard panel. `KpiHistoryOptions` (`ScadaBridge:KpiHistory`): SampleInterval 60s, RetentionDays 90, PurgeInterval 1d, DefaultMaxSeriesPoints 200; validated. **Hourly rollups shipped 2026-07-10 (deferred #22):** a second `KpiRollupHourly` table folded by a third recorder tick (`kpi-rollup`, `RollupInterval` 1h, re-folds `RollupLookbackHours` window via idempotent failover-self-healing upsert + one-shot backfill on start), per-metric gauge-vs-rate fold (`KpiMetricAggregationCatalog`), and raw-vs-rollup query routing by `RollupThresholdHours` (168h) unblock long-range **30d/90d** trend windows (rollups retained `RollupRetentionDays` 365 ≥ raw 90, dual daily purge). M6's T9 and T10 were originally deferred. **T9/T10 delivered 2026-06-19** as an SMS (Twilio) adapter — Teams was evaluated and dropped (outbound-only backends cannot send 1:1 chat bodies via Graph without a Bot Framework inbound endpoint; SMS is inherently per-person and outbound-only). `NotificationType.Sms` added; `SmsNotificationDeliveryAdapter` (Twilio REST, no SDK) registered alongside Email; Central UI Type selector (adapter-gated) + SMS recipient input; CLI `--type sms --phones` on list create/update; `notification sms list|update` commands. ### Code Organization - Entity classes are persistence-ignorant POCOs in Commons; EF mappings in Configuration Database. diff --git a/docs/plans/2026-06-17-m6-kpi-history-design.md b/docs/plans/2026-06-17-m6-kpi-history-design.md index d5f276dc..6fbc56e7 100644 --- a/docs/plans/2026-06-17-m6-kpi-history-design.md +++ b/docs/plans/2026-06-17-m6-kpi-history-design.md @@ -95,6 +95,13 @@ skips that source's samples for the tick — it never throws into or disrupts th `RetentionDays` (default 90), reusing the existing purge-scheduler shape. Hourly downsampling beyond N days is deferred (YAGNI). +> **Delivered 2026-07-10** (deferred-work register #22): the hourly-downsampling deferral is +> closed. A `KpiRollupHourly` table folded by a third recorder tick, per-metric gauge-vs-rate +> aggregation, raw-vs-rollup query routing by `RollupThresholdHours`, longer rollup retention, +> a one-shot backfill, and 30 d / 90 d trend windows shipped per +> `docs/plans/2026-07-10-kpi-history-hourly-rollups-plan.md`. See `Component-KpiHistory.md` for +> the shipped design. + ### Sample sources - **`NotificationOutboxKpiSampleSource`** (in NotificationOutbox) → diff --git a/docs/plans/2026-07-08-deferred-work-register.md b/docs/plans/2026-07-08-deferred-work-register.md index 946af816..f688acbc 100644 --- a/docs/plans/2026-07-08-deferred-work-register.md +++ b/docs/plans/2026-07-08-deferred-work-register.md @@ -26,7 +26,6 @@ them and are removed from this table when that plan's task lands. | 17 | Unified notifications+site-calls outbox page | stillpending :118 | Explicit M9 decision to keep two pages | Operator confusion reports | | 18 | Folder drag-drop | same, [PERM] | Permanently closed; menu reorder shipped | — (closed) | | 19 | Bundle signing / cluster-to-cluster pull / differential bundles | transport-design :402 | v1 manifest hash + AES-GCM held sufficient | Non-repudiation requirement across orgs | -| 22 | KPI history hourly rollups | Component-KpiHistory.md | YAGNI; 90-day retention bounds table | KpiSample query latency on dashboards | ## Resolved (verified against the code 2026-07-10) Rows removed from the Deferred table above once confirmed shipped. Kept here for traceability. @@ -41,6 +40,7 @@ Rows removed from the Deferred table above once confirmed shipped. Kept here for | 16 | StubOpcUaClient throws on browse | Already resolved: `StubOpcUaClient` supports browse + address-space search, covered by `StubOpcUaClientBrowseTests`/`StubOpcUaClientSearchTests`. | | 20 | Deployment EXPIRED-row purge | Already resolved (PLAN-04): `PendingDeploymentPurgeActor` central singleton (spawned in `AkkaHostedService`) ticks `IDeploymentManagerRepository.PurgeExpiredPendingDeploymentsAsync` every `CommunicationOptions.PendingDeploymentPurgeInterval` (default 1h), options-validated, tested. | | 21 | SiteAuditBacklogReporter threshold consolidation | Shipped 2026-07-10: `SqliteAuditWriterOptions.BacklogPollIntervalSeconds` (default 30) now drives the reporter's poll cadence; explicit ctor override still wins (tests), non-positive falls back to the 30 s default. Cadence tests added; stale "hard-code / follow-up" class-doc corrected. | +| 22 | KPI history hourly rollups | Shipped 2026-07-10 (`docs/plans/2026-07-10-kpi-history-hourly-rollups-plan.md`, T1–T8): new `KpiRollupHourly` table (migration `20260710153953`) folded by a third recorder tick (`kpi-rollup`, `RollupInterval` default 1h) over a re-folded `RollupLookbackHours` window via an idempotent, failover-self-healing upsert; per-metric gauge-vs-rate aggregation (`KpiMetricAggregationCatalog`); a one-shot backfill of the retention window on start; raw-vs-rollup query routing by `RollupThresholdHours` (default 168h); longer rollup retention (`RollupRetentionDays` default 365 ≥ `RetentionDays`, dual daily purge); and 30 d / 90 d trend windows added to the four surfaces. Options + validator, docs (`Component-KpiHistory.md`), and tests shipped. | ## New deferrals from review 08 (this plan) | Item | Rationale | Revisit trigger | diff --git a/docs/requirements/Component-KpiHistory.md b/docs/requirements/Component-KpiHistory.md index ee55c51c..86099993 100644 --- a/docs/requirements/Component-KpiHistory.md +++ b/docs/requirements/Component-KpiHistory.md @@ -4,7 +4,7 @@ The KPI History component is the central, reusable **KPI-history backbone** — a tall / EAV time-series store, a periodic recorder singleton, a bucketed query API, and a reusable custom SVG trend-chart component. It turns the system's existing point-in-time KPIs into trends, and ships those trends for the **Notification Outbox (#21)**, **Site Call Audit (#22)**, **Audit Log (#23)**, and **Site Health (#11)** sources. -This supersedes the earlier "KPI history — point-in-time only, no separate time-series store is added" stance carried by the Notification Outbox and Site Call Audit KPI sections. M6 explicitly introduces a store. It lives in **central MS SQL** — the existing HA store — so it adds **no new infrastructure dependency**: a single `KpiSample` table, an EF mapping + migration, and a central cluster singleton that samples every minute. +This supersedes the earlier "KPI history — point-in-time only, no separate time-series store is added" stance carried by the Notification Outbox and Site Call Audit KPI sections. M6 explicitly introduces a store. It lives in **central MS SQL** — the existing HA store — so it adds **no new infrastructure dependency**: a raw `KpiSample` table plus an hourly `KpiRollupHourly` rollup table (the latter added for the deferred #22 long-range-query slice), EF mappings + migrations, and a central cluster singleton that samples every minute and folds hourly rollups. The backbone is deliberately source-agnostic. Each owning component contributes an `IKpiSampleSource` registered into DI; the recorder enumerates them. KPI History therefore does **not** reference every component, and every source reuses the KPI reads its owner already computes — no per-source schema or storage work. @@ -12,19 +12,20 @@ The backbone is deliberately source-agnostic. Each owning component contributes - `src/ZB.MOM.WW.ScadaBridge.KpiHistory` — the component project: the `KpiHistoryRecorderActor`, `KpiHistoryOptions` + validator, and the DI/options wiring (`ServiceCollectionExtensions`). It owns the recorder, the options, and consumes the `IKpiSampleSource` abstraction (defined in Commons). - **`IKpiSampleSource` implementations live with their owners**, not here — `NotificationOutboxKpiSampleSource` (in NotificationOutbox), `SiteCallAuditKpiSampleSource` (in SiteCallAudit), `AuditLogKpiSampleSource` (in AuditLog), `SiteHealthKpiSampleSource` (in HealthMonitoring). Each registers itself via `TryAddEnumerable`. -- **Commons** — the `KpiSample` POCO entity (`Entities/Kpi`), the `IKpiSampleSource` and `IKpiHistoryRepository` interfaces (`Interfaces/Kpi`), and the `KpiSources` / `KpiScopes` constant catalogs + `KpiSeriesPoint` / `KpiSeriesBucketer` types (`Types/Kpi`). -- **Configuration Database** — the EF mapping (`KpiSampleEntityTypeConfiguration`), the migration that creates the `KpiSample` table + indexes, and the `KpiHistoryRepository` implementation. +- **Commons** — the `KpiSample` and `KpiRollupHourly` POCO entities (`Entities/Kpi`), the `IKpiSampleSource` and `IKpiHistoryRepository` interfaces (`Interfaces/Kpi`), and the `KpiSources` / `KpiScopes` constant catalogs + `KpiSeriesPoint` / `KpiSeriesBucketer` types and the `KpiRollupAggregation` enum + `KpiMetricAggregationCatalog` (`Types/Kpi`). +- **Configuration Database** — the EF mappings (`KpiSampleEntityTypeConfiguration`, `KpiRollupHourlyEntityTypeConfiguration`), the migrations that create the `KpiSample` and `KpiRollupHourly` tables + indexes, and the `KpiHistoryRepository` implementation (raw record/query/purge plus the hourly fold / rollup query / rollup purge). - **Central UI** — the `KpiHistoryQueryService` query service and the reusable `KpiTrendChart.razor` component, plus the trend sections embedded on four surfaces. The recorder is a **singleton on the active central node**, consistent with the other central singletons (Notification Outbox, Site Call Audit, purge actors). ## Responsibilities -- Own the `KpiSample` table — the central tall / EAV KPI-history store in MS SQL. +- Own the `KpiSample` and `KpiRollupHourly` tables — the central tall / EAV KPI-history store (raw + hourly rollups) in MS SQL. - Run the recorder loop: every `SampleInterval`, enumerate all registered `IKpiSampleSource`s and persist their samples stamped with one shared tick timestamp. - Isolate sources from one another and from the store: a failure in any one source (or in the write) is logged and skipped for that tick and never disrupts the source component or the rest of the tick (best-effort observability). -- Purge aged rows on a daily cadence (`PurgeInterval`) older than `RetentionDays`. -- Provide a bucketed series-query API (`IKpiHistoryRepository.GetRawSeriesAsync` + `KpiSeriesBucketer`) and the Central UI query service + reusable trend chart that consume it. +- Fold raw samples into hourly rollups on a `RollupInterval` cadence (with a one-shot backfill on start), applying per-metric gauge-vs-rate aggregation via an idempotent, failover-self-healing upsert. +- Purge aged rows on a daily cadence (`PurgeInterval`): raw rows older than `RetentionDays` and rollup rows older than the longer `RollupRetentionDays`. +- Provide a bucketed series-query API (`IKpiHistoryRepository.GetRawSeriesAsync` / `GetHourlySeriesAsync` + `KpiSeriesBucketer`, routed raw-vs-rollup by window width) and the Central UI query service + reusable trend chart that consume it. KPI History is **observability, never a user-facing critical path** — neither recording nor querying may ever break a hosting page or disrupt a source component. @@ -49,6 +50,32 @@ Two named indexes back the access paths: - **`IX_KpiSample_Series` (`Source`, `Metric`, `Scope`, `ScopeKey`, `CapturedAtUtc`)** — the per-series range query (one series scanned in time order). - **`IX_KpiSample_Captured` (`CapturedAtUtc`)** — the retention purge. +## Schema — `KpiRollupHourly` (hourly rollups) + +Long-range trend queries (30 d / 90 d) would fetch and sort tens of thousands of raw `KpiSample` rows per series (~43k at 30 d, ~130k at 90 d @ 1/min) before bucketing. The recorder therefore pre-aggregates raw samples into an **hourly rollup** table, `KpiRollupHourly` — a second tall / EAV table in the same central MS SQL database (no new infrastructure), keyed by the **identical four-tuple series key** plus a truncated `HourStartUtc`. A 90 d read becomes a small indexed scan (~2,160 rows/series vs ~130k). One row per `(Source, Metric, Scope, ScopeKey)` per UTC hour: + +| Column | Type | Notes | +|---|---|---| +| `Id` | `bigint` PK identity | Surrogate key assigned by the store. | +| `Source` | `varchar(64)` | Same `KpiSources` constant as `KpiSample`. | +| `Metric` | `varchar(64)` | Same per-source metric name as `KpiSample`. | +| `Scope` | `varchar(16)` | A `KpiScopes` constant: `Global` / `Site` / `Node`. | +| `ScopeKey` | `varchar(64)` NULL | Site id / node name; `NULL` for `Global`. | +| `HourStartUtc` | `datetime2` | Hour-truncated UTC bucket start (minutes/seconds/ticks zeroed). | +| `Value` | `float` (`double`) | The folded aggregate — a **sum-per-hour** for rate metrics or a **last-value-per-hour** for gauge metrics (see Query — aggregation intent). | +| `MinValue` | `float` (`double`) | Minimum raw sample value observed within the hour. | +| `MaxValue` | `float` (`double`) | Maximum raw sample value observed within the hour. | +| `SampleCount` | `int` | Number of raw `KpiSample` rows folded into the hour. | + +`MinValue` / `MaxValue` / `SampleCount` preserve the fold's fidelity and unblock future min/max/avg charting; the current `KpiTrendChart` still plots the single `Value` series (see Query). + +Two named indexes back the access paths, mirroring `KpiSample`: + +- **`IX_KpiRollupHourly_Series` (`Source`, `Metric`, `Scope`, `ScopeKey`, `HourStartUtc`)** — the UNIQUE index that is **both** the idempotent upsert key and the covering index for the bucketed range read. It is declared with `HasFilter(null)` to **suppress EF's default `[ScopeKey] IS NOT NULL` filtered index**, so `Global`-scope rows (null `ScopeKey`) participate in the uniqueness constraint and the per-`(series, hour)` upsert holds for global rollups too. +- **`IX_KpiRollupHourly_Hour` (`HourStartUtc`)** — the retention purge. + +The table is non-partitioned, on the standard `[PRIMARY]` filegroup, with no DB-role restriction — operational history, not audit (mirrors `KpiSample`). Entity + EF mapping in Commons / Configuration Database; created by migration `20260710153953_AddKpiRollupHourlyTable`. + ## Recorder — `KpiHistoryRecorderActor` The recorder is the Akka.NET cluster singleton **`kpi-history-recorder`** (singleton-manager actor `kpi-history-recorder-singleton`), running on the active central node. It is **not readiness-gated** — the recorder is pure observability and must never gate `/health/ready`, so it is started outside the readiness barrier (unlike the operational singletons). On graceful shutdown it drains via a `CoordinatedShutdown` task for clean singleton handover. @@ -61,7 +88,11 @@ A timer fires every `SampleInterval` (default 60s; an immediate first tick prime **Best-effort, per-source isolation.** Each source call and the write are individually guarded. A throwing source is logged and its samples skipped for that tick; it never aborts the tick, the other sources, or the source component itself. This is the same `IEnumerable<>`-of-adapters decoupling pattern used by `INotificationDeliveryAdapter`. -**Retention.** A daily purge timer (`PurgeInterval`, default 24h) deletes rows older than `RetentionDays` (default 90) via `IKpiHistoryRepository.PurgeOlderThanAsync`, reusing the existing purge-scheduler shape. Hourly/longer-range downsampling is deferred (YAGNI). +**Hourly rollup tick.** A third timer (`kpi-rollup`) fires every `RollupInterval` (default 1h). On each tick the recorder opens a per-tick DI scope and calls `IKpiHistoryRepository.FoldHourlyRollupsAsync` over the window `[TruncateToHour(now) − RollupLookbackHours, TruncateToHour(now))` — an **exclusive** upper bound at the current hour start, so the in-progress hour is never folded. The fold groups raw `KpiSample` rows by `(series, hour)`, applies the per-metric aggregation intent (sum for rate metrics, last-value for gauges — see Query), and **idempotently upserts** each `(Source, Metric, Scope, ScopeKey, HourStartUtc)` row. Re-folding the trailing lookback window (default 3h) rather than only the last hour is the self-heal for a **singleton-failover-missed tick**: because the upsert overwrites the aggregate in place, re-running the same window produces identical values (no double-count), so any complete hour a missed tick skipped is refilled on the next fold. A `_rollupInFlight` guard (mirroring `_sampleInFlight`) skips a tick while a prior fold is still running so two idempotent upserts never race on overlapping rows. + +**One-shot backfill.** So the 30 d / 90 d charts are not blank until enough wall-clock passes, the recorder runs a **single** backfill on start (`kpi-rollup-backfill` timer) that folds the full retention window once, `[TruncateToHour(now) − RetentionDays, TruncateToHour(now))`, reusing the same `FoldHourlyRollupsAsync` path. A `_backfillDone` latch makes it at-most-once per actor lifetime, and it defers (re-arms) while a periodic fold is in flight; being an idempotent upsert, it is cheap and safe to re-run on a later failover restart. + +**Retention.** A daily purge timer (`PurgeInterval`, default 24h) now runs **both** purges: raw `KpiSample` rows older than `RetentionDays` (default 90) via `IKpiHistoryRepository.PurgeOlderThanAsync`, **and** hourly rollup rows older than `RollupRetentionDays` (default 365) via `IKpiHistoryRepository.PurgeRollupsOlderThanAsync` (a one-hour-sliced batched DELETE mirroring the raw purge). Rollups deliberately **outlive** raw samples so long-range trend charts have data after the raw rows are purged; the options validator enforces `RollupRetentionDays >= RetentionDays` so a window never falls into a hole where raw is purged but no rollup exists. ## Sample Sources @@ -117,15 +148,29 @@ Reads `ICentralHealthAggregator.GetAllSiteStates()` (in-memory, no DB). Scope: * ### Bucketed query -`IKpiHistoryRepository.GetRawSeriesAsync(source, metric, scope, scopeKey, fromUtc, toUtc, …)` returns the raw points for one series over `[fromUtc, toUtc]`. `KpiSeriesBucketer.Bucket(raw, fromUtc, toUtc, maxPoints)` then partitions the window into ≤ `maxPoints` time buckets and returns the **last value per bucket** as `KpiSeriesPoint(BucketStartUtc, Value)`. Last-value is correct for gauge metrics; v1 ships exactly one aggregation — avg / min / max are deferred. +`IKpiHistoryRepository.GetRawSeriesAsync(source, metric, scope, scopeKey, fromUtc, toUtc, …)` returns the raw points for one series over `[fromUtc, toUtc]`. `GetHourlySeriesAsync(…)` returns the same ascending `KpiSeriesPoint` shape from `KpiRollupHourly` (projecting `HourStartUtc` → the point timestamp, `Value`), so the bucketer is agnostic to the source. `KpiSeriesBucketer.Bucket(series, fromUtc, toUtc, maxPoints)` then partitions the window into ≤ `maxPoints` time buckets and returns the **last value per bucket** as `KpiSeriesPoint(BucketStartUtc, Value)` — unchanged, and still applied on top of a rollup series (a 90 d rollup is ~2,160 points > the 200-point cap, so bucketing still thins it). The chart plots the single `Value` series; avg / min / max charting is still deferred, though the rollup's `MinValue` / `MaxValue` / `SampleCount` columns are now stored to unblock it. + +### Raw-vs-rollup range routing + +`GetRawSeriesAsync` and `GetHourlySeriesAsync` are selected by window width in the query service (below), governed by `RollupThresholdHours` (default 168 = 7 d). Windows **at or below** the threshold read raw samples (preserving intra-minute detail on the 24 h / 7 d charts); windows **strictly wider** than the threshold read the pre-aggregated hourly rollups (≈60× fewer rows scanned on the 30 d / 90 d charts). The boundary is strict — a window of exactly `RollupThresholdHours` stays on the raw path. + +### Aggregation intent — gauge vs. rate + +Not every metric folds the same way, and choosing wrong silently corrupts long-range totals. `KpiMetricAggregationCatalog.Resolve(source, metric)` classifies each `(Source, Metric)` pair as `KpiRollupAggregation.Gauge` (fold = **last-value-per-hour**) or `KpiRollupAggregation.Rate` (fold = **sum-per-hour**): + +- **Gauge** (last-value) — instantaneous readings where a per-hour sum is meaningless: `queueDepth`, `buffered`, `stuck`/`stuckCount`, `parked*`, `oldestPendingAgeSeconds`, `connectionsUp`/`connectionsDown`, `deployedInstances`, and every other metric not explicitly listed as a rate. This is the **safe default**: an unknown / newly-added metric resolves to Gauge and never crashes the fold or invents a total. +- **Rate** (sum-per-hour) — already-windowed per-interval deltas whose hour total is the sum of its minute samples: `NotificationOutbox/deliveredLastInterval`, `SiteCallAudit/failedLastInterval`, `SiteCallAudit/deliveredLastInterval`, and the per-report `SiteHealth` error counts `scriptErrors` / `alarmEvalErrors` / `deadLetters` / `eventLogWriteFailures`. Folding these as last-value would keep one minute's delta and discard the other 59, under-counting every long-range total. +- **Deliberately Gauge, not Rate** — `AuditLog/totalEventsLastHour` and `errorEventsLastHour` read as rate-like but are computed as a **rolling trailing-one-hour** count re-evaluated every capture; consecutive minutes overlap by 59 minutes, so summing would over-count ~60×. Last-value is correct. + +The catalog is keyed by `(source, metric)` — not metric alone — because `deliveredLastInterval` is emitted by both Notification Outbox and Site Call Audit, so a metric name is not globally unique. ### `KpiHistoryQueryService` (Central UI) -A scoped-repository direct read with a **dual-constructor test seam** (one ctor resolves a scoped `IKpiHistoryRepository` per call; the other accepts an injected repository for tests) — the same shape as `AuditLogQueryService`. `GetSeriesAsync` resolves the effective point cap (caller override or `KpiHistoryOptions.DefaultMaxSeriesPoints`), fetches the raw series, and reduces it via `KpiSeriesBucketer`. A query failure surfaces as an unavailable chart (em-dash / message), mirroring how the existing KPI tiles surface transient failures — it never breaks the hosting page. +A scoped-repository direct read with a **dual-constructor test seam** (one ctor resolves a scoped `IKpiHistoryRepository` per call; the other accepts an injected repository for tests) — the same shape as `AuditLogQueryService`. `GetSeriesAsync` resolves the effective point cap (caller override or `KpiHistoryOptions.DefaultMaxSeriesPoints`), fetches the series — **routing raw vs. hourly-rollup by window width** against `RollupThresholdHours` (see Raw-vs-rollup range routing, above) — and reduces it via `KpiSeriesBucketer`. The callers pass `fromUtc` / `toUtc` unchanged; the long-window buttons transparently exercise the rollup path with no fetch-call change. A query failure surfaces as an unavailable chart (em-dash / message), mirroring how the existing KPI tiles surface transient failures — it never breaks the hosting page. ### `KpiTrendChart.razor` (Central UI) -A reusable **custom inline-SVG** line/area chart — a polyline path with min/max + time-range axis labels, a responsive `viewBox`, and clean corporate styling. There is **no third-party charting library** (per the CLAUDE.md no-third-party-component-framework rule). The time window (e.g. 24h / 7d) is owned by the parent page. +A reusable **custom inline-SVG** line/area chart — a polyline path with min/max + time-range axis labels, a responsive `viewBox`, and clean corporate styling. There is **no third-party charting library** (per the CLAUDE.md no-third-party-component-framework rule). The time window is owned by the parent page; each of the four surfaces offers **24 h / 7 d / 30 d (720 h) / 90 d (2160 h)** toggle buttons — the two long windows are what exercise the rollup read path. ### Surfaces @@ -143,16 +188,20 @@ Bound from the `ScadaBridge:KpiHistory` section on the central host (Options pat | Option | Default | Notes | |---|---|---| | `SampleInterval` | `60s` | Recorder tick cadence. Must be `> 0`. | -| `RetentionDays` | `90` | Rows older than this are purged. Bounded to `[1, 3650]` days. | -| `PurgeInterval` | `1d` | Daily purge cadence. Must be `> 0`. | +| `RetentionDays` | `90` | Raw `KpiSample` rows older than this are purged. Bounded to `[1, 3650]` days. | +| `PurgeInterval` | `1d` | Daily purge cadence (drives **both** raw and rollup purges). Must be `> 0`. | | `DefaultMaxSeriesPoints` | `200` | Default bucket cap for a series query when the caller does not override it. Bounded to `[2, 5000]`. | +| `RollupInterval` | `1h` | Hourly rollup fold cadence (`kpi-rollup` timer). Must be `> 0`. | +| `RollupLookbackHours` | `3` | Trailing whole hours each fold re-processes (idempotent self-heal for a missed tick). Bounded to `[1, 168]`. | +| `RollupRetentionDays` | `365` | `KpiRollupHourly` rows older than this are purged. Bounded to `[1, 3650]` days **and** required `>= RetentionDays`. | +| `RollupThresholdHours` | `168` | Raw-vs-rollup query-routing boundary (7 d); windows strictly wider read rollups. Minimum `24`. | -Validation fails fast at startup on a non-positive `SampleInterval` / `PurgeInterval` (which would stall the recorder / purge), an out-of-range `RetentionDays` (too short loses history; too long defeats retention), or an out-of-range `DefaultMaxSeriesPoints`. +Validation fails fast at startup on a non-positive `SampleInterval` / `PurgeInterval` / `RollupInterval` (which would stall the recorder / purge / fold), an out-of-range `RetentionDays` / `RollupLookbackHours` / `RollupRetentionDays` / `DefaultMaxSeriesPoints`, a `RollupThresholdHours` below 24 h, or a `RollupRetentionDays < RetentionDays` (which would leave long-range charts a hole where raw is purged but no rollup remains). ## Dependencies - **Commons**: defines the `KpiSample` entity, the `IKpiSampleSource` and `IKpiHistoryRepository` interfaces, the `KpiSources` / `KpiScopes` catalogs, and the `KpiSeriesPoint` / `KpiSeriesBucketer` query types. -- **Configuration Database**: hosts the `KpiSample` table, its EF mapping, the migration, and the `KpiHistoryRepository` implementation. +- **Configuration Database**: hosts the `KpiSample` and `KpiRollupHourly` tables, their EF mappings, the migrations, and the `KpiHistoryRepository` implementation (raw + rollup fold / query / purge). - **Cluster Infrastructure**: hosts the `kpi-history-recorder` cluster singleton with active/standby failover. - **Host**: binds `KpiHistoryOptions`, registers the component on the central role, and starts the recorder singleton **outside** the readiness barrier. - **Notification Outbox / Site Call Audit / Audit Log / Health Monitoring**: each contributes an `IKpiSampleSource` and the KPI/aggregator reads it reuses. KPI History depends on the `IKpiSampleSource` abstraction, not on these components directly.