ScadaBridge/docs/requirements/Component-KpiHistory.md

# Component: KPI History

## Purpose

The KPI History component is the central, reusable **KPI-history backbone** — a tall / EAV time-series store, a periodic recorder singleton, a bucketed query API, and a reusable custom SVG trend-chart component. It turns the system's existing point-in-time KPIs into trends, and ships those trends for the **Notification Outbox (#21)**, **Site Call Audit (#22)**, **Audit Log (#23)**, and **Site Health (#11)** sources.

This supersedes the earlier "KPI history — point-in-time only, no separate time-series store is added" stance carried by the Notification Outbox and Site Call Audit KPI sections. M6 explicitly introduces a store. It lives in **central MS SQL** — the existing HA store — so it adds **no new infrastructure dependency**: a raw `KpiSample` table plus an hourly `KpiRollupHourly` rollup table (the latter added for the deferred #22 long-range-query slice), EF mappings + migrations, and a central cluster singleton that samples every minute and folds hourly rollups.

The backbone is deliberately source-agnostic. Each owning component contributes an `IKpiSampleSource` registered into DI; the recorder enumerates them. KPI History therefore does **not** reference every component, and every source reuses the KPI reads its owner already computes — no per-source schema or storage work.

## Location

- `src/ZB.MOM.WW.ScadaBridge.KpiHistory` — the component project: the `KpiHistoryRecorderActor`, `KpiHistoryOptions` + validator, and the DI/options wiring (`ServiceCollectionExtensions`). It owns the recorder, the options, and consumes the `IKpiSampleSource` abstraction (defined in Commons).
- **`IKpiSampleSource` implementations live with their owners**, not here — `NotificationOutboxKpiSampleSource` (in NotificationOutbox), `SiteCallAuditKpiSampleSource` (in SiteCallAudit), `AuditLogKpiSampleSource` (in AuditLog), `SiteHealthKpiSampleSource` (in HealthMonitoring). Each registers itself via `TryAddEnumerable`.
- **Commons** — the `KpiSample` and `KpiRollupHourly` POCO entities (`Entities/Kpi`), the `IKpiSampleSource` and `IKpiHistoryRepository` interfaces (`Interfaces/Kpi`), and the `KpiSources` / `KpiScopes` constant catalogs + `KpiSeriesPoint` / `KpiSeriesBucketer` types and the `KpiRollupAggregation` enum + `KpiMetricAggregationCatalog` (`Types/Kpi`).
- **Configuration Database** — the EF mappings (`KpiSampleEntityTypeConfiguration`, `KpiRollupHourlyEntityTypeConfiguration`), the migrations that create the `KpiSample` and `KpiRollupHourly` tables + indexes, and the `KpiHistoryRepository` implementation (raw record/query/purge plus the hourly fold / rollup query / rollup purge).
- **Central UI** — the `KpiHistoryQueryService` query service and the reusable `KpiTrendChart.razor` component, plus the trend sections embedded on four surfaces.

The recorder is a **singleton on the active central node**, consistent with the other central singletons (Notification Outbox, Site Call Audit, purge actors).

## Responsibilities

- Own the `KpiSample` and `KpiRollupHourly` tables — the central tall / EAV KPI-history store (raw + hourly rollups) in MS SQL.
- Run the recorder loop: every `SampleInterval`, enumerate all registered `IKpiSampleSource`s and persist their samples stamped with one shared tick timestamp.
- Isolate sources from one another and from the store: a failure in any one source (or in the write) is logged and skipped for that tick and never disrupts the source component or the rest of the tick (best-effort observability).
- Fold raw samples into hourly rollups on a `RollupInterval` cadence (with a one-shot backfill on start), applying per-metric gauge-vs-rate aggregation via an idempotent, failover-self-healing upsert.
- Purge aged rows on a daily cadence (`PurgeInterval`): raw rows older than `RetentionDays` and rollup rows older than the longer `RollupRetentionDays`.
- Provide a bucketed series-query API (`IKpiHistoryRepository.GetRawSeriesAsync` / `GetHourlySeriesAsync` + `KpiSeriesBucketer`, routed raw-vs-rollup by window width) and the Central UI query service + reusable trend chart that consume it.

KPI History is **observability, never a user-facing critical path** — neither recording nor querying may ever break a hosting page or disrupt a source component.

## Schema — `KpiSample` (tall / EAV)

A persistence-ignorant POCO in Commons; EF mapping + migration in Configuration Database; one table in central MS SQL. One row per `(Source, Metric, Scope, ScopeKey)` per recorder tick:

| Column | Type | Notes |
|---|---|---|
| `Id` | `bigint` PK identity | Surrogate key assigned by the store. |
| `Source` | `varchar(64)` | Owning source — a `KpiSources` constant: `NotificationOutbox` / `SiteCallAudit` / `AuditLog` / `SiteHealth`. |
| `Metric` | `varchar(64)` | Per-source metric name, e.g. `queueDepth`, `parkedCount`, `deadLetters` — drawn from each source's own metric catalog. |
| `Scope` | `varchar(16)` | A `KpiScopes` constant: `Global` / `Site` / `Node`. |
| `ScopeKey` | `varchar(64)` NULL | Site id (for `Site`) or node name (for `Node`); `NULL` for `Global`. |
| `Value` | `float` (`double`) | Counts carried exactly within range; ages stored as **seconds**. |
| `CapturedAtUtc` | `datetime2` | The recorder tick timestamp (UTC), shared across every sample in one tick. |

All timestamps are UTC, consistent with the system-wide convention.

Two named indexes back the access paths:

- **`IX_KpiSample_Series` (`Source`, `Metric`, `Scope`, `ScopeKey`, `CapturedAtUtc`)** — the per-series range query (one series scanned in time order).
- **`IX_KpiSample_Captured` (`CapturedAtUtc`)** — the retention purge.

## Schema — `KpiRollupHourly` (hourly rollups)

Long-range trend queries (30 d / 90 d) would fetch and sort tens of thousands of raw `KpiSample` rows per series (~43k at 30 d, ~130k at 90 d @ 1/min) before bucketing. The recorder therefore pre-aggregates raw samples into an **hourly rollup** table, `KpiRollupHourly` — a second tall / EAV table in the same central MS SQL database (no new infrastructure), keyed by the **identical four-tuple series key** plus a truncated `HourStartUtc`. A 90 d read becomes a small indexed scan (~2,160 rows/series vs ~130k). One row per `(Source, Metric, Scope, ScopeKey)` per UTC hour:

| Column | Type | Notes |
|---|---|---|
| `Id` | `bigint` PK identity | Surrogate key assigned by the store. |
| `Source` | `varchar(64)` | Same `KpiSources` constant as `KpiSample`. |
| `Metric` | `varchar(64)` | Same per-source metric name as `KpiSample`. |
| `Scope` | `varchar(16)` | A `KpiScopes` constant: `Global` / `Site` / `Node`. |
| `ScopeKey` | `varchar(64)` NULL | Site id / node name; `NULL` for `Global`. |
| `HourStartUtc` | `datetime2` | Hour-truncated UTC bucket start (minutes/seconds/ticks zeroed). |
| `Value` | `float` (`double`) | The folded aggregate — a **sum-per-hour** for rate metrics or a **last-value-per-hour** for gauge metrics (see Query — aggregation intent). |
| `MinValue` | `float` (`double`) | Minimum raw sample value observed within the hour. |
| `MaxValue` | `float` (`double`) | Maximum raw sample value observed within the hour. |
| `SampleCount` | `int` | Number of raw `KpiSample` rows folded into the hour. |

`MinValue` / `MaxValue` / `SampleCount` preserve the fold's fidelity and unblock future min/max/avg charting; the current `KpiTrendChart` still plots the single `Value` series (see Query).

Two named indexes back the access paths, mirroring `KpiSample`:

- **`IX_KpiRollupHourly_Series` (`Source`, `Metric`, `Scope`, `ScopeKey`, `HourStartUtc`)** — the UNIQUE index that is **both** the idempotent upsert key and the covering index for the bucketed range read. It is declared with `HasFilter(null)` to **suppress EF's default `[ScopeKey] IS NOT NULL` filtered index**, so `Global`-scope rows (null `ScopeKey`) participate in the uniqueness constraint and the per-`(series, hour)` upsert holds for global rollups too.
- **`IX_KpiRollupHourly_Hour` (`HourStartUtc`)** — the retention purge.

The table is non-partitioned, on the standard `[PRIMARY]` filegroup, with no DB-role restriction — operational history, not audit (mirrors `KpiSample`). Entity + EF mapping in Commons / Configuration Database; created by migration `20260710153953_AddKpiRollupHourlyTable`.

## Recorder — `KpiHistoryRecorderActor`

The recorder is the Akka.NET cluster singleton **`kpi-history-recorder`** (singleton-manager actor `kpi-history-recorder-singleton`), running on the active central node. It is **not readiness-gated** — the recorder is pure observability and must never gate `/health/ready`, so it is started outside the readiness barrier (unlike the operational singletons). On graceful shutdown it drains via a `CoordinatedShutdown` task for clean singleton handover.

A timer fires every `SampleInterval` (default 60s; an immediate first tick primes the series, then it settles into the periodic cadence). On each tick the recorder:

1. Opens a **per-tick DI scope** (scoped `DbContext`/repository — the same scope-per-sweep pattern as the `NotificationOutboxActor`).
2. Enumerates the registered `IEnumerable<IKpiSampleSource>`. Each source returns an `IReadOnlyList<KpiSample>` stamped with the tick's single `CapturedAtUtc`.
3. Writes all collected samples via `IKpiHistoryRepository.RecordSamplesAsync`.

**Best-effort, per-source isolation.** Each source call and the write are individually guarded. A throwing source is logged and its samples skipped for that tick; it never aborts the tick, the other sources, or the source component itself. This is the same `IEnumerable<>`-of-adapters decoupling pattern used by `INotificationDeliveryAdapter`.

**Hourly rollup tick.** A third timer (`kpi-rollup`) fires every `RollupInterval` (default 1h). On each tick the recorder opens a per-tick DI scope and calls `IKpiHistoryRepository.FoldHourlyRollupsAsync` over the window `[TruncateToHour(now) − RollupLookbackHours, TruncateToHour(now))` — an **exclusive** upper bound at the current hour start, so the in-progress hour is never folded. The fold groups raw `KpiSample` rows by `(series, hour)`, applies the per-metric aggregation intent (sum for rate metrics, last-value for gauges — see Query), and **idempotently upserts** each `(Source, Metric, Scope, ScopeKey, HourStartUtc)` row. Re-folding the trailing lookback window (default 3h) rather than only the last hour is the self-heal for a **singleton-failover-missed tick**: because the upsert overwrites the aggregate in place, re-running the same window produces identical values (no double-count), so any complete hour a missed tick skipped is refilled on the next fold. A `_rollupInFlight` guard (mirroring `_sampleInFlight`) skips a tick while a prior fold is still running so two idempotent upserts never race on overlapping rows. During a **failover overlap** — the old singleton incarnation's in-flight fold and the new node's first fold both `Add`-ing the same `(series, hour)` row — the loser of the unique `IX_KpiRollupHourly_Series` upsert race discards its **whole fold pass** (a single `SaveChanges`), logged at Information and repaired by the next idempotent fold; when reading fold logs after a failover, the failure grain is the **pass, not the row** (arch-review 04 round 2, R5).

**One-shot backfill.** So the 30 d / 90 d charts are not blank until enough wall-clock passes, the recorder runs a backfill on start (`kpi-rollup-backfill` timer) that folds the retention window, reusing the same `FoldHourlyRollupsAsync` path. The `FoldHourlyRollupsAsync` repository call materializes its whole window in memory, so the backfill folds the window in **≤24 h slices, oldest-first** — one bounded slice per fold — rather than handing the fold the full 90-day window at once (which at fleet volume is tens of millions of rows in one tracked pass). Between slices it lowers its in-flight guard so periodic folds **interleave** instead of stalling behind the historical pass; a periodic fold and a backfill slice strictly alternate through the single `_rollupInFlight`/`_backfillInFlight` gate pair, so the two idempotent upserts never race on overlapping rows. On a **failover restart** the plan first consults the `IKpiHistoryRepository.GetLatestRollupHourAsync` watermark: if the newest folded hour is already within `RollupLookbackHours` of now (the common case) the whole historical pass is **skipped** — the periodic fold's lookback covers the tail — otherwise the window is **shrunk to the un-rolled tail** `[watermark, TruncateToHour(now))` (re-folding the watermark hour itself as an idempotent safety margin) instead of the full `RetentionDays` floor. A `_backfillDone` latch makes the whole pass at-most-once per actor lifetime, and it defers (re-arms) while a periodic fold is in flight. Being an idempotent upsert, it is safe (idempotent) to re-run on a later failover restart; the watermark fast-path makes the failover re-run cheap as well.

**Retention.** A daily purge timer (`PurgeInterval`, default 24h) now runs **both** purges: raw `KpiSample` rows older than `RetentionDays` (default 90) via `IKpiHistoryRepository.PurgeOlderThanAsync`, **and** hourly rollup rows older than `RollupRetentionDays` (default 365) via `IKpiHistoryRepository.PurgeRollupsOlderThanAsync` (a one-hour-sliced batched DELETE mirroring the raw purge). Rollups deliberately **outlive** raw samples so long-range trend charts have data after the raw rows are purged; the options validator enforces `RollupRetentionDays >= RetentionDays` so a window never falls into a hole where raw is purged but no rollup exists.

## Sample Sources

Each `IKpiSampleSource` lives in its owning component and is registered into DI with `TryAddEnumerable` (idempotent, additive). Each reuses the KPI reads its owner already performs — the Notification Outbox / Site Call Audit / Audit Log sources call their owners' existing `Compute…KpisAsync` aggregator reads; the Site Health source reads the in-memory `ICentralHealthAggregator` (no DB read). `Value` carries counts exactly and ages as seconds; all metric names below are the exact shipped strings.

### `NotificationOutboxKpiSampleSource` (in NotificationOutbox)

Scopes: **Global + per-Site + per-Node** (the per-node breakdown reuses the M5 `ComputePerNodeKpisAsync`).

- `queueDepth`
- `stuckCount`
- `parkedCount`
- `deliveredLastInterval`
- `oldestPendingAgeSeconds`

### `SiteCallAuditKpiSampleSource` (in SiteCallAudit)

Scopes: **Global + per-Site + per-Node**.

- `buffered`
- `parked`
- `failedLastInterval`
- `deliveredLastInterval`
- `stuck`
- `oldestPendingAgeSeconds`

### `AuditLogKpiSampleSource` (in AuditLog)

Scope: **Global**.

- `totalEventsLastHour`
- `errorEventsLastHour`
- `backlogTotal`

### `SiteHealthKpiSampleSource` (in HealthMonitoring)

Reads `ICentralHealthAggregator.GetAllSiteStates()` (in-memory, no DB). Scope: **per-Site** — the largest latent win, since Site Health was previously sequence-numbered every 30s but its history discarded.

- `connectionsUp`
- `connectionsDown`
- `scriptErrors`
- `alarmEvalErrors`
- `sfBufferDepth`
- `deadLetters`
- `parkedMessages`
- `deployedInstances`
- `enabledInstances`
- `disabledInstances`
- `auditBacklogPending`
- `eventLogWriteFailures`

## Query + UI

### Bucketed query

`IKpiHistoryRepository.GetRawSeriesAsync(source, metric, scope, scopeKey, fromUtc, toUtc, …)` returns the raw points for one series over `[fromUtc, toUtc]`. `GetHourlySeriesAsync(…)` returns the same ascending `KpiSeriesPoint` shape from `KpiRollupHourly` (projecting `HourStartUtc` → the point timestamp, `Value`), so the bucketer is agnostic to the source. `KpiSeriesBucketer.Bucket(series, fromUtc, toUtc, maxPoints, aggregation)` then partitions the window into ≤ `maxPoints` time buckets and reduces each bucket **per-metric**: it keeps the **last value** for a **Gauge** series and **sums per bucket** for a **Rate** series, driven by the same `KpiMetricAggregationCatalog` the hourly fold consults. The previous unconditional last-value-per-bucket kept **1 of ~11 hourly sums** per 90 d bucket on the rollup path — the exact fold error the catalog's own doc warns about, re-introduced one layer up — and is fixed per arch-review 04 round 2, R3. The reduction is still applied on top of a rollup series (a 90 d rollup is ~2,160 points > the 200-point cap, so bucketing still thins it). The chart plots the single `Value` series; avg / min / max charting is still deferred, though the rollup's `MinValue` / `MaxValue` / `SampleCount` columns are now stored to unblock it.

### Raw-vs-rollup range routing

`GetRawSeriesAsync` and `GetHourlySeriesAsync` are selected by window width in the query service (below), governed by `RollupThresholdHours` (default 168 = 7 d). Windows **at or below** the threshold read raw samples (preserving intra-minute detail on the 24 h / 7 d charts); windows **strictly wider** than the threshold read the pre-aggregated hourly rollups (≈60× fewer rows scanned on the 30 d / 90 d charts). The boundary is strict — a window of exactly `RollupThresholdHours` stays on the raw path.

Rate charts keep **one unit (events per chart bucket)** on both sides of the boundary: the catalog-driven per-bucket sum reduction is applied to the raw-minute deltas and the hourly rollup sums alike, so widening 7 d → 30 d no longer changes a Rate chart's magnitude ~60×. Residual: an unbucketed just-over-threshold rollup series plots native per-hour sums vs. the at-threshold raw path's ~50-minute bucket sums (≤~1.2×) — accepted and documented, not hidden.

### Aggregation intent — gauge vs. rate

Not every metric folds the same way, and choosing wrong silently corrupts long-range totals. `KpiMetricAggregationCatalog.Resolve(source, metric)` classifies each `(Source, Metric)` pair as `KpiRollupAggregation.Gauge` (fold = **last-value-per-hour**) or `KpiRollupAggregation.Rate` (fold = **sum-per-hour**). The catalog now has **two consumers**: the hourly rollup fold (which picks last-value vs. sum per hour) and the chart-time bucket reduction in `KpiSeriesBucketer` (which picks last-value vs. sum per chart bucket) — one classification keeps both layers consistent:

- **Gauge** (last-value) — instantaneous readings where a per-hour sum is meaningless: `queueDepth`, `buffered`, `stuck`/`stuckCount`, `parked*`, `oldestPendingAgeSeconds`, `connectionsUp`/`connectionsDown`, `deployedInstances`, and every other metric not explicitly listed as a rate. This is the **safe default**: an unknown / newly-added metric resolves to Gauge and never crashes the fold or invents a total.
- **Rate** (sum-per-hour) — already-windowed per-interval deltas whose hour total is the sum of its minute samples: `NotificationOutbox/deliveredLastInterval`, `SiteCallAudit/failedLastInterval`, `SiteCallAudit/deliveredLastInterval`, and the per-report `SiteHealth` error counts `scriptErrors` / `alarmEvalErrors` / `deadLetters` / `eventLogWriteFailures`. Folding these as last-value would keep one minute's delta and discard the other 59, under-counting every long-range total.
- **Deliberately Gauge, not Rate** — `AuditLog/totalEventsLastHour` and `errorEventsLastHour` read as rate-like but are computed as a **rolling trailing-one-hour** count re-evaluated every capture; consecutive minutes overlap by 59 minutes, so summing would over-count ~60×. Last-value is correct.

The catalog is keyed by `(source, metric)` — not metric alone — because `deliveredLastInterval` is emitted by both Notification Outbox and Site Call Audit, so a metric name is not globally unique.

### `KpiHistoryQueryService` (Central UI)

A scoped-repository direct read with a **dual-constructor test seam** (one ctor resolves a scoped `IKpiHistoryRepository` per call; the other accepts an injected repository for tests) — the same shape as `AuditLogQueryService`. `GetSeriesAsync` resolves the effective point cap (caller override or `KpiHistoryOptions.DefaultMaxSeriesPoints`), fetches the series — **routing raw vs. hourly-rollup by window width** against `RollupThresholdHours` (see Raw-vs-rollup range routing, above) — and reduces it via `KpiSeriesBucketer`. The callers pass `fromUtc` / `toUtc` unchanged; the long-window buttons transparently exercise the rollup path with no fetch-call change. A query failure surfaces as an unavailable chart (em-dash / message), mirroring how the existing KPI tiles surface transient failures — it never breaks the hosting page.

### `KpiTrendChart.razor` (Central UI)

A reusable **custom inline-SVG** line/area chart — a polyline path with min/max + time-range axis labels, a responsive `viewBox`, and clean corporate styling. There is **no third-party charting library** (per the CLAUDE.md no-third-party-component-framework rule). The time window is owned by the parent page; each of the four surfaces offers **24 h / 7 d / 30 d (720 h) / 90 d (2160 h)** toggle buttons — the two long windows are what exercise the rollup read path.

### Surfaces

Trend sections render on four pages, each feeding `KpiTrendChart` from `KpiHistoryQueryService`:

- **Notification Outbox** page — outbox KPI trends.
- **Site Calls** page — cached-call KPI trends.
- **Audit Log** page — audit volume / error / backlog trends.
- **Health dashboard** — a per-site Site Health trend panel.

## Configuration — `KpiHistoryOptions`

Bound from the `ScadaBridge:KpiHistory` section on the central host (Options pattern), validated on startup by `KpiHistoryOptionsValidator`:

| Option | Default | Notes |
|---|---|---|
| `SampleInterval` | `60s` | Recorder tick cadence. Must be `> 0`. |
| `RetentionDays` | `90` | Raw `KpiSample` rows older than this are purged. Bounded to `[1, 3650]` days. |
| `PurgeInterval` | `1d` | Daily purge cadence (drives **both** raw and rollup purges). Must be `> 0`. |
| `DefaultMaxSeriesPoints` | `200` | Default bucket cap for a series query when the caller does not override it. Bounded to `[2, 5000]`. |
| `RollupInterval` | `1h` | Hourly rollup fold cadence (`kpi-rollup` timer). Must be `> 0`. |
| `RollupLookbackHours` | `3` | Trailing whole hours each fold re-processes (idempotent self-heal for a missed tick). Bounded to `[1, 168]`. |
| `RollupRetentionDays` | `365` | `KpiRollupHourly` rows older than this are purged. Bounded to `[1, 3650]` days **and** required `>= RetentionDays`. |
| `RollupThresholdHours` | `168` | Raw-vs-rollup query-routing boundary (7 d); windows strictly wider read rollups. Minimum `24`. |

Validation fails fast at startup on a non-positive `SampleInterval` / `PurgeInterval` / `RollupInterval` (which would stall the recorder / purge / fold), an out-of-range `RetentionDays` / `RollupLookbackHours` / `RollupRetentionDays` / `DefaultMaxSeriesPoints`, a `RollupThresholdHours` below 24 h, or a `RollupRetentionDays < RetentionDays` (which would leave long-range charts a hole where raw is purged but no rollup remains).

## Dependencies

- **Commons**: defines the `KpiSample` entity, the `IKpiSampleSource` and `IKpiHistoryRepository` interfaces, the `KpiSources` / `KpiScopes` catalogs, and the `KpiSeriesPoint` / `KpiSeriesBucketer` query types.
- **Configuration Database**: hosts the `KpiSample` and `KpiRollupHourly` tables, their EF mappings, the migrations, and the `KpiHistoryRepository` implementation (raw + rollup fold / query / purge).
- **Cluster Infrastructure**: hosts the `kpi-history-recorder` cluster singleton with active/standby failover.
- **Host**: binds `KpiHistoryOptions`, registers the component on the central role, and starts the recorder singleton **outside** the readiness barrier.
- **Notification Outbox / Site Call Audit / Audit Log / Health Monitoring**: each contributes an `IKpiSampleSource` and the KPI/aggregator reads it reuses. KPI History depends on the `IKpiSampleSource` abstraction, not on these components directly.
- **Central UI**: hosts `KpiHistoryQueryService` and the `KpiTrendChart` component.

## Interactions

- **Notification Outbox (#21)**: registers `NotificationOutboxKpiSampleSource` (Global / Site / Node), sampled each recorder tick; its trends render on the Notification Outbox page.
- **Site Call Audit (#22)**: registers `SiteCallAuditKpiSampleSource` (Global / Site / Node); its trends render on the Site Calls page.
- **Audit Log (#23)**: registers `AuditLogKpiSampleSource` (Global); its trends render on the Audit Log page.
- **Health Monitoring (#11)**: registers `SiteHealthKpiSampleSource` (per-Site), reading the in-memory central health aggregator; its trends render in the Health dashboard's per-site panel.
- **Central UI (#9)**: renders the reusable `KpiTrendChart` fed by `KpiHistoryQueryService` across the four trend surfaces; a query failure degrades to an unavailable-chart placeholder rather than breaking the page.
- **Cluster Infrastructure (#13)**: provides the active/standby singleton hosting for the recorder, which drains on `CoordinatedShutdown` for clean handover.