ScadaBridge/docs/requirements/Component-KpiHistory.md

Component: KPI History

Purpose

The KPI History component is the central, reusable KPI-history backbone — a tall / EAV time-series store, a periodic recorder singleton, a bucketed query API, and a reusable custom SVG trend-chart component. It turns the system's existing point-in-time KPIs into trends, and ships those trends for the Notification Outbox (#21), Site Call Audit (#22), Audit Log (#23), and Site Health (#11) sources.

This supersedes the earlier "KPI history — point-in-time only, no separate time-series store is added" stance carried by the Notification Outbox and Site Call Audit KPI sections. M6 explicitly introduces a store. It lives in central MS SQL — the existing HA store — so it adds no new infrastructure dependency: a raw KpiSample table plus an hourly KpiRollupHourly rollup table (the latter added for the deferred #22 long-range-query slice), EF mappings + migrations, and a central cluster singleton that samples every minute and folds hourly rollups.

The backbone is deliberately source-agnostic. Each owning component contributes an IKpiSampleSource registered into DI; the recorder enumerates them. KPI History therefore does not reference every component, and every source reuses the KPI reads its owner already computes — no per-source schema or storage work.

Location

• src/ZB.MOM.WW.ScadaBridge.KpiHistory — the component project: the KpiHistoryRecorderActor, KpiHistoryOptions + validator, and the DI/options wiring (ServiceCollectionExtensions). It owns the recorder, the options, and consumes the IKpiSampleSource abstraction (defined in Commons).
• IKpiSampleSource implementations live with their owners, not here — NotificationOutboxKpiSampleSource (in NotificationOutbox), SiteCallAuditKpiSampleSource (in SiteCallAudit), AuditLogKpiSampleSource (in AuditLog), SiteHealthKpiSampleSource (in HealthMonitoring). Each registers itself via TryAddEnumerable.
• Commons — the KpiSample and KpiRollupHourly POCO entities (Entities/Kpi), the IKpiSampleSource and IKpiHistoryRepository interfaces (Interfaces/Kpi), and the KpiSources / KpiScopes constant catalogs + KpiSeriesPoint / KpiSeriesBucketer types and the KpiRollupAggregation enum + KpiMetricAggregationCatalog (Types/Kpi).
• Configuration Database — the EF mappings (KpiSampleEntityTypeConfiguration, KpiRollupHourlyEntityTypeConfiguration), the migrations that create the KpiSample and KpiRollupHourly tables + indexes, and the KpiHistoryRepository implementation (raw record/query/purge plus the hourly fold / rollup query / rollup purge).
• Central UI — the KpiHistoryQueryService query service and the reusable KpiTrendChart.razor component, plus the trend sections embedded on four surfaces.

The recorder is a singleton on the active central node, consistent with the other central singletons (Notification Outbox, Site Call Audit, purge actors).

Responsibilities

• Own the KpiSample and KpiRollupHourly tables — the central tall / EAV KPI-history store (raw + hourly rollups) in MS SQL.
• Run the recorder loop: every SampleInterval, enumerate all registered IKpiSampleSources and persist their samples stamped with one shared tick timestamp.
• Isolate sources from one another and from the store: a failure in any one source (or in the write) is logged and skipped for that tick and never disrupts the source component or the rest of the tick (best-effort observability).
• Fold raw samples into hourly rollups on a RollupInterval cadence (with a one-shot backfill on start), applying per-metric gauge-vs-rate aggregation via an idempotent, failover-self-healing upsert.
• Purge aged rows on a daily cadence (PurgeInterval): raw rows older than RetentionDays and rollup rows older than the longer RollupRetentionDays.
• Provide a bucketed series-query API (IKpiHistoryRepository.GetRawSeriesAsync / GetHourlySeriesAsync + KpiSeriesBucketer, routed raw-vs-rollup by window width) and the Central UI query service + reusable trend chart that consume it.

KPI History is observability, never a user-facing critical path — neither recording nor querying may ever break a hosting page or disrupt a source component.

Schema — KpiSample (tall / EAV)

A persistence-ignorant POCO in Commons; EF mapping + migration in Configuration Database; one table in central MS SQL. One row per (Source, Metric, Scope, ScopeKey) per recorder tick:

Column	Type	Notes
Id	bigint PK identity	Surrogate key assigned by the store.
Source	varchar(64)	Owning source — a KpiSources constant: NotificationOutbox / SiteCallAudit / AuditLog / SiteHealth.
Metric	varchar(64)	Per-source metric name, e.g. queueDepth, parkedCount, deadLetters — drawn from each source's own metric catalog.
Scope	varchar(16)	A KpiScopes constant: Global / Site / Node.
ScopeKey	varchar(64) NULL	Site id (for Site) or node name (for Node); NULL for Global.
Value	float (double)	Counts carried exactly within range; ages stored as seconds.
CapturedAtUtc	datetime2	The recorder tick timestamp (UTC), shared across every sample in one tick.

All timestamps are UTC, consistent with the system-wide convention.

Two named indexes back the access paths:

• IX_KpiSample_Series (Source, Metric, Scope, ScopeKey, CapturedAtUtc) — the per-series range query (one series scanned in time order).
• IX_KpiSample_Captured (CapturedAtUtc) — the retention purge.

Schema — KpiRollupHourly (hourly rollups)

Long-range trend queries (30 d / 90 d) would fetch and sort tens of thousands of raw KpiSample rows per series (~43k at 30 d, ~130k at 90 d @ 1/min) before bucketing. The recorder therefore pre-aggregates raw samples into an hourly rollup table, KpiRollupHourly — a second tall / EAV table in the same central MS SQL database (no new infrastructure), keyed by the identical four-tuple series key plus a truncated HourStartUtc. A 90 d read becomes a small indexed scan (~2,160 rows/series vs ~130k). One row per (Source, Metric, Scope, ScopeKey) per UTC hour:

Column	Type	Notes
Id	bigint PK identity	Surrogate key assigned by the store.
Source	varchar(64)	Same KpiSources constant as KpiSample.
Metric	varchar(64)	Same per-source metric name as KpiSample.
Scope	varchar(16)	A KpiScopes constant: Global / Site / Node.
ScopeKey	varchar(64) NULL	Site id / node name; NULL for Global.
HourStartUtc	datetime2	Hour-truncated UTC bucket start (minutes/seconds/ticks zeroed).
Value	float (double)	The folded aggregate — a sum-per-hour for rate metrics or a last-value-per-hour for gauge metrics (see Query — aggregation intent).
MinValue	float (double)	Minimum raw sample value observed within the hour.
MaxValue	float (double)	Maximum raw sample value observed within the hour.
SampleCount	int	Number of raw KpiSample rows folded into the hour.

MinValue / MaxValue / SampleCount preserve the fold's fidelity and unblock future min/max/avg charting; the current KpiTrendChart still plots the single Value series (see Query).

Two named indexes back the access paths, mirroring KpiSample:

• IX_KpiRollupHourly_Series (Source, Metric, Scope, ScopeKey, HourStartUtc) — the UNIQUE index that is both the idempotent upsert key and the covering index for the bucketed range read. It is declared with HasFilter(null) to suppress EF's default [ScopeKey] IS NOT NULL filtered index, so Global-scope rows (null ScopeKey) participate in the uniqueness constraint and the per-(series, hour) upsert holds for global rollups too.
• IX_KpiRollupHourly_Hour (HourStartUtc) — the retention purge.

The table is non-partitioned, on the standard [PRIMARY] filegroup, with no DB-role restriction — operational history, not audit (mirrors KpiSample). Entity + EF mapping in Commons / Configuration Database; created by migration 20260710153953_AddKpiRollupHourlyTable.

Recorder — KpiHistoryRecorderActor

The recorder is the Akka.NET cluster singleton kpi-history-recorder (singleton-manager actor kpi-history-recorder-singleton), running on the active central node. It is not readiness-gated — the recorder is pure observability and must never gate /health/ready, so it is started outside the readiness barrier (unlike the operational singletons). On graceful shutdown it drains via a CoordinatedShutdown task for clean singleton handover.

A timer fires every SampleInterval (default 60s; an immediate first tick primes the series, then it settles into the periodic cadence). On each tick the recorder:

1. Opens a per-tick DI scope (scoped DbContext/repository — the same scope-per-sweep pattern as the NotificationOutboxActor).
2. Enumerates the registered IEnumerable<IKpiSampleSource>. Each source returns an IReadOnlyList<KpiSample> stamped with the tick's single CapturedAtUtc.
3. Writes all collected samples via IKpiHistoryRepository.RecordSamplesAsync.

Best-effort, per-source isolation. Each source call and the write are individually guarded. A throwing source is logged and its samples skipped for that tick; it never aborts the tick, the other sources, or the source component itself. This is the same IEnumerable<>-of-adapters decoupling pattern used by INotificationDeliveryAdapter.

Hourly rollup tick. A third timer (kpi-rollup) fires every RollupInterval (default 1h). On each tick the recorder opens a per-tick DI scope and calls IKpiHistoryRepository.FoldHourlyRollupsAsync over the window [TruncateToHour(now) − RollupLookbackHours, TruncateToHour(now)) — an exclusive upper bound at the current hour start, so the in-progress hour is never folded. The fold groups raw KpiSample rows by (series, hour), applies the per-metric aggregation intent (sum for rate metrics, last-value for gauges — see Query), and idempotently upserts each (Source, Metric, Scope, ScopeKey, HourStartUtc) row. Re-folding the trailing lookback window (default 3h) rather than only the last hour is the self-heal for a singleton-failover-missed tick: because the upsert overwrites the aggregate in place, re-running the same window produces identical values (no double-count), so any complete hour a missed tick skipped is refilled on the next fold. A _rollupInFlight guard (mirroring _sampleInFlight) skips a tick while a prior fold is still running so two idempotent upserts never race on overlapping rows.

One-shot backfill. So the 30 d / 90 d charts are not blank until enough wall-clock passes, the recorder runs a backfill on start (kpi-rollup-backfill timer) that folds the retention window, reusing the same FoldHourlyRollupsAsync path. The FoldHourlyRollupsAsync repository call materializes its whole window in memory, so the backfill folds the window in ≤24 h slices, oldest-first — one bounded slice per fold — rather than handing the fold the full 90-day window at once (which at fleet volume is tens of millions of rows in one tracked pass). Between slices it lowers its in-flight guard so periodic folds interleave instead of stalling behind the historical pass; a periodic fold and a backfill slice strictly alternate through the single _rollupInFlight/_backfillInFlight gate pair, so the two idempotent upserts never race on overlapping rows. On a failover restart the plan first consults the IKpiHistoryRepository.GetLatestRollupHourAsync watermark: if the newest folded hour is already within RollupLookbackHours of now (the common case) the whole historical pass is skipped — the periodic fold's lookback covers the tail — otherwise the window is shrunk to the un-rolled tail [watermark, TruncateToHour(now)) (re-folding the watermark hour itself as an idempotent safety margin) instead of the full RetentionDays floor. A _backfillDone latch makes the whole pass at-most-once per actor lifetime, and it defers (re-arms) while a periodic fold is in flight. Being an idempotent upsert, it is safe (idempotent) to re-run on a later failover restart; the watermark fast-path makes the failover re-run cheap as well.

Retention. A daily purge timer (PurgeInterval, default 24h) now runs both purges: raw KpiSample rows older than RetentionDays (default 90) via IKpiHistoryRepository.PurgeOlderThanAsync, and hourly rollup rows older than RollupRetentionDays (default 365) via IKpiHistoryRepository.PurgeRollupsOlderThanAsync (a one-hour-sliced batched DELETE mirroring the raw purge). Rollups deliberately outlive raw samples so long-range trend charts have data after the raw rows are purged; the options validator enforces RollupRetentionDays >= RetentionDays so a window never falls into a hole where raw is purged but no rollup exists.

Sample Sources

Each IKpiSampleSource lives in its owning component and is registered into DI with TryAddEnumerable (idempotent, additive). Each reuses the KPI reads its owner already performs — the Notification Outbox / Site Call Audit / Audit Log sources call their owners' existing Compute…KpisAsync aggregator reads; the Site Health source reads the in-memory ICentralHealthAggregator (no DB read). Value carries counts exactly and ages as seconds; all metric names below are the exact shipped strings.

NotificationOutboxKpiSampleSource (in NotificationOutbox)

Scopes: Global + per-Site + per-Node (the per-node breakdown reuses the M5 ComputePerNodeKpisAsync).

• queueDepth
• stuckCount
• parkedCount
• deliveredLastInterval
• oldestPendingAgeSeconds

SiteCallAuditKpiSampleSource (in SiteCallAudit)

Scopes: Global + per-Site + per-Node.

• buffered
• parked
• failedLastInterval
• deliveredLastInterval
• stuck
• oldestPendingAgeSeconds

AuditLogKpiSampleSource (in AuditLog)

Scope: Global.

• totalEventsLastHour
• errorEventsLastHour
• backlogTotal

SiteHealthKpiSampleSource (in HealthMonitoring)

Reads ICentralHealthAggregator.GetAllSiteStates() (in-memory, no DB). Scope: per-Site — the largest latent win, since Site Health was previously sequence-numbered every 30s but its history discarded.

• connectionsUp
• connectionsDown
• scriptErrors
• alarmEvalErrors
• sfBufferDepth
• deadLetters
• parkedMessages
• deployedInstances
• enabledInstances
• disabledInstances
• auditBacklogPending
• eventLogWriteFailures

Query + UI

Bucketed query

IKpiHistoryRepository.GetRawSeriesAsync(source, metric, scope, scopeKey, fromUtc, toUtc, …) returns the raw points for one series over [fromUtc, toUtc]. GetHourlySeriesAsync(…) returns the same ascending KpiSeriesPoint shape from KpiRollupHourly (projecting HourStartUtc → the point timestamp, Value), so the bucketer is agnostic to the source. KpiSeriesBucketer.Bucket(series, fromUtc, toUtc, maxPoints) then partitions the window into ≤ maxPoints time buckets and returns the last value per bucket as KpiSeriesPoint(BucketStartUtc, Value) — unchanged, and still applied on top of a rollup series (a 90 d rollup is ~2,160 points > the 200-point cap, so bucketing still thins it). The chart plots the single Value series; avg / min / max charting is still deferred, though the rollup's MinValue / MaxValue / SampleCount columns are now stored to unblock it.

Raw-vs-rollup range routing

GetRawSeriesAsync and GetHourlySeriesAsync are selected by window width in the query service (below), governed by RollupThresholdHours (default 168 = 7 d). Windows at or below the threshold read raw samples (preserving intra-minute detail on the 24 h / 7 d charts); windows strictly wider than the threshold read the pre-aggregated hourly rollups (≈60× fewer rows scanned on the 30 d / 90 d charts). The boundary is strict — a window of exactly RollupThresholdHours stays on the raw path.

Aggregation intent — gauge vs. rate

Not every metric folds the same way, and choosing wrong silently corrupts long-range totals. KpiMetricAggregationCatalog.Resolve(source, metric) classifies each (Source, Metric) pair as KpiRollupAggregation.Gauge (fold = last-value-per-hour) or KpiRollupAggregation.Rate (fold = sum-per-hour):

• Gauge (last-value) — instantaneous readings where a per-hour sum is meaningless: queueDepth, buffered, stuck/stuckCount, parked*, oldestPendingAgeSeconds, connectionsUp/connectionsDown, deployedInstances, and every other metric not explicitly listed as a rate. This is the safe default: an unknown / newly-added metric resolves to Gauge and never crashes the fold or invents a total.
• Rate (sum-per-hour) — already-windowed per-interval deltas whose hour total is the sum of its minute samples: NotificationOutbox/deliveredLastInterval, SiteCallAudit/failedLastInterval, SiteCallAudit/deliveredLastInterval, and the per-report SiteHealth error counts scriptErrors / alarmEvalErrors / deadLetters / eventLogWriteFailures. Folding these as last-value would keep one minute's delta and discard the other 59, under-counting every long-range total.
• Deliberately Gauge, not Rate — AuditLog/totalEventsLastHour and errorEventsLastHour read as rate-like but are computed as a rolling trailing-one-hour count re-evaluated every capture; consecutive minutes overlap by 59 minutes, so summing would over-count ~60×. Last-value is correct.

The catalog is keyed by (source, metric) — not metric alone — because deliveredLastInterval is emitted by both Notification Outbox and Site Call Audit, so a metric name is not globally unique.

KpiHistoryQueryService (Central UI)

A scoped-repository direct read with a dual-constructor test seam (one ctor resolves a scoped IKpiHistoryRepository per call; the other accepts an injected repository for tests) — the same shape as AuditLogQueryService. GetSeriesAsync resolves the effective point cap (caller override or KpiHistoryOptions.DefaultMaxSeriesPoints), fetches the series — routing raw vs. hourly-rollup by window width against RollupThresholdHours (see Raw-vs-rollup range routing, above) — and reduces it via KpiSeriesBucketer. The callers pass fromUtc / toUtc unchanged; the long-window buttons transparently exercise the rollup path with no fetch-call change. A query failure surfaces as an unavailable chart (em-dash / message), mirroring how the existing KPI tiles surface transient failures — it never breaks the hosting page.

KpiTrendChart.razor (Central UI)

A reusable custom inline-SVG line/area chart — a polyline path with min/max + time-range axis labels, a responsive viewBox, and clean corporate styling. There is no third-party charting library (per the CLAUDE.md no-third-party-component-framework rule). The time window is owned by the parent page; each of the four surfaces offers 24 h / 7 d / 30 d (720 h) / 90 d (2160 h) toggle buttons — the two long windows are what exercise the rollup read path.

Surfaces

Trend sections render on four pages, each feeding KpiTrendChart from KpiHistoryQueryService:

• Notification Outbox page — outbox KPI trends.
• Site Calls page — cached-call KPI trends.
• Audit Log page — audit volume / error / backlog trends.
• Health dashboard — a per-site Site Health trend panel.

Configuration — KpiHistoryOptions

Bound from the ScadaBridge:KpiHistory section on the central host (Options pattern), validated on startup by KpiHistoryOptionsValidator:

Option	Default	Notes
SampleInterval	60s	Recorder tick cadence. Must be > 0.
RetentionDays	90	Raw KpiSample rows older than this are purged. Bounded to [1, 3650] days.
PurgeInterval	1d	Daily purge cadence (drives both raw and rollup purges). Must be > 0.
DefaultMaxSeriesPoints	200	Default bucket cap for a series query when the caller does not override it. Bounded to [2, 5000].
RollupInterval	1h	Hourly rollup fold cadence (kpi-rollup timer). Must be > 0.
RollupLookbackHours	3	Trailing whole hours each fold re-processes (idempotent self-heal for a missed tick). Bounded to [1, 168].
RollupRetentionDays	365	KpiRollupHourly rows older than this are purged. Bounded to [1, 3650] days and required >= RetentionDays.
RollupThresholdHours	168	Raw-vs-rollup query-routing boundary (7 d); windows strictly wider read rollups. Minimum 24.

Validation fails fast at startup on a non-positive SampleInterval / PurgeInterval / RollupInterval (which would stall the recorder / purge / fold), an out-of-range RetentionDays / RollupLookbackHours / RollupRetentionDays / DefaultMaxSeriesPoints, a RollupThresholdHours below 24 h, or a RollupRetentionDays < RetentionDays (which would leave long-range charts a hole where raw is purged but no rollup remains).

Dependencies

• Commons: defines the KpiSample entity, the IKpiSampleSource and IKpiHistoryRepository interfaces, the KpiSources / KpiScopes catalogs, and the KpiSeriesPoint / KpiSeriesBucketer query types.
• Configuration Database: hosts the KpiSample and KpiRollupHourly tables, their EF mappings, the migrations, and the KpiHistoryRepository implementation (raw + rollup fold / query / purge).
• Cluster Infrastructure: hosts the kpi-history-recorder cluster singleton with active/standby failover.
• Host: binds KpiHistoryOptions, registers the component on the central role, and starts the recorder singleton outside the readiness barrier.
• Notification Outbox / Site Call Audit / Audit Log / Health Monitoring: each contributes an IKpiSampleSource and the KPI/aggregator reads it reuses. KPI History depends on the IKpiSampleSource abstraction, not on these components directly.
• Central UI: hosts KpiHistoryQueryService and the KpiTrendChart component.

Interactions

• Notification Outbox (#21): registers NotificationOutboxKpiSampleSource (Global / Site / Node), sampled each recorder tick; its trends render on the Notification Outbox page.
• Site Call Audit (#22): registers SiteCallAuditKpiSampleSource (Global / Site / Node); its trends render on the Site Calls page.
• Audit Log (#23): registers AuditLogKpiSampleSource (Global); its trends render on the Audit Log page.
• Health Monitoring (#11): registers SiteHealthKpiSampleSource (per-Site), reading the in-memory central health aggregator; its trends render in the Health dashboard's per-site panel.
• Central UI (#9): renders the reusable KpiTrendChart fed by KpiHistoryQueryService across the four trend surfaces; a query failure degrades to an unavailable-chart placeholder rather than breaking the page.
• Cluster Infrastructure (#13): provides the active/standby singleton hosting for the recorder, which drains on CoordinatedShutdown for clean handover.