ScadaBridge/docs/components/KpiHistory.md

# KPI History

KPI History (#26) is a central-only component that records periodic point-in-time KPI samples into the configuration database, folds them into hourly rollups, and purges both on a retention schedule so the Central UI can draw trend charts. The project owns the recording singleton and its options; the sample sources, repository contract and query service live elsewhere.

## Overview

The component lives in `src/ZB.MOM.WW.ScadaBridge.KpiHistory/` and references only Commons (plus Akka and `ZB.MOM.WW.Configuration`). Its pieces are:

- `KpiHistoryRecorderActor` — a `ReceiveActor` with `IWithTimers`, created as the `kpi-history-recorder` cluster singleton by `AkkaHostedService` on the central role. The Host logs it as "not readiness-gated", so it never affects `/health/ready`.
- `KpiHistoryOptions` and `KpiHistoryOptionsValidator` — the `ScadaBridge:KpiHistory` section and its startup validation.
- `KpiSourceAdmission` — root-provider-owned admission so a replaced actor cannot start a second collection while an earlier one is still running.
- `ServiceCollectionExtensions.AddKpiHistory` — binds and validates the options and registers `KpiSourceAdmission` as a singleton. The actor itself is not registered in DI.

## Key Concepts

### Sources, samples, repository

The recorder is generic over sources: each pass it enumerates every DI-registered `IKpiSampleSource` (`Source` name plus `CollectAsync(capturedAtUtc, ct)` returning `KpiSample` rows) and writes the combined batch via `IKpiHistoryRepository.RecordSamplesAsync`. The sources registered in the codebase are `NotificationOutboxKpiSampleSource`, `SiteCallAuditKpiSampleSource`, `AuditLogKpiSampleSource` and `SiteHealthKpiSampleSource` (each a scoped registration via `TryAddEnumerable`); the matching names are the constants in `KpiSources` (`NotificationOutbox`, `SiteCallAudit`, `AuditLog`, `SiteHealth`). The repository implementation is `KpiHistoryRepository` in the Configuration Database project.

### Four timers

`PreStart` arms four timers:

| Timer | First fire | Cadence | Work |
|---|---|---|---|
| `kpi-sample` | 5 s | `SampleInterval` | Collect from all sources and record one batch stamped with a single `capturedAtUtc`. |
| `kpi-purge` | `PurgeTimerSchedule.InitialDelay(PurgeInterval)` (the interval, capped at 5 minutes) | `PurgeInterval` | `PurgeOlderThanAsync` for raw rows older than `RetentionDays`, then `PurgeRollupsOlderThanAsync` for rollups older than `RollupRetentionDays`. |
| `kpi-rollup` | 10 s | `RollupInterval` | `FoldHourlyRollupsAsync` over `[now truncated to hour − RollupLookbackHours, now truncated to hour)`; the in-progress hour is excluded. |
| `kpi-rollup-backfill` | 15 s, single-shot | once per actor lifetime | Fold the raw-retention window in slices. |

The fold is an idempotent upsert, so a tick missed during a singleton handover self-heals on the next fold.

### In-flight guards

A sample tick is skipped (debug log) while a previous sample pass is still running, and a rollup tick is skipped while either a rollup fold or the backfill is running. Every pass is piped back to the actor with a `...Complete` message on success or failure, so a faulted pass cannot wedge the guard.

### Source isolation

Each source is collected in its own DI scope through `KpiSourceAdmission.TryStart`, keyed by registration index. If an earlier collection for that index has not completed, no new one starts and the running task is awaited instead. The wait is bounded by `SourceTimeout` (via `WaitAsync`); the collection also gets a linked token that cancels after `SourceTimeout`. On timeout the source is skipped with a warning while its work stays owned; any other source exception is logged and the pass continues. If no samples were collected nothing is written. A failed batch write is logged and the batch dropped.

### Backfill

The backfill first reads `GetLatestRollupHourAsync` (the watermark; failures fall back to a null watermark, meaning the full window). `HandleBackfillPlan` sets the floor to the later of the watermark and `now − RetentionDays`. If that floor is at or after `now − RollupLookbackHours`, rollups are already current and the backfill is skipped. Otherwise the range is cut into 24-hour slices and folded one per tick, with the periodic fold interleaving between slices (a slice tick re-arms after 5 seconds while a periodic fold runs). The backfill latches done when the queue empties.

### Shutdown

`PostStop` cancels a shared `CancellationTokenSource`; cancellation-triggered exits are treated as non-failures throughout.

## Architecture

### Cross-validation in Host

`KpiRateFoldIntervalCrossValidator` (Host project) is registered against `IValidateOptions<KpiHistoryOptions>` and fails startup unless `KpiHistory:SampleInterval`, `NotificationOutbox:DeliveredKpiWindow` and `SiteCallAudit:KpiInterval` are all equal. The Notification Outbox and Site Call Audit sample sources rescale their delivered/failed-last-interval counts onto `SampleInterval`, so the hourly sum stays correct regardless; the equality check keeps charted values matching the live Health tiles.

### Query side

The Central UI registers `KpiHistoryQueryService` (scoped, in `CentralUI/ServiceCollectionExtensions.cs`), which reads `KpiHistoryOptions` and the repository. `RollupThresholdHours` and `DefaultMaxSeriesPoints` are consumed there; the recorder does not use them. Per the option docs, windows up to `RollupThresholdHours` read raw `KpiSample` rows and wider windows read hourly rollups.

## Usage

The component exposes no message API. Wiring is `services.AddKpiHistory(configuration)` in the Host's central composition, and the singleton is started by `AkkaHostedService` with `Props.Create(() => new KpiHistoryRecorderActor(serviceProvider, options, logger))`. To add a metric source, implement `IKpiSampleSource` and register it as a scoped service with `TryAddEnumerable`.

## Configuration

`KpiHistoryOptions` is bound from `ScadaBridge:KpiHistory` and validated on start.

| Key | Default | Validation |
|---|---|---|
| `SampleInterval` | `00:01:00` | > 0 |
| `SourceTimeout` | `00:00:10` | > 0 and <= 1 day |
| `RetentionDays` | `90` | [1, 3650] |
| `PurgeInterval` | `1.00:00:00` | > 0 |
| `DefaultMaxSeriesPoints` | `200` | [2, 5000] |
| `RollupInterval` | `01:00:00` | > 0 |
| `RollupLookbackHours` | `3` | [1, 168] |
| `RollupRetentionDays` | `365` | [1, 3650] and >= `RetentionDays` |
| `RollupThresholdHours` | `168` | >= 24 |

## Dependencies & Interactions

- [Commons (#16)](./Commons.md) — `IKpiSampleSource`, `IKpiHistoryRepository`, `KpiSample`, `KpiSources`, `PurgeTimerSchedule`.
- [Configuration Database (#17)](./ConfigurationDatabase.md) — `KpiHistoryRepository`.
- [Notification Outbox (#21)](./NotificationOutbox.md), [Site Call Audit (#22)](./SiteCallAudit.md), [Audit Log (#23)](./AuditLog.md), [Health Monitoring (#11)](./HealthMonitoring.md) — each contributes an `IKpiSampleSource`.
- [Central UI (#9)](./CentralUI.md) — trend charts through `KpiHistoryQueryService`.
- [Host (#15)](./Host.md) — registration, singleton creation and the rate-fold cross-validator.
- [Cluster Infrastructure (#13)](./ClusterInfrastructure.md) — singleton failover.

## Troubleshooting

### Host fails to start with "KPI rate-fold interval mismatch"

`SampleInterval`, `DeliveredKpiWindow` and `KpiInterval` differ; set all three equal.

### Startup fails on `RollupRetentionDays`

It must be at least `RetentionDays`, otherwise long-range charts would hit a window where raw rows are purged and no rollup exists.

### A source never appears in charts

Look for "exceeded its collection deadline" warnings. A source that ignores cancellation keeps ownership of its slot and is not restarted until the earlier call completes.

### Long-range charts blank after a deploy

The backfill runs 15 seconds after singleton start and processes one 24-hour slice per tick; check for the "KPI rollup backfill" log lines.

## Related Documentation

- [KPI History design specification](../requirements/Component-KpiHistory.md)
- [Notification Outbox](./NotificationOutbox.md)
- [Site Call Audit](./SiteCallAudit.md)
- [Audit Log](./AuditLog.md)
- [Configuration Database](./ConfigurationDatabase.md)
- [Commons](./Commons.md)
- [Host](./Host.md)