Joseph Doherty
8.1 KiB
KPI History
KPI History (#26) is a central-only component that records periodic point-in-time KPI samples into the configuration database, folds them into hourly rollups, and purges both on a retention schedule so the Central UI can draw trend charts. The project owns the recording singleton and its options; the sample sources, repository contract and query service live elsewhere.
Overview
The component lives in src/ZB.MOM.WW.ScadaBridge.KpiHistory/ and references only Commons (plus Akka and ZB.MOM.WW.Configuration). Its pieces are:
KpiHistoryRecorderActor— aReceiveActorwithIWithTimers, created as thekpi-history-recordercluster singleton byAkkaHostedServiceon the central role. The Host logs it as "not readiness-gated", so it never affects/health/ready.KpiHistoryOptionsandKpiHistoryOptionsValidator— theScadaBridge:KpiHistorysection and its startup validation.KpiSourceAdmission— root-provider-owned admission so a replaced actor cannot start a second collection while an earlier one is still running.ServiceCollectionExtensions.AddKpiHistory— binds and validates the options and registersKpiSourceAdmissionas a singleton. The actor itself is not registered in DI.
Key Concepts
Sources, samples, repository
The recorder is generic over sources: each pass it enumerates every DI-registered IKpiSampleSource (Source name plus CollectAsync(capturedAtUtc, ct) returning KpiSample rows) and writes the combined batch via IKpiHistoryRepository.RecordSamplesAsync. The sources registered in the codebase are NotificationOutboxKpiSampleSource, SiteCallAuditKpiSampleSource, AuditLogKpiSampleSource and SiteHealthKpiSampleSource (each a scoped registration via TryAddEnumerable); the matching names are the constants in KpiSources (NotificationOutbox, SiteCallAudit, AuditLog, SiteHealth). The repository implementation is KpiHistoryRepository in the Configuration Database project.
Four timers
PreStart arms four timers:
| Timer | First fire | Cadence | Work |
|---|---|---|---|
kpi-sample |
5 s | SampleInterval |
Collect from all sources and record one batch stamped with a single capturedAtUtc. |
kpi-purge |
PurgeTimerSchedule.InitialDelay(PurgeInterval) (the interval, capped at 5 minutes) |
PurgeInterval |
PurgeOlderThanAsync for raw rows older than RetentionDays, then PurgeRollupsOlderThanAsync for rollups older than RollupRetentionDays. |
kpi-rollup |
10 s | RollupInterval |
FoldHourlyRollupsAsync over [now truncated to hour − RollupLookbackHours, now truncated to hour); the in-progress hour is excluded. |
kpi-rollup-backfill |
15 s, single-shot | once per actor lifetime | Fold the raw-retention window in slices. |
The fold is an idempotent upsert, so a tick missed during a singleton handover self-heals on the next fold.
In-flight guards
A sample tick is skipped (debug log) while a previous sample pass is still running, and a rollup tick is skipped while either a rollup fold or the backfill is running. Every pass is piped back to the actor with a ...Complete message on success or failure, so a faulted pass cannot wedge the guard.
Source isolation
Each source is collected in its own DI scope through KpiSourceAdmission.TryStart, keyed by registration index. If an earlier collection for that index has not completed, no new one starts and the running task is awaited instead. The wait is bounded by SourceTimeout (via WaitAsync); the collection also gets a linked token that cancels after SourceTimeout. On timeout the source is skipped with a warning while its work stays owned; any other source exception is logged and the pass continues. If no samples were collected nothing is written. A failed batch write is logged and the batch dropped.
Backfill
The backfill first reads GetLatestRollupHourAsync (the watermark; failures fall back to a null watermark, meaning the full window). HandleBackfillPlan sets the floor to the later of the watermark and now − RetentionDays. If that floor is at or after now − RollupLookbackHours, rollups are already current and the backfill is skipped. Otherwise the range is cut into 24-hour slices and folded one per tick, with the periodic fold interleaving between slices (a slice tick re-arms after 5 seconds while a periodic fold runs). The backfill latches done when the queue empties.
Shutdown
PostStop cancels a shared CancellationTokenSource; cancellation-triggered exits are treated as non-failures throughout.
Architecture
Cross-validation in Host
KpiRateFoldIntervalCrossValidator (Host project) is registered against IValidateOptions<KpiHistoryOptions> and fails startup unless KpiHistory:SampleInterval, NotificationOutbox:DeliveredKpiWindow and SiteCallAudit:KpiInterval are all equal. The Notification Outbox and Site Call Audit sample sources rescale their delivered/failed-last-interval counts onto SampleInterval, so the hourly sum stays correct regardless; the equality check keeps charted values matching the live Health tiles.
Query side
The Central UI registers KpiHistoryQueryService (scoped, in CentralUI/ServiceCollectionExtensions.cs), which reads KpiHistoryOptions and the repository. RollupThresholdHours and DefaultMaxSeriesPoints are consumed there; the recorder does not use them. Per the option docs, windows up to RollupThresholdHours read raw KpiSample rows and wider windows read hourly rollups.
Usage
The component exposes no message API. Wiring is services.AddKpiHistory(configuration) in the Host's central composition, and the singleton is started by AkkaHostedService with Props.Create(() => new KpiHistoryRecorderActor(serviceProvider, options, logger)). To add a metric source, implement IKpiSampleSource and register it as a scoped service with TryAddEnumerable.
Configuration
KpiHistoryOptions is bound from ScadaBridge:KpiHistory and validated on start.
| Key | Default | Validation |
|---|---|---|
SampleInterval |
00:01:00 |
> 0 |
SourceTimeout |
00:00:10 |
> 0 and <= 1 day |
RetentionDays |
90 |
[1, 3650] |
PurgeInterval |
1.00:00:00 |
> 0 |
DefaultMaxSeriesPoints |
200 |
[2, 5000] |
RollupInterval |
01:00:00 |
> 0 |
RollupLookbackHours |
3 |
[1, 168] |
RollupRetentionDays |
365 |
[1, 3650] and >= RetentionDays |
RollupThresholdHours |
168 |
>= 24 |
Dependencies & Interactions
- Commons (#16) —
IKpiSampleSource,IKpiHistoryRepository,KpiSample,KpiSources,PurgeTimerSchedule. - Configuration Database (#17) —
KpiHistoryRepository. - Notification Outbox (#21), Site Call Audit (#22), Audit Log (#23), Health Monitoring (#11) — each contributes an
IKpiSampleSource. - Central UI (#9) — trend charts through
KpiHistoryQueryService. - Host (#15) — registration, singleton creation and the rate-fold cross-validator.
- Cluster Infrastructure (#13) — singleton failover.
Troubleshooting
Host fails to start with "KPI rate-fold interval mismatch"
SampleInterval, DeliveredKpiWindow and KpiInterval differ; set all three equal.
Startup fails on RollupRetentionDays
It must be at least RetentionDays, otherwise long-range charts would hit a window where raw rows are purged and no rollup exists.
A source never appears in charts
Look for "exceeded its collection deadline" warnings. A source that ignores cancellation keeps ownership of its slot and is not restarted until the earlier call completes.
Long-range charts blank after a deploy
The backfill runs 15 seconds after singleton start and processes one 24-hour slice per tick; check for the "KPI rollup backfill" log lines.