ScadaBridge/docs/plans/2026-06-17-m6-kpi-history-design.md

M6 — KPI History & Trends — Design

Date: 2026-06-17 Milestone: M6 (from docs/plans/2026-06-15-stillpending-completion-design.md) Status: Approved — proceed to implementation plan.

Summary

M6 was originally scoped as Notifications (T9 Teams adapter, T10 NotificationType enum values + UI Type selector, T11 historical/trend KPI charts). During brainstorming the scope was reshaped:

• T9 (Teams + other non-Email delivery adapters) — DEFERRED to next major version (user decision 2026-06-17). The INotificationDeliveryAdapter seam already exists; no code now. Transport choice (Incoming Webhook vs Microsoft Graph) and the Teams list-targeting model remain to be designed.
• T10 (NotificationType enum values + Central UI list "Type" selector) — DEFERRED with T9. A Type selector has no purpose until a second delivery type exists.
• T11 (historical/trend KPI charts) — promoted from a notifications-only feature into a reusable common KPI-history backbone with trend charts shipped for all current KPI sources.

This supersedes the original docs/plans/notif.md "KPI history — point-in-time only … no separate time-series store is added" decision: the completion plan (T11) explicitly introduces a store. We keep it in central MS SQL (the existing HA store) — no new infra dependency.

Goal

A reusable central KPI-history backbone — tall/EAV store + periodic recorder + bucketed query API + a custom SVG trend-chart component — with trend charts delivered for Notification Outbox, Site Call Audit, Audit Log, and Site Health.

Reuse map (why a common backbone)

Every source below is point-in-time only today (computed on demand, or only the latest kept in memory). The backbone turns each into a trend with no per-source schema work:

Source	Metrics (already computed)	Scope	Persisted today?
Notification Outbox	queue depth, stuck, parked, delivered/interval, oldest-pending age	global / per-site / per-node	on-demand only
Site Call Audit	buffered, parked, failed/interval, delivered/interval, oldest-pending, stuck	global / per-site / per-node	on-demand only
Audit Log	events/hour, error events/hour, backlog total	global	on-demand only
Site Health	connection up/down, tag-resolution good/bad, script errors, alarm-eval errors, S&F buffer depth, dead-letter count, parked count, deployed/enabled/disabled instances, audit backlog (pending/oldest/bytes), event-log write failures	per-site	only latest, in-memory

Site Health is the largest latent win — sequence-numbered every 30s, but history is currently discarded.

Architecture

New component: #26 KpiHistory

A small central component (mirrors NotificationOutbox / SiteCallAudit / AuditLog being their own projects). Owns the recorder, options, and the IKpiSampleSource abstraction.

Decoupling. Each owning component registers its own IKpiSampleSource into DI; the recorder enumerates IEnumerable<IKpiSampleSource> (same pattern as INotificationDeliveryAdapter). So KpiHistory does not reference every component — each source impl lives with its owner and calls that owner's existing Compute…KpisAsync methods (or, for Site Health, reads the in-memory aggregator). CLAUDE.md component count goes 25 → 26.

Schema — KpiSample (tall / EAV)

Commons entity, EF mapping + migration in ConfigurationDatabase, central MS SQL:

Column	Type	Notes
Id	bigint PK identity
Source	varchar(64)	NotificationOutbox / SiteCallAudit / AuditLog / SiteHealth
Metric	varchar(64)	per-source constant, e.g. queueDepth, parkedCount, deadLetterCount
Scope	varchar(16)	Global / Site / Node
ScopeKey	varchar(64) NULL	site id / node name; NULL for Global
Value	float	counts exact within range; ages stored as seconds
CapturedAtUtc	datetime2	recorder tick timestamp (UTC)

Indexes:

• IX_KpiSample_Series (Source, Metric, Scope, ScopeKey, CapturedAtUtc) — per-series range query.
• IX_KpiSample_Captured (CapturedAtUtc) — retention purge.

Recorder — KpiHistoryRecorderActor (central cluster singleton)

Runs on the active central node (consistent with the existing central singletons — Notification Outbox, Site Call Audit, purge actors). Timer fires every SampleIntervalSeconds (default 60s). Per tick:

1. Open a DI scope (scoped DbContext/repository — mirrors NotificationOutboxActor's scope-per-sweep pattern).
2. Enumerate registered IKpiSampleSources; each returns IReadOnlyList<KpiSample> stamped with the tick's CapturedAtUtc.
3. Write all samples via IKpiHistoryRepository.RecordSamplesAsync.

Best-effort (R2). Each source call and the write are individually wrapped; a failure logs and skips that source's samples for the tick — it never throws into or disrupts the source component.

Retention. Daily purge (PurgeIntervalHours, default 24) deletes rows older than RetentionDays (default 90), reusing the existing purge-scheduler shape. Hourly downsampling beyond N days is deferred (YAGNI).

Sample sources

• NotificationOutboxKpiSampleSource (in NotificationOutbox) → queueDepth, stuckCount, parkedCount, deliveredLastInterval, oldestPendingAgeSeconds; Global + per-Site + per-Node (reuses the M5 ComputePerNodeKpisAsync).
• SiteCallAuditKpiSampleSource (in SiteCallAudit) → buffered, parked, failedLastInterval, deliveredLastInterval, oldestPendingAgeSeconds, stuck; Global + per-Site + per-Node.
• AuditLogKpiSampleSource (in AuditLog) → totalEventsLastHour, errorEventsLastHour, backlogTotal; Global.
• SiteHealthKpiSampleSource (in HealthMonitoring) → reads ICentralHealthAggregator.GetAllSiteStates() (in-memory, no DB): per-Site connectionsUp/Down, tagsGood/Bad, scriptErrors, alarmEvalErrors, sfBufferDepth, deadLetters, parkedMessages, deployedInstances/enabledInstances/disabledInstances, auditBacklogPending, eventLogWriteFailures.

Query + UI

• IKpiHistoryRepository.GetSeriesAsync(source, metric, scope, scopeKey, fromUtc, toUtc, maxPoints) → buckets [from, toUtc] into ≤ maxPoints buckets and returns last-value per bucket (KpiSeriesPoint(BucketStartUtc, Value)). Last-value is correct for gauge metrics; v1 uses one aggregation, avg/min/max deferred.
• KpiHistoryQueryService (CentralUI) — scoped-repo direct read with a dual-ctor test seam, exactly like AuditLogQueryService.
• KpiTrendChart.razor — reusable custom SVG line/area component: polyline path, min/max
  • time-range axis labels, responsive viewBox, clean corporate styling (no third-party charting lib, per CLAUDE.md). The time window (24h / 7d) is owned by the parent page.
• Surfaces: trend charts on the Notification Outbox, Site Calls, and Audit Log pages, plus a per-site trend panel on the Health dashboard.

Config — KpiHistoryOptions

SampleIntervalSeconds (60), RetentionDays (90), PurgeIntervalHours (24), DefaultMaxSeriesPoints (200), with an options validator; bound on the central role in Host.

Error handling

• Recorder: best-effort, per-source isolation (above). The KPI history is observability, never on a user-facing critical path.
• Query: a query failure surfaces in the UI as an unavailable chart (em-dash / message), mirroring how the existing KPI tiles surface transient failures — it never breaks the hosting page.

Testing

• Recorder writes samples; best-effort source-failure isolation (a throwing source does not abort the tick or other sources).
• Repository range/bucket-query correctness; retention purge deletes only aged rows.
• KpiTrendChart SVG render (unit/bUnit-style).
• One Playwright trend-chart UI test (per the M5–M10 testing strategy).
• Targeted tests per task (filtered tests + per-project builds); full-solution build at integration.

Docs & deploy

• New docs/requirements/Component-KpiHistory.md (#26).
• README component table + CLAUDE.md (25 → 26 + KPI-history bullet).
• Interactions updated in Component-NotificationOutbox / -SiteCallAudit / -AuditLog / -HealthMonitoring / -CentralUI.
• Update docs/plans/2026-06-15-stillpending-completion-design.md (T9/T10 deferred; T11 → KPI-history backbone).
• EF migration auto-applies in dev; cluster rebuild via bash docker/deploy.sh at integration.

Execution housekeeping

• Work in the dedicated worktree m6-kpi-history, branched off local main (639e331, includes unpushed M5) — not origin/main.
• Implementers commit pathspec form (git commit -m "…" -- <paths>), retry on .git/index.lock.
• Keep concurrent committers to ≤ 2–3 and run a post-wave HEAD-presence check per the concurrent-commit ref-race lesson.

Deferred (next major version)

• T9 — Teams (and other non-Email) delivery adapter behind INotificationDeliveryAdapter.
• T10 — NotificationType enum values + Central UI list "Type" selector.

Task list

• MK-1 (first priority): Common KPI-history/rollup store — backbone (schema + recorder + repository + retention + chart component).
• MK-2: Notification Outbox trend charts (T11 first consumer) — blocked by MK-1.
• MK-3: Trend consumers — Site Call Audit / Audit Log / Site Health — blocked by MK-1.

(With the "charts for all sources" scope, MK-2 and MK-3 together deliver the full UI this milestone; the implementation plan sequences them.)