docs(m6): KPI History & Trends design — reusable tall/EAV KPI-history backbone + trend charts for all sources; T9/T10 (Teams) deferred to next major version

docs/plans/2026-06-17-m6-kpi-history-design.md

@@ -0,0 +1,182 @@
+# M6 — KPI History & Trends — Design
+
+**Date:** 2026-06-17
+**Milestone:** M6 (from `docs/plans/2026-06-15-stillpending-completion-design.md`)
+**Status:** Approved — proceed to implementation plan.
+
+## Summary
+
+M6 was originally scoped as Notifications (T9 Teams adapter, T10 `NotificationType` enum
+values + UI Type selector, T11 historical/trend KPI charts). During brainstorming the scope
+was reshaped:
+
+- **T9 (Teams + other non-Email delivery adapters)** — **DEFERRED to next major version**
+  (user decision 2026-06-17). The `INotificationDeliveryAdapter` seam already exists; no code
+  now. Transport choice (Incoming Webhook vs Microsoft Graph) and the Teams list-targeting
+  model remain to be designed.
+- **T10 (`NotificationType` enum values + Central UI list "Type" selector)** — **DEFERRED with
+  T9.** A Type selector has no purpose until a second delivery type exists.
+- **T11 (historical/trend KPI charts)** — promoted from a notifications-only feature into a
+  **reusable common KPI-history backbone** with trend charts shipped for **all** current KPI
+  sources.
+
+This supersedes the original `docs/plans/notif.md` "KPI history — point-in-time only … no
+separate time-series store is added" decision: the completion plan (T11) explicitly introduces
+a store. We keep it in **central MS SQL** (the existing HA store) — no new infra dependency.
+
+## Goal
+
+A reusable central KPI-history backbone — tall/EAV store + periodic recorder + bucketed query
+API + a custom SVG trend-chart component — with trend charts delivered for Notification Outbox,
+Site Call Audit, Audit Log, and Site Health.
+
+## Reuse map (why a common backbone)
+
+Every source below is **point-in-time only today** (computed on demand, or only the latest kept
+in memory). The backbone turns each into a trend with no per-source schema work:
+
+| Source | Metrics (already computed) | Scope | Persisted today? |
+|---|---|---|---|
+| **Notification Outbox** | queue depth, stuck, parked, delivered/interval, oldest-pending age | global / per-site / per-node | on-demand only |
+| **Site Call Audit** | buffered, parked, failed/interval, delivered/interval, oldest-pending, stuck | global / per-site / per-node | on-demand only |
+| **Audit Log** | events/hour, error events/hour, backlog total | global | on-demand only |
+| **Site Health** | connection up/down, tag-resolution good/bad, script errors, alarm-eval errors, S&F buffer depth, dead-letter count, parked count, deployed/enabled/disabled instances, audit backlog (pending/oldest/bytes), event-log write failures | per-site | only latest, in-memory |
+
+Site Health is the largest latent win — sequence-numbered every 30s, but history is currently
+discarded.
+
+## Architecture
+
+### New component: `#26 KpiHistory`
+
+A small central component (mirrors NotificationOutbox / SiteCallAudit / AuditLog being their own
+projects). Owns the recorder, options, and the `IKpiSampleSource` abstraction.
+
+**Decoupling.** Each owning component registers its own `IKpiSampleSource` into DI; the recorder
+enumerates `IEnumerable<IKpiSampleSource>` (same pattern as `INotificationDeliveryAdapter`). So
+KpiHistory does **not** reference every component — each source impl lives with its owner and
+calls that owner's existing `Compute…KpisAsync` methods (or, for Site Health, reads the
+in-memory aggregator). CLAUDE.md component count goes 25 → 26.
+
+### Schema — `KpiSample` (tall / EAV)
+
+Commons entity, EF mapping + migration in ConfigurationDatabase, central MS SQL:
+
+| Column | Type | Notes |
+|---|---|---|
+| `Id` | `bigint` PK identity | |
+| `Source` | `varchar(64)` | `NotificationOutbox` / `SiteCallAudit` / `AuditLog` / `SiteHealth` |
+| `Metric` | `varchar(64)` | per-source constant, e.g. `queueDepth`, `parkedCount`, `deadLetterCount` |
+| `Scope` | `varchar(16)` | `Global` / `Site` / `Node` |
+| `ScopeKey` | `varchar(64)` NULL | site id / node name; `NULL` for `Global` |
+| `Value` | `float` | counts exact within range; ages stored as **seconds** |
+| `CapturedAtUtc` | `datetime2` | recorder tick timestamp (UTC) |
+
+Indexes:
+- `IX_KpiSample_Series (Source, Metric, Scope, ScopeKey, CapturedAtUtc)` — per-series range query.
+- `IX_KpiSample_Captured (CapturedAtUtc)` — retention purge.
+
+### Recorder — `KpiHistoryRecorderActor` (central cluster singleton)
+
+Runs on the active central node (consistent with the existing central singletons — Notification
+Outbox, Site Call Audit, purge actors). Timer fires every `SampleIntervalSeconds` (default 60s).
+Per tick:
+
+1. Open a DI scope (scoped `DbContext`/repository — mirrors `NotificationOutboxActor`'s
+   scope-per-sweep pattern).
+2. Enumerate registered `IKpiSampleSource`s; each returns `IReadOnlyList<KpiSample>` stamped with
+   the tick's `CapturedAtUtc`.
+3. Write all samples via `IKpiHistoryRepository.RecordSamplesAsync`.
+
+**Best-effort (R2).** Each source call and the write are individually wrapped; a failure logs and
+skips that source's samples for the tick — it never throws into or disrupts the source component.
+
+**Retention.** Daily purge (`PurgeIntervalHours`, default 24) deletes rows older than
+`RetentionDays` (default 90), reusing the existing purge-scheduler shape. Hourly downsampling
+beyond N days is deferred (YAGNI).
+
+### Sample sources
+
+- **`NotificationOutboxKpiSampleSource`** (in NotificationOutbox) →
+  `queueDepth, stuckCount, parkedCount, deliveredLastInterval, oldestPendingAgeSeconds`;
+  Global + per-Site + per-Node (reuses the M5 `ComputePerNodeKpisAsync`).
+- **`SiteCallAuditKpiSampleSource`** (in SiteCallAudit) →
+  `buffered, parked, failedLastInterval, deliveredLastInterval, oldestPendingAgeSeconds, stuck`;
+  Global + per-Site + per-Node.
+- **`AuditLogKpiSampleSource`** (in AuditLog) →
+  `totalEventsLastHour, errorEventsLastHour, backlogTotal`; Global.
+- **`SiteHealthKpiSampleSource`** (in HealthMonitoring) → reads
+  `ICentralHealthAggregator.GetAllSiteStates()` (in-memory, no DB): per-Site
+  `connectionsUp/Down, tagsGood/Bad, scriptErrors, alarmEvalErrors, sfBufferDepth, deadLetters,
+  parkedMessages, deployedInstances/enabledInstances/disabledInstances, auditBacklogPending,
+  eventLogWriteFailures`.
+
+### Query + UI
+
+- **`IKpiHistoryRepository.GetSeriesAsync(source, metric, scope, scopeKey, fromUtc, toUtc,
+  maxPoints)`** → buckets `[from, toUtc]` into ≤ `maxPoints` buckets and returns
+  **last-value per bucket** (`KpiSeriesPoint(BucketStartUtc, Value)`). Last-value is correct for
+  gauge metrics; v1 uses one aggregation, avg/min/max deferred.
+- **`KpiHistoryQueryService`** (CentralUI) — scoped-repo direct read with a dual-ctor test seam,
+  exactly like `AuditLogQueryService`.
+- **`KpiTrendChart.razor`** — reusable custom **SVG** line/area component: polyline path, min/max
+  + time-range axis labels, responsive `viewBox`, clean corporate styling (no third-party
+  charting lib, per CLAUDE.md). The time window (24h / 7d) is owned by the parent page.
+- **Surfaces:** trend charts on the **Notification Outbox**, **Site Calls**, and **Audit Log**
+  pages, plus a per-site trend panel on the **Health dashboard**.
+
+### Config — `KpiHistoryOptions`
+
+`SampleIntervalSeconds` (60), `RetentionDays` (90), `PurgeIntervalHours` (24),
+`DefaultMaxSeriesPoints` (200), with an options validator; bound on the central role in Host.
+
+## Error handling
+
+- Recorder: best-effort, per-source isolation (above). The KPI history is observability, never on
+  a user-facing critical path.
+- Query: a query failure surfaces in the UI as an unavailable chart (em-dash / message), mirroring
+  how the existing KPI tiles surface transient failures — it never breaks the hosting page.
+
+## Testing
+
+- Recorder writes samples; **best-effort source-failure isolation** (a throwing source does not
+  abort the tick or other sources).
+- Repository range/bucket-query correctness; retention purge deletes only aged rows.
+- `KpiTrendChart` SVG render (unit/bUnit-style).
+- One Playwright trend-chart UI test (per the M5–M10 testing strategy).
+- Targeted tests per task (filtered tests + per-project builds); full-solution build at
+  integration.
+
+## Docs & deploy
+
+- New `docs/requirements/Component-KpiHistory.md` (#26).
+- README component table + CLAUDE.md (25 → 26 + KPI-history bullet).
+- Interactions updated in Component-NotificationOutbox / -SiteCallAudit / -AuditLog /
+  -HealthMonitoring / -CentralUI.
+- Update `docs/plans/2026-06-15-stillpending-completion-design.md` (T9/T10 deferred; T11 →
+  KPI-history backbone).
+- EF migration auto-applies in dev; cluster rebuild via `bash docker/deploy.sh` at integration.
+
+## Execution housekeeping
+
+- Work in the dedicated worktree `m6-kpi-history`, branched off **local `main` (639e331,
+  includes unpushed M5)** — not `origin/main`.
+- Implementers commit **pathspec form** (`git commit -m "…" -- <paths>`), retry on
+  `.git/index.lock`.
+- Keep concurrent committers to ≤ 2–3 and run a **post-wave HEAD-presence check** per the
+  concurrent-commit ref-race lesson.
+
+## Deferred (next major version)
+
+- **T9** — Teams (and other non-Email) delivery adapter behind `INotificationDeliveryAdapter`.
+- **T10** — `NotificationType` enum values + Central UI list "Type" selector.
+
+## Task list
+
+- **MK-1 (first priority):** Common KPI-history/rollup store — backbone (schema + recorder +
+  repository + retention + chart component).
+- **MK-2:** Notification Outbox trend charts (T11 first consumer) — blocked by MK-1.
+- **MK-3:** Trend consumers — Site Call Audit / Audit Log / Site Health — blocked by MK-1.
+
+(With the "charts for all sources" scope, MK-2 and MK-3 together deliver the full UI this
+milestone; the implementation plan sequences them.)