merge(r2): r2-plan04

docs/requirements/Component-KpiHistory.md

@@ -88,9 +88,9 @@ A timer fires every `SampleInterval` (default 60s; an immediate first tick prime
 
 **Best-effort, per-source isolation.** Each source call and the write are individually guarded. A throwing source is logged and its samples skipped for that tick; it never aborts the tick, the other sources, or the source component itself. This is the same `IEnumerable<>`-of-adapters decoupling pattern used by `INotificationDeliveryAdapter`.
 
-**Hourly rollup tick.** A third timer (`kpi-rollup`) fires every `RollupInterval` (default 1h). On each tick the recorder opens a per-tick DI scope and calls `IKpiHistoryRepository.FoldHourlyRollupsAsync` over the window `[TruncateToHour(now) − RollupLookbackHours, TruncateToHour(now))` — an **exclusive** upper bound at the current hour start, so the in-progress hour is never folded. The fold groups raw `KpiSample` rows by `(series, hour)`, applies the per-metric aggregation intent (sum for rate metrics, last-value for gauges — see Query), and **idempotently upserts** each `(Source, Metric, Scope, ScopeKey, HourStartUtc)` row. Re-folding the trailing lookback window (default 3h) rather than only the last hour is the self-heal for a **singleton-failover-missed tick**: because the upsert overwrites the aggregate in place, re-running the same window produces identical values (no double-count), so any complete hour a missed tick skipped is refilled on the next fold. A `_rollupInFlight` guard (mirroring `_sampleInFlight`) skips a tick while a prior fold is still running so two idempotent upserts never race on overlapping rows.
+**Hourly rollup tick.** A third timer (`kpi-rollup`) fires every `RollupInterval` (default 1h). On each tick the recorder opens a per-tick DI scope and calls `IKpiHistoryRepository.FoldHourlyRollupsAsync` over the window `[TruncateToHour(now) − RollupLookbackHours, TruncateToHour(now))` — an **exclusive** upper bound at the current hour start, so the in-progress hour is never folded. The fold groups raw `KpiSample` rows by `(series, hour)`, applies the per-metric aggregation intent (sum for rate metrics, last-value for gauges — see Query), and **idempotently upserts** each `(Source, Metric, Scope, ScopeKey, HourStartUtc)` row. Re-folding the trailing lookback window (default 3h) rather than only the last hour is the self-heal for a **singleton-failover-missed tick**: because the upsert overwrites the aggregate in place, re-running the same window produces identical values (no double-count), so any complete hour a missed tick skipped is refilled on the next fold. A `_rollupInFlight` guard (mirroring `_sampleInFlight`) skips a tick while a prior fold is still running so two idempotent upserts never race on overlapping rows. During a **failover overlap** — the old singleton incarnation's in-flight fold and the new node's first fold both `Add`-ing the same `(series, hour)` row — the loser of the unique `IX_KpiRollupHourly_Series` upsert race discards its **whole fold pass** (a single `SaveChanges`), logged at Information and repaired by the next idempotent fold; when reading fold logs after a failover, the failure grain is the **pass, not the row** (arch-review 04 round 2, R5).
 
-**One-shot backfill.** So the 30 d / 90 d charts are not blank until enough wall-clock passes, the recorder runs a **single** backfill on start (`kpi-rollup-backfill` timer) that folds the full retention window once, `[TruncateToHour(now) − RetentionDays, TruncateToHour(now))`, reusing the same `FoldHourlyRollupsAsync` path. A `_backfillDone` latch makes it at-most-once per actor lifetime, and it defers (re-arms) while a periodic fold is in flight; being an idempotent upsert, it is cheap and safe to re-run on a later failover restart.
+**One-shot backfill.** So the 30 d / 90 d charts are not blank until enough wall-clock passes, the recorder runs a backfill on start (`kpi-rollup-backfill` timer) that folds the retention window, reusing the same `FoldHourlyRollupsAsync` path. The `FoldHourlyRollupsAsync` repository call materializes its whole window in memory, so the backfill folds the window in **≤24 h slices, oldest-first** — one bounded slice per fold — rather than handing the fold the full 90-day window at once (which at fleet volume is tens of millions of rows in one tracked pass). Between slices it lowers its in-flight guard so periodic folds **interleave** instead of stalling behind the historical pass; a periodic fold and a backfill slice strictly alternate through the single `_rollupInFlight`/`_backfillInFlight` gate pair, so the two idempotent upserts never race on overlapping rows. On a **failover restart** the plan first consults the `IKpiHistoryRepository.GetLatestRollupHourAsync` watermark: if the newest folded hour is already within `RollupLookbackHours` of now (the common case) the whole historical pass is **skipped** — the periodic fold's lookback covers the tail — otherwise the window is **shrunk to the un-rolled tail** `[watermark, TruncateToHour(now))` (re-folding the watermark hour itself as an idempotent safety margin) instead of the full `RetentionDays` floor. A `_backfillDone` latch makes the whole pass at-most-once per actor lifetime, and it defers (re-arms) while a periodic fold is in flight. Being an idempotent upsert, it is safe (idempotent) to re-run on a later failover restart; the watermark fast-path makes the failover re-run cheap as well.
 
 **Retention.** A daily purge timer (`PurgeInterval`, default 24h) now runs **both** purges: raw `KpiSample` rows older than `RetentionDays` (default 90) via `IKpiHistoryRepository.PurgeOlderThanAsync`, **and** hourly rollup rows older than `RollupRetentionDays` (default 365) via `IKpiHistoryRepository.PurgeRollupsOlderThanAsync` (a one-hour-sliced batched DELETE mirroring the raw purge). Rollups deliberately **outlive** raw samples so long-range trend charts have data after the raw rows are purged; the options validator enforces `RollupRetentionDays >= RetentionDays` so a window never falls into a hole where raw is purged but no rollup exists.
 
@@ -148,15 +148,17 @@ Reads `ICentralHealthAggregator.GetAllSiteStates()` (in-memory, no DB). Scope: *
 
 ### Bucketed query
 
-`IKpiHistoryRepository.GetRawSeriesAsync(source, metric, scope, scopeKey, fromUtc, toUtc, …)` returns the raw points for one series over `[fromUtc, toUtc]`. `GetHourlySeriesAsync(…)` returns the same ascending `KpiSeriesPoint` shape from `KpiRollupHourly` (projecting `HourStartUtc` → the point timestamp, `Value`), so the bucketer is agnostic to the source. `KpiSeriesBucketer.Bucket(series, fromUtc, toUtc, maxPoints)` then partitions the window into ≤ `maxPoints` time buckets and returns the **last value per bucket** as `KpiSeriesPoint(BucketStartUtc, Value)` — unchanged, and still applied on top of a rollup series (a 90 d rollup is ~2,160 points > the 200-point cap, so bucketing still thins it). The chart plots the single `Value` series; avg / min / max charting is still deferred, though the rollup's `MinValue` / `MaxValue` / `SampleCount` columns are now stored to unblock it.
+`IKpiHistoryRepository.GetRawSeriesAsync(source, metric, scope, scopeKey, fromUtc, toUtc, …)` returns the raw points for one series over `[fromUtc, toUtc]`. `GetHourlySeriesAsync(…)` returns the same ascending `KpiSeriesPoint` shape from `KpiRollupHourly` (projecting `HourStartUtc` → the point timestamp, `Value`), so the bucketer is agnostic to the source. `KpiSeriesBucketer.Bucket(series, fromUtc, toUtc, maxPoints, aggregation)` then partitions the window into ≤ `maxPoints` time buckets and reduces each bucket **per-metric**: it keeps the **last value** for a **Gauge** series and **sums per bucket** for a **Rate** series, driven by the same `KpiMetricAggregationCatalog` the hourly fold consults. The previous unconditional last-value-per-bucket kept **1 of ~11 hourly sums** per 90 d bucket on the rollup path — the exact fold error the catalog's own doc warns about, re-introduced one layer up — and is fixed per arch-review 04 round 2, R3. The reduction is still applied on top of a rollup series (a 90 d rollup is ~2,160 points > the 200-point cap, so bucketing still thins it). The chart plots the single `Value` series; avg / min / max charting is still deferred, though the rollup's `MinValue` / `MaxValue` / `SampleCount` columns are now stored to unblock it.
 
 ### Raw-vs-rollup range routing
 
 `GetRawSeriesAsync` and `GetHourlySeriesAsync` are selected by window width in the query service (below), governed by `RollupThresholdHours` (default 168 = 7 d). Windows **at or below** the threshold read raw samples (preserving intra-minute detail on the 24 h / 7 d charts); windows **strictly wider** than the threshold read the pre-aggregated hourly rollups (≈60× fewer rows scanned on the 30 d / 90 d charts). The boundary is strict — a window of exactly `RollupThresholdHours` stays on the raw path.
 
+Rate charts keep **one unit (events per chart bucket)** on both sides of the boundary: the catalog-driven per-bucket sum reduction is applied to the raw-minute deltas and the hourly rollup sums alike, so widening 7 d → 30 d no longer changes a Rate chart's magnitude ~60×. Residual: an unbucketed just-over-threshold rollup series plots native per-hour sums vs. the at-threshold raw path's ~50-minute bucket sums (≤~1.2×) — accepted and documented, not hidden.
+
 ### Aggregation intent — gauge vs. rate
 
-Not every metric folds the same way, and choosing wrong silently corrupts long-range totals. `KpiMetricAggregationCatalog.Resolve(source, metric)` classifies each `(Source, Metric)` pair as `KpiRollupAggregation.Gauge` (fold = **last-value-per-hour**) or `KpiRollupAggregation.Rate` (fold = **sum-per-hour**):
+Not every metric folds the same way, and choosing wrong silently corrupts long-range totals. `KpiMetricAggregationCatalog.Resolve(source, metric)` classifies each `(Source, Metric)` pair as `KpiRollupAggregation.Gauge` (fold = **last-value-per-hour**) or `KpiRollupAggregation.Rate` (fold = **sum-per-hour**). The catalog now has **two consumers**: the hourly rollup fold (which picks last-value vs. sum per hour) and the chart-time bucket reduction in `KpiSeriesBucketer` (which picks last-value vs. sum per chart bucket) — one classification keeps both layers consistent:
 
 - **Gauge** (last-value) — instantaneous readings where a per-hour sum is meaningless: `queueDepth`, `buffered`, `stuck`/`stuckCount`, `parked*`, `oldestPendingAgeSeconds`, `connectionsUp`/`connectionsDown`, `deployedInstances`, and every other metric not explicitly listed as a rate. This is the **safe default**: an unknown / newly-added metric resolves to Gauge and never crashes the fold or invents a total.
 - **Rate** (sum-per-hour) — already-windowed per-interval deltas whose hour total is the sum of its minute samples: `NotificationOutbox/deliveredLastInterval`, `SiteCallAudit/failedLastInterval`, `SiteCallAudit/deliveredLastInterval`, and the per-report `SiteHealth` error counts `scriptErrors` / `alarmEvalErrors` / `deadLetters` / `eventLogWriteFailures`. Folding these as last-value would keep one minute's delta and discard the other 59, under-counting every long-range total.