diff --git a/docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md b/docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md new file mode 100644 index 00000000..f5d28893 --- /dev/null +++ b/docs/plans/2026-07-10-aggregated-live-alarm-stream-plan.md @@ -0,0 +1,183 @@ +# Plan: Aggregated Live Alarm Stream for Alarm Summary (deferred #10) + +**Status:** Draft plan (not yet executed) — 2026-07-10 +**Register row:** `docs/plans/2026-07-08-deferred-work-register.md` #10 +**Revisit trigger that fired this plan:** *"Alarm Summary latency complaints or >~50 instances/site."* +**Owning component:** Central UI (#9) + Central–Site Communication (#5); no new component. + +--- + +## 1. Problem + +The operator Alarm Summary page (`/monitoring/alarms`, `RequireDeployment`, read-only) +currently rebuilds its whole view every 15 s by fanning out a per-instance +`DebugViewSnapshot` Ask to **every Enabled instance on the selected site** and +aggregating client-side (`AlarmSummaryService`, `MaxConcurrentFetches = 8`). At +current instance counts this is acceptable; past **~50 instances/site** the poll +does 50 cross-cluster Asks every 15 s (worst case 50 × ClusterClient round-trips +gated 8-at-a-time), so freshness degrades and the active central node carries a +periodic burst. We want an **aggregated live feed** so the page reflects alarm +transitions in near-real-time without re-polling the whole site. + +## 2. Hard constraint (locked decision — do not violate) + +`docs/plans/2026-05-29-native-alarms-design.md:24` marks **"no central alarm +tables, no central history"** as `[PERM]`. Therefore this plan introduces **no +persisted central alarm store** — only a **transient in-memory per-site live +cache** on the active central node, seeded from the existing snapshot fan-out and +kept warm by a live gRPC delta stream. If a persisted store is ever wanted, that +is a separate re-decision, not part of this plan. + +## 3. Chosen approach + +**A shared, reference-counted, per-site central live alarm cache**, seeded by the +existing snapshot fan-out and updated by a new **site-wide, alarm-only** gRPC +stream, pushed to the Blazor page via an in-process singleton event (the +`IDeploymentStatusNotifier` pattern — no new SignalR hub; the Blazor circuit +already pushes). + +### Why site-wide RPC (Option B) over N per-instance streams (Option A) + +The revisit trigger is *>~50 instances/site* — exactly where opening one gRPC +stream per instance (Option A) strains `GrpcMaxConcurrentStreams = 100` and +duplicates the fan-out problem in streaming form. A single **`SubscribeSite`** +stream per site (alarm events only; attributes are far higher-volume and the +summary never shows them) is the scalable answer and is an *additive* proto +change. It reuses the existing `SiteStreamManager` broadcast hub by adding a +subscribe variant that drops the per-instance `InstanceUniqueName` filter +(`SiteStreamManager.cs:101`) and keeps only `AlarmStateChanged` events. + +### Lifecycle & failover + +- **Shared per-site cache**, reference-counted by active viewers: first viewer of + a site starts the subscription + snapshot seed; last viewer leaving stops it + (with a short linger to avoid thrash). One gRPC stream per *site*, not per + browser circuit. +- **Seed-then-stream** ordering copied from `DebugStreamBridgeActor`: open the + stream first, buffer live deltas, run the snapshot fan-out once, flush buffer + with dedup, then live pass-through. Dedup identity = `(InstanceUniqueName, + AlarmName, SourceReference)` (matches `DebugStreamBridgeActor.AlarmKey`). +- **NodeA↔NodeB reconnect** with retry budget + stability window, reusing the + reconnect logic already proven in `DebugStreamBridgeActor`. +- **Placeholder reconciliation:** live stream drops `IsConfiguredPlaceholder` + rows (`StreamRelayActor.cs:63`); snapshots include them. Cache seeds + placeholders from the snapshot and never expects them on the live stream. +- **Instance-set drift:** on deploy/enable/disable/delete the site-wide stream + naturally starts/stops carrying that instance's events; a periodic (e.g. 60 s) + reconcile snapshot corrects any missed enable/disable so the cache can't drift + indefinitely. This replaces the old "re-query everything every 15 s" with + "stream + occasional reconcile." + +## 4. Task breakdown + +Classification legend: `trivial` / `small` (code review) / `standard` (spec+code +review) / `high-risk` (serial reviews + integration). Times are single-task +estimates. + +### Task 1 — Site-wide alarm-only stream on the site runtime `[standard]` +- **Files:** `src/ZB.MOM.WW.ScadaBridge.SiteRuntime/Streaming/SiteStreamManager.cs` +- Add `SubscribeSiteAlarms(ISiteStreamSubscriber subscriber)` — same broadcast-hub + wiring as `Subscribe(...)` but **without** the `InstanceUniqueName` filter and + **filtering to `AlarmStateChanged` only** (skip `AttributeValueChanged`). +- Return an `IDisposable`/kill-switch symmetric with the existing per-instance + subscribe. Unit-test that alarm events for *all* instances arrive and attribute + events do not. + +### Task 2 — `SubscribeSite` gRPC contract + server handler `[standard]` +- **Files:** `src/ZB.MOM.WW.ScadaBridge.Communication/Protos/sitestream.proto`, + `src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcServer.cs`, + `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/StreamRelayActor.cs` +- Add **additive** `rpc SubscribeSite(SiteStreamRequest) returns (stream + SiteStreamEvent)` + `SiteStreamRequest { correlation_id }` (no instance name). + Keep `bundleFormatVersion`/message numbering additive — do not renumber + existing fields. +- Server handler mirrors `SubscribeInstance` (bounded `Channel(1000, + DropOldest)`, `GrpcMaxConcurrentStreams`/`GrpcMaxStreamLifetime` limits, the + `StreamRelayActor` mapping, `SiteConnectionOpened/Closed` telemetry) but calls + `SubscribeSiteAlarms`. `StreamRelayActor` already drops placeholder rows and + emits enriched `AlarmStateUpdate` — reuse unchanged. + +### Task 3 — Central per-site subscription client `[standard]` +- **Files:** `src/ZB.MOM.WW.ScadaBridge.Communication/Grpc/SiteStreamGrpcClient.cs`, + `.../Grpc/SiteStreamGrpcClientFactory.cs` +- Add `SubscribeSiteAsync(correlationId, onAlarmEvent, onError, ct)` alongside + `SubscribeAsync`, reusing `ConvertToDomainEvent` (maps proto → domain + `AlarmStateChanged`). Factory already caches channels per `(site, endpoint)` for + failover — no change needed beyond the new call. + +### Task 4 — Central live alarm cache actor/service `[high-risk]` +- **Files (new):** `src/ZB.MOM.WW.ScadaBridge.Communication/Actors/SiteAlarmAggregatorActor.cs`, + `src/ZB.MOM.WW.ScadaBridge.Communication/SiteAlarmLiveCacheService.cs`, + `src/ZB.MOM.WW.ScadaBridge.Communication/ISiteAlarmLiveCache.cs` +- **Files (edit):** `src/ZB.MOM.WW.ScadaBridge.Communication/ServiceCollectionExtensions.cs` +- Per-site aggregator: seed-then-stream (buffer → snapshot fan-out via existing + `CommunicationService.RequestDebugSnapshotAsync` → flush+dedup → live), holds a + `Dictionary` snapshot of current alarm state. + Reference-counted start/stop keyed by site. NodeA↔NodeB reconnect + periodic + reconcile snapshot (60 s). Reuse `DebugStreamBridgeActor` reconnect logic + (extract shared helper or copy with attribution). +- Expose `ISiteAlarmLiveCache` (singleton) with `Subscribe(siteId, onChange)` → + `IDisposable`, raising an `IDeploymentStatusNotifier`-style event on each + applied delta. This is the **highest-risk** task (concurrency, failover, + seed/stream ordering) — spawn a code-reviewer subagent and add integration + coverage. + +### Task 5 — Wire Alarm Summary page to the live cache `[standard]` +- **Files:** `src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/AlarmSummaryService.cs`, + `.../Services/IAlarmSummaryService.cs`, + `.../Components/Pages/Monitoring/AlarmSummary.razor`, + `.../ServiceCollectionExtensions.cs` +- Add `SubscribeSiteAlarmsAsync(siteId, onChange)` to the service, delegating to + `ISiteAlarmLiveCache`. Page: on site select, subscribe; `onChange` updates + `_rows` + recomputes `_rollup`/`_visibleRows` and `StateHasChanged()` (over the + Blazor circuit). **Keep the 15 s poll as a fallback** but gate it behind a + feature flag / degrade path when the live cache is healthy, so a stream failure + never blanks the page. Unsubscribe in `Dispose()`. `AlarmStateBadges` and all + client-side filters/sort are unchanged (they already consume `AlarmStateChanged`). + +### Task 6 — Options, limits, and telemetry `[small]` +- **Files:** communication options class (grep `CommunicationOptions` / + `GrpcMaxConcurrentStreams`), `SiteAlarmLiveCacheService` +- Add options: live-cache linger, reconcile interval, per-site subscriber cap. + Validate (eager options validation is the house rule). Surface a gauge for + active per-site aggregators + reconnect count on the health snapshot, mirroring + `SiteConnectionUp` gauges. + +### Task 7 — Tests + integration trace `[standard]` +- **Files:** `tests/.../Communication.Tests/*`, `tests/.../CentralUI.Tests/*`, + `tests/.../SiteRuntime.Tests/Streaming/SiteStreamManagerTests.cs` +- Unit: `SubscribeSiteAlarms` filtering; `SubscribeSite` server/client mapping; + aggregator seed/stream/dedup/placeholder/reconnect; page subscribe/unsubscribe. +- Integration: end-to-end site→central alarm delta reaches the cache; failover + flip re-seeds; last-viewer stop tears the stream down. Live-smoke on the docker + cluster (`bash docker/deploy.sh`) with >1 instance raising alarms. + +### Task 8 — Docs `[trivial]` +- **Files:** `docs/requirements/Component-CentralUI.md` (Alarm Summary section — + replace "no aggregated live stream" with the shipped design), + `docs/requirements/Component-Communication.md` (new `SubscribeSite` RPC), + `docs/plans/2026-06-18-m7-opcua-mxgateway-ux-design.md` (mark T13 follow-up + delivered), and move register #10 to **Resolved**. + +## 5. Risks / call-outs + +- **`[PERM]` no-central-store** — the cache is explicitly transient/in-memory. + Reviewer must confirm no EF entity/table/migration is introduced. +- **Proto evolution** — `SubscribeSite` must be purely additive (new RPC + new + message; no field renumbering). PLAN-08's ClusterClient/gRPC contract-lock + tests apply. +- **Stream-vs-snapshot race** — the seed-then-stream dedup is the classic bug + surface; reuse the `DebugStreamBridgeActor` ordering exactly. +- **Failover gap** — a NodeA↔NodeB flip must re-seed (not silently serve stale); + the 60 s reconcile snapshot is the backstop. +- **Effort estimate:** ~6–9 focused tasks; Task 4 is the only genuinely hard one. + Everything else reuses existing, proven machinery + (`SiteStreamManager`/`SiteStreamGrpcClientFactory`/`DebugStreamBridgeActor`/ + `IDeploymentStatusNotifier`/`AlarmSummaryService`). + +## 6. Explicitly out of scope +- Persisted central alarm store or history (locked `[PERM]`). +- Aggregated attribute stream (summary needs alarms only). +- CLI/SignalR surface for the aggregated feed (Blazor-only; revisit if a CLI + consumer appears). +- Ack-back / write path (native alarms remain read-only mirrors). diff --git a/docs/plans/2026-07-10-kpi-history-hourly-rollups-plan.md b/docs/plans/2026-07-10-kpi-history-hourly-rollups-plan.md new file mode 100644 index 00000000..290a1bc1 --- /dev/null +++ b/docs/plans/2026-07-10-kpi-history-hourly-rollups-plan.md @@ -0,0 +1,210 @@ +# Plan: KPI History Hourly Rollups (deferred #22) + +**Status:** Draft plan (not yet executed) — 2026-07-10 +**Register row:** `docs/plans/2026-07-08-deferred-work-register.md` #22 +**Revisit trigger that fired this plan:** *"KpiSample query latency on dashboards."* +**Owning component:** KPI History (#26); touches Configuration Database (#17), +Commons (#16), Central UI (#9), and every `IKpiSampleSource` owner. + +--- + +## 1. Problem + +`KpiSample` is a tall/EAV table sampled once per minute per series +(`(Source, Metric, Scope, ScopeKey)` 4-tuple). Long-range trend queries fetch and +sort every raw row in the window before `KpiSeriesBucketer` downsamples to +`DefaultMaxSeriesPoints = 200`: + +| Window | Raw rows/series @ 1/min | +|--------|-------------------------| +| 24 h | 1,440 | +| 7 d | 10,080 | +| 30 d | 43,200 | +| 90 d | 129,600 | + +The four trend surfaces (Notification Outbox, Site Calls, Audit Log, Health) today +offer only **24 h** and **7 d** windows, so the worst case is ~10k rows/series. +The latency trigger only truly bites once **30 d / 90 d** windows are added — so +this plan must decide whether to add those windows *and* pre-aggregate them, or +rollups have no caller. + +## 2. Key facts that shape the design (from the code) + +- **The bucketer is not the bottleneck.** `KpiSeriesBucketer.Bucket` derives bucket + width purely from `(to − from) / maxPoints` *after* rows are in memory. The cost + is entirely in `GetRawSeriesAsync` fetching/sorting rows. Hourly rollups plug in + by feeding the bucketer a **pre-thinned ≤1-row/hour series** (≈60× fewer rows). +- **`KpiHistoryQueryService.GetSeriesAsync` already owns fetch→bucket + options** — + the natural place to branch raw-vs-rollup by range. +- **`KpiHistoryRecorderActor` takes the root `IServiceProvider` and opens a + per-tick scope** — a new hourly rollup timer needs **no constructor change**, + just a new timer + repo method (same shape as the existing sample/purge timers). +- **Metrics are not all gauges.** Most are instantaneous gauges (`queueDepth`, + `buffered`, `connectionsDown`, `oldestPendingAgeSeconds`) where last-value-per- + hour is correct. But several are **already-windowed rate counters** + (`deliveredLastInterval`, `failedLastInterval`, `*LastHour`, `scriptErrors`, + `deadLetters`) where last-value-per-hour **undercounts** (keeps one minute's + delta, discards the other 59). This is the single biggest correctness decision. +- Some metrics are **conditionally emitted** (`oldestPendingAgeSeconds` skipped + when null) — a rollup must not assume a row exists every minute. +- **This is greenfield** — no `rollup`/`KpiRollup` table/column/code exists. + +## 3. Chosen approach + +**A separate `KpiRollupHourly` table, populated by an hourly recorder tick, with +per-metric aggregation intent, queried via a range threshold, retained longer than +raw samples, and backfilled once at migration.** Also add **30 d / 90 d** windows +to the four surfaces so the rollups have a caller. + +Rationale for each decision (the research surfaced these as the required choices): + +**(a) Separate table vs. computed-on-read → separate `KpiRollupHourly` table.** +Computed-on-read (`GROUP BY DATEADD(hour…)`) needs no migration/backfill but still +**scans all raw rows** in SQL every query — it shrinks rows *returned*, not rows +*scanned*, so it only partially addresses the latency trigger. A pre-aggregated +table (design-consistent with the non-partitioned `KpiSample` model) makes a 90 d +read a small indexed scan (~2,160 rows vs ~130k). + +**(b) Who computes → recorder-side hourly tick.** Add a third Akka timer +(`RollupTick`) to `KpiHistoryRecorderActor`. Fold the **trailing N hours** +(lookback window, not just "last hour") so a singleton-failover handover that +misses a tick re-folds on recovery. Idempotent **upsert** keyed on +`(Source, Metric, Scope, ScopeKey, HourStartUtc)`. Query-time fold was rejected +because it doesn't reduce SQL scan (see (a)). + +**(c) Range routing → in `KpiHistoryQueryService.GetSeriesAsync`.** New option +`RollupThresholdHours` (default 168 = 7 d): windows ≤ threshold → `GetRawSeriesAsync` +(preserves intra-minute detail on 24 h/7 d); windows > threshold → +`GetHourlySeriesAsync`. `KpiSeriesBucketer.Bucket` then runs unchanged on whichever +series (note 90 d rollup = 2,160 rows > 200, so bucketing still applies on top). + +**(d) Retention → rollups outlive raw.** Keep raw `KpiSample` at 90 d (existing +purge). Retain `KpiRollupHourly` much longer (new `RollupRetentionDays`, default +365) via a **second purge pass** mirroring the 1-hour-sliced batch DELETE in +`PurgeOlderThanAsync`. Validator must enforce `RollupRetentionDays ≥ RetentionDays`. +This delivers *both* read-speed and history depth; without it the feature only +buys speed. + +**(e) Backfill → one-shot fold at rollout.** Fold existing ≤90 d of `KpiSample` +into `KpiRollupHourly` once (data migration or idempotent startup fold), so +30 d/90 d charts aren't blank until enough wall-clock passes. The idempotent +upsert key makes a re-run safe. + +**Cross-cutting correctness — per-metric aggregation.** Classify each charted +metric as **gauge** (store last-value-per-hour) or **rate** (store sum-per-hour); +store `Value` plus `Min`/`Max`/`Count` so the fold is faithful and future avg/min/ +max charts are unblocked. A metric→kind classification lives next to the +`KpiMetrics` catalog. Rate metrics folded with `last-value` would silently +undercount long-range totals — this must be explicit, not incidental. + +## 4. Task breakdown + +### Task 1 — `KpiRollupHourly` entity + EF config + migration `[standard]` +- **Files (new):** `src/ZB.MOM.WW.ScadaBridge.Commons/Entities/Kpi/KpiRollupHourly.cs`, + `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Configurations/KpiRollupHourlyEntityTypeConfiguration.cs`, + a new `Migrations/*AddKpiRollupHourlyTable.cs` (append after `20260710…`). +- **Files (edit):** `.../ConfigurationDatabase/ScadaBridgeDbContext.cs` (DbSet). +- Columns: 4-tuple series key (same varchar sizes as `KpiSample`) + `HourStartUtc` + (`datetime2`, hour-truncated UTC) + `Value` (folded aggregate) + `MinValue` + + `MaxValue` + `SampleCount`. Unique index on + `(Source, Metric, Scope, ScopeKey, HourStartUtc)` (upsert key + covers range + reads); secondary index on `(HourStartUtc)` for purge. Non-partitioned, + `[PRIMARY]`, no DB-role restriction (mirror `KpiSample`). **Build before + scaffolding the migration** (repo gotcha: `--no-build` emits an empty migration). + +### Task 2 — Metric aggregation classification `[small]` +- **Files:** `src/ZB.MOM.WW.ScadaBridge.Commons/Types/Kpi/KpiMetrics.cs` (or a new + `KpiMetricAggregation.cs` beside it). +- Add a `Gauge`/`Rate` classification and a lookup by `(Source, Metric)` covering + all charted metrics across the four sources. Rate: `deliveredLastInterval`, + `failedLastInterval`, `totalEventsLastHour`, `errorEventsLastHour`, + `scriptErrors`, `alarmEvalErrors`, `deadLetters`, `eventLogWriteFailures`. + Gauge: everything else (`queueDepth`, `buffered`, `stuck`, `parked*`, + `oldestPendingAgeSeconds`, `connectionsUp/Down`, `deployedInstances`, …). + Unknown metric → default `Gauge` (last-value) + log, so a new metric can't crash + the fold. Unit-test the classification is total over the current catalog. + +### Task 3 — Rollup fold repository method `[standard]` +- **Files:** `src/ZB.MOM.WW.ScadaBridge.Commons/Interfaces/Repositories/IKpiHistoryRepository.cs`, + `src/ZB.MOM.WW.ScadaBridge.ConfigurationDatabase/Repositories/KpiHistoryRepository.cs` +- `FoldHourlyRollupsAsync(DateTime fromHourUtc, DateTime toHourUtc, ct)`: + group `KpiSample` in the window by `(series, hour)`, apply per-metric + aggregation (Task 2) → `Value`, plus `MinValue`/`MaxValue`/`SampleCount`; + **idempotent upsert** into `KpiRollupHourly` on the unique key. Skip the current + (incomplete) hour. Handle the conditionally-emitted metric (missing rows in some + minutes) gracefully. +- `GetHourlySeriesAsync(source, metric, scope, scopeKey, fromUtc, toUtc, ct)` → + `IReadOnlyList` (project `HourStartUtc`→`CapturedAtUtc`, + `Value`), ordered ascending — same contract as `GetRawSeriesAsync` so the + bucketer is agnostic. +- `PurgeRollupsOlderThanAsync(before, ct)` — 1-hour-sliced batch DELETE (mirror + `PurgeOlderThanAsync`). + +### Task 4 — Recorder hourly tick + rollup purge `[standard]` +- **Files:** `src/ZB.MOM.WW.ScadaBridge.KpiHistory/KpiHistoryRecorderActor.cs`, + `.../KpiHistoryOptions.cs`, `.../KpiHistoryOptionsValidator.cs` +- New Akka timer `RollupTick` at `RollupInterval` (default 1 h): per-tick scope → + `FoldHourlyRollupsAsync(now − RollupLookbackHours, now)`. Re-fold lookback + (default 3 h) makes failover-missed hours self-heal via the idempotent upsert. + `_rollupInFlight` guard (mirror `_sampleInFlight`). Extend the daily purge tick + to also call `PurgeRollupsOlderThanAsync(now − RollupRetentionDays)`. +- Options: `RollupInterval` (1 h), `RollupLookbackHours` (3), `RollupRetentionDays` + (365), `RollupThresholdHours` (168). Validator: all `> 0`; `RollupRetentionDays` + bound `[1, 3650]` **and ≥ `RetentionDays`**; `RollupThresholdHours ≥ 24`. + +### Task 5 — Query service range routing `[small]` +- **Files:** `src/ZB.MOM.WW.ScadaBridge.CentralUI/Services/KpiHistoryQueryService.cs` +- In `GetSeriesAsync`: if `(toUtc − fromUtc).TotalHours > RollupThresholdHours`, + call `GetHourlySeriesAsync`, else `GetRawSeriesAsync`; then `Bucket` as today. + Both ctors (production `IServiceScopeFactory`, test `IKpiHistoryRepository`) + keep working. Unit-test the routing boundary (168 h raw vs 169 h rollup). + +### Task 6 — Backfill existing samples once `[standard]` +- **Files:** startup fold in Host (grep `AkkaHostedService` KPI wiring) OR a data + migration. +- One idempotent `FoldHourlyRollupsAsync(now − RetentionDays, now)` at first + startup after deploy (guarded so it runs once / is cheap to re-run). Log rows + folded. Prefer the startup fold over a data-migration so it reuses Task 3 logic. + +### Task 7 — Add 30 d / 90 d windows to the four surfaces `[standard]` +- **Files:** `.../Pages/Audit/AuditLogPage.razor.cs`, + `.../Pages/SiteCalls/SiteCallsReport.razor.cs`, + `.../Pages/Notifications/NotificationKpis.razor`, + `.../Pages/Monitoring/Health.razor` +- Add 30 d (720 h) and 90 d (2160 h) toggle buttons alongside the existing 24 h / + 7 d. No fetch-call changes — they already pass `fromUtc/toUtc` to the query + service, which now transparently routes to rollups for the long windows. This is + what actually *exercises* the rollup path (without it, rollups have no caller). + +### Task 8 — Tests + docs `[standard]` +- **Files:** `tests/.../Commons.Tests/*` (bucketer unchanged; classification; + fold aggregation math — especially rate-sum vs gauge-last), + `tests/.../ConfigurationDatabase.Tests/*` (fold upsert idempotency, purge, + `GetHourlySeriesAsync`), `tests/.../KpiHistory.Tests/*` (rollup tick + purge + + in-flight guard + failover re-fold), `tests/.../CentralUI.Tests/*` (query + routing; new window buttons). +- Docs: `docs/requirements/Component-KpiHistory.md` (schema section for the rollup + table, retention section, query section: raw≤threshold / rollup>threshold, new + windows), `docs/plans/2026-06-17-m6-kpi-history-design.md` (mark downsampling + deferral delivered), and move register #22 to **Resolved**. + +## 5. Risks / call-outs +- **Rate-metric undercount** — the plan's biggest correctness risk; Task 2's + classification + Task 3's per-metric fold must be reviewed together. A rate + metric folded as last-value silently misreports long-range totals. +- **Failover-missed hour** — the lookback re-fold + idempotent upsert is the + backstop; verify a simulated missed tick self-heals. +- **Retention coherence** — `RollupRetentionDays ≥ RetentionDays` must be + validator-enforced or long-range charts get holes where raw is purged but + rollups weren't yet written. +- **Migration hygiene** — build before `migrations add`; delete empty scaffolds + (repo gotcha). +- **No new infra** — like `KpiSample`, the rollup table lives in central MS SQL, + non-partitioned; no new services. + +## 6. Explicitly out of scope +- Multi-resolution rollups (daily/weekly) — hourly only; revisit if 1-year+ + windows appear. +- Charting `Min`/`Max`/`Avg` bands — the columns are stored to unblock it, but the + `KpiTrendChart` still plots a single series in this plan. +- Changing live-sample cadence or the point-in-time KPI reads (unaffected).