fix(historian): #683 review — bound the StartEnd pick joins to the window; usable picks re-check usability

- SqlServer StartEnd join-back: every pick join also carries the window predicate
  (forward >= @start AND < @end, reverse > @start AND <= @end). Keyed on the pick
  time alone a hash/merge/adaptive join read the tag's whole history (review:
  43,657 vs 361 logical reads), and parameter sniffing reused that plan for every
  tag. Forced OPTION (HASH JOIN), 1d@1h on 6M rows: 152,524 -> 6,756 reads.
- Usable picks (x/y) re-check the exact u predicate in the join, so a concurrent
  HistoryUpdate Replace that turned the picked sample Bad/Empty reads as absent
  (BadNoData), never a Bad row's value under Delta.
- StartEndJoinBackTests (integration): 300k history rows either side of a 1 h
  window, forced HASH and MERGE joins, results identical to the unhinted plan and
  Start/End correct, session logical reads < 1,000 (183 bounded; 8,055 with the
  bound removed — the test fails then).
- TagIndexTests: the refused-ADD-INDEX test asserts the Warning naming the ALTER.
- Historian.md: re-measured table incl. long-history windows (output identical by
  EXCEPT both ways in all 7 cases) and the forced-join read counts.
This commit is contained in:
Joseph Doherty
2026-10-12 11:10:30 -04:00
parent 910befc84c
commit 4a3b12564a
4 changed files with 185 additions and 15 deletions
+23 -10
View File
@@ -1607,19 +1607,32 @@ Every query filters `NOT deleted`; every value goes through `StoredSample`, so t
the five-reference form on long windows. It is now **aggregate, then join back on the key**: one `GROUP BY` computes the counts and each
bucket's four pick *times* (`MIN`/`MAX(SourceTimeUtc)`, any quality and usable), and each pick's columns are fetched by a seek on the
clustered key `(TagId, SourceTimeUtc)`. Output is identical (`EXCEPT` both ways empty on 6 M rows, forward and reverse); measured on
6 M rows / 30 days, one tag, average of 3 runs over 2 rounds:
6 M rows / 30 days, one tag (so a 1 d or 7 d window leaves up to 5.8 M rows of the tag's history outside it), average of 3 runs
over 2 rounds, plans compiled by the 30 d read and reused (sniffed) by the rest:
| Window | windowed form | join-back |
| Window | windowed form (before) | join-back |
|---|---|---|
| 30 d @ 1 h (720 buckets) | 18.6–31.7 s | 3.4 s |
| 30 d @ 1 h, reverse | 18.1–29.9 s | 3.4–5.2 s |
| 7 d @ 1 d (7 buckets) | 5.2–6.1 s | 0.79–0.80 s |
| 1 d @ 1 min (1440 buckets) | 0.31–0.40 s | 0.16 s |
| 1 d @ 7 min, reverse, sub-µs start | 0.46–0.47 s | 0.12 s |
| 30 d @ 1 h (720 buckets) | 19.0–21.7 s | 3.4–3.5 s |
| 30 d @ 1 h, reverse | 24.7–30.3 s | 3.4 s |
| 7 d @ 1 d (7 buckets) | 5.5–6.0 s | 0.78–0.81 s |
| 7 d @ 1 min (10,080 buckets) | 2.1–2.3 s | 1.08–1.11 s |
| 1 d @ 1 h (24 buckets) | 0.55–0.60 s | 0.12–0.13 s |
| 1 d @ 1 h, reverse | 0.52–0.55 s | 0.11 s |
| 1 d @ 1 min (1440 buckets) | 0.29–0.32 s | 0.16 s |
(The dev SQL container is slow in absolute terms — the ratio is the point.) ⚠️ The seeks are separate reads of the table within one
statement, so a sample **deleted** by HistoryUpdate between the aggregate and its seek leaves that pick NULL while the counts include
it: the null-tolerant reader below answers that bucket's Start/End/Delta `BadNoData`, transiently. Min/MaxActualTime
⚠️ **Every pick join is bounded to the window** (`@start`/`@end`, the same predicate as the window CTE), not only keyed on the pick
time. Keyed alone, the optimizer may choose a hash/merge/adaptive join whose inner input is the tag's **whole history**, and
parameter sniffing then reuses that plan for every tag — a review measured 43,657 logical reads vs 361 for a 1 d @ 1 h read of a tag
with 3 M rows outside the window. Forcing that shape here, 1 d @ 1 h: **152,524** session logical reads unbounded vs **6,756** bounded
under `OPTION (HASH JOIN)` (6,655 under `MERGE JOIN`; the unforced plan reads 1,846 either way). `StartEndJoinBackTests` pins it:
a tag with 300 k rows either side of a 1 h window, forced hash and merge joins, < 1,000 reads (183 bounded, 8,055 with the bound
removed). Wider windows trade reads for time — 7 d @ 1 min reads 174 k pages (four key seeks per bucket) against the windowed form's
46 k, yet runs in half the time. (The dev SQL container is slow in absolute terms — the ratios are the point.)
⚠️ The seeks are separate reads of the table within one statement, so a sample **deleted** by HistoryUpdate between the aggregate
and its seek leaves that pick NULL while the counts include it: the null-tolerant reader below answers that bucket's Start/End/Delta
`BadNoData`, transiently. A usable pick's join also re-checks usability (the exact `u` predicate), so a HistoryUpdate **Replace**
that turned the picked sample Bad/Empty in between reads the same way — never a Bad row's value under Delta. Min/MaxActualTime
keep their counts + filtered-rank form on both backends: every single-scan variant measured 1.2–2.4× slower, and QuestDB's
`arg_max` cannot express the earliest-time tie-break. So, on both backends, a row whose picks are absent anyway maps to "no such
sample" — `BadNoData` for that interval, never an exception and never a value-less Good. `LiveWriteReadRaceStressTests` (QuestDB