fix(historian): #683 review — bound the StartEnd pick joins to the window; usable picks re-check usability
- SqlServer StartEnd join-back: every pick join also carries the window predicate (forward >= @start AND < @end, reverse > @start AND <= @end). Keyed on the pick time alone a hash/merge/adaptive join read the tag's whole history (review: 43,657 vs 361 logical reads), and parameter sniffing reused that plan for every tag. Forced OPTION (HASH JOIN), 1d@1h on 6M rows: 152,524 -> 6,756 reads. - Usable picks (x/y) re-check the exact u predicate in the join, so a concurrent HistoryUpdate Replace that turned the picked sample Bad/Empty reads as absent (BadNoData), never a Bad row's value under Delta. - StartEndJoinBackTests (integration): 300k history rows either side of a 1 h window, forced HASH and MERGE joins, results identical to the unhinted plan and Start/End correct, session logical reads < 1,000 (183 bounded; 8,055 with the bound removed — the test fails then). - TagIndexTests: the refused-ADD-INDEX test asserts the Warning naming the ALTER. - Historian.md: re-measured table incl. long-history windows (output identical by EXCEPT both ways in all 7 cases) and the forced-join read counts.
This commit is contained in:
+23
-10
@@ -1607,19 +1607,32 @@ Every query filters `NOT deleted`; every value goes through `StoredSample`, so t
|
||||
the five-reference form on long windows. It is now **aggregate, then join back on the key**: one `GROUP BY` computes the counts and each
|
||||
bucket's four pick *times* (`MIN`/`MAX(SourceTimeUtc)`, any quality and usable), and each pick's columns are fetched by a seek on the
|
||||
clustered key `(TagId, SourceTimeUtc)`. Output is identical (`EXCEPT` both ways empty on 6 M rows, forward and reverse); measured on
|
||||
6 M rows / 30 days, one tag, average of 3 runs over 2 rounds:
|
||||
6 M rows / 30 days, one tag (so a 1 d or 7 d window leaves up to 5.8 M rows of the tag's history outside it), average of 3 runs
|
||||
over 2 rounds, plans compiled by the 30 d read and reused (sniffed) by the rest:
|
||||
|
||||
| Window | windowed form | join-back |
|
||||
| Window | windowed form (before) | join-back |
|
||||
|---|---|---|
|
||||
| 30 d @ 1 h (720 buckets) | 18.6–31.7 s | 3.4 s |
|
||||
| 30 d @ 1 h, reverse | 18.1–29.9 s | 3.4–5.2 s |
|
||||
| 7 d @ 1 d (7 buckets) | 5.2–6.1 s | 0.79–0.80 s |
|
||||
| 1 d @ 1 min (1440 buckets) | 0.31–0.40 s | 0.16 s |
|
||||
| 1 d @ 7 min, reverse, sub-µs start | 0.46–0.47 s | 0.12 s |
|
||||
| 30 d @ 1 h (720 buckets) | 19.0–21.7 s | 3.4–3.5 s |
|
||||
| 30 d @ 1 h, reverse | 24.7–30.3 s | 3.4 s |
|
||||
| 7 d @ 1 d (7 buckets) | 5.5–6.0 s | 0.78–0.81 s |
|
||||
| 7 d @ 1 min (10,080 buckets) | 2.1–2.3 s | 1.08–1.11 s |
|
||||
| 1 d @ 1 h (24 buckets) | 0.55–0.60 s | 0.12–0.13 s |
|
||||
| 1 d @ 1 h, reverse | 0.52–0.55 s | 0.11 s |
|
||||
| 1 d @ 1 min (1440 buckets) | 0.29–0.32 s | 0.16 s |
|
||||
|
||||
(The dev SQL container is slow in absolute terms — the ratio is the point.) ⚠️ The seeks are separate reads of the table within one
|
||||
statement, so a sample **deleted** by HistoryUpdate between the aggregate and its seek leaves that pick NULL while the counts include
|
||||
it: the null-tolerant reader below answers that bucket's Start/End/Delta `BadNoData`, transiently. Min/MaxActualTime
|
||||
⚠️ **Every pick join is bounded to the window** (`@start`/`@end`, the same predicate as the window CTE), not only keyed on the pick
|
||||
time. Keyed alone, the optimizer may choose a hash/merge/adaptive join whose inner input is the tag's **whole history**, and
|
||||
parameter sniffing then reuses that plan for every tag — a review measured 43,657 logical reads vs 361 for a 1 d @ 1 h read of a tag
|
||||
with 3 M rows outside the window. Forcing that shape here, 1 d @ 1 h: **152,524** session logical reads unbounded vs **6,756** bounded
|
||||
under `OPTION (HASH JOIN)` (6,655 under `MERGE JOIN`; the unforced plan reads 1,846 either way). `StartEndJoinBackTests` pins it:
|
||||
a tag with 300 k rows either side of a 1 h window, forced hash and merge joins, < 1,000 reads (183 bounded, 8,055 with the bound
|
||||
removed). Wider windows trade reads for time — 7 d @ 1 min reads 174 k pages (four key seeks per bucket) against the windowed form's
|
||||
46 k, yet runs in half the time. (The dev SQL container is slow in absolute terms — the ratios are the point.)
|
||||
|
||||
⚠️ The seeks are separate reads of the table within one statement, so a sample **deleted** by HistoryUpdate between the aggregate
|
||||
and its seek leaves that pick NULL while the counts include it: the null-tolerant reader below answers that bucket's Start/End/Delta
|
||||
`BadNoData`, transiently. A usable pick's join also re-checks usability (the exact `u` predicate), so a HistoryUpdate **Replace**
|
||||
that turned the picked sample Bad/Empty in between reads the same way — never a Bad row's value under Delta. Min/MaxActualTime
|
||||
keep their counts + filtered-rank form on both backends: every single-scan variant measured 1.2–2.4× slower, and QuestDB's
|
||||
`arg_max` cannot express the earliest-time tie-break. So, on both backends, a row whose picks are absent anyway maps to "no such
|
||||
sample" — `BadNoData` for that interval, never an exception and never a value-less Good. `LiveWriteReadRaceStressTests` (QuestDB
|
||||
|
||||
Reference in New Issue
Block a user