perf(historian): #683 SqlServer StartEnd join-back on PK
Start/End/Delta: one GROUP BY computes the counts and each bucket's four pick
times (MIN/MAX SourceTimeUtc, any quality and usable); each pick's columns are
then fetched by a seek on the clustered key (TagId, SourceTimeUtc). Replaces the
windowed ROW_NUMBER form (live-gate 8 F1), whose four window sorts spilled.
Output identical (EXCEPT both ways empty, 6M rows, forward + reverse incl. a
sub-microsecond reverse start). 6M rows / 30 days, avg of 3 x 2 rounds:
30d@1h 18.6-31.7 s -> 3.4 s (reverse 18.1-29.9 s -> 3.4-5.2 s)
7d@1d 5.2-6.1 s -> 0.8 s
1d@1min 0.31-0.40 s -> 0.16 s (1d@7min reverse 0.46 -> 0.12 s)
The seeks are separate reads within one statement: under READ COMMITTED a
HistoryUpdate delete between the aggregate and the seek nulls that pick; the
null-tolerant reader (2ceed561) makes that bucket's Start/End/Delta BadNoData,
transiently. Unit test pins it; documented in the query remarks + Historian.md.
QuestDB single-scan StartEnd unchanged.
This commit is contained in:
+18
-2
@@ -1602,8 +1602,24 @@ Every query filters `NOT deleted`; every value goes through `StoredSample`, so t
|
||||
query now reads the table **once**: one `GROUP BY` with `first()`/`last()` for the first/last sample of any quality and
|
||||
`arg_min`/`arg_max` keyed on the usable rows' time for the first/last usable one (~15 % faster than before; the usable picks carry
|
||||
no `str_value` — `arg_min` has no VARCHAR form, and only Delta reads them, never with a usable string in the bucket). On `SqlServer`
|
||||
(READ COMMITTED gives no cross-reference guarantee either, though it was never observed there) Start/End/Delta was reshaped the same
|
||||
way with a windowed pass + conditional `MAX` — 1.4–3× *faster* than the old five-reference form on a 1 M-row tag. Min/MaxActualTime
|
||||
(READ COMMITTED gives no cross-reference guarantee either, though it was never observed there) Start/End/Delta was first reshaped
|
||||
into a windowed pass + conditional `MAX`; **#683 replaced that** — its four `ROW_NUMBER` sorts spilled and it measured *slower* than
|
||||
the five-reference form on long windows. It is now **aggregate, then join back on the key**: one `GROUP BY` computes the counts and each
|
||||
bucket's four pick *times* (`MIN`/`MAX(SourceTimeUtc)`, any quality and usable), and each pick's columns are fetched by a seek on the
|
||||
clustered key `(TagId, SourceTimeUtc)`. Output is identical (`EXCEPT` both ways empty on 6 M rows, forward and reverse); measured on
|
||||
6 M rows / 30 days, one tag, average of 3 runs over 2 rounds:
|
||||
|
||||
| Window | windowed form | join-back |
|
||||
|---|---|---|
|
||||
| 30 d @ 1 h (720 buckets) | 18.6–31.7 s | 3.4 s |
|
||||
| 30 d @ 1 h, reverse | 18.1–29.9 s | 3.4–5.2 s |
|
||||
| 7 d @ 1 d (7 buckets) | 5.2–6.1 s | 0.79–0.80 s |
|
||||
| 1 d @ 1 min (1440 buckets) | 0.31–0.40 s | 0.16 s |
|
||||
| 1 d @ 7 min, reverse, sub-µs start | 0.46–0.47 s | 0.12 s |
|
||||
|
||||
(The dev SQL container is slow in absolute terms — the ratio is the point.) ⚠️ The seeks are separate reads of the table within one
|
||||
statement, so a sample **deleted** by HistoryUpdate between the aggregate and its seek leaves that pick NULL while the counts include
|
||||
it: the null-tolerant reader below answers that bucket's Start/End/Delta `BadNoData`, transiently. Min/MaxActualTime
|
||||
keep their counts + filtered-rank form on both backends: every single-scan variant measured 1.2–2.4× slower, and QuestDB's
|
||||
`arg_max` cannot express the earliest-time tie-break. So, on both backends, a row whose picks are absent anyway maps to "no such
|
||||
sample" — `BadNoData` for that interval, never an exception and never a value-less Good. `LiveWriteReadRaceStressTests` (QuestDB
|
||||
|
||||
Reference in New Issue
Block a user