fix(historian): #683 live gate 9 F1 — force an exact-key seek for every StartEnd pick
The window-bounded pick joins (4a3b1256) let the optimizer choose, under
understated statistics, a nested-loops RANGE seek over the window with the pick
time as a residual, once per bucket (O(buckets x window rows)); the cached plan
was then reused for wide reads: gate 9 1d@1min 20 s, here 47 s / 3.6M reads.
Each pick join now carries WITH (FORCESEEK (PK_Sample (TagId, SourceTimeUtc))):
an exact-key seek on both columns per bucket under any statistics, and no
hash/merge join over the tag's history can exist (forcing one is error 8622), so
the window bound is dropped from the join. Output identical (EXCEPT both ways,
fwd/rev, incl. a mixed Double/Int64/Boolean/String/DateTime/outage tag).
Measured (6M-row tag + 86k mixed-kinds tag; ms / logical reads), understated
stats: 1d@1min 47,223/3.6M -> 253/25.6k; mixed 1d@1min 20,164/1.66M -> 137/24k;
accurate stats: 1d@1min 121 -> 160 ms, 1d@1h 79 -> 117 ms. Rejected: OUTER
APPLY TOP(1) FORCESEEK (identical), OPTION (RECOMPILE) (still 20 s under the bad
stats, and a compile per read).
StartEndJoinBackTests rewritten: understated statistics + compile-small/reuse-wide
(18,645 reads; the merged query read 1,645,845 / 23 s and failed it), accurate
statistics with 300k history rows either side (22,616), forced HASH/MERGE
refused (8622 — also failed against the merged query).
Docs: perf table + explanation; SDK-parity N2 (*ActualTime Calculated bit) and
N3 (Good String => BadAggregateInvalidInputs, SDK skips it).
This commit is contained in:
+38
-20
@@ -1337,6 +1337,12 @@ bounding values back to the source integer type, so its `Interpolative` returns
|
||||
`TimeAverage`/`TimeAverage2`/`Total`/`Total2` differ from ours by the truncated bound values (~0.002–0.02 on the gate's data). Ours
|
||||
always computes and returns `Double` (as stated above for every aggregate but Count and Start/End), which is the exact Part 13
|
||||
value; the parity datasets are all `Double`, so this one is not masked there but recorded here.
|
||||
Two more, found by live gate 9 (both kept deliberately): **Minimum/MaximumActualTime carry the Calculated bit on every interval**
|
||||
(`0x…401`, `0x…405` with Partial) where the SDK returns the picked raw sample's own status (`0x…000`, `0x…404`) — the same value and
|
||||
timestamp, and the same treatment as the Min/Max-at-interval-start case above; masked in `SdkAggregateParityTests` (class remarks).
|
||||
And an interval holding a **Good String (or DateTime) sample** is `BadAggregateInvalidInputs` for Average/Minimum/Maximum/Range/
|
||||
*ActualTime here (the documented #684 refusal rule — a Good non-numeric value is not a number to aggregate), while the SDK skips the
|
||||
string and serves the numeric samples (e.g. Average 4 Good); an Uncertain non-numeric sample refuses nothing on either side.
|
||||
|
||||
**Pushdown trade-off.** Unlike the per-sample aggregates, the time-weighted family is **not** pushed down to SQL: the backend streams
|
||||
the window's rows plus the four bounds (one batch of five result sets on SQL Server; a bounds `UNION ALL` + a window select on QuestDB)
|
||||
@@ -1606,28 +1612,40 @@ Every query filters `NOT deleted`; every value goes through `StoredSample`, so t
|
||||
into a windowed pass + conditional `MAX`; **#683 replaced that** — its four `ROW_NUMBER` sorts spilled and it measured *slower* than
|
||||
the five-reference form on long windows. It is now **aggregate, then join back on the key**: one `GROUP BY` computes the counts and each
|
||||
bucket's four pick *times* (`MIN`/`MAX(SourceTimeUtc)`, any quality and usable), and each pick's columns are fetched by a seek on the
|
||||
clustered key `(TagId, SourceTimeUtc)`. Output is identical (`EXCEPT` both ways empty on 6 M rows, forward and reverse); measured on
|
||||
6 M rows / 30 days, one tag (so a 1 d or 7 d window leaves up to 5.8 M rows of the tag's history outside it), average of 3 runs
|
||||
over 2 rounds, plans compiled by the 30 d read and reused (sniffed) by the rest:
|
||||
clustered key `(TagId, SourceTimeUtc)` — **forced** with `WITH (FORCESEEK (PK_Sample (TagId, SourceTimeUtc)))`, so every pick is a
|
||||
nested-loops seek on both key columns, one row per bucket per pick, whatever the statistics say. The optimizer's own choices were
|
||||
both pathological, and plans are cached and reused across reads and tags: keyed on the pick time alone, a hash/merge join whose inner
|
||||
input was the tag's **whole history** (#683 review: 43,657 logical reads vs 361 for a 1 d @ 1 h read of a tag with 3 M rows outside the
|
||||
window); bounded to the window as well, a nested-loops **range** seek over the window with the pick time as a residual, once per
|
||||
bucket — O(buckets × window rows) — compiled whenever the statistics understate the table (a fresh or freshly bulk-loaded
|
||||
historian, ascending-key estimates for recent windows) and then reused for a wide read (**live gate 9 F1**: 1 d @ 1 min 20 s). With
|
||||
the hint neither shape exists: a forced `HASH`/`MERGE JOIN` is refused (error 8622). ⚠️ The hint names the primary key `PK_Sample`
|
||||
created by `001_initial.sql`; a hand-built schema must keep that name.
|
||||
|
||||
| Window | windowed form (before) | join-back |
|
||||
|---|---|---|
|
||||
| 30 d @ 1 h (720 buckets) | 19.0–21.7 s | 3.4–3.5 s |
|
||||
| 30 d @ 1 h, reverse | 24.7–30.3 s | 3.4 s |
|
||||
| 7 d @ 1 d (7 buckets) | 5.5–6.0 s | 0.78–0.81 s |
|
||||
| 7 d @ 1 min (10,080 buckets) | 2.1–2.3 s | 1.08–1.11 s |
|
||||
| 1 d @ 1 h (24 buckets) | 0.55–0.60 s | 0.12–0.13 s |
|
||||
| 1 d @ 1 h, reverse | 0.52–0.55 s | 0.11 s |
|
||||
| 1 d @ 1 min (1440 buckets) | 0.29–0.32 s | 0.16 s |
|
||||
Output is identical to every earlier form (`EXCEPT` both ways empty, forward and reverse, including a tag of mixed Double/Int64/
|
||||
Boolean/String/DateTime/outage samples). Measured on 6 M rows / 30 days for one tag (a 1 d or 7 d window leaves up to 5.8 M rows of
|
||||
that tag outside it) plus an 86 k-row mixed-kinds tag; elapsed ms / session logical reads, each variant's plan compiled on its first
|
||||
read and reused by the rest:
|
||||
|
||||
⚠️ **Every pick join is bounded to the window** (`@start`/`@end`, the same predicate as the window CTE), not only keyed on the pick
|
||||
time. Keyed alone, the optimizer may choose a hash/merge/adaptive join whose inner input is the tag's **whole history**, and
|
||||
parameter sniffing then reuses that plan for every tag — a review measured 43,657 logical reads vs 361 for a 1 d @ 1 h read of a tag
|
||||
with 3 M rows outside the window. Forcing that shape here, 1 d @ 1 h: **152,524** session logical reads unbounded vs **6,756** bounded
|
||||
under `OPTION (HASH JOIN)` (6,655 under `MERGE JOIN`; the unforced plan reads 1,846 either way). `StartEndJoinBackTests` pins it:
|
||||
a tag with 300 k rows either side of a 1 h window, forced hash and merge joins, < 1,000 reads (183 bounded, 8,055 with the bound
|
||||
removed). Wider windows trade reads for time — 7 d @ 1 min reads 174 k pages (four key seeks per bucket) against the windowed form's
|
||||
46 k, yet runs in half the time. (The dev SQL container is slow in absolute terms — the ratios are the point.)
|
||||
| Window | before #683 (windowed) | #683 as merged, accurate stats | #683 as merged, understated stats ‡ | **now (FORCESEEK)**, accurate | **now**, understated ‡ |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 d @ 1 min (1440 buckets) | 329 / 6.7 k | 121 / 14.9 k | **47,223 / 3.6 M** | 160 / 25.0 k | 253 / 25.6 k |
|
||||
| 1 d @ 1 min, reverse | 371 / 6.6 k | 129 / 14.9 k | **46,806 / 3.6 M** | 167 / 25.0 k | 246 / 25.6 k |
|
||||
| 1 d @ 1 h (24 buckets) | 578 / 5.4 k | 79 / 5.5 k | 984 / 63 k | 117 / 1.8 k | 206 / 3.6 k |
|
||||
| 7 d @ 1 min (10,080 buckets) | 3,426 / 103 k | 1,019 / 92 k | (hours — not run) | 1,472 / 180 k | 1,819 / 179 k |
|
||||
| 30 d @ 1 h (720 buckets) † | 35,339 / 322 k | 1,487 / 191 k | (hours — not run) | 5,720 / 73 k | 6,921 / 101 k |
|
||||
| mixed kinds 1 d @ 1 min | 190 / 5.9 k | 92 / 12.0 k | **20,164 / 1.66 M** | 107 / 24.4 k | 137 / 24.0 k |
|
||||
| mixed kinds 1 d @ 7 min, reverse | 199 / 1.5 k | 71 / 3.9 k | 2,977 / 238 k | 59 / 4.2 k | 93 / 4.7 k |
|
||||
|
||||
‡ `UPDATE STATISTICS … WITH ROWCOUNT = 34, PAGECOUNT = 1`, then the plan compiled on a 1 h @ 1 min read and reused — gate 9's
|
||||
recipe. † The 30 d row is dominated by the window `GROUP BY`, not the picks: compiled on the 30 d read itself, the merged query and
|
||||
the hinted one both take ~6.9 s (serial, a small memory grant, spills); the merged query's 1.5 s above is a plan compiled on the
|
||||
1 d read and reused. Alternatives measured and rejected: the same seek written as `OUTER APPLY (SELECT TOP (1) … WITH (FORCESEEK
|
||||
…))` (identical plans and numbers — the join form is shorter); `OPTION (RECOMPILE)` on the merged query (compiles under the bad
|
||||
statistics on every call: still 20.4 s / 1.66 M reads on the mixed-kinds tag, and a compile per read). `StartEndJoinBackTests`
|
||||
pins it — one sample a second over a day with 300 k rows of history either side: under understated statistics 18,645 reads
|
||||
(the merged query: **1,645,845 reads, 23 s**, so the test failed against it), with accurate statistics 22,616, and forced
|
||||
`HASH`/`MERGE` joins refused with 8622. (The dev SQL container is slow in absolute terms — the ratios are the point.)
|
||||
|
||||
⚠️ The seeks are separate reads of the table within one statement, so a sample **deleted** by HistoryUpdate between the aggregate
|
||||
and its seek leaves that pick NULL while the counts include it: the null-tolerant reader below answers that bucket's Start/End/Delta
|
||||
|
||||
Reference in New Issue
Block a user