perf(historian): #683 SqlServer StartEnd join-back on PK

Start/End/Delta: one GROUP BY computes the counts and each bucket's four pick
times (MIN/MAX SourceTimeUtc, any quality and usable); each pick's columns are
then fetched by a seek on the clustered key (TagId, SourceTimeUtc). Replaces the
windowed ROW_NUMBER form (live-gate 8 F1), whose four window sorts spilled.

Output identical (EXCEPT both ways empty, 6M rows, forward + reverse incl. a
sub-microsecond reverse start). 6M rows / 30 days, avg of 3 x 2 rounds:
  30d@1h      18.6-31.7 s -> 3.4 s      (reverse 18.1-29.9 s -> 3.4-5.2 s)
  7d@1d        5.2-6.1 s  -> 0.8 s
  1d@1min     0.31-0.40 s -> 0.16 s     (1d@7min reverse 0.46 -> 0.12 s)

The seeks are separate reads within one statement: under READ COMMITTED a
HistoryUpdate delete between the aggregate and the seek nulls that pick; the
null-tolerant reader (2ceed561) makes that bucket's Start/End/Delta BadNoData,
transiently. Unit test pins it; documented in the query remarks + Historian.md.
QuestDB single-scan StartEnd unchanged.
This commit is contained in:
Joseph Doherty
2026-10-12 10:33:28 -04:00
parent cc0cc8cd89
commit 910befc84c
3 changed files with 66 additions and 25 deletions
@@ -76,6 +76,25 @@ public sealed class BucketRowMappingTests
AggregateEvaluator.Evaluate(agg, b, T, T).StatusCode.ShouldBe(NoDataCalc);
}
[Fact]
public void A_pick_nulled_by_a_delete_racing_the_key_join_is_no_data_for_that_end_only()
{
// #683 join-back: the bucket's first sample (usable, so both first picks) was deleted between the GROUP BY and its
// key seek — the counts still include it, its pick columns are NULL; the last picks joined normally.
var t = StartEndTable();
var row = Counts(t, 0, 2);
row["k1"] = (byte)1; row["n1"] = 20d; row["t1"] = T.AddSeconds(1);
row["at1"] = T.AddSeconds(1); row["ak1"] = (byte)1; row["an1"] = 20d; row["st1"] = 0;
t.Rows.Add(row);
var b = Map(t, AggregateFamily.StartEnd);
b.FirstAny.ShouldBeNull();
b.First.Kind.ShouldBe(HistorianValueKind.Empty);
AggregateEvaluator.Evaluate(HistoryAggregateType.Start, b, T, T).StatusCode.ShouldBe(NoDataCalc);
AggregateEvaluator.Evaluate(HistoryAggregateType.Delta, b, T, T).StatusCode.ShouldBe(NoDataCalc);
AggregateEvaluator.Evaluate(HistoryAggregateType.End, b, T, T).Value.ShouldBe(20d);
}
[Fact]
public void A_consistent_row_still_maps_every_pick()
{