dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

perf(historian): #683 SqlServer StartEnd join-back on PK

Browse Source 
Start/End/Delta: one GROUP BY computes the counts and each bucket's four pick
times (MIN/MAX SourceTimeUtc, any quality and usable); each pick's columns are
then fetched by a seek on the clustered key (TagId, SourceTimeUtc). Replaces the
windowed ROW_NUMBER form (live-gate 8 F1), whose four window sorts spilled.

Output identical (EXCEPT both ways empty, 6M rows, forward + reverse incl. a
sub-microsecond reverse start). 6M rows / 30 days, avg of 3 x 2 rounds:
  30d@1h      18.6-31.7 s -> 3.4 s      (reverse 18.1-29.9 s -> 3.4-5.2 s)
  7d@1d        5.2-6.1 s  -> 0.8 s
  1d@1min     0.31-0.40 s -> 0.16 s     (1d@7min reverse 0.46 -> 0.12 s)

The seeks are separate reads within one statement: under READ COMMITTED a
HistoryUpdate delete between the aggregate and the seek nulls that pick; the
null-tolerant reader (2ceed561) makes that bucket's Start/End/Delta BadNoData,
transiently. Unit test pins it; documented in the query remarks + Historian.md.
QuestDB single-scan StartEnd unchanged.
This commit is contained in:
Joseph Doherty
2026-10-12 10:33:28 -04:00
parent cc0cc8cd89
commit 910befc84c
3 changed files with 66 additions and 25 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Historian.md
+18 -2
View File
@@ -1602,8 +1602,24 @@ Every query filters `NOT deleted`; every value goes through `StoredSample`, so t
query now reads the table **once**: one `GROUP BY` with `first()`/`last()` for the first/last sample of any quality and
`arg_min`/`arg_max` keyed on the usable rows' time for the first/last usable one (~15 % faster than before; the usable picks carry
no `str_value` — `arg_min` has no VARCHAR form, and only Delta reads them, never with a usable string in the bucket). On `SqlServer`
(READ COMMITTED gives no cross-reference guarantee either, though it was never observed there) Start/End/Delta was reshaped the same
way with a windowed pass + conditional `MAX` — 1.4–3× *faster* than the old five-reference form on a 1 M-row tag. Min/MaxActualTime
(READ COMMITTED gives no cross-reference guarantee either, though it was never observed there) Start/End/Delta was first reshaped
into a windowed pass + conditional `MAX`; **#683 replaced that** — its four `ROW_NUMBER` sorts spilled and it measured *slower* than
the five-reference form on long windows. It is now **aggregate, then join back on the key**: one `GROUP BY` computes the counts and each
bucket's four pick *times* (`MIN`/`MAX(SourceTimeUtc)`, any quality and usable), and each pick's columns are fetched by a seek on the
clustered key `(TagId, SourceTimeUtc)`. Output is identical (`EXCEPT` both ways empty on 6 M rows, forward and reverse); measured on
6 M rows / 30 days, one tag, average of 3 runs over 2 rounds:
| Window | windowed form | join-back |
|---|---|---|
| 30 d @ 1 h (720 buckets) | 18.6–31.7 s | 3.4 s |
| 30 d @ 1 h, reverse | 18.1–29.9 s | 3.4–5.2 s |
| 7 d @ 1 d (7 buckets) | 5.2–6.1 s | 0.79–0.80 s |
| 1 d @ 1 min (1440 buckets) | 0.31–0.40 s | 0.16 s |
| 1 d @ 7 min, reverse, sub-µs start | 0.46–0.47 s | 0.12 s |
(The dev SQL container is slow in absolute terms — the ratio is the point.) ⚠️ The seeks are separate reads of the table within one
statement, so a sample **deleted** by HistoryUpdate between the aggregate and its seek leaves that pick NULL while the counts include
it: the null-tolerant reader below answers that bucket's Start/End/Delta `BadNoData`, transiently. Min/MaxActualTime
keep their counts + filtered-rank form on both backends: every single-scan variant measured 1.2–2.4× slower, and QuestDB's
`arg_max` cannot express the earliest-time tie-break. So, on both backends, a row whose picks are absent anyway maps to "no such
sample" — `BadNoData` for that interval, never an exception and never a value-less Good. `LiveWriteReadRaceStressTests` (QuestDB
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlAggregateQueries.cs
+29 -23
View File
@@ -78,36 +78,42 @@ internal static class SqlAggregateQueries
/// StatusCode</c>) and <c>22..27</c> the last (same order) — present for every returned (non-empty) bucket.
/// </summary>
/// <remarks>
/// <b>One scan (live-gate 8 F1).</b> The counts and all four picked samples come from one reference to <c>s</c>:
/// a windowed pass ranks each row within its bucket (<c>rfa</c>/<c>rla</c> over every sample, <c>rfu</c>/<c>rlu</c>
/// within its usable / not-usable partition) and the outer <c>GROUP BY</c> both counts and lifts each picked row's
/// columns with a conditional <c>MAX</c> (one row per bucket matches each pick). The previous form referenced the
/// CTE five times; SQL Server expands each reference into its own scan, and under READ COMMITTED nothing makes those
/// scans see the same rows, so a sample committed between two of them could be counted but not picked (QuestDB hit
/// exactly that). It is also fewer scans of the key range.
/// <para>
/// <b>Aggregate, then join back on the key (#683).</b> One <c>GROUP BY</c> over the window computes the counts and
/// each bucket's four pick TIMES — <c>MIN</c>/<c>MAX(SourceTimeUtc)</c> over every sample (any quality) and over
/// the usable ones — and each pick's columns are then fetched by a seek on the clustered key
/// <c>(TagId, SourceTimeUtc)</c> (unique, so one row per pick; a NULL usable time joins nothing ⇒ NULL columns,
/// as before). Output is identical to the previous windowed form (one <c>ROW_NUMBER</c> pass + conditional
/// <c>MAX</c>, live-gate 8 F1) — <c>EXCEPT</c> both ways is empty on 6M rows, forward and reverse — without that
/// form's four window sorts, which spilled: on 6M rows / 30 days, 30 d @ 1 h ~3.4 s vs 18–32 s, 7 d @ 1 d
/// ~0.8 s vs 5.2–6.1 s, 1 d @ 1 min ~0.16 s vs 0.31–0.40 s (docs/Historian.md).
/// </para>
/// <para>
/// ⚠️ The key seeks are separate reads of the table within one statement, so under READ COMMITTED a sample
/// deleted (HistoryUpdate DeleteRaw/DeleteAtTime) between the aggregate and its seek leaves that pick NULL while
/// the counts still include it. The reader maps a NULL pick to "absent" (<c>ReadStoredOrNull</c>, live-gate 8 F1)
/// and the evaluator answers that bucket's Start/End/Delta <c>BadNoData</c> — transient, until the next read. A
/// sample replaced in between is picked with its new value (same time); a sample inserted in between is in
/// neither half (the pick times come from the aggregate).
/// </para>
/// </remarks>
/// <param name="sample">Schema-qualified <c>Sample</c> table name.</param>
/// <returns>The SQL text.</returns>
/// <param name="reverse">True for a reverse read (window <c>(@start, @end]</c>).</param>
public static string StartEnd(string sample, bool reverse = false)
{
static string Pick(string when, bool any) =>
(any ? $"MAX(CASE WHEN {when} THEN SourceTimeUtc END), " : "") +
$"MAX(CASE WHEN {when} THEN ValueKind END), MAX(CASE WHEN {when} THEN NumValue END), " +
$"MAX(CASE WHEN {when} THEN IntValue END), MAX(CASE WHEN {when} THEN StrValue END), " +
(any ? $"MAX(CASE WHEN {when} THEN StatusCode END)" : $"MAX(CASE WHEN {when} THEN SourceTimeUtc END)");
// A pick's columns from its joined row, in the established order: usable picks (kind, num, int, str, time), any-
// quality picks (time, kind, num, int, str, status).
static string Usable(string a) => $"{a}.ValueKind, {a}.NumValue, {a}.IntValue, {a}.StrValue, {a}.SourceTimeUtc";
static string Any(string a) => $"{a}.SourceTimeUtc, {a}.ValueKind, {a}.NumValue, {a}.IntValue, {a}.StrValue, {a}.StatusCode";
string Join(string a, string time) => $"LEFT JOIN {sample} {a} ON {a}.TagId = @id AND {a}.SourceTimeUtc = st.{time} ";
return Classified(sample, reverse) +
", w AS (SELECT b, SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode, u, g, " +
"ROW_NUMBER() OVER (PARTITION BY b ORDER BY SourceTimeUtc ASC) AS rfa, " +
"ROW_NUMBER() OVER (PARTITION BY b ORDER BY SourceTimeUtc DESC) AS rla, " +
"ROW_NUMBER() OVER (PARTITION BY b, u ORDER BY SourceTimeUtc ASC) AS rfu, " +
"ROW_NUMBER() OVER (PARTITION BY b, u ORDER BY SourceTimeUtc DESC) AS rlu FROM s) " +
"SELECT b, " + CommonCounts() + ", " +
Pick("u = 1 AND rfu = 1", any: false) + ", " +
Pick("u = 1 AND rlu = 1", any: false) + ", " +
Pick("rfa = 1", any: true) + ", " +
Pick("rla = 1", any: true) + " " +
"FROM w GROUP BY b ORDER BY b;";
", st AS (SELECT b, " + CommonCounts() + ", " +
"MIN(CASE WHEN u = 1 THEN SourceTimeUtc END) AS fu, MAX(CASE WHEN u = 1 THEN SourceTimeUtc END) AS lu, " +
"MIN(SourceTimeUtc) AS fa, MAX(SourceTimeUtc) AS la FROM s GROUP BY b) " +
"SELECT st.b, st.TotalCount, st.UsableCount, st.GoodCount, st.BadCount, st.NonNumericCount, " +
Usable("x") + ", " + Usable("y") + ", " + Any("p") + ", " + Any("q") + " FROM st " +
Join("x", "fu") + Join("y", "lu") + Join("p", "fa") + Join("q", "la") + "ORDER BY st.b;";
}
/// <summary>
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/BucketRowMappingTests.cs
+19
View File
@@ -76,6 +76,25 @@ public sealed class BucketRowMappingTests
AggregateEvaluator.Evaluate(agg, b, T, T).StatusCode.ShouldBe(NoDataCalc);
}
[Fact]
public void A_pick_nulled_by_a_delete_racing_the_key_join_is_no_data_for_that_end_only()
{
// #683 join-back: the bucket's first sample (usable, so both first picks) was deleted between the GROUP BY and its
// key seek — the counts still include it, its pick columns are NULL; the last picks joined normally.
var t = StartEndTable();
var row = Counts(t, 0, 2);
row["k1"] = (byte)1; row["n1"] = 20d; row["t1"] = T.AddSeconds(1);
row["at1"] = T.AddSeconds(1); row["ak1"] = (byte)1; row["an1"] = 20d; row["st1"] = 0;
t.Rows.Add(row);
var b = Map(t, AggregateFamily.StartEnd);
b.FirstAny.ShouldBeNull();
b.First.Kind.ShouldBe(HistorianValueKind.Empty);
AggregateEvaluator.Evaluate(HistoryAggregateType.Start, b, T, T).StatusCode.ShouldBe(NoDataCalc);
AggregateEvaluator.Evaluate(HistoryAggregateType.Delta, b, T, T).StatusCode.ShouldBe(NoDataCalc);
AggregateEvaluator.Evaluate(HistoryAggregateType.End, b, T, T).Value.ShouldBe(20d);
}
[Fact]
public void A_consistent_row_still_maps_every_pick()
{