fix(historian): #683 review — bound the StartEnd pick joins to the window; usable picks re-check usability
- SqlServer StartEnd join-back: every pick join also carries the window predicate (forward >= @start AND < @end, reverse > @start AND <= @end). Keyed on the pick time alone a hash/merge/adaptive join read the tag's whole history (review: 43,657 vs 361 logical reads), and parameter sniffing reused that plan for every tag. Forced OPTION (HASH JOIN), 1d@1h on 6M rows: 152,524 -> 6,756 reads. - Usable picks (x/y) re-check the exact u predicate in the join, so a concurrent HistoryUpdate Replace that turned the picked sample Bad/Empty reads as absent (BadNoData), never a Bad row's value under Delta. - StartEndJoinBackTests (integration): 300k history rows either side of a 1 h window, forced HASH and MERGE joins, results identical to the unhinted plan and Start/End correct, session logical reads < 1,000 (183 bounded; 8,055 with the bound removed — the test fails then). - TagIndexTests: the refused-ADD-INDEX test asserts the Warning naming the ALTER. - Historian.md: re-measured table incl. long-history windows (output identical by EXCEPT both ways in all 7 cases) and the forced-join read counts.
This commit is contained in:
@@ -93,8 +93,15 @@ internal static class SqlAggregateQueries
|
||||
/// deleted (HistoryUpdate DeleteRaw/DeleteAtTime) between the aggregate and its seek leaves that pick NULL while
|
||||
/// the counts still include it. The reader maps a NULL pick to "absent" (<c>ReadStoredOrNull</c>, live-gate 8 F1)
|
||||
/// and the evaluator answers that bucket's Start/End/Delta <c>BadNoData</c> — transient, until the next read. A
|
||||
/// sample replaced in between is picked with its new value (same time); a sample inserted in between is in
|
||||
/// neither half (the pick times come from the aggregate).
|
||||
/// sample replaced in between is picked with its new value (same time) — except that a usable pick re-checks
|
||||
/// usability in its join (the exact <c>u</c> predicate), so a Replace that turned it Bad/Empty reads as absent
|
||||
/// (<c>BadNoData</c>), never as a Bad row's value under Delta. A sample inserted in between is in neither half
|
||||
/// (the pick times come from the aggregate).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// Every join is also bounded to the window (the same <c>@start</c>/<c>@end</c> predicate as <c>s</c>): keyed
|
||||
/// on the pick time alone, the optimizer may choose a hash/merge/adaptive join whose inner input is the tag's
|
||||
/// whole history, and parameter sniffing then reuses that plan for every tag.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="sample">Schema-qualified <c>Sample</c> table name.</param>
|
||||
@@ -106,14 +113,24 @@ internal static class SqlAggregateQueries
|
||||
// quality picks (time, kind, num, int, str, status).
|
||||
static string Usable(string a) => $"{a}.ValueKind, {a}.NumValue, {a}.IntValue, {a}.StrValue, {a}.SourceTimeUtc";
|
||||
static string Any(string a) => $"{a}.SourceTimeUtc, {a}.ValueKind, {a}.NumValue, {a}.IntValue, {a}.StrValue, {a}.StatusCode";
|
||||
string Join(string a, string time) => $"LEFT JOIN {sample} {a} ON {a}.TagId = @id AND {a}.SourceTimeUtc = st.{time} ";
|
||||
// Each seek is bounded to the window as well as keyed on the pick time: without the bounds the optimizer may pick a
|
||||
// hash/merge/adaptive join whose inner input is the tag's WHOLE history (TagId = @id only), and parameter sniffing
|
||||
// then reuses that plan for every tag (review of 910befc8: 43,657 logical reads vs 361 on a tag with 3M rows of
|
||||
// history outside a 1 d window). A usable pick also re-checks usability (the exact `u` predicate), so a sample a
|
||||
// HistoryUpdate Replace turned Bad/Empty in between joins nothing ⇒ absent ⇒ BadNoData, never a Bad row's value.
|
||||
string Window(string a) => reverse
|
||||
? $"{a}.SourceTimeUtc > @start AND {a}.SourceTimeUtc <= @end"
|
||||
: $"{a}.SourceTimeUtc >= @start AND {a}.SourceTimeUtc < @end";
|
||||
string Join(string a, string time, bool usable) =>
|
||||
$"LEFT JOIN {sample} {a} ON {a}.TagId = @id AND {a}.SourceTimeUtc = st.{time} AND {Window(a)}" +
|
||||
(usable ? $" AND {a}.StatusCode >= 0 AND {a}.ValueKind <> 0" : "") + " ";
|
||||
return Classified(sample, reverse) +
|
||||
", st AS (SELECT b, " + CommonCounts() + ", " +
|
||||
"MIN(CASE WHEN u = 1 THEN SourceTimeUtc END) AS fu, MAX(CASE WHEN u = 1 THEN SourceTimeUtc END) AS lu, " +
|
||||
"MIN(SourceTimeUtc) AS fa, MAX(SourceTimeUtc) AS la FROM s GROUP BY b) " +
|
||||
"SELECT st.b, st.TotalCount, st.UsableCount, st.GoodCount, st.BadCount, st.NonNumericCount, " +
|
||||
Usable("x") + ", " + Usable("y") + ", " + Any("p") + ", " + Any("q") + " FROM st " +
|
||||
Join("x", "fu") + Join("y", "lu") + Join("p", "fa") + Join("q", "la") + "ORDER BY st.b;";
|
||||
Join("x", "fu", usable: true) + Join("y", "lu", usable: true) + Join("p", "fa", usable: false) + Join("q", "la", usable: false) + "ORDER BY st.b;";
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
|
||||
Reference in New Issue
Block a user