fix(historian): #683 review — bound the StartEnd pick joins to the window; usable picks re-check usability

- SqlServer StartEnd join-back: every pick join also carries the window predicate
  (forward >= @start AND < @end, reverse > @start AND <= @end). Keyed on the pick
  time alone a hash/merge/adaptive join read the tag's whole history (review:
  43,657 vs 361 logical reads), and parameter sniffing reused that plan for every
  tag. Forced OPTION (HASH JOIN), 1d@1h on 6M rows: 152,524 -> 6,756 reads.
- Usable picks (x/y) re-check the exact u predicate in the join, so a concurrent
  HistoryUpdate Replace that turned the picked sample Bad/Empty reads as absent
  (BadNoData), never a Bad row's value under Delta.
- StartEndJoinBackTests (integration): 300k history rows either side of a 1 h
  window, forced HASH and MERGE joins, results identical to the unhinted plan and
  Start/End correct, session logical reads < 1,000 (183 bounded; 8,055 with the
  bound removed — the test fails then).
- TagIndexTests: the refused-ADD-INDEX test asserts the Warning naming the ALTER.
- Historian.md: re-measured table incl. long-history windows (output identical by
  EXCEPT both ways in all 7 cases) and the forced-join read counts.
This commit is contained in:
Joseph Doherty
2026-10-12 11:10:30 -04:00
parent 910befc84c
commit 4a3b12564a
4 changed files with 185 additions and 15 deletions
@@ -93,8 +93,15 @@ internal static class SqlAggregateQueries
/// deleted (HistoryUpdate DeleteRaw/DeleteAtTime) between the aggregate and its seek leaves that pick NULL while
/// the counts still include it. The reader maps a NULL pick to "absent" (<c>ReadStoredOrNull</c>, live-gate 8 F1)
/// and the evaluator answers that bucket's Start/End/Delta <c>BadNoData</c> — transient, until the next read. A
/// sample replaced in between is picked with its new value (same time); a sample inserted in between is in
/// neither half (the pick times come from the aggregate).
/// sample replaced in between is picked with its new value (same time) — except that a usable pick re-checks
/// usability in its join (the exact <c>u</c> predicate), so a Replace that turned it Bad/Empty reads as absent
/// (<c>BadNoData</c>), never as a Bad row's value under Delta. A sample inserted in between is in neither half
/// (the pick times come from the aggregate).
/// </para>
/// <para>
/// Every join is also bounded to the window (the same <c>@start</c>/<c>@end</c> predicate as <c>s</c>): keyed
/// on the pick time alone, the optimizer may choose a hash/merge/adaptive join whose inner input is the tag's
/// whole history, and parameter sniffing then reuses that plan for every tag.
/// </para>
/// </remarks>
/// <param name="sample">Schema-qualified <c>Sample</c> table name.</param>
@@ -106,14 +113,24 @@ internal static class SqlAggregateQueries
// quality picks (time, kind, num, int, str, status).
static string Usable(string a) => $"{a}.ValueKind, {a}.NumValue, {a}.IntValue, {a}.StrValue, {a}.SourceTimeUtc";
static string Any(string a) => $"{a}.SourceTimeUtc, {a}.ValueKind, {a}.NumValue, {a}.IntValue, {a}.StrValue, {a}.StatusCode";
string Join(string a, string time) => $"LEFT JOIN {sample} {a} ON {a}.TagId = @id AND {a}.SourceTimeUtc = st.{time} ";
// Each seek is bounded to the window as well as keyed on the pick time: without the bounds the optimizer may pick a
// hash/merge/adaptive join whose inner input is the tag's WHOLE history (TagId = @id only), and parameter sniffing
// then reuses that plan for every tag (review of 910befc8: 43,657 logical reads vs 361 on a tag with 3M rows of
// history outside a 1 d window). A usable pick also re-checks usability (the exact `u` predicate), so a sample a
// HistoryUpdate Replace turned Bad/Empty in between joins nothing ⇒ absent ⇒ BadNoData, never a Bad row's value.
string Window(string a) => reverse
? $"{a}.SourceTimeUtc > @start AND {a}.SourceTimeUtc <= @end"
: $"{a}.SourceTimeUtc >= @start AND {a}.SourceTimeUtc < @end";
string Join(string a, string time, bool usable) =>
$"LEFT JOIN {sample} {a} ON {a}.TagId = @id AND {a}.SourceTimeUtc = st.{time} AND {Window(a)}" +
(usable ? $" AND {a}.StatusCode >= 0 AND {a}.ValueKind <> 0" : "") + " ";
return Classified(sample, reverse) +
", st AS (SELECT b, " + CommonCounts() + ", " +
"MIN(CASE WHEN u = 1 THEN SourceTimeUtc END) AS fu, MAX(CASE WHEN u = 1 THEN SourceTimeUtc END) AS lu, " +
"MIN(SourceTimeUtc) AS fa, MAX(SourceTimeUtc) AS la FROM s GROUP BY b) " +
"SELECT st.b, st.TotalCount, st.UsableCount, st.GoodCount, st.BadCount, st.NonNumericCount, " +
Usable("x") + ", " + Usable("y") + ", " + Any("p") + ", " + Any("q") + " FROM st " +
Join("x", "fu") + Join("y", "lu") + Join("p", "fa") + Join("q", "la") + "ORDER BY st.b;";
Join("x", "fu", usable: true) + Join("y", "lu", usable: true) + Join("p", "fa", usable: false) + Join("q", "la", usable: false) + "ORDER BY st.b;";
}
/// <summary>