dohertj2/lmxopcua
1
0
Fork 0
Code Issues 32 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(historian): live-gate 8 F1 — per-sample counts and picks from one table version (#680 follow-up)

Browse Source 
QuestDB Start/End/Delta reads of a tag being written intermittently failed
BadHistoryOperationUnsupported (InvalidCastException: Column 'ts_3' is null):
the StartEnd query referenced its CTE four times and QuestDB opens a table
reader per reference, so a row committed between two of them was counted but
not picked. MinimumActualTime/MaximumActualTime had the same race and returned
a value-less Good.

(a) SQL
- QuestDB StartEnd: one scan, one GROUP BY — first()/last() for the
  any-quality picks, arg_min/arg_max keyed on the usable rows' time for the
  usable picks (no str_value: arg_min has no VARCHAR form; only Delta reads
  them). Byte-identical output to the old query on a 1M-row tag, ~15% faster.
- SqlServer StartEnd: one windowed pass + conditional MAX (was five CTE
  references); identical output, 1.4-3x faster.
- ActualTime kept on both backends: every single-scan form measured 1.2-2.4x
  slower; its inconsistency is absorbed by (b).

(b) Defence in depth on both backends
- ReadBucket takes DbDataReader and reads absent picks as null
  (ReadStoredOrNull) instead of throwing.
- AggregateEvaluator: Delta with a missing first/last usable pick and
  ActualTime with a missing extreme are BadNoData (were BadAggregateInvalidInputs
  / a value-less Good); Start/End with no pick already were.
- Unit tests (Sql.Common evaluator, QuestDb + SqlServer mappers over a
  DataTableReader) fail with the guards removed.

LiveWriteReadRaceStressTests (QuestDB integration, opt-in via
HISTORIAN_QUESTDB_STRESS_SECONDS, Category=SlowUnit): single-row writer vs
4 tight-loop readers. Old code: 6 faults in ~105k reads; new: 0 in 106k.

Docs: Historian.md QuestDB processed-read note; F2 (Min/Max on the interval
start: SDK raw, ours Calculated) and F3 (integer tags: SDK casts interpolated
values to the integer type, ours Double) added to the SDK-parity deviations.
This commit is contained in:
Joseph Doherty
2026-10-11 18:28:29 -04:00
parent 11f792538e
commit 2ceed561ad
13 changed files with 549 additions and 54 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlAggregateQueries.cs
+37 -17
View File
@@ -71,33 +71,53 @@ internal static class SqlAggregateQueries
/// Family columns: <c>6..10</c> first usable sample (<c>ValueKind, NumValue, IntValue, StrValue, SourceTimeUtc</c>),
/// <c>11..15</c> last usable sample (same order) — all NULL when the bucket has no usable sample; then
/// <c>16..21</c> the first sample of any quality (<c>SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue,
/// StatusCode</c>) and <c>22..27</c> the last (same order) — never NULL for a returned (non-empty) bucket.
/// StatusCode</c>) and <c>22..27</c> the last (same order) — present for every returned (non-empty) bucket.
/// </summary>
/// <remarks>
/// <b>One scan (live-gate 8 F1).</b> The counts and all four picked samples come from one reference to <c>s</c>:
/// a windowed pass ranks each row within its bucket (<c>rfa</c>/<c>rla</c> over every sample, <c>rfu</c>/<c>rlu</c>
/// within its usable / not-usable partition) and the outer <c>GROUP BY</c> both counts and lifts each picked row's
/// columns with a conditional <c>MAX</c> (one row per bucket matches each pick). The previous form referenced the
/// CTE five times; SQL Server expands each reference into its own scan, and under READ COMMITTED nothing makes those
/// scans see the same rows, so a sample committed between two of them could be counted but not picked (QuestDB hit
/// exactly that). It is also fewer scans of the key range.
/// </remarks>
/// <param name="sample">Schema-qualified <c>Sample</c> table name.</param>
/// <returns>The SQL text.</returns>
/// <param name="reverse">True for a reverse read (window <c>(@start, @end]</c>).</param>
public static string StartEnd(string sample, bool reverse = false) =>
Classified(sample, reverse) +
", st AS (SELECT b, " + CommonCounts() + " FROM s GROUP BY b)" +
", f AS (SELECT b, ValueKind, NumValue, IntValue, StrValue, SourceTimeUtc, " +
"ROW_NUMBER() OVER (PARTITION BY b ORDER BY SourceTimeUtc ASC) AS rf, " +
"ROW_NUMBER() OVER (PARTITION BY b ORDER BY SourceTimeUtc DESC) AS rl FROM s WHERE u = 1)" +
", a AS (SELECT b, SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode, " +
"ROW_NUMBER() OVER (PARTITION BY b ORDER BY SourceTimeUtc ASC) AS rf, " +
"ROW_NUMBER() OVER (PARTITION BY b ORDER BY SourceTimeUtc DESC) AS rl FROM s) " +
"SELECT st.*, " +
"x.ValueKind, x.NumValue, x.IntValue, x.StrValue, x.SourceTimeUtc, " +
"y.ValueKind, y.NumValue, y.IntValue, y.StrValue, y.SourceTimeUtc, " +
"p.SourceTimeUtc, p.ValueKind, p.NumValue, p.IntValue, p.StrValue, p.StatusCode, " +
"q.SourceTimeUtc, q.ValueKind, q.NumValue, q.IntValue, q.StrValue, q.StatusCode " +
"FROM st LEFT JOIN f x ON x.b = st.b AND x.rf = 1 LEFT JOIN f y ON y.b = st.b AND y.rl = 1 " +
"JOIN a p ON p.b = st.b AND p.rf = 1 JOIN a q ON q.b = st.b AND q.rl = 1 ORDER BY st.b;";
public static string StartEnd(string sample, bool reverse = false)
{
static string Pick(string when, bool any) =>
(any ? $"MAX(CASE WHEN {when} THEN SourceTimeUtc END), " : "") +
$"MAX(CASE WHEN {when} THEN ValueKind END), MAX(CASE WHEN {when} THEN NumValue END), " +
$"MAX(CASE WHEN {when} THEN IntValue END), MAX(CASE WHEN {when} THEN StrValue END), " +
(any ? $"MAX(CASE WHEN {when} THEN StatusCode END)" : $"MAX(CASE WHEN {when} THEN SourceTimeUtc END)");
return Classified(sample, reverse) +
", w AS (SELECT b, SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode, u, g, " +
"ROW_NUMBER() OVER (PARTITION BY b ORDER BY SourceTimeUtc ASC) AS rfa, " +
"ROW_NUMBER() OVER (PARTITION BY b ORDER BY SourceTimeUtc DESC) AS rla, " +
"ROW_NUMBER() OVER (PARTITION BY b, u ORDER BY SourceTimeUtc ASC) AS rfu, " +
"ROW_NUMBER() OVER (PARTITION BY b, u ORDER BY SourceTimeUtc DESC) AS rlu FROM s) " +
"SELECT b, " + CommonCounts() + ", " +
Pick("u = 1 AND rfu = 1", any: false) + ", " +
Pick("u = 1 AND rlu = 1", any: false) + ", " +
Pick("rfa = 1", any: true) + ", " +
Pick("rla = 1", any: true) + " " +
"FROM w GROUP BY b ORDER BY b;";
}
/// <summary>
/// MinimumActualTime / MaximumActualTime: the usable sample with the smallest (largest) <c>NumValue</c>,
/// ties broken by the earliest <c>SourceTimeUtc</c>. Family columns: <c>6 NumValue, 7 SourceTimeUtc</c>
/// (NULL when the bucket has no usable numeric sample).
/// </summary>
/// <remarks>
/// Deliberately still two references to <c>s</c> (counts + the ranked extreme), unlike <see cref="StartEnd"/>:
/// every single-scan form measured (a <c>ROW_NUMBER</c> or windowed <c>MIN</c>/<c>MAX</c> over every row) was
/// 1.2–2× slower on day-plus windows than ranking only the filtered usable rows. Live-gate 8 F1's inconsistency —
/// a sample committed between the two scans, counted but not ranked — is instead absorbed by the reader: a
/// NULL extreme in a bucket with usable samples evaluates to <c>BadNoData</c> (never a value-less Good).
/// </remarks>
/// <param name="sample">Schema-qualified <c>Sample</c> table name.</param>
/// <param name="maximum">True for MaximumActualTime.</param>
/// <returns>The SQL text.</returns>