fix(historian): live-gate 8 F1 — per-sample counts and picks from one table version (#680 follow-up)

QuestDB Start/End/Delta reads of a tag being written intermittently failed
BadHistoryOperationUnsupported (InvalidCastException: Column 'ts_3' is null):
the StartEnd query referenced its CTE four times and QuestDB opens a table
reader per reference, so a row committed between two of them was counted but
not picked. MinimumActualTime/MaximumActualTime had the same race and returned
a value-less Good.

(a) SQL
- QuestDB StartEnd: one scan, one GROUP BY — first()/last() for the
  any-quality picks, arg_min/arg_max keyed on the usable rows' time for the
  usable picks (no str_value: arg_min has no VARCHAR form; only Delta reads
  them). Byte-identical output to the old query on a 1M-row tag, ~15% faster.
- SqlServer StartEnd: one windowed pass + conditional MAX (was five CTE
  references); identical output, 1.4-3x faster.
- ActualTime kept on both backends: every single-scan form measured 1.2-2.4x
  slower; its inconsistency is absorbed by (b).

(b) Defence in depth on both backends
- ReadBucket takes DbDataReader and reads absent picks as null
  (ReadStoredOrNull) instead of throwing.
- AggregateEvaluator: Delta with a missing first/last usable pick and
  ActualTime with a missing extreme are BadNoData (were BadAggregateInvalidInputs
  / a value-less Good); Start/End with no pick already were.
- Unit tests (Sql.Common evaluator, QuestDb + SqlServer mappers over a
  DataTableReader) fail with the guards removed.

LiveWriteReadRaceStressTests (QuestDB integration, opt-in via
HISTORIAN_QUESTDB_STRESS_SECONDS, Category=SlowUnit): single-row writer vs
4 tight-loop readers. Old code: 6 faults in ~105k reads; new: 0 in 106k.

Docs: Historian.md QuestDB processed-read note; F2 (Min/Max on the interval
start: SDK raw, ours Calculated) and F3 (integer tags: SDK casts interpolated
values to the integer type, ours Double) added to the SDK-parity deviations.
This commit is contained in:
Joseph Doherty
2026-10-11 18:28:29 -04:00
parent 11f792538e
commit 2ceed561ad
13 changed files with 549 additions and 54 deletions
@@ -25,6 +25,12 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// sample is not usable.
/// </para>
/// <para>
/// <b>Inconsistent rows</b> (live-gate 8 F1): a row whose counts say a bucket holds samples while a picked sample
/// (first/last of any quality, first/last usable, the extreme) is absent is answered from what is present —
/// Start/End/Delta/ActualTime without the sample they need are <c>BadNoData</c> — never an exception. The backends'
/// queries read counts and picks in one scan so this should not arise; the evaluator does not rely on it.
/// </para>
/// <para>
/// Values: Count is <see cref="int"/>; Start/End keep the sample's type; every other aggregate is
/// <see cref="double"/> computed over the numeric value (Boolean = 0/1). These are the simple per-sample
/// definitions; the Part 13 time-weighted aggregates (TimeAverage, TimeAverage2, Total, Total2, Interpolative)
@@ -76,12 +82,20 @@ public static class AggregateEvaluator
case HistoryAggregateType.Delta:
// #680 (SDK ComputeDelta): last usable − first usable; Uncertain only when the interval's first or last
// sample was skipped (not usable) — a Bad sample in the middle does not degrade it.
// Live-gate 8 F1: a row whose counts promise a usable sample its picks do not carry (a backend that read
// the counts and the picks from two table versions) has no consistent first/last — no data, not a throw
// and not a wrong-input verdict.
if (r.First.Kind == HistorianValueKind.Empty || r.Last.Kind == HistorianValueKind.Empty)
return NoData(bucketStart, now);
if (!r.First.TryGetDouble(out var a) || !r.Last.TryGetDouble(out var b))
return Calc(null, HistorianStatusCodes.BadAggregateInvalidInputs);
var bounded = r.FirstAny is { IsUsable: true } && r.LastAny is { IsUsable: true };
return Calc(b - a, bounded ? HistorianStatusCodes.Good : HistorianStatusCodes.UncertainDataSubNormal);
case HistoryAggregateType.MinimumActualTime:
case HistoryAggregateType.MaximumActualTime:
// Live-gate 8 F1: usable numeric samples counted but no extreme picked (an inconsistent row) is no data,
// never a value-less Good.
if (r.Extreme is null || r.ExtremeTime is null) return NoData(bucketStart, now);
return Calc(r.Extreme, status, r.ExtremeTime);
default:
throw new HistoryAggregateNotSupportedException(aggregate);