Start/End/Delta: one GROUP BY computes the counts and each bucket's four pick
times (MIN/MAX SourceTimeUtc, any quality and usable); each pick's columns are
then fetched by a seek on the clustered key (TagId, SourceTimeUtc). Replaces the
windowed ROW_NUMBER form (live-gate 8 F1), whose four window sorts spilled.
Output identical (EXCEPT both ways empty, 6M rows, forward + reverse incl. a
sub-microsecond reverse start). 6M rows / 30 days, avg of 3 x 2 rounds:
30d@1h 18.6-31.7 s -> 3.4 s (reverse 18.1-29.9 s -> 3.4-5.2 s)
7d@1d 5.2-6.1 s -> 0.8 s
1d@1min 0.31-0.40 s -> 0.16 s (1d@7min reverse 0.46 -> 0.12 s)
The seeks are separate reads within one statement: under READ COMMITTED a
HistoryUpdate delete between the aggregate and the seek nulls that pick; the
null-tolerant reader (2ceed561) makes that bucket's Start/End/Delta BadNoData,
transiently. Unit test pins it; documented in the query remarks + Historian.md.
QuestDB single-scan StartEnd unchanged.
- Average/Minimum/Maximum/Range/*ActualTime refuse (BadAggregateInvalidInputs) only a GOOD
String/DateTime sample (new GoodNonNumericCount / good_non_numeric_count column); an Uncertain
one is skipped and makes the min/max family Uncertain, matching the SDK (Average skips it,
ComputeMinMax flags the uncastable value). Delta keeps its usable-sample refusal.
- ProcessedConformance: an Uncertain value above the Good maximum (pins UncMax/unc_max through
the real SQL) and an Uncertain String through the real SQL.
- The_uncertain_dataset_is_not_vacuous also proves the min/max STATUS discrimination.
Co-Authored-By: Claude <noreply@anthropic.com>
Part 13 defines Average/Minimum/Maximum/Range/*ActualTime over Good raw values
(independent of TreatUncertainAsBad) and the SDK's calculators skip Uncertain values
under the server's TreatUncertainAsBad=false. SqlServer/QuestDB value columns now use
the good flag (g) instead of usable (u); new UncMin/UncMax columns feed the Part 13
Minimum/Maximum/Range status rule (Uncertain on a Bad sample or an Uncertain value
beyond the Good extreme -- the SDK applies only the Bad half). An Uncertain-only
interval is BadNoData. Uncertain-bearing parity datasets (in-memory + real SQL) match
the SDK unmasked; the beyond-extreme divergence is pinned explicitly.
Co-Authored-By: Claude <noreply@anthropic.com>
QuestDB Start/End/Delta reads of a tag being written intermittently failed
BadHistoryOperationUnsupported (InvalidCastException: Column 'ts_3' is null):
the StartEnd query referenced its CTE four times and QuestDB opens a table
reader per reference, so a row committed between two of them was counted but
not picked. MinimumActualTime/MaximumActualTime had the same race and returned
a value-less Good.
(a) SQL
- QuestDB StartEnd: one scan, one GROUP BY — first()/last() for the
any-quality picks, arg_min/arg_max keyed on the usable rows' time for the
usable picks (no str_value: arg_min has no VARCHAR form; only Delta reads
them). Byte-identical output to the old query on a 1M-row tag, ~15% faster.
- SqlServer StartEnd: one windowed pass + conditional MAX (was five CTE
references); identical output, 1.4-3x faster.
- ActualTime kept on both backends: every single-scan form measured 1.2-2.4x
slower; its inconsistency is absorbed by (b).
(b) Defence in depth on both backends
- ReadBucket takes DbDataReader and reads absent picks as null
(ReadStoredOrNull) instead of throwing.
- AggregateEvaluator: Delta with a missing first/last usable pick and
ActualTime with a missing extreme are BadNoData (were BadAggregateInvalidInputs
/ a value-less Good); Start/End with no pick already were.
- Unit tests (Sql.Common evaluator, QuestDb + SqlServer mappers over a
DataTableReader) fail with the guards removed.
LiveWriteReadRaceStressTests (QuestDB integration, opt-in via
HISTORIAN_QUESTDB_STRESS_SECONDS, Category=SlowUnit): single-row writer vs
4 tight-loop readers. Old code: 6 faults in ~105k reads; new: 0 in 106k.
Docs: Historian.md QuestDB processed-read note; F2 (Min/Max on the interval
start: SDK raw, ours Calculated) and F3 (integer tags: SDK casts interpolated
values to the integer type, ours Double) added to the SDK-parity deviations.
Joseph Doherty