perf(historian): index the QuestDB sample tag column; read the data range before the buckets (#683)

QuestDB: {p}sample.tag is now SYMBOL INDEX. The processed-read data-range
lookups (EarliestSample / LatestSample) were partition scans of the unindexed
symbol; on 53.6M rows / 120 tags / 61 daily partitions the indexed seeks are
1-1.5 ms warm / 2-4 ms cold whatever the tag's history (unindexed: up to 11 ms
warm, 14 ms cold; cost grew with the table). The bootstrapper adds the index to a
pre-#683 table (ALTER ... ADD INDEX, a WAL txn, idempotent, ~3 s for 68M rows);
AutoCreateSchema=false only logs the statement. Disk +~12%; write apply time
within noise; main per-sample/raw/time-weighted queries unchanged or faster.

Rejected by measurement/analysis: bounded existence probes (still full scans
when a side is absent), LATEST ON (slowest unindexed), per-tag first/last side
table (cannot track TTL drops or deletes INSERT-only).

Both backends now read the data range before the bucket statement: an append
racing the read reaches the buckets but not the range, so it can no longer
produce a Good Count of 0 for an interval that has samples.

Tests: TagIndexTests (new-table index, bound-param seek uses the index,
tombstones excluded / outage marker counted, pre-#683 migration with identical
processed results across 10 aggregates x 4 windows x both directions).
This commit is contained in:
Joseph Doherty
2026-10-12 09:40:28 -04:00
parent 197246c3c1
commit 549d2d68b3
9 changed files with 331 additions and 32 deletions
@@ -450,7 +450,9 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource
/// the SDK: window <c>(end, start]</c>, buckets stepping back from <c>start</c>, the partial one earliest, each stamped
/// with its later edge, returned latest first. One statement per node returns the <see cref="BucketRow"/> statistics
/// of each non-empty bucket (bucket index <c>(cast(ts AS LONG)*10 − BucketBaseUnixTicks) / IntervalTicks</c>, never
/// <c>SAMPLE BY</c>); a second fetches the tag's earliest sample for the start-of-data Partial bit; and
/// <c>SAMPLE BY</c>); before it, when <see cref="AggregateEvaluator.NeedsDataRange"/>, two tag-index seeks fetch the
/// tag's data range (earliest / latest live sample) for the start/end-of-data Partial bits and Count's in-range
/// empty interval; and
/// <see cref="AggregateEvaluator.EvaluateAll"/> produces the values and fills the empty buckets with <c>BadNoData</c>. The time-weighted family (#671) instead streams the window plus four bounding
/// rows into <see cref="TimeWeightedAggregator"/>, exactly as the SQL Server backend does.
/// </remarks>
@@ -488,6 +490,20 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource
_ => QuestDbAggregateQueries.ActualTime(sample, aggregate == HistoryAggregateType.MaximumActualTime, plan.Reversed),
};
await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false);
// The tag's data range (earliest + latest live sample): the start/end-of-data Partial rules (#677/#680 —
// any per-sample aggregate but Average, either direction) and Count's in-range empty interval (#680); skipped
// otherwise. Index seeks on tag (#683). Read BEFORE the bucket statement, which is a separate table reader:
// a sample appended in between (the common race — a live tag read up to now) then reaches the buckets but
// not the range — at worst a stale end-of-data Partial or a BadNoData where a 0 belongs — instead of a
// range covering a bucket the rows do not show, which reported a Good Count of 0 for an interval that has
// samples. (A concurrent HistoryUpdate delete is the mirror case and can still do that, transiently.)
if (AggregateEvaluator.NeedsDataRange(aggregate, plan))
{
earliest = await ReadBoundAsync(conn, QuestDbAggregateQueries.EarliestSample(sample), fullReference,
plan.EndUtc, cancellationToken).ConfigureAwait(false);
latest = await ReadBoundAsync(conn, QuestDbAggregateQueries.LatestSample(sample), fullReference,
plan.StartUtc, cancellationToken).ConfigureAwait(false);
}
await using (var cmd = _conn.CreateCommand(conn, sql))
{
QuestDbConnection.BindLong(cmd, plan.BucketBaseUnixTicks);
@@ -499,16 +515,6 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource
while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false))
rows.Add(ReadBucket(reader, family));
}
// The tag's data range (earliest + latest live sample): the start/end-of-data Partial rules (#677/#680 —
// any per-sample aggregate but Average, either direction) and Count's in-range empty interval (#680); skipped
// otherwise.
if (AggregateEvaluator.NeedsDataRange(aggregate, plan))
{
earliest = await ReadBoundAsync(conn, QuestDbAggregateQueries.EarliestSample(sample), fullReference,
plan.EndUtc, cancellationToken).ConfigureAwait(false);
latest = await ReadBoundAsync(conn, QuestDbAggregateQueries.LatestSample(sample), fullReference,
plan.StartUtc, cancellationToken).ConfigureAwait(false);
}
}
var result = AggregateEvaluator.EvaluateAll(aggregate, plan, rows, earliest, latest, now);