fix(historian): #677 review — start-of-data Partial forward-only, bounded/skipped earliest-sample query, wider SDK parity

F1: the SDK applies its m_startOfData Partial rule to forward reads only; ours
now does too (AggregateEvaluator.EvaluateAll + TimeWeightedAggregator). New
reverse late-start parity tests through the SDK oracle (unit) and real SQL
(conformance, both backends).
F2: AggregateEvaluator.NeedsEarliestSample — the earliest-sample query runs
only for forward reads of aggregates other than Average, and is bounded by
the window end (SqlServer < @end, QuestDB ts < $2).
F3: per-sample SDK parity extended to Range, Start, End, Delta, Min/Max
ActualTime and PercentGood/Bad (oracle maps StatusAggregateCalculator);
pre-existing definitional differences masked and marked #680.
Docs: Historian.md Partial/SDK-parity sections, CLAUDE.md bullet.
This commit is contained in:
Joseph Doherty
2026-10-11 11:47:22 -04:00
parent 9120182314
commit 7349260231
12 changed files with 198 additions and 41 deletions
+17 -10
View File
@@ -918,7 +918,7 @@ point (see [Continuation-point paging (Events)](#continuation-point-paging-event
> 80 s → 0 s read at 20 s used to answer `[60,80) [40,60) [20,40) [0,20)` stamped 60/40/20/0; it now answers
> `(60,80] (40,60] (20,40] (0,20]` stamped 80/60/40/20 — a sample exactly on an interval edge now belongs to the interval it *ends*, a
> sample at `start` is included and one at `end` is not (the same window a reverse raw read uses). Forward reads are unchanged apart
> from the new start-of-data Partial bit below. A client that paired reverse results with forward ones by timestamp must shift by one
> from the new (forward-only) start-of-data Partial bit below. A client that paired reverse results with forward ones by timestamp must shift by one
> interval.
**Bucketing.** Forward: one result per interval of `[start, end)`; bucket *i* = `[start + i·interval, start + (i+1)·interval)`,
@@ -933,11 +933,14 @@ the later time and makes the buckets later-edge-inclusive — `BucketPlan.IndexO
window of more than **100 000 intervals** is refused before any I/O and reported `BadHistoryOperationUnsupported` for that node. Every
returned status carries the historian *Calculated* info bit, `BadNoData` included.
**Partial bit (#677).** Every aggregate except plain `Average` sets the **Partial** bit (`0x4`) on a non-Bad result for the interval the
tag's data **starts inside** — its earliest stored sample (any quality) lies strictly between the interval's edges, so there is no
bounding value before it and the data does not cover the whole interval. This is the SDK's `m_startOfData` rule (the SDK turns it off for
`Average` only). Each per-sample read therefore also fetches the tag's earliest sample (one key seek). An earliest sample exactly on an
edge, or before the interval, sets nothing. The time-weighted aggregates also keep setting Partial on the truncated interval (below).
**Partial bit (#677).** On a **forward** read, every aggregate except plain `Average` sets the **Partial** bit (`0x4`) on a non-Bad
result for the interval the tag's data **starts inside** — its earliest stored sample (any quality) lies strictly between the interval's
edges, so there is no bounding value before it and the data does not cover the whole interval. This follows the SDK's `m_startOfData`
rule, which the SDK turns off for `Average` and in effect applies to forward reads only (a reverse read is fed latest-first, so its
"start of data" is the latest sample, past the window). A reverse read therefore sets no start-of-data Partial — matching the SDK. Only a
forward per-sample read of an aggregate other than `Average` fetches the tag's earliest sample (one seek, bounded by the window end);
every other read skips that query. An earliest sample exactly on an edge, or before the interval, sets nothing. The time-weighted
aggregates also keep setting Partial on the truncated interval (below).
**Sample classes.** *Usable* = severity Good or Uncertain **and** a stored value (not `Empty`). *Good* = usable with severity Good.
*Bad* = not usable.
@@ -1035,14 +1038,18 @@ value at **10:00**, not 09:00, and no point ever falls before the window. The in
`ProcessedConformance.Processed_reads_match_the_SDK_aggregate_calculators` (both SQL backends, through the real SQL) run the pinned SDK's
own calculators (`AverageAggregateCalculator`, `MinMaxAggregateCalculator`, `CountAggregateCalculator`, the base `AggregateCalculator`
for Interpolative) over the same samples, forward and reverse, and compare every interval's timestamp, value, status code and Partial
bit. Known differences that remain — documented in the test, deliberately not changed here because forward results were to stay as
they are — are: the SDK reports `TimeAverage`/`Total` Uncertain whenever any region of the interval is non-Good, where ours applies the
bit. The per-sample comparison covers Average, Minimum, Maximum, Count, Range, Start, End, Delta, Minimum/MaximumActualTime and
PercentGood/Bad; reverse reads over data that starts inside the window are compared too. Known differences that remain (tracked as
#680) — documented and masked in the tests, deliberately not changed here because forward results were to stay as they are — are:
`Start`/`End` (the SDK returns the raw first/last sample, of any quality, with its own timestamp and stored status; ours returns the
first/last usable value stamped with the interval, under the per-sample status rule — the values agree wherever the SDK's is not Bad);
`Delta` status (the SDK's stays Good with a Bad sample in the interval, ours is Uncertain); `PercentGood`/`PercentBad` (ours count
samples, the SDK measures duration — different definitions, so only timestamp and status are compared); the SDK reports `TimeAverage`/`Total` Uncertain whenever any region of the interval is non-Good, where ours applies the
time-based rule (a fully-covered interval bridged across a Bad sample is Good here); `Total` over an interval the data starts inside is
`TimeAverage × interval` here but the integral of the covered part in the SDK; ours flags a truncated interval Partial, the SDK does
not; the SDK's forward end-of-data Partial rule is not implemented; the SDK's per-sample Count answers 0/Good for an empty interval
inside the data and its per-sample aggregates use Good samples only; and on a reverse `Interpolative` read the SDK marks a sample exactly
at a point Interpolated (Uncertain if a Bad sample follows) where ours — like the SDK forward — returns the raw sample. On a reverse read
the SDK's own start-of-data rule keys on the *latest* sample (the first one it is fed); ours keys on the earliest in both directions.
at a point Interpolated (Uncertain if a Bad sample follows) where ours — like the SDK forward — returns the raw sample.
**Pushdown trade-off.** Unlike the per-sample aggregates, the time-weighted family is **not** pushed down to SQL: the backend streams
the window's rows plus the four bounds (one batch of five result sets on SQL Server; a bounds `UNION ALL` + a window select on QuestDB)