fix(historian): #677 review — start-of-data Partial forward-only, bounded/skipped earliest-sample query, wider SDK parity
F1: the SDK applies its m_startOfData Partial rule to forward reads only; ours now does too (AggregateEvaluator.EvaluateAll + TimeWeightedAggregator). New reverse late-start parity tests through the SDK oracle (unit) and real SQL (conformance, both backends). F2: AggregateEvaluator.NeedsEarliestSample — the earliest-sample query runs only for forward reads of aggregates other than Average, and is bounded by the window end (SqlServer < @end, QuestDB ts < $2). F3: per-sample SDK parity extended to Range, Start, End, Delta, Min/Max ActualTime and PercentGood/Bad (oracle maps StatusAggregateCalculator); pre-existing definitional differences masked and marked #680. Docs: Historian.md Partial/SDK-parity sections, CLAUDE.md bullet.
This commit is contained in:
+17
-10
@@ -918,7 +918,7 @@ point (see [Continuation-point paging (Events)](#continuation-point-paging-event
|
||||
> 80 s → 0 s read at 20 s used to answer `[60,80) [40,60) [20,40) [0,20)` stamped 60/40/20/0; it now answers
|
||||
> `(60,80] (40,60] (20,40] (0,20]` stamped 80/60/40/20 — a sample exactly on an interval edge now belongs to the interval it *ends*, a
|
||||
> sample at `start` is included and one at `end` is not (the same window a reverse raw read uses). Forward reads are unchanged apart
|
||||
> from the new start-of-data Partial bit below. A client that paired reverse results with forward ones by timestamp must shift by one
|
||||
> from the new (forward-only) start-of-data Partial bit below. A client that paired reverse results with forward ones by timestamp must shift by one
|
||||
> interval.
|
||||
|
||||
**Bucketing.** Forward: one result per interval of `[start, end)`; bucket *i* = `[start + i·interval, start + (i+1)·interval)`,
|
||||
@@ -933,11 +933,14 @@ the later time and makes the buckets later-edge-inclusive — `BucketPlan.IndexO
|
||||
window of more than **100 000 intervals** is refused before any I/O and reported `BadHistoryOperationUnsupported` for that node. Every
|
||||
returned status carries the historian *Calculated* info bit, `BadNoData` included.
|
||||
|
||||
**Partial bit (#677).** Every aggregate except plain `Average` sets the **Partial** bit (`0x4`) on a non-Bad result for the interval the
|
||||
tag's data **starts inside** — its earliest stored sample (any quality) lies strictly between the interval's edges, so there is no
|
||||
bounding value before it and the data does not cover the whole interval. This is the SDK's `m_startOfData` rule (the SDK turns it off for
|
||||
`Average` only). Each per-sample read therefore also fetches the tag's earliest sample (one key seek). An earliest sample exactly on an
|
||||
edge, or before the interval, sets nothing. The time-weighted aggregates also keep setting Partial on the truncated interval (below).
|
||||
**Partial bit (#677).** On a **forward** read, every aggregate except plain `Average` sets the **Partial** bit (`0x4`) on a non-Bad
|
||||
result for the interval the tag's data **starts inside** — its earliest stored sample (any quality) lies strictly between the interval's
|
||||
edges, so there is no bounding value before it and the data does not cover the whole interval. This follows the SDK's `m_startOfData`
|
||||
rule, which the SDK turns off for `Average` and in effect applies to forward reads only (a reverse read is fed latest-first, so its
|
||||
"start of data" is the latest sample, past the window). A reverse read therefore sets no start-of-data Partial — matching the SDK. Only a
|
||||
forward per-sample read of an aggregate other than `Average` fetches the tag's earliest sample (one seek, bounded by the window end);
|
||||
every other read skips that query. An earliest sample exactly on an edge, or before the interval, sets nothing. The time-weighted
|
||||
aggregates also keep setting Partial on the truncated interval (below).
|
||||
|
||||
**Sample classes.** *Usable* = severity Good or Uncertain **and** a stored value (not `Empty`). *Good* = usable with severity Good.
|
||||
*Bad* = not usable.
|
||||
@@ -1035,14 +1038,18 @@ value at **10:00**, not 09:00, and no point ever falls before the window. The in
|
||||
`ProcessedConformance.Processed_reads_match_the_SDK_aggregate_calculators` (both SQL backends, through the real SQL) run the pinned SDK's
|
||||
own calculators (`AverageAggregateCalculator`, `MinMaxAggregateCalculator`, `CountAggregateCalculator`, the base `AggregateCalculator`
|
||||
for Interpolative) over the same samples, forward and reverse, and compare every interval's timestamp, value, status code and Partial
|
||||
bit. Known differences that remain — documented in the test, deliberately not changed here because forward results were to stay as
|
||||
they are — are: the SDK reports `TimeAverage`/`Total` Uncertain whenever any region of the interval is non-Good, where ours applies the
|
||||
bit. The per-sample comparison covers Average, Minimum, Maximum, Count, Range, Start, End, Delta, Minimum/MaximumActualTime and
|
||||
PercentGood/Bad; reverse reads over data that starts inside the window are compared too. Known differences that remain (tracked as
|
||||
#680) — documented and masked in the tests, deliberately not changed here because forward results were to stay as they are — are:
|
||||
`Start`/`End` (the SDK returns the raw first/last sample, of any quality, with its own timestamp and stored status; ours returns the
|
||||
first/last usable value stamped with the interval, under the per-sample status rule — the values agree wherever the SDK's is not Bad);
|
||||
`Delta` status (the SDK's stays Good with a Bad sample in the interval, ours is Uncertain); `PercentGood`/`PercentBad` (ours count
|
||||
samples, the SDK measures duration — different definitions, so only timestamp and status are compared); the SDK reports `TimeAverage`/`Total` Uncertain whenever any region of the interval is non-Good, where ours applies the
|
||||
time-based rule (a fully-covered interval bridged across a Bad sample is Good here); `Total` over an interval the data starts inside is
|
||||
`TimeAverage × interval` here but the integral of the covered part in the SDK; ours flags a truncated interval Partial, the SDK does
|
||||
not; the SDK's forward end-of-data Partial rule is not implemented; the SDK's per-sample Count answers 0/Good for an empty interval
|
||||
inside the data and its per-sample aggregates use Good samples only; and on a reverse `Interpolative` read the SDK marks a sample exactly
|
||||
at a point Interpolated (Uncertain if a Bad sample follows) where ours — like the SDK forward — returns the raw sample. On a reverse read
|
||||
the SDK's own start-of-data rule keys on the *latest* sample (the first one it is fed); ours keys on the earliest in both directions.
|
||||
at a point Interpolated (Uncertain if a Bad sample follows) where ours — like the SDK forward — returns the raw sample.
|
||||
|
||||
**Pushdown trade-off.** Unlike the per-sample aggregates, the time-weighted family is **not** pushed down to SQL: the backend streams
|
||||
the window's rows plus the four bounds (one batch of five result sets on SQL Server; a bounds `UNION ALL` + a window select on QuestDB)
|
||||
|
||||
Reference in New Issue
Block a user