fix(historian): #680 lane B — per-sample Count/Start/End/Delta per Part 13 + forward end-of-data Partial
Count: Good samples only, value-based status (all Bad => generic Bad), 0/Good on an empty interval inside the tag's data range [first, last] (both directions). Start/End: the raw first/last sample of any quality (own value, status, timestamp). Delta: Uncertain only when the interval's first or last sample is not usable. Forward end-of-data Partial (SDK m_endOfData, earlier edge inclusive) for every per-sample aggregate but Average (PercentGood/Bad left to lane A). Backends push down first/last-any-quality columns and fetch the tag's data range (earliest <= window end, latest >= window start) only when NeedsDataRange. SDK parity masks for Start/End/Delta/Count removed; new parity datasets.
This commit is contained in:
+47
-15
@@ -935,6 +935,24 @@ point (see [Continuation-point paging (Events)](#continuation-point-paging-event
|
||||
> from the new (forward-only) start-of-data Partial bit below. A client that paired reverse results with forward ones by timestamp must shift by one
|
||||
> interval.
|
||||
|
||||
> ⚠️ **#680 (client-visible): `Count`, `Start`, `End`, `Delta` and the end-of-data Partial bit now follow Part 13 / the SDK** on the
|
||||
> `SqlServer` and `QuestDb` backends:
|
||||
> - **`Count`** counts **Good** samples only (it used to count Good + Uncertain); its status is value-based — Good when every sample in
|
||||
> the interval is Good, the generic **Bad** (`0x80000000`, value null) when every one is Bad (it used to be `BadNoData`), else
|
||||
> Uncertain. An **empty interval inside the tag's data range** `[first sample, last sample]` is now **`0`, Good** (it used to be
|
||||
> `BadNoData`); an empty interval wholly before the first or after the last sample stays `BadNoData`. Both directions.
|
||||
> - **`Start` / `End`** return the interval's **raw** first / last sample of **any** quality — its own value (null when Bad), its own
|
||||
> stored status (Raw: no Calculated bit) and **its own timestamp** (they used to return the first/last *usable* value stamped with
|
||||
> the interval start under the per-sample status rule). A reverse read returns the same samples: `Start` is still the earliest.
|
||||
> - **`Delta`** (last usable − first usable) is Good unless the interval's first or last sample is not usable (then Uncertain); a Bad
|
||||
> sample in the middle no longer degrades it, and neither does an Uncertain sample.
|
||||
> - **End-of-data Partial:** the interval holding the tag's **last** sample carries the Partial bit for every per-sample aggregate but
|
||||
> `Average` — forward with the interval's earlier edge included, reverse only when the sample lies strictly inside (the SDK's rules) —
|
||||
> so a "live" tag's latest interval is now flagged.
|
||||
>
|
||||
> A client that read `Count` as "usable samples", keyed `Start`/`End` on the interval start, or treated `Count = BadNoData` as "no
|
||||
> samples" must adjust. Not changed: `Average`/`Minimum`/`Maximum` still use Good + Uncertain samples (the SDK uses Good only).
|
||||
|
||||
**Bucketing.** Forward: one result per interval of `[start, end)`; bucket *i* = `[start + i·interval, start + (i+1)·interval)`,
|
||||
start-inclusive / end-exclusive to the tick, the last bucket truncated at `end`, each stamped with its start. Reverse (`end < start`,
|
||||
#677): the window is `(end, start]`; result *k* = `(start − (k+1)·interval, start − k·interval]`, clipped at `end` (the **earliest**
|
||||
@@ -951,29 +969,43 @@ returned status carries the historian *Calculated* info bit, `BadNoData` include
|
||||
result for the interval the tag's data **starts inside** — its earliest stored sample (any quality) lies strictly between the interval's
|
||||
edges, so there is no bounding value before it and the data does not cover the whole interval. This follows the SDK's `m_startOfData`
|
||||
rule, which the SDK turns off for `Average` and in effect applies to forward reads only (a reverse read is fed latest-first, so its
|
||||
"start of data" is the latest sample, past the window). A reverse read therefore sets no start-of-data Partial — matching the SDK
|
||||
(the time-weighted family does set it, on a reverse read, on the interval holding the *latest* sample when nothing follows it — #680,
|
||||
see [Time-weighted aggregates](#time-weighted-aggregates-part-13-671)). Only a
|
||||
forward per-sample read of an aggregate other than `Average` fetches the tag's earliest sample (one seek, bounded by the window end);
|
||||
every other read skips that query. An earliest sample exactly on an edge, or before the interval, sets nothing. The time-weighted
|
||||
"start of data" is the latest sample, past the window). A reverse read therefore sets no start-of-data Partial on the interval holding
|
||||
the *earliest* sample — matching the SDK. An earliest sample exactly on an edge, or before the interval, sets nothing. The time-weighted
|
||||
aggregates also keep setting Partial on the truncated interval (below).
|
||||
|
||||
**End-of-data Partial (#680).** Every aggregate except `Average` (and `Interpolative`) also sets Partial on a non-Bad result for the
|
||||
interval the tag's data **ends inside** — nothing stored after it:
|
||||
- **forward:** the latest stored sample (any quality) lies at or after the interval's earlier edge and before its later one (Part 13
|
||||
§5.3.3.2; the SDK's `m_endOfData` rule — the earlier edge is *included*);
|
||||
- **reverse:** the latest sample lies *strictly* inside the interval (the SDK's reverse read keys its `m_startOfData` rule on the first
|
||||
value it is fed — the latest — so a latest sample exactly on an interval edge sets nothing). The time-weighted family does the same
|
||||
(see [Time-weighted aggregates](#time-weighted-aggregates-part-13-671)).
|
||||
|
||||
Data continuing past the window sets nothing. Two deliberate limits, kept: the per-sample aggregates **never** set Partial on a merely
|
||||
*truncated* interval (only the time-weighted ones do), and **a Bad result never carries Partial** (the SDK sets it on Bad results too).
|
||||
|
||||
**Data-range queries.** The start/end-of-data rules and `Count`'s in-range empty interval need the tag's **data range**: its earliest
|
||||
stored sample (any quality) at or before the window's later edge and its latest at or after the earlier edge — two single-key seeks
|
||||
(SQL Server: one extra result set of two `MIN`/`MAX` subqueries; QuestDB: two `LIMIT 1` statements). They run only when the answer can
|
||||
change a result (`AggregateEvaluator.NeedsDataRange`): a per-sample read of an aggregate other than `Average` (either direction), or
|
||||
`Count`; a per-sample `Average` read skips them.
|
||||
|
||||
**Sample classes.** *Usable* = severity Good or Uncertain **and** a stored value (not `Empty`). *Good* = usable with severity Good.
|
||||
*Bad* = not usable.
|
||||
|
||||
| Aggregate | Computed over | Value timestamp |
|
||||
|---|---|---|
|
||||
| `Average`, `Minimum`, `Maximum` | `AVG` / `MIN` / `MAX` of `NumValue` over usable samples | bucket start (reverse read: its later edge, as every row marked "bucket start" here) |
|
||||
| `Count` | number of usable samples (`Int32`) | bucket start |
|
||||
| `Count` | number of **Good** samples (`Int32`; #680) | bucket start |
|
||||
| `Range` | `Maximum − Minimum` | bucket start |
|
||||
| `Start`, `End` | first / last usable sample by time (keeps the sample's type) | bucket start |
|
||||
| `Delta` | `End − Start` (numeric samples) | bucket start |
|
||||
| `Start`, `End` | the **raw** first / last sample of any quality (its type, its own status; value null when Bad — #680) | **the sample's own time** |
|
||||
| `Delta` | last usable − first usable value (numeric samples) | bucket start |
|
||||
| `MinimumActualTime`, `MaximumActualTime` | the usable sample with the smallest / largest `NumValue`; ties ⇒ earliest | **the sample's own time** |
|
||||
|
||||
Status and edge rules:
|
||||
|
||||
- An **empty interval** (no samples) ⇒ `BadNoData`. **`Count` — and every value aggregate — over an interval whose samples are all Bad ⇒ `BadNoData`**, not `0`.
|
||||
- Value aggregates are `Good` when every sample in the bucket is Good and `UncertainDataSubNormal` when the bucket also holds any Bad **or Uncertain** sample.
|
||||
- An **empty interval** (no samples) ⇒ `BadNoData` — except **`Count`**, which is `0`, Good, when the interval lies inside the tag's data range `[first sample, last sample]` (both edges inclusive, either direction; #680). **Every value aggregate over an interval whose samples are all Bad ⇒ `BadNoData`**; `Count` there is the generic **Bad** (value null).
|
||||
- Value aggregates are `Good` when every sample in the bucket is Good and `UncertainDataSubNormal` when the bucket also holds any Bad **or Uncertain** sample. `Count`'s status is value-based (all Good ⇒ Good, all Bad ⇒ Bad, else Uncertain); `Start`/`End` carry the raw sample's own status; `Delta` is Good unless the interval's first or last sample is not usable (then Uncertain) — #680.
|
||||
- `Count`, `Start`, `End` accept any value kind; the other aggregates (and `Delta`) on a bucket that holds a usable `String`/`DateTime` sample ⇒ `BadAggregateInvalidInputs` for that interval.
|
||||
- `PercentGood` / `PercentBad` are **duration-based since #680** — see [Time-weighted aggregates](#time-weighted-aggregates-part-13-671).
|
||||
- Everything except `Count` and `Start`/`End` is a `double`; `Int64` samples aggregate as doubles and `Boolean` as 0/1.
|
||||
@@ -1118,10 +1150,8 @@ Start, End, Delta and Minimum/MaximumActualTime; the time-weighted one TimeAvera
|
||||
PercentGood/Bad (#680), including datasets whose data ends inside the window and with an interval wholly inside an outage; reverse
|
||||
reads over data that starts inside the window are compared too. Known differences that remain (tracked as
|
||||
#680) — documented and masked in the tests, deliberately not changed here because forward results were to stay as they are — are:
|
||||
`Start`/`End` (the SDK returns the raw first/last sample, of any quality, with its own timestamp and stored status; ours returns the
|
||||
first/last usable value stamped with the interval, under the per-sample status rule — the values agree wherever the SDK's is not Bad);
|
||||
`Delta` status (the SDK's stays Good with a Bad sample in the interval, ours is Uncertain); the SDK's per-sample Count answers
|
||||
0/Good for an empty interval inside the data and its per-sample aggregates use Good samples only. Time-weighted differences (#680 closed
|
||||
the SDK's per-sample Average/Minimum/Maximum use Good samples only, ours Good + Uncertain; the SDK sets Partial on Bad per-sample
|
||||
results too, ours never (#680 closed the Start/End, Delta, Count and end-of-data Partial differences). Time-weighted differences (#680 closed
|
||||
the TimeAverage/Total status, `Total` coverage, end-of-data and PercentGood/Bad ones), each deliberate: ours flags a truncated interval
|
||||
Partial (Part 13 §5.3.3.2 Case 3), the SDK does not; ours carries the Calculated bit on a `BadNoData`; `TimeAverage2`/`Total2` over an
|
||||
interval wholly inside an outage are generic `Bad` here (Part 13's PercentDataBad rule), `BadNoData` in the SDK; in the end-of-data
|
||||
@@ -1370,7 +1400,9 @@ Every query filters `NOT deleted`; every value goes through `StoredSample`, so t
|
||||
- **Processed:** one query per node; bucket index `(cast(ts AS LONG) * 10 - bucketBaseUnixTicks) / intervalTicks` (never `SAMPLE BY`, whose alignment
|
||||
differs), so bucket edges are identical to `SqlServer` (forward start-inclusive / end-exclusive with the partial bucket last; reverse anchored at the
|
||||
later time, later-edge-inclusive, partial bucket earliest — #677; interval 0 ⇒ one bucket, 100,000-bucket cap). On a forward read of any aggregate
|
||||
but Average, a second `LIMIT 1` statement (capped at the window end) fetches the tag's earliest live sample for the start-of-data Partial bit.
|
||||
but Average, and for Count in either direction, two `LIMIT 1` statements fetch the tag's data range — its earliest live sample at or before the
|
||||
window's later edge and its latest at or after its earlier edge — for the start/end-of-data Partial bits and Count's in-range empty interval
|
||||
(#677/#680). Start/End/Delta also return each bucket's first and last sample of any quality (`row_number()` over every row).
|
||||
Min/max times use `row_number()` windows with the earliest-timestamp tie-break.
|
||||
|
||||
### Retention (TTL)
|
||||
|
||||
Reference in New Issue
Block a user