dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

feat(historian): #677 Part 13/SDK reverse anchoring, start-of-data Partial, AggregateConfiguration, IntegralDivisor

Browse Source 
- BucketPlan: reverse processed reads anchored at the later time for every
  aggregate: window (end, start], partial interval earliest, results stamped
  with the later edge, latest first (SDK CreateSlice/GetTimestamp/backward
  GetValues). SqlServer adds @bucketAdjustTicks + flips the window predicate;
  QuestDB binds BucketBaseUnixTicks. IndexOf is the C# statement of the math.
- Partial bit on the interval the tag's data starts inside (SDK m_startOfData),
  every aggregate but Average; per-sample reads fetch the earliest sample.
- Node manager: non-default client AggregateConfiguration =>
  BadAggregateConfigurationRejected per node (ServerAggregateConfiguration,
  pinned to the SDK AggregateManager defaults).
- GatewayTagProvisioner sends IntegralDivisor = 1.
- SDK parity oracle (Opc.Ua.Server calculators) in unit + both backends'
  conformance suites, forward and reverse; remaining divergences documented.
- docs/Historian.md + CLAUDE.md release note (client-visible).
This commit is contained in:
Joseph Doherty
2026-10-11 11:26:00 -04:00
parent 825393299e
commit 27ca6cd791
24 changed files with 1134 additions and 135 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Historian.md
+66 -21
View File
@@ -701,6 +701,12 @@ RetrievalMode.Integral`) — no client-side scaling. (This replaces the retired
likewise a native gateway mode. Bucket status codes and timestamps are preserved unchanged; empty / null
buckets surface as `BadNoData`.
**Integral unit (#677).** AVEVA's integral is `∫ value dt` (seconds) ÷ the tag's `IntegralDivisor`. The proto3 `double` default for
`HistorianTagDefinition.IntegralDivisor` is `0`, so `GatewayTagProvisioner` now sends **`IntegralDivisor = 1`** explicitly on every
`EnsureTags` — the value·seconds unit the SQL backends' `Total` uses. A tag provisioned earlier keeps its divisor until its next
`EnsureTags` (an upsert). The live magnitude check against AVEVA (a constant 1 integrated over 1 s reads 1) is still open — it needs the
VPN. Reverse processed reads on the `Gateway` backend are answered by AVEVA and are not re-anchored by #677.
### Aggregate set and `BadAggregateNotSupported` (#285)
`HistoryAggregateType` carries the wider Part 13 set (Interpolative, TimeAverage, Start, End, Delta, Range,
@@ -860,21 +866,41 @@ point (see [Continuation-point paging (Events)](#continuation-point-paging-event
> that relied on the old sum gets a different number for the same data (e.g. two samples 10 and 20 ten seconds apart no longer total
> 30). There is no switch back; `Count` × `Average` reproduces the old sum.
**Bucketing.** One result per interval of `[start, end)`; bucket *i* = `[start + i·interval, start + (i+1)·interval)`, start-inclusive /
end-exclusive to the tick, the last bucket truncated at `end`. `interval <= 0` ⇒ a single interval `[start, end)`; `start == end` ⇒ no
> ⚠️ **Release note (#677): reverse processed reads changed — client-visible.** A reverse read (`end < start`) is now anchored at the
> request's start (its *later* time) for **every** aggregate, as OPC UA Part 13 and the SDK's own `AggregateCalculator` do: intervals
> step back from `start`, the window is `(end, start]`, the truncated interval is the **earliest** one, and each result is stamped with its
> interval's **later** edge. Before #677 only `Interpolative` did this; every other aggregate swapped the window, anchored it at the
> earlier time (`[end, start)`, the partial interval at the latest end) and stamped each interval with its earlier edge. So a reverse
> 80 s → 0 s read at 20 s used to answer `[60,80) [40,60) [20,40) [0,20)` stamped 60/40/20/0; it now answers
> `(60,80] (40,60] (20,40] (0,20]` stamped 80/60/40/20 — a sample exactly on an interval edge now belongs to the interval it *ends*, a
> sample at `start` is included and one at `end` is not (the same window a reverse raw read uses). Forward reads are unchanged apart
> from the new start-of-data Partial bit below. A client that paired reverse results with forward ones by timestamp must shift by one
> interval.
**Bucketing.** Forward: one result per interval of `[start, end)`; bucket *i* = `[start + i·interval, start + (i+1)·interval)`,
start-inclusive / end-exclusive to the tick, the last bucket truncated at `end`, each stamped with its start. Reverse (`end < start`,
#677): the window is `(end, start]`; result *k* = `(start − (k+1)·interval, start − k·interval]`, clipped at `end` (the **earliest**
interval is the truncated one), stamped with its **later** edge `start − k·interval`, returned latest first. `interval <= 0` ⇒ a single interval `[start, end)`; `start == end` ⇒ no
intervals. The bucket index is computed in **100 ns ticks** (not milliseconds): `DATEDIFF_BIG(microsecond, @start, t) × 10` plus a
sub-microsecond correction from the two tick digits, divided by the interval in ticks. (`DATEDIFF` counts unit *boundaries crossed*, so a
millisecond formulation misplaces samples near a bucket edge, and a nanosecond one overflows past ≈292 years.) A window of more than
**100 000 intervals** is refused before any I/O and reported `BadHistoryOperationUnsupported` for that node. **Reverse windows**
(`end < start`) are swapped, computed forward **anchored at the earlier time**, and returned in **descending** bucket order. Every returned
status carries the historian *Calculated* info bit, `BadNoData` included.
millisecond formulation misplaces samples near a bucket edge, and a nanosecond one overflows past ≈292 years.) A reverse read adds one
constant to that elapsed-ticks dividend (`BucketPlan.BucketIndexAdjustTicks`; QuestDB folds it into its base) that re-anchors the grid at
the later time and makes the buckets later-edge-inclusive — `BucketPlan.IndexOf` is the C# statement of the math both dialects mirror. A
window of more than **100 000 intervals** is refused before any I/O and reported `BadHistoryOperationUnsupported` for that node. Every
returned status carries the historian *Calculated* info bit, `BadNoData` included.
**Partial bit (#677).** Every aggregate except plain `Average` sets the **Partial** bit (`0x4`) on a non-Bad result for the interval the
tag's data **starts inside** — its earliest stored sample (any quality) lies strictly between the interval's edges, so there is no
bounding value before it and the data does not cover the whole interval. This is the SDK's `m_startOfData` rule (the SDK turns it off for
`Average` only). Each per-sample read therefore also fetches the tag's earliest sample (one key seek). An earliest sample exactly on an
edge, or before the interval, sets nothing. The time-weighted aggregates also keep setting Partial on the truncated interval (below).
**Sample classes.** *Usable* = severity Good or Uncertain **and** a stored value (not `Empty`). *Good* = usable with severity Good.
*Bad* = not usable.
| Aggregate | Computed over | Value timestamp |
|---|---|---|
| `Average`, `Minimum`, `Maximum` | `AVG` / `MIN` / `MAX` of `NumValue` over usable samples | bucket start |
| `Average`, `Minimum`, `Maximum` | `AVG` / `MIN` / `MAX` of `NumValue` over usable samples | bucket start (reverse read: its later edge, as every row marked "bucket start" here) |
| `Count` | number of usable samples (`Int32`) | bucket start |
| `Range` | `Maximum − Minimum` | bucket start |
| `Start`, `End` | first / last usable sample by time (keeps the sample's type) | bucket start |
@@ -896,11 +922,16 @@ Status and edge rules:
`TimeAverage`, `TimeAverage2`, `Total`, `Total2` and `Interpolative`, evaluated by the shared
`Driver.Historian.Sql.Common/TimeWeightedAggregator.cs` for **both** SQL backends (so `SqlServer` and `QuestDb` agree exactly).
**This whole family — `Total` included — is evaluated in C#, not pushed down to SQL** (see *Pushdown trade-off* and the raw-row cap
below). Bucketing (edges, truncated last bucket, the 100 000-interval cap) is the same as above; reverse windows are covered below.
below). Bucketing (edges, truncated interval, reverse anchoring, the 100 000-interval cap) is the same as above.
**Server defaults, fixed:** `TreatUncertainAsBad = false`, `PercentDataBad = PercentDataGood = 100`, `UseSlopedExtrapolation = false`.
**A client's `AggregateConfiguration` in the HistoryRead request is ignored** (the node manager does not forward it); every read uses
these defaults.
**Server defaults, fixed:** `TreatUncertainAsBad = false`, `PercentDataBad = PercentDataGood = 100`, `UseSlopedExtrapolation = false`
(`ServerAggregateConfiguration` in the OPC UA server project — the SDK's own `AggregateManager` defaults, which a test pins). **A
client's `AggregateConfiguration` (#677):** `UseServerCapabilitiesDefaults = true`, or the four values spelled out equal to these defaults
⇒ served; anything else ⇒ **`BadAggregateConfigurationRejected`** for each node of the request (the source is never called). It used to
be silently ignored, so a client asking for, say, `TreatUncertainAsBad = true` got answers computed without it. The check is server-wide
(it applies on the `Gateway` backend too). A client sending an all-zero structure without `UseServerCapabilitiesDefaults` is asking for
`PercentDataBad = PercentDataGood = 0` and is refused; the SDK's default-constructed `AggregateConfiguration` sets
`UseServerCapabilitiesDefaults = true` and is served.
**Bounding values.** Besides the window `[start, end)`, each read fetches four single-row seeks: the latest *usable* sample before
`start`, the latest sample of any quality before `start` (they differ when an outage straddles the start), the first sample of any
@@ -934,9 +965,9 @@ PercentDataGood.
- No data at all ⇒ `BadNoData`.
- A usable `String`/`DateTime` sample feeding the interval ⇒ `BadAggregateInvalidInputs`.
All carry the Calculated bit; a non-Bad result for the **truncated last interval** (shorter than the processing interval) also carries
the **Partial** bit (`0x4`, `AggregateBits.Partial`). Timestamp = interval start; value `double` (Boolean 0/1). (The per-sample
aggregates do not set Partial — a follow-up.)
All carry the Calculated bit; a non-Bad result for the **truncated interval** (shorter than the processing interval — the latest one
forward, the earliest one reverse) or for the interval the tag's **data starts inside** (#677, see *Partial bit* above) also carries the
**Partial** bit (`0x4`, `AggregateBits.Partial`). Timestamp = interval start (reverse: its later edge); value `double` (Boolean 0/1).
> **A live tag's latest interval.** Past the tag's last stored sample the value is held (stepped extrapolation, Uncertain). Under
> the SDK rule that Uncertain time is Good time, so the latest interval of a live tag normally reads **`Good`** — not Uncertain —
@@ -951,11 +982,23 @@ aggregates do not set Partial — a follow-up.)
- `Good` on a segment whose defining samples are Good;
- `UncertainDataSubNormal` inside or up to an outage (bridged) and past the last sample (extrapolated).
**Reverse reads (`end < start`).** Part 13 anchors a reverse read at the request's start, which is the later time. `Interpolative`
follows that: its points are `start, start − interval, …` (in the forward window's terms `End − i·interval`), latest first. A
reverse read 10:00 → 09:00 at 1 h returns the value at **10:00**, not 09:00, and no point ever falls before the window. The interval
aggregates (TimeAverage, Total, …) keep the existing backend-wide reverse rule: computed forward with intervals anchored at the
earlier time, returned in descending order. Re-anchoring those at the later time for every aggregate is a follow-up.
**Reverse reads (`end < start`).** Part 13 anchors a reverse read at the request's start, which is the later time — for every aggregate
since #677. `Interpolative`'s points are `start, start − interval, …`, latest first: a reverse read 10:00 → 09:00 at 1 h returns the
value at **10:00**, not 09:00, and no point ever falls before the window. The interval aggregates use the intervals
`(start − (k+1)·interval, start − k·interval]`, the earliest one truncated, each stamped with its later edge.
**SDK parity (#677).** `SdkAggregateParityTests` (unit, `Sql.Common.Tests`) and
`ProcessedConformance.Processed_reads_match_the_SDK_aggregate_calculators` (both SQL backends, through the real SQL) run the pinned SDK's
own calculators (`AverageAggregateCalculator`, `MinMaxAggregateCalculator`, `CountAggregateCalculator`, the base `AggregateCalculator`
for Interpolative) over the same samples, forward and reverse, and compare every interval's timestamp, value, status code and Partial
bit. Known differences that remain — documented in the test, deliberately not changed here because forward results were to stay as
they are — are: the SDK reports `TimeAverage`/`Total` Uncertain whenever any region of the interval is non-Good, where ours applies the
time-based rule (a fully-covered interval bridged across a Bad sample is Good here); `Total` over an interval the data starts inside is
`TimeAverage × interval` here but the integral of the covered part in the SDK; ours flags a truncated interval Partial, the SDK does
not; the SDK's forward end-of-data Partial rule is not implemented; the SDK's per-sample Count answers 0/Good for an empty interval
inside the data and its per-sample aggregates use Good samples only; and on a reverse `Interpolative` read the SDK marks a sample exactly
at a point Interpolated (Uncertain if a Bad sample follows) where ours — like the SDK forward — returns the raw sample. On a reverse read
the SDK's own start-of-data rule keys on the *latest* sample (the first one it is fed); ours keys on the earliest in both directions.
**Pushdown trade-off.** Unlike the per-sample aggregates, the time-weighted family is **not** pushed down to SQL: the backend streams
the window's rows plus the four bounds (one batch of five result sets on SQL Server; a bounds `UNION ALL` + a window select on QuestDB)
@@ -1180,8 +1223,10 @@ Every query filters `NOT deleted`; every value goes through `StoredSample`, so t
- **AtTime:** previous-or-equal and next neighbour of each time by `LIMIT 1` seeks, `UNION ALL`-ed, **50 times per statement** (QuestDB 10.0.1 refuses
more than 126 branches). `AtTimeEvaluator` decides each value, as on `SqlServer`.
- **Events:** `[start, end)` by `ts`, ordered `(ts, event_id)`, `maxEvents + 1` fetched to produce the continuation point.
- **Processed:** one query per node; bucket index `(cast(ts AS LONG) * 10 - startUnixTicks) / intervalTicks` (never `SAMPLE BY`, whose alignment
differs), so bucket edges are identical to `SqlServer` (start-inclusive, end-exclusive, partial last bucket, interval 0 ⇒ one bucket, 100,000-bucket cap).
- **Processed:** one query per node; bucket index `(cast(ts AS LONG) * 10 - bucketBaseUnixTicks) / intervalTicks` (never `SAMPLE BY`, whose alignment
differs), so bucket edges are identical to `SqlServer` (forward start-inclusive / end-exclusive with the partial bucket last; reverse anchored at the
later time, later-edge-inclusive, partial bucket earliest — #677; interval 0 ⇒ one bucket, 100,000-bucket cap). A second `LIMIT 1` statement fetches
the tag's earliest live sample for the start-of-data Partial bit.
Min/max times use `row_number()` windows with the earliest-timestamp tie-break.
### Retention (TTL)