fix(historian): #685 Gateway native TimeAverage/Total — drop AVEVA's pre-window row, keep Uncertain values

AVEVA answers one extra row for the interval before the window; it was mapped as
a result stamped before StartTime (N+1 values). Rows whose interval lies outside
the requested span (either direction) are now dropped after the drained stream.
An Uncertain (opc 64-191) interval kept no value (opc >= 192 was the only data
test); it now carries its value as UncertainDataSubNormal | Calculated.
Docs: AVEVA's TWA excludes NULL-gap time while the SQL backends bridge it;
end-to-end per-interval values also need historiangw#27.
This commit is contained in:
Joseph Doherty
2026-10-12 14:31:31 -04:00
parent 9ced6455ad
commit 44a03bedc0
5 changed files with 249 additions and 24 deletions
+28 -5
View File
@@ -795,17 +795,40 @@ The OPC UA `Total` aggregate is **supported** over the HistorianGateway backend.
native **`Integral`** retrieval mode, so `Total` maps straight to it (`HistoryAggregateType.Total →
RetrievalMode.Integral`) — no client-side scaling. (This replaces the retired Wonderware path, which had no
`Total` column and derived it client-side as time-weighted `Average × interval-seconds`.) Values pass through unchanged; a Good
bucket is `Good | Calculated` (`0x00000401`) and an empty / null bucket `BadNoData | Calculated` (#681), stamped with its interval
start (see below).
bucket is `Good | Calculated` (`0x00000401`), an Uncertain one `UncertainDataSubNormal | Calculated` (`0x40A40401`) **with its
value** (#685), and an empty / null bucket `BadNoData | Calculated` (#681), stamped with its interval start (see below).
**Native `TimeAverage` / `Total` rows (#685 — client-visible).** Two mapping defects found by the live AVEVA gate (2026-10-12) are
fixed:
- **One result per interval in `[Start, End)`.** AVEVA always answers one extra row for the interval *before* the window
(`[start − interval, start)` — a real value when data exists there, a no-data row otherwise). It used to be passed through, so a
client got N + 1 values, the first stamped before its StartTime. Rows whose interval start lies outside the requested span are now
dropped (a row with no start time is placed by its end time, `(lo, hi]`); the span is taken in either order, so the filter does
not depend on the read direction. The stream is still drained in full and filtered afterwards — never broken out of.
- **An Uncertain interval keeps its value.** AVEVA marks an interval that holds a NULL/Bad sample `opc_quality = 64` (Uncertain) and
still supplies its value. It used to be mapped `BadNoData` with the value discarded (`opc_quality >= 192` was the only "has data"
test); every Uncertain-family quality (64–191, low byte) now returns the value as `UncertainDataSubNormal | Calculated` — the
sub-code the SQL backends give an interval that includes non-good data. Only a Bad-family quality (< 64) is `BadNoData`, null.
- ⚠️ **AVEVA's semantics differ from the SQL backends' across a NULL gap.** AVEVA's `TimeWeightedAverage` holds the previous value
up to the NULL sample and **excludes the gap time from both numerator and denominator** (live: an interval with 10 good seconds
of (5 → 1) answers 3, `Integral` 30); the `SqlServer`/`QuestDb` evaluator bridges across the outage (Part 13 interpolated
bounding — 6.08 Uncertain for the same samples, see [Time-weighted aggregates](#time-weighted-aggregates-part-13-671)).
Extrapolation past the last sample is `Good` on AVEVA and `Uncertain` on the SQL backends. On all-Good spans the magnitudes
match exactly (both sloped/trapezoidal).
- ⚠️ **End-to-end correctness also needs the gateway fix historiangw#27:** the gateway sends the aggregate interval to AVEVA in
ticks where AVEVA reads milliseconds (10,000× too large), so today a native read returns one row for the whole window (plus the
pre-window row, now dropped). OtOpcUa does not work around it; per-interval values return once the gateway is fixed.
**Integral unit (#677).** AVEVA's integral is `∫ value dt` (seconds) ÷ the tag's `IntegralDivisor`. The proto3 `double` default for
`HistorianTagDefinition.IntegralDivisor` is `0`, so `GatewayTagProvisioner` now sends **`IntegralDivisor = 1`** explicitly on every
`EnsureTags` — the value·seconds unit the SQL backends' `Total` uses. A tag provisioned earlier keeps its divisor until its next
`EnsureTags` (an upsert). The live magnitude check against AVEVA (a constant 1 integrated over 1 s reads 1) is still open — it needs the
VPN. Reverse processed reads of the AVEVA-native `TimeAverage`/`Total` are not re-anchored by #677 (the gateway refuses `end < start`);
`EnsureTags` (an upsert). The live magnitude check against AVEVA (a constant 1 integrated over 1 s reads 1) passed on 2026-10-12 at the AVEVA
level; end to end it waits on historiangw#27 (above). Reverse processed reads of the AVEVA-native `TimeAverage`/`Total` are not re-anchored by #677 (the gateway refuses `end < start`);
the raw-evaluated per-sample aggregates (#681) do follow #677. Since #681 every native bucket is stamped with its interval **start**
(Part 13; it was the interval end) and carries the Calculated info bits (`Good` ⇒ `0x00000401`, no data ⇒ `BadNoData | 0x401`).
⚠️ **The interval-start stamp is pending the live leg:** it reads the aggregate row's `start_time`, which histsdk decodes from a row
The live AVEVA gate (2026-10-12) confirmed `start_time` is the interval start (and the `value·s` unit). Previously noted as
pending: it reads the aggregate row's `start_time`, which histsdk decodes from a row
field (`tail + 43`) its own reverse-engineering notes still mark unverified (HistorianGateway
`docs/plans/2026-07-13-archreview2-D-histsdk-residue-design.md`); if that field proves wrong on live AVEVA the stamp is wrong with it.