fix(historian): #687 review — one bounded re-read instead of failing, raw budget unclamped, exclusive forward end

- M1/M2: when the 1 ms pre-window margin (boundary row + stored samples, e.g. a
  >16-sample tie cluster at start-1ms / lastTs-1ms) uses up the cap + 16 budget,
  re-read ONCE asking for cap + m + 16 (m = pre-window rows seen); fail
  BadTooManyOperations only if the re-read is still short because of them. The
  raw budget is no longer clamped at the gateway default ceiling (AVEVA always
  emits the boundary row, so page sizes 999,985-1,000,000 always came back
  short); the #688 clamp stays on the per-sample aggregate path only.
- Merge trap: NumValuesPerNode 0 => int.MaxValue, never through the clamp.
- L3: the forward end is exclusive ([start, end), the IHistorianDataSource
  contract); a point read (start == end, the tie-cluster over-fetch) inclusive.
- RequestStartFor never goes before the Unix epoch (#686's walk-back floor).
- L4 (docs): an Uncertain native row for an all-NULL interval is unverified live.
This commit is contained in:
Joseph Doherty
2026-10-12 15:19:34 -04:00
parent 6796ba4126
commit 8114f13a28
5 changed files with 202 additions and 63 deletions
+25 -12
View File
@@ -682,18 +682,28 @@ data.
> `18:06:21 v=30 Good`), a window before the tag's history answered `v=0 Bad` instead of `GoodNoData`, and a start-only read led
> with the fabricated row. It is now dropped; such windows read `GoodNoData`.
- **How.** The same classification the raw-evaluated aggregates use (#681, `RawAggregateReader.RequestStartFor` /
`IsWindowSample` — one authority): the gateway is asked from **1 ms before** the window, so the boundary row lands before it, and
only rows stamped at or after StartTime are returned. The decision is by **timestamp**, not by value or quality — a stored NULL
sample looks exactly like the no-data boundary row on the wire (value 0, Bad), and a stored NULL inside the window is still
returned (Bad), as it is a stored sample.
- **Paging (#669) is unaffected.** The gateway is asked for `NumValuesPerNode + 16` rows (`NumValuesPerNode = 0` stays unlimited),
so the boundary row cannot shorten a page: a full page still holds exactly `NumValuesPerNode` stored samples and carries a
continuation point, and a resumed page (which re-reads from the last returned timestamp inclusive, and the tie-cluster over-fetch
`start == end` read) no longer gets a fabricated row at its resume point. If more than 16 rows lie in the 1 ms before the window
and exhaust that allowance, the page cannot be read completely and the node fails (`HistoryReadLimitExceededException`, not
counted as a backend fault) rather than answering a short page that would end the client's paging early. Every gateway stream is
still drained to completion.
- **How.** The same request start the raw-evaluated aggregates use (#681, `RawAggregateReader.RequestStartFor` — one authority):
the gateway is asked from **1 ms before** the window (never before the Unix epoch, the start of the historian's time encoding),
so the boundary row lands before it, and only rows `RawAggregateReader.IsWindowSample` are returned. The decision is by
**timestamp**, not by value or quality — a stored NULL sample looks exactly like the no-data boundary row on the wire (value 0,
Bad), and a stored NULL inside the window is still returned (Bad), as it is a stored sample.
- **The forward end is exclusive** (`[StartTime, EndTime)`, the `IHistorianDataSource` contract and what the SQL backends return;
the gateway's own `ReadRaw` end is inclusive, so a sample stored exactly at EndTime used to be returned). A point read
(`StartTime == EndTime` — the server's tie-cluster over-fetch) still returns the samples at that instant. The #686 reverse
walk-back is exact either way.
- **Row budget.** `NumValuesPerNode = 0` ("all values") asks the gateway for an unbounded `max_values` (`int.MaxValue` — the client
package and the gateway reject 0, #686). A cap asks for `NumValuesPerNode + 16` rows, **not** clamped at the gateway's default
ceiling (#688 clamps only the per-sample aggregate budget): AVEVA always emits the boundary row, so a clamped budget would make
every page size from 999,985 to 1,000,000 come back short. A gateway with `AllowStreamingBeyondCeiling=false` refuses an
over-ceiling budget instead ⇒ `BadTooManyOperations` (not a backend fault); a default gateway serves it.
- **Paging (#669) is unaffected.** A full page still holds exactly `NumValuesPerNode` stored samples and carries a continuation
point, and a resumed page (which re-reads from the last returned timestamp inclusive, and the tie-cluster over-fetch
`start == end` read) no longer gets a fabricated row at its resume point. A read is complete when the stream came back short,
filled the page, or reached EndTime. If rows **before** the window — the boundary row plus stored samples inside the 1 ms margin,
e.g. a tie cluster at `StartTime − 1 ms` or at a resume point `− 1 ms` — used up the budget, the window is re-read **once**
asking for `NumValuesPerNode + m + 16` (m = the pre-window rows the first read saw, unclamped). Only if that re-read is still
short because of pre-window rows does the node fail `BadTooManyOperations` (not a backend fault) rather than answer a short page
that would end the client's paging early. Every gateway stream is still drained to completion.
- **`ReturnBounds` is not honoured** — on any backend: the server does not pass it to the data source, so no bounding values are
returned (see [Known limitations](#known-limitations)). The AVEVA boundary row is not used as a bound either: it is stamped at the
query start rather than carrying the time of the real prior sample, so it could only ever be a Part 11 *interpolated* bound, which
@@ -845,6 +855,9 @@ fixed:
- ⚠️ **End-to-end correctness also needs the gateway fix historiangw#27:** the gateway sends the aggregate interval to AVEVA in
ticks where AVEVA reads milliseconds (10,000× too large), so today a native read returns one row for the whole window (plus the
pre-window row, now dropped). OtOpcUa does not work around it; per-interval values return once the gateway is fixed.
- **Not verified live:** what AVEVA answers for an interval holding **only** NULL samples (no good time at all). If it marks such a
row Uncertain (`opc_quality` 64–191) with a placeholder value, that value is now passed through as Uncertain; the gate only
observed Uncertain rows that also held good time. Re-check on the VPN once historiangw#27 is fixed.
**Integral unit (#677).** AVEVA's integral is `∫ value dt` (seconds) ÷ the tag's `IntegralDivisor`. The proto3 `double` default for
`HistorianTagDefinition.IntegralDivisor` is `0`, so `GatewayTagProvisioner` now sends **`IntegralDivisor = 1`** explicitly on every