docs(historian): #692 raw paging fills every page; lookahead row; marked placeholders

This commit is contained in:
Joseph Doherty
2026-10-13 00:42:13 -04:00
parent 8dfa165110
commit 3d7eea6280
+25 -7
View File
@@ -582,23 +582,35 @@ gateway is provisioned, without producing error spikes in connected clients.
`NumValuesPerNode` samples with no continuation point of its own), so the server synthesises
paging time-based:
- A page that returns **exactly** `NumValuesPerNode` samples (with `NumValuesPerNode > 0`) MAY
have more behind it, so the server stores a resume cursor and returns an opaque continuation
- Every page's backend read asks for **one lookahead row** past the page (`NumValuesPerNode + 1`, plus the boundary
ties a resumed page is about to trim — see below). When the lookahead row comes back there is more data: the page
returns exactly `NumValuesPerNode` values, and the server stores a resume cursor and returns an opaque continuation
point (16 bytes). The client hands it back to fetch the next page.
- A short page (fewer than the cap) is the last page — no continuation point.
- When it does not, the page is the **last** — no continuation point, even when it is exactly full. (#692: a total that
is an exact multiple of the cap used to end with an extra empty `GoodNoData` page.)
- `NumValuesPerNode == 0` ("all values, no limit") is never paged; the whole window returns in one
shot.
- The resume cursor is **tie-safe**: the next page resumes from the last returned sample's
SourceTimestamp *inclusive* and drops the boundary samples already emitted, so samples sharing
the boundary timestamp are neither duplicated nor skipped.
the boundary timestamp are neither duplicated nor skipped. Because it re-reads them, a resumed page asks for
`NumValuesPerNode + skip + 1` rows, so it still carries a full page.
- ⚠️ **#692 (client-visible): every page is full.** A resumed page used to ask for only `NumValuesPerNode` rows, so after
the trim it netted `N − skip` values — and at `NumValuesPerNode = 1` **nothing**: the lone re-read boundary sample was
trimmed, the empty "full" page was taken for a stalled tie cluster, and a `MaxTieClusterOverfetch + 1` (65,537-row)
over-fetch stepped one tick to return an **empty page with a continuation point between every value** (live, AVEVA:
8 values = 17 pages / 25 backend reads; on every backend, forward and reverse, with or without `ReturnBounds`). Now 8
values at N = 1 is 8 pages and 8 reads. Clients that drained to the end were unaffected in content, only in cost; a
client that stopped at the first empty page lost data.
> **Oversized tie clusters — within-timestamp paging.** When more samples share one
> `SourceTimestamp` than the current page cap, the server detects that the cursor has stalled on
> a tie cluster (the last returned timestamp equals the resume timestamp). It then **over-fetches
> a tie cluster (every row of the resumed `N + skip + 1` read past the trimmed ties — the lookahead row included — still
> sits on the resume timestamp; since #692 nothing else trips it). It then **over-fetches
> the entire cluster** at that single timestamp up to a bounded ceiling controlled by
> `ServerHistorian:MaxTieClusterOverfetch` (default **65 536**), then serves the cluster
> `NumValuesPerNode` samples at a time across successive pages, advancing the cursor one tick past
> the timestamp once the cluster is fully drained.
> the timestamp once the cluster is fully drained. A cursor left inside the cluster is marked so (`WithinTieCluster`), and
> its next page over-fetches the cluster directly instead of first re-reading rows it knows are ties.
>
> **Reverse reads** (`start > end`, newest first) page the same way in the opposite direction: a drained
> cluster steps the cursor one tick **down**, the read ends when that would reach the (excluded) `end`, and
@@ -792,7 +804,13 @@ switch `ServerHistorian:HonorReturnBounds = false` restores the pre-#691 behavio
- **Counting and paging.** Bounds **count toward `NumValuesPerNode`** (§6.5.3.2: "any Bounding Values returned are included in the
numValuesPerNode count"; `NumValuesPerNode = 1` returns the start bound alone). The start bound leads the **first** page and the
end bound ends the **last**: the pending end bound rides the continuation point, and when a page is exactly full the end bound is
the whole next page. Paging a bounded read at any page size concatenates to the unpaged answer.
the whole next page. Paging a bounded read at any page size concatenates to the unpaged answer. At `NumValuesPerNode = 1`
with no start bound found, the server first checks the read holds anything (else `GoodNoData`); since #692 what that check
learns rides the continuation point — the walk-back hint and, when there is no data, the end bound it found — so the next
page reads nothing it already read.
- **Placeholders are marked, not recognised by shape (#692).** Only a `Bad_BoundNotFound` value the server synthesised is a
placeholder; a stored row that carries `Bad_BoundNotFound` with a null value (writable through HistoryUpdate) is data, and
never turns a read into the "no data and no bound" `GoodNoData` answer.
- **How it is served — one authority, `HistoryBounds` in the OPC UA server.** "At or before T" is a newest-first read of `(0, T]`
capped at one — native on SqlServer/QuestDB, the #686 walk-back of bounded forward reads on the Gateway (a tag with no sample
before StartTime is proven empty down to 1970 by the #689 walk — after two empty segments it asks once for the whole remainder —