fix(historian): #689 size reverse-walk segments from the page, not the tie budget

ReverseRawReader aimed every segment at half of MaxTieClusterOverfetch
(~32k rows by default) and carried that width page to page, so a paged
'latest N' read on a dense Gateway tag fetched ~32,770 rows per page
(2-5 s live) to return N. The walk now lives in a generic ReverseWalk
core: segments are aimed at ~N+8 rows (page budget 2N+16), a discovery
segment (first without a hint / after an empty one) may read 1,024, and
the tie budget is only the ceiling a single-timestamp cluster escalates
to. A truncated read never re-enters the stretch below its newest row
until the gap above it is bisected. Tests: per-page row counts on a
dense tag (node manager + walk), randomized exactness fuzz (walk and
node-manager paging), dense-tag-below-a-long-gap read bound.
This commit is contained in:
Joseph Doherty
2026-10-12 16:25:07 -04:00
parent 71d031fcd0
commit d33992034d
8 changed files with 663 additions and 183 deletions
+20 -9
View File
@@ -639,15 +639,26 @@ backend:
`[⌊segLo⌋, ⌈segHi + 1 tick⌉]`, both rounded **outward to the millisecond**, and filtered on the samples' own
timestamps — exact for an inclusive or exclusive forward end, and the row AVEVA fabricates at a query's (possibly
rounded-up) start can never land inside the segment. The first segment is **1 minute** (or the width the previous
page ended on — it rides the continuation point); each next width is aimed at about half the row budget from the
density just observed, growing at most ×4. So "the last N values" of a live tag is normally one read, a dense tag
costs about one read per page, and sparse history is found in a few dozen tiny ones.
- **Never answered from a truncated read.** The row budget is `M = max(1, MaxTieClusterOverfetch)`; a segment read asks
for `M + 2` rows (room for the fabricated row and one to see past the segment). Rows stream ascending, so a read is
complete when it came back short or its last row is past the segment; otherwise it is discarded — re-read once with
room for the rows before the segment that crowded it (a neighbouring tie cluster, at most `2M + 4` rows), or narrowed
to the density it showed. Only **one timestamp holding more than `M` samples** is refused, **`BadTooManyOperations`**
(a request limit, not a backend fault); a cluster of exactly `M`, or neighbouring clusters each within `M`, is served.
page ended on — it rides the continuation point); each next width is aimed from the density just observed at about
half the **page budget** `2N + 16` (≈ N + 8 rows), growing at most ×4. So "the last N values" of a live tag is
normally one read of about N rows, a dense tag costs about one N-sized read per page, and sparse history is found in
a few dozen tiny ones.
- **Read sizes (#689).** A segment read asks for its budget + 2 rows (room for the fabricated row and one to see past
the segment). The budget is the page budget `2N + 16`, except a **discovery** segment — the first of a page with no
carried width, or one after an empty segment, whose ×4 growth may overshoot into dense data — which may read up to
**1,024** rows. The tie budget `M = max(1, MaxTieClusterOverfetch)` is only a **ceiling**: a single timestamp holding
more rows than the segment budget escalates to `M`.
⚠️ Before #689 every segment was aimed at half of `M` (≈ 32k rows by default) and that width was carried page to
page, so on a 1 Hz tag from about page 5 every page of a "latest N" read fetched ~32,770 rows and took 2–5 s to
return N.
- **Never answered from a truncated read.** Rows stream ascending, so a read is complete when it came back short or its
last row is past the segment; otherwise it is discarded — re-read once with room for the rows before the segment
that crowded it (a neighbouring tie cluster, at most twice the budget + 4 rows), or narrowed: to the density it
showed, and never again below the newest row it saw until what lies above that row is read (the gap above it is
bisected — a dense tag that went quiet days ago is found in about 20 reads rather than re-reading the same oldest
rows of its burst over and over). Only **one timestamp holding more than `M` samples** is refused,
**`BadTooManyOperations`** (a request limit, not a backend fault); a cluster of exactly `M`, or neighbouring
clusters each within `M`, is served.
- **Bounds.** The walk starts no later than **now + 1 day** (a `StartTime = DateTime.MaxValue` read does not send
year-9999 query starts) and never reads before the **Unix epoch** (AVEVA holds nothing earlier): a walked-back read
returns only samples strictly after 1970-01-01T00:00:00Z and no later than now + 1 day.