fix(historian): #693 review — bisecting walk-back, small jump reads, read cap, DateTime/NaN hints

H1: the walk-back reset its width after a full segment and discarded a full
whole-remainder read, so a dense tag that stopped reporting weeks ago gave up
after 65 reads (BadTooManyOperations). It now gallops (x4, whole-remainder read
after two empties) and then BISECTS once any stored row is known below — a full
segment's (incl. a full whole-remainder read's) last row, or an earlier window
read's last row: (mid, hi] empty => hi = mid; full => known = its last row;
complete with rows => done; one density-aimed segment after a full read until an
aimed segment comes back empty; below 2 ms the whole stretch from the known row
(never a 100 ns step). Stopped 1 h..365 d with 5 y history: 6-18 reads at
1/min, 21-25 at 1 Hz, 25-31 at 10 Hz.
M1: a jump read asks for 64 rows (1,000 times over 10 y: ~66k rows, was ~1M).
L1: per-request read cap 64 + 4 x distinct times with a clear message; a strict
ceiling below 4 rows is refused naming the ceiling.
L2: DateTime hints are stepped. L3: under an integer hint a NaN/inf or
out-of-Int64-range value is unusable, never saturated Good.
L5: conformance Int64 AtTime asserts the type. L4: docs name the backend
differences. Nit: the window read end is clamped to now + 1 day.

Reviewer harnesses (Scratch693Fuzz/Cost, not committed): 0 differential diffs
over 2,400 cases.
This commit is contained in:
Joseph Doherty
2026-10-13 01:09:54 -04:00
parent 0d249f3ee9
commit 68a1f31397
4 changed files with 379 additions and 96 deletions
+23 -9
View File
@@ -780,14 +780,23 @@ data.
**probe** (18 rows) for the sample after the last. Measured on the fake: ten times one second apart on a 1 Hz tag cost **1 read**
(all exact hits) or **3** (interpolated); ten times an hour apart on a 1 Hz tag **12** (≈ 9.5k rows, never the 32k between); one
time 400 days after the only samples **5**; a tag with no history **4**. A window read that fills its budget resolves the times before
its last row; the next time either continues from there (if the rows in between should fit one read) or jumps to it with a read
that starts a density-sized margin earlier (so its `prev` is normally in the same read).
- **Walk-back** (the gateway reads forward only): segments stepping back from the time, the first sized from the window read's density
(else 1 minute), growing ×4 when empty; after two empty segments the whole remainder down to the epoch is asked once (a complete
answer ends the walk — sparse history and no history are found in a handful of reads). A segment that fills its budget is narrowed
towards the time to about a quarter of a read at the density it showed — never below a row it saw — so dense history is bisected,
not paged (a 1 kHz tag behind a 10-minute gap: 7 reads, < 10k rows). At most 64 reads; more than 1,024 stored samples within one
millisecond fail the node `BadTooManyOperations` (not counted as a backend failure).
its last row; the next time either continues from there (if the rows in between should fit one read) or **jumps** to it with a
small read (64 rows) that starts a density-sized margin earlier (so its `prev` is normally in the same read) — 1,000 times
spread over ten years of a 1 Hz tag cost ≈ 1,000 reads of ≤ 64 rows (≈ 66k rows; it used to be 1,024-row reads).
- **Walk-back** (the gateway reads forward only), in two phases. **Gallop:** segments stepping back from the time, the first sized
from the window read's density (else 1 minute), growing ×4 when empty; after two empty segments the whole remainder down to the
epoch is asked once — a complete answer ends the walk, a **full** one seeds the bisection with its last row. **Bisect** (once a
stored row is known below — a full segment's, or an earlier window read's last row): read `(mid, hi]`; empty ⇒ `hi = mid`; full ⇒
the known row moves up to its last row; complete with rows ⇒ done. Right after a full read one segment is aimed at about a quarter
of a read at the density it showed (until an aimed segment comes back empty — a gap — then plain halving); below 2 ms the segment
is the whole stretch from the known row (never a 100 ns step). Measured (review harness, 5 years of history, time = now − 1 min):
a tag stopped 1 h / 1 d / 7 d / 30 d / 365 d ago costs 6–18 reads at 1/min, 21–25 at 1 Hz and 25–31 at 10 Hz; a 1 kHz tag behind
a 10-minute gap 6 reads (< 10k rows). (Before the review fix a full segment reset the width and a full whole-remainder read was
discarded, so a dense tag that stopped weeks ago gave up after 65 reads.) More than 1,024 stored samples within one millisecond
fail the node `BadTooManyOperations` (not counted as a backend failure).
- **Bounded.** One request takes at most **64 + 4 × distinct times** raw reads; past it the node fails `BadTooManyOperations`
("needed more than N raw reads", naming a small strict-gateway ceiling when one bounded the reads). A strict gateway whose ceiling is
below 4 rows is refused up front with a message naming the ceiling; a small one (e.g. 10) is served by paging within the cap.
- **Wire shape.** Every request bound is a whole millisecond and a row counts only **strictly inside** its request, so AVEVA's synthetic
boundary row at the query start (#687) — and any row at the query end — is never taken for a stored sample. A Bad row is value-less
(#690). Nothing is asked past now + 1 day (the gateway's AVEVA may refuse a far-future end; a time beyond it is stepped from the last
@@ -800,7 +809,12 @@ data.
**before** the shared evaluator runs, so its own type rules apply exactly as on the SQL backends: a **Boolean** node
(value ≠ 0) is **stepped** — never 0.5; an **integer** node is interpolated, rounded half away from zero and returned as the
node's own integer type (`Int32` ⇒ `int`); Float / Double interpolate as Double (as the SQL backends return them). With no
hint (an unknown node) every numeric tag interpolates as Double.
hint (an unknown node) every numeric tag interpolates as Double. A **DateTime** node is stepped. Under an integer hint a NaN, ±∞ or
out-of-Int64-range stored value is no integer: it is treated as unusable (as `BadNoData`), so a neighbour like that is never
interpolated or saturated into a Good value.
- **What differs between backends (by design, not changed by #693).** Only the **Gateway** converts an AtTime result to the node's
integer type (`Int32` ⇒ `int`): the SQL backends store and return integer nodes as `Int64` (and Boolean as Boolean). Gateway
**Raw** reads still return a Boolean node's samples as the stored Double 0/1 (the hint is passed only on AtTime).
- ⚠️ **Shared-evaluator fix (all backends that use it):** `AtTimeEvaluator` returned every interpolated `Int64` boxed as a
**Double** (a C# `cond ? (long)x : double` conditional widens the long); its "keeping the Int64 type" tests passed because
Shouldly's `ShouldBe` compares numbers across types. SqlServer / QuestDB interpolated `Int64` AtTime values are now `Int64`,