fix(historian): #693 review — bisecting walk-back, small jump reads, read cap, DateTime/NaN hints
H1: the walk-back reset its width after a full segment and discarded a full whole-remainder read, so a dense tag that stopped reporting weeks ago gave up after 65 reads (BadTooManyOperations). It now gallops (x4, whole-remainder read after two empties) and then BISECTS once any stored row is known below — a full segment's (incl. a full whole-remainder read's) last row, or an earlier window read's last row: (mid, hi] empty => hi = mid; full => known = its last row; complete with rows => done; one density-aimed segment after a full read until an aimed segment comes back empty; below 2 ms the whole stretch from the known row (never a 100 ns step). Stopped 1 h..365 d with 5 y history: 6-18 reads at 1/min, 21-25 at 1 Hz, 25-31 at 10 Hz. M1: a jump read asks for 64 rows (1,000 times over 10 y: ~66k rows, was ~1M). L1: per-request read cap 64 + 4 x distinct times with a clear message; a strict ceiling below 4 rows is refused naming the ceiling. L2: DateTime hints are stepped. L3: under an integer hint a NaN/inf or out-of-Int64-range value is unusable, never saturated Good. L5: conformance Int64 AtTime asserts the type. L4: docs name the backend differences. Nit: the window read end is clamped to now + 1 day. Reviewer harnesses (Scratch693Fuzz/Cost, not committed): 0 differential diffs over 2,400 cases.
This commit is contained in:
+23
-9
@@ -780,14 +780,23 @@ data.
|
||||
**probe** (18 rows) for the sample after the last. Measured on the fake: ten times one second apart on a 1 Hz tag cost **1 read**
|
||||
(all exact hits) or **3** (interpolated); ten times an hour apart on a 1 Hz tag **12** (≈ 9.5k rows, never the 32k between); one
|
||||
time 400 days after the only samples **5**; a tag with no history **4**. A window read that fills its budget resolves the times before
|
||||
its last row; the next time either continues from there (if the rows in between should fit one read) or jumps to it with a read
|
||||
that starts a density-sized margin earlier (so its `prev` is normally in the same read).
|
||||
- **Walk-back** (the gateway reads forward only): segments stepping back from the time, the first sized from the window read's density
|
||||
(else 1 minute), growing ×4 when empty; after two empty segments the whole remainder down to the epoch is asked once (a complete
|
||||
answer ends the walk — sparse history and no history are found in a handful of reads). A segment that fills its budget is narrowed
|
||||
towards the time to about a quarter of a read at the density it showed — never below a row it saw — so dense history is bisected,
|
||||
not paged (a 1 kHz tag behind a 10-minute gap: 7 reads, < 10k rows). At most 64 reads; more than 1,024 stored samples within one
|
||||
millisecond fail the node `BadTooManyOperations` (not counted as a backend failure).
|
||||
its last row; the next time either continues from there (if the rows in between should fit one read) or **jumps** to it with a
|
||||
small read (64 rows) that starts a density-sized margin earlier (so its `prev` is normally in the same read) — 1,000 times
|
||||
spread over ten years of a 1 Hz tag cost ≈ 1,000 reads of ≤ 64 rows (≈ 66k rows; it used to be 1,024-row reads).
|
||||
- **Walk-back** (the gateway reads forward only), in two phases. **Gallop:** segments stepping back from the time, the first sized
|
||||
from the window read's density (else 1 minute), growing ×4 when empty; after two empty segments the whole remainder down to the
|
||||
epoch is asked once — a complete answer ends the walk, a **full** one seeds the bisection with its last row. **Bisect** (once a
|
||||
stored row is known below — a full segment's, or an earlier window read's last row): read `(mid, hi]`; empty ⇒ `hi = mid`; full ⇒
|
||||
the known row moves up to its last row; complete with rows ⇒ done. Right after a full read one segment is aimed at about a quarter
|
||||
of a read at the density it showed (until an aimed segment comes back empty — a gap — then plain halving); below 2 ms the segment
|
||||
is the whole stretch from the known row (never a 100 ns step). Measured (review harness, 5 years of history, time = now − 1 min):
|
||||
a tag stopped 1 h / 1 d / 7 d / 30 d / 365 d ago costs 6–18 reads at 1/min, 21–25 at 1 Hz and 25–31 at 10 Hz; a 1 kHz tag behind
|
||||
a 10-minute gap 6 reads (< 10k rows). (Before the review fix a full segment reset the width and a full whole-remainder read was
|
||||
discarded, so a dense tag that stopped weeks ago gave up after 65 reads.) More than 1,024 stored samples within one millisecond
|
||||
fail the node `BadTooManyOperations` (not counted as a backend failure).
|
||||
- **Bounded.** One request takes at most **64 + 4 × distinct times** raw reads; past it the node fails `BadTooManyOperations`
|
||||
("needed more than N raw reads", naming a small strict-gateway ceiling when one bounded the reads). A strict gateway whose ceiling is
|
||||
below 4 rows is refused up front with a message naming the ceiling; a small one (e.g. 10) is served by paging within the cap.
|
||||
- **Wire shape.** Every request bound is a whole millisecond and a row counts only **strictly inside** its request, so AVEVA's synthetic
|
||||
boundary row at the query start (#687) — and any row at the query end — is never taken for a stored sample. A Bad row is value-less
|
||||
(#690). Nothing is asked past now + 1 day (the gateway's AVEVA may refuse a far-future end; a time beyond it is stepped from the last
|
||||
@@ -800,7 +809,12 @@ data.
|
||||
**before** the shared evaluator runs, so its own type rules apply exactly as on the SQL backends: a **Boolean** node
|
||||
(value ≠ 0) is **stepped** — never 0.5; an **integer** node is interpolated, rounded half away from zero and returned as the
|
||||
node's own integer type (`Int32` ⇒ `int`); Float / Double interpolate as Double (as the SQL backends return them). With no
|
||||
hint (an unknown node) every numeric tag interpolates as Double.
|
||||
hint (an unknown node) every numeric tag interpolates as Double. A **DateTime** node is stepped. Under an integer hint a NaN, ±∞ or
|
||||
out-of-Int64-range stored value is no integer: it is treated as unusable (as `BadNoData`), so a neighbour like that is never
|
||||
interpolated or saturated into a Good value.
|
||||
- **What differs between backends (by design, not changed by #693).** Only the **Gateway** converts an AtTime result to the node's
|
||||
integer type (`Int32` ⇒ `int`): the SQL backends store and return integer nodes as `Int64` (and Boolean as Boolean). Gateway
|
||||
**Raw** reads still return a Boolean node's samples as the stored Double 0/1 (the hint is passed only on AtTime).
|
||||
- ⚠️ **Shared-evaluator fix (all backends that use it):** `AtTimeEvaluator` returned every interpolated `Int64` boxed as a
|
||||
**Double** (a C# `cond ? (long)x : double` conditional widens the long); its "keeping the Int64 type" tests passed because
|
||||
Shouldly's `ShouldBe` compares numbers across types. SqlServer / QuestDB interpolated `Int64` AtTime values are now `Int64`,
|
||||
|
||||
Reference in New Issue
Block a user