fix(historian): #691.4 review — search the gap above a truncated read by gallop then bisect, never below 1 ms

Bisecting the gap above a truncated read halved below the 1 ms request
granularity (requests are ms-rounded, so it re-sent identical requests)
and bisected the WHOLE gap even when the burst ended right above the
row it saw: a 1,026-row burst ending 100 days ago took 46 reads. After
an empty top-anchored segment the walk now gallops up from the known
row ((known + d, top], d x4, following each truncated read), then
bisects the bracket, with halving floored at the granularity (the Min
kept; a crowded gallop falls back to top-anchored halving). Burst: 18
reads, 0 repeated requests; dense tag stopped 2 days ago: 26 (gallop
alone crawled: 94). Walk fuzzed at 4,000 seeds locally (400 in-tree).
Docs: measured counts; CLAUDE.md gateway prerequisites list
RuntimeDb:EventReadMaxRows.
This commit is contained in:
Joseph Doherty
2026-10-12 17:37:32 -04:00
parent ccda27d220
commit d28e3e64f1
4 changed files with 119 additions and 18 deletions
+9 -6
View File
@@ -654,9 +654,12 @@ backend:
- **Never answered from a truncated read.** Rows stream ascending, so a read is complete when it came back short or its
last row is past the segment; otherwise it is discarded — re-read once with room for the rows before the segment
that crowded it (a neighbouring tie cluster, at most twice the budget + 4 rows), or narrowed: to the density it
showed, and never again below the newest row it saw until what lies above that row is read (the gap above it is
bisected — a dense tag that went quiet days ago is found in about 20 reads rather than re-reading the same oldest
rows of its burst over and over). Only **one timestamp holding more than `M` samples** is refused,
showed, and never again below the newest row it saw until what lies above that row is read. The gap above it is
searched: a segment at that density ending on the top (data running right up to it); if empty, a **gallop** up from
that row (`(row + d, top]`, `d` ×4 per step, following each truncated read) until a read comes back empty, then a
**bisection** of the bracket — never narrower than the 1 ms request granularity, which only re-sent the same request.
Measured: a dense tag that stopped two days ago ≈ **26** reads, a burst of just over 1,024 rows that ended 100 days ago
≈ **18** (was 46 bisecting the whole gap) — rather than re-reading the oldest rows of a burst over and over. Only **one timestamp holding more than `M` samples** is refused,
**`BadTooManyOperations`** (a request limit, not a backend fault); a cluster of exactly `M`, or neighbouring
clusters each within `M`, is served.
- **Proving the rest empty (#691).** Walking down through empty history to the floor used to cost one read per ×4
@@ -778,8 +781,8 @@ says is complete carries no continuation point. `NumValuesPerNode == 0` stays un
is forward `[MinValue, EndTime)` everywhere.
- **Backends.** The server checks `IHistorianDataSource.SupportsReverseEvents`. SqlServer and QuestDB serve reverse natively
(`… > lo AND … <= hi ORDER BY time DESC, id DESC`, the `ReadEventsAsync(…, bool reverse, …)` overload) and report `true`.
**Gateway and Null report `false`** — the gateway streams ascending only and rejects `end < start` — and are never sent a
reverse read: since **#691.4** the server walks it back from bounded **forward** reads with the raw walk (#686/#689/#691:
**Gateway reports `false`** — the gateway streams ascending only and rejects `end < start` — and is never sent a reverse
read (Null reports `true` and answers empty either way, as for raw): since **#691.4** the server walks it back from bounded **forward** reads with the raw walk (#686/#689/#691:
page-sized ms-aligned segments filtered to `(segLo, segHi]`, a read the source flagged truncated is never answered from,
the now + 1 day / 1970 bounds, the remainder probe and the proven-empty stretch carried in the cursor). Event specifics:
- **Ties** are sorted by `EventId` (ordinal) on every read — the gateway orders by `EventTimeUtc` only, in any order within a
@@ -796,7 +799,7 @@ says is complete carries no continuation point. `NumValuesPerNode == 0` stays un
That cap truncates **silently** (the stream just ends), so a segment it cut would look complete and drop its newest events —
the same blindness a forward read has to it, but on the end a reverse read is asking for.
⚠️ **Client-visible (#691.4).** On Gateway (the default backend) an explicit reverse events read used to be refused
⚠️ **Client-visible (#691.4).** On Gateway (the default backend) and Null an explicit reverse events read used to be refused
`BadHistoryOperationUnsupported` (#675), and an end-only capped events read returned the **oldest** N events ascending. Both
now return the latest events newest-first. Because the capability is checked, not a caught exception, a
`NotSupportedException` thrown from inside a backend (a driver library) is still logged as a backend fault. Every