fix(historian): #691.4 review — search the gap above a truncated read by gallop then bisect, never below 1 ms

Bisecting the gap above a truncated read halved below the 1 ms request
granularity (requests are ms-rounded, so it re-sent identical requests)
and bisected the WHOLE gap even when the burst ended right above the
row it saw: a 1,026-row burst ending 100 days ago took 46 reads. After
an empty top-anchored segment the walk now gallops up from the known
row ((known + d, top], d x4, following each truncated read), then
bisects the bracket, with halving floored at the granularity (the Min
kept; a crowded gallop falls back to top-anchored halving). Burst: 18
reads, 0 repeated requests; dense tag stopped 2 days ago: 26 (gallop
alone crawled: 94). Walk fuzzed at 4,000 seeds locally (400 in-tree).
Docs: measured counts; CLAUDE.md gateway prerequisites list
RuntimeDb:EventReadMaxRows.
This commit is contained in:
Joseph Doherty
2026-10-12 17:37:32 -04:00
parent ccda27d220
commit d28e3e64f1
4 changed files with 119 additions and 18 deletions
+4 -1
View File
@@ -1353,7 +1353,7 @@ historian tagname, which is the tag's driver `FullName`.
- **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; per-sample aggregates (10, simple definitions) are pushed down to T-SQL with bucket math in **100 ns ticks**; the Part 13 time-weighted seven (TimeAverage/TimeAverage2/Total/Total2/Interpolative, #671 — ⚠️ `Total` is now the time-weighted integral, no longer `SUM` — plus the **duration-based** PercentGood/PercentBad, #680) stream window + 4 bound rows into the shared `TimeWeightedAggregator` (both SQL backends, identical results; reverse `Interpolative` anchored at the later time; window capped by `ServerHistorian:MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations`, percentages included). ⚠️ **#680 status rules (client-visible):** TimeAverage/Total are Uncertain if any part of the interval uses an Uncertain value, is bridged across a Bad sample, is extrapolated past the last sample (⇒ a live tag's latest interval is Uncertain) or is not covered; TimeAverage2/Total2 keep the SDK time-based rule (Uncertain time counts as Good) but never extrapolate past the last sample — that interval is Partial, later ones `BadNoData` (a report-by-exception tag with nothing after its last sample reads `BadNoData` until its next sample arrives). `docs/Historian.md` §"Time-weighted aggregates". ⚠️ **#680 per-sample (client-visible):** `Count` counts **Good** samples only (value-based status; an all-Bad interval is generic Bad, value null) and an empty interval **inside the tag's data range** is `0`/Good (outside it still `BadNoData`); `Start`/`End` return the **raw** first/last sample of any quality — its own timestamp and status, value null when Bad; `Delta` is Uncertain only when the interval's first or last sample is unusable; ⚠️ #684: `Average`/`Minimum`/`Maximum`/`Range`/`*ActualTime` use **Good samples only** (was Good + Uncertain; an Uncertain-only interval is `BadNoData`), Min/Max-family status Uncertain on a Bad sample, an Uncertain value beyond the Good extreme (Part 13 — the SDK ignores that) or an Uncertain non-numeric sample, and only a **Good** String/DateTime sample refuses them (`BadAggregateInvalidInputs`; Delta still refuses any usable one); and every per-sample aggregate but `Average` sets **end-of-data Partial** on the interval holding the last sample (forward incl. its earlier edge, reverse strictly inside), never on a Bad result. The data-range lookup is a key seek on SQL Server and a `tag` symbol-index seek on QuestDB (#683 — `{p}sample.tag` is `SYMBOL INDEX`; the bootstrapper ADDs it to a pre-#683 table, idempotent), read BEFORE the bucket statement so a racing append can never yield a Good `Count` of 0 (a concurrent HistoryUpdate delete is the mirror case and still can, transiently). `docs/Historian.md` §"Aggregates". Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. **Retention (#667):** opt-in `SqlServer:RetentionDays` (+ `PurgeInterval` 1h, `PurgeBatchSize` 4000) -> `HistorianRetentionService` purges old `hist.Sample` (per tag) / `hist.Event` in `DELETE TOP(n)` batches, Primary-gated on `IHistorizationRoleView`, one purger per DB via `sp_getapplock`; counter `otopcua.historian.retention.purged{table}`. Optional `IHistorianBackendProvider.CreateRetention` (default null). `docs/Historian.md` §Retention. - **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; per-sample aggregates (10, simple definitions) are pushed down to T-SQL with bucket math in **100 ns ticks**; the Part 13 time-weighted seven (TimeAverage/TimeAverage2/Total/Total2/Interpolative, #671 — ⚠️ `Total` is now the time-weighted integral, no longer `SUM` — plus the **duration-based** PercentGood/PercentBad, #680) stream window + 4 bound rows into the shared `TimeWeightedAggregator` (both SQL backends, identical results; reverse `Interpolative` anchored at the later time; window capped by `ServerHistorian:MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations`, percentages included). ⚠️ **#680 status rules (client-visible):** TimeAverage/Total are Uncertain if any part of the interval uses an Uncertain value, is bridged across a Bad sample, is extrapolated past the last sample (⇒ a live tag's latest interval is Uncertain) or is not covered; TimeAverage2/Total2 keep the SDK time-based rule (Uncertain time counts as Good) but never extrapolate past the last sample — that interval is Partial, later ones `BadNoData` (a report-by-exception tag with nothing after its last sample reads `BadNoData` until its next sample arrives). `docs/Historian.md` §"Time-weighted aggregates". ⚠️ **#680 per-sample (client-visible):** `Count` counts **Good** samples only (value-based status; an all-Bad interval is generic Bad, value null) and an empty interval **inside the tag's data range** is `0`/Good (outside it still `BadNoData`); `Start`/`End` return the **raw** first/last sample of any quality — its own timestamp and status, value null when Bad; `Delta` is Uncertain only when the interval's first or last sample is unusable; ⚠️ #684: `Average`/`Minimum`/`Maximum`/`Range`/`*ActualTime` use **Good samples only** (was Good + Uncertain; an Uncertain-only interval is `BadNoData`), Min/Max-family status Uncertain on a Bad sample, an Uncertain value beyond the Good extreme (Part 13 — the SDK ignores that) or an Uncertain non-numeric sample, and only a **Good** String/DateTime sample refuses them (`BadAggregateInvalidInputs`; Delta still refuses any usable one); and every per-sample aggregate but `Average` sets **end-of-data Partial** on the interval holding the last sample (forward incl. its earlier edge, reverse strictly inside), never on a Bad result. The data-range lookup is a key seek on SQL Server and a `tag` symbol-index seek on QuestDB (#683 — `{p}sample.tag` is `SYMBOL INDEX`; the bootstrapper ADDs it to a pre-#683 table, idempotent), read BEFORE the bucket statement so a racing append can never yield a Good `Count` of 0 (a concurrent HistoryUpdate delete is the mirror case and still can, transiently). `docs/Historian.md` §"Aggregates". Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. **Retention (#667):** opt-in `SqlServer:RetentionDays` (+ `PurgeInterval` 1h, `PurgeBatchSize` 4000) -> `HistorianRetentionService` purges old `hist.Sample` (per tag) / `hist.Event` in `DELETE TOP(n)` batches, Primary-gated on `IHistorizationRoleView`, one purger per DB via `sp_getapplock`; counter `otopcua.historian.retention.purged{table}`. Optional `IHistorianBackendProvider.CreateRetention` (default null). `docs/Historian.md` §Retention.
- **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend". - **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend".
- ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); on forward reads Partial also marks the interval the data starts inside (not Average — the SDK's rule, forward-only); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates. - ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); on forward reads Partial also marks the interval the data starts inside (not Average — the SDK's rule, forward-only); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates.
- ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` serves it from FORWARD reads: uncapped ⇒ ONE forward read reversed in memory; capped ⇒ a bounded walk back in segments (ms-aligned request bounds, width aimed from observed density at ≈ N + 8 rows and carried page to page in the continuation point — #689: it used to aim at half `MaxTieClusterOverfetch`, ~32k rows per page on a dense tag; a read asks for `2N + 18` rows (a discovery segment ≤ 1,026), `MaxTieClusterOverfetch` is only the tie-cluster ceiling — a full read is re-read, never answered truncated; only one timestamp holding more than max(1, `MaxTieClusterOverfetch`) samples ⇒ `BadTooManyOperations`; walks no later than now + 1 day and never before the Unix epoch; #691: after two empty segments it asks once for the whole remainder, and the stretch a page proved empty rides the continuation point — the last pages of a deep page-through no longer each walk ~14 reads to 1970). The raw/events tie-path oversized-cluster refusal is now `BadTooManyOperations` too (was `BadHistoryOperationUnsupported`). Uncapped Gateway raw reads (forward too) used to send `max_values = 0`, which the client/gateway reject — now unbounded (`int.MaxValue`; a ceiling refusal ⇒ `BadTooManyOperations`), and a cancelled read is no longer a backend failure. Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events follow since #691.4: a reverse/end-only capped events read on Gateway/Null (`SupportsReverseEvents` false) is walked back the same way (ties sorted by EventId — the gateway orders by time only; one extra event found ⇒ continuation point) — it used to be refused `BadHistoryOperationUnsupported` (explicit reverse) or answered with the OLDEST N ascending (end-only); needs the gateway's `RuntimeDb:EventReadMaxRows` 0/unlimited (it truncates silently). `docs/Historian.md` §"Open-ended and reverse raw reads". - ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` serves it from FORWARD reads: uncapped ⇒ ONE forward read reversed in memory; capped ⇒ a bounded walk back in segments (ms-aligned request bounds, width aimed from observed density at ≈ N + 8 rows and carried page to page in the continuation point — #689: it used to aim at half `MaxTieClusterOverfetch`, ~32k rows per page on a dense tag; a read asks for `2N + 18` rows (a discovery segment ≤ 1,026), `MaxTieClusterOverfetch` is only the tie-cluster ceiling — a full read is re-read, never answered truncated; only one timestamp holding more than max(1, `MaxTieClusterOverfetch`) samples ⇒ `BadTooManyOperations`; walks no later than now + 1 day and never before the Unix epoch; #691: after two empty segments it asks once for the whole remainder, and the stretch a page proved empty rides the continuation point — the last pages of a deep page-through no longer each walk ~14 reads to 1970). The raw/events tie-path oversized-cluster refusal is now `BadTooManyOperations` too (was `BadHistoryOperationUnsupported`). Uncapped Gateway raw reads (forward too) used to send `max_values = 0`, which the client/gateway reject — now unbounded (`int.MaxValue`; a ceiling refusal ⇒ `BadTooManyOperations`), and a cancelled read is no longer a backend failure. Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events follow since #691.4: a reverse/end-only capped events read on Gateway (`SupportsReverseEvents` false; Null now reports `true` and answers empty) is walked back the same way (ties sorted by EventId — the gateway orders by time only; one extra event found ⇒ continuation point) — it used to be refused `BadHistoryOperationUnsupported` (explicit reverse) or answered with the OLDEST N ascending (end-only); needs the gateway's `RuntimeDb:EventReadMaxRows` 0/unlimited (it truncates silently). Uncapped Gateway events reads (forward too) used to throw "Array dimensions exceeded" (result list pre-sized to `int.MaxValue`) — fixed. Above a truncated read the walk gallops up from the newest row seen, then bisects (never below 1 ms): a burst that ended 100 days ago ≈ 18 reads, a dense tag that stopped days ago ≈ 26. `docs/Historian.md` §"Open-ended and reverse raw reads".
- **Advertised aggregates (#682):** `IHistorianDataSource.SupportedAggregates` (Gateway 4 via `GatewayHistorianDataSource.ServedAggregates`, SqlServer/QuestDB 17 via `AggregateFamilies.Served`, Null **none**; a null set ⇒ all) is the one authority for `HistoryServerCapabilities/AggregateFunctions` (i=11201 — only its folder references are trimmed) and the up-front `BadAggregateNotSupported` in HistoryRead-Processed. ⚠️ `ServerCapabilities/AggregateFunctions` (i=2997) is deliberately NOT trimmed: it keeps the SDK's 37 because live `AggregateFilter` subscriptions still compute them all. ⚠️ Historian disabled ⇒ Processed is now `BadAggregateNotSupported`, not `GoodNoData`. `docs/Historian.md` §Advertised aggregates. - **Advertised aggregates (#682):** `IHistorianDataSource.SupportedAggregates` (Gateway 4 via `GatewayHistorianDataSource.ServedAggregates`, SqlServer/QuestDB 17 via `AggregateFamilies.Served`, Null **none**; a null set ⇒ all) is the one authority for `HistoryServerCapabilities/AggregateFunctions` (i=11201 — only its folder references are trimmed) and the up-front `BadAggregateNotSupported` in HistoryRead-Processed. ⚠️ `ServerCapabilities/AggregateFunctions` (i=2997) is deliberately NOT trimmed: it keeps the SDK's 37 because live `AggregateFilter` subscriptions still compute them all. ⚠️ Historian disabled ⇒ Processed is now `BadAggregateNotSupported`, not `GoodNoData`. `docs/Historian.md` §Advertised aggregates.
- **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good. - **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good.
- **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node. - **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node.
@@ -1455,6 +1455,9 @@ The target HistorianGateway OtOpcUa points at **must** run with:
- `RuntimeDb:Enabled=true` — enables the `WriteLiveValues` SQL live path (continuous historization). - `RuntimeDb:Enabled=true` — enables the `WriteLiveValues` SQL live path (continuous historization).
- `RuntimeDb:EventReadsEnabled=true` — enables `ReadEvents` from `Runtime.dbo.Events` (alarm-history reads). - `RuntimeDb:EventReadsEnabled=true` — enables `ReadEvents` from `Runtime.dbo.Events` (alarm-history reads).
- `RuntimeDb:EventReadMaxRows=0` (its default, unlimited) — or larger than the reverse-events walk's biggest read
(`2·MaxTieClusterOverfetch + 4`). The cap truncates **silently**, so a walked-back reverse events read (#691.4) would
lose the newest events of a segment it cut.
- An API key carrying scopes **`historian:read`**, **`historian:write`**, **`historian:tags:write`**. - An API key carrying scopes **`historian:read`**, **`historian:write`**, **`historian:tags:write`**.
### Migration note (deployments upgrading from the Wonderware backend) ### Migration note (deployments upgrading from the Wonderware backend)
+9 -6
View File
@@ -654,9 +654,12 @@ backend:
- **Never answered from a truncated read.** Rows stream ascending, so a read is complete when it came back short or its - **Never answered from a truncated read.** Rows stream ascending, so a read is complete when it came back short or its
last row is past the segment; otherwise it is discarded — re-read once with room for the rows before the segment last row is past the segment; otherwise it is discarded — re-read once with room for the rows before the segment
that crowded it (a neighbouring tie cluster, at most twice the budget + 4 rows), or narrowed: to the density it that crowded it (a neighbouring tie cluster, at most twice the budget + 4 rows), or narrowed: to the density it
showed, and never again below the newest row it saw until what lies above that row is read (the gap above it is showed, and never again below the newest row it saw until what lies above that row is read. The gap above it is
bisected — a dense tag that went quiet days ago is found in about 20 reads rather than re-reading the same oldest searched: a segment at that density ending on the top (data running right up to it); if empty, a **gallop** up from
rows of its burst over and over). Only **one timestamp holding more than `M` samples** is refused, that row (`(row + d, top]`, `d` ×4 per step, following each truncated read) until a read comes back empty, then a
**bisection** of the bracket — never narrower than the 1 ms request granularity, which only re-sent the same request.
Measured: a dense tag that stopped two days ago ≈ **26** reads, a burst of just over 1,024 rows that ended 100 days ago
≈ **18** (was 46 bisecting the whole gap) — rather than re-reading the oldest rows of a burst over and over. Only **one timestamp holding more than `M` samples** is refused,
**`BadTooManyOperations`** (a request limit, not a backend fault); a cluster of exactly `M`, or neighbouring **`BadTooManyOperations`** (a request limit, not a backend fault); a cluster of exactly `M`, or neighbouring
clusters each within `M`, is served. clusters each within `M`, is served.
- **Proving the rest empty (#691).** Walking down through empty history to the floor used to cost one read per ×4 - **Proving the rest empty (#691).** Walking down through empty history to the floor used to cost one read per ×4
@@ -778,8 +781,8 @@ says is complete carries no continuation point. `NumValuesPerNode == 0` stays un
is forward `[MinValue, EndTime)` everywhere. is forward `[MinValue, EndTime)` everywhere.
- **Backends.** The server checks `IHistorianDataSource.SupportsReverseEvents`. SqlServer and QuestDB serve reverse natively - **Backends.** The server checks `IHistorianDataSource.SupportsReverseEvents`. SqlServer and QuestDB serve reverse natively
(`… > lo AND … <= hi ORDER BY time DESC, id DESC`, the `ReadEventsAsync(…, bool reverse, …)` overload) and report `true`. (`… > lo AND … <= hi ORDER BY time DESC, id DESC`, the `ReadEventsAsync(…, bool reverse, …)` overload) and report `true`.
**Gateway and Null report `false`** — the gateway streams ascending only and rejects `end < start` — and are never sent a **Gateway reports `false`** — the gateway streams ascending only and rejects `end < start` — and is never sent a reverse
reverse read: since **#691.4** the server walks it back from bounded **forward** reads with the raw walk (#686/#689/#691: read (Null reports `true` and answers empty either way, as for raw): since **#691.4** the server walks it back from bounded **forward** reads with the raw walk (#686/#689/#691:
page-sized ms-aligned segments filtered to `(segLo, segHi]`, a read the source flagged truncated is never answered from, page-sized ms-aligned segments filtered to `(segLo, segHi]`, a read the source flagged truncated is never answered from,
the now + 1 day / 1970 bounds, the remainder probe and the proven-empty stretch carried in the cursor). Event specifics: the now + 1 day / 1970 bounds, the remainder probe and the proven-empty stretch carried in the cursor). Event specifics:
- **Ties** are sorted by `EventId` (ordinal) on every read — the gateway orders by `EventTimeUtc` only, in any order within a - **Ties** are sorted by `EventId` (ordinal) on every read — the gateway orders by `EventTimeUtc` only, in any order within a
@@ -796,7 +799,7 @@ says is complete carries no continuation point. `NumValuesPerNode == 0` stays un
That cap truncates **silently** (the stream just ends), so a segment it cut would look complete and drop its newest events — That cap truncates **silently** (the stream just ends), so a segment it cut would look complete and drop its newest events —
the same blindness a forward read has to it, but on the end a reverse read is asking for. the same blindness a forward read has to it, but on the end a reverse read is asking for.
⚠️ **Client-visible (#691.4).** On Gateway (the default backend) an explicit reverse events read used to be refused ⚠️ **Client-visible (#691.4).** On Gateway (the default backend) and Null an explicit reverse events read used to be refused
`BadHistoryOperationUnsupported` (#675), and an end-only capped events read returned the **oldest** N events ascending. Both `BadHistoryOperationUnsupported` (#675), and an end-only capped events read returned the **oldest** N events ascending. Both
now return the latest events newest-first. Because the capability is checked, not a caught exception, a now return the latest events newest-first. Because the capability is checked, not a caught exception, a
`NotSupportedException` thrown from inside a backend (a driver library) is still logged as a backend fault. Every `NotSupportedException` thrown from inside a backend (a driver library) is still logged as a backend fault. Every
@@ -156,10 +156,19 @@ internal static class ReverseWalk
var discoveryBudget = Math.Min(budget, Math.Max(pageBudget, DiscoveryRows)); var discoveryBudget = Math.Min(budget, Math.Max(pageBudget, DiscoveryRows));
// `known` > 0: a truncated read saw rows up to this timestamp (below segHi), and nothing is known of (known, segHi]. // `known` > 0: a truncated read saw rows up to this timestamp (below segHi), and nothing is known of (known, segHi].
// Segments stay above it (bisecting the gap) until that stretch is read; then `knownWidth` (the density the // Segments stay above it until that stretch is read; then `knownWidth` (the density the truncated read showed)
// truncated read showed) sizes the segment that ends on it. // sizes the segment that ends on it. Searching the gap: first a top-anchored segment at that density (data running
// right up to segHi); if it is empty, the data stops somewhere above `known`, so GALLOP up from it — read
// (known + lowerBy, segHi] with lowerBy growing ×4, `known` following what each truncated read saw — until a read
// comes back empty (the top is now bracketed below its segLo), then BISECT that bracket. Any complete read holding
// rows ends the search. A burst that ended long before segHi is found in a few reads instead of bisecting the whole
// gap down to the millisecond, and a long dense stretch is crossed geometrically, not a budget at a time (#691.4
// review). `noGallop` falls back to plain top-anchored halving if a galloping read is crowded by rows before it.
var known = 0L; var known = 0L;
var knownWidth = 0L; var knownWidth = 0L;
var lowerBy = 0L;
var bisecting = false;
var noGallop = false;
var segHi = hiTicks; var segHi = hiTicks;
// #691: the remainder probe (once per page, after two empty segments) and the empty stretches. // #691: the remainder probe (once per page, after two empty segments) and the empty stretches.
@@ -178,7 +187,20 @@ internal static class ReverseWalk
var probe = !probed && known == 0 && emptyRun >= 2 && segHi - stopTicks > width; var probe = !probed && known == 0 && emptyRun >= 2 && segHi - stopTicks > width;
var w = probe ? segHi - stopTicks : width; var w = probe ? segHi - stopTicks : width;
if (known > 0) w = Math.Min(w, Math.Max(1, (segHi - known) / 2)); var lowered = false;
var bisected = false;
if (known > 0)
{
var gap = segHi - known;
lowered = lowerBy > 0 && lowerBy < gap;
bisected = !lowered && (bisecting || lowerBy > 0) && gap > 2 * Math.Max(granularity, knownWidth);
// Never narrower than the request granularity when halving: requests are rounded outward to it, so a
// narrower half re-sends the same request (#691.4 review). Keep the Min: a truncated read crowded by rows
// before the segment must still shrink the segment.
w = lowered ? gap - lowerBy
: bisected ? gap / 2
: Math.Min(w, gap <= 2 * granularity ? gap : gap / 2);
}
var segLo = segHi - stopTicks <= w ? stopTicks : segHi - w; var segLo = segHi - stopTicks <= w ? stopTicks : segHi - w;
var start = Utc(AlignDown(segLo, granularity)); var start = Utc(AlignDown(segLo, granularity));
var end = Utc(AlignUp(segHi + 1, granularity)); var end = Utc(AlignUp(segHi + 1, granularity));
@@ -244,6 +266,14 @@ internal static class ReverseWalk
if (inner > 0 && lastInner == segHi) if (inner > 0 && lastInner == segHi)
{ {
width = resolution; // a tie cluster at the top of the segment: read that one timestamp width = resolution; // a tie cluster at the top of the segment: read that one timestamp
lowerBy = 0;
bisecting = false;
}
else if (inner > 0 && (lowered || bisected))
{
// Data reaches past this segment's bottom: move `known` up to what this read saw; a gallop goes further.
known = lastInner;
if (lowered) lowerBy = lowerBy > maxWidth / 4 ? maxWidth : lowerBy * 4;
} }
else if (inner > 0) else if (inner > 0)
{ {
@@ -255,10 +285,17 @@ internal static class ReverseWalk
known = lastInner; known = lastInner;
knownWidth = Math.Max(resolution, aimed); knownWidth = Math.Max(resolution, aimed);
width = Math.Max(resolution, Math.Min(aimed, segHi - segLo - 1)); width = Math.Max(resolution, Math.Min(aimed, segHi - segLo - 1));
lowerBy = 0;
bisecting = false;
} }
else else
{ {
width = Math.Max(resolution, (segHi - segLo) / 4); // only rows before the segment: shrink it // Only rows before the segment: shrink it — top-anchored from here on (a gallop would re-send the same
// crowded read).
width = Math.Max(resolution, Math.Min(width, segHi - segLo) / 4);
if (lowered || bisected) noGallop = true;
lowerBy = 0;
bisecting = false;
} }
discovering = false; discovering = false;
@@ -283,9 +320,29 @@ internal static class ReverseWalk
var grown = width > maxWidth / 4 ? maxWidth : width * 4; var grown = width > maxWidth / 4 ? maxWidth : width * 4;
if (count == 0) if (count == 0)
{ {
// ×4 growth may overshoot into dense data; above a known row, bisect the rest of the gap instead. discovering = true; // ×4 growth may overshoot into dense data
width = known > 0 ? maxWidth : grown; if (known > 0)
discovering = true; {
// An empty top-anchored segment: gallop up from `known`. An empty galloping read: the top is bracketed
// in (known, segLo] — bisect it (an empty bisecting read keeps bisecting).
if (lowered || bisected)
{
lowerBy = 0;
bisecting = true;
}
else if (noGallop)
{
width = maxWidth; // halve the gap, top-anchored
}
else
{
lowerBy = Math.Max(knownWidth, granularity);
}
}
else
{
width = grown;
}
} }
else else
{ {
@@ -293,6 +350,9 @@ internal static class ReverseWalk
width = (long)Math.Clamp((double)(segHi - segLo) * Math.Max(1.0, pageBudget / 2.0) / count, resolution, grown); width = (long)Math.Clamp((double)(segHi - segLo) * Math.Max(1.0, pageBudget / 2.0) / count, resolution, grown);
discovering = false; discovering = false;
known = 0; known = 0;
lowerBy = 0;
bisecting = false;
noGallop = false;
} }
segHi = segLo; segHi = segLo;
@@ -300,6 +360,9 @@ internal static class ReverseWalk
{ {
// Everything above the newest row the truncated read saw is read: the next segment ends on it, at its density. // Everything above the newest row the truncated read saw is read: the next segment ends on it, at its density.
known = 0; known = 0;
lowerBy = 0;
bisecting = false;
noGallop = false;
width = knownWidth; width = knownWidth;
discovering = false; discovering = false;
} }
@@ -120,7 +120,7 @@ public sealed class ReverseWalkTests
/// <summary> /// <summary>
/// #689: a dense tag that stopped long ago. The walk's ×4 growth across the empty stretch overshoots into far more /// #689: a dense tag that stopped long ago. The walk's ×4 growth across the empty stretch overshoots into far more
/// rows than one read takes; the truncated read only shows the OLDEST of them, so the walk must not keep re-entering /// rows than one read takes; the truncated read only shows the OLDEST of them, so the walk must not keep re-entering
/// that stretch — it bisects the gap above the newest row it saw. /// that stretch — it searches the gap above the newest row it saw (gallop up, then bisect). Measured: 26 reads.
/// </summary> /// </summary>
[Fact] [Fact]
public async Task A_dense_tag_below_a_long_empty_stretch_is_found_in_a_bounded_number_of_reads() public async Task A_dense_tag_below_a_long_empty_stretch_is_found_in_a_bounded_number_of_reads()
@@ -133,8 +133,8 @@ public sealed class ReverseWalkTests
var page = await ReverseRawReader.ReadAsync(source, "T", T0, DateTime.MinValue, 3, 65536, T0, default(ReverseWalkHint), Ct); var page = await ReverseRawReader.ReadAsync(source, "T", T0, DateTime.MinValue, 3, 65536, T0, default(ReverseWalkHint), Ct);
page.Samples.Select(s => (double)s.Value!).ShouldBe([107_999d, 107_998d, 107_997d]); page.Samples.Select(s => (double)s.Value!).ShouldBe([107_999d, 107_998d, 107_997d]);
source.Reads.ShouldBeLessThanOrEqualTo(25); source.Reads.ShouldBeLessThanOrEqualTo(30);
source.RowsReturned.ShouldBeLessThanOrEqualTo(25 * (ReverseWalk.DiscoveryRows + 2)); source.RowsReturned.ShouldBeLessThanOrEqualTo(30 * (ReverseWalk.DiscoveryRows + 2));
} }
/// <summary> /// <summary>
@@ -185,6 +185,27 @@ public sealed class ReverseWalkTests
} }
} }
/// <summary>
/// #691.4 review: a burst just over the discovery read, sub-millisecond spaced, ending 100 days ago. The search for
/// the top of the burst above the truncated read never re-sends an identical request (halving below the 1 ms request
/// granularity did), and gallops up from the truncated read's newest row instead of bisecting the whole 100-day gap
/// down to the millisecond (was 46 reads).
/// </summary>
[Fact]
public async Task Bisecting_above_a_truncated_read_never_resends_an_identical_request()
{
var end = T0.AddDays(-100);
var count = ReverseWalk.DiscoveryRows + 2;
var series = Enumerable.Range(0, count).Select(i => At(end.Ticks - (count - 1 - i) * 1_000, i)).ToList(); // 0.1 ms apart
var source = new AvevaShapedSource(series);
var page = await ReverseRawReader.ReadAsync(source, "T", T0, DateTime.MinValue, 3, 65536, T0, default(ReverseWalkHint), Ct);
page.Samples.Select(s => (double)s.Value!).ShouldBe([count - 1d, count - 2d, count - 3d]);
source.RepeatedRequests.ShouldBe(0, string.Join("\n", source.Log));
source.Reads.ShouldBeLessThanOrEqualTo(22, string.Join("\n", source.Log));
}
private static long Ts(DataValueSnapshot s) => s.SourceTimestampUtc!.Value.Ticks; private static long Ts(DataValueSnapshot s) => s.SourceTimestampUtc!.Value.Ticks;
/// <summary>Random ascending series: bursts, gaps, sub-millisecond stamps and tie clusters.</summary> /// <summary>Random ascending series: bursts, gaps, sub-millisecond stamps and tie clusters.</summary>
@@ -221,7 +242,15 @@ public sealed class ReverseWalkTests
public int Reads { get; private set; } public int Reads { get; private set; }
public int BadRequests { get; private set; } public int BadRequests { get; private set; }
public void Reset() { MaxCap = 0; RowsReturned = 0; Reads = 0; } /// <summary>Requests that repeated an earlier request exactly (start, end, cap) — read-only, so pure waste.</summary>
public int RepeatedRequests { get; private set; }
private readonly HashSet<(DateTime, DateTime, uint)> _seen = [];
/// <summary>Every request, for failure messages.</summary>
public List<string> Log { get; } = [];
public void Reset() { MaxCap = 0; RowsReturned = 0; Reads = 0; RepeatedRequests = 0; _seen.Clear(); Log.Clear(); }
public bool SupportsReverseRaw => false; public bool SupportsReverseRaw => false;
@@ -229,6 +258,9 @@ public sealed class ReverseWalkTests
CancellationToken cancellationToken) CancellationToken cancellationToken)
{ {
Reads++; Reads++;
if (!_seen.Add((startUtc, endUtc, maxValuesPerNode))) RepeatedRequests++;
Log.Add($"[{startUtc:O} .. {endUtc:O}] cap {maxValuesPerNode}");
if (startUtc > endUtc || maxValuesPerNode == 0 || startUtc < DateTime.UnixEpoch) BadRequests++; if (startUtc > endUtc || maxValuesPerNode == 0 || startUtc < DateTime.UnixEpoch) BadRequests++;
MaxCap = Math.Max(MaxCap, maxValuesPerNode); MaxCap = Math.Max(MaxCap, maxValuesPerNode);
var stamp = roundUp && startUtc.Ticks % TimeSpan.TicksPerMillisecond != 0 var stamp = roundUp && startUtc.Ticks % TimeSpan.TicksPerMillisecond != 0