fix(historian): #691.4 walk reverse events reads back on forward-only backends (Gateway)

A reverse or end-only capped events read on a backend without
SupportsReverseEvents (Gateway, Null) was refused
BadHistoryOperationUnsupported (explicit reverse) or answered with the
OLDEST N ascending (end-only, #675). It is now served newest-first by
the same ReverseWalk as raw reads: page-sized ms-aligned segments,
never answered from a read the source flagged truncated, ties sorted by
EventId (the gateway orders by event time only - the fake shuffles
them per call), one extra event walked to signal 'more', uncapped = one
forward read reversed, and the width + proven-empty stretch carried in
the events cursor. The oversized-cluster over-fetch keeps exactly the
events at T in the same order. A tie cluster over the budget is
BadTooManyOperations. Docs note the gateway's EventReadMaxRows must be
unlimited (it truncates silently).
This commit is contained in:
Joseph Doherty
2026-10-12 16:48:05 -04:00
parent 186b7d5ea5
commit d277f34a96
9 changed files with 251 additions and 82 deletions
+31 -19
View File
@@ -677,8 +677,7 @@ backend:
> ⚠️ **Client-visible (#686).** Since #669 an end-only capped raw read, and any explicit reverse raw read, on the Gateway
> backend failed the node (`BadHistoryOperationUnsupported`) and was counted as a historian backend failure (the gateway
> refused `start > end`). Before #669 an end-only read on Gateway returned the **oldest** N values ascending. Both now return
> the Part 11 answer — the latest values at or before `EndTime`, newest first. **Events differ:** an end-only events read on
> Gateway still keeps its forward meaning and an explicit reverse events read is refused (see below).
> the Part 11 answer — the latest values at or before `EndTime`, newest first. **Events** follow since #691.4 (see below).
>
> ⚠️ **Also fixed with #686: uncapped raw reads on Gateway.** `NumValuesPerNode = 0` ("all values") used to be forwarded as
> `max_values = 0`, which the client package and the gateway reject — every uncapped raw read (forward too) failed the node
@@ -763,30 +762,43 @@ says is complete carries no continuation point. `NumValuesPerNode == 0` stays un
that many more rows than the cap, so every page short of the last is a FULL `NumValuesPerNode` page (#675; resumed pages used
to come back short by the skip).
- A tie cluster larger than the cap is paged within by over-fetching the half-open window `[T, T + resolution)` up to
`MaxTieClusterOverfetch`; beyond that bound the node answers `BadHistoryOperationUnsupported` (raise the bound).
`MaxTieClusterOverfetch`; beyond that bound the node answers `BadTooManyOperations` (raise the bound).
- **Tie exactness depends on the backend.** SqlServer and QuestDB order by `(EventTimeUtc, EventId)`, so the skip count is exact.
The Gateway backend has no stable intra-timestamp order (and only a truncation flag), so ties at a page boundary are
best-effort there.
The Gateway backend has no stable intra-timestamp order (and only a truncation flag), so ties at a page boundary of a
**forward** read are best-effort there; a walked-back **reverse** read (below) sorts them by `EventId` itself and is exact.
- **Reverse event reads page newest-first (#675).** `StartTime > EndTime` reads the Part 11 reverse window `(EndTime, StartTime]`
with events in DESCENDING `(EventTimeUtc, EventId)` order, so a cap keeps the **latest** N and a continuation point is issued
when the source flags more. The direction is stored **in the cursor**: a reverse point resumes reverse (and a forward one
forward) whatever the resume request's details say. Reverse tie clusters are over-fetched forward and sliced from the newest
end, exactly as Raw does. Before #675 a capped reverse read silently returned the EARLIEST N, ascending, Good, with no
continuation point.
- **End-only events reads: backward on SqlServer/QuestDB, forward on Gateway (#675).** `StartTime` unset
(`DateTime.MinValue`) with an `EndTime` and `NumValuesPerNode > 0` pages backward from `EndTime` over `(MinValue, EndTime]` —
the same rule as Raw — **only on a backend that reports `SupportsReverseEvents`** (SqlServer, QuestDB). On Gateway (the
default backend) and Null it keeps its pre-#675 forward `[MinValue, EndTime)` meaning, paged oldest-first, rather than
becoming a refused reverse read. Start-only reads page forward to an open upper bound (`DateTime.MaxValue`, UTC) everywhere.
Without a cap an end-only read is forward `[MinValue, EndTime)` everywhere.
- **Backends.** The server checks `IHistorianDataSource.SupportsReverseEvents` and then calls
`ReadEventsAsync(…, bool reverse, …)`. SqlServer and QuestDB serve reverse natively (`… > lo AND … <= hi ORDER BY time DESC,
id DESC`) and report `true`. **Gateway and Null report `false`**: an explicit reverse read (`StartTime > EndTime`) is answered
`BadHistoryOperationUnsupported` **without calling the backend** — the gateway streams ascending only and rejects `end < start`,
and draining the whole window to keep its tail would be exactly the unbounded read the cap prevents. Because the refusal is a
capability check, not a caught exception, a `NotSupportedException` thrown from inside a backend (a driver library) is logged
as a backend fault like any other. Every implementation (and any future wrapper) must declare both members itself — the
interface defaults (`false` / throw) would silently disable reverse reads; `HistorianDataSourceReverseEventsGuardTests` pins it.
- **End-only events reads page backward on every backend (#675, #691.4).** `StartTime` unset (`DateTime.MinValue`) with an
`EndTime` and `NumValuesPerNode > 0` pages backward from `EndTime` over `(MinValue, EndTime]` — the same rule as Raw.
Start-only reads page forward to an open upper bound (`DateTime.MaxValue`, UTC) everywhere. Without a cap an end-only read
is forward `[MinValue, EndTime)` everywhere.
- **Backends.** The server checks `IHistorianDataSource.SupportsReverseEvents`. SqlServer and QuestDB serve reverse natively
(`… > lo AND … <= hi ORDER BY time DESC, id DESC`, the `ReadEventsAsync(…, bool reverse, …)` overload) and report `true`.
**Gateway and Null report `false`** — the gateway streams ascending only and rejects `end < start` — and are never sent a
reverse read: since **#691.4** the server walks it back from bounded **forward** reads with the raw walk (#686/#689/#691:
page-sized ms-aligned segments filtered to `(segLo, segHi]`, a read the source flagged truncated is never answered from,
the now + 1 day / 1970 bounds, the remainder probe and the proven-empty stretch carried in the cursor). Event specifics:
- **Ties** are sorted by `EventId` (ordinal) on every read — the gateway orders by `EventTimeUtc` only, in any order within a
timestamp — so the page is DESCENDING `(EventTimeUtc, EventId)` and a resumed page's boundary skip lines up; the
oversized-cluster over-fetch keeps exactly the events at `T` (the gateway's end is inclusive) in the same order.
- **"More exists"** is found by walking for one event past the page (the walk has no source truncation flag to hand on).
- **Uncapped** reverse reads are one forward read of the window, sorted and reversed.
- A timestamp holding more than `MaxTieClusterOverfetch` events ⇒ **`BadTooManyOperations`** for that node.
- ⚠️ **Gateway prerequisite:** the gateway's `RuntimeDb:EventReadMaxRows` must be `0` (its default, unlimited) or larger than
the walk's biggest read (`2N + 18` rows, a discovery read 1,026, a tie-cluster read up to `2·MaxTieClusterOverfetch + 4`).
That cap truncates **silently** (the stream just ends), so a segment it cut would look complete and drop its newest events —
the same blindness a forward read has to it, but on the end a reverse read is asking for.
⚠️ **Client-visible (#691.4).** On Gateway (the default backend) an explicit reverse events read used to be refused
`BadHistoryOperationUnsupported` (#675), and an end-only capped events read returned the **oldest** N events ascending. Both
now return the latest events newest-first. Because the capability is checked, not a caught exception, a
`NotSupportedException` thrown from inside a backend (a driver library) is still logged as a backend fault. Every
implementation (and any future wrapper) must declare both members itself — the interface defaults (`false` / throw) would
silently send a native-reverse backend through the walk; `HistorianDataSourceReverseEventsGuardTests` pins it.
- **Gateway open upper bound.** The `DateTime.MaxValue` (UTC) bound of a start-only forward read is pinned offline through the
published client's proto `Timestamp` conversion (captured by an in-process gRPC server) and the FILETIME encoding
(`GatewayEventBoundsTests`). What a live AVEVA Historian does with a year-9999 bound is still unverified (needs the VPN).