fix(historian): #691.4 walk reverse events reads back on forward-only backends (Gateway)

A reverse or end-only capped events read on a backend without
SupportsReverseEvents (Gateway, Null) was refused
BadHistoryOperationUnsupported (explicit reverse) or answered with the
OLDEST N ascending (end-only, #675). It is now served newest-first by
the same ReverseWalk as raw reads: page-sized ms-aligned segments,
never answered from a read the source flagged truncated, ties sorted by
EventId (the gateway orders by event time only - the fake shuffles
them per call), one extra event walked to signal 'more', uncapped = one
forward read reversed, and the width + proven-empty stretch carried in
the events cursor. The oversized-cluster over-fetch keeps exactly the
events at T in the same order. A tie cluster over the budget is
BadTooManyOperations. Docs note the gateway's EventReadMaxRows must be
unlimited (it truncates silently).
This commit is contained in:
Joseph Doherty
2026-10-12 16:48:05 -04:00
parent 186b7d5ea5
commit d277f34a96
9 changed files with 251 additions and 82 deletions
+1 -1
View File
@@ -1353,7 +1353,7 @@ historian tagname, which is the tag's driver `FullName`.
- **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; per-sample aggregates (10, simple definitions) are pushed down to T-SQL with bucket math in **100 ns ticks**; the Part 13 time-weighted seven (TimeAverage/TimeAverage2/Total/Total2/Interpolative, #671 — ⚠️ `Total` is now the time-weighted integral, no longer `SUM` — plus the **duration-based** PercentGood/PercentBad, #680) stream window + 4 bound rows into the shared `TimeWeightedAggregator` (both SQL backends, identical results; reverse `Interpolative` anchored at the later time; window capped by `ServerHistorian:MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations`, percentages included). ⚠️ **#680 status rules (client-visible):** TimeAverage/Total are Uncertain if any part of the interval uses an Uncertain value, is bridged across a Bad sample, is extrapolated past the last sample (⇒ a live tag's latest interval is Uncertain) or is not covered; TimeAverage2/Total2 keep the SDK time-based rule (Uncertain time counts as Good) but never extrapolate past the last sample — that interval is Partial, later ones `BadNoData` (a report-by-exception tag with nothing after its last sample reads `BadNoData` until its next sample arrives). `docs/Historian.md` §"Time-weighted aggregates". ⚠️ **#680 per-sample (client-visible):** `Count` counts **Good** samples only (value-based status; an all-Bad interval is generic Bad, value null) and an empty interval **inside the tag's data range** is `0`/Good (outside it still `BadNoData`); `Start`/`End` return the **raw** first/last sample of any quality — its own timestamp and status, value null when Bad; `Delta` is Uncertain only when the interval's first or last sample is unusable; ⚠️ #684: `Average`/`Minimum`/`Maximum`/`Range`/`*ActualTime` use **Good samples only** (was Good + Uncertain; an Uncertain-only interval is `BadNoData`), Min/Max-family status Uncertain on a Bad sample, an Uncertain value beyond the Good extreme (Part 13 — the SDK ignores that) or an Uncertain non-numeric sample, and only a **Good** String/DateTime sample refuses them (`BadAggregateInvalidInputs`; Delta still refuses any usable one); and every per-sample aggregate but `Average` sets **end-of-data Partial** on the interval holding the last sample (forward incl. its earlier edge, reverse strictly inside), never on a Bad result. The data-range lookup is a key seek on SQL Server and a `tag` symbol-index seek on QuestDB (#683 — `{p}sample.tag` is `SYMBOL INDEX`; the bootstrapper ADDs it to a pre-#683 table, idempotent), read BEFORE the bucket statement so a racing append can never yield a Good `Count` of 0 (a concurrent HistoryUpdate delete is the mirror case and still can, transiently). `docs/Historian.md` §"Aggregates". Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. **Retention (#667):** opt-in `SqlServer:RetentionDays` (+ `PurgeInterval` 1h, `PurgeBatchSize` 4000) -> `HistorianRetentionService` purges old `hist.Sample` (per tag) / `hist.Event` in `DELETE TOP(n)` batches, Primary-gated on `IHistorizationRoleView`, one purger per DB via `sp_getapplock`; counter `otopcua.historian.retention.purged{table}`. Optional `IHistorianBackendProvider.CreateRetention` (default null). `docs/Historian.md` §Retention.
- **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend".
- ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); on forward reads Partial also marks the interval the data starts inside (not Average — the SDK's rule, forward-only); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates.
- ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` serves it from FORWARD reads: uncapped ⇒ ONE forward read reversed in memory; capped ⇒ a bounded walk back in segments (ms-aligned request bounds, width aimed from observed density at ≈ N + 8 rows and carried page to page in the continuation point — #689: it used to aim at half `MaxTieClusterOverfetch`, ~32k rows per page on a dense tag; a read asks for `2N + 18` rows (a discovery segment ≤ 1,026), `MaxTieClusterOverfetch` is only the tie-cluster ceiling — a full read is re-read, never answered truncated; only one timestamp holding more than max(1, `MaxTieClusterOverfetch`) samples ⇒ `BadTooManyOperations`; walks no later than now + 1 day and never before the Unix epoch; #691: after two empty segments it asks once for the whole remainder, and the stretch a page proved empty rides the continuation point — the last pages of a deep page-through no longer each walk ~14 reads to 1970). The raw/events tie-path oversized-cluster refusal is now `BadTooManyOperations` too (was `BadHistoryOperationUnsupported`). Uncapped Gateway raw reads (forward too) used to send `max_values = 0`, which the client/gateway reject — now unbounded (`int.MaxValue`; a ceiling refusal ⇒ `BadTooManyOperations`), and a cancelled read is no longer a backend failure. Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events still differ (end-only forward / explicit reverse refused on Gateway, #675). `docs/Historian.md` §"Open-ended and reverse raw reads".
- ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` serves it from FORWARD reads: uncapped ⇒ ONE forward read reversed in memory; capped ⇒ a bounded walk back in segments (ms-aligned request bounds, width aimed from observed density at ≈ N + 8 rows and carried page to page in the continuation point — #689: it used to aim at half `MaxTieClusterOverfetch`, ~32k rows per page on a dense tag; a read asks for `2N + 18` rows (a discovery segment ≤ 1,026), `MaxTieClusterOverfetch` is only the tie-cluster ceiling — a full read is re-read, never answered truncated; only one timestamp holding more than max(1, `MaxTieClusterOverfetch`) samples ⇒ `BadTooManyOperations`; walks no later than now + 1 day and never before the Unix epoch; #691: after two empty segments it asks once for the whole remainder, and the stretch a page proved empty rides the continuation point — the last pages of a deep page-through no longer each walk ~14 reads to 1970). The raw/events tie-path oversized-cluster refusal is now `BadTooManyOperations` too (was `BadHistoryOperationUnsupported`). Uncapped Gateway raw reads (forward too) used to send `max_values = 0`, which the client/gateway reject — now unbounded (`int.MaxValue`; a ceiling refusal ⇒ `BadTooManyOperations`), and a cancelled read is no longer a backend failure. Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events follow since #691.4: a reverse/end-only capped events read on Gateway/Null (`SupportsReverseEvents` false) is walked back the same way (ties sorted by EventId — the gateway orders by time only; one extra event found ⇒ continuation point) — it used to be refused `BadHistoryOperationUnsupported` (explicit reverse) or answered with the OLDEST N ascending (end-only); needs the gateway's `RuntimeDb:EventReadMaxRows` 0/unlimited (it truncates silently). `docs/Historian.md` §"Open-ended and reverse raw reads".
- **Advertised aggregates (#682):** `IHistorianDataSource.SupportedAggregates` (Gateway 4 via `GatewayHistorianDataSource.ServedAggregates`, SqlServer/QuestDB 17 via `AggregateFamilies.Served`, Null **none**; a null set ⇒ all) is the one authority for `HistoryServerCapabilities/AggregateFunctions` (i=11201 — only its folder references are trimmed) and the up-front `BadAggregateNotSupported` in HistoryRead-Processed. ⚠️ `ServerCapabilities/AggregateFunctions` (i=2997) is deliberately NOT trimmed: it keeps the SDK's 37 because live `AggregateFilter` subscriptions still compute them all. ⚠️ Historian disabled ⇒ Processed is now `BadAggregateNotSupported`, not `GoodNoData`. `docs/Historian.md` §Advertised aggregates.
- **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good.
- **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node.
+31 -19
View File
@@ -677,8 +677,7 @@ backend:
> ⚠️ **Client-visible (#686).** Since #669 an end-only capped raw read, and any explicit reverse raw read, on the Gateway
> backend failed the node (`BadHistoryOperationUnsupported`) and was counted as a historian backend failure (the gateway
> refused `start > end`). Before #669 an end-only read on Gateway returned the **oldest** N values ascending. Both now return
> the Part 11 answer — the latest values at or before `EndTime`, newest first. **Events differ:** an end-only events read on
> Gateway still keeps its forward meaning and an explicit reverse events read is refused (see below).
> the Part 11 answer — the latest values at or before `EndTime`, newest first. **Events** follow since #691.4 (see below).
>
> ⚠️ **Also fixed with #686: uncapped raw reads on Gateway.** `NumValuesPerNode = 0` ("all values") used to be forwarded as
> `max_values = 0`, which the client package and the gateway reject — every uncapped raw read (forward too) failed the node
@@ -763,30 +762,43 @@ says is complete carries no continuation point. `NumValuesPerNode == 0` stays un
that many more rows than the cap, so every page short of the last is a FULL `NumValuesPerNode` page (#675; resumed pages used
to come back short by the skip).
- A tie cluster larger than the cap is paged within by over-fetching the half-open window `[T, T + resolution)` up to
`MaxTieClusterOverfetch`; beyond that bound the node answers `BadHistoryOperationUnsupported` (raise the bound).
`MaxTieClusterOverfetch`; beyond that bound the node answers `BadTooManyOperations` (raise the bound).
- **Tie exactness depends on the backend.** SqlServer and QuestDB order by `(EventTimeUtc, EventId)`, so the skip count is exact.
The Gateway backend has no stable intra-timestamp order (and only a truncation flag), so ties at a page boundary are
best-effort there.
The Gateway backend has no stable intra-timestamp order (and only a truncation flag), so ties at a page boundary of a
**forward** read are best-effort there; a walked-back **reverse** read (below) sorts them by `EventId` itself and is exact.
- **Reverse event reads page newest-first (#675).** `StartTime > EndTime` reads the Part 11 reverse window `(EndTime, StartTime]`
with events in DESCENDING `(EventTimeUtc, EventId)` order, so a cap keeps the **latest** N and a continuation point is issued
when the source flags more. The direction is stored **in the cursor**: a reverse point resumes reverse (and a forward one
forward) whatever the resume request's details say. Reverse tie clusters are over-fetched forward and sliced from the newest
end, exactly as Raw does. Before #675 a capped reverse read silently returned the EARLIEST N, ascending, Good, with no
continuation point.
- **End-only events reads: backward on SqlServer/QuestDB, forward on Gateway (#675).** `StartTime` unset
(`DateTime.MinValue`) with an `EndTime` and `NumValuesPerNode > 0` pages backward from `EndTime` over `(MinValue, EndTime]` —
the same rule as Raw — **only on a backend that reports `SupportsReverseEvents`** (SqlServer, QuestDB). On Gateway (the
default backend) and Null it keeps its pre-#675 forward `[MinValue, EndTime)` meaning, paged oldest-first, rather than
becoming a refused reverse read. Start-only reads page forward to an open upper bound (`DateTime.MaxValue`, UTC) everywhere.
Without a cap an end-only read is forward `[MinValue, EndTime)` everywhere.
- **Backends.** The server checks `IHistorianDataSource.SupportsReverseEvents` and then calls
`ReadEventsAsync(…, bool reverse, …)`. SqlServer and QuestDB serve reverse natively (`… > lo AND … <= hi ORDER BY time DESC,
id DESC`) and report `true`. **Gateway and Null report `false`**: an explicit reverse read (`StartTime > EndTime`) is answered
`BadHistoryOperationUnsupported` **without calling the backend** — the gateway streams ascending only and rejects `end < start`,
and draining the whole window to keep its tail would be exactly the unbounded read the cap prevents. Because the refusal is a
capability check, not a caught exception, a `NotSupportedException` thrown from inside a backend (a driver library) is logged
as a backend fault like any other. Every implementation (and any future wrapper) must declare both members itself — the
interface defaults (`false` / throw) would silently disable reverse reads; `HistorianDataSourceReverseEventsGuardTests` pins it.
- **End-only events reads page backward on every backend (#675, #691.4).** `StartTime` unset (`DateTime.MinValue`) with an
`EndTime` and `NumValuesPerNode > 0` pages backward from `EndTime` over `(MinValue, EndTime]` — the same rule as Raw.
Start-only reads page forward to an open upper bound (`DateTime.MaxValue`, UTC) everywhere. Without a cap an end-only read
is forward `[MinValue, EndTime)` everywhere.
- **Backends.** The server checks `IHistorianDataSource.SupportsReverseEvents`. SqlServer and QuestDB serve reverse natively
(`… > lo AND … <= hi ORDER BY time DESC, id DESC`, the `ReadEventsAsync(…, bool reverse, …)` overload) and report `true`.
**Gateway and Null report `false`** — the gateway streams ascending only and rejects `end < start` — and are never sent a
reverse read: since **#691.4** the server walks it back from bounded **forward** reads with the raw walk (#686/#689/#691:
page-sized ms-aligned segments filtered to `(segLo, segHi]`, a read the source flagged truncated is never answered from,
the now + 1 day / 1970 bounds, the remainder probe and the proven-empty stretch carried in the cursor). Event specifics:
- **Ties** are sorted by `EventId` (ordinal) on every read — the gateway orders by `EventTimeUtc` only, in any order within a
timestamp — so the page is DESCENDING `(EventTimeUtc, EventId)` and a resumed page's boundary skip lines up; the
oversized-cluster over-fetch keeps exactly the events at `T` (the gateway's end is inclusive) in the same order.
- **"More exists"** is found by walking for one event past the page (the walk has no source truncation flag to hand on).
- **Uncapped** reverse reads are one forward read of the window, sorted and reversed.
- A timestamp holding more than `MaxTieClusterOverfetch` events ⇒ **`BadTooManyOperations`** for that node.
- ⚠️ **Gateway prerequisite:** the gateway's `RuntimeDb:EventReadMaxRows` must be `0` (its default, unlimited) or larger than
the walk's biggest read (`2N + 18` rows, a discovery read 1,026, a tie-cluster read up to `2·MaxTieClusterOverfetch + 4`).
That cap truncates **silently** (the stream just ends), so a segment it cut would look complete and drop its newest events —
the same blindness a forward read has to it, but on the end a reverse read is asking for.
⚠️ **Client-visible (#691.4).** On Gateway (the default backend) an explicit reverse events read used to be refused
`BadHistoryOperationUnsupported` (#675), and an end-only capped events read returned the **oldest** N events ascending. Both
now return the latest events newest-first. Because the capability is checked, not a caught exception, a
`NotSupportedException` thrown from inside a backend (a driver library) is still logged as a backend fault. Every
implementation (and any future wrapper) must declare both members itself — the interface defaults (`false` / throw) would
silently send a native-reverse backend through the walk; `HistorianDataSourceReverseEventsGuardTests` pins it.
- **Gateway open upper bound.** The `DateTime.MaxValue` (UTC) bound of a start-only forward read is pinned offline through the
published client's proto `Timestamp` conversion (captured by an in-process gRPC server) and the FILETIME encoding
(`GatewayEventBoundsTests`). What a live AVEVA Historian does with a year-9999 bound is still unverified (needs the VPN).
@@ -120,9 +120,9 @@ public interface IHistorianDataSource : IDisposable
/// <summary>
/// Whether this backend can serve a reverse (newest-first) events read — the reverse overload of
/// <see cref="ReadEventsAsync(string?, DateTime, DateTime, int, bool, CancellationToken)"/> (Gitea #675). The server
/// checks it BEFORE calling the source: an explicit reverse read (<c>StartTime &gt; EndTime</c>) on a backend that
/// cannot is answered <c>BadHistoryOperationUnsupported</c> without a call, and an end-only capped read, which Part 11
/// reads backward, keeps its forward <c>[MinValue, EndTime)</c> meaning there. Default <c>false</c>; an
/// checks it BEFORE calling the source: a reverse read (explicit <c>StartTime &gt; EndTime</c>, or an end-only capped
/// read, which Part 11 reads backward) on a backend that cannot is never sent the reverse overload — since Gitea
/// #691.4 the server walks it back from bounded FORWARD reads instead (#675 refused it). Default <c>false</c>; an
/// implementation (or a wrapper) declares it itself — a guard test pins that.
/// </summary>
bool SupportsReverseEvents => false;
@@ -144,9 +144,9 @@ public interface IHistorianDataSource : IDisposable
/// <para>
/// The default forwards a forward read to the five-argument overload and throws
/// <see cref="NotSupportedException"/> for a reverse one — a backend that cannot read newest-first (Gateway,
/// Null). The server never relies on that throw: it consults <see cref="SupportsReverseEvents"/> first and
/// answers <c>BadHistoryOperationUnsupported</c> without calling, rather than returning the earliest events in
/// ascending order as if they were the answer. A backend that serves reverse must override BOTH members.
/// Null). The server never relies on that throw: it consults <see cref="SupportsReverseEvents"/> first and walks
/// such a read back from forward reads (#691.4), never returning the earliest events in ascending order as if
/// they were the answer. A backend that serves reverse natively must override BOTH members.
/// </para>
/// </remarks>
/// <param name="sourceName">The source name to filter events, or null to return events from all sources.</param>
@@ -61,8 +61,8 @@ public sealed class NullHistorianDataSource : IHistorianDataSource
public bool SupportsReverseRaw => true;
/// <inheritdoc />
/// <remarks><c>false</c> (Gitea #675): there is no history to read in either direction, and an explicit reverse read
/// is answered <c>BadHistoryOperationUnsupported</c> like any backend that cannot read newest-first.</remarks>
/// <remarks><c>false</c> (Gitea #675): there is no history to read in either direction; the server walks a reverse
/// read back from (empty) forward reads like any backend that cannot read newest-first (#691.4).</remarks>
public bool SupportsReverseEvents => false;
/// <inheritdoc />
@@ -393,8 +393,9 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
/// <inheritdoc />
/// <remarks>
/// <c>false</c>: the gateway's <c>ReadEvents</c> streams ascending only (Gitea #675). The server therefore refuses an
/// explicit reverse events read without calling, and keeps an end-only capped read forward.
/// <c>false</c>: the gateway's <c>ReadEvents</c> streams ascending only (Gitea #675). The server never sends it a
/// reverse read; since #691.4 it walks one back from bounded forward reads (ties sorted by EventId — the gateway
/// orders by event time only).
/// </remarks>
public bool SupportsReverseEvents => false;
@@ -404,10 +405,9 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
/// <inheritdoc />
/// <remarks>
/// A reverse (newest-first) read is refused with <see cref="NotSupportedException"/> — the server reports
/// <c>BadHistoryOperationUnsupported</c> (Gitea #675). The gateway's <c>ReadEvents</c> streams ascending only and
/// rejects <c>end &lt; start</c>, so the only way to serve "the latest N" would be to drain the whole window and
/// keep its tail, an unbounded read the cap exists to prevent. Nothing is sent.
/// A reverse (newest-first) read is refused with <see cref="NotSupportedException"/> (Gitea #675) — the gateway's
/// <c>ReadEvents</c> streams ascending only and rejects <c>end &lt; start</c>. The server never calls this with
/// <c>reverse</c>: it walks a reverse read back from bounded forward reads (#691.4). Nothing is sent.
/// </remarks>
public Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, bool reverse,
@@ -5248,6 +5248,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
var endUtc = details.EndTime;
var numValuesPerNode = details.NumValuesPerNode;
var boundarySkip = 0;
var walkHint = default(ReverseWalkHint); // #691.4: a walked-back read's width + proven-empty stretch
bool reverse;
if (resumed)
@@ -5271,35 +5272,30 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// The direction is the CURSOR's (Gitea #675), never re-derived from the resume request's details: a
// reverse read resumes reverse, a forward one forward.
reverse = state.Reverse;
walkHint = state.WalkHint;
}
else
{
// Part 11: with one bound unset, an unset EndTime (MinValue) with a StartTime means "page FORWARD from
// StartTime" — normalise it to an open upper bound, or the cursor would store MinValue and the read would
// look reversed. An unset StartTime with an EndTime and a count cap means "page BACKWARD from EndTime"
// (#675, mirroring raw): the reverse window (MinValue, EndTime] — but only on a backend that can read
// newest-first. Elsewhere (Gateway, the default backend) and without a count cap an end-only read keeps its
// historical forward [MinValue, EndTime) meaning rather than becoming a refused reverse read.
// (#675, mirroring raw): the reverse window (MinValue, EndTime] — on every backend since #691.4 (a backend
// that cannot read newest-first is walked back below). Without a count cap an end-only read keeps its
// forward [MinValue, EndTime) meaning.
if (endUtc == DateTime.MinValue && startUtc != DateTime.MinValue)
endUtc = new DateTime(DateTime.MaxValue.Ticks, DateTimeKind.Utc); // Utc kind: a proto Timestamp conversion rejects Unspecified
else if (numValuesPerNode > 0 && startUtc == DateTime.MinValue && endUtc != DateTime.MinValue
&& source.SupportsReverseEvents)
else if (numValuesPerNode > 0 && startUtc == DateTime.MinValue && endUtc != DateTime.MinValue)
(startUtc, endUtc) = (endUtc, new DateTime(0, DateTimeKind.Utc));
// Part 11: StartTime later than EndTime is a reverse read — events newest-first over (end, start].
reverse = startUtc > endUtc;
}
// An explicit reverse read (or a reverse cursor resumed after the source was swapped) on a backend that cannot
// read newest-first: refuse it BEFORE calling the source — never the earliest events in ascending order as if
// they were the answer. Checked here rather than by catching an exception, so a NotSupportedException thrown
// from inside a backend (a driver library, the projection) stays a backend fault.
if (reverse && !source.SupportsReverseEvents)
{
errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadHistoryOperationUnsupported };
return;
}
// A reverse read on a backend that cannot read newest-first (Gateway, Null) is never sent there (#675 refused it
// here); since #691.4 it is walked back from bounded forward reads — the raw walk (#686), with ties ordered by
// EventId because the gateway orders by event time only. Decided by the capability, not by catching an
// exception, so a NotSupportedException thrown from inside a backend stays a backend fault.
var walked = reverse && !source.SupportsReverseEvents;
// NOT under the node-manager Lock — awaiting the async source is safe here.
// NumValuesPerNode==0 means "no limit — return ALL values" per OPC UA Part 4/11, so it is translated
@@ -5309,8 +5305,39 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
var fetch = EventMaxEvents(numValuesPerNode);
if (numValuesPerNode > 0 && boundarySkip > 0)
fetch = (int)Math.Min(int.MaxValue - 1L, (long)fetch + boundarySkip);
var sourceResult = await source.ReadEventsAsync(
sourceName, startUtc, endUtc, fetch, reverse, ct).ConfigureAwait(false);
HistoricalEventsResult sourceResult;
if (walked)
{
// Walk for one more than the fetch: finding it is the "more events exist" signal a source gives with its
// continuation point (the walk itself cannot flag truncation).
var want = numValuesPerNode == 0 ? 0u : (uint)Math.Min(uint.MaxValue - 1L, (long)fetch + 1);
var budget = Math.Max(1, MaxTieClusterOverfetch);
var walkSource = sourceName;
var (rows, next) = await ReverseWalk.WalkAsync<HistoricalEvent>(
async (s, e, cap, token) =>
{
var r = await source.ReadEventsAsync(walkSource, s, e, cap == 0 ? int.MaxValue : cap, reverse: false, token)
.ConfigureAwait(false);
return new ForwardSegmentRead<HistoricalEvent>(r.Events, r.ContinuationPoint is not null);
},
ev => ev.EventTimeUtc.Ticks, EventIdTieOrder.Instance, startUtc, endUtc, want, budget,
source.TimestampResolutionTicks, Clock.GetUtcNow().UtcDateTime, walkHint,
at => new HistoryReadLimitExceededException(budget,
$"A reverse events read of '{walkSource}' at {at:O} found more than {budget} events at one " +
"timestamp; the backend reads forward only, so the latest of them cannot be selected. Increase " +
"ServerHistorian:MaxTieClusterOverfetch."),
ct)
.ConfigureAwait(false);
walkHint = next;
var more = want > 0 && rows.Count > fetch;
if (more) rows.RemoveRange(fetch, rows.Count - fetch);
sourceResult = new HistoricalEventsResult(rows, more ? [0x01] : null);
}
else
{
sourceResult = await source.ReadEventsAsync(
sourceName, startUtc, endUtc, fetch, reverse, ct).ConfigureAwait(false);
}
// Truncation is the SOURCE's word (non-null continuation point) — never inferred from the count, so a
// page that happens to be exactly cap-sized but complete never emits a point.
@@ -5350,6 +5377,12 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
sourceName, startUtc, startUtc.AddTicks(resolution), overfetchCap, reverse: false, ct)
.ConfigureAwait(false)).Events;
// A walked read (#691.4) emitted its ties sorted by EventId; the gateway returns them in any order and its
// end is inclusive — keep exactly the cluster, in the walk's order, so the skip indexes the same events.
if (walked)
cluster = cluster.Where(e => e.EventTimeUtc == startUtc)
.OrderBy(e => e, EventIdTieOrder.Instance).ToList();
// The over-fetch reads the cluster FORWARD (ascending EventId), but a reverse read emitted its ties — and
// counted boundarySkip — in descending order. Slice in the read's own direction, as raw does.
if (reverse) cluster = cluster.Reverse().ToList();
@@ -5378,7 +5411,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
{
pendingCursor = new HistoryContinuationState(
sourceName ?? string.Empty, resumeAt, endUtc, nextSkip, numValuesPerNode, HistoryCursorKind.Events,
Reverse: reverse);
Reverse: reverse, SegmentWidthTicks: walkHint.WidthTicks, EmptyFromTicks: walkHint.EmptyFromTicks,
EmptyToTicks: walkHint.EmptyToTicks);
}
}
else if (truncated && events.Count > 0)
@@ -5390,7 +5424,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
for (var i = events.Count - 1; i >= 0 && events[i].EventTimeUtc == boundary; i--) skip++;
pendingCursor = new HistoryContinuationState(
sourceName ?? string.Empty, boundary, endUtc, skip, numValuesPerNode, HistoryCursorKind.Events,
Reverse: reverse);
Reverse: reverse, SegmentWidthTicks: walkHint.WidthTicks, EmptyFromTicks: walkHint.EmptyFromTicks,
EmptyToTicks: walkHint.EmptyToTicks);
}
var historyEvent = ProjectEvents(events, selectClauses);
@@ -5411,6 +5446,14 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
errors[handle.Index] = StatusCodes.BadTimeout;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadTimeout };
}
catch (HistoryReadLimitExceededException ex)
{
// A request limit (#691.4: a walked-back reverse read met a tie cluster over the budget) — BadTooManyOperations
// for THIS node, as on the raw path; not a backend fault.
_logger.LogWarning("OtOpcUaNodeManager: HistoryReadEvents limit exceeded for node {NodeId}: {Message}", handle.NodeId, ex.Message);
errors[handle.Index] = StatusCodes.BadTooManyOperations;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadTooManyOperations };
}
catch (Exception ex)
{
// One node's backend failure must not throw out of the batch — surface Bad for THIS node
@@ -1,3 +1,5 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
/// <summary>What one walked-back page tells the next page of the same read (carried in the continuation point).</summary>
@@ -364,3 +366,16 @@ internal static class ReverseWalk
return down > DateTime.MaxValue.Ticks - step ? DateTime.MaxValue.Ticks : down + step;
}
}
/// <summary>
/// Orders events within one timestamp by <see cref="HistoricalEvent.EventId"/> (ordinal) — the tie order a walked-back
/// events read (#691.4) and its oversized-cluster over-fetch share, because the Gateway orders by event time only.
/// </summary>
internal sealed class EventIdTieOrder : IComparer<HistoricalEvent>
{
/// <summary>The shared instance.</summary>
public static readonly EventIdTieOrder Instance = new();
/// <inheritdoc />
public int Compare(HistoricalEvent? x, HistoricalEvent? y) => string.CompareOrdinal(x?.EventId, y?.EventId);
}
@@ -35,7 +35,7 @@ public sealed class GatewayEventBoundsTests
{
var fake = new FakeHistorianGatewayClient();
IHistorianDataSource ds = new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
ds.SupportsReverseEvents.ShouldBeFalse("the server must keep end-only reads forward on the default backend");
ds.SupportsReverseEvents.ShouldBeFalse("the server must walk reverse reads back from forward reads on the default backend (#691.4)");
await Should.ThrowAsync<NotSupportedException>(() =>
ds.ReadEventsAsync("Pump1", DateTime.UtcNow, DateTime.UtcNow.AddHours(-1), 10, reverse: true, Ct));
@@ -333,12 +333,16 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable
}
[Fact]
public async Task A_backend_without_reverse_support_answers_BadHistoryOperationUnsupported_not_the_earliest_events()
public async Task A_backend_without_reverse_support_walks_a_reverse_read_back_never_answering_the_earliest_events()
{
// #691.4 (was refused BadHistoryOperationUnsupported since #675): served newest-first from forward reads.
var (host, nm, node, src) = await SetupAsync(Distinct(10));
nm.HistorianDataSource = new ForwardOnlyEventSource(src);
ReadEvents(nm, node, max: 3, inboundCp: null, end: Early, start: Late).Error.StatusCode.Code
.ShouldBe(StatusCodes.BadHistoryOperationUnsupported);
var fwd = new ForwardOnlyEventSource(src);
nm.HistorianDataSource = fwd;
var r = ReadEvents(nm, node, max: 3, inboundCp: null, end: Early, start: Late);
r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good);
r.Ids.ShouldBe(["e0009", "e0008", "e0007"]);
r.Cp.ShouldNotBeNull();
// Forward reads on the same backend are unaffected.
DrainAsync(nm, node, max: 4).ShouldBe(src.All.Select(e => e.EventId));
await host.DisposeAsync();
@@ -349,30 +353,115 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable
// --- #675 review follow-ups: the reverse-events capability ------------------------------------------------
// --- #691.4: reverse events on a backend that reads forward only (the Gateway) are walked back ------------------
[Fact]
public async Task End_only_capped_read_on_a_backend_without_reverse_support_pages_forward_as_before()
public async Task End_only_capped_read_on_a_forward_only_backend_pages_backward_from_EndTime()
{
// Gateway (the default backend) cannot read newest-first: an end-only read keeps its pre-#675 forward
// [MinValue, EndTime) meaning there rather than turning into a refused reverse read.
var (host, nm, node, src) = await SetupAsync(Distinct(25));
var fwd = new NonReverseSource(src);
nm.HistorianDataSource = fwd;
// Part 11 (as raw, #686): the latest events at or before EndTime, newest first — on Gateway this used to keep its
// pre-#675 forward [MinValue, EndTime) meaning, i.e. the OLDEST events. The backend is never sent a reverse read.
var (host, nm, node, _) = await SetupAsync([]);
var gw = new GatewayShapedEventSource(Distinct(25));
nm.HistorianDataSource = gw;
var ids = DrainAsync(nm, node, max: 10, start: DateTime.MinValue, end: Epoch.AddSeconds(20));
ids.ShouldBe(Enumerable.Range(0, 20).Select(i => $"e{i:0000}"));
fwd.ReverseCalls.ShouldBe(0);
ids.ShouldBe(Enumerable.Range(0, 21).Reverse().Select(i => $"e{i:0000}"));
gw.ReverseCalls.ShouldBe(0);
gw.BadRequests.ShouldBe(0);
await host.DisposeAsync();
}
[Fact]
public async Task Explicit_reverse_read_on_a_backend_without_reverse_support_is_refused_without_calling_it()
public async Task Explicit_reverse_read_on_a_forward_only_backend_pages_the_reverse_window_newest_first()
{
var (host, nm, node, src) = await SetupAsync(Distinct(10));
var fwd = new NonReverseSource(src);
nm.HistorianDataSource = fwd;
ReadEvents(nm, node, max: 3, inboundCp: null, end: Early, start: Late).Error.StatusCode.Code
.ShouldBe(StatusCodes.BadHistoryOperationUnsupported);
fwd.ReverseCalls.ShouldBe(0, "the capability is checked BEFORE the source is called");
fwd.ForwardCalls.ShouldBe(0);
// (EndTime, StartTime]: StartTime inclusive, EndTime exclusive.
var (host, nm, node, _) = await SetupAsync([]);
var gw = new GatewayShapedEventSource(Distinct(30));
nm.HistorianDataSource = gw;
var ids = DrainAsync(nm, node, max: 4, start: Epoch.AddSeconds(25), end: Epoch.AddSeconds(5));
ids.ShouldBe(Enumerable.Range(6, 20).Reverse().Select(i => $"e{i:0000}"));
gw.ReverseCalls.ShouldBe(0);
await host.DisposeAsync();
}
[Fact]
public async Task Reverse_ties_on_a_forward_only_backend_page_in_a_stable_order_although_the_backend_shuffles_them()
{
// The gateway orders events by EventTimeUtc only, so ties come back in any order — this fake shuffles them on every
// call. The walk sorts ties by EventId (descending, newest-first) on every read, and the oversized-cluster over-fetch
// does the same, so a boundary-tie skip always lines up. Cap 3 cuts through a 7-way cluster.
var events = new List<HistoricalEvent> { Ev("a0", 0) };
for (var i = 0; i < 7; i++) events.Add(Ev($"t{i}", 5));
events.Add(Ev("z0", 9));
var (host, nm, node, _) = await SetupAsync([]);
nm.HistorianDataSource = new GatewayShapedEventSource(events, shuffleSeed: 7);
DrainAsync(nm, node, max: 3, start: DateTime.MinValue, end: Epoch.AddSeconds(30)).ShouldBe(
["z0", "t6", "t5", "t4", "t3", "t2", "t1", "t0", "a0"]);
await host.DisposeAsync();
}
[Fact]
public async Task Uncapped_reverse_read_on_a_forward_only_backend_is_one_forward_read()
{
var (host, nm, node, _) = await SetupAsync([]);
var gw = new GatewayShapedEventSource(Distinct(30));
nm.HistorianDataSource = gw;
var r = ReadEvents(nm, node, max: 0, inboundCp: null, end: Epoch.AddSeconds(5), start: Epoch.AddSeconds(25));
r.Ids.ShouldBe(Enumerable.Range(6, 20).Reverse().Select(i => $"e{i:0000}"));
r.Cp.ShouldBeNull();
gw.ForwardCalls.ShouldBe(1);
await host.DisposeAsync();
}
[Fact]
public async Task Reverse_paging_on_a_forward_only_backend_is_exact_on_random_data()
{
var (host, nm, node, _) = await SetupAsync([]);
for (var seed = 0; seed < 80; seed++)
{
var rng = new Random(seed);
var events = new List<HistoricalEvent>();
var t = Epoch.Ticks;
var n = rng.Next(1, 120);
while (events.Count < n)
{
t += rng.Next(4) switch
{
0 => rng.Next(1, 10),
1 => rng.Next(1, 5) * TimeSpan.TicksPerMillisecond,
2 => rng.Next(1, 90) * TimeSpan.TicksPerMinute,
_ => rng.Next(1, 2000) * TimeSpan.TicksPerMillisecond,
};
var ties = rng.Next(5) == 0 ? rng.Next(2, 8) : 1;
for (var k = 0; k < ties; k++)
{
var at = new DateTime(t, DateTimeKind.Utc);
events.Add(new HistoricalEvent($"{Guid.NewGuid():N}", "src", at, at, "m", 500));
}
}
nm.HistorianDataSource = new GatewayShapedEventSource(events, shuffleSeed: seed);
nm.MaxTieClusterOverfetch = rng.Next(2) == 0 ? 8 : 65536;
var max = (uint)rng.Next(1, 9);
var end = new DateTime(t + rng.Next(0, 3) * TimeSpan.TicksPerSecond, DateTimeKind.Utc);
var expected = events.OrderByDescending(e => e.EventTimeUtc).ThenByDescending(e => e.EventId, StringComparer.Ordinal)
.Select(e => e.EventId).ToList();
var all = new List<string>();
byte[]? cp = null;
var pages = 0;
do
{
var r = ReadEvents(nm, node, max, cp, end, DateTime.MinValue);
r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good, $"seed {seed}");
all.AddRange(r.Ids);
cp = r.Cp;
(++pages).ShouldBeLessThan(2000);
}
while (cp is not null);
all.ShouldBe(expected, $"seed {seed}, N = {max}");
}
await host.DisposeAsync();
}
@@ -391,29 +480,39 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable
await host.DisposeAsync();
}
/// <summary>Declares the reverse overload (so a call would be visible) but reports no reverse capability.</summary>
private sealed class NonReverseSource(IHistorianDataSource inner) : IHistorianDataSource
/// <summary>
/// Shaped like the Gateway's SQL event read (#691.4): forward only (<see cref="SupportsReverseEvents"/> false — a
/// reverse call is counted and refused), <c>[start, end]</c> INCLUSIVE, ordered by event time only with ties in an
/// arbitrary order (shuffled on every call when a seed is given), capped with a truncation flag.
/// </summary>
private sealed class GatewayShapedEventSource(List<HistoricalEvent> events, int? shuffleSeed = null) : IHistorianDataSource
{
private readonly Random? _rng = shuffleSeed is { } s ? new Random(s) : null;
public int ReverseCalls { get; private set; }
public int ForwardCalls { get; private set; }
public int BadRequests { get; private set; }
public bool SupportsReverseEvents => false;
public Task<HistoricalEventsResult> ReadEventsAsync(string? n, DateTime s, DateTime e, int m, CancellationToken ct)
{
ForwardCalls++;
return inner.ReadEventsAsync(n, s, e, m, ct);
if (s > e || s < DateTime.UnixEpoch) BadRequests++;
var window = events.Where(x => x.EventTimeUtc >= s && x.EventTimeUtc <= e)
.OrderBy(x => x.EventTimeUtc).ThenBy(_ => _rng?.Next() ?? 0).ToList();
var cap = m <= 0 ? int.MaxValue : m;
return Task.FromResult(new HistoricalEventsResult(window.Take(cap).ToList(), window.Count > cap ? new byte[] { 1 } : null));
}
public Task<HistoricalEventsResult> ReadEventsAsync(string? n, DateTime s, DateTime e, int m, bool reverse, CancellationToken ct)
{
if (reverse) { ReverseCalls++; return inner.ReadEventsAsync(n, s, e, m, true, ct); }
if (reverse) { ReverseCalls++; throw new NotSupportedException("forward only"); }
return ReadEventsAsync(n, s, e, m, ct);
}
public Task<HistorianRead> ReadRawAsync(string r, DateTime s, DateTime e, uint m, CancellationToken ct) =>
inner.ReadRawAsync(r, s, e, m, ct);
NullHistorianDataSource.Instance.ReadRawAsync(r, s, e, m, ct);
public Task<HistorianRead> ReadProcessedAsync(string r, DateTime s, DateTime e, TimeSpan i, HistoryAggregateType a, CancellationToken ct) =>
inner.ReadProcessedAsync(r, s, e, i, a, ct);
NullHistorianDataSource.Instance.ReadProcessedAsync(r, s, e, i, a, ct);
public Task<HistorianRead> ReadAtTimeAsync(string r, IReadOnlyList<DateTime> t, CancellationToken ct) =>
inner.ReadAtTimeAsync(r, t, ct);
public HistorianHealthSnapshot GetHealthSnapshot() => inner.GetHealthSnapshot();
NullHistorianDataSource.Instance.ReadAtTimeAsync(r, t, ct);
public HistorianHealthSnapshot GetHealthSnapshot() => NullHistorianDataSource.Instance.GetHealthSnapshot();
public void Dispose() { }
}