fix(historian): #694 page raw reads at a strict gateway's learned ceiling; opt-in EventReadHorizon

Item 1: IHistorianDataSource.MaxRawValuesPerRead (Gateway: learned ceiling - 16 = L; 0 elsewhere, guard-tested).
RawPageLimit plans every raw page on top of #692's lookahead: with L > 0 the read (page P + boundary ties k
+ lookahead) is <= L, P = min(N, L - k - 1) (uncapped: L - k - 1); no room => straight to the tie-cluster
over-fetch (never an uncapped or underflowing request). One re-plan + re-read when the read taught a new
limit or was refused for crowding pre-window rows (HistoryReadLimitExceededException.PreWindowRows): the
page shrinks by them. Over-fetch bound min(MaxTieClusterOverfetch, L - 1); slice size never 0.
Review fixes: F1 a window starting at/before the epoch counts AVEVA's synthetic row as pre-window (a
ceiling-truncated read was reported complete); F2 the page path disables ReverseRawReader's own re-run
(cold end-only ceiling 20/N=3000 cost 3,023 reads, now <= 40); F3 crowded look-back re-read around, not
refused; F4 the #687 crowding re-read room at least doubles (up to 8 re-reads), so a cluster of hundreds
just before a resume point no longer fails on a default gateway; a crowding refusal that leaves no room
is BadTooManyOperations, never a tick-by-tick cursor.

Item 2: ServerHistorian:EventReadHorizon (default 00:00:00 = exact, unchanged). When set, every fresh
HistoryReadEvents window's earlier bound is raised to now - horizon (both directions, all backends).
Lossy by design; documented.

Item 3 (native TimeAverage/Total re-check) stays blocked on historiangw#27.
This commit is contained in:
Joseph Doherty
2026-10-13 00:52:12 -04:00
parent a5a0496176
commit 3c3513ea3e
23 changed files with 1461 additions and 79 deletions
@@ -93,6 +93,14 @@ public sealed class HistoryReadLimitExceededException : InvalidOperationExceptio
/// <summary>The configured limit that was exceeded.</summary>
public long Limit { get; }
/// <summary>
/// Gitea #694: a raw read refused because rows BEFORE its window (inside the backend's look-back margin — a tie
/// cluster just before a page start, the HistorianGateway's boundary row) crowded a ceiling-bounded row budget: how
/// many such rows the read saw (a lower bound when the read was itself truncated). <c>0</c> = not a crowding refusal.
/// The server re-reads such a page once, smaller by this many values (<c>RawPageLimit</c>).
/// </summary>
public int PreWindowRows { get; init; }
/// <summary>Creates the exception.</summary>
/// <param name="limit">The configured limit.</param>
/// <param name="message">Operator-facing description.</param>
@@ -46,6 +46,18 @@ public interface IHistorianDataSource : IDisposable
/// </summary>
bool SupportsReverseRaw => false;
/// <summary>
/// The most values one capped <see cref="ReadRawAsync"/> can currently return without failing
/// <see cref="HistoryReadLimitExceededException"/> on a BACKEND row ceiling (Gitea #694); <c>0</c> = no known limit.
/// The server sizes its raw pages to it — a page is at most this many values and carries a continuation point when
/// full (Part 11 allows fewer than <c>NumValuesPerNode</c> with a continuation point), instead of failing a page size
/// the backend cannot read in one go. It may change at runtime (a strict HistorianGateway's ceiling is learned from
/// its first refusal, #691.3), so it is read per read. A backend whose reads are never ceiling-bound returns <c>0</c>.
/// The source still never truncates: a read asking for more than this is answered in full or refused. Default
/// <c>0</c>; a backend declares it itself (a guard test pins that).
/// </summary>
int MaxRawValuesPerRead => 0;
/// <summary>
/// Read processed (interval-bucketed) samples — average / min / max / count / etc.
/// A bucket with no source data returns a sample whose
@@ -60,6 +60,10 @@ public sealed class NullHistorianDataSource : IHistorianDataSource
/// direction — no need for the server to walk back through forward reads to find nothing.</remarks>
public bool SupportsReverseRaw => true;
/// <inheritdoc />
/// <remarks><c>0</c>: no history, no ceiling (Gitea #694).</remarks>
public int MaxRawValuesPerRead => 0;
/// <inheritdoc />
/// <remarks><c>true</c>, like <see cref="SupportsReverseRaw"/> (Gitea #691.4 review; <c>false</c> under #675): there is no
/// history to read in either direction — no need for the server to walk back through empty forward reads.</remarks>
@@ -51,6 +51,9 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
/// <summary>#693: AtTime from raw reads (true) or AVEVA's own interpolation (the kill switch).</summary>
internal bool AtTimeFromRaw => _atTimeFromRaw;
/// <summary>The most #687 crowding re-reads one raw read makes (#694 F4): each at least doubles the pre-window room.</summary>
private const int MaxCrowdedReReads = 8;
/// <summary>A strict gateway's row ceiling, learned from its first refusal (#691.3).</summary>
private readonly GatewayRowCeiling _ceiling = new();
@@ -191,29 +194,46 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
throw new HistoryReadLimitExceededException(cap,
$"The raw read of '{fullReference}' holds more than the HistorianGateway's row ceiling ({all.Budget}, its " +
"Historian:ReadLimits:MaxValuesCeiling with AllowStreamingBeyondCeiling=false); set NumValuesPerNode to page, " +
"or narrow the window.");
"or narrow the window.")
{ PreWindowRows = Crowding(all.Result.PreWindowRows) };
}
var asked = (int)Math.Min(int.MaxValue, (long)cap + RawAggregateReader.PreWindowAllowance);
var first = await PassAsync(asked).ConfigureAwait(false);
if (first.Result.Complete) return first.Result.Samples;
var pass = await PassAsync(asked).ConfigureAwait(false);
if (pass.Result.Complete) return pass.Result.Samples;
// A strict gateway's ceiling already bounded that read (#691.3): a larger budget would only be refused again.
if (first.Budget < asked)
// A strict gateway's ceiling already bounded that read (#691.3): a larger budget would only be refused again. The
// refusal carries the pre-window rows it saw (#694): the server re-reads the page that many values smaller.
if (pass.Budget < asked)
throw new HistoryReadLimitExceededException(cap,
$"The raw read of '{fullReference}' could not be read completely within the HistorianGateway's row ceiling " +
$"({first.Budget}, its Historian:ReadLimits:MaxValuesCeiling with AllowStreamingBeyondCeiling=false); lower " +
"NumValuesPerNode or narrow the window.");
$"({pass.Budget}, its Historian:ReadLimits:MaxValuesCeiling with AllowStreamingBeyondCeiling=false); lower " +
"NumValuesPerNode or narrow the window.")
{ PreWindowRows = Crowding(pass.Result.PreWindowRows) };
// The margin crowded the window out: one bounded re-read with room for the pre-window rows already seen.
var second = await PassAsync(
(int)Math.Min(int.MaxValue, (long)cap + first.Result.PreWindowRows + RawAggregateReader.PreWindowAllowance))
.ConfigureAwait(false);
if (second.Result.Complete) return second.Result.Samples;
// The margin crowded the window out: re-read with room for the pre-window rows. A read that was itself cut short saw
// only some of them (#694 F4: a tie cluster of hundreds just before a resume point used to fail even with no
// ceiling), so the room at least doubles per re-read — bounded by MaxCrowdedReReads, and never past a known ceiling.
var preWindow = pass.Result.PreWindowRows;
var room = Math.Max(1, preWindow);
for (var attempt = 0; attempt < MaxCrowdedReReads; attempt++)
{
var budget = (int)Math.Min(int.MaxValue, (long)cap + room + RawAggregateReader.PreWindowAllowance);
if (_ceiling.Learned is var known and > 0 && budget > known) break; // over a strict ceiling: refused anyway
pass = await PassAsync(budget).ConfigureAwait(false);
if (pass.Result.Complete) return pass.Result.Samples;
preWindow = Math.Max(preWindow, pass.Result.PreWindowRows);
if (pass.Budget < budget) break; // bounded by a ceiling learned meanwhile
room = (int)Math.Min(int.MaxValue / 2, Math.Max(preWindow, 2L * room));
}
throw new HistoryReadLimitExceededException(cap,
$"The raw read of '{fullReference}' could not be read completely: more than {second.Result.PreWindowRows} rows lie in " +
$"the {RawAggregateReader.BoundaryMargin.TotalMilliseconds} ms before the window and exhausted the row budget.");
$"The raw read of '{fullReference}' could not be read completely: more than {pass.Result.PreWindowRows} rows lie in " +
$"the {RawAggregateReader.BoundaryMargin.TotalMilliseconds} ms before the window and exhausted the row budget.")
{ PreWindowRows = preWindow };
// Pre-window rows the allowance did not cover are what crowded the read; fewer are not the cause (#694).
static int Crowding(int preWindowRows) => preWindowRows > RawAggregateReader.PreWindowAllowance ? preWindowRows : 0;
// One drained pass; a strict gateway's refusal of the budget is retried once at the ceiling it names (#691.3).
Task<((List<HistorianSample> Samples, int PreWindowRows, bool Complete) Result, int Budget)> PassAsync(int budget) =>
@@ -240,7 +260,11 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
{
if (cap == 0 || samples.Count < cap) samples.Add(sample);
}
else if (sample.Timestamp is { } ts && ts.ToDateTime() < startUtc) preWindow++;
// A row before the window — or, for a window starting at/before the epoch (no look-back), AVEVA's synthetic row at
// the query start, which is no sample either (#694 F1: it used to set reachedEnd, so a ceiling-truncated read
// passed as complete) — is a pre-window row.
else if (sample.Timestamp is { } ts
&& (ts.ToDateTime() < startUtc || !RawAggregateReader.IsAfterFloor(ts.ToDateTime()))) preWindow++;
else if (sample.Timestamp is not null) reachedEnd = true;
}
@@ -453,6 +477,17 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
/// </remarks>
public bool SupportsReverseRaw => false;
/// <inheritdoc />
/// <remarks>
/// Gitea #694: <c>0</c> until a strict gateway (<c>AllowStreamingBeyondCeiling=false</c>) has refused a budget and
/// named its ceiling (#691.3, <see cref="GatewayRowCeiling"/>); then that ceiling less the
/// <see cref="RawAggregateReader.PreWindowAllowance"/> a capped read adds for AVEVA's boundary row — so a page of this
/// size asks for exactly the ceiling. Back to <c>0</c> when a re-probe finds the gateway streaming again.
/// </remarks>
public int MaxRawValuesPerRead => _ceiling.Learned is var c and > 0
? Math.Max(1, c - RawAggregateReader.PreWindowAllowance)
: 0;
/// <inheritdoc />
/// <remarks>
/// <c>false</c>: the gateway's <c>ReadEvents</c> streams ascending only (Gitea #675). The server never sends it a
@@ -359,6 +359,10 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource
/// <remarks>Reverse raw reads are served natively (<c>(end, start]</c>, <c>ORDER BY … DESC</c>; #669/#686).</remarks>
public bool SupportsReverseRaw => true;
/// <inheritdoc />
/// <remarks><c>0</c>: a capped raw read is one bounded statement — no backend row ceiling (Gitea #694).</remarks>
public int MaxRawValuesPerRead => 0;
/// <inheritdoc />
/// <remarks>Reverse events reads are served natively (<c>ORDER BY … DESC</c>, Gitea #675).</remarks>
public bool SupportsReverseEvents => true;
@@ -316,6 +316,10 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
/// <remarks>Reverse raw reads are served natively (<c>(end, start]</c>, <c>ORDER BY … DESC</c>; #669/#686).</remarks>
public bool SupportsReverseRaw => true;
/// <inheritdoc />
/// <remarks><c>0</c>: a capped raw read is one bounded statement — no backend row ceiling (Gitea #694).</remarks>
public int MaxRawValuesPerRead => 0;
/// <inheritdoc />
/// <remarks>Reverse events reads are served natively (<c>ORDER BY … DESC</c>, Gitea #675).</remarks>
public bool SupportsReverseEvents => true;
@@ -320,6 +320,9 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
// Gitea #691: the ReturnBounds kill switch (default on).
_server.NodeManager.HonorReturnBounds = HonorReturnBounds;
// Gitea #694: the opt-in HistoryReadEvents horizon (default off = exact to each read's own window).
_server.NodeManager.EventReadHorizon = EventReadHorizon;
// Propagate the S3 HistoryRead concurrency bounds (arch-review 03/S3): bounded per-node fan-out
// within a batch, the process-wide concurrent-batch limiter, and the per-request deadline. Defaults
// (4 / 16 / 60 s) survive when the ServerHistorian section is absent.
@@ -356,6 +359,12 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
/// </summary>
internal bool HonorReturnBounds => _serverHistorianOptions.HonorReturnBounds;
/// <summary>
/// Gitea #694 — the <c>ServerHistorian:EventReadHorizon</c> <c>StartAsync</c> applies to the node manager (default
/// <see cref="TimeSpan.Zero"/> = none). Exposed so the wiring is testable without booting the SDK.
/// </summary>
internal TimeSpan EventReadHorizon => _serverHistorianOptions.EventReadHorizon;
/// <summary>
/// Gitea #674 — what <c>StartAsync</c> wires onto the node manager for HistoryUpdate: the Primary gate (read from
/// the <see cref="IHistoryUpdateRoleView"/> on every call; null when none is registered, i.e. allow), the
@@ -35,7 +35,9 @@
"AllowUntrustedServerCertificate": false,
"CaCertificatePath": null,
"CallTimeout": "00:00:30",
"MaxTieClusterOverfetch": 65536
"MaxTieClusterOverfetch": 65536,
"EventReadHorizon": "00:00:00",
"_EventReadHorizonComment": "Gitea #694, opt-in. 00:00:00 = no horizon: HistoryReadEvents is exact to each read's window (on the Gateway a reverse/end-only events read proves 'nothing older' with one read down to 1970 - seconds on a large Runtime.dbo.Events). A positive span (e.g. 365.00:00:00) never reads events older than now minus it, in either direction - LOSSY: older events become invisible to HistoryReadEvents."
},
"ContinuousHistorization": {
"_comment": "Continuous historization of driver (non-Galaxy) tag values: a crash-safe FasterLog outbox + recorder draining to the ServerHistorian gateway's WriteLiveValues. Disabled => no recorder is spawned. Requires ServerHistorian to be configured; the gateway connection (endpoint/key/TLS) is sourced from the ServerHistorian section, not here.",
@@ -0,0 +1,43 @@
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
/// <summary>
/// Gitea #694 item 2: the opt-in <c>ServerHistorian:EventReadHorizon</c> — HistoryReadEvents never reads events older than
/// <c>now − horizon</c>. Off (<see cref="TimeSpan.Zero"/>, the default) a read is exact down to whatever its window says,
/// which on the Gateway means a reverse / end-only read walked back with forward reads proves "nothing older" with one
/// read down to the Unix epoch (#691) — ~3.8 s live on <c>Runtime.dbo.Events</c>, and an unfiltered one can hit the
/// gateway's 30 s SQL timeout. A forward read whose StartTime is far in the past costs the same on that table. The
/// horizon bounds both, at a stated price: events older than it are not returned, in either direction, on any backend.
/// </summary>
internal static class EventHorizon
{
/// <summary>
/// Raises the EARLIER bound of a fresh events window to the horizon's floor (<c>now − horizon</c>, inclusive): the
/// start of a forward <c>[start, end)</c> window, the exclusive lower end of a reverse <c>(end, start]</c> one.
/// </summary>
/// <param name="startUtc">The window start (the later bound of a reverse read).</param>
/// <param name="endUtc">The window end (the exclusive earlier bound of a reverse read).</param>
/// <param name="reverse">Whether the read is newest-first (<c>start &gt; end</c>).</param>
/// <param name="horizon">The horizon; non-positive = none (nothing changes).</param>
/// <param name="nowUtc">Now.</param>
/// <returns><c>false</c> when the whole window lies before the floor — the read is empty and must not be sent.</returns>
public static bool TryClamp(ref DateTime startUtc, ref DateTime endUtc, bool reverse, TimeSpan horizon, DateTime nowUtc)
{
if (horizon <= TimeSpan.Zero) return true;
var floor = nowUtc.Ticks - DateTime.MinValue.Ticks > horizon.Ticks
? new DateTime(nowUtc.Ticks - horizon.Ticks, DateTimeKind.Utc)
: new DateTime(0, DateTimeKind.Utc);
if (floor.Ticks == 0) return true;
if (reverse)
{
if (startUtc < floor) return false; // (end, start] lies wholly before the floor
var below = floor.AddTicks(-1); // exclusive lower end ⇒ the floor instant itself is read
if (endUtc < below) endUtc = below;
return true;
}
if (endUtc <= floor) return false; // [start, end) lies wholly before the floor
if (startUtc < floor) startUtc = floor;
return true;
}
}
@@ -552,6 +552,13 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
/// </summary>
public bool HonorReturnBounds { get; set; } = true;
/// <summary>
/// Gitea #694: <c>ServerHistorian:EventReadHorizon</c> — HistoryReadEvents never reads events older than
/// <c>now − EventReadHorizon</c> (<see cref="EventHorizon"/>). <see cref="TimeSpan.Zero"/> (the default) = no horizon:
/// every read is exact down to its own window (a walked Gateway reverse read proves "nothing older" down to 1970).
/// </summary>
public TimeSpan EventReadHorizon { get; set; } = TimeSpan.Zero;
/// <summary>
/// Maximum historized nodes served concurrently within one HistoryRead batch (arch-review 03/S3).
/// Mirrors <c>ServerHistorianOptions.HistoryReadBatchParallelism</c>; the Host sets it at
@@ -5339,6 +5346,19 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// Part 11: StartTime later than EndTime is a reverse read — events newest-first over (end, start].
reverse = startUtc > endUtc;
// #694: the opt-in events horizon raises the window's earlier bound (the cursor then carries it). A window
// wholly older than it is empty and is not sent (a forward [start, end) clamped past its end would be refused).
if (!EventHorizon.TryClamp(ref startUtc, ref endUtc, reverse, EventReadHorizon, Clock.GetUtcNow().UtcDateTime))
{
results[handle.Index] = new SdkHistoryReadResult
{
StatusCode = StatusCodes.GoodNoData,
HistoryData = new ExtensionObject(ProjectEvents([], selectClauses)),
};
errors[handle.Index] = ServiceResult.Good;
return;
}
}
// A reverse read on a backend that cannot read newest-first (the Gateway) is never sent there (#675 refused it
@@ -6450,58 +6470,77 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// "full" empty page was taken for a stalled tie cluster, and a MaxTieClusterOverfetch + 1 over-fetch stepped one
// tick to return an EMPTY page with a continuation point — between every value. The lookahead row is what says
// "more data follows": a page whose read found no row past it is the last, so the read no longer ends with an
// empty trailing page either.
// empty trailing page either. (RawPageLimit.Plan computes the request: page + boundary ties + 1.)
var resumed = inboundCp is { Length: > 0 };
var requestCap = readCap == 0
? 0u
: (uint)Math.Min(uint.MaxValue, (long)readCap + (resumed ? Math.Max(0, boundarySkip) : 0) + 1L);
var boundaryTies = resumed ? Math.Max(0, boundarySkip) : 0;
var want = readCap; // N, less the start bound on a bounded first page; 0 = all values
// HistoryRead is NOT under the node-manager Lock — awaiting the async source is safe.
// A reverse window (explicit, resumed, or the end-only read above) goes to the backend only when it declares it
// can read newest-first; otherwise it is walked back from bounded forward reads (#686 — the Gateway rejects
// start > end, which failed the node and counted as a backend fault).
// A cursor left inside an oversized tie cluster (#692) skips this read: every row it could return is a tie.
// #694: a backend with a row ceiling (MaxRawValuesPerRead = L > 0, a strict gateway) sizes the page to it — the
// read asks for page + boundary ties + lookahead ≤ L, and a full page carries a continuation point as always
// (uncapped reads included). No room left after the boundary ties ⇒ the cursor sits in a tie cluster as large as
// a read: straight to the over-fetch. See RawPageLimit.
var source = HistorianDataSource;
IReadOnlyList<DataValueSnapshot> samples = [];
var backendFull = false;
int pageLimit;
if (!withinTieCluster)
{
HistorianRead sourceResult;
if (startUtc > endUtc && !source.SupportsReverseRaw)
var read = await RawPageLimit.ReadAsync(source, want, boundaryTies, async plan =>
{
var page = await ReverseRawReader.ReadAsync(
source, tagname, startUtc, endUtc, requestCap, MaxTieClusterOverfetch,
nowUtc, walkHint, ct)
.ConfigureAwait(false);
walkHint = page.Hint;
sourceResult = new HistorianRead(page.Samples, ContinuationPoint: null);
if (startUtc > endUtc && !source.SupportsReverseRaw)
{
var page = await ReverseRawReader.ReadAsync(
source, tagname, startUtc, endUtc, plan.RequestCap, MaxTieClusterOverfetch,
nowUtc, walkHint, ct, retryOnLearnedLimit: false)
.ConfigureAwait(false);
walkHint = page.Hint;
return new HistorianRead(page.Samples, ContinuationPoint: null);
}
return await source.ReadRawAsync(tagname, startUtc, endUtc, plan.RequestCap, ct).ConfigureAwait(false);
}).ConfigureAwait(false);
pageLimit = read.Plan.Limit;
if (read.Plan.NoRoom)
{
if (!resumed)
throw new HistoryReadLimitExceededException(pageLimit,
$"The backend's row ceiling ({pageLimit} values per read) leaves no room for a raw page of '{tagname}'.");
withinTieCluster = true; // every row a read could return is a boundary tie already emitted
}
else
{
sourceResult = await source
.ReadRawAsync(tagname, startUtc, endUtc, requestCap, ct)
.ConfigureAwait(false);
}
readCap = read.Plan.PageCap;
var sourceResult = read.Page!;
backendFull = HistoryPaging.IsFullPage(sourceResult.Samples.Count, read.Plan.RequestCap);
backendFull = HistoryPaging.IsFullPage(sourceResult.Samples.Count, requestCap);
// On a resume read, drop the boundary ties already returned on the prior page.
samples = resumed
? HistoryPaging.TrimBoundaryDuplicates(sourceResult.Samples, startUtc, boundarySkip)
: sourceResult.Samples;
// On a resume read, drop the boundary ties already returned on the prior page.
samples = resumed
? HistoryPaging.TrimBoundaryDuplicates(sourceResult.Samples, startUtc, boundarySkip)
: sourceResult.Samples;
// #691: a bounded read's data window excludes the end time (a sample there is the end bound). A backend whose
// forward end is inclusive (the gateway's own) returns it as data; dropping it also means the data is drained.
if (endBound is not null)
{
var inside = samples.Where(x => HistoryBounds.InDataWindow(x, endBound, resolution)).ToList();
if (inside.Count != samples.Count)
// #691: a bounded read's data window excludes the end time (a sample there is the end bound). A backend whose
// forward end is inclusive (the gateway's own) returns it as data; dropping it also means the data is drained.
if (endBound is not null)
{
samples = inside;
backendFull = false;
var inside = samples.Where(x => HistoryBounds.InDataWindow(x, endBound, resolution)).ToList();
if (inside.Count != samples.Count)
{
samples = inside;
backendFull = false;
}
}
}
}
else
{
pageLimit = source.MaxRawValuesPerRead;
}
// Oversized tie cluster: a resume read whose FULL backend read does not advance past the resume boundary
// timestamp `startUtc` — every row it returned past the trimmed ties, the lookahead row included, still sits on
@@ -6523,7 +6562,12 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// lets us DETECT a cluster strictly larger than the bound (the absurd-burst backstop below).
// Compute in uint so +1 can never wrap: Math.Max(1,…) also guards against a zero/negative
// config value slipping through (Validate() warns but does not block startup).
var overfetchCap = (uint)Math.Max(1, MaxTieClusterOverfetch) + 1u;
// #694: on a backend with a row ceiling the bound is also at most L − 1, so the over-fetch (bound + 1 rows)
// fits one read and an oversized cluster is refused here rather than by a gateway refusal.
var overfetchBound = pageLimit > 0
? Math.Min(Math.Max(1, MaxTieClusterOverfetch), Math.Max(1, pageLimit - 1))
: Math.Max(1, MaxTieClusterOverfetch);
var overfetchCap = (uint)overfetchBound + 1u;
IReadOnlyList<DataValueSnapshot> cluster = (await HistorianDataSource
.ReadRawAsync(tagname, startUtc, startUtc, overfetchCap, ct)
.ConfigureAwait(false)).Samples;
@@ -6537,12 +6581,13 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// Absurd burst: more ties than we're willing to buffer in memory. Preserve today's loud-fail
// for that node rather than over-fetch an unbounded cluster; the operator's remedy is a
// larger ServerHistorian:MaxTieClusterOverfetch (or NumValuesPerNode).
if (cluster.Count > MaxTieClusterOverfetch)
if (cluster.Count > overfetchBound)
{
_logger.LogError(
"OtOpcUaNodeManager: HistoryReadRaw tie cluster at {StartUtc:O} for tag '{Tagname}' has {Count} samples, " +
"exceeding MaxTieClusterOverfetch={Max}; cannot page within it. Increase MaxTieClusterOverfetch.",
startUtc, tagname, cluster.Count, MaxTieClusterOverfetch);
"exceeding {Max} (MaxTieClusterOverfetch, or the backend's row ceiling less one); cannot page within it. " +
"Increase MaxTieClusterOverfetch (or the HistorianGateway MaxValuesCeiling).",
startUtc, tagname, cluster.Count, overfetchBound);
errors[handle.Index] = StatusCodes.BadTooManyOperations;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadTooManyOperations };
return;
@@ -6550,8 +6595,15 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// Direction comes from the window itself (Part 11: start > end is a reverse read). A resumed reverse
// read keeps start > end, because a reverse resume point is always strictly after endUtc.
// #694: an uncapped read pages only at the backend's limit, so it slices at that page size — never 0 (a zero
// slice hands back the same cursor forever; a limit that reverted to 0 under an uncapped cursor in a cluster
// falls back to the cluster's own size).
var sliceSize = numValuesPerNode > 0 ? numValuesPerNode
: readCap > 0 ? readCap
: pageLimit > 0 ? (uint)pageLimit
: (uint)Math.Max(1, cluster.Count);
HistoryPaging.SliceTieCluster(
cluster.Count, boundarySkip, numValuesPerNode, startUtc, endUtc,
cluster.Count, boundarySkip, sliceSize, startUtc, endUtc,
out var sliceStart, out var sliceCount, out var nextStartUtc, out var nextSkip,
reverse: reverse, resolutionTicks: Math.Max(1L, HistorianDataSource.TimestampResolutionTicks));
@@ -0,0 +1,97 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
/// <summary>How one HistoryRead-Raw page reads (<see cref="RawPageLimit.Plan"/>).</summary>
/// <param name="Limit">The backend's <see cref="IHistorianDataSource.MaxRawValuesPerRead"/> the plan was made under (0 = none).</param>
/// <param name="PageCap">The page: at most this many values are returned (0 = all values — only without a limit).</param>
/// <param name="RequestCap">What the read asks for: page + the boundary ties it will trim + one lookahead row (0 = all).</param>
/// <param name="NoRoom">The boundary ties alone fill a read — nothing can be read past them; serve the tie cluster.</param>
internal readonly record struct RawPagePlan(int Limit, uint PageCap, uint RequestCap, bool NoRoom);
/// <summary>
/// Gitea #694: sizes a HistoryRead-Raw page to what the backend can return in one read
/// (<see cref="IHistorianDataSource.MaxRawValuesPerRead"/> = L — a strict HistorianGateway's learned row ceiling, less
/// its boundary-row allowance). With a limit the read — page P + boundary ties k + the #692 lookahead row — is at most L,
/// so P = min(want, L − k − 1) (all values: P = L − k − 1); a full page carries a continuation point like any other, as
/// Part 11 allows fewer values than <c>NumValuesPerNode</c> with one. Before #694 a cap above a strict gateway's ceiling
/// (or an uncapped read of a window holding more) failed <c>BadTooManyOperations</c>.
/// </summary>
/// <remarks>
/// <para>
/// <b>Never a silent truncation.</b> The backend still answers a read in full or refuses it; only the server's page is
/// smaller, and a page is judged full against what was asked for. A read the backend cannot serve at all (a tie
/// cluster larger than one read) stays <c>BadTooManyOperations</c>.
/// </para>
/// <para>
/// <b>One re-read, planned again.</b> A refused read (<see cref="HistoryReadLimitExceededException"/>) is re-planned and
/// re-issued ONCE when (a) the backend learned a different limit while it ran — the first read after start-up teaches
/// the ceiling — or (b) it was refused because rows before the window crowded the budget
/// (<see cref="HistoryReadLimitExceededException.PreWindowRows"/> m): the page is then shrunk by m so that the
/// backend's look-back fits beside it. Anything else is the answer.
/// </para>
/// </remarks>
internal static class RawPageLimit
{
/// <summary>Plans one page.</summary>
/// <param name="want">The page the client asked for (N, less a start bound; 0 = all values).</param>
/// <param name="boundaryTies">Ties at the resume timestamp already returned, which the read re-reads and trims (k).</param>
/// <param name="limit">The backend's limit L (&lt;= 0 = none).</param>
/// <param name="sizeLimit">The limit to size the page by (L, or L − m after a crowded refusal); defaults to <paramref name="limit"/>.</param>
/// <returns>The plan.</returns>
public static RawPagePlan Plan(uint want, int boundaryTies, int limit, int? sizeLimit = null)
{
var k = Math.Max(0, boundaryTies);
if (limit <= 0)
return new RawPagePlan(0, want, want == 0 ? 0u : (uint)Math.Min(uint.MaxValue, (long)want + k + 1L), false);
var room = (long)(sizeLimit ?? limit) - k - 1;
if (room < 1) return new RawPagePlan(limit, 0, 0, true);
var page = want == 0 ? room : Math.Min(want, room);
return new RawPagePlan(limit, (uint)page, (uint)(page + k + 1), false);
}
/// <summary>
/// Reads one page under the backend's current limit, re-planning and re-reading ONCE as described in the remarks.
/// A <see cref="RawPagePlan.NoRoom"/> plan is returned without a read.
/// </summary>
/// <typeparam name="T">The page.</typeparam>
/// <param name="source">The backend.</param>
/// <param name="want">The page the client asked for (0 = all values).</param>
/// <param name="boundaryTies">The boundary ties the read re-reads and trims.</param>
/// <param name="read">One read with the given plan (it asks for <see cref="RawPagePlan.RequestCap"/>).</param>
/// <returns>The page (null when <see cref="RawPagePlan.NoRoom"/>) and the plan it was read with.</returns>
public static async Task<(T? Page, RawPagePlan Plan)> ReadAsync<T>(
IHistorianDataSource source, uint want, int boundaryTies, Func<RawPagePlan, Task<T>> read)
{
var before = source.MaxRawValuesPerRead;
var plan = Plan(want, boundaryTies, before);
if (plan.NoRoom) return (default, plan);
HistoryReadLimitExceededException refused;
try
{
return (await read(plan).ConfigureAwait(false), plan);
}
catch (HistoryReadLimitExceededException ex) when (source.MaxRawValuesPerRead is var now and > 0
&& (now != before || ex.PreWindowRows > 0))
{
refused = ex; // re-read outside the catch: the second refusal is the answer
}
var limit = source.MaxRawValuesPerRead;
var shrink = Math.Max(0, refused.PreWindowRows);
plan = Plan(want, boundaryTies, limit, (int)Math.Max(0L, (long)limit - shrink));
// No room left once the crowding rows are allowed for: those rows are not (necessarily) the boundary ties, so this is
// NOT "the cursor sits in a tie cluster" — the refusal is the answer (stepping the cursor through it a tick per page
// would never end).
// (A limit that changed under an un-crowded read leaves the boundary ties alone filling a read: that IS the tie
// cluster at the cursor, served by the over-fetch.)
if (plan.NoRoom)
{
if (shrink > 0) throw refused;
return (default, plan);
}
return (await read(plan).ConfigureAwait(false), plan);
}
}
@@ -58,25 +58,72 @@ internal static class ReverseRawReader
/// <param name="ct">Cancellation.</param>
/// <returns>The page, newest first, and the hint for the next page.</returns>
/// <exception cref="HistoryReadLimitExceededException">One timestamp holds more than the tie budget.</exception>
/// <remarks>
/// Gitea #694: every segment read fits the backend's <see cref="IHistorianDataSource.MaxRawValuesPerRead"/> (a strict
/// gateway's learned ceiling) — the walk's tie budget is bounded by it so its segments are sized for it, and a
/// crowded re-read asking more is cut to it (and counts as truncated, never as complete). A walk that TEACHES the
/// backend its limit (the first read after start-up) and is refused for it is re-run once with segments sized for the
/// new limit — unless <paramref name="retryOnLearnedLimit"/> is false: the HistoryRead page path passes false, so the
/// refusal reaches <see cref="RawPageLimit"/>, which shrinks the PAGE to the limit as well (re-running the walk for a
/// page of N far above the limit cost thousands of segment reads). The bound lookups (a page of one) keep the re-run.
/// The page size itself is never cut here: a short answer would read as the last page. A segment refused because rows
/// just before it crowded the backend's budget (<see cref="HistoryReadLimitExceededException.PreWindowRows"/>) is
/// re-read once that many rows smaller and counts as truncated when full. An uncapped walk (one forward read of the
/// whole window) is never cut: it is answered in full or refused.
/// </remarks>
/// <param name="retryOnLearnedLimit">Re-run the walk once when it taught the backend its limit (see the remarks).</param>
public static async Task<ReverseRawPage> ReadAsync(
IHistorianDataSource source, string tagname, DateTime hi, DateTime lo, uint maxValues, int maxTieCluster,
DateTime nowUtc, ReverseWalkHint hint, CancellationToken ct)
DateTime nowUtc, ReverseWalkHint hint, CancellationToken ct, bool retryOnLearnedLimit = true)
{
var budget = Math.Max(1, maxTieCluster);
var (rows, next) = await ReverseWalk.WalkAsync(
async (start, end, cap, token) =>
{
var samples = (await source.ReadRawAsync(tagname, start, end, (uint)cap, token).ConfigureAwait(false)).Samples;
return new ForwardSegmentRead<DataValueSnapshot>(samples, cap > 0 && samples.Count >= cap);
},
Ticks, tieOrder: null, hi, lo, maxValues, budget, source.TimestampResolutionTicks, nowUtc, hint,
at => new HistoryReadLimitExceededException(budget,
$"A reverse raw read of '{tagname}' at {at:O} found more than {budget} samples within one millisecond; the " +
"backend reads forward only, so the latest of them cannot be selected. Increase " +
"ServerHistorian:MaxTieClusterOverfetch."),
ct)
.ConfigureAwait(false);
return new ReverseRawPage(rows, next.WidthTicks) { Hint = next };
var before = source.MaxRawValuesPerRead;
try
{
return await WalkAsync(before).ConfigureAwait(false);
}
catch (HistoryReadLimitExceededException) when (retryOnLearnedLimit && maxValues > 0
&& source.MaxRawValuesPerRead is var now and > 0 && now != before)
{
// fall through (outside the catch): the walk taught the backend its limit — re-run it sized for that limit once.
// Not an uncapped walk: it is one read of the whole window, which the limit does not change.
}
return await WalkAsync(source.MaxRawValuesPerRead).ConfigureAwait(false);
async Task<ReverseRawPage> WalkAsync(int limit)
{
// A segment asks for at most its budget + 2 rows (ReverseWalk), so the budget is bounded 2 under the limit.
var budget = Math.Max(1, maxTieCluster);
if (limit > 0) budget = Math.Min(budget, Math.Max(1, limit - 2));
var (rows, next) = await ReverseWalk.WalkAsync(
async (start, end, cap, token) =>
{
var asked = limit > 0 && cap > limit ? limit : cap; // cap 0 (the whole window) is never cut
IReadOnlyList<DataValueSnapshot> samples;
try
{
samples = (await source.ReadRawAsync(tagname, start, end, (uint)asked, token).ConfigureAwait(false)).Samples;
}
catch (HistoryReadLimitExceededException ex) when (asked > 0 && ex.PreWindowRows > 0 && asked - ex.PreWindowRows >= 1)
{
// #694: rows just before the segment crowded the backend's budget — read it smaller; a full answer
// then counts as truncated (the walk narrows), never as complete.
asked -= ex.PreWindowRows;
samples = (await source.ReadRawAsync(tagname, start, end, (uint)asked, token).ConfigureAwait(false)).Samples;
}
return new ForwardSegmentRead<DataValueSnapshot>(samples, asked > 0 && samples.Count >= asked);
},
Ticks, tieOrder: null, hi, lo, maxValues, budget, source.TimestampResolutionTicks, nowUtc, hint,
at => new HistoryReadLimitExceededException(budget,
$"A reverse raw read of '{tagname}' at {at:O} found more than {budget} samples within one millisecond; the " +
"backend reads forward only, so the latest of them cannot be selected. Increase " +
(budget < Math.Max(1, maxTieCluster)
? "the backend's row ceiling (HistorianGateway Historian:ReadLimits:MaxValuesCeiling)."
: "ServerHistorian:MaxTieClusterOverfetch.")),
ct)
.ConfigureAwait(false);
return new ReverseRawPage(rows, next.WidthTicks) { Hint = next };
}
}
private static long Ticks(DataValueSnapshot s) => (s.SourceTimestampUtc ?? DateTime.MinValue).Ticks;
@@ -106,6 +106,18 @@ public sealed class ServerHistorianOptions
/// </summary>
public bool GatewayAtTimeFromRaw { get; init; } = true;
/// <summary>
/// Gitea #694 (opt-in): HistoryReadEvents never reads events older than <c>now − EventReadHorizon</c> — the earlier
/// bound of every fresh events window is raised to it (both directions, every backend; a window wholly older is
/// <c>GoodNoData</c>). Default <see cref="TimeSpan.Zero"/> = no horizon: every read is exact to its own window. It
/// exists for the Gateway, where proving "no older events" costs a read down to the Unix epoch on
/// <c>Runtime.dbo.Events</c> (~3.8 s live per walked reverse / end-only read; an unfiltered one can hit the
/// gateway's 30 s SQL timeout) and a forward read from a far-past StartTime costs the same. ⚠️ Lossy by design:
/// events older than the horizon are invisible to HistoryReadEvents while it is set. Negative = none (a
/// <see cref="Validate"/> warning).
/// </summary>
public TimeSpan EventReadHorizon { get; init; } = TimeSpan.Zero;
/// <summary>
/// Maximum number of historized nodes served CONCURRENTLY within one HistoryRead batch (arch-review
/// 03/S3). The SDK's <c>HistoryRead*</c> override surface is synchronous, so each arm block-bridges
@@ -196,6 +208,8 @@ public sealed class ServerHistorianOptions
warnings.Add($"ServerHistorian:MaxConcurrentHistoryReadBatches is {MaxConcurrentHistoryReadBatches} — must be > 0; the process-wide HistoryRead batch limiter is disabled.");
if (MaxTimeWeightedRawRows <= 0)
warnings.Add($"ServerHistorian:MaxTimeWeightedRawRows is {MaxTimeWeightedRawRows} — must be > 0; time-weighted aggregate reads on the SqlServer/QuestDb backends stream an unbounded number of raw samples.");
if (EventReadHorizon < TimeSpan.Zero)
warnings.Add($"ServerHistorian:EventReadHorizon is {EventReadHorizon} — negative is treated as no horizon (00:00:00); HistoryReadEvents reads every window in full.");
if (HistoryReadDeadline <= TimeSpan.Zero)
warnings.Add($"ServerHistorian:HistoryReadDeadline is {HistoryReadDeadline} — must be > 0; HistoryRead requests run unbounded (no per-request deadline).");
return warnings;