dohertj2/lmxopcua
1
0
Fork 0
Code Issues 32 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(historian): #694 page raw reads at a strict gateway's learned ceiling; opt-in EventReadHorizon

Browse Source 
Item 1: IHistorianDataSource.MaxRawValuesPerRead (Gateway: learned ceiling - 16 = L; 0 elsewhere, guard-tested).
RawPageLimit plans every raw page on top of #692's lookahead: with L > 0 the read (page P + boundary ties k
+ lookahead) is <= L, P = min(N, L - k - 1) (uncapped: L - k - 1); no room => straight to the tie-cluster
over-fetch (never an uncapped or underflowing request). One re-plan + re-read when the read taught a new
limit or was refused for crowding pre-window rows (HistoryReadLimitExceededException.PreWindowRows): the
page shrinks by them. Over-fetch bound min(MaxTieClusterOverfetch, L - 1); slice size never 0.
Review fixes: F1 a window starting at/before the epoch counts AVEVA's synthetic row as pre-window (a
ceiling-truncated read was reported complete); F2 the page path disables ReverseRawReader's own re-run
(cold end-only ceiling 20/N=3000 cost 3,023 reads, now <= 40); F3 crowded look-back re-read around, not
refused; F4 the #687 crowding re-read room at least doubles (up to 8 re-reads), so a cluster of hundreds
just before a resume point no longer fails on a default gateway; a crowding refusal that leaves no room
is BadTooManyOperations, never a tick-by-tick cursor.

Item 2: ServerHistorian:EventReadHorizon (default 00:00:00 = exact, unchanged). When set, every fresh
HistoryReadEvents window's earlier bound is raised to now - horizon (both directions, all backends).
Lossy by design; documented.

Item 3 (native TimeAverage/Total re-check) stays blocked on historiangw#27.
This commit is contained in:
Joseph Doherty
2026-10-13 00:52:12 -04:00
parent a5a0496176
commit 3c3513ea3e
23 changed files with 1461 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs
+87 -35
View File
@@ -552,6 +552,13 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
/// </summary>
public bool HonorReturnBounds { get; set; } = true;
/// <summary>
/// Gitea #694: <c>ServerHistorian:EventReadHorizon</c> — HistoryReadEvents never reads events older than
/// <c>now − EventReadHorizon</c> (<see cref="EventHorizon"/>). <see cref="TimeSpan.Zero"/> (the default) = no horizon:
/// every read is exact down to its own window (a walked Gateway reverse read proves "nothing older" down to 1970).
/// </summary>
public TimeSpan EventReadHorizon { get; set; } = TimeSpan.Zero;
/// <summary>
/// Maximum historized nodes served concurrently within one HistoryRead batch (arch-review 03/S3).
/// Mirrors <c>ServerHistorianOptions.HistoryReadBatchParallelism</c>; the Host sets it at
@@ -5339,6 +5346,19 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// Part 11: StartTime later than EndTime is a reverse read — events newest-first over (end, start].
reverse = startUtc > endUtc;
// #694: the opt-in events horizon raises the window's earlier bound (the cursor then carries it). A window
// wholly older than it is empty and is not sent (a forward [start, end) clamped past its end would be refused).
if (!EventHorizon.TryClamp(ref startUtc, ref endUtc, reverse, EventReadHorizon, Clock.GetUtcNow().UtcDateTime))
{
results[handle.Index] = new SdkHistoryReadResult
{
StatusCode = StatusCodes.GoodNoData,
HistoryData = new ExtensionObject(ProjectEvents([], selectClauses)),
};
errors[handle.Index] = ServiceResult.Good;
return;
}
}
// A reverse read on a backend that cannot read newest-first (the Gateway) is never sent there (#675 refused it
@@ -6450,58 +6470,77 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// "full" empty page was taken for a stalled tie cluster, and a MaxTieClusterOverfetch + 1 over-fetch stepped one
// tick to return an EMPTY page with a continuation point — between every value. The lookahead row is what says
// "more data follows": a page whose read found no row past it is the last, so the read no longer ends with an
// empty trailing page either.
// empty trailing page either. (RawPageLimit.Plan computes the request: page + boundary ties + 1.)
var resumed = inboundCp is { Length: > 0 };
var requestCap = readCap == 0
? 0u
: (uint)Math.Min(uint.MaxValue, (long)readCap + (resumed ? Math.Max(0, boundarySkip) : 0) + 1L);
var boundaryTies = resumed ? Math.Max(0, boundarySkip) : 0;
var want = readCap; // N, less the start bound on a bounded first page; 0 = all values
// HistoryRead is NOT under the node-manager Lock — awaiting the async source is safe.
// A reverse window (explicit, resumed, or the end-only read above) goes to the backend only when it declares it
// can read newest-first; otherwise it is walked back from bounded forward reads (#686 — the Gateway rejects
// start > end, which failed the node and counted as a backend fault).
// A cursor left inside an oversized tie cluster (#692) skips this read: every row it could return is a tie.
// #694: a backend with a row ceiling (MaxRawValuesPerRead = L > 0, a strict gateway) sizes the page to it — the
// read asks for page + boundary ties + lookahead ≤ L, and a full page carries a continuation point as always
// (uncapped reads included). No room left after the boundary ties ⇒ the cursor sits in a tie cluster as large as
// a read: straight to the over-fetch. See RawPageLimit.
var source = HistorianDataSource;
IReadOnlyList<DataValueSnapshot> samples = [];
var backendFull = false;
int pageLimit;
if (!withinTieCluster)
{
HistorianRead sourceResult;
if (startUtc > endUtc && !source.SupportsReverseRaw)
var read = await RawPageLimit.ReadAsync(source, want, boundaryTies, async plan =>
{
var page = await ReverseRawReader.ReadAsync(
source, tagname, startUtc, endUtc, requestCap, MaxTieClusterOverfetch,
nowUtc, walkHint, ct)
.ConfigureAwait(false);
walkHint = page.Hint;
sourceResult = new HistorianRead(page.Samples, ContinuationPoint: null);
if (startUtc > endUtc && !source.SupportsReverseRaw)
{
var page = await ReverseRawReader.ReadAsync(
source, tagname, startUtc, endUtc, plan.RequestCap, MaxTieClusterOverfetch,
nowUtc, walkHint, ct, retryOnLearnedLimit: false)
.ConfigureAwait(false);
walkHint = page.Hint;
return new HistorianRead(page.Samples, ContinuationPoint: null);
}
return await source.ReadRawAsync(tagname, startUtc, endUtc, plan.RequestCap, ct).ConfigureAwait(false);
}).ConfigureAwait(false);
pageLimit = read.Plan.Limit;
if (read.Plan.NoRoom)
{
if (!resumed)
throw new HistoryReadLimitExceededException(pageLimit,
$"The backend's row ceiling ({pageLimit} values per read) leaves no room for a raw page of '{tagname}'.");
withinTieCluster = true; // every row a read could return is a boundary tie already emitted
}
else
{
sourceResult = await source
.ReadRawAsync(tagname, startUtc, endUtc, requestCap, ct)
.ConfigureAwait(false);
}
readCap = read.Plan.PageCap;
var sourceResult = read.Page!;
backendFull = HistoryPaging.IsFullPage(sourceResult.Samples.Count, read.Plan.RequestCap);
backendFull = HistoryPaging.IsFullPage(sourceResult.Samples.Count, requestCap);
// On a resume read, drop the boundary ties already returned on the prior page.
samples = resumed
? HistoryPaging.TrimBoundaryDuplicates(sourceResult.Samples, startUtc, boundarySkip)
: sourceResult.Samples;
// On a resume read, drop the boundary ties already returned on the prior page.
samples = resumed
? HistoryPaging.TrimBoundaryDuplicates(sourceResult.Samples, startUtc, boundarySkip)
: sourceResult.Samples;
// #691: a bounded read's data window excludes the end time (a sample there is the end bound). A backend whose
// forward end is inclusive (the gateway's own) returns it as data; dropping it also means the data is drained.
if (endBound is not null)
{
var inside = samples.Where(x => HistoryBounds.InDataWindow(x, endBound, resolution)).ToList();
if (inside.Count != samples.Count)
// #691: a bounded read's data window excludes the end time (a sample there is the end bound). A backend whose
// forward end is inclusive (the gateway's own) returns it as data; dropping it also means the data is drained.
if (endBound is not null)
{
samples = inside;
backendFull = false;
var inside = samples.Where(x => HistoryBounds.InDataWindow(x, endBound, resolution)).ToList();
if (inside.Count != samples.Count)
{
samples = inside;
backendFull = false;
}
}
}
}
else
{
pageLimit = source.MaxRawValuesPerRead;
}
// Oversized tie cluster: a resume read whose FULL backend read does not advance past the resume boundary
// timestamp `startUtc` — every row it returned past the trimmed ties, the lookahead row included, still sits on
@@ -6523,7 +6562,12 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// lets us DETECT a cluster strictly larger than the bound (the absurd-burst backstop below).
// Compute in uint so +1 can never wrap: Math.Max(1,…) also guards against a zero/negative
// config value slipping through (Validate() warns but does not block startup).
var overfetchCap = (uint)Math.Max(1, MaxTieClusterOverfetch) + 1u;
// #694: on a backend with a row ceiling the bound is also at most L − 1, so the over-fetch (bound + 1 rows)
// fits one read and an oversized cluster is refused here rather than by a gateway refusal.
var overfetchBound = pageLimit > 0
? Math.Min(Math.Max(1, MaxTieClusterOverfetch), Math.Max(1, pageLimit - 1))
: Math.Max(1, MaxTieClusterOverfetch);
var overfetchCap = (uint)overfetchBound + 1u;
IReadOnlyList<DataValueSnapshot> cluster = (await HistorianDataSource
.ReadRawAsync(tagname, startUtc, startUtc, overfetchCap, ct)
.ConfigureAwait(false)).Samples;
@@ -6537,12 +6581,13 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// Absurd burst: more ties than we're willing to buffer in memory. Preserve today's loud-fail
// for that node rather than over-fetch an unbounded cluster; the operator's remedy is a
// larger ServerHistorian:MaxTieClusterOverfetch (or NumValuesPerNode).
if (cluster.Count > MaxTieClusterOverfetch)
if (cluster.Count > overfetchBound)
{
_logger.LogError(
"OtOpcUaNodeManager: HistoryReadRaw tie cluster at {StartUtc:O} for tag '{Tagname}' has {Count} samples, " +
"exceeding MaxTieClusterOverfetch={Max}; cannot page within it. Increase MaxTieClusterOverfetch.",
startUtc, tagname, cluster.Count, MaxTieClusterOverfetch);
"exceeding {Max} (MaxTieClusterOverfetch, or the backend's row ceiling less one); cannot page within it. " +
"Increase MaxTieClusterOverfetch (or the HistorianGateway MaxValuesCeiling).",
startUtc, tagname, cluster.Count, overfetchBound);
errors[handle.Index] = StatusCodes.BadTooManyOperations;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadTooManyOperations };
return;
@@ -6550,8 +6595,15 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// Direction comes from the window itself (Part 11: start > end is a reverse read). A resumed reverse
// read keeps start > end, because a reverse resume point is always strictly after endUtc.
// #694: an uncapped read pages only at the backend's limit, so it slices at that page size — never 0 (a zero
// slice hands back the same cursor forever; a limit that reverted to 0 under an uncapped cursor in a cluster
// falls back to the cluster's own size).
var sliceSize = numValuesPerNode > 0 ? numValuesPerNode
: readCap > 0 ? readCap
: pageLimit > 0 ? (uint)pageLimit
: (uint)Math.Max(1, cluster.Count);
HistoryPaging.SliceTieCluster(
cluster.Count, boundarySkip, numValuesPerNode, startUtc, endUtc,
cluster.Count, boundarySkip, sliceSize, startUtc, endUtc,
out var sliceStart, out var sliceCount, out var nextStartUtc, out var nextSkip,
reverse: reverse, resolutionTicks: Math.Max(1L, HistorianDataSource.TimestampResolutionTicks));