feat(historian): #691 HistoryRead-Raw ReturnBounds — Part 11 bounding values on every backend

The node manager composes the start/end bounds from each backend's existing raw
reads (one authority, HistoryBounds): at-or-before = a capped reverse read
(native, or ReverseRawReader's walk-back on a forward-only backend — never
AVEVA's synthetic boundary row), at-or-after = a capped forward read. Bounds
count toward NumValuesPerNode, the start bound leads the first page, the end
bound ends the last (carried in the continuation point). Missing bound =
Bad_BoundNotFound (null, at the bound time; one-sided reads ±1 s from the
previous value, Table 1 footnotes a/b). A modified read with ReturnBounds is
Bad_InvalidArgument (Part 11 6.5.3.3).

The .NET SDK's ReadRawModifiedDetails defaults ReturnBounds to TRUE; the
existing tests built details without setting it and now set it false.
Oracle: every row of Part 11 Table 1 over three backend shapes, plus paging
concatenation, ties, Bad bounds, microsecond truncation; conformance pins the
lookup primitives on SqlServer + QuestDB.
This commit is contained in:
Joseph Doherty
2026-10-12 16:25:00 -04:00
parent 88141d6ed5
commit 60febbf2dc
14 changed files with 918 additions and 11 deletions
@@ -0,0 +1,197 @@
using Opc.Ua;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
/// <summary>
/// The end bound still owed to a HistoryRead-Raw read with <c>ReturnBounds = true</c> (Gitea #691). It rides the
/// continuation point, because Part 11 returns the end bound with the LAST page only.
/// </summary>
/// <param name="AtUtc">The request's end time (the later bound of a forward read, the earlier of a reverse one).</param>
/// <param name="Reverse">The read is newest-first (<c>StartTime &gt; EndTime</c>, including an end-only read).</param>
/// <param name="Inclusive">
/// <c>false</c> only for <c>StartTime == EndTime</c>: the samples AT that instant are the data, so the end bound is the
/// first sample strictly after it (Part 11 Table 1 footnote c). Otherwise the end bound is the sample at or beyond the
/// end time, and the data window excludes the end time.
/// </param>
/// <param name="Open">The end time was not given (a one-sided read's sentinel): there is no end bound to look up.</param>
/// <param name="DataExhausted">The data is drained and the end bound did not fit on the previous page.</param>
/// <param name="LastReturnedUtc">The last value's time returned on an earlier page (stamps an open bound's placeholder).</param>
internal sealed record HistoryEndBound(
DateTime AtUtc, bool Reverse, bool Inclusive, bool Open, bool DataExhausted = false, DateTime? LastReturnedUtc = null);
/// <summary>
/// Gitea #691: OPC UA Part 11 bounding values for HistoryRead-Raw (<c>ReadRawModifiedDetails.ReturnBounds</c>), composed
/// in the server from every backend's existing raw read — the ONE authority, so SqlServer, QuestDB, Gateway and Null
/// answer identically.
/// </summary>
/// <remarks>
/// <para>
/// <b>What a bound is</b> (Part 11 §4.6, Table 1). Forward: the start bound is the stored sample AT or immediately
/// BEFORE StartTime, the end bound the one AT or immediately AFTER EndTime. Reverse mirrors it: the start bound is at
/// or after StartTime, the end bound at or before EndTime. A sample exactly on a bound IS the bound: on the start it
/// is already the first sample of the data (the window includes StartTime), so nothing extra is added; on the end it is
/// added (the window excludes EndTime). <c>StartTime == EndTime</c> reads forward, and its end bound is the first
/// sample strictly after the instant. A bound is a stored sample of any quality — a Bad sample is returned as stored.
/// </para>
/// <para>
/// <b>Missing bound.</b> <c>Bad_BoundNotFound</c>, a null value, stamped with the bound's time. A one-sided read's
/// missing side (no EndTime / no StartTime) is never looked up; it is <c>Bad_BoundNotFound</c> stamped one second after
/// (forward) or before (reverse) the previous returned value (Table 1 footnotes a and b).
/// </para>
/// <para>
/// <b>Lookups.</b> "At or before T" is a newest-first read of <c>(0, T]</c> capped at one — native on a backend that
/// declares <see cref="IHistorianDataSource.SupportsReverseRaw"/>, otherwise <see cref="ReverseRawReader"/>'s walk-back
/// from bounded forward reads (the Gateway: NOT AVEVA's synthetic boundary row, which is stamped at the query start
/// rather than at the prior sample's own time). "At or after T" is a forward read of <c>[T, MaxValue)</c> capped at one;
/// "strictly after T" starts one storage resolution later (lossless: nothing is stored in between). Inside a tie
/// cluster the bound is the cluster's first sample in the read's direction — the forward read's first, the reverse
/// read's first (the forward order's last).
/// </para>
/// <para>
/// <b>Resolution.</b> "On the bound" compares at the backend's storage resolution, because a microsecond backend
/// (QuestDB) truncates the window bounds it is sent: a sample at <c>⌊T⌋</c> is inside a window starting at <c>T</c>
/// there, so it must not also be returned as the start bound.
/// </para>
/// </remarks>
internal static class HistoryBounds
{
private static readonly TimeSpan OpenBoundStep = TimeSpan.FromSeconds(1);
/// <summary>The end bound owed by a fresh read over the normalised window.</summary>
/// <param name="startUtc">The (normalised) start: the later time of a reverse read.</param>
/// <param name="endUtc">The (normalised) end; a one-sided read's sentinel (tick 0 / MaxValue) is an open bound.</param>
/// <returns>The pending end bound.</returns>
public static HistoryEndBound For(DateTime startUtc, DateTime endUtc) =>
new(endUtc, Reverse: startUtc > endUtc, Inclusive: startUtc != endUtc, Open: IsOpen(endUtc));
/// <summary>The pending end bound to store behind a continuation point, remembering the page's last returned time.</summary>
/// <param name="bound">The pending end bound (null = bounds not requested).</param>
/// <param name="page">The page being returned.</param>
/// <returns>The bound to carry, or null.</returns>
public static HistoryEndBound? Carry(HistoryEndBound? bound, IReadOnlyList<DataValueSnapshot> page) =>
bound is null || page.Count == 0 || page[^1].SourceTimestampUtc is not { } last
? bound
: bound with { LastReturnedUtc = last };
/// <summary>The <c>Bad_BoundNotFound</c> placeholder (null value) stamped at <paramref name="atUtc"/>.</summary>
/// <param name="atUtc">The bound's time.</param>
/// <returns>The placeholder.</returns>
public static DataValueSnapshot NotFound(DateTime atUtc) =>
new(null, StatusCodes.BadBoundNotFound, atUtc, atUtc);
/// <summary>
/// The start bound to return ahead of the data, or <c>null</c> when a stored sample sits on StartTime (that sample is
/// the bound, and it is already the data's first sample).
/// </summary>
public static async Task<DataValueSnapshot?> StartBoundAsync(
IHistorianDataSource source, string tagname, DateTime startUtc, bool reverse, int maxTieCluster, DateTime nowUtc,
CancellationToken ct)
{
if (IsOpen(startUtc)) return NotFound(startUtc);
var found = reverse
? await EarliestAtOrAfterAsync(source, tagname, startUtc, ct).ConfigureAwait(false)
: await LatestAtOrBeforeAsync(source, tagname, startUtc, maxTieCluster, nowUtc, ct).ConfigureAwait(false);
if (found is null) return NotFound(startUtc);
return SameInstant(Ticks(found), startUtc.Ticks, Resolution(source)) ? null : found;
}
/// <summary>The end bound: the stored sample on or beyond the end time, else the placeholder.</summary>
/// <param name="source">The backend.</param>
/// <param name="tagname">Historian tag.</param>
/// <param name="bound">The pending end bound.</param>
/// <param name="previousReturnedUtc">The last value's time already returned (stamps an open bound's placeholder).</param>
/// <param name="maxTieCluster">The walk-back's row budget (forward-only backends).</param>
/// <param name="nowUtc">Now (the walk-back's look-ahead clamp).</param>
/// <param name="ct">Cancellation.</param>
public static async Task<DataValueSnapshot> EndBoundAsync(
IHistorianDataSource source, string tagname, HistoryEndBound bound, DateTime previousReturnedUtc,
int maxTieCluster, DateTime nowUtc, CancellationToken ct)
{
if (bound.Open)
{
var ticks = bound.Reverse
? Math.Max(0L, previousReturnedUtc.Ticks - OpenBoundStep.Ticks)
: Math.Min(DateTime.MaxValue.Ticks, previousReturnedUtc.Ticks + OpenBoundStep.Ticks);
return NotFound(new DateTime(ticks, DateTimeKind.Utc));
}
DataValueSnapshot? found;
if (bound.Reverse)
found = await LatestAtOrBeforeAsync(source, tagname, bound.AtUtc, maxTieCluster, nowUtc, ct).ConfigureAwait(false);
else if (bound.Inclusive)
found = await EarliestAtOrAfterAsync(source, tagname, bound.AtUtc, ct).ConfigureAwait(false);
else
{
var resolution = Resolution(source);
found = bound.AtUtc.Ticks > DateTime.MaxValue.Ticks - resolution
? null
: await EarliestAtOrAfterAsync(source, tagname, bound.AtUtc.AddTicks(resolution), ct).ConfigureAwait(false);
}
return found ?? NotFound(bound.AtUtc);
}
/// <summary>
/// Whether a data sample lies inside the data window of a bounded read — a forward window excludes the end time (a
/// sample there is the end bound), compared at the storage resolution. A backend whose forward end is inclusive, or a
/// resumed read that steps onto the end time, would otherwise return the end bound twice.
/// </summary>
public static bool InDataWindow(DataValueSnapshot sample, HistoryEndBound bound, long resolution)
{
if (!bound.Inclusive || bound.Open) return true;
var ts = Align(Ticks(sample), resolution);
var end = Align(bound.AtUtc.Ticks, resolution);
return bound.Reverse ? ts > end : ts < end;
}
/// <summary>Whether a resumed read's next start is already past the data window (only the end bound remains).</summary>
public static bool DataWindowDrained(DateTime nextStartUtc, HistoryEndBound bound, long resolution)
{
if (!bound.Inclusive || bound.Open) return false;
var next = Align(nextStartUtc.Ticks, resolution);
var end = Align(bound.AtUtc.Ticks, resolution);
return bound.Reverse ? next <= end : next >= end;
}
/// <summary>The latest stored sample at or before <paramref name="atUtc"/>, or null.</summary>
internal static async Task<DataValueSnapshot?> LatestAtOrBeforeAsync(
IHistorianDataSource source, string tagname, DateTime atUtc, int maxTieCluster, DateTime nowUtc, CancellationToken ct)
{
if (atUtc.Ticks == 0) return null;
var floor = new DateTime(0, DateTimeKind.Utc);
IReadOnlyList<DataValueSnapshot> rows = source.SupportsReverseRaw
? (await source.ReadRawAsync(tagname, atUtc, floor, 1, ct).ConfigureAwait(false)).Samples
: (await ReverseRawReader.ReadAsync(source, tagname, atUtc, floor, 1, maxTieCluster, nowUtc, 0, ct)
.ConfigureAwait(false)).Samples;
var limit = Align(atUtc.Ticks, Resolution(source));
foreach (var row in rows)
if (Align(Ticks(row), Resolution(source)) <= limit)
return row;
return null;
}
/// <summary>The earliest stored sample at or after <paramref name="atUtc"/>, or null.</summary>
internal static async Task<DataValueSnapshot?> EarliestAtOrAfterAsync(
IHistorianDataSource source, string tagname, DateTime atUtc, CancellationToken ct)
{
if (atUtc.Ticks >= DateTime.MaxValue.Ticks) return null;
var rows = (await source.ReadRawAsync(tagname, atUtc, new DateTime(DateTime.MaxValue.Ticks, DateTimeKind.Utc), 1, ct)
.ConfigureAwait(false)).Samples;
var limit = Align(atUtc.Ticks, Resolution(source));
foreach (var row in rows)
if (Align(Ticks(row), Resolution(source)) >= limit)
return row;
return null;
}
private static bool IsOpen(DateTime t) => t.Ticks == 0 || t.Ticks == DateTime.MaxValue.Ticks;
private static bool SameInstant(long a, long b, long resolution) => Align(a, resolution) == Align(b, resolution);
private static long Resolution(IHistorianDataSource source) => Math.Max(1L, source.TimestampResolutionTicks);
private static long Ticks(DataValueSnapshot s) => (s.SourceTimestampUtc ?? DateTime.MinValue).Ticks;
private static long Align(long ticks, long resolution) => ticks - ticks % resolution;
}
@@ -62,6 +62,10 @@ namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
/// serialized), so a new field with a "none" default is all the versioning it needs. /// serialized), so a new field with a "none" default is all the versioning it needs.
/// </param> /// </param>
/// <param name="EmptyToTicks">The exclusive upper end of that stretch.</param> /// <param name="EmptyToTicks">The exclusive upper end of that stretch.</param>
/// <param name="PendingEndBound">
/// <see cref="HistoryCursorKind.Raw"/> with <c>ReturnBounds = true</c> only (Gitea #691): the end bound still owed —
/// Part 11 returns it with the last page. <c>null</c> = bounds not requested.
/// </param>
internal sealed record HistoryContinuationState( internal sealed record HistoryContinuationState(
string Tagname, string Tagname,
DateTime NextStartUtc, DateTime NextStartUtc,
@@ -74,7 +78,8 @@ internal sealed record HistoryContinuationState(
bool Reverse = false, bool Reverse = false,
long SegmentWidthTicks = 0, long SegmentWidthTicks = 0,
long EmptyFromTicks = 0, long EmptyFromTicks = 0,
long EmptyToTicks = 0) long EmptyToTicks = 0,
HistoryEndBound? PendingEndBound = null)
{ {
/// <summary>The walk-back hint this cursor carries (#686/#689/#691).</summary> /// <summary>The walk-back hint this cursor carries (#686/#689/#691).</summary>
public ReverseWalkHint WalkHint => new(SegmentWidthTicks, EmptyFromTicks, EmptyToTicks); public ReverseWalkHint WalkHint => new(SegmentWidthTicks, EmptyFromTicks, EmptyToTicks);
@@ -5056,6 +5056,15 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
var work = new List<Func<Task>>(nodesToProcess.Count); var work = new List<Func<Task>>(nodesToProcess.Count);
foreach (var handle in nodesToProcess) foreach (var handle in nodesToProcess)
{ {
if (details.IsReadModified && details.ReturnBounds)
{
// Part 11 §6.5.3.3: returnBounds "shall be set to FALSE" for a modified read, "otherwise the Server returns
// a Bad_InvalidArgument" (#691).
errors[handle.Index] = StatusCodes.BadInvalidArgument;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadInvalidArgument };
continue;
}
if (details.IsReadModified) if (details.IsReadModified)
{ {
// Part 11 modified-value history (#668): the prior values every HistoryUpdate recorded. Gated by the // Part 11 modified-value history (#668): the prior values every HistoryUpdate recorded. Gated by the
@@ -5070,7 +5079,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
work.Add(() => ServeRawPagedAsync( work.Add(() => ServeRawPagedAsync(
handle, session, nodesToRead, results, errors, handle, session, nodesToRead, results, errors,
details.StartTime, details.EndTime, details.NumValuesPerNode, details.StartTime, details.EndTime, details.NumValuesPerNode,
ct)); ct, details.ReturnBounds));
} }
RunBounded(work, HistoryReadBatchParallelism); RunBounded(work, HistoryReadBatchParallelism);
@@ -6197,6 +6206,11 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
/// <param name="startTimeUtc">The request window's (inclusive) lower bound, used for a fresh read.</param> /// <param name="startTimeUtc">The request window's (inclusive) lower bound, used for a fresh read.</param>
/// <param name="endUtc">The (inclusive) upper bound of the read window; unchanged across pages.</param> /// <param name="endUtc">The (inclusive) upper bound of the read window; unchanged across pages.</param>
/// <param name="numValuesPerNode">The client's per-page cap; <c>0</c> means "all values, no paging".</param> /// <param name="numValuesPerNode">The client's per-page cap; <c>0</c> means "all values, no paging".</param>
/// <param name="ct">Cancellation (the per-request deadline).</param>
/// <param name="returnBounds">
/// Part 11 bounding values (#691, <see cref="HistoryBounds"/>): the start bound leads the first page, the end bound
/// ends the last, and both count toward <paramref name="numValuesPerNode"/>. A resumed read takes it from the cursor.
/// </param>
private async Task ServeRawPagedAsync( private async Task ServeRawPagedAsync(
NodeHandle handle, NodeHandle handle,
ISession? session, ISession? session,
@@ -6206,7 +6220,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
DateTime startTimeUtc, DateTime startTimeUtc,
DateTime endUtc, DateTime endUtc,
uint numValuesPerNode, uint numValuesPerNode,
CancellationToken ct) CancellationToken ct,
bool returnBounds = false)
{ {
var inboundCp = nodesToRead[handle.Index].ContinuationPoint; var inboundCp = nodesToRead[handle.Index].ContinuationPoint;
@@ -6215,6 +6230,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
DateTime startUtc; DateTime startUtc;
var boundarySkip = 0; var boundarySkip = 0;
var walkHint = default(ReverseWalkHint); // #686/#689/#691: the walk-back's width + proven-empty stretch, page to page var walkHint = default(ReverseWalkHint); // #686/#689/#691: the walk-back's width + proven-empty stretch, page to page
HistoryEndBound? endBound = null; // #691: the end bound still owed (null = bounds not requested)
string tagname; string tagname;
if (inboundCp is { Length: > 0 }) if (inboundCp is { Length: > 0 })
@@ -6236,6 +6252,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
endUtc = state.EndUtc; endUtc = state.EndUtc;
numValuesPerNode = state.NumValuesPerNode; numValuesPerNode = state.NumValuesPerNode;
walkHint = state.WalkHint; walkHint = state.WalkHint;
endBound = state.PendingEndBound;
} }
else else
{ {
@@ -6261,6 +6278,78 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
else if (startUtc == DateTime.MinValue && endUtc != DateTime.MinValue) else if (startUtc == DateTime.MinValue && endUtc != DateTime.MinValue)
(startUtc, endUtc) = (endUtc, new DateTime(0, DateTimeKind.Utc)); (startUtc, endUtc) = (endUtc, new DateTime(0, DateTimeKind.Utc));
} }
if (returnBounds) endBound = HistoryBounds.For(startUtc, endUtc);
}
var nowUtc = Clock.GetUtcNow().UtcDateTime;
var resolution = Math.Max(1L, HistorianDataSource.TimestampResolutionTicks);
// #691: the data is drained — the end bound ends this page when it has room (Part 11: bounds count toward
// NumValuesPerNode), else it is the whole next page. Returns the continuation point to emit, if any.
async Task<byte[]?> AppendEndBoundAsync(List<DataValueSnapshot> page, DateTime resumedAtUtc)
{
// The previous returned value stamps an open bound's placeholder (Part 11 Table 1 footnotes a/b): this
// page's last, else the last of an earlier page (carried in the cursor), else where the read resumed.
var previous = (page.Count > 0 ? page[^1].SourceTimestampUtc : null)
?? endBound!.LastReturnedUtc ?? resumedAtUtc;
if (numValuesPerNode == 0 || page.Count < numValuesPerNode)
{
page.Add(await HistoryBounds.EndBoundAsync(
HistorianDataSource, tagname, endBound!, previous, MaxTieClusterOverfetch, nowUtc, ct).ConfigureAwait(false));
return null;
}
return _historyContinuationStore.Save(session, new HistoryContinuationState(
tagname, previous, endUtc, 0, numValuesPerNode, SegmentWidthTicks: walkHint.WidthTicks,
EmptyFromTicks: walkHint.EmptyFromTicks, EmptyToTicks: walkHint.EmptyToTicks,
PendingEndBound: HistoryBounds.Carry(endBound!, page)! with { DataExhausted = true }));
}
// #691: a resumed bounded read whose data is drained (the end bound did not fit on the previous page, or the
// resume point already sits on the end time) owes only the end bound.
if (endBound is not null && inboundCp is { Length: > 0 }
&& (endBound.DataExhausted || HistoryBounds.DataWindowDrained(startUtc, endBound, resolution)))
{
var onlyBound = new List<DataValueSnapshot>(1);
var afterCp = await AppendEndBoundAsync(onlyBound, startUtc).ConfigureAwait(false);
results[handle.Index] = new SdkHistoryReadResult
{
StatusCode = StatusCodes.Good,
HistoryData = new ExtensionObject(ToHistoryDataFromSamples(onlyBound)),
ContinuationPoint = afterCp,
};
errors[handle.Index] = ServiceResult.Good;
return;
}
// #691: the start bound leads the first page and counts toward the cap; null when a sample sits on StartTime
// (that sample is the bound, and it is the data's first).
DataValueSnapshot? startBound = null;
var readCap = numValuesPerNode;
if (endBound is not null && inboundCp is not { Length: > 0 })
{
startBound = await HistoryBounds.StartBoundAsync(
HistorianDataSource, tagname, startUtc, endBound.Reverse, MaxTieClusterOverfetch, nowUtc, ct)
.ConfigureAwait(false);
if (startBound is not null && numValuesPerNode == 1)
{
// The page is the start bound alone; the data starts at StartTime on the next page.
var boundOnlyCp = _historyContinuationStore.Save(session, new HistoryContinuationState(
tagname, startUtc, endUtc, 0, numValuesPerNode, SegmentWidthTicks: walkHint.WidthTicks,
EmptyFromTicks: walkHint.EmptyFromTicks, EmptyToTicks: walkHint.EmptyToTicks,
PendingEndBound: HistoryBounds.Carry(endBound, [startBound])));
results[handle.Index] = new SdkHistoryReadResult
{
StatusCode = StatusCodes.Good,
HistoryData = new ExtensionObject(ToHistoryDataFromSamples([startBound])),
ContinuationPoint = boundOnlyCp,
};
errors[handle.Index] = ServiceResult.Good;
return;
}
if (startBound is not null && numValuesPerNode > 0) readCap = numValuesPerNode - 1;
} }
// HistoryRead is NOT under the node-manager Lock — awaiting the async source is safe. // HistoryRead is NOT under the node-manager Lock — awaiting the async source is safe.
@@ -6272,8 +6361,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
if (startUtc > endUtc && !source.SupportsReverseRaw) if (startUtc > endUtc && !source.SupportsReverseRaw)
{ {
var page = await ReverseRawReader.ReadAsync( var page = await ReverseRawReader.ReadAsync(
source, tagname, startUtc, endUtc, numValuesPerNode, MaxTieClusterOverfetch, source, tagname, startUtc, endUtc, readCap, MaxTieClusterOverfetch,
Clock.GetUtcNow().UtcDateTime, walkHint, ct) nowUtc, walkHint, ct)
.ConfigureAwait(false); .ConfigureAwait(false);
walkHint = page.Hint; walkHint = page.Hint;
sourceResult = new HistorianRead(page.Samples, ContinuationPoint: null); sourceResult = new HistorianRead(page.Samples, ContinuationPoint: null);
@@ -6281,17 +6370,29 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
else else
{ {
sourceResult = await source sourceResult = await source
.ReadRawAsync(tagname, startUtc, endUtc, numValuesPerNode, ct) .ReadRawAsync(tagname, startUtc, endUtc, readCap, ct)
.ConfigureAwait(false); .ConfigureAwait(false);
} }
var backendFull = HistoryPaging.IsFullPage(sourceResult.Samples.Count, numValuesPerNode); var backendFull = HistoryPaging.IsFullPage(sourceResult.Samples.Count, readCap);
// On a resume read, drop the boundary ties already returned on the prior page. // On a resume read, drop the boundary ties already returned on the prior page.
var samples = inboundCp is { Length: > 0 } var samples = inboundCp is { Length: > 0 }
? HistoryPaging.TrimBoundaryDuplicates(sourceResult.Samples, startUtc, boundarySkip) ? HistoryPaging.TrimBoundaryDuplicates(sourceResult.Samples, startUtc, boundarySkip)
: sourceResult.Samples; : sourceResult.Samples;
// #691: a bounded read's data window excludes the end time (a sample there is the end bound). A backend whose
// forward end is inclusive (the gateway's own) returns it as data; dropping it also means the data is drained.
if (endBound is not null)
{
var inside = samples.Where(x => HistoryBounds.InDataWindow(x, endBound, resolution)).ToList();
if (inside.Count != samples.Count)
{
samples = inside;
backendFull = false;
}
}
// Oversized tie cluster: a resume read whose FULL backend page does not advance past the resume // Oversized tie cluster: a resume read whose FULL backend page does not advance past the resume
// boundary timestamp `startUtc`. That happens when more samples share `startUtc` than the page // boundary timestamp `startUtc`. That happens when more samples share `startUtc` than the page
// cap — a tie cluster larger than NumValuesPerNode. The fixed-(start,end,cap) backend can only // cap — a tie cluster larger than NumValuesPerNode. The fixed-(start,end,cap) backend can only
@@ -6355,9 +6456,15 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
{ {
var clusterState = new HistoryContinuationState( var clusterState = new HistoryContinuationState(
tagname, resumeAt, endUtc, nextSkip, numValuesPerNode, SegmentWidthTicks: walkHint.WidthTicks, tagname, resumeAt, endUtc, nextSkip, numValuesPerNode, SegmentWidthTicks: walkHint.WidthTicks,
EmptyFromTicks: walkHint.EmptyFromTicks, EmptyToTicks: walkHint.EmptyToTicks); EmptyFromTicks: walkHint.EmptyFromTicks, EmptyToTicks: walkHint.EmptyToTicks,
PendingEndBound: HistoryBounds.Carry(endBound, slice));
clusterCp = _historyContinuationStore.Save(session, clusterState); clusterCp = _historyContinuationStore.Save(session, clusterState);
} }
else if (endBound is not null)
{
// #691: the data is drained — the end bound ends this page, or is the next one when this one is full.
clusterCp = await AppendEndBoundAsync(slice, startUtc).ConfigureAwait(false);
}
results[handle.Index] = new SdkHistoryReadResult results[handle.Index] = new SdkHistoryReadResult
{ {
@@ -6371,6 +6478,14 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// The "full page" test is against the RAW backend count (before trimming): the backend honoured // The "full page" test is against the RAW backend count (before trimming): the backend honoured
// the cap, so a full backend page ⇒ there may be more even if we trimmed some boundary ties. // the cap, so a full backend page ⇒ there may be more even if we trimmed some boundary ties.
// #691: the start bound (first page only) leads the data.
if (startBound is not null)
{
var led = new List<DataValueSnapshot>(samples.Count + 2) { startBound };
led.AddRange(samples);
samples = led;
}
var historyData = ToHistoryDataFromSamples(samples); var historyData = ToHistoryDataFromSamples(samples);
byte[]? outboundCp = null; byte[]? outboundCp = null;
@@ -6379,11 +6494,21 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
HistoryPaging.ComputeResumeCursor(samples, out var nextStart, out var skip); HistoryPaging.ComputeResumeCursor(samples, out var nextStart, out var skip);
var nextState = new HistoryContinuationState( var nextState = new HistoryContinuationState(
tagname, nextStart, endUtc, skip, numValuesPerNode, SegmentWidthTicks: walkHint.WidthTicks, tagname, nextStart, endUtc, skip, numValuesPerNode, SegmentWidthTicks: walkHint.WidthTicks,
EmptyFromTicks: walkHint.EmptyFromTicks, EmptyToTicks: walkHint.EmptyToTicks); EmptyFromTicks: walkHint.EmptyFromTicks, EmptyToTicks: walkHint.EmptyToTicks,
PendingEndBound: HistoryBounds.Carry(endBound, samples));
// Save may return null (no session on this request) ⇒ degrade to single-shot for this node. // Save may return null (no session on this request) ⇒ degrade to single-shot for this node.
// Built AFTER historyData so a failure projecting samples can never orphan a stored cursor. // Built AFTER historyData so a failure projecting samples can never orphan a stored cursor.
outboundCp = _historyContinuationStore.Save(session, nextState); outboundCp = _historyContinuationStore.Save(session, nextState);
} }
else if (endBound is not null)
{
// #691: the data is drained — the end bound ends this page, or is the next one when this one is full.
var page = new List<DataValueSnapshot>(samples.Count + 1);
page.AddRange(samples);
outboundCp = await AppendEndBoundAsync(page, startUtc).ConfigureAwait(false);
samples = page;
historyData = ToHistoryDataFromSamples(samples);
}
results[handle.Index] = new SdkHistoryReadResult results[handle.Index] = new SdkHistoryReadResult
{ {
@@ -0,0 +1,100 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
/// <summary>
/// Gitea #691: the server composes Part 11 bounding values (HistoryRead-Raw <c>ReturnBounds</c>) from exactly these raw
/// reads on a backend that reads newest-first natively (<c>HistoryBounds</c> in the OPC UA server) — so every backend must
/// answer them precisely: "at or before T" = a reverse read of <c>(0, T]</c> capped at one, "at or after T" = a forward read
/// of <c>[T, MaxValue)</c> capped at one, "strictly after T" = the same from <c>T + resolution</c>. A bound is a stored
/// sample of any quality, so a Bad row or an outage marker comes back as the bound.
/// </summary>
public abstract class BoundingValueConformance : ConformanceBase
{
private static readonly CancellationToken Ct = CancellationToken.None;
private static readonly DateTime Floor = new(0, DateTimeKind.Utc);
private static readonly DateTime Ceiling = new(DateTime.MaxValue.Ticks, DateTimeKind.Utc);
private async Task<DataValueSnapshot?> AtOrBeforeAsync(IHistorianDataSource src, string tag, DateTime t) =>
(await src.ReadRawAsync(tag, t, Floor, 1, Ct)).Samples.SingleOrDefault();
private async Task<DataValueSnapshot?> AtOrAfterAsync(IHistorianDataSource src, string tag, DateTime t) =>
(await src.ReadRawAsync(tag, t, Ceiling, 1, Ct)).Samples.SingleOrDefault();
[Fact]
public async Task The_sample_at_or_before_a_time_is_the_latest_one_on_or_before_it()
{
await ConformanceData.SeedAAsync(B);
var src = B.CreateDataSource();
(await AtOrBeforeAsync(src, "A", At(10)))!.Value.ShouldBe(20d); // exactly on it
(await AtOrBeforeAsync(src, "A", At(15)))!.Value.ShouldBe(20d); // between samples
(await AtOrBeforeAsync(src, "A", At(1000)))!.Value.ShouldBe(50d); // past the last
(await AtOrBeforeAsync(src, "A", At(-1))).ShouldBeNull(); // before the first: no bound
(await AtOrBeforeAsync(src, "A", At(15)))!.SourceTimestampUtc!.Value.Ticks.ShouldBe(B.Round(At(10)).Ticks);
}
[Fact]
public async Task The_sample_at_or_after_a_time_is_the_earliest_one_on_or_after_it()
{
await ConformanceData.SeedAAsync(B);
var src = B.CreateDataSource();
(await AtOrAfterAsync(src, "A", At(10)))!.Value.ShouldBe(20d); // exactly on it
(await AtOrAfterAsync(src, "A", At(15)))!.Value.ShouldBe(30d); // between samples
(await AtOrAfterAsync(src, "A", At(-100)))!.Value.ShouldBe(10d); // before the first
(await AtOrAfterAsync(src, "A", At(41))).ShouldBeNull(); // past the last: no bound
}
[Fact]
public async Task The_sample_strictly_after_a_time_starts_one_storage_resolution_later()
{
await ConformanceData.SeedAAsync(B);
var src = B.CreateDataSource();
(await AtOrAfterAsync(src, "A", At(10).AddTicks(B.ResolutionTicks)))!.Value.ShouldBe(30d);
(await AtOrAfterAsync(src, "A", At(40).AddTicks(B.ResolutionTicks))).ShouldBeNull();
}
[Fact]
public async Task A_time_between_storage_ticks_finds_the_sample_the_backend_stores_at_its_truncation()
{
// The server compares "on the bound" at the storage resolution, so the lookups must agree with the window reads:
// a time a fraction of a resolution step after a stored sample is that sample's instant on a coarse backend.
await ConformanceData.SeedAAsync(B);
var src = B.CreateDataSource();
var t = At(10).AddTicks(B.ResolutionTicks - 1); // == At(10) on a 1-tick backend
(await AtOrBeforeAsync(src, "A", t))!.Value.ShouldBe(20d);
var after = await AtOrAfterAsync(src, "A", t);
after!.SourceTimestampUtc!.Value.Ticks.ShouldBeGreaterThanOrEqualTo(B.Round(t).Ticks);
after.Value.ShouldBe(B.Round(t) == B.Round(At(10)) ? 20d : 30d);
}
[Fact]
public async Task A_bad_sample_or_an_outage_marker_is_a_bound_as_stored()
{
await B.SeedAsync("M",
(At(0), HistorianValue.FromDouble(10), Good),
(At(5), HistorianValue.Empty, BadNoCommunication),
(At(7), HistorianValue.FromDouble(99), Bad));
var src = B.CreateDataSource();
var before = await AtOrBeforeAsync(src, "M", At(6));
before!.Value.ShouldBeNull();
before.StatusCode.ShouldBe(0x80310000u);
var after = await AtOrAfterAsync(src, "M", At(6));
after!.Value.ShouldBe(99d);
after.StatusCode.ShouldBe(0x80000000u);
}
[Fact]
public async Task An_unknown_tag_has_no_bounds()
{
var src = B.CreateDataSource();
(await AtOrBeforeAsync(src, "nope", At(10))).ShouldBeNull();
(await AtOrAfterAsync(src, "nope", At(10))).ShouldBeNull();
}
}
@@ -67,3 +67,10 @@ public sealed class QuestDbEventReadConformance(QuestDbFixture fx) : EventReadCo
/// <inheritdoc/> /// <inheritdoc/>
protected override Task<IConformanceBackend> CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx); protected override Task<IConformanceBackend> CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx);
} }
[Collection(QuestDbCollection.Name)]
public sealed class QuestDbBoundingValueConformance(QuestDbFixture fx) : BoundingValueConformance
{
/// <inheritdoc/>
protected override Task<IConformanceBackend> CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx);
}
@@ -67,3 +67,10 @@ public sealed class SqlServerEventReadConformance(SqlHistorianFixture fx) : Even
/// <inheritdoc/> /// <inheritdoc/>
protected override Task<IConformanceBackend> CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx); protected override Task<IConformanceBackend> CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx);
} }
[Collection(SqlHistorianCollection.Name)]
public sealed class SqlServerBoundingValueConformance(SqlHistorianFixture fx) : BoundingValueConformance
{
/// <inheritdoc/>
protected override Task<IConformanceBackend> CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx);
}
@@ -97,6 +97,7 @@ public sealed class HistoryUpdateOverTheWireTests
var details = new ReadRawModifiedDetails var details = new ReadRawModifiedDetails
{ {
ReturnBounds = false,
StartTime = T1.AddHours(-1), EndTime = T1.AddHours(1), NumValuesPerNode = 100, IsReadModified = true, StartTime = T1.AddHours(-1), EndTime = T1.AddHours(1), NumValuesPerNode = 100, IsReadModified = true,
}; };
var response = await session.HistoryReadAsync( var response = await session.HistoryReadAsync(
@@ -384,6 +384,7 @@ public sealed class NodeManagerAclEnforcementTests : IDisposable
context, context,
new ReadRawModifiedDetails new ReadRawModifiedDetails
{ {
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1), StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow, EndTime = DateTime.UtcNow,
NumValuesPerNode = 10, NumValuesPerNode = 10,
@@ -0,0 +1,450 @@
using Microsoft.Extensions.Logging.Abstractions;
using Opc.Ua;
using Opc.Ua.Server;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using HistorianRead = ZB.MOM.WW.OtOpcUa.Core.Abstractions.HistoryReadResult;
using SdkHistoryReadResult = Opc.Ua.HistoryReadResult;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests;
/// <summary>Boots ONE real SDK server for the whole bounds suite (the cases swap the historian source per read).</summary>
public sealed class BoundsServerFixture : IAsyncLifetime
{
private readonly string _pkiRoot = Path.Combine(Path.GetTempPath(), $"otopcua-bounds-{Guid.NewGuid():N}");
private OpcUaApplicationHost? _host;
/// <summary>The node manager under test.</summary>
public OtOpcUaNodeManager Nm { get; private set; } = null!;
/// <summary>The historized variable every case reads.</summary>
public NodeId NodeId { get; private set; } = null!;
/// <summary>Serialises the cases (they share the node manager's source).</summary>
public SemaphoreSlim Gate { get; } = new(1, 1);
public async ValueTask InitializeAsync()
{
var started = await OpcUaHostStart.StartAsync(
port => new OpcUaApplicationHost(
new OpcUaApplicationHostOptions
{
ApplicationName = "OtOpcUa.BoundsTest",
ApplicationUri = $"urn:OtOpcUa.BoundsTest:{Guid.NewGuid():N}",
OpcUaPort = port,
PublicHostname = "localhost",
PkiStoreRoot = _pkiRoot,
},
NullLogger<OpcUaApplicationHost>.Instance),
() => new OtOpcUaSdkServer(), CancellationToken.None);
_host = started.Host;
Nm = started.Server.NodeManager!;
Nm.HistoryContinuationStore = new InMemoryHistoryContinuationStore();
Nm.EnsureVariable("eq-1/bounds", parentFolderNodeId: null, displayName: "Bounds", dataType: "Double",
writable: false, historianTagname: "WW.Bounds", realm: AddressSpaceRealm.Uns);
NodeId = Nm.TryGetVariable("eq-1/bounds")!.NodeId;
}
public async ValueTask DisposeAsync()
{
if (_host is not null) await _host.DisposeAsync();
if (Directory.Exists(_pkiRoot))
{
try { Directory.Delete(_pkiRoot, recursive: true); }
catch { /* best-effort */ }
}
}
}
/// <summary>
/// Gitea #691: HistoryRead-Raw <c>ReturnBounds</c> (OPC UA Part 11 §4.6 bounding values, §6.5.3.2). The oracle is Part 11
/// Table 1 itself — every row, over its data set (samples at 5:00, 5:02, 5:03, 5:05, 5:06) — run against three backend
/// shapes: one that reads newest-first natively (SqlServer-like), one that reads forward only (Gateway-like, walked back
/// by <see cref="ReverseRawReader"/>) and a microsecond one (QuestDB-like,
/// truncating its window bounds).
/// </summary>
public sealed class NodeManagerHistoryReadBoundsTests(BoundsServerFixture fx) : IClassFixture<BoundsServerFixture>
{
private static readonly DateTime Day = new(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
private static readonly DateTime Unspecified = DateTime.MinValue;
private static DateTime T(string hhmm)
{
var parts = hhmm.Split(':');
return Day.AddHours(int.Parse(parts[0])).AddMinutes(int.Parse(parts[1]));
}
/// <summary>Part 11 Table 1's data: value = minutes past 5:00.</summary>
private static DataValueSnapshot[] TableData() =>
new[] { "5:00", "5:02", "5:03", "5:05", "5:06" }
.Select(s => new DataValueSnapshot((double)(T(s) - T("5:00")).TotalMinutes, StatusCodes.Good, T(s), T(s)))
.ToArray();
public enum Shape { Reverse, ForwardOnly, Microsecond }
private static IHistorianDataSource Source(Shape shape, IReadOnlyList<DataValueSnapshot> series) => shape switch
{
Shape.Reverse => new SeriesSource(series, reverse: true, inclusiveForwardEnd: false, resolution: 1),
Shape.ForwardOnly => new SeriesSource(series, reverse: false, inclusiveForwardEnd: false, resolution: 1),
_ => new SeriesSource(series, reverse: true, inclusiveForwardEnd: false, resolution: 10),
};
/// <summary>
/// Part 11 Table 1, every row, verbatim. "FIRST"/"LAST" are <c>Bad_BoundNotFound</c> placeholders (null value) at the
/// StartTime/EndTime; a/b footnotes: one second after/before the previous returned value when that side is unspecified.
/// </summary>
public static TheoryData<string, string, uint, bool, string> Table1 => new()
{
{ "5:00", "5:05", 0, true, "5:00, 5:02, 5:03, 5:05" },
{ "5:00", "5:05", 0, false, "5:00, 5:02, 5:03" },
{ "5:01", "5:04", 0, true, "5:00, 5:02, 5:03, 5:05" },
{ "5:01", "5:04", 0, false, "5:02, 5:03" },
{ "5:05", "5:00", 0, true, "5:05, 5:03, 5:02, 5:00" },
{ "5:05", "5:00", 0, false, "5:05, 5:03, 5:02" },
{ "5:04", "5:01", 0, true, "5:05, 5:03, 5:02, 5:00" },
{ "5:04", "5:01", 0, false, "5:03, 5:02" },
{ "4:59", "5:05", 0, true, "FIRST, 5:00, 5:02, 5:03, 5:05" },
{ "4:59", "5:05", 0, false, "5:00, 5:02, 5:03" },
{ "5:01", "5:07", 0, true, "5:00, 5:02, 5:03, 5:05, 5:06, LAST" },
{ "5:01", "5:07", 0, false, "5:02, 5:03, 5:05, 5:06" },
{ "5:00", "5:05", 3, true, "5:00, 5:02, 5:03" },
{ "5:00", "5:05", 3, false, "5:00, 5:02, 5:03" },
{ "5:01", "5:04", 3, true, "5:00, 5:02, 5:03" },
{ "5:01", "5:04", 3, false, "5:02, 5:03" },
{ "5:05", "5:00", 3, true, "5:05, 5:03, 5:02" },
{ "5:05", "5:00", 3, false, "5:05, 5:03, 5:02" },
{ "5:04", "5:01", 3, true, "5:05, 5:03, 5:02" },
{ "5:04", "5:01", 3, false, "5:03, 5:02" },
{ "4:59", "5:05", 3, true, "FIRST, 5:00, 5:02" },
{ "4:59", "5:05", 3, false, "5:00, 5:02, 5:03" },
{ "5:01", "5:07", 3, true, "5:00, 5:02, 5:03" },
{ "5:01", "5:07", 3, false, "5:02, 5:03, 5:05" },
{ "5:00", "", 3, true, "5:00, 5:02, 5:03" },
{ "5:00", "", 3, false, "5:00, 5:02, 5:03" },
{ "5:00", "", 6, true, "5:00, 5:02, 5:03, 5:05, 5:06, LAST+" },
{ "5:00", "", 6, false, "5:00, 5:02, 5:03, 5:05, 5:06" },
{ "5:07", "", 6, true, "5:06, LAST+" },
{ "5:07", "", 6, false, "" },
{ "", "5:06", 3, true, "5:06, 5:05, 5:03" },
{ "", "5:06", 3, false, "5:06, 5:05, 5:03" },
{ "", "5:06", 6, true, "5:06, 5:05, 5:03, 5:02, 5:00, FIRST-" },
{ "", "5:06", 6, false, "5:06, 5:05, 5:03, 5:02, 5:00" },
{ "", "4:48", 6, true, "5:00, FIRST-" },
{ "", "4:48", 6, false, "" },
{ "4:48", "4:48", 0, true, "FIRST, 5:00" },
{ "4:48", "4:48", 0, false, "" },
{ "4:48", "4:48", 1, true, "FIRST" },
{ "4:48", "4:48", 1, false, "" },
{ "4:48", "4:48", 2, true, "FIRST, 5:00" },
{ "5:00", "5:00", 0, true, "5:00, 5:02" },
{ "5:00", "5:00", 0, false, "5:00" },
{ "5:00", "5:00", 1, true, "5:00" },
{ "5:00", "5:00", 1, false, "5:00" },
{ "5:01", "5:01", 0, true, "5:00, 5:02" },
{ "5:01", "5:01", 0, false, "" },
{ "5:01", "5:01", 1, true, "5:00" },
{ "5:01", "5:01", 1, false, "" },
};
[Theory]
[MemberData(nameof(Table1))]
public async Task Part11_table1_first_page_on_a_reverse_capable_backend(string start, string end, uint max, bool bounds, string expected) =>
await AssertFirstPage(Shape.Reverse, start, end, max, bounds, expected);
[Theory]
[MemberData(nameof(Table1))]
public async Task Part11_table1_first_page_on_a_forward_only_backend(string start, string end, uint max, bool bounds, string expected) =>
await AssertFirstPage(Shape.ForwardOnly, start, end, max, bounds, expected);
[Theory]
[MemberData(nameof(Table1))]
public async Task Part11_table1_first_page_on_a_microsecond_backend(string start, string end, uint max, bool bounds, string expected) =>
await AssertFirstPage(Shape.Microsecond, start, end, max, bounds, expected);
private async Task AssertFirstPage(Shape shape, string start, string end, uint max, bool bounds, string expected)
{
var s = start.Length == 0 ? Unspecified : T(start);
var e = end.Length == 0 ? Unspecified : T(end);
var source = Source(shape, TableData());
var (values, error, _) = await ReadAsync(source, s, e, max, bounds, cp: null);
error.StatusCode.Code.ShouldBe(StatusCodes.Good);
Render(values, s, e).ShouldBe(expected);
(source as SeriesSource)?.ReverseWindowsOnForwardOnly.ShouldBe(0);
}
/// <summary>
/// Part 11: bounds count toward NumValuesPerNode, the start bound comes with the first page and the end bound with the
/// last — so paging a bounded two-sided read at any page size concatenates to the unpaged (N = 0) answer.
/// </summary>
[Theory]
[InlineData(Shape.Reverse)]
[InlineData(Shape.ForwardOnly)]
[InlineData(Shape.Microsecond)]
public async Task Paging_a_bounded_read_concatenates_to_the_unpaged_answer(Shape shape)
{
var windows = new[]
{
("5:00", "5:05"), ("5:01", "5:04"), ("5:05", "5:00"), ("5:04", "5:01"), ("4:59", "5:05"), ("5:01", "5:07"),
("4:48", "4:48"), ("5:00", "5:00"), ("5:01", "5:01"), ("5:07", "4:00"), ("5:06", "5:06"),
};
foreach (var (start, end) in windows)
{
var source = Source(shape, TableData());
var all = await ReadAsync(source, T(start), T(end), 0, true, cp: null);
all.Cp.ShouldBeNull();
var expected = Render(all.Values, T(start), T(end));
for (uint n = 1; n <= 6; n++)
{
var paged = await DrainAsync(source, T(start), T(end), n, bounds: true);
Render(paged, T(start), T(end)).ShouldBe(expected, $"{start}->{end} paged by {n}");
}
}
}
/// <summary>One-sided reads page to the open placeholder exactly once (footnotes a/b), never past it.</summary>
[Theory]
[InlineData(Shape.Reverse)]
[InlineData(Shape.ForwardOnly)]
[InlineData(Shape.Microsecond)]
public async Task Paging_a_one_sided_bounded_read_ends_with_the_open_placeholder(Shape shape)
{
for (uint n = 1; n <= 7; n++)
{
var fwd = await DrainAsync(Source(shape, TableData()), T("5:01"), Unspecified, n, bounds: true);
Render(fwd, T("5:01"), Unspecified).ShouldBe("5:00, 5:02, 5:03, 5:05, 5:06, LAST+", $"forward by {n}");
var rev = await DrainAsync(Source(shape, TableData()), Unspecified, T("5:04"), n, bounds: true);
Render(rev, Unspecified, T("5:04")).ShouldBe("5:05, 5:03, 5:02, 5:00, FIRST-", $"reverse by {n}");
}
}
/// <summary>A bound is a stored sample of any quality: a Bad sample before StartTime is the start bound, as stored.</summary>
[Theory]
[InlineData(Shape.Reverse)]
[InlineData(Shape.ForwardOnly)]
[InlineData(Shape.Microsecond)]
public async Task A_bad_sample_is_returned_as_the_bound_as_stored(Shape shape)
{
var series = TableData();
series[0] = new DataValueSnapshot(null, StatusCodes.BadNoCommunication, T("5:00"), T("5:00"));
var (values, _, _) = await ReadAsync(Source(shape, series), T("5:01"), T("5:04"), 0, true, cp: null);
values[0].StatusCode.Code.ShouldBe(StatusCodes.BadNoCommunication);
values[0].SourceTimestamp.ShouldBe(T("5:00"));
values.Count.ShouldBe(4);
}
/// <summary>
/// Ties on a bound: the start bound before a forward window is the LAST of a tie cluster (the sample immediately
/// before StartTime in the read's order); the end bound after it is the FIRST of one; and ties ON StartTime are all
/// data (the bound is the first of them, so nothing extra is added).
/// </summary>
[Theory]
[InlineData(Shape.Reverse)]
[InlineData(Shape.ForwardOnly)]
[InlineData(Shape.Microsecond)]
public async Task Tie_clusters_on_either_side_yield_one_bound_each(Shape shape)
{
var series = new List<DataValueSnapshot>();
void Add(string t, double v) => series.Add(new DataValueSnapshot(v, StatusCodes.Good, T(t), T(t)));
Add("5:00", 1); Add("5:00", 2); Add("5:00", 3);
Add("5:02", 4);
Add("5:05", 5); Add("5:05", 6);
var (fwd, _, _) = await ReadAsync(Source(shape, series), T("5:01"), T("5:04"), 0, true, cp: null);
fwd.Select(v => (double)v.Value).ShouldBe([3d, 4d, 5d]);
var (rev, _, _) = await ReadAsync(Source(shape, series), T("5:04"), T("5:01"), 0, true, cp: null);
rev.Select(v => (double)v.Value).ShouldBe([5d, 4d, 3d]);
var (onStart, _, _) = await ReadAsync(Source(shape, series), T("5:00"), T("5:03"), 0, true, cp: null);
onStart.Select(v => (double)v.Value).ShouldBe([1d, 2d, 3d, 4d, 5d]);
}
/// <summary>
/// QuestDB truncates the window bounds it is sent to the microsecond: a sample at ⌊StartTime⌋ is inside a window that
/// starts a few ticks later, so it is data — never returned a second time as the start bound.
/// </summary>
[Fact]
public async Task A_microsecond_backend_does_not_return_a_sample_at_the_truncated_start_twice()
{
var series = TableData();
var (values, _, _) = await ReadAsync(Source(Shape.Microsecond, series), T("5:00").AddTicks(7), T("5:04"), 0, true, cp: null);
values.Select(v => (double)v.Value).ShouldBe([0d, 2d, 3d, 5d]);
}
/// <summary>
/// A backend whose forward end is inclusive (the gateway's own <c>ReadRaw</c>), or a resumed read that steps onto
/// EndTime, returns the sample AT EndTime as data — it must come back once, as the end bound.
/// </summary>
[Theory]
[InlineData(0u)]
[InlineData(1u)]
[InlineData(2u)]
[InlineData(3u)]
public async Task An_inclusive_forward_end_does_not_return_the_end_bound_twice(uint max)
{
var source = new SeriesSource(TableData(), reverse: false, inclusiveForwardEnd: true, resolution: 1);
var values = await DrainAsync(source, T("5:00"), T("5:05"), max, bounds: true);
Render(values, T("5:00"), T("5:05")).ShouldBe("5:00, 5:02, 5:03, 5:05");
}
/// <summary>Part 11 §6.5.3.3: a modified read with ReturnBounds is <c>Bad_InvalidArgument</c>.</summary>
[Fact]
public async Task A_modified_read_with_return_bounds_is_an_invalid_argument()
{
await fx.Gate.WaitAsync(TestContext.Current.CancellationToken);
try
{
fx.Nm.HistorianDataSource = Source(Shape.Reverse, TableData());
var (_, error, cp) = Read(T("5:00"), T("5:05"), 0, bounds: true, cp: null, modified: true);
error.StatusCode.Code.ShouldBe(StatusCodes.BadInvalidArgument);
cp.ShouldBeNull();
}
finally { fx.Gate.Release(); }
}
// --- helpers --------------------------------------------------------------------------------
private static string Render(IReadOnlyList<DataValue> values, DateTime start, DateTime end)
{
var parts = new List<string>();
for (var i = 0; i < values.Count; i++)
{
var v = values[i];
if (v.StatusCode.Code != StatusCodes.BadBoundNotFound)
{
v.Value.ShouldNotBeNull();
parts.Add($"{v.SourceTimestamp.Hour}:{v.SourceTimestamp.Minute:00}");
continue;
}
v.Value.ShouldBeNull("a missing bound carries a null value");
var isFirst = i == 0 && start != Unspecified && v.SourceTimestamp == start;
var isLast = end != Unspecified && v.SourceTimestamp == end;
if (isFirst) { parts.Add("FIRST"); continue; }
if (isLast) { parts.Add("LAST"); continue; }
// Open side (footnotes a/b): one second from the previous returned value.
i.ShouldBeGreaterThan(0);
var prev = values[i - 1].SourceTimestamp;
if (v.SourceTimestamp == prev.AddSeconds(1)) parts.Add("LAST+");
else if (v.SourceTimestamp == prev.AddSeconds(-1)) parts.Add("FIRST-");
else parts.Add($"?{v.SourceTimestamp:O}");
}
return string.Join(", ", parts);
}
private async Task<List<DataValue>> DrainAsync(IHistorianDataSource source, DateTime start, DateTime end, uint max, bool bounds)
{
var all = new List<DataValue>();
byte[]? cp = null;
var pages = 0;
do
{
var (values, error, next) = await ReadAsync(source, start, end, max, bounds, cp);
error.StatusCode.Code.ShouldBe(StatusCodes.Good);
if (max > 0) values.Count.ShouldBeLessThanOrEqualTo((int)max, "bounds count toward NumValuesPerNode");
all.AddRange(values);
cp = next;
(++pages).ShouldBeLessThan(50, "paging must terminate");
}
while (cp is not null);
return all;
}
private async Task<(IReadOnlyList<DataValue> Values, ServiceResult Error, byte[]? Cp)> ReadAsync(
IHistorianDataSource source, DateTime start, DateTime end, uint max, bool bounds, byte[]? cp)
{
await fx.Gate.WaitAsync(TestContext.Current.CancellationToken);
try
{
fx.Nm.HistorianDataSource = source;
return Read(start, end, max, bounds, cp, modified: false);
}
finally { fx.Gate.Release(); }
}
private (IReadOnlyList<DataValue> Values, ServiceResult Error, byte[]? Cp) Read(
DateTime start, DateTime end, uint max, bool bounds, byte[]? cp, bool modified)
{
var context = new OperationContext(new RequestHeader(), secureChannelContext: null, RequestType.HistoryRead, identity: null);
var details = new ReadRawModifiedDetails
{
StartTime = start, EndTime = end, NumValuesPerNode = max, ReturnBounds = bounds, IsReadModified = modified,
};
var nodesToRead = new List<HistoryReadValueId> { new() { NodeId = fx.NodeId, ContinuationPoint = cp } };
var results = new List<SdkHistoryReadResult> { null! };
var errors = new List<ServiceResult> { null! };
fx.Nm.HistoryRead(context, details, TimestampsToReturn.Both, releaseContinuationPoints: false, nodesToRead, results, errors);
var values = new List<DataValue>();
if (results[0]?.HistoryData is { } ho && !ExtensionObject.IsNull(ho))
values.AddRange(((HistoryData)ExtensionObject.ToEncodeable(ho)).DataValues);
return (values, errors[0], results[0]?.ContinuationPoint);
}
/// <summary>
/// A series-backed historian. <paramref name="reverse"/>: serves <c>start &gt; end</c> as <c>(end, start]</c>
/// newest-first (SqlServer/QuestDB); otherwise rejects it (the Gateway) and counts that. The forward read is
/// <c>[start, end)</c>, or <c>[start, end]</c> when <paramref name="inclusiveForwardEnd"/> (the gateway's own end);
/// a point read is the instant. <paramref name="resolution"/> &gt; 1 truncates every window bound to it first (QuestDB
/// µs); the series must already be stored at that resolution.
/// </summary>
private sealed class SeriesSource(IReadOnlyList<DataValueSnapshot> series, bool reverse, bool inclusiveForwardEnd, long resolution)
: IHistorianDataSource
{
public int ReverseWindowsOnForwardOnly;
public bool SupportsReverseRaw => reverse;
public long TimestampResolutionTicks => resolution;
private DateTime Trunc(DateTime t) => new(t.Ticks - t.Ticks % resolution, DateTimeKind.Utc);
public Task<HistorianRead> ReadRawAsync(
string fullReference, DateTime startUtc, DateTime endUtc, uint maxValuesPerNode, CancellationToken cancellationToken)
{
static DateTime Ts(DataValueSnapshot s) => s.SourceTimestampUtc ?? DateTime.MinValue;
startUtc = Trunc(startUtc);
endUtc = Trunc(endUtc);
List<DataValueSnapshot> window;
if (startUtc > endUtc)
{
if (!reverse)
{
Interlocked.Increment(ref ReverseWindowsOnForwardOnly);
throw new InvalidOperationException("start_time must be less than or equal to end_time. (InvalidArgument)");
}
window = series.Where(s => Ts(s) > endUtc && Ts(s) <= startUtc).Reverse().ToList();
}
else if (startUtc == endUtc) window = series.Where(s => Ts(s) == startUtc).ToList();
else window = series.Where(s => Ts(s) >= startUtc && (inclusiveForwardEnd ? Ts(s) <= endUtc : Ts(s) < endUtc)).ToList();
var page = maxValuesPerNode == 0 ? window : window.Take((int)maxValuesPerNode).ToList();
return Task.FromResult(new HistorianRead(page, null));
}
public Task<HistorianRead> ReadProcessedAsync(
string fullReference, DateTime startUtc, DateTime endUtc, TimeSpan interval,
HistoryAggregateType aggregate, CancellationToken cancellationToken) =>
Task.FromResult(new HistorianRead(Array.Empty<DataValueSnapshot>(), null));
public Task<HistorianRead> ReadAtTimeAsync(
string fullReference, IReadOnlyList<DateTime> timestampsUtc, CancellationToken cancellationToken) =>
Task.FromResult(new HistorianRead(Array.Empty<DataValueSnapshot>(), null));
public Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, CancellationToken cancellationToken) =>
Task.FromResult(new HistoricalEventsResult(Array.Empty<HistoricalEvent>(), null));
public HistorianHealthSnapshot GetHealthSnapshot() => NullHistorianDataSource.Instance.GetHealthSnapshot();
public void Dispose() { }
}
}
@@ -45,6 +45,7 @@ public sealed class NodeManagerHistoryReadConcurrencyTests : IDisposable
var details = new ReadRawModifiedDetails var details = new ReadRawModifiedDetails
{ {
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1), StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow, EndTime = DateTime.UtcNow,
NumValuesPerNode = 10, NumValuesPerNode = 10,
@@ -78,6 +79,7 @@ public sealed class NodeManagerHistoryReadConcurrencyTests : IDisposable
var nodeIds = MaterializeHistorizedNodes(nm, count: 12); var nodeIds = MaterializeHistorizedNodes(nm, count: 12);
var details = new ReadRawModifiedDetails var details = new ReadRawModifiedDetails
{ {
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1), StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow, EndTime = DateTime.UtcNow,
NumValuesPerNode = 10, NumValuesPerNode = 10,
@@ -115,6 +117,7 @@ public sealed class NodeManagerHistoryReadConcurrencyTests : IDisposable
var nodeIds = MaterializeHistorizedNodes(nm, count: 8); var nodeIds = MaterializeHistorizedNodes(nm, count: 8);
var details = new ReadRawModifiedDetails var details = new ReadRawModifiedDetails
{ {
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1), StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow, EndTime = DateTime.UtcNow,
NumValuesPerNode = 10, NumValuesPerNode = 10,
@@ -157,6 +160,7 @@ public sealed class NodeManagerHistoryReadConcurrencyTests : IDisposable
var nodeIds = MaterializeHistorizedNodes(nm, count: 4); var nodeIds = MaterializeHistorizedNodes(nm, count: 4);
var details = new ReadRawModifiedDetails var details = new ReadRawModifiedDetails
{ {
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1), StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow, EndTime = DateTime.UtcNow,
NumValuesPerNode = 10, NumValuesPerNode = 10,
@@ -190,6 +194,7 @@ public sealed class NodeManagerHistoryReadConcurrencyTests : IDisposable
var ids = MaterializeHistorizedNodes(nm, count: 2); var ids = MaterializeHistorizedNodes(nm, count: 2);
var details = new ReadRawModifiedDetails var details = new ReadRawModifiedDetails
{ {
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1), StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow, EndTime = DateTime.UtcNow,
NumValuesPerNode = 10, NumValuesPerNode = 10,
@@ -160,7 +160,7 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable
var ctx = new OperationContext(new RequestHeader(), secureChannelContext: null, RequestType.HistoryRead, identity: null); var ctx = new OperationContext(new RequestHeader(), secureChannelContext: null, RequestType.HistoryRead, identity: null);
var res = new List<SdkHistoryReadResult> { null! }; var res = new List<SdkHistoryReadResult> { null! };
var errs = new List<ServiceResult> { null! }; var errs = new List<ServiceResult> { null! };
nm.HistoryRead(ctx, new ReadRawModifiedDetails { StartTime = Epoch, EndTime = Epoch.AddHours(1), NumValuesPerNode = 2, IsReadModified = false }, nm.HistoryRead(ctx, new ReadRawModifiedDetails { ReturnBounds = false, StartTime = Epoch, EndTime = Epoch.AddHours(1), NumValuesPerNode = 2, IsReadModified = false },
TimestampsToReturn.Both, false, TimestampsToReturn.Both, false,
new List<HistoryReadValueId> { new() { NodeId = n, ContinuationPoint = cp } }, res, errs); new List<HistoryReadValueId> { new() { NodeId = n, ContinuationPoint = cp } }, res, errs);
err = errs[0]; err = errs[0];
@@ -271,7 +271,7 @@ public sealed class NodeManagerHistoryReadModifiedTests : IDisposable
var results = new List<SdkHistoryReadResult> { null! }; var results = new List<SdkHistoryReadResult> { null! };
var errors = new List<ServiceResult> { null! }; var errors = new List<ServiceResult> { null! };
nm.HistoryRead(context, nm.HistoryRead(context,
new ReadRawModifiedDetails { StartTime = start, EndTime = end, NumValuesPerNode = max, IsReadModified = modified }, new ReadRawModifiedDetails { ReturnBounds = false, StartTime = start, EndTime = end, NumValuesPerNode = max, IsReadModified = modified },
TimestampsToReturn.Both, false, TimestampsToReturn.Both, false,
new List<HistoryReadValueId> { new() { NodeId = node, ContinuationPoint = cp } }, results, errors); new List<HistoryReadValueId> { new() { NodeId = node, ContinuationPoint = cp } }, results, errors);
@@ -967,6 +967,7 @@ public sealed class NodeManagerHistoryReadPagingTests : IDisposable
var details = new ReadRawModifiedDetails var details = new ReadRawModifiedDetails
{ {
ReturnBounds = false,
StartTime = start, StartTime = start,
EndTime = end, EndTime = end,
NumValuesPerNode = max, NumValuesPerNode = max,
@@ -1000,6 +1001,7 @@ public sealed class NodeManagerHistoryReadPagingTests : IDisposable
new RequestHeader(), secureChannelContext: null, RequestType.HistoryRead, identity: null); new RequestHeader(), secureChannelContext: null, RequestType.HistoryRead, identity: null);
var details = new ReadRawModifiedDetails var details = new ReadRawModifiedDetails
{ {
ReturnBounds = false,
StartTime = Epoch, EndTime = Epoch.AddHours(1), NumValuesPerNode = 100, IsReadModified = false, StartTime = Epoch, EndTime = Epoch.AddHours(1), NumValuesPerNode = 100, IsReadModified = false,
}; };
var nodesToRead = new List<HistoryReadValueId> { new() { NodeId = nodeId, ContinuationPoint = cp } }; var nodesToRead = new List<HistoryReadValueId> { new() { NodeId = nodeId, ContinuationPoint = cp } };
@@ -52,6 +52,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
var end = DateTime.UtcNow; var end = DateTime.UtcNow;
var details = new ReadRawModifiedDetails var details = new ReadRawModifiedDetails
{ {
ReturnBounds = false,
StartTime = start, StartTime = start,
EndTime = end, EndTime = end,
NumValuesPerNode = 100, NumValuesPerNode = 100,
@@ -97,6 +98,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
var details = new ReadRawModifiedDetails var details = new ReadRawModifiedDetails
{ {
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1), StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow, EndTime = DateTime.UtcNow,
NumValuesPerNode = 10, NumValuesPerNode = 10,
@@ -129,6 +131,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
var details = new ReadRawModifiedDetails var details = new ReadRawModifiedDetails
{ {
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1), StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow, EndTime = DateTime.UtcNow,
NumValuesPerNode = 10, NumValuesPerNode = 10,
@@ -162,6 +165,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
var details = new ReadRawModifiedDetails var details = new ReadRawModifiedDetails
{ {
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1), StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow, EndTime = DateTime.UtcNow,
NumValuesPerNode = 10, NumValuesPerNode = 10,
@@ -195,6 +199,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
var details = new ReadRawModifiedDetails var details = new ReadRawModifiedDetails
{ {
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1), StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow, EndTime = DateTime.UtcNow,
NumValuesPerNode = 10, NumValuesPerNode = 10,
@@ -613,6 +618,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
var details = new ReadRawModifiedDetails var details = new ReadRawModifiedDetails
{ {
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1), StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow, EndTime = DateTime.UtcNow,
NumValuesPerNode = 10, NumValuesPerNode = 10,
@@ -658,6 +664,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
var details = new ReadRawModifiedDetails var details = new ReadRawModifiedDetails
{ {
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1), StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow, EndTime = DateTime.UtcNow,
NumValuesPerNode = 10, NumValuesPerNode = 10,