fix(historian): #693 review — bisecting walk-back, small jump reads, read cap, DateTime/NaN hints

H1: the walk-back reset its width after a full segment and discarded a full
whole-remainder read, so a dense tag that stopped reporting weeks ago gave up
after 65 reads (BadTooManyOperations). It now gallops (x4, whole-remainder read
after two empties) and then BISECTS once any stored row is known below — a full
segment's (incl. a full whole-remainder read's) last row, or an earlier window
read's last row: (mid, hi] empty => hi = mid; full => known = its last row;
complete with rows => done; one density-aimed segment after a full read until an
aimed segment comes back empty; below 2 ms the whole stretch from the known row
(never a 100 ns step). Stopped 1 h..365 d with 5 y history: 6-18 reads at
1/min, 21-25 at 1 Hz, 25-31 at 10 Hz.
M1: a jump read asks for 64 rows (1,000 times over 10 y: ~66k rows, was ~1M).
L1: per-request read cap 64 + 4 x distinct times with a clear message; a strict
ceiling below 4 rows is refused naming the ceiling.
L2: DateTime hints are stepped. L3: under an integer hint a NaN/inf or
out-of-Int64-range value is unusable, never saturated Good.
L5: conformance Int64 AtTime asserts the type. L4: docs name the backend
differences. Nit: the window read end is clamped to now + 1 day.

Reviewer harnesses (Scratch693Fuzz/Cost, not committed): 0 differential diffs
over 2,400 cases.
This commit is contained in:
Joseph Doherty
2026-10-13 01:09:54 -04:00
parent 0d249f3ee9
commit 68a1f31397
4 changed files with 379 additions and 96 deletions
+23 -9
View File
@@ -780,14 +780,23 @@ data.
**probe** (18 rows) for the sample after the last. Measured on the fake: ten times one second apart on a 1 Hz tag cost **1 read** **probe** (18 rows) for the sample after the last. Measured on the fake: ten times one second apart on a 1 Hz tag cost **1 read**
(all exact hits) or **3** (interpolated); ten times an hour apart on a 1 Hz tag **12** (≈ 9.5k rows, never the 32k between); one (all exact hits) or **3** (interpolated); ten times an hour apart on a 1 Hz tag **12** (≈ 9.5k rows, never the 32k between); one
time 400 days after the only samples **5**; a tag with no history **4**. A window read that fills its budget resolves the times before time 400 days after the only samples **5**; a tag with no history **4**. A window read that fills its budget resolves the times before
its last row; the next time either continues from there (if the rows in between should fit one read) or jumps to it with a read its last row; the next time either continues from there (if the rows in between should fit one read) or **jumps** to it with a
that starts a density-sized margin earlier (so its `prev` is normally in the same read). small read (64 rows) that starts a density-sized margin earlier (so its `prev` is normally in the same read) — 1,000 times
- **Walk-back** (the gateway reads forward only): segments stepping back from the time, the first sized from the window read's density spread over ten years of a 1 Hz tag cost ≈ 1,000 reads of ≤ 64 rows (≈ 66k rows; it used to be 1,024-row reads).
(else 1 minute), growing ×4 when empty; after two empty segments the whole remainder down to the epoch is asked once (a complete - **Walk-back** (the gateway reads forward only), in two phases. **Gallop:** segments stepping back from the time, the first sized
answer ends the walk — sparse history and no history are found in a handful of reads). A segment that fills its budget is narrowed from the window read's density (else 1 minute), growing ×4 when empty; after two empty segments the whole remainder down to the
towards the time to about a quarter of a read at the density it showed — never below a row it saw — so dense history is bisected, epoch is asked once — a complete answer ends the walk, a **full** one seeds the bisection with its last row. **Bisect** (once a
not paged (a 1 kHz tag behind a 10-minute gap: 7 reads, < 10k rows). At most 64 reads; more than 1,024 stored samples within one stored row is known below — a full segment's, or an earlier window read's last row): read `(mid, hi]`; empty ⇒ `hi = mid`; full ⇒
millisecond fail the node `BadTooManyOperations` (not counted as a backend failure). the known row moves up to its last row; complete with rows ⇒ done. Right after a full read one segment is aimed at about a quarter
of a read at the density it showed (until an aimed segment comes back empty — a gap — then plain halving); below 2 ms the segment
is the whole stretch from the known row (never a 100 ns step). Measured (review harness, 5 years of history, time = now − 1 min):
a tag stopped 1 h / 1 d / 7 d / 30 d / 365 d ago costs 6–18 reads at 1/min, 21–25 at 1 Hz and 25–31 at 10 Hz; a 1 kHz tag behind
a 10-minute gap 6 reads (< 10k rows). (Before the review fix a full segment reset the width and a full whole-remainder read was
discarded, so a dense tag that stopped weeks ago gave up after 65 reads.) More than 1,024 stored samples within one millisecond
fail the node `BadTooManyOperations` (not counted as a backend failure).
- **Bounded.** One request takes at most **64 + 4 × distinct times** raw reads; past it the node fails `BadTooManyOperations`
("needed more than N raw reads", naming a small strict-gateway ceiling when one bounded the reads). A strict gateway whose ceiling is
below 4 rows is refused up front with a message naming the ceiling; a small one (e.g. 10) is served by paging within the cap.
- **Wire shape.** Every request bound is a whole millisecond and a row counts only **strictly inside** its request, so AVEVA's synthetic - **Wire shape.** Every request bound is a whole millisecond and a row counts only **strictly inside** its request, so AVEVA's synthetic
boundary row at the query start (#687) — and any row at the query end — is never taken for a stored sample. A Bad row is value-less boundary row at the query start (#687) — and any row at the query end — is never taken for a stored sample. A Bad row is value-less
(#690). Nothing is asked past now + 1 day (the gateway's AVEVA may refuse a far-future end; a time beyond it is stepped from the last (#690). Nothing is asked past now + 1 day (the gateway's AVEVA may refuse a far-future end; a time beyond it is stepped from the last
@@ -800,7 +809,12 @@ data.
**before** the shared evaluator runs, so its own type rules apply exactly as on the SQL backends: a **Boolean** node **before** the shared evaluator runs, so its own type rules apply exactly as on the SQL backends: a **Boolean** node
(value ≠ 0) is **stepped** — never 0.5; an **integer** node is interpolated, rounded half away from zero and returned as the (value ≠ 0) is **stepped** — never 0.5; an **integer** node is interpolated, rounded half away from zero and returned as the
node's own integer type (`Int32` ⇒ `int`); Float / Double interpolate as Double (as the SQL backends return them). With no node's own integer type (`Int32` ⇒ `int`); Float / Double interpolate as Double (as the SQL backends return them). With no
hint (an unknown node) every numeric tag interpolates as Double. hint (an unknown node) every numeric tag interpolates as Double. A **DateTime** node is stepped. Under an integer hint a NaN, ±∞ or
out-of-Int64-range stored value is no integer: it is treated as unusable (as `BadNoData`), so a neighbour like that is never
interpolated or saturated into a Good value.
- **What differs between backends (by design, not changed by #693).** Only the **Gateway** converts an AtTime result to the node's
integer type (`Int32` ⇒ `int`): the SQL backends store and return integer nodes as `Int64` (and Boolean as Boolean). Gateway
**Raw** reads still return a Boolean node's samples as the stored Double 0/1 (the hint is passed only on AtTime).
- ⚠️ **Shared-evaluator fix (all backends that use it):** `AtTimeEvaluator` returned every interpolated `Int64` boxed as a - ⚠️ **Shared-evaluator fix (all backends that use it):** `AtTimeEvaluator` returned every interpolated `Int64` boxed as a
**Double** (a C# `cond ? (long)x : double` conditional widens the long); its "keeping the Int64 type" tests passed because **Double** (a C# `cond ? (long)x : double` conditional widens the long); its "keeping the Int64 type" tests passed because
Shouldly's `ShouldBe` compares numbers across types. SqlServer / QuestDB interpolated `Int64` AtTime values are now `Int64`, Shouldly's `ShouldBe` compares numbers across types. SqlServer / QuestDB interpolated `Int64` AtTime values are now `Int64`,
@@ -30,12 +30,22 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway;
/// apart on a 1 Hz tag cost 1 read (all exact hits) or 3 (interpolated); ten times an hour apart about one read each. /// apart on a 1 Hz tag cost 1 read (all exact hits) or 3 (interpolated); ten times an hour apart about one read each.
/// </para> /// </para>
/// <para> /// <para>
/// <b>Walk-back</b> (the gateway reads forward only): segments <c>(lo, hi]</c> stepping back from <c>hi</c>, each one /// <b>Walk-back</b> (the gateway reads forward only), in two phases. <i>Gallop:</i> segments <c>(lo, hi]</c> stepping
/// forward read; an empty segment grows the next ×4 (from the density the window read showed, else /// back from <c>hi</c>, each one forward read, the first sized from the density the window read showed (else
/// <see cref="InitialWalkWidth"/>), and after two empty segments the whole remainder down to the epoch is asked once /// <see cref="InitialWalkWidth"/>), an empty one growing the next ×4; after two empty segments the whole remainder down
/// (a complete answer ends the walk). A segment that fills its budget is narrowed towards <c>hi</c> to a width sized /// to the epoch is asked once — a complete answer ends the walk, a FULL one seeds the bisection with its last row.
/// from the density it showed, never below a row it saw (rows exist there) — so dense history is bisected, not paged. /// <i>Bisect</i> (once a stored row is known below <c>hi</c> — a full segment's last row, or the last row of an earlier
/// At most <see cref="MaxWalkReads"/> reads. /// window read): read <c>(mid, hi]</c>; empty ⇒ <c>hi = mid</c>; full ⇒ the known row moves up to its last row;
/// complete with rows ⇒ its last row is the answer. Right after a full read the next segment is aimed at about a
/// quarter of a read at the density it showed (one try — an empty aimed segment falls back to halving), and below
/// 2 ms the segment is the whole stretch from the known row (never a 100 ns step).
/// </para>
/// <para>
/// <b>Bounded.</b> One request may take at most <c>64 + 4 × distinct times</c> raw reads (<see cref="BaseReads"/>,
/// <see cref="ReadsPerTime"/>); past it — or with a strict-gateway ceiling below <see cref="MinBudget"/> rows — the
/// node fails <c>BadTooManyOperations</c> with a message naming the limit. A jump read (to a time far past the previous
/// read) asks for only <see cref="JumpRows"/> rows, so a thousand times spread over ten years cost about a thousand
/// small reads, not a thousand 1,024-row reads.
/// </para> /// </para>
/// <para> /// <para>
/// <b>Wire shape.</b> Every request bound is a whole millisecond (AVEVA stores milliseconds and truncates request /// <b>Wire shape.</b> Every request bound is a whole millisecond (AVEVA stores milliseconds and truncates request
@@ -57,23 +67,34 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway;
/// </remarks> /// </remarks>
internal static class GatewayAtTimeReader internal static class GatewayAtTimeReader
{ {
/// <summary>The row budget of a window read and a walk-back segment.</summary> /// <summary>The row budget of a window read (the first, or one continuing from a full one) and a walk-back segment.</summary>
internal const int ReadRows = 1024; internal const int ReadRows = 1024;
/// <summary>The row budget of a jump read (a window read restarting at a time far past the previous read's last row).</summary>
internal const int JumpRows = 64;
/// <summary>The row budget of the forward probe for the sample after the last time (boundary row + ties + one).</summary> /// <summary>The row budget of the forward probe for the sample after the last time (boundary row + ties + one).</summary>
internal const int ProbeRows = 2 + RawAggregateReader.PreWindowAllowance; internal const int ProbeRows = 2 + RawAggregateReader.PreWindowAllowance;
/// <summary>The smallest row budget a read can work with (the boundary row, a tie, one sample beyond).</summary>
internal const int MinBudget = 4;
/// <summary>Raw reads one request may always take …</summary>
internal const int BaseReads = 64;
/// <summary>… plus this many per distinct requested time.</summary>
internal const int ReadsPerTime = 4;
/// <summary>The first walk-back segment's width when the window read showed no density.</summary> /// <summary>The first walk-back segment's width when the window read showed no density.</summary>
internal static readonly TimeSpan InitialWalkWidth = TimeSpan.FromMinutes(1); internal static readonly TimeSpan InitialWalkWidth = TimeSpan.FromMinutes(1);
/// <summary>The widest walk-back segment sized from density, before ×4 growth.</summary> /// <summary>The widest walk-back segment sized from density, before ×4 growth.</summary>
private static readonly TimeSpan MaxSeededWidth = TimeSpan.FromDays(1); private static readonly TimeSpan MaxSeededWidth = TimeSpan.FromDays(1);
/// <summary>The most reads one walk-back may take.</summary>
internal const int MaxWalkReads = 64;
private const long Ms = TimeSpan.TicksPerMillisecond; private const long Ms = TimeSpan.TicksPerMillisecond;
private static readonly TimeSpan OneMs = TimeSpan.FromMilliseconds(1);
private static readonly DateTime Floor = RawAggregateReader.ProbeFloor; private static readonly DateTime Floor = RawAggregateReader.ProbeFloor;
/// <summary>One drained read: the rows strictly inside the request (ascending) and whether the stream ended short.</summary> /// <summary>One drained read: the rows strictly inside the request (ascending) and whether the stream ended short.</summary>
@@ -86,28 +107,49 @@ internal static class GatewayAtTimeReader
/// <param name="timestampsUtc">Requested times (any kind; treated as UTC, as on the SQL backends).</param> /// <param name="timestampsUtc">Requested times (any kind; treated as UTC, as on the SQL backends).</param>
/// <param name="now">Server timestamp, the beyond-now rule and the look-ahead horizon.</param> /// <param name="now">Server timestamp, the beyond-now rule and the look-ahead horizon.</param>
/// <param name="ct">Cancellation.</param> /// <param name="ct">Cancellation.</param>
/// <returns>One value per requested time, stamped with it.</returns>
/// <exception cref="HistoryReadLimitExceededException">More than <see cref="ReadRows"/> rows within one millisecond.</exception>
/// <param name="valueType">The node's scalar value type (the hinted <c>IHistorianDataSource.ReadAtTimeAsync</c>), or null.</param> /// <param name="valueType">The node's scalar value type (the hinted <c>IHistorianDataSource.ReadAtTimeAsync</c>), or null.</param>
/// <param name="maxReads">The request's raw-read cap; null = <see cref="BaseReads"/> + <see cref="ReadsPerTime"/> per distinct time.</param>
/// <returns>One value per requested time, stamped with it.</returns>
/// <exception cref="HistoryReadLimitExceededException">More than <see cref="ReadRows"/> rows within one millisecond, the
/// read cap, or a strict gateway ceiling below <see cref="MinBudget"/>.</exception>
public static async Task<IReadOnlyList<DataValueSnapshot>> ReadAsync(IHistorianGatewayClient client, GatewayRowCeiling ceiling, public static async Task<IReadOnlyList<DataValueSnapshot>> ReadAsync(IHistorianGatewayClient client, GatewayRowCeiling ceiling,
string tag, IReadOnlyList<DateTime> timestampsUtc, DateTime now, CancellationToken ct, TypeCode? valueType = null) string tag, IReadOnlyList<DateTime> timestampsUtc, DateTime now, CancellationToken ct, TypeCode? valueType = null,
int? maxReads = null)
{ {
var requested = timestampsUtc.Select(t => DateTime.SpecifyKind(t, DateTimeKind.Utc)).ToArray(); var requested = timestampsUtc.Select(t => DateTime.SpecifyKind(t, DateTimeKind.Utc)).ToArray();
if (requested.Length == 0) return []; if (requested.Length == 0) return [];
var neighbours = await ResolveAsync(new Ctx(client, ceiling, tag, ct, valueType), requested.Distinct().Order().ToArray(), now) var times = requested.Distinct().Order().ToArray();
.ConfigureAwait(false); var ctx = new Ctx(client, ceiling, tag, ct, valueType, maxReads ?? BaseReads + ReadsPerTime * times.Length);
var neighbours = await ResolveAsync(ctx, times, now).ConfigureAwait(false);
var result = new DataValueSnapshot[requested.Length]; var result = new DataValueSnapshot[requested.Length];
for (var i = 0; i < requested.Length; i++) for (var i = 0; i < requested.Length; i++)
{ {
var t = requested[i]; var t = requested[i];
var (prev, next) = neighbours[t.Ticks]; var (prev, next) = neighbours[t.Ticks];
// A DateTime node is stepped (as on the SQL backends): a usable non-numeric next sends the evaluator down its
// stepped branch, with the next's own status.
if (valueType == TypeCode.DateTime && next is { IsUsable: true } n)
next = n with { Value = HistorianValue.FromString(string.Empty) };
result[i] = ToNodeType(AtTimeEvaluator.Evaluate(t, prev, next, now, now), valueType); result[i] = ToNodeType(AtTimeEvaluator.Evaluate(t, prev, next, now, now), valueType);
} }
return result; return result;
} }
private sealed record Ctx(IHistorianGatewayClient Client, GatewayRowCeiling Ceiling, string Tag, CancellationToken Ct, /// <summary>One request's read context: the client, the type hint, and the read count against its cap.</summary>
TypeCode? ValueType); private sealed class Ctx(IHistorianGatewayClient client, GatewayRowCeiling ceiling, string tag, CancellationToken ct,
TypeCode? valueType, int maxReads)
{
public IHistorianGatewayClient Client { get; } = client;
public GatewayRowCeiling Ceiling { get; } = ceiling;
public string Tag { get; } = tag;
public CancellationToken Ct { get; } = ct;
public TypeCode? ValueType { get; } = valueType;
public int MaxReads { get; } = maxReads;
public int Reads { get; set; }
/// <summary>The smallest budget a read was served with (a strict gateway's ceiling shows here).</summary>
public int SmallestBudget { get; set; } = int.MaxValue;
}
/// <summary>True for the integer type codes (evaluated as Int64, returned as the node's own type).</summary> /// <summary>True for the integer type codes (evaluated as Int64, returned as the node's own type).</summary>
private static bool IsInteger(TypeCode? t) => private static bool IsInteger(TypeCode? t) =>
@@ -145,21 +187,22 @@ internal static class GatewayAtTimeReader
for (; i < times.Length && times[i] <= Floor; i++) result[times[i].Ticks] = (null, null); for (; i < times.Length && times[i] <= Floor; i++) result[times[i].Ticks] = (null, null);
if (i == times.Length) return result; if (i == times.Length) return result;
// Reads are planned on times clamped to the horizon (nothing is stored beyond it). // Reads are planned on times clamped to the horizon (nothing is stored beyond it, and no read ends past it).
var qe = MsFloor(Min(times[^1], horizon)) + TimeSpan.FromMilliseconds(1); var qe = Min(MsFloor(Min(times[^1], horizon)) + OneMs, horizon);
var qs = RequestStart(Min(times[i], horizon)); var qs = RequestStart(Min(times[i], horizon));
var budget = ReadRows;
StoredSample? before = null; // the latest stored sample at or before qs … StoredSample? before = null; // the latest stored sample at or before qs …
var beforeKnown = false; // … once known var beforeKnown = false; // … once known
DateTime? walkKnown = null; // a stored sample exists here (at or before qs): the walk-back never goes below it DateTime? walkKnown = null; // a stored sample exists here (at or before qs): the walk-back bisects above it
var walkWidth = InitialWalkWidth; var walkWidth = InitialWalkWidth;
var probeFrom = -1; // first index whose next lies at or after qe (complete last read) var probeFrom = -1; // first index whose next lies at or after qe (complete last read)
while (true) while (true)
{ {
var read = await ReadTrustedAsync(ctx, qs, qe, ReadRows).ConfigureAwait(false); var read = await ReadTrustedAsync(ctx, qs, qe, budget).ConfigureAwait(false);
var rows = read.Rows; var rows = read.Rows;
if (!read.Complete && rows.Count == 0) if (!read.Complete && rows.Count == 0)
throw TooDense(ctx.Tag, qs); throw TooDense(ctx, qs);
if (rows.Count >= 2 && !beforeKnown && walkKnown is null) if (rows.Count >= 2 && !beforeKnown && walkKnown is null)
walkWidth = Clamp(TimeSpan.FromTicks(4 * (rows[^1].TimeUtc - rows[0].TimeUtc).Ticks / (rows.Count - 1))); walkWidth = Clamp(TimeSpan.FromTicks(4 * (rows[^1].TimeUtc - rows[0].TimeUtc).Ticks / (rows.Count - 1)));
@@ -197,15 +240,17 @@ internal static class GatewayAtTimeReader
if (k == times.Length) break; if (k == times.Length) break;
// Truncated before times[k]: continue from the last row, or jump to times[k] with a density-sized margin. // Truncated before times[k]: continue from the last row, or jump to times[k] with a density-sized margin and a
// small budget (it needs that time's neighbours, and whatever later times follow closely).
var tn = Min(times[k], horizon); var tn = Min(times[k], horizon);
var spanTicks = Math.Max(1, (x - rows[0].TimeUtc).Ticks); var spanTicks = Math.Max(1, (x - rows[0].TimeUtc).Ticks);
var density = (double)rows.Count / spanTicks; // rows per tick var density = (double)rows.Count / spanTicks; // rows per tick
var gapRows = density * (tn - x).Ticks; var gapRows = density * (tn - x).Ticks;
var margin = Clamp(TimeSpan.FromTicks((long)(ReadRows / 4 / Math.Max(density, 1e-18)))); var margin = Clamp(TimeSpan.FromTicks((long)Math.Min(TimeSpan.MaxValue.Ticks / 2.0, JumpRows / 4.0 / Math.Max(density, 1e-18))));
var resume = MsFloor(x) - TimeSpan.FromMilliseconds(1); var resume = MsFloor(x) - OneMs;
var nextQs = gapRows < ReadRows / 2.0 ? resume : Max(RequestStart(tn - Min(margin, tn - Floor)), resume); var jump = gapRows >= ReadRows / 2.0;
if (nextQs <= qs) throw TooDense(ctx.Tag, x); var nextQs = jump ? Max(RequestStart(tn - Min(margin, tn - Floor)), resume) : resume;
if (nextQs <= qs) throw TooDense(ctx, x);
if (nextQs < x) if (nextQs < x)
{ {
@@ -225,13 +270,14 @@ internal static class GatewayAtTimeReader
walkKnown = x; walkKnown = x;
walkWidth = margin; walkWidth = margin;
} }
budget = jump ? JumpRows : ReadRows;
qs = nextQs; qs = nextQs;
i = k; i = k;
} }
if (probeFrom >= 0 && qe - TimeSpan.FromMilliseconds(1) < horizon) if (probeFrom >= 0 && qe < horizon)
{ {
var after = await FirstAfterAsync(ctx, qe - TimeSpan.FromMilliseconds(1), horizon, Min(times[^1], horizon)).ConfigureAwait(false); var after = await FirstAfterAsync(ctx, qe - OneMs, horizon, Min(times[^1], horizon)).ConfigureAwait(false);
if (after is not null) if (after is not null)
for (var k = probeFrom; k < times.Length; k++) for (var k = probeFrom; k < times.Length; k++)
{ {
@@ -247,70 +293,92 @@ internal static class GatewayAtTimeReader
/// <summary>The evaluator reads <c>next</c> only for a usable <c>prev</c> that is not an exact hit.</summary> /// <summary>The evaluator reads <c>next</c> only for a usable <c>prev</c> that is not an exact hit.</summary>
private static bool NeedsNext(DateTime t, StoredSample? prev) => prev is { IsUsable: true } p && p.TimeUtc != t; private static bool NeedsNext(DateTime t, StoredSample? prev) => prev is { IsUsable: true } p && p.TimeUtc != t;
/// <summary>One walk-back segment <c>(lo, hi]</c>: its oldest and newest rows, and whether the read was complete.</summary>
private static async Task<(StoredSample? First, StoredSample? Last, bool Complete)> SegmentAsync(
Ctx ctx, DateTime lo, DateTime hi)
{
var read = await ReadTrustedAsync(ctx, MsFloor(lo), MsFloor(hi) + OneMs, ReadRows).ConfigureAwait(false);
StoredSample? first = null, last = null;
foreach (var r in read.Rows)
if (r.TimeUtc > lo && r.TimeUtc <= hi)
{
first ??= r;
last = r;
}
return (first, last, read.Complete);
}
/// <summary> /// <summary>
/// The latest stored sample with time in <c>(Floor, hi]</c>, walking back in forward-read segments (see the type /// The latest stored sample with time in <c>(Floor, hi]</c> (see the type remarks: gallop, then bisect).
/// remarks). <paramref name="known"/>: a stored sample exists at that time (≤ <paramref name="hi"/>). /// <paramref name="known"/>: a stored sample exists at that time (≤ <paramref name="hi"/>) — the walk then bisects
/// above it straight away, its first segment aimed at <paramref name="width"/>.
/// </summary> /// </summary>
private static async Task<StoredSample?> WalkBackAsync(Ctx ctx, DateTime hi, DateTime? known, TimeSpan width) private static async Task<StoredSample?> WalkBackAsync(Ctx ctx, DateTime hi, DateTime? known, TimeSpan width)
{ {
TimeSpan? aim = known is null ? null : width;
// Gallop: no stored row known below hi yet.
var empties = 0; var empties = 0;
var triedRest = false; while (known is null)
for (var reads = 0; reads < MaxWalkReads; reads++)
{ {
if (hi <= Floor) return null; if (hi <= Floor) return null;
var rest = known is null && !triedRest && empties >= 2; var rest = empties >= 2;
DateTime lo; var lo = rest || (hi - Floor).Ticks <= width.Ticks ? Floor : hi - width;
if (rest) var seg = await SegmentAsync(ctx, lo, hi).ConfigureAwait(false);
if (seg.Complete)
{ {
lo = Floor; if (seg.Last is { } found) return found;
triedRest = true;
}
else
{
lo = (hi - Floor).Ticks > width.Ticks ? hi - width : Floor;
if (known is { } kn && lo < kn) lo = kn.AddTicks(-1); // never below a row known to exist (it is the floor)
}
var read = await ReadTrustedAsync(ctx, MsFloor(lo), MsFloor(hi) + TimeSpan.FromMilliseconds(1), ReadRows)
.ConfigureAwait(false);
StoredSample? first = null, last = null;
DateTime? lastAny = null;
foreach (var r in read.Rows)
{
if (r.TimeUtc > lo && r.TimeUtc <= hi)
{
first ??= r;
last = r;
}
lastAny = r.TimeUtc;
}
if (read.Complete)
{
if (last is not null) return last;
if (lo <= Floor) return null; if (lo <= Floor) return null;
hi = lo; hi = lo;
empties++; empties++;
width = Grow(width); width = Grow(width);
continue; continue;
} }
// Full — a whole-remainder read of deep history too: its last row is a stored row below hi. Bisect above it.
// Truncated. A whole-remainder read of deep history proves nothing: carry on with segments. if (seg.Last is not { } seen) throw TooDense(ctx, lo);
if (rest) continue;
if (last is not { } seen || (known is { } k0 && seen.TimeUtc <= k0))
throw TooDense(ctx.Tag, lastAny ?? lo);
var spanTicks = Math.Max(1, (seen.TimeUtc - first!.Value.TimeUtc).Ticks); // the density of the rows seen
known = seen.TimeUtc; known = seen.TimeUtc;
// Aim at about a quarter of a read from the density just seen, and never wider than half the open stretch. aim = AimFrom(seg.First!.Value.TimeUtc, seen.TimeUtc);
var aimed = TimeSpan.FromTicks(Math.Max(Ms, spanTicks / 4)); }
var half = TimeSpan.FromTicks(Math.Max(Ms, (hi - seen.TimeUtc).Ticks / 2));
width = aimed < half ? aimed : half; // Bisect: a stored row exists at `known`, nothing newer than `hi` matters; the answer is the newest row in [known, hi].
empties = 0; var aimMissed = false;
while (true)
{
var k = known.Value;
DateTime lo;
var aimed = false;
if (hi - k < TimeSpan.FromMilliseconds(2))
lo = k.AddTicks(-1); // the whole stretch from the known row: never a sub-millisecond step
else
{
lo = k + TimeSpan.FromTicks((hi - k).Ticks / 2);
if (aim is { } a && hi - a > lo)
{
lo = hi - a; // an aimed try right after a full read — until one comes back empty (a gap: halve from then on)
aimed = true;
}
}
aim = null;
var seg = await SegmentAsync(ctx, lo, hi).ConfigureAwait(false);
if (seg.Complete)
{
if (seg.Last is { } found) return found;
if (lo < k) return null; // the known row is gone (deleted meanwhile): nothing is left to find
hi = lo;
aimMissed |= aimed;
continue;
}
if (seg.Last is not { } seen || seen.TimeUtc <= k) throw TooDense(ctx, k);
known = seen.TimeUtc;
if (!aimMissed) aim = AimFrom(seg.First!.Value.TimeUtc, seen.TimeUtc);
} }
throw new HistoryReadLimitExceededException(MaxWalkReads,
$"The AtTime read of '{ctx.Tag}' could not find the stored sample before a requested time within {MaxWalkReads} raw reads.");
} }
/// <summary>About a quarter of a read at the density a full read showed (its rows from <paramref name="first"/> to <paramref name="last"/>).</summary>
private static TimeSpan AimFrom(DateTime first, DateTime last) =>
TimeSpan.FromTicks(Math.Max(Ms, (last - first).Ticks / 4));
/// <summary>The first stored sample after <paramref name="after"/> in <c>(qs, end)</c>, from a small forward read.</summary> /// <summary>The first stored sample after <paramref name="after"/> in <c>(qs, end)</c>, from a small forward read.</summary>
private static async Task<StoredSample?> FirstAfterAsync(Ctx ctx, DateTime qs, DateTime end, DateTime after) private static async Task<StoredSample?> FirstAfterAsync(Ctx ctx, DateTime qs, DateTime end, DateTime after)
{ {
@@ -321,21 +389,35 @@ internal static class GatewayAtTimeReader
if (r.TimeUtc > after) return r; if (r.TimeUtc > after) return r;
if (read.Complete) return null; if (read.Complete) return null;
} }
throw TooDense(ctx.Tag, qs); throw TooDense(ctx, qs);
} }
/// <summary> /// <summary>
/// One forward read <c>[qs, qe]</c> (whole milliseconds), drained, keeping only rows strictly inside it and after the /// One forward read <c>[qs, qe]</c> (whole milliseconds), drained, keeping only rows strictly inside it and after the
/// epoch. A strict gateway's ceiling is learned on refusal and then asked for directly; a full read is not re-probed /// epoch — counted against the request's read cap. A strict gateway's ceiling is learned on refusal and then asked for
/// (the caller pages on, it does not need the rest of this read). /// directly; a full read is not re-probed (the caller pages on, it does not need the rest of this read).
/// </summary> /// </summary>
private static async Task<Read> ReadTrustedAsync(Ctx ctx, DateTime qs, DateTime qe, int budget) private static async Task<Read> ReadTrustedAsync(Ctx ctx, DateTime qs, DateTime qe, int budget)
{ {
var (read, _) = await ctx.Ceiling.ReadAsync(budget, b => PassAsync(ctx, qs, qe, b), (_, _) => false) if (++ctx.Reads > ctx.MaxReads)
throw new HistoryReadLimitExceededException(ctx.MaxReads,
$"The AtTime read of '{ctx.Tag}' needed more than {ctx.MaxReads} raw reads ({BaseReads} + {ReadsPerTime} per " +
"requested time)" + CeilingNote(ctx) + "; the Gateway backend serves AtTime from raw reads — request fewer times.");
var (read, used) = await ctx.Ceiling.ReadAsync(budget, b => PassAsync(ctx, qs, qe, b), (_, _) => false)
.ConfigureAwait(false); .ConfigureAwait(false);
if (used < ctx.SmallestBudget) ctx.SmallestBudget = used;
if (used < MinBudget)
throw new HistoryReadLimitExceededException(used,
$"The HistorianGateway's row ceiling ({used}, its Historian:ReadLimits:MaxValuesCeiling with " +
$"AllowStreamingBeyondCeiling=false) is below the {MinBudget} rows an AtTime raw read of '{ctx.Tag}' needs.");
return read; return read;
} }
private static string CeilingNote(Ctx ctx) =>
ctx.SmallestBudget < JumpRows
? $", with reads bounded by the HistorianGateway's row ceiling of {ctx.SmallestBudget} (Historian:ReadLimits:MaxValuesCeiling)"
: string.Empty;
private static async Task<Read> PassAsync(Ctx ctx, DateTime qs, DateTime qe, int budget) private static async Task<Read> PassAsync(Ctx ctx, DateTime qs, DateTime qe, int budget)
{ {
var rows = new List<StoredSample>(); var rows = new List<StoredSample>();
@@ -369,9 +451,11 @@ internal static class GatewayAtTimeReader
{ {
TypeCode.Boolean => HistorianValue.FromBoolean(v != 0), TypeCode.Boolean => HistorianValue.FromBoolean(v != 0),
TypeCode.String => HistorianValue.FromString(v.ToString("R", System.Globalization.CultureInfo.InvariantCulture)), TypeCode.String => HistorianValue.FromString(v.ToString("R", System.Globalization.CultureInfo.InvariantCulture)),
// In Int64 range only (2^63 is exactly representable; anything at or beyond it stays a Double). // In Int64 range only (2^63 is exactly representable); NaN, ±∞ or out of range is no integer at all — Empty, which
_ when IsInteger(valueType) && v >= -9.2233720368547758E18 && v < 9.2233720368547758E18 => // StoredSample.FromStored makes BadNoData (unusable), never a saturated Good value.
HistorianValue.FromInt64((long)Math.Round(v, MidpointRounding.AwayFromZero)), _ when IsInteger(valueType) => v >= -9.2233720368547758E18 && v < 9.2233720368547758E18
? HistorianValue.FromInt64((long)Math.Round(v, MidpointRounding.AwayFromZero))
: HistorianValue.Empty,
_ => HistorianValue.FromDouble(v), _ => HistorianValue.FromDouble(v),
}; };
@@ -395,8 +479,12 @@ internal static class GatewayAtTimeReader
private static TimeSpan Grow(TimeSpan w) => w.Ticks > TimeSpan.MaxValue.Ticks / 4 ? TimeSpan.MaxValue : w * 4; private static TimeSpan Grow(TimeSpan w) => w.Ticks > TimeSpan.MaxValue.Ticks / 4 ? TimeSpan.MaxValue : w * 4;
private static HistoryReadLimitExceededException TooDense(string tag, DateTime at) => private static HistoryReadLimitExceededException TooDense(Ctx ctx, DateTime at)
new(ReadRows, {
$"The AtTime read of '{tag}' met more than {ReadRows} stored samples within about one millisecond of " + var rows = Math.Min(ReadRows, ctx.SmallestBudget);
$"{at:O}; the Gateway backend serves AtTime from raw reads."); return new(rows,
$"The AtTime read of '{ctx.Tag}' met more than {rows} stored samples within about one millisecond of {at:O}" +
(rows < ReadRows ? $" (reads are bounded by the HistorianGateway's row ceiling of {rows})" : string.Empty) +
"; the Gateway backend serves AtTime from raw reads.");
}
} }
@@ -139,6 +139,7 @@ public abstract class AtTimeConformance : ConformanceBase
(At(10), HistorianValue.FromInt64(15), Good)); (At(10), HistorianValue.FromInt64(15), Good));
var s = (await B.CreateDataSource().ReadAtTimeAsync("I", [At(5), At(2)], Ct)).Samples; var s = (await B.CreateDataSource().ReadAtTimeAsync("I", [At(5), At(2)], Ct)).Samples;
s[0].Value.ShouldBe(13L); // 12.5 ⇒ 13 (away from zero) s[0].Value.ShouldBe(13L); // 12.5 ⇒ 13 (away from zero)
s[0].Value.ShouldBeOfType<long>(); // ShouldBe compares numbers across types — the type needs its own check (#693)
s[1].Value.ShouldBe(11L); // 11.0 s[1].Value.ShouldBe(11L); // 11.0
s[0].StatusCode.ShouldBe(GoodInterpolated); s[0].StatusCode.ShouldBe(GoodInterpolated);
} }
@@ -409,7 +409,7 @@ public sealed class GatewayAtTimeTests
var s = (await AtTime(fake, T0.AddMinutes(-5))).Single(); var s = (await AtTime(fake, T0.AddMinutes(-5))).Single();
s.Value.ShouldBe(599_999d); s.Value.ShouldBe(599_999d);
s.StatusCode.ShouldBe(GoodInterpolated); // stepped from the last sample (before now) s.StatusCode.ShouldBe(GoodInterpolated); // stepped from the last sample (before now)
fake.ReadRawCallCount.ShouldBe(7); // window, 1m, 4m, remainder (truncated), 16m (truncated), 256 ms, probe fake.ReadRawCallCount.ShouldBe(6); // window, 1m, 4m, remainder (full: seeds the bisection), one aimed segment, probe
store.RowsStreamed.ShouldBeLessThan(10_000); store.RowsStreamed.ShouldBeLessThan(10_000);
} }
@@ -446,6 +446,186 @@ public sealed class GatewayAtTimeTests
ds.GetHealthSnapshot().TotalFailures.ShouldBe(1); ds.GetHealthSnapshot().TotalFailures.ShouldBe(1);
} }
// ---- long histories (review H1/M1/L1): a generated store, too large to materialise -----------------------------
/// <summary>
/// A regular series (value = sample index) over <c>[from, to]</c> every <paramref name="step"/>, optionally with a
/// hole <c>(gapFrom, gapTo)</c>, served in the AVEVA wire shape (boundary row at the query start, ms bounds only,
/// <c>max_values &gt; 0</c>, <c>start &lt;= end</c>, nothing past now + 2 days).
/// </summary>
private sealed class SeriesStore(DateTime from, DateTime to, TimeSpan step, DateTime? gapFrom = null, DateTime? gapTo = null)
{
public long RowsStreamed;
private bool InGap(DateTime t) => gapFrom is { } a && gapTo is { } b && t > a && t < b;
private DateTime? FirstAtOrAfter(DateTime t)
{
if (t > to) return null;
var k = t <= from ? 0 : ((t - from).Ticks + step.Ticks - 1) / step.Ticks;
var r = from.AddTicks(k * step.Ticks);
if (InGap(r)) return FirstAtOrAfter(gapTo!.Value);
return r <= to ? r : null;
}
private DateTime? LastBefore(DateTime t)
{
if (t <= from) return null;
var k = ((Min(t, to.AddTicks(1)) - from).Ticks - 1) / step.Ticks;
var r = from.AddTicks(k * step.Ticks);
return InGap(r) ? LastBefore(gapFrom!.Value.AddTicks(1)) : r;
}
private static DateTime Min(DateTime a, DateTime b) => a <= b ? a : b;
/// <summary>The stored row at <paramref name="t"/>, stamped <paramref name="at"/> (the boundary row) when given.</summary>
private HistorianSample Row(DateTime t, uint quality = 0, DateTime? at = null) => new()
{
Tag = "T", Timestamp = Timestamp.FromDateTime(at ?? t), NumericValue = (t - from).Ticks / step.Ticks, OpcQuality = 192,
Quality = quality,
};
public FakeHistorianGatewayClient Client(int? ceiling = null) => new() { RawSource = Serve, MaxValuesCeiling = ceiling };
private IEnumerable<HistorianSample> Serve(DateTime start, DateTime end, int max)
{
if (max <= 0 || end < start || end > Now.AddDays(2)
|| start.Ticks % TimeSpan.TicksPerMillisecond != 0 || end.Ticks % TimeSpan.TicksPerMillisecond != 0)
throw new Grpc.Core.RpcException(new Grpc.Core.Status(Grpc.Core.StatusCode.InvalidArgument, "refused"));
var rows = new List<HistorianSample>();
if (FirstAtOrAfter(start) != start)
rows.Add(LastBefore(start) is { } p
? Row(p, 133, start)
: new HistorianSample { Tag = "T", Timestamp = Timestamp.FromDateTime(start), OpcQuality = 0, Quality = 1, NumericValue = 0 });
for (var t = FirstAtOrAfter(start); t is { } u && u <= end && rows.Count < max; t = FirstAtOrAfter(u.AddTicks(1)))
rows.Add(Row(u));
RowsStreamed += rows.Count;
return rows;
}
}
[Theory] // H1: a dense tag that stopped reporting — the walk crosses the silence by bisection, not by regrowing segments
[InlineData(1.0, 30)]
[InlineData(1.0, 365)]
[InlineData(10.0, 30)]
[InlineData(10.0, 365)]
public async Task A_dense_tag_that_stopped_long_ago_is_found_in_a_bounded_number_of_reads(double hz, int stoppedDaysAgo)
{
var step = TimeSpan.FromTicks((long)(TimeSpan.TicksPerSecond / hz));
var end = Now.AddDays(-stoppedDaysAgo);
var fake = new SeriesStore(end.AddYears(-5), end, step).Client();
var s = (await AtTime(fake, Now.AddMinutes(-1))).Single();
s.Value.ShouldBe((double)((end - end.AddYears(-5)).Ticks / step.Ticks)); // the last sample, held
s.StatusCode.ShouldBe(GoodInterpolated);
fake.ReadRawCallCount.ShouldBeLessThan(40);
}
[Fact] // H1: a time inside a month-long hole in the middle of a dense history
public async Task A_time_inside_a_mid_history_gap_interpolates_across_it()
{
var from = Now.AddYears(-2);
var gapFrom = Now.AddDays(-200);
var gapTo = gapFrom.AddDays(30);
var fake = new SeriesStore(from, Now, TimeSpan.FromSeconds(1), gapFrom, gapTo).Client();
var t = gapFrom.AddDays(15);
var s = (await AtTime(fake, t)).Single();
var before = (gapFrom - from).Ticks / TimeSpan.TicksPerSecond;
var after = (gapTo - from).Ticks / TimeSpan.TicksPerSecond;
((double)s.Value!).ShouldBe(before + (after - before) * 0.5, 1e-6);
s.StatusCode.ShouldBe(GoodInterpolated);
fake.ReadRawCallCount.ShouldBeLessThan(40);
}
[Fact] // H1: a jump to a time in a gap — the walk starts from the earlier read's last row, far below
public async Task A_jump_into_a_gap_bisects_up_from_the_earlier_reads_last_row()
{
var from = Now.AddDays(-30);
var gapFrom = from.AddDays(2);
var gapTo = gapFrom.AddDays(10);
var fake = new SeriesStore(from, Now, TimeSpan.FromSeconds(1), gapFrom, gapTo).Client();
var t0 = from.AddSeconds(10.5);
var t1 = gapFrom.AddDays(5);
var s = await AtTime(fake, t0, t1);
s[0].Value.ShouldBe(10.5d);
var before = (gapFrom - from).Ticks / TimeSpan.TicksPerSecond;
var after = (gapTo - from).Ticks / TimeSpan.TicksPerSecond;
((double)s[1].Value!).ShouldBe(before + (after - before) * 0.5, 1e-6);
fake.ReadRawCallCount.ShouldBeLessThan(40);
}
[Fact] // M1: widely spread times cost one SMALL read each, not one 1,024-row read each
public async Task A_thousand_times_over_ten_years_cost_one_small_read_each()
{
var store = new SeriesStore(Now.AddYears(-10), Now, TimeSpan.FromSeconds(1));
var fake = store.Client();
var rnd = new Random(1);
var times = Enumerable.Range(0, 1000).Select(_ => Now.AddYears(-9).AddTicks((long)(rnd.NextDouble() * TimeSpan.TicksPerDay * 3000))).ToArray();
var s = await AtTime(fake, times);
s.ShouldAllBe(x => x.StatusCode == GoodInterpolated);
fake.ReadRawCallCount.ShouldBeLessThanOrEqualTo(1010);
store.RowsStreamed.ShouldBeLessThan(80_000); // ~64 rows a read (was ~1,000,000)
}
[Fact] // L1: a strict-gateway ceiling too small to work with is refused with a message naming it
public async Task A_ceiling_below_a_usable_read_is_refused_naming_the_ceiling()
{
var fake = new SeriesStore(Now.AddDays(-1), Now, TimeSpan.FromSeconds(1)).Client(ceiling: 2);
var ex = await Should.ThrowAsync<HistoryReadLimitExceededException>(
() => Source(fake).ReadAtTimeAsync("T", [Now.AddHours(-1)], Ct));
ex.Message.ShouldContain("row ceiling (2");
}
[Fact] // L1: a small ceiling is served (paging), within the read cap
public async Task A_small_ceiling_is_served_within_the_read_cap()
{
var fake = new SeriesStore(Now.AddYears(-1), Now, TimeSpan.FromSeconds(1)).Client(ceiling: 10);
var times = Enumerable.Range(0, 50).Select(i => Now.AddDays(-300 + i * 5).AddSeconds(0.5)).ToArray();
var s = await AtTime(fake, times);
s.ShouldAllBe(x => x.StatusCode == GoodInterpolated);
fake.ReadRawCallCount.ShouldBeLessThanOrEqualTo(GatewayAtTimeReader.BaseReads + GatewayAtTimeReader.ReadsPerTime * 50);
}
[Fact] // L1: the request's read cap stops a walk with a clear error, never a hang (here a cap of 3 vs. a 5-read walk)
public async Task The_read_cap_stops_a_request_with_a_clear_error()
{
var fake = new AvevaStore([(T0.AddDays(-400), 42, false), (T0.AddDays(-400).AddSeconds(10), 43, false)]).Client();
var ex = await Should.ThrowAsync<HistoryReadLimitExceededException>(() => GatewayAtTimeReader.ReadAsync(
fake, new GatewayRowCeiling(), "T", [T0], Now, Ct, maxReads: 3));
ex.Message.ShouldContain("needed more than 3 raw reads");
fake.ReadRawCallCount.ShouldBe(3);
fake.RawStreamsCompleted.ShouldBe(3);
}
[Fact] // the default cap scales with the request: 64 + 4 per distinct time
public async Task The_default_read_cap_is_64_plus_4_per_distinct_time()
{
(GatewayAtTimeReader.BaseReads, GatewayAtTimeReader.ReadsPerTime).ShouldBe((64, 4));
await Task.CompletedTask;
}
// ---- type hint edge cases (review L2/L3) ------------------------------------------------------------------------
[Fact]
public async Task A_DateTime_node_is_stepped()
{
var fake = new AvevaStore([(At(0), 0, false), (At(10), 10, false)]).Client();
var s = (await AtTimeTyped(fake, TypeCode.DateTime, At(5))).Single();
s.Value.ShouldBe(0d);
s.StatusCode.ShouldBe(GoodInterpolated);
}
[Theory]
[InlineData(double.PositiveInfinity)]
[InlineData(double.NaN)]
[InlineData(1e30)]
public async Task Under_an_integer_hint_a_non_integer_next_value_is_unusable_not_saturated(double next)
{
var fake = new AvevaStore([(At(0), 100, false), (At(10), next, false)]).Client();
var s = (await AtTimeTyped(fake, TypeCode.Int32, At(5))).Single();
s.Value.ShouldBeOfType<int>().ShouldBe(100); // held before an unusable neighbour
s.StatusCode.ShouldBe(UncertainInterpolated);
}
// ---- kill switch ----------------------------------------------------------------------------------------------- // ---- kill switch -----------------------------------------------------------------------------------------------
[Fact] [Fact]