fix(historian): #693 review — bisecting walk-back, small jump reads, read cap, DateTime/NaN hints
H1: the walk-back reset its width after a full segment and discarded a full whole-remainder read, so a dense tag that stopped reporting weeks ago gave up after 65 reads (BadTooManyOperations). It now gallops (x4, whole-remainder read after two empties) and then BISECTS once any stored row is known below — a full segment's (incl. a full whole-remainder read's) last row, or an earlier window read's last row: (mid, hi] empty => hi = mid; full => known = its last row; complete with rows => done; one density-aimed segment after a full read until an aimed segment comes back empty; below 2 ms the whole stretch from the known row (never a 100 ns step). Stopped 1 h..365 d with 5 y history: 6-18 reads at 1/min, 21-25 at 1 Hz, 25-31 at 10 Hz. M1: a jump read asks for 64 rows (1,000 times over 10 y: ~66k rows, was ~1M). L1: per-request read cap 64 + 4 x distinct times with a clear message; a strict ceiling below 4 rows is refused naming the ceiling. L2: DateTime hints are stepped. L3: under an integer hint a NaN/inf or out-of-Int64-range value is unusable, never saturated Good. L5: conformance Int64 AtTime asserts the type. L4: docs name the backend differences. Nit: the window read end is clamped to now + 1 day. Reviewer harnesses (Scratch693Fuzz/Cost, not committed): 0 differential diffs over 2,400 cases.
This commit is contained in:
+23
-9
@@ -780,14 +780,23 @@ data.
|
|||||||
**probe** (18 rows) for the sample after the last. Measured on the fake: ten times one second apart on a 1 Hz tag cost **1 read**
|
**probe** (18 rows) for the sample after the last. Measured on the fake: ten times one second apart on a 1 Hz tag cost **1 read**
|
||||||
(all exact hits) or **3** (interpolated); ten times an hour apart on a 1 Hz tag **12** (≈ 9.5k rows, never the 32k between); one
|
(all exact hits) or **3** (interpolated); ten times an hour apart on a 1 Hz tag **12** (≈ 9.5k rows, never the 32k between); one
|
||||||
time 400 days after the only samples **5**; a tag with no history **4**. A window read that fills its budget resolves the times before
|
time 400 days after the only samples **5**; a tag with no history **4**. A window read that fills its budget resolves the times before
|
||||||
its last row; the next time either continues from there (if the rows in between should fit one read) or jumps to it with a read
|
its last row; the next time either continues from there (if the rows in between should fit one read) or **jumps** to it with a
|
||||||
that starts a density-sized margin earlier (so its `prev` is normally in the same read).
|
small read (64 rows) that starts a density-sized margin earlier (so its `prev` is normally in the same read) — 1,000 times
|
||||||
- **Walk-back** (the gateway reads forward only): segments stepping back from the time, the first sized from the window read's density
|
spread over ten years of a 1 Hz tag cost ≈ 1,000 reads of ≤ 64 rows (≈ 66k rows; it used to be 1,024-row reads).
|
||||||
(else 1 minute), growing ×4 when empty; after two empty segments the whole remainder down to the epoch is asked once (a complete
|
- **Walk-back** (the gateway reads forward only), in two phases. **Gallop:** segments stepping back from the time, the first sized
|
||||||
answer ends the walk — sparse history and no history are found in a handful of reads). A segment that fills its budget is narrowed
|
from the window read's density (else 1 minute), growing ×4 when empty; after two empty segments the whole remainder down to the
|
||||||
towards the time to about a quarter of a read at the density it showed — never below a row it saw — so dense history is bisected,
|
epoch is asked once — a complete answer ends the walk, a **full** one seeds the bisection with its last row. **Bisect** (once a
|
||||||
not paged (a 1 kHz tag behind a 10-minute gap: 7 reads, < 10k rows). At most 64 reads; more than 1,024 stored samples within one
|
stored row is known below — a full segment's, or an earlier window read's last row): read `(mid, hi]`; empty ⇒ `hi = mid`; full ⇒
|
||||||
millisecond fail the node `BadTooManyOperations` (not counted as a backend failure).
|
the known row moves up to its last row; complete with rows ⇒ done. Right after a full read one segment is aimed at about a quarter
|
||||||
|
of a read at the density it showed (until an aimed segment comes back empty — a gap — then plain halving); below 2 ms the segment
|
||||||
|
is the whole stretch from the known row (never a 100 ns step). Measured (review harness, 5 years of history, time = now − 1 min):
|
||||||
|
a tag stopped 1 h / 1 d / 7 d / 30 d / 365 d ago costs 6–18 reads at 1/min, 21–25 at 1 Hz and 25–31 at 10 Hz; a 1 kHz tag behind
|
||||||
|
a 10-minute gap 6 reads (< 10k rows). (Before the review fix a full segment reset the width and a full whole-remainder read was
|
||||||
|
discarded, so a dense tag that stopped weeks ago gave up after 65 reads.) More than 1,024 stored samples within one millisecond
|
||||||
|
fail the node `BadTooManyOperations` (not counted as a backend failure).
|
||||||
|
- **Bounded.** One request takes at most **64 + 4 × distinct times** raw reads; past it the node fails `BadTooManyOperations`
|
||||||
|
("needed more than N raw reads", naming a small strict-gateway ceiling when one bounded the reads). A strict gateway whose ceiling is
|
||||||
|
below 4 rows is refused up front with a message naming the ceiling; a small one (e.g. 10) is served by paging within the cap.
|
||||||
- **Wire shape.** Every request bound is a whole millisecond and a row counts only **strictly inside** its request, so AVEVA's synthetic
|
- **Wire shape.** Every request bound is a whole millisecond and a row counts only **strictly inside** its request, so AVEVA's synthetic
|
||||||
boundary row at the query start (#687) — and any row at the query end — is never taken for a stored sample. A Bad row is value-less
|
boundary row at the query start (#687) — and any row at the query end — is never taken for a stored sample. A Bad row is value-less
|
||||||
(#690). Nothing is asked past now + 1 day (the gateway's AVEVA may refuse a far-future end; a time beyond it is stepped from the last
|
(#690). Nothing is asked past now + 1 day (the gateway's AVEVA may refuse a far-future end; a time beyond it is stepped from the last
|
||||||
@@ -800,7 +809,12 @@ data.
|
|||||||
**before** the shared evaluator runs, so its own type rules apply exactly as on the SQL backends: a **Boolean** node
|
**before** the shared evaluator runs, so its own type rules apply exactly as on the SQL backends: a **Boolean** node
|
||||||
(value ≠ 0) is **stepped** — never 0.5; an **integer** node is interpolated, rounded half away from zero and returned as the
|
(value ≠ 0) is **stepped** — never 0.5; an **integer** node is interpolated, rounded half away from zero and returned as the
|
||||||
node's own integer type (`Int32` ⇒ `int`); Float / Double interpolate as Double (as the SQL backends return them). With no
|
node's own integer type (`Int32` ⇒ `int`); Float / Double interpolate as Double (as the SQL backends return them). With no
|
||||||
hint (an unknown node) every numeric tag interpolates as Double.
|
hint (an unknown node) every numeric tag interpolates as Double. A **DateTime** node is stepped. Under an integer hint a NaN, ±∞ or
|
||||||
|
out-of-Int64-range stored value is no integer: it is treated as unusable (as `BadNoData`), so a neighbour like that is never
|
||||||
|
interpolated or saturated into a Good value.
|
||||||
|
- **What differs between backends (by design, not changed by #693).** Only the **Gateway** converts an AtTime result to the node's
|
||||||
|
integer type (`Int32` ⇒ `int`): the SQL backends store and return integer nodes as `Int64` (and Boolean as Boolean). Gateway
|
||||||
|
**Raw** reads still return a Boolean node's samples as the stored Double 0/1 (the hint is passed only on AtTime).
|
||||||
- ⚠️ **Shared-evaluator fix (all backends that use it):** `AtTimeEvaluator` returned every interpolated `Int64` boxed as a
|
- ⚠️ **Shared-evaluator fix (all backends that use it):** `AtTimeEvaluator` returned every interpolated `Int64` boxed as a
|
||||||
**Double** (a C# `cond ? (long)x : double` conditional widens the long); its "keeping the Int64 type" tests passed because
|
**Double** (a C# `cond ? (long)x : double` conditional widens the long); its "keeping the Int64 type" tests passed because
|
||||||
Shouldly's `ShouldBe` compares numbers across types. SqlServer / QuestDB interpolated `Int64` AtTime values are now `Int64`,
|
Shouldly's `ShouldBe` compares numbers across types. SqlServer / QuestDB interpolated `Int64` AtTime values are now `Int64`,
|
||||||
|
|||||||
@@ -30,12 +30,22 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway;
|
|||||||
/// apart on a 1 Hz tag cost 1 read (all exact hits) or 3 (interpolated); ten times an hour apart about one read each.
|
/// apart on a 1 Hz tag cost 1 read (all exact hits) or 3 (interpolated); ten times an hour apart about one read each.
|
||||||
/// </para>
|
/// </para>
|
||||||
/// <para>
|
/// <para>
|
||||||
/// <b>Walk-back</b> (the gateway reads forward only): segments <c>(lo, hi]</c> stepping back from <c>hi</c>, each one
|
/// <b>Walk-back</b> (the gateway reads forward only), in two phases. <i>Gallop:</i> segments <c>(lo, hi]</c> stepping
|
||||||
/// forward read; an empty segment grows the next ×4 (from the density the window read showed, else
|
/// back from <c>hi</c>, each one forward read, the first sized from the density the window read showed (else
|
||||||
/// <see cref="InitialWalkWidth"/>), and after two empty segments the whole remainder down to the epoch is asked once
|
/// <see cref="InitialWalkWidth"/>), an empty one growing the next ×4; after two empty segments the whole remainder down
|
||||||
/// (a complete answer ends the walk). A segment that fills its budget is narrowed towards <c>hi</c> to a width sized
|
/// to the epoch is asked once — a complete answer ends the walk, a FULL one seeds the bisection with its last row.
|
||||||
/// from the density it showed, never below a row it saw (rows exist there) — so dense history is bisected, not paged.
|
/// <i>Bisect</i> (once a stored row is known below <c>hi</c> — a full segment's last row, or the last row of an earlier
|
||||||
/// At most <see cref="MaxWalkReads"/> reads.
|
/// window read): read <c>(mid, hi]</c>; empty ⇒ <c>hi = mid</c>; full ⇒ the known row moves up to its last row;
|
||||||
|
/// complete with rows ⇒ its last row is the answer. Right after a full read the next segment is aimed at about a
|
||||||
|
/// quarter of a read at the density it showed (one try — an empty aimed segment falls back to halving), and below
|
||||||
|
/// 2 ms the segment is the whole stretch from the known row (never a 100 ns step).
|
||||||
|
/// </para>
|
||||||
|
/// <para>
|
||||||
|
/// <b>Bounded.</b> One request may take at most <c>64 + 4 × distinct times</c> raw reads (<see cref="BaseReads"/>,
|
||||||
|
/// <see cref="ReadsPerTime"/>); past it — or with a strict-gateway ceiling below <see cref="MinBudget"/> rows — the
|
||||||
|
/// node fails <c>BadTooManyOperations</c> with a message naming the limit. A jump read (to a time far past the previous
|
||||||
|
/// read) asks for only <see cref="JumpRows"/> rows, so a thousand times spread over ten years cost about a thousand
|
||||||
|
/// small reads, not a thousand 1,024-row reads.
|
||||||
/// </para>
|
/// </para>
|
||||||
/// <para>
|
/// <para>
|
||||||
/// <b>Wire shape.</b> Every request bound is a whole millisecond (AVEVA stores milliseconds and truncates request
|
/// <b>Wire shape.</b> Every request bound is a whole millisecond (AVEVA stores milliseconds and truncates request
|
||||||
@@ -57,23 +67,34 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway;
|
|||||||
/// </remarks>
|
/// </remarks>
|
||||||
internal static class GatewayAtTimeReader
|
internal static class GatewayAtTimeReader
|
||||||
{
|
{
|
||||||
/// <summary>The row budget of a window read and a walk-back segment.</summary>
|
/// <summary>The row budget of a window read (the first, or one continuing from a full one) and a walk-back segment.</summary>
|
||||||
internal const int ReadRows = 1024;
|
internal const int ReadRows = 1024;
|
||||||
|
|
||||||
|
/// <summary>The row budget of a jump read (a window read restarting at a time far past the previous read's last row).</summary>
|
||||||
|
internal const int JumpRows = 64;
|
||||||
|
|
||||||
/// <summary>The row budget of the forward probe for the sample after the last time (boundary row + ties + one).</summary>
|
/// <summary>The row budget of the forward probe for the sample after the last time (boundary row + ties + one).</summary>
|
||||||
internal const int ProbeRows = 2 + RawAggregateReader.PreWindowAllowance;
|
internal const int ProbeRows = 2 + RawAggregateReader.PreWindowAllowance;
|
||||||
|
|
||||||
|
/// <summary>The smallest row budget a read can work with (the boundary row, a tie, one sample beyond).</summary>
|
||||||
|
internal const int MinBudget = 4;
|
||||||
|
|
||||||
|
/// <summary>Raw reads one request may always take …</summary>
|
||||||
|
internal const int BaseReads = 64;
|
||||||
|
|
||||||
|
/// <summary>… plus this many per distinct requested time.</summary>
|
||||||
|
internal const int ReadsPerTime = 4;
|
||||||
|
|
||||||
/// <summary>The first walk-back segment's width when the window read showed no density.</summary>
|
/// <summary>The first walk-back segment's width when the window read showed no density.</summary>
|
||||||
internal static readonly TimeSpan InitialWalkWidth = TimeSpan.FromMinutes(1);
|
internal static readonly TimeSpan InitialWalkWidth = TimeSpan.FromMinutes(1);
|
||||||
|
|
||||||
/// <summary>The widest walk-back segment sized from density, before ×4 growth.</summary>
|
/// <summary>The widest walk-back segment sized from density, before ×4 growth.</summary>
|
||||||
private static readonly TimeSpan MaxSeededWidth = TimeSpan.FromDays(1);
|
private static readonly TimeSpan MaxSeededWidth = TimeSpan.FromDays(1);
|
||||||
|
|
||||||
/// <summary>The most reads one walk-back may take.</summary>
|
|
||||||
internal const int MaxWalkReads = 64;
|
|
||||||
|
|
||||||
private const long Ms = TimeSpan.TicksPerMillisecond;
|
private const long Ms = TimeSpan.TicksPerMillisecond;
|
||||||
|
|
||||||
|
private static readonly TimeSpan OneMs = TimeSpan.FromMilliseconds(1);
|
||||||
|
|
||||||
private static readonly DateTime Floor = RawAggregateReader.ProbeFloor;
|
private static readonly DateTime Floor = RawAggregateReader.ProbeFloor;
|
||||||
|
|
||||||
/// <summary>One drained read: the rows strictly inside the request (ascending) and whether the stream ended short.</summary>
|
/// <summary>One drained read: the rows strictly inside the request (ascending) and whether the stream ended short.</summary>
|
||||||
@@ -86,28 +107,49 @@ internal static class GatewayAtTimeReader
|
|||||||
/// <param name="timestampsUtc">Requested times (any kind; treated as UTC, as on the SQL backends).</param>
|
/// <param name="timestampsUtc">Requested times (any kind; treated as UTC, as on the SQL backends).</param>
|
||||||
/// <param name="now">Server timestamp, the beyond-now rule and the look-ahead horizon.</param>
|
/// <param name="now">Server timestamp, the beyond-now rule and the look-ahead horizon.</param>
|
||||||
/// <param name="ct">Cancellation.</param>
|
/// <param name="ct">Cancellation.</param>
|
||||||
/// <returns>One value per requested time, stamped with it.</returns>
|
|
||||||
/// <exception cref="HistoryReadLimitExceededException">More than <see cref="ReadRows"/> rows within one millisecond.</exception>
|
|
||||||
/// <param name="valueType">The node's scalar value type (the hinted <c>IHistorianDataSource.ReadAtTimeAsync</c>), or null.</param>
|
/// <param name="valueType">The node's scalar value type (the hinted <c>IHistorianDataSource.ReadAtTimeAsync</c>), or null.</param>
|
||||||
|
/// <param name="maxReads">The request's raw-read cap; null = <see cref="BaseReads"/> + <see cref="ReadsPerTime"/> per distinct time.</param>
|
||||||
|
/// <returns>One value per requested time, stamped with it.</returns>
|
||||||
|
/// <exception cref="HistoryReadLimitExceededException">More than <see cref="ReadRows"/> rows within one millisecond, the
|
||||||
|
/// read cap, or a strict gateway ceiling below <see cref="MinBudget"/>.</exception>
|
||||||
public static async Task<IReadOnlyList<DataValueSnapshot>> ReadAsync(IHistorianGatewayClient client, GatewayRowCeiling ceiling,
|
public static async Task<IReadOnlyList<DataValueSnapshot>> ReadAsync(IHistorianGatewayClient client, GatewayRowCeiling ceiling,
|
||||||
string tag, IReadOnlyList<DateTime> timestampsUtc, DateTime now, CancellationToken ct, TypeCode? valueType = null)
|
string tag, IReadOnlyList<DateTime> timestampsUtc, DateTime now, CancellationToken ct, TypeCode? valueType = null,
|
||||||
|
int? maxReads = null)
|
||||||
{
|
{
|
||||||
var requested = timestampsUtc.Select(t => DateTime.SpecifyKind(t, DateTimeKind.Utc)).ToArray();
|
var requested = timestampsUtc.Select(t => DateTime.SpecifyKind(t, DateTimeKind.Utc)).ToArray();
|
||||||
if (requested.Length == 0) return [];
|
if (requested.Length == 0) return [];
|
||||||
var neighbours = await ResolveAsync(new Ctx(client, ceiling, tag, ct, valueType), requested.Distinct().Order().ToArray(), now)
|
var times = requested.Distinct().Order().ToArray();
|
||||||
.ConfigureAwait(false);
|
var ctx = new Ctx(client, ceiling, tag, ct, valueType, maxReads ?? BaseReads + ReadsPerTime * times.Length);
|
||||||
|
var neighbours = await ResolveAsync(ctx, times, now).ConfigureAwait(false);
|
||||||
var result = new DataValueSnapshot[requested.Length];
|
var result = new DataValueSnapshot[requested.Length];
|
||||||
for (var i = 0; i < requested.Length; i++)
|
for (var i = 0; i < requested.Length; i++)
|
||||||
{
|
{
|
||||||
var t = requested[i];
|
var t = requested[i];
|
||||||
var (prev, next) = neighbours[t.Ticks];
|
var (prev, next) = neighbours[t.Ticks];
|
||||||
|
// A DateTime node is stepped (as on the SQL backends): a usable non-numeric next sends the evaluator down its
|
||||||
|
// stepped branch, with the next's own status.
|
||||||
|
if (valueType == TypeCode.DateTime && next is { IsUsable: true } n)
|
||||||
|
next = n with { Value = HistorianValue.FromString(string.Empty) };
|
||||||
result[i] = ToNodeType(AtTimeEvaluator.Evaluate(t, prev, next, now, now), valueType);
|
result[i] = ToNodeType(AtTimeEvaluator.Evaluate(t, prev, next, now, now), valueType);
|
||||||
}
|
}
|
||||||
return result;
|
return result;
|
||||||
}
|
}
|
||||||
|
|
||||||
private sealed record Ctx(IHistorianGatewayClient Client, GatewayRowCeiling Ceiling, string Tag, CancellationToken Ct,
|
/// <summary>One request's read context: the client, the type hint, and the read count against its cap.</summary>
|
||||||
TypeCode? ValueType);
|
private sealed class Ctx(IHistorianGatewayClient client, GatewayRowCeiling ceiling, string tag, CancellationToken ct,
|
||||||
|
TypeCode? valueType, int maxReads)
|
||||||
|
{
|
||||||
|
public IHistorianGatewayClient Client { get; } = client;
|
||||||
|
public GatewayRowCeiling Ceiling { get; } = ceiling;
|
||||||
|
public string Tag { get; } = tag;
|
||||||
|
public CancellationToken Ct { get; } = ct;
|
||||||
|
public TypeCode? ValueType { get; } = valueType;
|
||||||
|
public int MaxReads { get; } = maxReads;
|
||||||
|
public int Reads { get; set; }
|
||||||
|
|
||||||
|
/// <summary>The smallest budget a read was served with (a strict gateway's ceiling shows here).</summary>
|
||||||
|
public int SmallestBudget { get; set; } = int.MaxValue;
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>True for the integer type codes (evaluated as Int64, returned as the node's own type).</summary>
|
/// <summary>True for the integer type codes (evaluated as Int64, returned as the node's own type).</summary>
|
||||||
private static bool IsInteger(TypeCode? t) =>
|
private static bool IsInteger(TypeCode? t) =>
|
||||||
@@ -145,21 +187,22 @@ internal static class GatewayAtTimeReader
|
|||||||
for (; i < times.Length && times[i] <= Floor; i++) result[times[i].Ticks] = (null, null);
|
for (; i < times.Length && times[i] <= Floor; i++) result[times[i].Ticks] = (null, null);
|
||||||
if (i == times.Length) return result;
|
if (i == times.Length) return result;
|
||||||
|
|
||||||
// Reads are planned on times clamped to the horizon (nothing is stored beyond it).
|
// Reads are planned on times clamped to the horizon (nothing is stored beyond it, and no read ends past it).
|
||||||
var qe = MsFloor(Min(times[^1], horizon)) + TimeSpan.FromMilliseconds(1);
|
var qe = Min(MsFloor(Min(times[^1], horizon)) + OneMs, horizon);
|
||||||
var qs = RequestStart(Min(times[i], horizon));
|
var qs = RequestStart(Min(times[i], horizon));
|
||||||
|
var budget = ReadRows;
|
||||||
StoredSample? before = null; // the latest stored sample at or before qs …
|
StoredSample? before = null; // the latest stored sample at or before qs …
|
||||||
var beforeKnown = false; // … once known
|
var beforeKnown = false; // … once known
|
||||||
DateTime? walkKnown = null; // a stored sample exists here (at or before qs): the walk-back never goes below it
|
DateTime? walkKnown = null; // a stored sample exists here (at or before qs): the walk-back bisects above it
|
||||||
var walkWidth = InitialWalkWidth;
|
var walkWidth = InitialWalkWidth;
|
||||||
var probeFrom = -1; // first index whose next lies at or after qe (complete last read)
|
var probeFrom = -1; // first index whose next lies at or after qe (complete last read)
|
||||||
|
|
||||||
while (true)
|
while (true)
|
||||||
{
|
{
|
||||||
var read = await ReadTrustedAsync(ctx, qs, qe, ReadRows).ConfigureAwait(false);
|
var read = await ReadTrustedAsync(ctx, qs, qe, budget).ConfigureAwait(false);
|
||||||
var rows = read.Rows;
|
var rows = read.Rows;
|
||||||
if (!read.Complete && rows.Count == 0)
|
if (!read.Complete && rows.Count == 0)
|
||||||
throw TooDense(ctx.Tag, qs);
|
throw TooDense(ctx, qs);
|
||||||
if (rows.Count >= 2 && !beforeKnown && walkKnown is null)
|
if (rows.Count >= 2 && !beforeKnown && walkKnown is null)
|
||||||
walkWidth = Clamp(TimeSpan.FromTicks(4 * (rows[^1].TimeUtc - rows[0].TimeUtc).Ticks / (rows.Count - 1)));
|
walkWidth = Clamp(TimeSpan.FromTicks(4 * (rows[^1].TimeUtc - rows[0].TimeUtc).Ticks / (rows.Count - 1)));
|
||||||
|
|
||||||
@@ -197,15 +240,17 @@ internal static class GatewayAtTimeReader
|
|||||||
|
|
||||||
if (k == times.Length) break;
|
if (k == times.Length) break;
|
||||||
|
|
||||||
// Truncated before times[k]: continue from the last row, or jump to times[k] with a density-sized margin.
|
// Truncated before times[k]: continue from the last row, or jump to times[k] with a density-sized margin and a
|
||||||
|
// small budget (it needs that time's neighbours, and whatever later times follow closely).
|
||||||
var tn = Min(times[k], horizon);
|
var tn = Min(times[k], horizon);
|
||||||
var spanTicks = Math.Max(1, (x - rows[0].TimeUtc).Ticks);
|
var spanTicks = Math.Max(1, (x - rows[0].TimeUtc).Ticks);
|
||||||
var density = (double)rows.Count / spanTicks; // rows per tick
|
var density = (double)rows.Count / spanTicks; // rows per tick
|
||||||
var gapRows = density * (tn - x).Ticks;
|
var gapRows = density * (tn - x).Ticks;
|
||||||
var margin = Clamp(TimeSpan.FromTicks((long)(ReadRows / 4 / Math.Max(density, 1e-18))));
|
var margin = Clamp(TimeSpan.FromTicks((long)Math.Min(TimeSpan.MaxValue.Ticks / 2.0, JumpRows / 4.0 / Math.Max(density, 1e-18))));
|
||||||
var resume = MsFloor(x) - TimeSpan.FromMilliseconds(1);
|
var resume = MsFloor(x) - OneMs;
|
||||||
var nextQs = gapRows < ReadRows / 2.0 ? resume : Max(RequestStart(tn - Min(margin, tn - Floor)), resume);
|
var jump = gapRows >= ReadRows / 2.0;
|
||||||
if (nextQs <= qs) throw TooDense(ctx.Tag, x);
|
var nextQs = jump ? Max(RequestStart(tn - Min(margin, tn - Floor)), resume) : resume;
|
||||||
|
if (nextQs <= qs) throw TooDense(ctx, x);
|
||||||
|
|
||||||
if (nextQs < x)
|
if (nextQs < x)
|
||||||
{
|
{
|
||||||
@@ -225,13 +270,14 @@ internal static class GatewayAtTimeReader
|
|||||||
walkKnown = x;
|
walkKnown = x;
|
||||||
walkWidth = margin;
|
walkWidth = margin;
|
||||||
}
|
}
|
||||||
|
budget = jump ? JumpRows : ReadRows;
|
||||||
qs = nextQs;
|
qs = nextQs;
|
||||||
i = k;
|
i = k;
|
||||||
}
|
}
|
||||||
|
|
||||||
if (probeFrom >= 0 && qe - TimeSpan.FromMilliseconds(1) < horizon)
|
if (probeFrom >= 0 && qe < horizon)
|
||||||
{
|
{
|
||||||
var after = await FirstAfterAsync(ctx, qe - TimeSpan.FromMilliseconds(1), horizon, Min(times[^1], horizon)).ConfigureAwait(false);
|
var after = await FirstAfterAsync(ctx, qe - OneMs, horizon, Min(times[^1], horizon)).ConfigureAwait(false);
|
||||||
if (after is not null)
|
if (after is not null)
|
||||||
for (var k = probeFrom; k < times.Length; k++)
|
for (var k = probeFrom; k < times.Length; k++)
|
||||||
{
|
{
|
||||||
@@ -247,70 +293,92 @@ internal static class GatewayAtTimeReader
|
|||||||
/// <summary>The evaluator reads <c>next</c> only for a usable <c>prev</c> that is not an exact hit.</summary>
|
/// <summary>The evaluator reads <c>next</c> only for a usable <c>prev</c> that is not an exact hit.</summary>
|
||||||
private static bool NeedsNext(DateTime t, StoredSample? prev) => prev is { IsUsable: true } p && p.TimeUtc != t;
|
private static bool NeedsNext(DateTime t, StoredSample? prev) => prev is { IsUsable: true } p && p.TimeUtc != t;
|
||||||
|
|
||||||
|
/// <summary>One walk-back segment <c>(lo, hi]</c>: its oldest and newest rows, and whether the read was complete.</summary>
|
||||||
|
private static async Task<(StoredSample? First, StoredSample? Last, bool Complete)> SegmentAsync(
|
||||||
|
Ctx ctx, DateTime lo, DateTime hi)
|
||||||
|
{
|
||||||
|
var read = await ReadTrustedAsync(ctx, MsFloor(lo), MsFloor(hi) + OneMs, ReadRows).ConfigureAwait(false);
|
||||||
|
StoredSample? first = null, last = null;
|
||||||
|
foreach (var r in read.Rows)
|
||||||
|
if (r.TimeUtc > lo && r.TimeUtc <= hi)
|
||||||
|
{
|
||||||
|
first ??= r;
|
||||||
|
last = r;
|
||||||
|
}
|
||||||
|
return (first, last, read.Complete);
|
||||||
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// The latest stored sample with time in <c>(Floor, hi]</c>, walking back in forward-read segments (see the type
|
/// The latest stored sample with time in <c>(Floor, hi]</c> (see the type remarks: gallop, then bisect).
|
||||||
/// remarks). <paramref name="known"/>: a stored sample exists at that time (≤ <paramref name="hi"/>).
|
/// <paramref name="known"/>: a stored sample exists at that time (≤ <paramref name="hi"/>) — the walk then bisects
|
||||||
|
/// above it straight away, its first segment aimed at <paramref name="width"/>.
|
||||||
/// </summary>
|
/// </summary>
|
||||||
private static async Task<StoredSample?> WalkBackAsync(Ctx ctx, DateTime hi, DateTime? known, TimeSpan width)
|
private static async Task<StoredSample?> WalkBackAsync(Ctx ctx, DateTime hi, DateTime? known, TimeSpan width)
|
||||||
{
|
{
|
||||||
|
TimeSpan? aim = known is null ? null : width;
|
||||||
|
|
||||||
|
// Gallop: no stored row known below hi yet.
|
||||||
var empties = 0;
|
var empties = 0;
|
||||||
var triedRest = false;
|
while (known is null)
|
||||||
for (var reads = 0; reads < MaxWalkReads; reads++)
|
|
||||||
{
|
{
|
||||||
if (hi <= Floor) return null;
|
if (hi <= Floor) return null;
|
||||||
var rest = known is null && !triedRest && empties >= 2;
|
var rest = empties >= 2;
|
||||||
DateTime lo;
|
var lo = rest || (hi - Floor).Ticks <= width.Ticks ? Floor : hi - width;
|
||||||
if (rest)
|
var seg = await SegmentAsync(ctx, lo, hi).ConfigureAwait(false);
|
||||||
|
if (seg.Complete)
|
||||||
{
|
{
|
||||||
lo = Floor;
|
if (seg.Last is { } found) return found;
|
||||||
triedRest = true;
|
|
||||||
}
|
|
||||||
else
|
|
||||||
{
|
|
||||||
lo = (hi - Floor).Ticks > width.Ticks ? hi - width : Floor;
|
|
||||||
if (known is { } kn && lo < kn) lo = kn.AddTicks(-1); // never below a row known to exist (it is the floor)
|
|
||||||
}
|
|
||||||
|
|
||||||
var read = await ReadTrustedAsync(ctx, MsFloor(lo), MsFloor(hi) + TimeSpan.FromMilliseconds(1), ReadRows)
|
|
||||||
.ConfigureAwait(false);
|
|
||||||
StoredSample? first = null, last = null;
|
|
||||||
DateTime? lastAny = null;
|
|
||||||
foreach (var r in read.Rows)
|
|
||||||
{
|
|
||||||
if (r.TimeUtc > lo && r.TimeUtc <= hi)
|
|
||||||
{
|
|
||||||
first ??= r;
|
|
||||||
last = r;
|
|
||||||
}
|
|
||||||
lastAny = r.TimeUtc;
|
|
||||||
}
|
|
||||||
|
|
||||||
if (read.Complete)
|
|
||||||
{
|
|
||||||
if (last is not null) return last;
|
|
||||||
if (lo <= Floor) return null;
|
if (lo <= Floor) return null;
|
||||||
hi = lo;
|
hi = lo;
|
||||||
empties++;
|
empties++;
|
||||||
width = Grow(width);
|
width = Grow(width);
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
|
// Full — a whole-remainder read of deep history too: its last row is a stored row below hi. Bisect above it.
|
||||||
// Truncated. A whole-remainder read of deep history proves nothing: carry on with segments.
|
if (seg.Last is not { } seen) throw TooDense(ctx, lo);
|
||||||
if (rest) continue;
|
|
||||||
if (last is not { } seen || (known is { } k0 && seen.TimeUtc <= k0))
|
|
||||||
throw TooDense(ctx.Tag, lastAny ?? lo);
|
|
||||||
var spanTicks = Math.Max(1, (seen.TimeUtc - first!.Value.TimeUtc).Ticks); // the density of the rows seen
|
|
||||||
known = seen.TimeUtc;
|
known = seen.TimeUtc;
|
||||||
// Aim at about a quarter of a read from the density just seen, and never wider than half the open stretch.
|
aim = AimFrom(seg.First!.Value.TimeUtc, seen.TimeUtc);
|
||||||
var aimed = TimeSpan.FromTicks(Math.Max(Ms, spanTicks / 4));
|
}
|
||||||
var half = TimeSpan.FromTicks(Math.Max(Ms, (hi - seen.TimeUtc).Ticks / 2));
|
|
||||||
width = aimed < half ? aimed : half;
|
// Bisect: a stored row exists at `known`, nothing newer than `hi` matters; the answer is the newest row in [known, hi].
|
||||||
empties = 0;
|
var aimMissed = false;
|
||||||
|
while (true)
|
||||||
|
{
|
||||||
|
var k = known.Value;
|
||||||
|
DateTime lo;
|
||||||
|
var aimed = false;
|
||||||
|
if (hi - k < TimeSpan.FromMilliseconds(2))
|
||||||
|
lo = k.AddTicks(-1); // the whole stretch from the known row: never a sub-millisecond step
|
||||||
|
else
|
||||||
|
{
|
||||||
|
lo = k + TimeSpan.FromTicks((hi - k).Ticks / 2);
|
||||||
|
if (aim is { } a && hi - a > lo)
|
||||||
|
{
|
||||||
|
lo = hi - a; // an aimed try right after a full read — until one comes back empty (a gap: halve from then on)
|
||||||
|
aimed = true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
aim = null;
|
||||||
|
|
||||||
|
var seg = await SegmentAsync(ctx, lo, hi).ConfigureAwait(false);
|
||||||
|
if (seg.Complete)
|
||||||
|
{
|
||||||
|
if (seg.Last is { } found) return found;
|
||||||
|
if (lo < k) return null; // the known row is gone (deleted meanwhile): nothing is left to find
|
||||||
|
hi = lo;
|
||||||
|
aimMissed |= aimed;
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (seg.Last is not { } seen || seen.TimeUtc <= k) throw TooDense(ctx, k);
|
||||||
|
known = seen.TimeUtc;
|
||||||
|
if (!aimMissed) aim = AimFrom(seg.First!.Value.TimeUtc, seen.TimeUtc);
|
||||||
}
|
}
|
||||||
throw new HistoryReadLimitExceededException(MaxWalkReads,
|
|
||||||
$"The AtTime read of '{ctx.Tag}' could not find the stored sample before a requested time within {MaxWalkReads} raw reads.");
|
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/// <summary>About a quarter of a read at the density a full read showed (its rows from <paramref name="first"/> to <paramref name="last"/>).</summary>
|
||||||
|
private static TimeSpan AimFrom(DateTime first, DateTime last) =>
|
||||||
|
TimeSpan.FromTicks(Math.Max(Ms, (last - first).Ticks / 4));
|
||||||
|
|
||||||
/// <summary>The first stored sample after <paramref name="after"/> in <c>(qs, end)</c>, from a small forward read.</summary>
|
/// <summary>The first stored sample after <paramref name="after"/> in <c>(qs, end)</c>, from a small forward read.</summary>
|
||||||
private static async Task<StoredSample?> FirstAfterAsync(Ctx ctx, DateTime qs, DateTime end, DateTime after)
|
private static async Task<StoredSample?> FirstAfterAsync(Ctx ctx, DateTime qs, DateTime end, DateTime after)
|
||||||
{
|
{
|
||||||
@@ -321,21 +389,35 @@ internal static class GatewayAtTimeReader
|
|||||||
if (r.TimeUtc > after) return r;
|
if (r.TimeUtc > after) return r;
|
||||||
if (read.Complete) return null;
|
if (read.Complete) return null;
|
||||||
}
|
}
|
||||||
throw TooDense(ctx.Tag, qs);
|
throw TooDense(ctx, qs);
|
||||||
}
|
}
|
||||||
|
|
||||||
/// <summary>
|
/// <summary>
|
||||||
/// One forward read <c>[qs, qe]</c> (whole milliseconds), drained, keeping only rows strictly inside it and after the
|
/// One forward read <c>[qs, qe]</c> (whole milliseconds), drained, keeping only rows strictly inside it and after the
|
||||||
/// epoch. A strict gateway's ceiling is learned on refusal and then asked for directly; a full read is not re-probed
|
/// epoch — counted against the request's read cap. A strict gateway's ceiling is learned on refusal and then asked for
|
||||||
/// (the caller pages on, it does not need the rest of this read).
|
/// directly; a full read is not re-probed (the caller pages on, it does not need the rest of this read).
|
||||||
/// </summary>
|
/// </summary>
|
||||||
private static async Task<Read> ReadTrustedAsync(Ctx ctx, DateTime qs, DateTime qe, int budget)
|
private static async Task<Read> ReadTrustedAsync(Ctx ctx, DateTime qs, DateTime qe, int budget)
|
||||||
{
|
{
|
||||||
var (read, _) = await ctx.Ceiling.ReadAsync(budget, b => PassAsync(ctx, qs, qe, b), (_, _) => false)
|
if (++ctx.Reads > ctx.MaxReads)
|
||||||
|
throw new HistoryReadLimitExceededException(ctx.MaxReads,
|
||||||
|
$"The AtTime read of '{ctx.Tag}' needed more than {ctx.MaxReads} raw reads ({BaseReads} + {ReadsPerTime} per " +
|
||||||
|
"requested time)" + CeilingNote(ctx) + "; the Gateway backend serves AtTime from raw reads — request fewer times.");
|
||||||
|
var (read, used) = await ctx.Ceiling.ReadAsync(budget, b => PassAsync(ctx, qs, qe, b), (_, _) => false)
|
||||||
.ConfigureAwait(false);
|
.ConfigureAwait(false);
|
||||||
|
if (used < ctx.SmallestBudget) ctx.SmallestBudget = used;
|
||||||
|
if (used < MinBudget)
|
||||||
|
throw new HistoryReadLimitExceededException(used,
|
||||||
|
$"The HistorianGateway's row ceiling ({used}, its Historian:ReadLimits:MaxValuesCeiling with " +
|
||||||
|
$"AllowStreamingBeyondCeiling=false) is below the {MinBudget} rows an AtTime raw read of '{ctx.Tag}' needs.");
|
||||||
return read;
|
return read;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
private static string CeilingNote(Ctx ctx) =>
|
||||||
|
ctx.SmallestBudget < JumpRows
|
||||||
|
? $", with reads bounded by the HistorianGateway's row ceiling of {ctx.SmallestBudget} (Historian:ReadLimits:MaxValuesCeiling)"
|
||||||
|
: string.Empty;
|
||||||
|
|
||||||
private static async Task<Read> PassAsync(Ctx ctx, DateTime qs, DateTime qe, int budget)
|
private static async Task<Read> PassAsync(Ctx ctx, DateTime qs, DateTime qe, int budget)
|
||||||
{
|
{
|
||||||
var rows = new List<StoredSample>();
|
var rows = new List<StoredSample>();
|
||||||
@@ -369,9 +451,11 @@ internal static class GatewayAtTimeReader
|
|||||||
{
|
{
|
||||||
TypeCode.Boolean => HistorianValue.FromBoolean(v != 0),
|
TypeCode.Boolean => HistorianValue.FromBoolean(v != 0),
|
||||||
TypeCode.String => HistorianValue.FromString(v.ToString("R", System.Globalization.CultureInfo.InvariantCulture)),
|
TypeCode.String => HistorianValue.FromString(v.ToString("R", System.Globalization.CultureInfo.InvariantCulture)),
|
||||||
// In Int64 range only (2^63 is exactly representable; anything at or beyond it stays a Double).
|
// In Int64 range only (2^63 is exactly representable); NaN, ±∞ or out of range is no integer at all — Empty, which
|
||||||
_ when IsInteger(valueType) && v >= -9.2233720368547758E18 && v < 9.2233720368547758E18 =>
|
// StoredSample.FromStored makes BadNoData (unusable), never a saturated Good value.
|
||||||
HistorianValue.FromInt64((long)Math.Round(v, MidpointRounding.AwayFromZero)),
|
_ when IsInteger(valueType) => v >= -9.2233720368547758E18 && v < 9.2233720368547758E18
|
||||||
|
? HistorianValue.FromInt64((long)Math.Round(v, MidpointRounding.AwayFromZero))
|
||||||
|
: HistorianValue.Empty,
|
||||||
_ => HistorianValue.FromDouble(v),
|
_ => HistorianValue.FromDouble(v),
|
||||||
};
|
};
|
||||||
|
|
||||||
@@ -395,8 +479,12 @@ internal static class GatewayAtTimeReader
|
|||||||
|
|
||||||
private static TimeSpan Grow(TimeSpan w) => w.Ticks > TimeSpan.MaxValue.Ticks / 4 ? TimeSpan.MaxValue : w * 4;
|
private static TimeSpan Grow(TimeSpan w) => w.Ticks > TimeSpan.MaxValue.Ticks / 4 ? TimeSpan.MaxValue : w * 4;
|
||||||
|
|
||||||
private static HistoryReadLimitExceededException TooDense(string tag, DateTime at) =>
|
private static HistoryReadLimitExceededException TooDense(Ctx ctx, DateTime at)
|
||||||
new(ReadRows,
|
{
|
||||||
$"The AtTime read of '{tag}' met more than {ReadRows} stored samples within about one millisecond of " +
|
var rows = Math.Min(ReadRows, ctx.SmallestBudget);
|
||||||
$"{at:O}; the Gateway backend serves AtTime from raw reads.");
|
return new(rows,
|
||||||
|
$"The AtTime read of '{ctx.Tag}' met more than {rows} stored samples within about one millisecond of {at:O}" +
|
||||||
|
(rows < ReadRows ? $" (reads are bounded by the HistorianGateway's row ceiling of {rows})" : string.Empty) +
|
||||||
|
"; the Gateway backend serves AtTime from raw reads.");
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -139,6 +139,7 @@ public abstract class AtTimeConformance : ConformanceBase
|
|||||||
(At(10), HistorianValue.FromInt64(15), Good));
|
(At(10), HistorianValue.FromInt64(15), Good));
|
||||||
var s = (await B.CreateDataSource().ReadAtTimeAsync("I", [At(5), At(2)], Ct)).Samples;
|
var s = (await B.CreateDataSource().ReadAtTimeAsync("I", [At(5), At(2)], Ct)).Samples;
|
||||||
s[0].Value.ShouldBe(13L); // 12.5 ⇒ 13 (away from zero)
|
s[0].Value.ShouldBe(13L); // 12.5 ⇒ 13 (away from zero)
|
||||||
|
s[0].Value.ShouldBeOfType<long>(); // ShouldBe compares numbers across types — the type needs its own check (#693)
|
||||||
s[1].Value.ShouldBe(11L); // 11.0
|
s[1].Value.ShouldBe(11L); // 11.0
|
||||||
s[0].StatusCode.ShouldBe(GoodInterpolated);
|
s[0].StatusCode.ShouldBe(GoodInterpolated);
|
||||||
}
|
}
|
||||||
|
|||||||
+181
-1
@@ -409,7 +409,7 @@ public sealed class GatewayAtTimeTests
|
|||||||
var s = (await AtTime(fake, T0.AddMinutes(-5))).Single();
|
var s = (await AtTime(fake, T0.AddMinutes(-5))).Single();
|
||||||
s.Value.ShouldBe(599_999d);
|
s.Value.ShouldBe(599_999d);
|
||||||
s.StatusCode.ShouldBe(GoodInterpolated); // stepped from the last sample (before now)
|
s.StatusCode.ShouldBe(GoodInterpolated); // stepped from the last sample (before now)
|
||||||
fake.ReadRawCallCount.ShouldBe(7); // window, 1m, 4m, remainder (truncated), 16m (truncated), 256 ms, probe
|
fake.ReadRawCallCount.ShouldBe(6); // window, 1m, 4m, remainder (full: seeds the bisection), one aimed segment, probe
|
||||||
store.RowsStreamed.ShouldBeLessThan(10_000);
|
store.RowsStreamed.ShouldBeLessThan(10_000);
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -446,6 +446,186 @@ public sealed class GatewayAtTimeTests
|
|||||||
ds.GetHealthSnapshot().TotalFailures.ShouldBe(1);
|
ds.GetHealthSnapshot().TotalFailures.ShouldBe(1);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ---- long histories (review H1/M1/L1): a generated store, too large to materialise -----------------------------
|
||||||
|
|
||||||
|
/// <summary>
|
||||||
|
/// A regular series (value = sample index) over <c>[from, to]</c> every <paramref name="step"/>, optionally with a
|
||||||
|
/// hole <c>(gapFrom, gapTo)</c>, served in the AVEVA wire shape (boundary row at the query start, ms bounds only,
|
||||||
|
/// <c>max_values > 0</c>, <c>start <= end</c>, nothing past now + 2 days).
|
||||||
|
/// </summary>
|
||||||
|
private sealed class SeriesStore(DateTime from, DateTime to, TimeSpan step, DateTime? gapFrom = null, DateTime? gapTo = null)
|
||||||
|
{
|
||||||
|
public long RowsStreamed;
|
||||||
|
|
||||||
|
private bool InGap(DateTime t) => gapFrom is { } a && gapTo is { } b && t > a && t < b;
|
||||||
|
|
||||||
|
private DateTime? FirstAtOrAfter(DateTime t)
|
||||||
|
{
|
||||||
|
if (t > to) return null;
|
||||||
|
var k = t <= from ? 0 : ((t - from).Ticks + step.Ticks - 1) / step.Ticks;
|
||||||
|
var r = from.AddTicks(k * step.Ticks);
|
||||||
|
if (InGap(r)) return FirstAtOrAfter(gapTo!.Value);
|
||||||
|
return r <= to ? r : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
private DateTime? LastBefore(DateTime t)
|
||||||
|
{
|
||||||
|
if (t <= from) return null;
|
||||||
|
var k = ((Min(t, to.AddTicks(1)) - from).Ticks - 1) / step.Ticks;
|
||||||
|
var r = from.AddTicks(k * step.Ticks);
|
||||||
|
return InGap(r) ? LastBefore(gapFrom!.Value.AddTicks(1)) : r;
|
||||||
|
}
|
||||||
|
|
||||||
|
private static DateTime Min(DateTime a, DateTime b) => a <= b ? a : b;
|
||||||
|
|
||||||
|
/// <summary>The stored row at <paramref name="t"/>, stamped <paramref name="at"/> (the boundary row) when given.</summary>
|
||||||
|
private HistorianSample Row(DateTime t, uint quality = 0, DateTime? at = null) => new()
|
||||||
|
{
|
||||||
|
Tag = "T", Timestamp = Timestamp.FromDateTime(at ?? t), NumericValue = (t - from).Ticks / step.Ticks, OpcQuality = 192,
|
||||||
|
Quality = quality,
|
||||||
|
};
|
||||||
|
|
||||||
|
public FakeHistorianGatewayClient Client(int? ceiling = null) => new() { RawSource = Serve, MaxValuesCeiling = ceiling };
|
||||||
|
|
||||||
|
private IEnumerable<HistorianSample> Serve(DateTime start, DateTime end, int max)
|
||||||
|
{
|
||||||
|
if (max <= 0 || end < start || end > Now.AddDays(2)
|
||||||
|
|| start.Ticks % TimeSpan.TicksPerMillisecond != 0 || end.Ticks % TimeSpan.TicksPerMillisecond != 0)
|
||||||
|
throw new Grpc.Core.RpcException(new Grpc.Core.Status(Grpc.Core.StatusCode.InvalidArgument, "refused"));
|
||||||
|
var rows = new List<HistorianSample>();
|
||||||
|
if (FirstAtOrAfter(start) != start)
|
||||||
|
rows.Add(LastBefore(start) is { } p
|
||||||
|
? Row(p, 133, start)
|
||||||
|
: new HistorianSample { Tag = "T", Timestamp = Timestamp.FromDateTime(start), OpcQuality = 0, Quality = 1, NumericValue = 0 });
|
||||||
|
for (var t = FirstAtOrAfter(start); t is { } u && u <= end && rows.Count < max; t = FirstAtOrAfter(u.AddTicks(1)))
|
||||||
|
rows.Add(Row(u));
|
||||||
|
RowsStreamed += rows.Count;
|
||||||
|
return rows;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
[Theory] // H1: a dense tag that stopped reporting — the walk crosses the silence by bisection, not by regrowing segments
|
||||||
|
[InlineData(1.0, 30)]
|
||||||
|
[InlineData(1.0, 365)]
|
||||||
|
[InlineData(10.0, 30)]
|
||||||
|
[InlineData(10.0, 365)]
|
||||||
|
public async Task A_dense_tag_that_stopped_long_ago_is_found_in_a_bounded_number_of_reads(double hz, int stoppedDaysAgo)
|
||||||
|
{
|
||||||
|
var step = TimeSpan.FromTicks((long)(TimeSpan.TicksPerSecond / hz));
|
||||||
|
var end = Now.AddDays(-stoppedDaysAgo);
|
||||||
|
var fake = new SeriesStore(end.AddYears(-5), end, step).Client();
|
||||||
|
var s = (await AtTime(fake, Now.AddMinutes(-1))).Single();
|
||||||
|
s.Value.ShouldBe((double)((end - end.AddYears(-5)).Ticks / step.Ticks)); // the last sample, held
|
||||||
|
s.StatusCode.ShouldBe(GoodInterpolated);
|
||||||
|
fake.ReadRawCallCount.ShouldBeLessThan(40);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact] // H1: a time inside a month-long hole in the middle of a dense history
|
||||||
|
public async Task A_time_inside_a_mid_history_gap_interpolates_across_it()
|
||||||
|
{
|
||||||
|
var from = Now.AddYears(-2);
|
||||||
|
var gapFrom = Now.AddDays(-200);
|
||||||
|
var gapTo = gapFrom.AddDays(30);
|
||||||
|
var fake = new SeriesStore(from, Now, TimeSpan.FromSeconds(1), gapFrom, gapTo).Client();
|
||||||
|
var t = gapFrom.AddDays(15);
|
||||||
|
var s = (await AtTime(fake, t)).Single();
|
||||||
|
var before = (gapFrom - from).Ticks / TimeSpan.TicksPerSecond;
|
||||||
|
var after = (gapTo - from).Ticks / TimeSpan.TicksPerSecond;
|
||||||
|
((double)s.Value!).ShouldBe(before + (after - before) * 0.5, 1e-6);
|
||||||
|
s.StatusCode.ShouldBe(GoodInterpolated);
|
||||||
|
fake.ReadRawCallCount.ShouldBeLessThan(40);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact] // H1: a jump to a time in a gap — the walk starts from the earlier read's last row, far below
|
||||||
|
public async Task A_jump_into_a_gap_bisects_up_from_the_earlier_reads_last_row()
|
||||||
|
{
|
||||||
|
var from = Now.AddDays(-30);
|
||||||
|
var gapFrom = from.AddDays(2);
|
||||||
|
var gapTo = gapFrom.AddDays(10);
|
||||||
|
var fake = new SeriesStore(from, Now, TimeSpan.FromSeconds(1), gapFrom, gapTo).Client();
|
||||||
|
var t0 = from.AddSeconds(10.5);
|
||||||
|
var t1 = gapFrom.AddDays(5);
|
||||||
|
var s = await AtTime(fake, t0, t1);
|
||||||
|
s[0].Value.ShouldBe(10.5d);
|
||||||
|
var before = (gapFrom - from).Ticks / TimeSpan.TicksPerSecond;
|
||||||
|
var after = (gapTo - from).Ticks / TimeSpan.TicksPerSecond;
|
||||||
|
((double)s[1].Value!).ShouldBe(before + (after - before) * 0.5, 1e-6);
|
||||||
|
fake.ReadRawCallCount.ShouldBeLessThan(40);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact] // M1: widely spread times cost one SMALL read each, not one 1,024-row read each
|
||||||
|
public async Task A_thousand_times_over_ten_years_cost_one_small_read_each()
|
||||||
|
{
|
||||||
|
var store = new SeriesStore(Now.AddYears(-10), Now, TimeSpan.FromSeconds(1));
|
||||||
|
var fake = store.Client();
|
||||||
|
var rnd = new Random(1);
|
||||||
|
var times = Enumerable.Range(0, 1000).Select(_ => Now.AddYears(-9).AddTicks((long)(rnd.NextDouble() * TimeSpan.TicksPerDay * 3000))).ToArray();
|
||||||
|
var s = await AtTime(fake, times);
|
||||||
|
s.ShouldAllBe(x => x.StatusCode == GoodInterpolated);
|
||||||
|
fake.ReadRawCallCount.ShouldBeLessThanOrEqualTo(1010);
|
||||||
|
store.RowsStreamed.ShouldBeLessThan(80_000); // ~64 rows a read (was ~1,000,000)
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact] // L1: a strict-gateway ceiling too small to work with is refused with a message naming it
|
||||||
|
public async Task A_ceiling_below_a_usable_read_is_refused_naming_the_ceiling()
|
||||||
|
{
|
||||||
|
var fake = new SeriesStore(Now.AddDays(-1), Now, TimeSpan.FromSeconds(1)).Client(ceiling: 2);
|
||||||
|
var ex = await Should.ThrowAsync<HistoryReadLimitExceededException>(
|
||||||
|
() => Source(fake).ReadAtTimeAsync("T", [Now.AddHours(-1)], Ct));
|
||||||
|
ex.Message.ShouldContain("row ceiling (2");
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact] // L1: a small ceiling is served (paging), within the read cap
|
||||||
|
public async Task A_small_ceiling_is_served_within_the_read_cap()
|
||||||
|
{
|
||||||
|
var fake = new SeriesStore(Now.AddYears(-1), Now, TimeSpan.FromSeconds(1)).Client(ceiling: 10);
|
||||||
|
var times = Enumerable.Range(0, 50).Select(i => Now.AddDays(-300 + i * 5).AddSeconds(0.5)).ToArray();
|
||||||
|
var s = await AtTime(fake, times);
|
||||||
|
s.ShouldAllBe(x => x.StatusCode == GoodInterpolated);
|
||||||
|
fake.ReadRawCallCount.ShouldBeLessThanOrEqualTo(GatewayAtTimeReader.BaseReads + GatewayAtTimeReader.ReadsPerTime * 50);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact] // L1: the request's read cap stops a walk with a clear error, never a hang (here a cap of 3 vs. a 5-read walk)
|
||||||
|
public async Task The_read_cap_stops_a_request_with_a_clear_error()
|
||||||
|
{
|
||||||
|
var fake = new AvevaStore([(T0.AddDays(-400), 42, false), (T0.AddDays(-400).AddSeconds(10), 43, false)]).Client();
|
||||||
|
var ex = await Should.ThrowAsync<HistoryReadLimitExceededException>(() => GatewayAtTimeReader.ReadAsync(
|
||||||
|
fake, new GatewayRowCeiling(), "T", [T0], Now, Ct, maxReads: 3));
|
||||||
|
ex.Message.ShouldContain("needed more than 3 raw reads");
|
||||||
|
fake.ReadRawCallCount.ShouldBe(3);
|
||||||
|
fake.RawStreamsCompleted.ShouldBe(3);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Fact] // the default cap scales with the request: 64 + 4 per distinct time
|
||||||
|
public async Task The_default_read_cap_is_64_plus_4_per_distinct_time()
|
||||||
|
{
|
||||||
|
(GatewayAtTimeReader.BaseReads, GatewayAtTimeReader.ReadsPerTime).ShouldBe((64, 4));
|
||||||
|
await Task.CompletedTask;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ---- type hint edge cases (review L2/L3) ------------------------------------------------------------------------
|
||||||
|
|
||||||
|
[Fact]
|
||||||
|
public async Task A_DateTime_node_is_stepped()
|
||||||
|
{
|
||||||
|
var fake = new AvevaStore([(At(0), 0, false), (At(10), 10, false)]).Client();
|
||||||
|
var s = (await AtTimeTyped(fake, TypeCode.DateTime, At(5))).Single();
|
||||||
|
s.Value.ShouldBe(0d);
|
||||||
|
s.StatusCode.ShouldBe(GoodInterpolated);
|
||||||
|
}
|
||||||
|
|
||||||
|
[Theory]
|
||||||
|
[InlineData(double.PositiveInfinity)]
|
||||||
|
[InlineData(double.NaN)]
|
||||||
|
[InlineData(1e30)]
|
||||||
|
public async Task Under_an_integer_hint_a_non_integer_next_value_is_unusable_not_saturated(double next)
|
||||||
|
{
|
||||||
|
var fake = new AvevaStore([(At(0), 100, false), (At(10), next, false)]).Client();
|
||||||
|
var s = (await AtTimeTyped(fake, TypeCode.Int32, At(5))).Single();
|
||||||
|
s.Value.ShouldBeOfType<int>().ShouldBe(100); // held before an unusable neighbour
|
||||||
|
s.StatusCode.ShouldBe(UncertainInterpolated);
|
||||||
|
}
|
||||||
|
|
||||||
// ---- kill switch -----------------------------------------------------------------------------------------------
|
// ---- kill switch -----------------------------------------------------------------------------------------------
|
||||||
|
|
||||||
[Fact]
|
[Fact]
|
||||||
|
|||||||
Reference in New Issue
Block a user