dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(historian): #694 page raw reads at a strict gateway's learned ceiling; opt-in EventReadHorizon

Browse Source 
Item 1: IHistorianDataSource.MaxRawValuesPerRead (Gateway: learned ceiling - 16 = L; 0 elsewhere, guard-tested).
RawPageLimit plans every raw page on top of #692's lookahead: with L > 0 the read (page P + boundary ties k
+ lookahead) is <= L, P = min(N, L - k - 1) (uncapped: L - k - 1); no room => straight to the tie-cluster
over-fetch (never an uncapped or underflowing request). One re-plan + re-read when the read taught a new
limit or was refused for crowding pre-window rows (HistoryReadLimitExceededException.PreWindowRows): the
page shrinks by them. Over-fetch bound min(MaxTieClusterOverfetch, L - 1); slice size never 0.
Review fixes: F1 a window starting at/before the epoch counts AVEVA's synthetic row as pre-window (a
ceiling-truncated read was reported complete); F2 the page path disables ReverseRawReader's own re-run
(cold end-only ceiling 20/N=3000 cost 3,023 reads, now <= 40); F3 crowded look-back re-read around, not
refused; F4 the #687 crowding re-read room at least doubles (up to 8 re-reads), so a cluster of hundreds
just before a resume point no longer fails on a default gateway; a crowding refusal that leaves no room
is BadTooManyOperations, never a tick-by-tick cursor.

Item 2: ServerHistorian:EventReadHorizon (default 00:00:00 = exact, unchanged). When set, every fresh
HistoryReadEvents window's earlier bound is raised to now - horizon (both directions, all backends).
Lossy by design; documented.

Item 3 (native TimeAverage/Total re-check) stays blocked on historiangw#27.
This commit is contained in:
Joseph Doherty
2026-10-13 00:52:12 -04:00
parent a5a0496176
commit 3c3513ea3e
23 changed files with 1461 additions and 79 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Historian.md
+54 -7
View File
@@ -113,6 +113,7 @@ Processed read is `BadAggregateNotSupported` and no history aggregate functions
"MaxTieClusterOverfetch": 65536,
"HonorReturnBounds": true,
"GatewayAtTimeFromRaw": true,
"EventReadHorizon": "00:00:00",
"HistoryReadBatchParallelism": 4,
"MaxConcurrentHistoryReadBatches": 16,
"HistoryReadDeadline": "00:01:00",
@@ -148,6 +149,7 @@ Processed read is `BadAggregateNotSupported` and no history aggregate functions
| `MaxTieClusterOverfetch` | int | `65536` | (Node-manager paging; all backends — a `SqlServer` PK rules out tie clusters, so it is moot there.) Maximum samples the server will fetch in one shot to page through a tie cluster (multiple samples sharing one `SourceTimestamp`). A cluster larger than this ceiling fails `BadTooManyOperations` (#686; was `BadHistoryOperationUnsupported`). Also the per-read row budget of a Gateway reverse raw read (walked back from forward reads, #686): one timestamp holding more samples is refused `BadTooManyOperations`. Values below 1 count as 1. Raise to handle abnormally large tie clusters; the default covers all normal-data cases. |
| `HonorReturnBounds` | bool | `true` | (#691, all backends.) Whether HistoryRead-Raw honours `ReturnBounds` (Part 11 bounding values, [Bounding values](#bounding-values-returnbounds-691)). `false` is the kill switch back to the pre-#691 behaviour: the flag is ignored (stored samples only) and a modified read carrying it is not refused `Bad_InvalidArgument`. Note the .NET SDK defaults both `ReturnBounds` and `IsReadModified` to `true`. |
| `GatewayAtTimeFromRaw` | bool | `true` | (#693, **Gateway backend only.**) Whether HistoryRead-AtTime is served from raw gateway reads evaluated with the SQL backends' AtTime rules ([Gateway AtTime](#gateway-attime-is-evaluated-from-raw-reads-693)). `false` is the kill switch back to the pre-#693 answer: AVEVA's own interpolation (the gateway's `ReadAtTime`) passed through unchanged. |
| `EventReadHorizon` | TimeSpan | `00:00:00` | (#694, all backends, **opt-in, lossy**.) `00:00:00` = no horizon: every HistoryReadEvents read is exact to its own window. A positive span raises the earlier bound of every fresh events window to `now − EventReadHorizon` (the floor instant itself included), in **both** directions — a window wholly older is `GoodNoData` and is not sent; a page-through keeps the first page's floor. It exists for the Gateway, where a walked reverse / end-only read proves "nothing older" down to the Unix epoch (~3.8 s live; an unfiltered one can hit the gateway's 30 s SQL timeout) and a forward read from a far-past `StartTime` scans the same range. ⚠️ While set, events older than the horizon are **invisible** to HistoryReadEvents. Raw/Processed/AtTime are unaffected. Negative ⇒ treated as none (a `Validate()` warning). See [Continuation-point paging (Events)](#continuation-point-paging-events-669). |
| `HistoryReadBatchParallelism` | int | `4` | Max historized nodes served **concurrently** within one HistoryRead batch (arch-review 03/S3). The SDK's `HistoryRead*` override surface is synchronous, so each arm bridges once at its boundary; this turns per-node serving from sequential (`N × CallTimeout` worst case) into `⌈N / P⌉ × CallTimeout`. Also caps the gateway-load multiplication a single client can command. `≤ 1` serves sequentially. |
| `MaxConcurrentHistoryReadBatches` | int | `16` | Process-wide limit on HistoryRead (and HistoryUpdate) batches served **concurrently** (arch-review 03/S3). A flood of history clients degrades to `BadTooManyOperations` rather than exhausting the SDK request-thread pool. With `HistoryReadBatchParallelism` this caps total in-flight gateway reads at `MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism` (default 16 × 4 = 64). A batch that cannot get a slot within `min(HistoryReadDeadline, 5 s)` fails every handle `BadTooManyOperations`. |
| `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` (#671), and the `Gateway` per-sample aggregates (#681 — every per-sample read there streams raw samples, so the cap applies to the window's raw rows; see [Aggregate set](#aggregate-set-and-badaggregatenotsupported-285)). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative, and — since #680 — **PercentGood/PercentBad**) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped on the SQL backends (pushed down). 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). `Gateway` (#688): the row budget it asks the gateway for (`cap + 18`) is clamped at the gateway's default `MaxValuesCeiling` (1,000,000) while the cap is at or under it, so the default works against a gateway with `AllowStreamingBeyondCeiling=false`; against a gateway with a *lower* ceiling the refused budget is retried once at the ceiling the gateway names (#691.3), so no change is needed (`ceiling − 18` only saves that round-trip). |
@@ -716,7 +718,8 @@ backend:
> and counted as a backend failure. It is now sent unbounded (`int.MaxValue`); a gateway configured with
> `AllowStreamingBeyondCeiling=false` refuses that over its `MaxValuesCeiling`; since #691.3 the read is retried once at the
> ceiling the refusal names and answered when provably complete, else **`BadTooManyOperations`** (not a backend fault) — see
> [#687](#gateway-raw-reads-return-stored-samples-only-687) "Strict gateway". A cancelled read (client cancel / `HistoryReadDeadline`) is no longer counted as a backend failure.
> [#687](#gateway-raw-reads-return-stored-samples-only-687) "Strict gateway". Since #694 the HistoryRead pages such a read at the
> learned ceiling with continuation points instead (same section, "Pages are sized to a strict gateway's ceiling"). A cancelled read (client cancel / `HistoryReadDeadline`) is no longer counted as a backend failure.
Continuation points are bound to the OPC UA session (the SDK's
`ServerConfiguration.MaxHistoryContinuationPoints` cap, default 100, with oldest-eviction; points
@@ -767,15 +770,50 @@ data.
when it is **provably complete** — the stream came back short of the ceiling, or reached `EndTime` (or, capped, filled the page);
a stream that delivered the whole ceiling short of `EndTime` may be truncated and fails **`BadTooManyOperations`** (not a backend
fault) — set `NumValuesPerNode` to page, or narrow the window. A refusal whose text carries no number falls back to the gateway's
default ceiling (1,000,000). Every stream is still drained. The #686 reverse walk-back (segment budget `MaxTieClusterOverfetch + 2`)
inherits this: against a ceiling below that budget a dense segment fails `BadTooManyOperations` instead of being narrowed.
- **Paging (#669) is unaffected.** A full page still holds exactly `NumValuesPerNode` stored samples and carries a continuation
default ceiling (1,000,000). Every stream is still drained.
- ⚠️ **Pages are sized to a strict gateway's ceiling (#694, client-visible).** Until #694 a page size above a strict gateway's
ceiling failed: with `MaxValuesCeiling = 1000`, `NumValuesPerNode = 5000` — or `0` over 3,600 rows — was `BadTooManyOperations`.
The data source now reports the learned ceiling to the server as `IHistorianDataSource.MaxRawValuesPerRead` = **L** = ceiling − 16
(`0` until a refusal has named a ceiling, and again after a re-probe finds the gateway streaming; `0` on every SQL backend). The
node manager plans each page with it (`RawPageLimit`): a page's read is the page **P** + the boundary ties **k** it re-reads and
trims + the #692 lookahead row, and with a limit that read is at most L — so **P = min(N, L − k − 1)** (uncapped: P = L − k − 1;
with the ceiling at 1000 a first page holds 983 values, a resumed one 982), and a full page carries a continuation point like any
other (Part 11 allows fewer than `NumValuesPerNode` with one). **This includes uncapped reads** (`NumValuesPerNode = 0`), which
page instead of being refused; a window that fits is still one page without a continuation point. Without a limit the #692 plan
is unchanged. Boundary ties that alone fill a read (k ≥ L − 1) go straight to the tie-cluster over-fetch — never an uncapped or
underflowing request.
- **First read after start-up:** the ceiling is unknown until refused once; the read that learned it is re-planned and re-issued
ONCE (cost on that read: the refused attempt, the ceiling re-read that could not hold the window, the page-sized read; later
pages are one read each). Reverse / end-only reads (the #686 walk-back) size their segments to L (tie budget
`min(MaxTieClusterOverfetch, L − 2)`); on the page path the refusal reaches the page planner, which shrinks the **page** — the
walk never re-runs itself at the ceiling with the page still at N (review F2: a cold end-only read with a ceiling of 20 and
N = 3000 cost 3,023 gateway reads in one HistoryRead; now ≤ 40). Bound lookups (a page of one) keep the walk's own re-run.
- **Crowded look-back (review F3).** A page asking for exactly the ceiling leaves no room for the #687 1 ms look-back, so more
than 16 rows just before a page start (e.g. 20 ties 1 ms before it) used to fail `BadTooManyOperations`. The data source now
reports the crowding rows on its refusal (`HistoryReadLimitExceededException.PreWindowRows`, only when they exceed the 16-row
allowance), and the planner re-reads the page once that many values smaller — the source's own crowding re-read then fits under
the ceiling. In the walk-back, a crowded segment is re-read that much smaller and counts as truncated.
- **Tie clusters.** The over-fetch that pages within a cluster reads at most `min(MaxTieClusterOverfetch, L − 1)` + 1 rows (so it
never trips a refusal); a larger cluster is **`BadTooManyOperations`** — the gateway cannot offset into it. A crowding refusal that
leaves no room for a page is that answer too (it is never mistaken for "the cursor sits in a cluster", which would step the
cursor a tick per page forever). Every sample before such a failure is served exactly once; nothing is truncated silently.
- **Never a truncation:** the gateway read is still answered in full or refused; only the server's page is smaller.
- A raised ceiling is not noticed by raw paging (its reads ask for at most the learned ceiling, so nothing re-probes); pages stay
at the old size until a read asking for more (a per-sample aggregate) re-probes it, or the server restarts — correct results
either way. A **lowered** ceiling is re-learned from the next page's refusal and that page re-planned.
- ⚠️ **A window starting at or before the Unix epoch (review F1, fixed with #694).** Such a window gets no 1 ms look-back, so
AVEVA's synthetic row lands exactly at its start; it is no sample, and it used to count as "past the window end" — a
ceiling-truncated read was then reported complete (an uncapped forward read from the epoch answered its first ceiling − 1 rows,
Good, no continuation point). It now counts as a pre-window row.
- **Paging (#669) is unaffected.** A full page still holds exactly the page size of stored samples and carries a continuation
point, and a resumed page (which re-reads from the last returned timestamp inclusive, and the tie-cluster over-fetch
`start == end` read) no longer gets a fabricated row at its resume point. A read is complete when the stream came back short,
filled the page, or reached EndTime. If rows **before** the window — the boundary row plus stored samples inside the 1 ms margin,
e.g. a tie cluster at `StartTime − 1 ms` or at a resume point `− 1 ms` — used up the budget, the window is re-read **once**
asking for `NumValuesPerNode + m + 16` (m = the pre-window rows the first read saw, unclamped). Only if that re-read is still
short because of pre-window rows does the node fail `BadTooManyOperations` (not a backend fault) rather than answer a short page
e.g. a tie cluster at `StartTime − 1 ms` or at a resume point `− 1 ms` — used up the budget, the window is re-read with room for
them: `cap + room + 16`, room = the pre-window rows the first read saw, at least **doubling** per re-read while a read that was
itself cut short keeps finding more (up to 8 re-reads, never past a known strict ceiling — review F4: a single re-read sized from
a truncated read's count failed a 300–1,700-row cluster just before a resume point even on a default gateway). Only if the
re-reads are still crowded does the node fail `BadTooManyOperations` (not a backend fault) rather than answer a short page
that would end the client's paging early. Every gateway stream is still drained to completion.
- **`ReturnBounds` is served by the server, not by this row** (#691, [Bounding values](#bounding-values-returnbounds-691)). The AVEVA
boundary row is never used as a bound: it is stamped at the query start rather than carrying the time of the real prior sample, so
@@ -951,6 +989,15 @@ says is complete carries no continuation point. `NumValuesPerNode == 0` stays un
the walk's biggest read (`2N + 18` rows, a discovery read 1,026, a tie-cluster read up to `2·MaxTieClusterOverfetch + 4`).
That cap truncates **silently** (the stream just ends), so a segment it cut would look complete and drop its newest events —
the same blindness a forward read has to it, but on the end a reverse read is asking for.
- ⚠️ **Cost of "nothing older" (#694 item 2).** A walked read that runs out of events proves the rest of its window empty with
one forward read down to its earlier bound — for an end-only read, the Unix epoch. On `Runtime.dbo.Events` that read is
measured at **~3.8 s** for a source with few events (vs ~0.7 s for a bounded window), and an **unfiltered** deep read hit the
gateway's 30 s SQL timeout; a forward read from a far-past `StartTime` scans the same range. The cost is the gateway's query
over `Runtime.dbo.Events` (its index use is a historiangw matter, not fixable here). Two ways to keep it down: give the read an
explicit `EndTime`/`StartTime` that bounds the window, or set the opt-in **`ServerHistorian:EventReadHorizon`** (below). The
default stays exact — there is no cache of "earliest event per source", deliberately: events reach `Runtime.dbo.Events` from
the alarm store-and-forward drain (which can deliver hours- or days-old events after an outage) and from AVEVA itself, so a
remembered "nothing before X" can go stale silently.
⚠️ **Client-visible (#691.4).** On Gateway (the default backend) and Null an explicit reverse events read used to be refused
`BadHistoryOperationUnsupported` (#675), and an end-only capped events read returned the **oldest** N events ascending. Both
src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/HistoryReadContracts.cs
+8
View File
@@ -93,6 +93,14 @@ public sealed class HistoryReadLimitExceededException : InvalidOperationExceptio
/// <summary>The configured limit that was exceeded.</summary>
public long Limit { get; }
/// <summary>
/// Gitea #694: a raw read refused because rows BEFORE its window (inside the backend's look-back margin — a tie
/// cluster just before a page start, the HistorianGateway's boundary row) crowded a ceiling-bounded row budget: how
/// many such rows the read saw (a lower bound when the read was itself truncated). <c>0</c> = not a crowding refusal.
/// The server re-reads such a page once, smaller by this many values (<c>RawPageLimit</c>).
/// </summary>
public int PreWindowRows { get; init; }
/// <summary>Creates the exception.</summary>
/// <param name="limit">The configured limit.</param>
/// <param name="message">Operator-facing description.</param>
src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataSource.cs
+12
View File
@@ -46,6 +46,18 @@ public interface IHistorianDataSource : IDisposable
/// </summary>
bool SupportsReverseRaw => false;
/// <summary>
/// The most values one capped <see cref="ReadRawAsync"/> can currently return without failing
/// <see cref="HistoryReadLimitExceededException"/> on a BACKEND row ceiling (Gitea #694); <c>0</c> = no known limit.
/// The server sizes its raw pages to it — a page is at most this many values and carries a continuation point when
/// full (Part 11 allows fewer than <c>NumValuesPerNode</c> with a continuation point), instead of failing a page size
/// the backend cannot read in one go. It may change at runtime (a strict HistorianGateway's ceiling is learned from
/// its first refusal, #691.3), so it is read per read. A backend whose reads are never ceiling-bound returns <c>0</c>.
/// The source still never truncates: a read asking for more than this is answered in full or refused. Default
/// <c>0</c>; a backend declares it itself (a guard test pins that).
/// </summary>
int MaxRawValuesPerRead => 0;
/// <summary>
/// Read processed (interval-bucketed) samples — average / min / max / count / etc.
/// A bucket with no source data returns a sample whose
src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/NullHistorianDataSource.cs
+4
View File
@@ -60,6 +60,10 @@ public sealed class NullHistorianDataSource : IHistorianDataSource
/// direction — no need for the server to walk back through forward reads to find nothing.</remarks>
public bool SupportsReverseRaw => true;
/// <inheritdoc />
/// <remarks><c>0</c>: no history, no ceiling (Gitea #694).</remarks>
public int MaxRawValuesPerRead => 0;
/// <inheritdoc />
/// <remarks><c>true</c>, like <see cref="SupportsReverseRaw"/> (Gitea #691.4 review; <c>false</c> under #675): there is no
/// history to read in either direction — no need for the server to walk back through empty forward reads.</remarks>
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/GatewayHistorianDataSource.cs
+50 -15
View File
@@ -51,6 +51,9 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
/// <summary>#693: AtTime from raw reads (true) or AVEVA's own interpolation (the kill switch).</summary>
internal bool AtTimeFromRaw => _atTimeFromRaw;
/// <summary>The most #687 crowding re-reads one raw read makes (#694 F4): each at least doubles the pre-window room.</summary>
private const int MaxCrowdedReReads = 8;
/// <summary>A strict gateway's row ceiling, learned from its first refusal (#691.3).</summary>
private readonly GatewayRowCeiling _ceiling = new();
@@ -191,29 +194,46 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
throw new HistoryReadLimitExceededException(cap,
$"The raw read of '{fullReference}' holds more than the HistorianGateway's row ceiling ({all.Budget}, its " +
"Historian:ReadLimits:MaxValuesCeiling with AllowStreamingBeyondCeiling=false); set NumValuesPerNode to page, " +
"or narrow the window.");
"or narrow the window.")
{ PreWindowRows = Crowding(all.Result.PreWindowRows) };
}
var asked = (int)Math.Min(int.MaxValue, (long)cap + RawAggregateReader.PreWindowAllowance);
var first = await PassAsync(asked).ConfigureAwait(false);
if (first.Result.Complete) return first.Result.Samples;
var pass = await PassAsync(asked).ConfigureAwait(false);
if (pass.Result.Complete) return pass.Result.Samples;
// A strict gateway's ceiling already bounded that read (#691.3): a larger budget would only be refused again.
if (first.Budget < asked)
// A strict gateway's ceiling already bounded that read (#691.3): a larger budget would only be refused again. The
// refusal carries the pre-window rows it saw (#694): the server re-reads the page that many values smaller.
if (pass.Budget < asked)
throw new HistoryReadLimitExceededException(cap,
$"The raw read of '{fullReference}' could not be read completely within the HistorianGateway's row ceiling " +
$"({first.Budget}, its Historian:ReadLimits:MaxValuesCeiling with AllowStreamingBeyondCeiling=false); lower " +
"NumValuesPerNode or narrow the window.");
$"({pass.Budget}, its Historian:ReadLimits:MaxValuesCeiling with AllowStreamingBeyondCeiling=false); lower " +
"NumValuesPerNode or narrow the window.")
{ PreWindowRows = Crowding(pass.Result.PreWindowRows) };
// The margin crowded the window out: one bounded re-read with room for the pre-window rows already seen.
var second = await PassAsync(
(int)Math.Min(int.MaxValue, (long)cap + first.Result.PreWindowRows + RawAggregateReader.PreWindowAllowance))
.ConfigureAwait(false);
if (second.Result.Complete) return second.Result.Samples;
// The margin crowded the window out: re-read with room for the pre-window rows. A read that was itself cut short saw
// only some of them (#694 F4: a tie cluster of hundreds just before a resume point used to fail even with no
// ceiling), so the room at least doubles per re-read — bounded by MaxCrowdedReReads, and never past a known ceiling.
var preWindow = pass.Result.PreWindowRows;
var room = Math.Max(1, preWindow);
for (var attempt = 0; attempt < MaxCrowdedReReads; attempt++)
{
var budget = (int)Math.Min(int.MaxValue, (long)cap + room + RawAggregateReader.PreWindowAllowance);
if (_ceiling.Learned is var known and > 0 && budget > known) break; // over a strict ceiling: refused anyway
pass = await PassAsync(budget).ConfigureAwait(false);
if (pass.Result.Complete) return pass.Result.Samples;
preWindow = Math.Max(preWindow, pass.Result.PreWindowRows);
if (pass.Budget < budget) break; // bounded by a ceiling learned meanwhile
room = (int)Math.Min(int.MaxValue / 2, Math.Max(preWindow, 2L * room));
}
throw new HistoryReadLimitExceededException(cap,
$"The raw read of '{fullReference}' could not be read completely: more than {second.Result.PreWindowRows} rows lie in " +
$"the {RawAggregateReader.BoundaryMargin.TotalMilliseconds} ms before the window and exhausted the row budget.");
$"The raw read of '{fullReference}' could not be read completely: more than {pass.Result.PreWindowRows} rows lie in " +
$"the {RawAggregateReader.BoundaryMargin.TotalMilliseconds} ms before the window and exhausted the row budget.")
{ PreWindowRows = preWindow };
// Pre-window rows the allowance did not cover are what crowded the read; fewer are not the cause (#694).
static int Crowding(int preWindowRows) => preWindowRows > RawAggregateReader.PreWindowAllowance ? preWindowRows : 0;
// One drained pass; a strict gateway's refusal of the budget is retried once at the ceiling it names (#691.3).
Task<((List<HistorianSample> Samples, int PreWindowRows, bool Complete) Result, int Budget)> PassAsync(int budget) =>
@@ -240,7 +260,11 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
{
if (cap == 0 || samples.Count < cap) samples.Add(sample);
}
else if (sample.Timestamp is { } ts && ts.ToDateTime() < startUtc) preWindow++;
// A row before the window — or, for a window starting at/before the epoch (no look-back), AVEVA's synthetic row at
// the query start, which is no sample either (#694 F1: it used to set reachedEnd, so a ceiling-truncated read
// passed as complete) — is a pre-window row.
else if (sample.Timestamp is { } ts
&& (ts.ToDateTime() < startUtc || !RawAggregateReader.IsAfterFloor(ts.ToDateTime()))) preWindow++;
else if (sample.Timestamp is not null) reachedEnd = true;
}
@@ -453,6 +477,17 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
/// </remarks>
public bool SupportsReverseRaw => false;
/// <inheritdoc />
/// <remarks>
/// Gitea #694: <c>0</c> until a strict gateway (<c>AllowStreamingBeyondCeiling=false</c>) has refused a budget and
/// named its ceiling (#691.3, <see cref="GatewayRowCeiling"/>); then that ceiling less the
/// <see cref="RawAggregateReader.PreWindowAllowance"/> a capped read adds for AVEVA's boundary row — so a page of this
/// size asks for exactly the ceiling. Back to <c>0</c> when a re-probe finds the gateway streaming again.
/// </remarks>
public int MaxRawValuesPerRead => _ceiling.Learned is var c and > 0
? Math.Max(1, c - RawAggregateReader.PreWindowAllowance)
: 0;
/// <inheritdoc />
/// <remarks>
/// <c>false</c>: the gateway's <c>ReadEvents</c> streams ascending only (Gitea #675). The server never sends it a
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs
+4
View File
@@ -359,6 +359,10 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource
/// <remarks>Reverse raw reads are served natively (<c>(end, start]</c>, <c>ORDER BY … DESC</c>; #669/#686).</remarks>
public bool SupportsReverseRaw => true;
/// <inheritdoc />
/// <remarks><c>0</c>: a capped raw read is one bounded statement — no backend row ceiling (Gitea #694).</remarks>
public int MaxRawValuesPerRead => 0;
/// <inheritdoc />
/// <remarks>Reverse events reads are served natively (<c>ORDER BY … DESC</c>, Gitea #675).</remarks>
public bool SupportsReverseEvents => true;
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataSource.cs
+4
View File
@@ -316,6 +316,10 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
/// <remarks>Reverse raw reads are served natively (<c>(end, start]</c>, <c>ORDER BY … DESC</c>; #669/#686).</remarks>
public bool SupportsReverseRaw => true;
/// <inheritdoc />
/// <remarks><c>0</c>: a capped raw read is one bounded statement — no backend row ceiling (Gitea #694).</remarks>
public int MaxRawValuesPerRead => 0;
/// <inheritdoc />
/// <remarks>Reverse events reads are served natively (<c>ORDER BY … DESC</c>, Gitea #675).</remarks>
public bool SupportsReverseEvents => true;
src/Server/ZB.MOM.WW.OtOpcUa.Host/OpcUa/OtOpcUaServerHostedService.cs
+9
View File
@@ -320,6 +320,9 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
// Gitea #691: the ReturnBounds kill switch (default on).
_server.NodeManager.HonorReturnBounds = HonorReturnBounds;
// Gitea #694: the opt-in HistoryReadEvents horizon (default off = exact to each read's own window).
_server.NodeManager.EventReadHorizon = EventReadHorizon;
// Propagate the S3 HistoryRead concurrency bounds (arch-review 03/S3): bounded per-node fan-out
// within a batch, the process-wide concurrent-batch limiter, and the per-request deadline. Defaults
// (4 / 16 / 60 s) survive when the ServerHistorian section is absent.
@@ -356,6 +359,12 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
/// </summary>
internal bool HonorReturnBounds => _serverHistorianOptions.HonorReturnBounds;
/// <summary>
/// Gitea #694 — the <c>ServerHistorian:EventReadHorizon</c> <c>StartAsync</c> applies to the node manager (default
/// <see cref="TimeSpan.Zero"/> = none). Exposed so the wiring is testable without booting the SDK.
/// </summary>
internal TimeSpan EventReadHorizon => _serverHistorianOptions.EventReadHorizon;
/// <summary>
/// Gitea #674 — what <c>StartAsync</c> wires onto the node manager for HistoryUpdate: the Primary gate (read from
/// the <see cref="IHistoryUpdateRoleView"/> on every call; null when none is registered, i.e. allow), the
src/Server/ZB.MOM.WW.OtOpcUa.Host/appsettings.json
+3 -1
View File
@@ -35,7 +35,9 @@
"AllowUntrustedServerCertificate": false,
"CaCertificatePath": null,
"CallTimeout": "00:00:30",
"MaxTieClusterOverfetch": 65536
"MaxTieClusterOverfetch": 65536,
"EventReadHorizon": "00:00:00",
"_EventReadHorizonComment": "Gitea #694, opt-in. 00:00:00 = no horizon: HistoryReadEvents is exact to each read's window (on the Gateway a reverse/end-only events read proves 'nothing older' with one read down to 1970 - seconds on a large Runtime.dbo.Events). A positive span (e.g. 365.00:00:00) never reads events older than now minus it, in either direction - LOSSY: older events become invisible to HistoryReadEvents."
},
"ContinuousHistorization": {
"_comment": "Continuous historization of driver (non-Galaxy) tag values: a crash-safe FasterLog outbox + recorder draining to the ServerHistorian gateway's WriteLiveValues. Disabled => no recorder is spawned. Requires ServerHistorian to be configured; the gateway connection (endpoint/key/TLS) is sourced from the ServerHistorian section, not here.",
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/EventHorizon.cs
+43
View File
@@ -0,0 +1,43 @@
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
/// <summary>
/// Gitea #694 item 2: the opt-in <c>ServerHistorian:EventReadHorizon</c> — HistoryReadEvents never reads events older than
/// <c>now − horizon</c>. Off (<see cref="TimeSpan.Zero"/>, the default) a read is exact down to whatever its window says,
/// which on the Gateway means a reverse / end-only read walked back with forward reads proves "nothing older" with one
/// read down to the Unix epoch (#691) — ~3.8 s live on <c>Runtime.dbo.Events</c>, and an unfiltered one can hit the
/// gateway's 30 s SQL timeout. A forward read whose StartTime is far in the past costs the same on that table. The
/// horizon bounds both, at a stated price: events older than it are not returned, in either direction, on any backend.
/// </summary>
internal static class EventHorizon
{
/// <summary>
/// Raises the EARLIER bound of a fresh events window to the horizon's floor (<c>now − horizon</c>, inclusive): the
/// start of a forward <c>[start, end)</c> window, the exclusive lower end of a reverse <c>(end, start]</c> one.
/// </summary>
/// <param name="startUtc">The window start (the later bound of a reverse read).</param>
/// <param name="endUtc">The window end (the exclusive earlier bound of a reverse read).</param>
/// <param name="reverse">Whether the read is newest-first (<c>start &gt; end</c>).</param>
/// <param name="horizon">The horizon; non-positive = none (nothing changes).</param>
/// <param name="nowUtc">Now.</param>
/// <returns><c>false</c> when the whole window lies before the floor — the read is empty and must not be sent.</returns>
public static bool TryClamp(ref DateTime startUtc, ref DateTime endUtc, bool reverse, TimeSpan horizon, DateTime nowUtc)
{
if (horizon <= TimeSpan.Zero) return true;
var floor = nowUtc.Ticks - DateTime.MinValue.Ticks > horizon.Ticks
? new DateTime(nowUtc.Ticks - horizon.Ticks, DateTimeKind.Utc)
: new DateTime(0, DateTimeKind.Utc);
if (floor.Ticks == 0) return true;
if (reverse)
{
if (startUtc < floor) return false; // (end, start] lies wholly before the floor
var below = floor.AddTicks(-1); // exclusive lower end ⇒ the floor instant itself is read
if (endUtc < below) endUtc = below;
return true;
}
if (endUtc <= floor) return false; // [start, end) lies wholly before the floor
if (startUtc < floor) startUtc = floor;
return true;
}
}
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs
+87 -35
View File
@@ -552,6 +552,13 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
/// </summary>
public bool HonorReturnBounds { get; set; } = true;
/// <summary>
/// Gitea #694: <c>ServerHistorian:EventReadHorizon</c> — HistoryReadEvents never reads events older than
/// <c>now − EventReadHorizon</c> (<see cref="EventHorizon"/>). <see cref="TimeSpan.Zero"/> (the default) = no horizon:
/// every read is exact down to its own window (a walked Gateway reverse read proves "nothing older" down to 1970).
/// </summary>
public TimeSpan EventReadHorizon { get; set; } = TimeSpan.Zero;
/// <summary>
/// Maximum historized nodes served concurrently within one HistoryRead batch (arch-review 03/S3).
/// Mirrors <c>ServerHistorianOptions.HistoryReadBatchParallelism</c>; the Host sets it at
@@ -5339,6 +5346,19 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// Part 11: StartTime later than EndTime is a reverse read — events newest-first over (end, start].
reverse = startUtc > endUtc;
// #694: the opt-in events horizon raises the window's earlier bound (the cursor then carries it). A window
// wholly older than it is empty and is not sent (a forward [start, end) clamped past its end would be refused).
if (!EventHorizon.TryClamp(ref startUtc, ref endUtc, reverse, EventReadHorizon, Clock.GetUtcNow().UtcDateTime))
{
results[handle.Index] = new SdkHistoryReadResult
{
StatusCode = StatusCodes.GoodNoData,
HistoryData = new ExtensionObject(ProjectEvents([], selectClauses)),
};
errors[handle.Index] = ServiceResult.Good;
return;
}
}
// A reverse read on a backend that cannot read newest-first (the Gateway) is never sent there (#675 refused it
@@ -6450,58 +6470,77 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// "full" empty page was taken for a stalled tie cluster, and a MaxTieClusterOverfetch + 1 over-fetch stepped one
// tick to return an EMPTY page with a continuation point — between every value. The lookahead row is what says
// "more data follows": a page whose read found no row past it is the last, so the read no longer ends with an
// empty trailing page either.
// empty trailing page either. (RawPageLimit.Plan computes the request: page + boundary ties + 1.)
var resumed = inboundCp is { Length: > 0 };
var requestCap = readCap == 0
? 0u
: (uint)Math.Min(uint.MaxValue, (long)readCap + (resumed ? Math.Max(0, boundarySkip) : 0) + 1L);
var boundaryTies = resumed ? Math.Max(0, boundarySkip) : 0;
var want = readCap; // N, less the start bound on a bounded first page; 0 = all values
// HistoryRead is NOT under the node-manager Lock — awaiting the async source is safe.
// A reverse window (explicit, resumed, or the end-only read above) goes to the backend only when it declares it
// can read newest-first; otherwise it is walked back from bounded forward reads (#686 — the Gateway rejects
// start > end, which failed the node and counted as a backend fault).
// A cursor left inside an oversized tie cluster (#692) skips this read: every row it could return is a tie.
// #694: a backend with a row ceiling (MaxRawValuesPerRead = L > 0, a strict gateway) sizes the page to it — the
// read asks for page + boundary ties + lookahead ≤ L, and a full page carries a continuation point as always
// (uncapped reads included). No room left after the boundary ties ⇒ the cursor sits in a tie cluster as large as
// a read: straight to the over-fetch. See RawPageLimit.
var source = HistorianDataSource;
IReadOnlyList<DataValueSnapshot> samples = [];
var backendFull = false;
int pageLimit;
if (!withinTieCluster)
{
HistorianRead sourceResult;
if (startUtc > endUtc && !source.SupportsReverseRaw)
var read = await RawPageLimit.ReadAsync(source, want, boundaryTies, async plan =>
{
var page = await ReverseRawReader.ReadAsync(
source, tagname, startUtc, endUtc, requestCap, MaxTieClusterOverfetch,
nowUtc, walkHint, ct)
.ConfigureAwait(false);
walkHint = page.Hint;
sourceResult = new HistorianRead(page.Samples, ContinuationPoint: null);
if (startUtc > endUtc && !source.SupportsReverseRaw)
{
var page = await ReverseRawReader.ReadAsync(
source, tagname, startUtc, endUtc, plan.RequestCap, MaxTieClusterOverfetch,
nowUtc, walkHint, ct, retryOnLearnedLimit: false)
.ConfigureAwait(false);
walkHint = page.Hint;
return new HistorianRead(page.Samples, ContinuationPoint: null);
}
return await source.ReadRawAsync(tagname, startUtc, endUtc, plan.RequestCap, ct).ConfigureAwait(false);
}).ConfigureAwait(false);
pageLimit = read.Plan.Limit;
if (read.Plan.NoRoom)
{
if (!resumed)
throw new HistoryReadLimitExceededException(pageLimit,
$"The backend's row ceiling ({pageLimit} values per read) leaves no room for a raw page of '{tagname}'.");
withinTieCluster = true; // every row a read could return is a boundary tie already emitted
}
else
{
sourceResult = await source
.ReadRawAsync(tagname, startUtc, endUtc, requestCap, ct)
.ConfigureAwait(false);
}
readCap = read.Plan.PageCap;
var sourceResult = read.Page!;
backendFull = HistoryPaging.IsFullPage(sourceResult.Samples.Count, read.Plan.RequestCap);
backendFull = HistoryPaging.IsFullPage(sourceResult.Samples.Count, requestCap);
// On a resume read, drop the boundary ties already returned on the prior page.
samples = resumed
? HistoryPaging.TrimBoundaryDuplicates(sourceResult.Samples, startUtc, boundarySkip)
: sourceResult.Samples;
// On a resume read, drop the boundary ties already returned on the prior page.
samples = resumed
? HistoryPaging.TrimBoundaryDuplicates(sourceResult.Samples, startUtc, boundarySkip)
: sourceResult.Samples;
// #691: a bounded read's data window excludes the end time (a sample there is the end bound). A backend whose
// forward end is inclusive (the gateway's own) returns it as data; dropping it also means the data is drained.
if (endBound is not null)
{
var inside = samples.Where(x => HistoryBounds.InDataWindow(x, endBound, resolution)).ToList();
if (inside.Count != samples.Count)
// #691: a bounded read's data window excludes the end time (a sample there is the end bound). A backend whose
// forward end is inclusive (the gateway's own) returns it as data; dropping it also means the data is drained.
if (endBound is not null)
{
samples = inside;
backendFull = false;
var inside = samples.Where(x => HistoryBounds.InDataWindow(x, endBound, resolution)).ToList();
if (inside.Count != samples.Count)
{
samples = inside;
backendFull = false;
}
}
}
}
else
{
pageLimit = source.MaxRawValuesPerRead;
}
// Oversized tie cluster: a resume read whose FULL backend read does not advance past the resume boundary
// timestamp `startUtc` — every row it returned past the trimmed ties, the lookahead row included, still sits on
@@ -6523,7 +6562,12 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// lets us DETECT a cluster strictly larger than the bound (the absurd-burst backstop below).
// Compute in uint so +1 can never wrap: Math.Max(1,…) also guards against a zero/negative
// config value slipping through (Validate() warns but does not block startup).
var overfetchCap = (uint)Math.Max(1, MaxTieClusterOverfetch) + 1u;
// #694: on a backend with a row ceiling the bound is also at most L − 1, so the over-fetch (bound + 1 rows)
// fits one read and an oversized cluster is refused here rather than by a gateway refusal.
var overfetchBound = pageLimit > 0
? Math.Min(Math.Max(1, MaxTieClusterOverfetch), Math.Max(1, pageLimit - 1))
: Math.Max(1, MaxTieClusterOverfetch);
var overfetchCap = (uint)overfetchBound + 1u;
IReadOnlyList<DataValueSnapshot> cluster = (await HistorianDataSource
.ReadRawAsync(tagname, startUtc, startUtc, overfetchCap, ct)
.ConfigureAwait(false)).Samples;
@@ -6537,12 +6581,13 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// Absurd burst: more ties than we're willing to buffer in memory. Preserve today's loud-fail
// for that node rather than over-fetch an unbounded cluster; the operator's remedy is a
// larger ServerHistorian:MaxTieClusterOverfetch (or NumValuesPerNode).
if (cluster.Count > MaxTieClusterOverfetch)
if (cluster.Count > overfetchBound)
{
_logger.LogError(
"OtOpcUaNodeManager: HistoryReadRaw tie cluster at {StartUtc:O} for tag '{Tagname}' has {Count} samples, " +
"exceeding MaxTieClusterOverfetch={Max}; cannot page within it. Increase MaxTieClusterOverfetch.",
startUtc, tagname, cluster.Count, MaxTieClusterOverfetch);
"exceeding {Max} (MaxTieClusterOverfetch, or the backend's row ceiling less one); cannot page within it. " +
"Increase MaxTieClusterOverfetch (or the HistorianGateway MaxValuesCeiling).",
startUtc, tagname, cluster.Count, overfetchBound);
errors[handle.Index] = StatusCodes.BadTooManyOperations;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadTooManyOperations };
return;
@@ -6550,8 +6595,15 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// Direction comes from the window itself (Part 11: start > end is a reverse read). A resumed reverse
// read keeps start > end, because a reverse resume point is always strictly after endUtc.
// #694: an uncapped read pages only at the backend's limit, so it slices at that page size — never 0 (a zero
// slice hands back the same cursor forever; a limit that reverted to 0 under an uncapped cursor in a cluster
// falls back to the cluster's own size).
var sliceSize = numValuesPerNode > 0 ? numValuesPerNode
: readCap > 0 ? readCap
: pageLimit > 0 ? (uint)pageLimit
: (uint)Math.Max(1, cluster.Count);
HistoryPaging.SliceTieCluster(
cluster.Count, boundarySkip, numValuesPerNode, startUtc, endUtc,
cluster.Count, boundarySkip, sliceSize, startUtc, endUtc,
out var sliceStart, out var sliceCount, out var nextStartUtc, out var nextSkip,
reverse: reverse, resolutionTicks: Math.Max(1L, HistorianDataSource.TimestampResolutionTicks));
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/RawPageLimit.cs
+97
View File
@@ -0,0 +1,97 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
/// <summary>How one HistoryRead-Raw page reads (<see cref="RawPageLimit.Plan"/>).</summary>
/// <param name="Limit">The backend's <see cref="IHistorianDataSource.MaxRawValuesPerRead"/> the plan was made under (0 = none).</param>
/// <param name="PageCap">The page: at most this many values are returned (0 = all values — only without a limit).</param>
/// <param name="RequestCap">What the read asks for: page + the boundary ties it will trim + one lookahead row (0 = all).</param>
/// <param name="NoRoom">The boundary ties alone fill a read — nothing can be read past them; serve the tie cluster.</param>
internal readonly record struct RawPagePlan(int Limit, uint PageCap, uint RequestCap, bool NoRoom);
/// <summary>
/// Gitea #694: sizes a HistoryRead-Raw page to what the backend can return in one read
/// (<see cref="IHistorianDataSource.MaxRawValuesPerRead"/> = L — a strict HistorianGateway's learned row ceiling, less
/// its boundary-row allowance). With a limit the read — page P + boundary ties k + the #692 lookahead row — is at most L,
/// so P = min(want, L − k − 1) (all values: P = L − k − 1); a full page carries a continuation point like any other, as
/// Part 11 allows fewer values than <c>NumValuesPerNode</c> with one. Before #694 a cap above a strict gateway's ceiling
/// (or an uncapped read of a window holding more) failed <c>BadTooManyOperations</c>.
/// </summary>
/// <remarks>
/// <para>
/// <b>Never a silent truncation.</b> The backend still answers a read in full or refuses it; only the server's page is
/// smaller, and a page is judged full against what was asked for. A read the backend cannot serve at all (a tie
/// cluster larger than one read) stays <c>BadTooManyOperations</c>.
/// </para>
/// <para>
/// <b>One re-read, planned again.</b> A refused read (<see cref="HistoryReadLimitExceededException"/>) is re-planned and
/// re-issued ONCE when (a) the backend learned a different limit while it ran — the first read after start-up teaches
/// the ceiling — or (b) it was refused because rows before the window crowded the budget
/// (<see cref="HistoryReadLimitExceededException.PreWindowRows"/> m): the page is then shrunk by m so that the
/// backend's look-back fits beside it. Anything else is the answer.
/// </para>
/// </remarks>
internal static class RawPageLimit
{
/// <summary>Plans one page.</summary>
/// <param name="want">The page the client asked for (N, less a start bound; 0 = all values).</param>
/// <param name="boundaryTies">Ties at the resume timestamp already returned, which the read re-reads and trims (k).</param>
/// <param name="limit">The backend's limit L (&lt;= 0 = none).</param>
/// <param name="sizeLimit">The limit to size the page by (L, or L − m after a crowded refusal); defaults to <paramref name="limit"/>.</param>
/// <returns>The plan.</returns>
public static RawPagePlan Plan(uint want, int boundaryTies, int limit, int? sizeLimit = null)
{
var k = Math.Max(0, boundaryTies);
if (limit <= 0)
return new RawPagePlan(0, want, want == 0 ? 0u : (uint)Math.Min(uint.MaxValue, (long)want + k + 1L), false);
var room = (long)(sizeLimit ?? limit) - k - 1;
if (room < 1) return new RawPagePlan(limit, 0, 0, true);
var page = want == 0 ? room : Math.Min(want, room);
return new RawPagePlan(limit, (uint)page, (uint)(page + k + 1), false);
}
/// <summary>
/// Reads one page under the backend's current limit, re-planning and re-reading ONCE as described in the remarks.
/// A <see cref="RawPagePlan.NoRoom"/> plan is returned without a read.
/// </summary>
/// <typeparam name="T">The page.</typeparam>
/// <param name="source">The backend.</param>
/// <param name="want">The page the client asked for (0 = all values).</param>
/// <param name="boundaryTies">The boundary ties the read re-reads and trims.</param>
/// <param name="read">One read with the given plan (it asks for <see cref="RawPagePlan.RequestCap"/>).</param>
/// <returns>The page (null when <see cref="RawPagePlan.NoRoom"/>) and the plan it was read with.</returns>
public static async Task<(T? Page, RawPagePlan Plan)> ReadAsync<T>(
IHistorianDataSource source, uint want, int boundaryTies, Func<RawPagePlan, Task<T>> read)
{
var before = source.MaxRawValuesPerRead;
var plan = Plan(want, boundaryTies, before);
if (plan.NoRoom) return (default, plan);
HistoryReadLimitExceededException refused;
try
{
return (await read(plan).ConfigureAwait(false), plan);
}
catch (HistoryReadLimitExceededException ex) when (source.MaxRawValuesPerRead is var now and > 0
&& (now != before || ex.PreWindowRows > 0))
{
refused = ex; // re-read outside the catch: the second refusal is the answer
}
var limit = source.MaxRawValuesPerRead;
var shrink = Math.Max(0, refused.PreWindowRows);
plan = Plan(want, boundaryTies, limit, (int)Math.Max(0L, (long)limit - shrink));
// No room left once the crowding rows are allowed for: those rows are not (necessarily) the boundary ties, so this is
// NOT "the cursor sits in a tie cluster" — the refusal is the answer (stepping the cursor through it a tick per page
// would never end).
// (A limit that changed under an un-crowded read leaves the boundary ties alone filling a read: that IS the tie
// cluster at the cursor, served by the over-fetch.)
if (plan.NoRoom)
{
if (shrink > 0) throw refused;
return (default, plan);
}
return (await read(plan).ConfigureAwait(false), plan);
}
}
src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/ReverseRawReader.cs
+63 -16
View File
@@ -58,25 +58,72 @@ internal static class ReverseRawReader
/// <param name="ct">Cancellation.</param>
/// <returns>The page, newest first, and the hint for the next page.</returns>
/// <exception cref="HistoryReadLimitExceededException">One timestamp holds more than the tie budget.</exception>
/// <remarks>
/// Gitea #694: every segment read fits the backend's <see cref="IHistorianDataSource.MaxRawValuesPerRead"/> (a strict
/// gateway's learned ceiling) — the walk's tie budget is bounded by it so its segments are sized for it, and a
/// crowded re-read asking more is cut to it (and counts as truncated, never as complete). A walk that TEACHES the
/// backend its limit (the first read after start-up) and is refused for it is re-run once with segments sized for the
/// new limit — unless <paramref name="retryOnLearnedLimit"/> is false: the HistoryRead page path passes false, so the
/// refusal reaches <see cref="RawPageLimit"/>, which shrinks the PAGE to the limit as well (re-running the walk for a
/// page of N far above the limit cost thousands of segment reads). The bound lookups (a page of one) keep the re-run.
/// The page size itself is never cut here: a short answer would read as the last page. A segment refused because rows
/// just before it crowded the backend's budget (<see cref="HistoryReadLimitExceededException.PreWindowRows"/>) is
/// re-read once that many rows smaller and counts as truncated when full. An uncapped walk (one forward read of the
/// whole window) is never cut: it is answered in full or refused.
/// </remarks>
/// <param name="retryOnLearnedLimit">Re-run the walk once when it taught the backend its limit (see the remarks).</param>
public static async Task<ReverseRawPage> ReadAsync(
IHistorianDataSource source, string tagname, DateTime hi, DateTime lo, uint maxValues, int maxTieCluster,
DateTime nowUtc, ReverseWalkHint hint, CancellationToken ct)
DateTime nowUtc, ReverseWalkHint hint, CancellationToken ct, bool retryOnLearnedLimit = true)
{
var budget = Math.Max(1, maxTieCluster);
var (rows, next) = await ReverseWalk.WalkAsync(
async (start, end, cap, token) =>
{
var samples = (await source.ReadRawAsync(tagname, start, end, (uint)cap, token).ConfigureAwait(false)).Samples;
return new ForwardSegmentRead<DataValueSnapshot>(samples, cap > 0 && samples.Count >= cap);
},
Ticks, tieOrder: null, hi, lo, maxValues, budget, source.TimestampResolutionTicks, nowUtc, hint,
at => new HistoryReadLimitExceededException(budget,
$"A reverse raw read of '{tagname}' at {at:O} found more than {budget} samples within one millisecond; the " +
"backend reads forward only, so the latest of them cannot be selected. Increase " +
"ServerHistorian:MaxTieClusterOverfetch."),
ct)
.ConfigureAwait(false);
return new ReverseRawPage(rows, next.WidthTicks) { Hint = next };
var before = source.MaxRawValuesPerRead;
try
{
return await WalkAsync(before).ConfigureAwait(false);
}
catch (HistoryReadLimitExceededException) when (retryOnLearnedLimit && maxValues > 0
&& source.MaxRawValuesPerRead is var now and > 0 && now != before)
{
// fall through (outside the catch): the walk taught the backend its limit — re-run it sized for that limit once.
// Not an uncapped walk: it is one read of the whole window, which the limit does not change.
}
return await WalkAsync(source.MaxRawValuesPerRead).ConfigureAwait(false);
async Task<ReverseRawPage> WalkAsync(int limit)
{
// A segment asks for at most its budget + 2 rows (ReverseWalk), so the budget is bounded 2 under the limit.
var budget = Math.Max(1, maxTieCluster);
if (limit > 0) budget = Math.Min(budget, Math.Max(1, limit - 2));
var (rows, next) = await ReverseWalk.WalkAsync(
async (start, end, cap, token) =>
{
var asked = limit > 0 && cap > limit ? limit : cap; // cap 0 (the whole window) is never cut
IReadOnlyList<DataValueSnapshot> samples;
try
{
samples = (await source.ReadRawAsync(tagname, start, end, (uint)asked, token).ConfigureAwait(false)).Samples;
}
catch (HistoryReadLimitExceededException ex) when (asked > 0 && ex.PreWindowRows > 0 && asked - ex.PreWindowRows >= 1)
{
// #694: rows just before the segment crowded the backend's budget — read it smaller; a full answer
// then counts as truncated (the walk narrows), never as complete.
asked -= ex.PreWindowRows;
samples = (await source.ReadRawAsync(tagname, start, end, (uint)asked, token).ConfigureAwait(false)).Samples;
}
return new ForwardSegmentRead<DataValueSnapshot>(samples, asked > 0 && samples.Count >= asked);
},
Ticks, tieOrder: null, hi, lo, maxValues, budget, source.TimestampResolutionTicks, nowUtc, hint,
at => new HistoryReadLimitExceededException(budget,
$"A reverse raw read of '{tagname}' at {at:O} found more than {budget} samples within one millisecond; the " +
"backend reads forward only, so the latest of them cannot be selected. Increase " +
(budget < Math.Max(1, maxTieCluster)
? "the backend's row ceiling (HistorianGateway Historian:ReadLimits:MaxValuesCeiling)."
: "ServerHistorian:MaxTieClusterOverfetch.")),
ct)
.ConfigureAwait(false);
return new ReverseRawPage(rows, next.WidthTicks) { Hint = next };
}
}
private static long Ticks(DataValueSnapshot s) => (s.SourceTimestampUtc ?? DateTime.MinValue).Ticks;
src/Server/ZB.MOM.WW.OtOpcUa.Runtime/Historian/ServerHistorianOptions.cs
+14
View File
@@ -106,6 +106,18 @@ public sealed class ServerHistorianOptions
/// </summary>
public bool GatewayAtTimeFromRaw { get; init; } = true;
/// <summary>
/// Gitea #694 (opt-in): HistoryReadEvents never reads events older than <c>now − EventReadHorizon</c> — the earlier
/// bound of every fresh events window is raised to it (both directions, every backend; a window wholly older is
/// <c>GoodNoData</c>). Default <see cref="TimeSpan.Zero"/> = no horizon: every read is exact to its own window. It
/// exists for the Gateway, where proving "no older events" costs a read down to the Unix epoch on
/// <c>Runtime.dbo.Events</c> (~3.8 s live per walked reverse / end-only read; an unfiltered one can hit the
/// gateway's 30 s SQL timeout) and a forward read from a far-past StartTime costs the same. ⚠️ Lossy by design:
/// events older than the horizon are invisible to HistoryReadEvents while it is set. Negative = none (a
/// <see cref="Validate"/> warning).
/// </summary>
public TimeSpan EventReadHorizon { get; init; } = TimeSpan.Zero;
/// <summary>
/// Maximum number of historized nodes served CONCURRENTLY within one HistoryRead batch (arch-review
/// 03/S3). The SDK's <c>HistoryRead*</c> override surface is synchronous, so each arm block-bridges
@@ -196,6 +208,8 @@ public sealed class ServerHistorianOptions
warnings.Add($"ServerHistorian:MaxConcurrentHistoryReadBatches is {MaxConcurrentHistoryReadBatches} — must be > 0; the process-wide HistoryRead batch limiter is disabled.");
if (MaxTimeWeightedRawRows <= 0)
warnings.Add($"ServerHistorian:MaxTimeWeightedRawRows is {MaxTimeWeightedRawRows} — must be > 0; time-weighted aggregate reads on the SqlServer/QuestDb backends stream an unbounded number of raw samples.");
if (EventReadHorizon < TimeSpan.Zero)
warnings.Add($"ServerHistorian:EventReadHorizon is {EventReadHorizon} — negative is treated as no horizon (00:00:00); HistoryReadEvents reads every window in full.");
if (HistoryReadDeadline <= TimeSpan.Zero)
warnings.Add($"ServerHistorian:HistoryReadDeadline is {HistoryReadDeadline} — must be > 0; HistoryRead requests run unbounded (no per-request deadline).");
return warnings;
tests/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions.Tests/Historian/HistorianDataSourceReverseEventsGuardTests.cs
+15
View File
@@ -94,4 +94,19 @@ public sealed class HistorianDataSourceReverseEventsGuardTests
$"{type.FullName} must declare SupportedAggregates itself, not inherit the interface default");
}
}
[Fact]
public void Every_implementation_declares_its_raw_page_limit_itself()
{
// Gitea #694: the default (0 = no limit) is what a strict-ceiling Gateway must NOT report — a wrapper that inherited
// it would bring back BadTooManyOperations for every page larger than the gateway's ceiling.
foreach (var type in ProductionImplementations())
{
var map = type.GetInterfaceMap(typeof(IHistorianDataSource));
var getter = map.TargetMethods[Array.FindIndex(map.InterfaceMethods,
m => m.Name == "get_" + nameof(IHistorianDataSource.MaxRawValuesPerRead))];
getter.DeclaringType.ShouldBe(type,
$"{type.FullName} must declare MaxRawValuesPerRead itself, not inherit the interface default");
}
}
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/GatewayCeilingPagingTests.cs
+499
View File
@@ -0,0 +1,499 @@
using Google.Protobuf.WellKnownTypes;
using Microsoft.Extensions.Logging.Abstractions;
using Opc.Ua;
using Opc.Ua.Server;
using Shouldly;
using Xunit;
using ZB.MOM.WW.HistorianGateway.Contracts.Grpc;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
using SdkHistoryReadResult = Opc.Ua.HistoryReadResult;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests;
/// <summary>
/// Gitea #694 item 1 (live gate F3): a strict gateway (<c>AllowStreamingBeyondCeiling=false</c>) with a LOWERED
/// <c>MaxValuesCeiling</c> (1000) refused ordinary page sizes — <c>NumValuesPerNode = 5000</c>, or an uncapped read over
/// 3,600 rows, came back <c>BadTooManyOperations</c>. The server now sizes the page to the learned ceiling
/// (<see cref="IHistorianDataSource.MaxRawValuesPerRead"/> L = ceiling − 16): the read — page P + boundary ties k + the
/// #692 lookahead row — is at most L, so a first page holds L − 1 values and a resumed one L − k − 1, each with a
/// continuation point. Driven end to end: a REAL SDK server's node manager → the REAL
/// <see cref="GatewayHistorianDataSource"/> → an AVEVA-shaped fake gateway client that refuses an over-ceiling
/// <c>max_values</c> exactly as the real gateway does (<c>InvalidArgument</c> "max_values exceeds the server ceiling of N",
/// wrapped by the client package) and rejects <c>max_values &lt;= 0</c> as the client package does.
/// </summary>
public sealed class GatewayCeilingPagingTests : IDisposable
{
private static CancellationToken Ct => TestContext.Current.CancellationToken;
private static readonly DateTime T0 = new(2026, 10, 12, 10, 0, 0, DateTimeKind.Utc);
private const int Ceiling = 1000;
private const int L = Ceiling - RawAggregateReader.PreWindowAllowance; // 984
private const int FirstPage = L - 1; // 983: page + lookahead fills a read
private const int NextPage = L - 2; // 982: + the one boundary tie a resumed page re-reads (distinct timestamps)
private readonly string _pkiRoot = Path.Combine(Path.GetTempPath(), $"otopcua-gwceiling-{Guid.NewGuid():N}");
private static DateTime At(int seconds) => T0.AddSeconds(seconds);
private static HistorianSample Row(DateTime t, double v, uint quality = 0) =>
new() { Tag = "T", Timestamp = Timestamp.FromDateTime(t), NumericValue = v, OpcQuality = 192, Quality = quality };
/// <summary>
/// AVEVA-shaped ReadRaw over <paramref name="stored"/>: a boundary row at the query start (the value in effect, or a
/// NULL row before the first sample), then <c>[start, end]</c> inclusive ascending, capped at <c>max_values</c>.
/// </summary>
private static FakeHistorianGatewayClient Gateway(IReadOnlyList<(DateTime T, double V)> stored, int? ceiling) => new()
{
MaxValuesCeiling = ceiling,
RawSource = (start, end, max) =>
{
if (max <= 0) throw new ArgumentOutOfRangeException(nameof(max), "max_values must be positive"); // the client package
var rows = new List<HistorianSample>();
if (stored.All(s => s.T != start))
{
var prior = stored.LastOrDefault(s => s.T < start);
rows.Add(prior.T == default
? new HistorianSample { Tag = "T", Timestamp = Timestamp.FromDateTime(start), NumericValue = 0, OpcQuality = 0, Quality = 1 }
: Row(start, prior.V, 133));
}
rows.AddRange(stored.Where(s => s.T >= start && s.T <= end).Select(s => Row(s.T, s.V)));
return rows.Take(max);
},
};
/// <summary>One sample per second, value = seconds, at +0 … +(count − 1).</summary>
private static List<(DateTime T, double V)> PerSecond(int count) =>
Enumerable.Range(0, count).Select(i => (At(i), (double)i)).ToList();
private static int Refusals(FakeHistorianGatewayClient fake) =>
fake.RawCalls.Count(c => fake.MaxValuesCeiling is { } n && c.MaxValues > n);
private static void EveryRequestFits(FakeHistorianGatewayClient fake) =>
fake.RawCalls.ShouldAllBe(c => c.MaxValues > 0 && c.MaxValues <= Ceiling); // never uncapped, never over the ceiling
// ---- the data source's page limit ------------------------------------------------------------------------------------
[Fact]
public async Task The_page_limit_is_unknown_until_a_refusal_names_the_ceiling_and_forgotten_when_the_gateway_streams()
{
var fake = Gateway(PerSecond(3600), Ceiling);
var ds = new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
ds.MaxRawValuesPerRead.ShouldBe(0);
await ds.ReadRawAsync("T", At(0), At(10), 0, Ct);
ds.MaxRawValuesPerRead.ShouldBe(L); // a read of this size asks for exactly the ceiling
(await ds.ReadRawAsync("T", At(0), At(3600), (uint)L, Ct)).Samples.Count.ShouldBe(L);
fake.RawCalls[^1].MaxValues.ShouldBe(Ceiling);
Refusals(fake).ShouldBe(1);
// The operator enabled AllowStreamingBeyondCeiling: a ceiling-full uncapped read re-probes (#691.3) and forgets it.
fake.MaxValuesCeiling = null;
(await ds.ReadRawAsync("T", At(0), At(3600), 0, Ct)).Samples.Count.ShouldBe(3600);
ds.MaxRawValuesPerRead.ShouldBe(0);
}
// ---- through the node manager ----------------------------------------------------------------------------------------
[Fact]
public async Task The_cold_first_read_costs_one_refusal_and_its_page_fits_the_learned_ceiling()
{
var fake = Gateway(PerSecond(3600), Ceiling);
var (host, nm, nodeId) = await BootAsync(fake);
try
{
var first = ReadRaw(nm, nodeId, At(0), At(3600), 5000, null);
first.Error.StatusCode.Code.ShouldBe(StatusCodes.Good);
first.Values.Count.ShouldBe(FirstPage);
first.Cp.ShouldNotBeNull();
// The budget the client asked for (+ lookahead + allowance, refused naming 1000), the ceiling re-read that cannot
// hold the window, then the page sized to the learned ceiling.
fake.RawCalls.Select(c => c.MaxValues).ShouldBe([5000 + 1 + RawAggregateReader.PreWindowAllowance, Ceiling, Ceiling]);
var (rest, _, status) = DrainFrom(nm, nodeId, At(0), At(3600), 5000, first.Cp);
status.Code.ShouldBe(StatusCodes.Good);
first.Values.Concat(rest).ShouldBe(PerSecond(3600).Select(s => s.Item2));
Refusals(fake).ShouldBe(1);
}
finally { await host.DisposeAsync(); }
}
[Theory]
[InlineData(5000u, false)]
[InlineData(0u, false)]
[InlineData(5000u, true)]
[InlineData(0u, true)]
public async Task With_the_ceiling_known_every_page_is_exactly_the_ceiling_page_and_nothing_is_refused(uint n, bool reverse)
{
var stored = PerSecond(3600);
var fake = Gateway(stored, Ceiling);
var (host, nm, nodeId) = await BootAsync(fake);
try
{
ReadRaw(nm, nodeId, At(0), At(3600), 5000, null); // learns the ceiling
fake.RawCalls.Clear();
var (values, pages, status) = reverse
? DrainFrom(nm, nodeId, At(3600), At(-1), n, null)
: DrainFrom(nm, nodeId, At(0), At(3600), n, null);
status.Code.ShouldBe(StatusCodes.Good);
var expected = stored.Select(s => s.Item2);
values.ShouldBe(reverse ? expected.Reverse() : expected);
pages[0].ShouldBe(FirstPage);
pages.Skip(1).SkipLast(1).ShouldAllBe(p => p == NextPage);
pages[^1].ShouldBeGreaterThan(0); // no empty trailing page
Refusals(fake).ShouldBe(0);
EveryRequestFits(fake);
if (!reverse) fake.RawCalls.Count.ShouldBe(pages.Count); // one read per page
}
finally { await host.DisposeAsync(); }
}
[Fact]
public async Task An_uncapped_read_that_fits_the_ceiling_is_still_one_page_without_a_continuation_point()
{
var fake = Gateway(PerSecond(3600), Ceiling);
var (host, nm, nodeId) = await BootAsync(fake);
try
{
var (values, pages, _) = DrainFrom(nm, nodeId, At(0), At(10), 0, null);
values.Count.ShouldBe(10);
pages.Count.ShouldBe(1);
fake.RawCalls.Select(c => c.MaxValues).ShouldBe([int.MaxValue, Ceiling]); // #691.3 unchanged
var again = DrainFrom(nm, nodeId, At(0), At(10), 0, null);
again.Pages.Count.ShouldBe(1);
fake.RawCalls[^1].MaxValues.ShouldBe(Ceiling); // the learned page size, one read
}
finally { await host.DisposeAsync(); }
}
[Fact]
public async Task A_default_gateway_is_unchanged_one_read_and_no_continuation_point()
{
var fake = Gateway(PerSecond(3600), ceiling: null);
var (host, nm, nodeId) = await BootAsync(fake);
try
{
var (values, pages, _) = DrainFrom(nm, nodeId, At(0), At(3600), 5000, null);
values.Count.ShouldBe(3600);
pages.Count.ShouldBe(1);
fake.RawCalls.Select(c => c.MaxValues).ShouldBe([5000 + 1 + RawAggregateReader.PreWindowAllowance]);
}
finally { await host.DisposeAsync(); }
}
/// <summary>
/// A tie cluster straddling a page boundary, at the sizes where the ceiling arithmetic turns: a read can carry L rows;
/// the over-fetch that pages within a cluster reads at most L − 1 (+1 to detect more). Every outcome is exact, or
/// <c>BadTooManyOperations</c> after an exact prefix — never an uncapped or over-ceiling request, never an endless read.
/// </summary>
[Theory]
[InlineData(L - 2, 5000u)]
[InlineData(L - 1, 5000u)]
[InlineData(L, 5000u)]
[InlineData(L + 1, 5000u)]
[InlineData(Ceiling - 17, 5000u)]
[InlineData(Ceiling - 16, 5000u)]
[InlineData(Ceiling + 1, 5000u)]
[InlineData(L - 2, 0u)]
[InlineData(L - 1, 0u)]
[InlineData(L + 1, 0u)]
[InlineData(Ceiling + 1, 0u)]
public async Task A_tie_cluster_at_a_page_boundary_is_exact_or_refused_never_truncated_or_endless(int size, uint n)
{
var stored = PerSecond(500)
.Concat(Enumerable.Range(0, size).Select(i => (At(600), 10_000.0 + i)))
.Concat(Enumerable.Range(0, 1500).Select(i => (At(700 + i), 20_000.0 + i)))
.ToList();
var fake = Gateway(stored, Ceiling);
var (host, nm, nodeId) = await BootAsync(fake);
try
{
ReadRaw(nm, nodeId, At(0), At(5), 5000, null); // learns the ceiling
fake.RawCalls.Clear();
var (values, pages, status) = DrainFrom(nm, nodeId, At(0), At(5000), n, null, maxPages: 60);
var expected = stored.Select(s => s.Item2).ToList();
if (status.Code == StatusCodes.Good) values.ShouldBe(expected);
else
{
status.Code.ShouldBe(StatusCodes.BadTooManyOperations);
values.ShouldBe(expected.Take(values.Count)); // an exact prefix
size.ShouldBeGreaterThan(L - 2, "a cluster that fits the over-fetch is always served");
}
pages.Count.ShouldBeLessThan(60);
EveryRequestFits(fake);
}
finally { await host.DisposeAsync(); }
}
[Fact]
public async Task An_uncapped_cursor_inside_a_tie_cluster_finishes_exactly_when_the_gateway_stops_being_strict()
{
// 100 samples, then 983 ties: the first page ends inside the cluster. Then the operator enables streaming and the
// data source forgets the ceiling (L back to 0) — the uncapped cursor must still finish exactly, not loop.
var stored = PerSecond(100)
.Concat(Enumerable.Range(0, L - 1).Select(i => (At(200), 10_000.0 + i)))
.Concat(Enumerable.Range(0, 300).Select(i => (At(300 + i), 20_000.0 + i)))
.ToList();
var fake = Gateway(stored, Ceiling);
var ds = new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
var (host, nm, nodeId) = await BootAsync(fake, ds);
try
{
ReadRaw(nm, nodeId, At(0), At(5), 5000, null); // learns the ceiling
var first = ReadRaw(nm, nodeId, At(0), At(5000), 0, null);
first.Values.Count.ShouldBe(FirstPage);
first.Cp.ShouldNotBeNull();
fake.MaxValuesCeiling = null;
await ds.ReadRawAsync("T", At(0), At(5000), 0, Ct); // a ceiling-full read re-probes and forgets
ds.MaxRawValuesPerRead.ShouldBe(0);
var (rest, pages, status) = DrainFrom(nm, nodeId, At(0), At(5000), 0, first.Cp, maxPages: 20);
status.Code.ShouldBe(StatusCodes.Good);
first.Values.Concat(rest).ShouldBe(stored.Select(s => s.Item2));
pages.Count.ShouldBeLessThan(20);
}
finally { await host.DisposeAsync(); }
}
[Fact]
public async Task A_ceiling_lowered_mid_page_through_is_re_learned_and_the_read_stays_exact()
{
var stored = PerSecond(3600);
var fake = Gateway(stored, Ceiling);
var (host, nm, nodeId) = await BootAsync(fake);
try
{
var got = new List<double>();
var p1 = ReadRaw(nm, nodeId, At(0), At(3600), 5000, null);
got.AddRange(p1.Values);
var p2 = ReadRaw(nm, nodeId, At(0), At(3600), 5000, p1.Cp);
got.AddRange(p2.Values);
fake.MaxValuesCeiling = 500; // the operator lowered it
fake.RawCalls.Clear();
var (rest, pages, status) = DrainFrom(nm, nodeId, At(0), At(3600), 5000, p2.Cp, maxPages: 20);
status.Code.ShouldBe(StatusCodes.Good);
got.AddRange(rest);
got.ShouldBe(stored.Select(s => s.Item2));
pages.ShouldAllBe(p => p <= 500 - RawAggregateReader.PreWindowAllowance);
Refusals(fake).ShouldBe(1); // the one that names the new ceiling
}
finally { await host.DisposeAsync(); }
}
[Fact]
public async Task Bounding_values_page_within_the_ceiling()
{
var stored = PerSecond(3600);
var fake = Gateway(stored, Ceiling);
var (host, nm, nodeId) = await BootAsync(fake);
try
{
ReadRaw(nm, nodeId, At(0), At(5), 5000, null); // learns the ceiling
fake.RawCalls.Clear();
// Start bound = the sample at +10 (before StartTime), end bound = the sample at +3001 (after EndTime).
var start = At(10).AddMilliseconds(500);
var end = At(3000).AddMilliseconds(500);
var (values, pages, status) = DrainFrom(nm, nodeId, start, end, 5000, null, returnBounds: true);
status.Code.ShouldBe(StatusCodes.Good);
values.ShouldBe(Enumerable.Range(10, 2992).Select(i => (double)i));
pages.ShouldAllBe(p => p <= L);
Refusals(fake).ShouldBe(0);
}
finally { await host.DisposeAsync(); }
}
[Theory]
[InlineData(1000, 100_000u, 10)]
[InlineData(1000, 5000u, 10)]
[InlineData(20, 3000u, 40)]
public async Task A_cold_end_only_read_with_a_huge_page_shrinks_the_page_not_the_walk(int ceiling, uint n, int maxReads)
{
// Review F2: the walk used to re-run itself at the learned ceiling while keeping the page at N — a cold end-only read
// with ceiling 20 and N = 3000 cost 3,023 gateway reads in ONE HistoryRead (ceiling 1000, N = 100,000: 207). The
// refusal now reaches the page planner, which shrinks the page to the ceiling.
var dense = Enumerable.Range(0, 200_000).Select(i => (T0.AddMilliseconds(i), (double)i)).ToList();
var fake = Gateway(dense, ceiling);
var (host, nm, nodeId) = await BootAsync(fake);
try
{
var r = ReadRaw(nm, nodeId, DateTime.MinValue, T0.AddMilliseconds(200_000), n, null);
r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good);
r.Cp.ShouldNotBeNull();
var page = ceiling - RawAggregateReader.PreWindowAllowance - 1;
r.Values.ShouldBe(Enumerable.Range(200_000 - r.Values.Count, r.Values.Count).Reverse().Select(i => (double)i));
r.Values.Count.ShouldBeLessThanOrEqualTo(page);
fake.RawCalls.Count.ShouldBeLessThanOrEqualTo(maxReads);
}
finally { await host.DisposeAsync(); }
}
// ---- F1: a window starting at/before the epoch gets no look-back (review of #694) -------------------------------------
private static List<(DateTime T, double V)> FromEpoch(int count) =>
Enumerable.Range(1, count).Select(i => (DateTime.UnixEpoch.AddSeconds(i), (double)i)).ToList();
[Fact]
public async Task A_ceiling_truncated_read_from_the_epoch_is_never_reported_complete()
{
// AVEVA's synthetic row lands AT the epoch start (no 1 ms look-back there). It used to count as "past the window
// end", so a ceiling-truncated uncapped read answered its first 999 rows, Good, as if that were everything.
var fake = Gateway(FromEpoch(3000), Ceiling);
var ds = new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
await Should.ThrowAsync<HistoryReadLimitExceededException>(() =>
ds.ReadRawAsync("T", DateTime.UnixEpoch, DateTime.UnixEpoch.AddDays(1), 0, Ct));
}
[Fact]
public async Task An_uncapped_read_from_the_epoch_pages_every_sample()
{
var stored = FromEpoch(3000);
var fake = Gateway(stored, Ceiling);
var (host, nm, nodeId) = await BootAsync(fake);
try
{
var (values, pages, status) = DrainFrom(nm, nodeId, DateTime.UnixEpoch, DateTime.UnixEpoch.AddDays(1), 0, null);
status.Code.ShouldBe(StatusCodes.Good);
values.ShouldBe(stored.Select(s => s.Item2));
pages.Count.ShouldBeGreaterThan(1);
}
finally { await host.DisposeAsync(); }
}
[Fact]
public async Task An_end_only_read_far_past_epoch_data_returns_the_latest_values()
{
var stored = FromEpoch(3000);
var fake = Gateway(stored, Ceiling);
var (host, nm, nodeId) = await BootAsync(fake);
try
{
ReadRaw(nm, nodeId, DateTime.UnixEpoch, DateTime.UnixEpoch.AddSeconds(5), 5000, null); // learns the ceiling
var r = ReadRaw(nm, nodeId, DateTime.MinValue, DateTime.UnixEpoch.AddDays(30), 10, null);
r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good);
r.Values.ShouldBe(Enumerable.Range(2991, 10).Reverse().Select(i => (double)i)); // the LATEST ten, newest first
}
finally { await host.DisposeAsync(); }
}
// ---- F3: rows crowding the look-back before a page start ---------------------------------------------------------------
[Theory]
[InlineData(5000u)]
[InlineData(0u)]
[InlineData((uint)L)]
[InlineData(900u)]
public async Task Ties_just_before_a_page_start_are_read_around_not_refused(uint n)
{
// 962 samples, 20 ties at X, one sample at X + 1 ms: the first page (983) ends exactly at X + 1 ms. The next page's
// 1 ms look-back holds the 20 ties — more than the 16-row allowance, and a ceiling-full page leaves no room for the
// #687 re-read, so it used to fail BadTooManyOperations. The source now names the crowding rows and the page is
// re-read that much smaller.
var x = At(1000);
var stored = Enumerable.Range(0, 962).Select(i => (At(i), (double)i))
.Concat(Enumerable.Range(0, 20).Select(i => (x, 962.0 + i)))
.Append((x.AddMilliseconds(1), 982.0))
.Concat(Enumerable.Range(0, 3000).Select(i => (x.AddSeconds(1 + i), 983.0 + i)))
.ToList();
var fake = Gateway(stored, Ceiling);
var (host, nm, nodeId) = await BootAsync(fake);
try
{
ReadRaw(nm, nodeId, At(0), At(5), 5000, null); // learns the ceiling
fake.RawCalls.Clear();
var (values, _, status) = DrainFrom(nm, nodeId, At(0), x.AddDays(1), n, null);
status.Code.ShouldBe(StatusCodes.Good);
values.ShouldBe(stored.Select(s => s.Item2));
Refusals(fake).ShouldBe(0);
EveryRequestFits(fake);
}
finally { await host.DisposeAsync(); }
}
// ---- helpers ---------------------------------------------------------------------------------------------------------
private async Task<(OpcUaApplicationHost Host, OtOpcUaNodeManager Nm, NodeId NodeId)> BootAsync(
FakeHistorianGatewayClient fake, GatewayHistorianDataSource? ds = null)
{
var started = await OpcUaHostStart.StartAsync(
port => new OpcUaApplicationHost(
new OpcUaApplicationHostOptions
{
ApplicationName = "OtOpcUa.GatewayCeilingPagingTest",
ApplicationUri = $"urn:OtOpcUa.GatewayCeilingPagingTest:{Guid.NewGuid():N}",
OpcUaPort = port,
PublicHostname = "localhost",
PkiStoreRoot = _pkiRoot,
},
NullLogger<OpcUaApplicationHost>.Instance),
() => new OtOpcUaSdkServer(), Ct);
var nm = started.Server.NodeManager!;
nm.HistoryContinuationStore = new InMemoryHistoryContinuationStore();
nm.HistorianDataSource = ds ?? new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
nm.EnsureVariable("eq-1/pv", parentFolderNodeId: null, displayName: "PV", dataType: "Double",
writable: false, historianTagname: "T", realm: AddressSpaceRealm.Uns);
return (started.Host, nm, nm.TryGetVariable("eq-1/pv")!.NodeId);
}
/// <summary>Pages until no continuation point or a Bad status; the page sizes and the final status.</summary>
private static (List<double> Values, List<int> Pages, StatusCode Status) DrainFrom(
OtOpcUaNodeManager nm, NodeId nodeId, DateTime start, DateTime end, uint max, byte[]? cp, int maxPages = 20,
bool returnBounds = false)
{
var values = new List<double>();
var pages = new List<int>();
ServiceResult error;
do
{
(var page, error, cp) = ReadRaw(nm, nodeId, start, end, max, cp, returnBounds);
values.AddRange(page);
pages.Add(page.Count);
pages.Count.ShouldBeLessThanOrEqualTo(maxPages, "paging must terminate");
}
while (cp is not null && StatusCode.IsGood(error.StatusCode));
return (values, pages, error.StatusCode);
}
private static (IReadOnlyList<double> Values, ServiceResult Error, byte[]? Cp) ReadRaw(
OtOpcUaNodeManager nm, NodeId nodeId, DateTime start, DateTime end, uint max, byte[]? inboundCp, bool returnBounds = false)
{
var context = new OperationContext(new RequestHeader(), secureChannelContext: null, RequestType.HistoryRead, identity: null);
var details = new ReadRawModifiedDetails
{
ReturnBounds = returnBounds,
IsReadModified = false,
StartTime = start,
EndTime = end,
NumValuesPerNode = max,
};
var nodesToRead = new List<HistoryReadValueId> { new() { NodeId = nodeId, ContinuationPoint = inboundCp } };
var results = new List<SdkHistoryReadResult> { null! };
var errors = new List<ServiceResult> { null! };
nm.HistoryRead(context, details, TimestampsToReturn.Both, releaseContinuationPoints: false, nodesToRead, results, errors);
var values = new List<double>();
if (results[0]?.HistoryData is { } ho && !ExtensionObject.IsNull(ho))
foreach (var dv in ((HistoryData)ExtensionObject.ToEncodeable(ho)).DataValues)
values.Add(Convert.ToDouble(dv.Value));
return (values, errors[0], results[0]?.ContinuationPoint);
}
public void Dispose()
{
if (Directory.Exists(_pkiRoot))
{
try { Directory.Delete(_pkiRoot, recursive: true); }
catch { /* best-effort */ }
}
}
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/GatewayRawBoundaryRowTests.cs
+24 -5
View File
@@ -222,17 +222,36 @@ public sealed class GatewayRawBoundaryRowTests
}
[Fact]
public async Task A_margin_still_crowding_the_re_read_fails_the_node_as_a_request_limit()
public async Task A_margin_crowding_a_truncated_read_is_re_read_with_growing_room()
{
// 200 rows in the margin: the single bounded re-read (5 + 21 + 16) is still short of the cap because of them.
// #694 F4: 200 rows in the margin. The first read (5 + 16) saw only 21 of them, so a single re-read sized from that
// count (5 + 21 + 16) was still short and the node failed — with no ceiling at all. The room now at least doubles per
// re-read (21, 42, 84, 168, 336), so the window is served in a bounded number of reads.
var margin = Enumerable.Range(0, 200).Select(i => (At(30).AddTicks(-1 - i), (double)i, false));
var fake = Aveva(margin.Append((At(35), 1d, false)).ToList());
var ds = Source(fake);
await Should.ThrowAsync<HistoryReadLimitExceededException>(() => ds.ReadRawAsync("T", At(30), At(40), 5, Ct));
var r = await ds.ReadRawAsync("T", At(30), At(40), 5, Ct);
fake.ReadRawCallCount.ShouldBe(2); // one bounded re-read, never more
fake.RawStreamsCompleted.ShouldBe(2);
r.Samples.Select(s => s.Value).ShouldBe([1d]);
fake.ReadRawCallCount.ShouldBe(6);
fake.RawStreamsCompleted.ShouldBe(6);
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
}
[Fact]
public async Task A_margin_still_crowding_the_bounded_re_reads_fails_the_node_as_a_request_limit()
{
// 20,000 rows in the margin: past the bounded re-reads (21 · 2^8 ≈ 5,400 rows of room).
var margin = Enumerable.Range(0, 20_000).Select(i => (At(30).AddTicks(-1 - i), (double)i, false));
var fake = Aveva(margin.Append((At(35), 1d, false)).ToList());
var ds = Source(fake);
var ex = await Should.ThrowAsync<HistoryReadLimitExceededException>(() => ds.ReadRawAsync("T", At(30), At(40), 5, Ct));
ex.PreWindowRows.ShouldBeGreaterThan(0);
fake.ReadRawCallCount.ShouldBe(9); // the read + MaxCrowdedReReads, never more
fake.RawStreamsCompleted.ShouldBe(9);
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
}
tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests.csproj
+5
View File
@@ -31,4 +31,9 @@
<ProjectReference Include="..\..\..\src\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway\ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.csproj"/>
</ItemGroup>
<ItemGroup>
<!-- #694: the node manager's HistoryRead paging over the REAL gateway data source boots a real SDK server. -->
<Compile Include="..\..\Server\ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests\OpcUaHostStart.cs" Link="OpcUaHostStart.cs"/>
</ItemGroup>
</Project>
tests/Server/ZB.MOM.WW.OtOpcUa.Host.Tests/OpcUa/OtOpcUaServerHostedServiceHistoryUpdateTests.cs
+12
View File
@@ -84,6 +84,18 @@ public sealed class OtOpcUaServerHostedServiceHistoryUpdateTests
Service(view: null, config: config).HonorReturnBounds.ShouldBeFalse();
}
/// <summary>#694: the events horizon is bound from <c>ServerHistorian:EventReadHorizon</c>, default none.</summary>
[Fact]
public void The_event_read_horizon_comes_from_options_and_defaults_off()
{
Service(view: null).EventReadHorizon.ShouldBe(TimeSpan.Zero);
var config = new ConfigurationBuilder().AddInMemoryCollection(new Dictionary<string, string?>
{
["ServerHistorian:EventReadHorizon"] = "365.00:00:00",
}).Build();
Service(view: null, config: config).EventReadHorizon.ShouldBe(TimeSpan.FromDays(365));
}
private static OtOpcUaServerHostedService Service(
IHistoryUpdateRoleView? view, IDriverPrimaryRoleSource? primary = null, IConfiguration? config = null,
TimeProvider? clock = null) => new(
tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/EventHorizonTests.cs
+63
View File
@@ -0,0 +1,63 @@
using Shouldly;
using Xunit;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests;
/// <summary>Gitea #694: <see cref="EventHorizon.TryClamp"/> — the pure window arithmetic of <c>ServerHistorian:EventReadHorizon</c>.</summary>
public sealed class EventHorizonTests
{
private static readonly DateTime Now = new(2026, 10, 13, 12, 0, 0, DateTimeKind.Utc);
private static readonly TimeSpan H = TimeSpan.FromDays(30);
private static readonly DateTime Floor = Now - H;
[Fact]
public void No_horizon_changes_nothing()
{
var (s, e) = (DateTime.MinValue, Now);
EventHorizon.TryClamp(ref s, ref e, reverse: false, TimeSpan.Zero, Now).ShouldBeTrue();
(s, e).ShouldBe((DateTime.MinValue, Now));
EventHorizon.TryClamp(ref s, ref e, reverse: false, TimeSpan.FromDays(-1), Now).ShouldBeTrue();
(s, e).ShouldBe((DateTime.MinValue, Now));
}
[Fact]
public void A_forward_window_starts_at_the_floor()
{
var (s, e) = (DateTime.UnixEpoch, Now);
EventHorizon.TryClamp(ref s, ref e, reverse: false, H, Now).ShouldBeTrue();
(s, e).ShouldBe((Floor, Now));
var (s2, e2) = (Floor.AddDays(1), Now);
EventHorizon.TryClamp(ref s2, ref e2, reverse: false, H, Now).ShouldBeTrue();
s2.ShouldBe(Floor.AddDays(1)); // already inside: unchanged
}
[Fact]
public void A_reverse_window_keeps_the_floor_instant()
{
var (s, e) = (Now, new DateTime(0, DateTimeKind.Utc));
EventHorizon.TryClamp(ref s, ref e, reverse: true, H, Now).ShouldBeTrue();
(s, e).ShouldBe((Now, Floor.AddTicks(-1))); // (end, start] — the floor itself is read
}
[Fact]
public void A_window_wholly_before_the_floor_is_empty()
{
var (s, e) = (Floor.AddDays(-2), Floor);
EventHorizon.TryClamp(ref s, ref e, reverse: false, H, Now).ShouldBeFalse(); // [start, end) ends at the floor
var (rs, re) = (Floor.AddTicks(-1), Floor.AddDays(-2));
EventHorizon.TryClamp(ref rs, ref re, reverse: true, H, Now).ShouldBeFalse();
var (ok, oe) = (Floor, Floor.AddDays(-2));
EventHorizon.TryClamp(ref ok, ref oe, reverse: true, H, Now).ShouldBeTrue(); // (…, floor] holds the floor
}
[Fact]
public void A_horizon_longer_than_the_calendar_is_no_horizon()
{
var (s, e) = (DateTime.MinValue, Now);
EventHorizon.TryClamp(ref s, ref e, reverse: false, TimeSpan.MaxValue, Now).ShouldBeTrue();
s.ShouldBe(DateTime.MinValue);
}
}
tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadEventsHorizonTests.cs
+231
View File
@@ -0,0 +1,231 @@
using Microsoft.Extensions.Logging.Abstractions;
using Opc.Ua;
using Opc.Ua.Server;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using HistorianRead = ZB.MOM.WW.OtOpcUa.Core.Abstractions.HistoryReadResult;
using SdkHistoryReadResult = Opc.Ua.HistoryReadResult;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests;
/// <summary>
/// Gitea #694 item 2 (live gate F4): a walked-back events read on the Gateway proves "nothing older" with one forward read
/// down to the Unix epoch (#691) — ~3.8 s live on <c>Runtime.dbo.Events</c>, and an unfiltered deep read hit the gateway's
/// 30 s SQL timeout. The opt-in <c>ServerHistorian:EventReadHorizon</c> (<see cref="OtOpcUaNodeManager.EventReadHorizon"/>)
/// bounds every events read at <c>now − horizon</c>, in both directions; off (the default) nothing changes. The backend is
/// shaped like the Gateway's SQL event read: forward only, <c>[start, end]</c> inclusive, refuses <c>start &gt; end</c>
/// (counted), records every requested window.
/// </summary>
public sealed class NodeManagerHistoryReadEventsHorizonTests : IDisposable
{
private static CancellationToken Ct => TestContext.Current.CancellationToken;
private static readonly DateTime Now = new(2026, 10, 13, 12, 0, 0, DateTimeKind.Utc);
private static readonly TimeSpan Horizon = TimeSpan.FromDays(30);
private static readonly DateTime Floor = Now - Horizon;
private readonly string _pkiRoot = Path.Combine(Path.GetTempPath(), $"otopcua-eventhorizon-{Guid.NewGuid():N}");
/// <summary>Five recent events (the last five minutes) — a source with little history, as in the live gate.</summary>
private static List<HistoricalEvent> Recent() =>
Enumerable.Range(0, 5).Select(i => Ev($"r{i}", Now.AddMinutes(-5 + i))).ToList();
[Fact]
public async Task Without_a_horizon_a_walked_reverse_read_proves_the_rest_empty_down_to_the_epoch()
{
// The positive control: today's exact behaviour, and the cost the horizon removes.
var (host, nm, node, gw) = await SetupAsync(Recent());
try
{
var r = ReadEvents(nm, node, max: 10, start: DateTime.MinValue, end: Now);
r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good);
r.Ids.ShouldBe(["r4", "r3", "r2", "r1", "r0"]);
gw.Requests.Min(q => q.Start).ShouldBe(DateTime.UnixEpoch);
gw.BadRequests.ShouldBe(0);
}
finally { await host.DisposeAsync(); }
}
[Fact]
public async Task With_a_horizon_the_walk_never_reads_before_it_and_answers_the_same()
{
var (host, nm, node, gw) = await SetupAsync(Recent());
try
{
nm.EventReadHorizon = Horizon;
var r = ReadEvents(nm, node, max: 10, start: DateTime.MinValue, end: Now);
r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good);
r.Ids.ShouldBe(["r4", "r3", "r2", "r1", "r0"]);
r.Cp.ShouldBeNull();
// Requests are rounded outward to the millisecond, so the earliest one may start up to 1 ms before the floor.
gw.Requests.Min(q => q.Start).ShouldBeGreaterThanOrEqualTo(Floor.AddMilliseconds(-1));
gw.BadRequests.ShouldBe(0);
}
finally { await host.DisposeAsync(); }
}
[Fact]
public async Task Events_older_than_the_horizon_are_not_returned_in_either_direction()
{
var events = new List<HistoricalEvent> { Ev("old", Floor.AddDays(-10)), Ev("edge", Floor), Ev("new", Now.AddDays(-1)) };
var (host, nm, node, gw) = await SetupAsync(events);
try
{
nm.EventReadHorizon = Horizon;
var reverse = ReadEvents(nm, node, max: 10, start: Now, end: DateTime.UnixEpoch);
reverse.Ids.ShouldBe(["new", "edge"]); // the floor instant itself is inside
var forward = ReadEvents(nm, node, max: 10, start: DateTime.UnixEpoch, end: Now);
forward.Ids.ShouldBe(["edge", "new"]);
gw.Requests.ShouldAllBe(q => q.Start >= Floor.AddMilliseconds(-1));
nm.EventReadHorizon = TimeSpan.Zero; // off again: exact to the window
ReadEvents(nm, node, max: 10, start: DateTime.UnixEpoch, end: Now).Ids.ShouldBe(["old", "edge", "new"]);
}
finally { await host.DisposeAsync(); }
}
[Fact]
public async Task A_window_wholly_older_than_the_horizon_is_GoodNoData_and_never_sent()
{
var (host, nm, node, gw) = await SetupAsync([Ev("old", Floor.AddDays(-10))]);
try
{
nm.EventReadHorizon = Horizon;
var forward = ReadEvents(nm, node, max: 10, start: Floor.AddDays(-20), end: Floor.AddDays(-1));
forward.Error.StatusCode.Code.ShouldBe(StatusCodes.Good);
forward.Status.Code.ShouldBe(StatusCodes.GoodNoData);
forward.Ids.ShouldBeEmpty();
var reverse = ReadEvents(nm, node, max: 10, start: Floor.AddDays(-1), end: Floor.AddDays(-20));
reverse.Status.Code.ShouldBe(StatusCodes.GoodNoData);
gw.Requests.ShouldBeEmpty();
}
finally { await host.DisposeAsync(); }
}
[Fact]
public async Task A_page_through_under_a_horizon_keeps_its_floor_across_continuation_points()
{
var events = Enumerable.Range(0, 25).Select(i => Ev($"e{i:00}", Now.AddMinutes(-i))).ToList();
events.Add(Ev("old", Floor.AddDays(-1)));
var (host, nm, node, gw) = await SetupAsync(events);
try
{
nm.EventReadHorizon = Horizon;
var all = new List<string>();
byte[]? cp = null;
var pages = 0;
do
{
var r = ReadEvents(nm, node, max: 10, start: DateTime.MinValue, end: Now, inboundCp: cp);
r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good);
all.AddRange(r.Ids);
cp = r.Cp;
(++pages).ShouldBeLessThan(10);
}
while (cp is not null);
all.ShouldBe(Enumerable.Range(0, 25).Select(i => $"e{i:00}"));
gw.Requests.ShouldAllBe(q => q.Start >= Floor.AddMilliseconds(-1));
}
finally { await host.DisposeAsync(); }
}
// ---- helpers ---------------------------------------------------------------------------------------------------------
private static HistoricalEvent Ev(string id, DateTime at) => new(id, "src", at, at, "m", 500);
private async Task<(OpcUaApplicationHost Host, OtOpcUaNodeManager Nm, NodeId Node, GatewayShapedEventSource Gw)> SetupAsync(
List<HistoricalEvent> events)
{
var started = await OpcUaHostStart.StartAsync(
port => new OpcUaApplicationHost(
new OpcUaApplicationHostOptions
{
ApplicationName = "OtOpcUa.EventHorizonTest",
ApplicationUri = $"urn:OtOpcUa.EventHorizonTest:{Guid.NewGuid():N}",
OpcUaPort = port,
PublicHostname = "localhost",
PkiStoreRoot = _pkiRoot,
},
NullLogger<OpcUaApplicationHost>.Instance),
() => new OtOpcUaSdkServer(), Ct);
var nm = started.Server.NodeManager!;
nm.HistoryContinuationStore = new InMemoryHistoryContinuationStore();
nm.Clock = new FixedClock(Now);
var gw = new GatewayShapedEventSource(events);
nm.HistorianDataSource = gw;
nm.EnsureFolder("eq-evt", parentNodeId: null, displayName: "Equipment", realm: AddressSpaceRealm.Uns);
nm.MaterialiseAlarmCondition("alarm-1", "eq-evt", "HighTemp", "OffNormalAlarm", severity: 700, realm: AddressSpaceRealm.Uns);
return (started.Host, nm, nm.TryGetFolder("eq-evt")!.NodeId, gw);
}
private static (List<string> Ids, ServiceResult Error, StatusCode Status, byte[]? Cp) ReadEvents(
OtOpcUaNodeManager nm, NodeId node, uint max, DateTime start, DateTime end, byte[]? inboundCp = null)
{
var context = new OperationContext(new RequestHeader(), secureChannelContext: null, RequestType.HistoryRead, identity: null);
var filter = new EventFilter();
filter.SelectClauses.Add(new SimpleAttributeOperand(ObjectTypeIds.BaseEventType, new QualifiedName("EventId"))
{
AttributeId = Attributes.Value,
});
var details = new ReadEventDetails { StartTime = start, EndTime = end, NumValuesPerNode = max, Filter = filter };
var nodesToRead = new List<HistoryReadValueId> { new() { NodeId = node, ContinuationPoint = inboundCp } };
var results = new List<SdkHistoryReadResult> { null! };
var errors = new List<ServiceResult> { null! };
nm.HistoryRead(context, details, TimestampsToReturn.Both, false, nodesToRead, results, errors);
var ids = new List<string>();
if (results[0]?.HistoryData is { } ho && !ExtensionObject.IsNull(ho))
foreach (var e in ((HistoryEvent)ExtensionObject.ToEncodeable(ho)).Events)
ids.Add(System.Text.Encoding.UTF8.GetString((byte[])e.EventFields[0].Value));
return (ids, errors[0], results[0]?.StatusCode ?? StatusCodes.Good, results[0]?.ContinuationPoint);
}
/// <summary>The Gateway's SQL event read: forward only, <c>[start, end]</c> inclusive, time-ordered, capped + flag.</summary>
private sealed class GatewayShapedEventSource(List<HistoricalEvent> events) : IHistorianDataSource
{
public List<(DateTime Start, DateTime End, int Max)> Requests { get; } = [];
public int BadRequests { get; private set; }
public bool SupportsReverseEvents => false;
public Task<HistoricalEventsResult> ReadEventsAsync(string? n, DateTime s, DateTime e, int m, CancellationToken ct)
{
Requests.Add((s, e, m));
if (s > e) BadRequests++; // the gateway's ValidateRange
var window = events.Where(x => x.EventTimeUtc >= s && x.EventTimeUtc <= e).OrderBy(x => x.EventTimeUtc).ToList();
var cap = m <= 0 ? int.MaxValue : m;
return Task.FromResult(new HistoricalEventsResult(window.Take(cap).ToList(), window.Count > cap ? new byte[] { 1 } : null));
}
public Task<HistoricalEventsResult> ReadEventsAsync(string? n, DateTime s, DateTime e, int m, bool reverse, CancellationToken ct) =>
reverse ? throw new NotSupportedException("forward only") : ReadEventsAsync(n, s, e, m, ct);
public Task<HistorianRead> ReadRawAsync(string r, DateTime s, DateTime e, uint m, CancellationToken ct) =>
NullHistorianDataSource.Instance.ReadRawAsync(r, s, e, m, ct);
public Task<HistorianRead> ReadProcessedAsync(string r, DateTime s, DateTime e, TimeSpan i, HistoryAggregateType a, CancellationToken ct) =>
NullHistorianDataSource.Instance.ReadProcessedAsync(r, s, e, i, a, ct);
public Task<HistorianRead> ReadAtTimeAsync(string r, IReadOnlyList<DateTime> t, CancellationToken ct) =>
NullHistorianDataSource.Instance.ReadAtTimeAsync(r, t, ct);
public HistorianHealthSnapshot GetHealthSnapshot() => NullHistorianDataSource.Instance.GetHealthSnapshot();
public void Dispose() { }
}
private sealed class FixedClock(DateTime now) : TimeProvider
{
public override DateTimeOffset GetUtcNow() => new(now);
}
public void Dispose()
{
if (Directory.Exists(_pkiRoot))
{
try { Directory.Delete(_pkiRoot, recursive: true); }
catch { /* best-effort */ }
}
}
}
tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/RawPageLimitTests.cs
+142
View File
@@ -0,0 +1,142 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests;
/// <summary>
/// Gitea #694: <see cref="RawPageLimit"/> — the page plan (P = min(want, L − k − 1), request = P + k + 1 ≤ L) and the one
/// re-plan after a refusal. The end-to-end behaviour over the real Gateway data source is pinned in
/// <c>GatewayCeilingPagingTests</c>.
/// </summary>
public sealed class RawPageLimitTests
{
[Theory]
// want, k, L ⇒ page, request, noRoom
[InlineData(5000u, 0, 0, 5000u, 5001u, false)] // no limit: #692 unchanged (page + ties + lookahead)
[InlineData(5000u, 3, 0, 5000u, 5004u, false)]
[InlineData(0u, 3, 0, 0u, 0u, false)] // no limit, uncapped: one uncapped read
[InlineData(5000u, 0, 984, 983u, 984u, false)] // over the limit: the request fills the limit exactly
[InlineData(5000u, 10, 984, 973u, 984u, false)] // boundary ties take room from the page
[InlineData(10u, 10, 984, 10u, 21u, false)] // under the limit: unchanged
[InlineData(0u, 0, 984, 983u, 984u, false)] // uncapped: a limit-sized page
[InlineData(0u, 982, 984, 1u, 984u, false)]
[InlineData(5000u, 983, 984, 0u, 0u, true)] // the ties alone fill a read: never a 0 (uncapped) request
[InlineData(5000u, 5000, 984, 0u, 0u, true)] // never an underflow
[InlineData(5000u, 0, 1, 0u, 0u, true)]
public void Plan_fits_page_ties_and_lookahead_into_the_limit(uint want, int k, int limit, uint page, uint request, bool noRoom)
{
var plan = RawPageLimit.Plan(want, k, limit);
plan.PageCap.ShouldBe(page);
plan.RequestCap.ShouldBe(request);
plan.NoRoom.ShouldBe(noRoom);
if (limit > 0 && !noRoom) plan.RequestCap.ShouldBeLessThanOrEqualTo((uint)limit);
}
[Fact]
public async Task A_read_that_teaches_the_limit_is_re_planned_once_at_it()
{
var source = new LimitSource();
var asked = new List<uint>();
var (page, plan) = await RawPageLimit.ReadAsync(source, 5000u, 0, p =>
{
asked.Add(p.RequestCap);
if (p.RequestCap > 984) { source.Limit = 984; throw new HistoryReadLimitExceededException(1000, "ceiling"); }
return Task.FromResult("ok");
});
page.ShouldBe("ok");
plan.PageCap.ShouldBe(983u);
asked.ShouldBe([5001u, 984u]);
}
[Fact]
public async Task A_crowded_refusal_shrinks_the_page_by_the_pre_window_rows()
{
var source = new LimitSource { Limit = 984 };
var asked = new List<uint>();
var (_, plan) = await RawPageLimit.ReadAsync(source, 0u, 2, p =>
{
asked.Add(p.RequestCap);
if (asked.Count == 1) throw new HistoryReadLimitExceededException(1000, "crowded") { PreWindowRows = 21 };
return Task.FromResult("ok");
});
asked.ShouldBe([984u, 963u]);
plan.PageCap.ShouldBe(960u);
}
[Fact]
public async Task A_refusal_at_an_unchanged_limit_without_crowding_is_the_answer()
{
var source = new LimitSource { Limit = 984 };
var calls = 0;
await Should.ThrowAsync<HistoryReadLimitExceededException>(() => RawPageLimit.ReadAsync<string>(source, 5000u, 0, _ =>
{
calls++;
throw new HistoryReadLimitExceededException(1000, "tie cluster");
}));
calls.ShouldBe(1);
}
[Fact]
public async Task A_second_refusal_is_not_retried_again()
{
var source = new LimitSource();
var calls = 0;
await Should.ThrowAsync<HistoryReadLimitExceededException>(() => RawPageLimit.ReadAsync<string>(source, 0u, 0, _ =>
{
calls++;
source.Limit = 984 - calls; // even if the limit keeps moving
throw new HistoryReadLimitExceededException(1000, "refused") { PreWindowRows = 5 };
}));
calls.ShouldBe(2);
}
[Fact]
public async Task No_room_is_returned_without_a_read()
{
var source = new LimitSource { Limit = 10 };
var calls = 0;
var (page, plan) = await RawPageLimit.ReadAsync(source, 5u, 9, _ => { calls++; return Task.FromResult("x"); });
plan.NoRoom.ShouldBeTrue();
page.ShouldBeNull();
calls.ShouldBe(0);
}
[Fact]
public async Task Other_failures_are_not_retried()
{
var source = new LimitSource();
var calls = 0;
await Should.ThrowAsync<InvalidOperationException>(() => RawPageLimit.ReadAsync<string>(source, 5000u, 0, _ =>
{
calls++;
source.Limit = 984;
throw new InvalidOperationException("backend fault");
}));
calls.ShouldBe(1);
}
/// <summary>Only <see cref="IHistorianDataSource.MaxRawValuesPerRead"/> matters here.</summary>
private sealed class LimitSource : IHistorianDataSource
{
public int Limit;
public int MaxRawValuesPerRead => Limit;
public Task<HistoryReadResult> ReadRawAsync(string fullReference, DateTime startUtc, DateTime endUtc, uint maxValuesPerNode,
CancellationToken cancellationToken) => throw new NotSupportedException();
public Task<HistoryReadResult> ReadProcessedAsync(string fullReference, DateTime startUtc, DateTime endUtc, TimeSpan interval,
HistoryAggregateType aggregate, CancellationToken cancellationToken) => throw new NotSupportedException();
public Task<HistoryReadResult> ReadAtTimeAsync(string fullReference, IReadOnlyList<DateTime> timestampsUtc,
CancellationToken cancellationToken) => throw new NotSupportedException();
public Task<HistoricalEventsResult> ReadEventsAsync(string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents,
CancellationToken cancellationToken) => throw new NotSupportedException();
public HistorianHealthSnapshot GetHealthSnapshot() => NullHistorianDataSource.Instance.GetHealthSnapshot();
public void Dispose() { }
}
}
tests/Server/ZB.MOM.WW.OtOpcUa.Runtime.Tests/Historian/ServerHistorianOptionsTests.cs
+18
View File
@@ -119,6 +119,24 @@ public sealed class ServerHistorianOptionsTests
{ Enabled = true, Endpoint = "https://h:5222", ApiKey = "histgw_x_y", HonorReturnBounds = false }.Validate());
}
/// <summary>#694: the events horizon defaults off, binds as a TimeSpan, and only a negative value warns.</summary>
[Fact]
public void EventReadHorizon_defaults_off_binds_and_only_a_negative_value_warns()
{
Assert.Equal(TimeSpan.Zero, new ServerHistorianOptions().EventReadHorizon);
var bound = new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?> { ["ServerHistorian:EventReadHorizon"] = "3650.00:00:00" })
.Build()
.GetSection(ServerHistorianOptions.SectionName)
.Get<ServerHistorianOptions>();
Assert.Equal(TimeSpan.FromDays(3650), bound!.EventReadHorizon);
Assert.Empty(new ServerHistorianOptions
{ Enabled = true, Endpoint = "https://h:5222", ApiKey = "histgw_x_y", EventReadHorizon = TimeSpan.FromDays(30) }.Validate());
var w = new ServerHistorianOptions
{ Enabled = true, Endpoint = "https://h:5222", ApiKey = "histgw_x_y", EventReadHorizon = TimeSpan.FromDays(-1) }.Validate();
Assert.Contains(w, m => m.Contains("EventReadHorizon"));
}
[Fact]
public void HistoryUpdateMaxFutureSkew_defaults_to_one_day_binds_and_zero_is_silent()
{