Compare commits

...

25 Commits

Author SHA1 Message Date
Joseph Doherty 15ccd9b722 docs(historian): #691.1 Gateway start-bound lookup inherits the per-millisecond tie limit
v2-ci / build (push) Successful in 5m1s
v2-ci / unit-tests (push) Failing after 24m6s
2026-10-12 18:20:14 -04:00
Joseph Doherty 942ca7bc14 Merge feat/691-return-bounds: Part 11 ReturnBounds on every backend (#691.1) 2026-10-12 18:14:06 -04:00
Joseph Doherty df0c1f3513 docs(historian): #691 bound walk-back cost after the #689 walk (rebase onto 88141d6e) 2026-10-12 17:55:34 -04:00
Joseph Doherty ba46f4f47a docs(historian): #691 review — breaking change names both SDK defaults (ReturnBounds + IsReadModified), GoodNoData, Gateway horizon, HonorReturnBounds 2026-10-12 17:52:34 -04:00
Joseph Doherty 934909b430 fix(historian): #691 review — clamp Gateway at-or-after read, GoodNoData when nothing exists, HonorReturnBounds kill switch
1. The at-or-after bound read on a forward-only backend (Gateway) ends at
   now + ReverseRawReader.LookAhead instead of DateTime.MaxValue (the gateway
   only checks start <= end, so a year-9999 end reached AVEVA); a bound beyond
   the horizon is not found without a read. SqlServer/QuestDB keep MaxValue:
   they bind it safely and may hold samples further ahead. The Gateway fake now
   refuses far-future ends, so the missing-end-bound test is no longer vacuous.
3. No data and no bound found (whole read empty) => GoodNoData, no values —
   the pre-bounds answer (6.5.3.2: success only if one or both bounds exist).
   A one-value page whose start bound is missing first checks there is data or
   a found end bound.
4. ServerHistorian:HonorReturnBounds (default true); false ignores ReturnBounds
   as before #691, including the modified-read Bad_InvalidArgument.
2026-10-12 17:52:34 -04:00
Joseph Doherty ebc6b8d4d9 chore(historian): #691 comment order in ServeRawPagedAsync 2026-10-12 17:52:34 -04:00
Joseph Doherty f6681544e7 docs(historian): #691 bounding values (ReturnBounds) — Historian.md + CLAUDE.md 2026-10-12 17:52:34 -04:00
Joseph Doherty 92aab7282c test(historian): #691 Gateway bounding values over the real source and an AVEVA-shaped wire
The start bound is the stored prior sample at its own time (walked back), never
the synthetic boundary row stamped at the query start; a stored NULL is a Bad
bound; reverse end bound walks back too.
2026-10-12 17:52:34 -04:00
Joseph Doherty 60febbf2dc feat(historian): #691 HistoryRead-Raw ReturnBounds — Part 11 bounding values on every backend
The node manager composes the start/end bounds from each backend's existing raw
reads (one authority, HistoryBounds): at-or-before = a capped reverse read
(native, or ReverseRawReader's walk-back on a forward-only backend — never
AVEVA's synthetic boundary row), at-or-after = a capped forward read. Bounds
count toward NumValuesPerNode, the start bound leads the first page, the end
bound ends the last (carried in the continuation point). Missing bound =
Bad_BoundNotFound (null, at the bound time; one-sided reads ±1 s from the
previous value, Table 1 footnotes a/b). A modified read with ReturnBounds is
Bad_InvalidArgument (Part 11 6.5.3.3).

The .NET SDK's ReadRawModifiedDetails defaults ReturnBounds to TRUE; the
existing tests built details without setting it and now set it false.
Oracle: every row of Part 11 Table 1 over three backend shapes, plus paging
concatenation, ties, Bad bounds, microsecond truncation; conformance pins the
lookup primitives on SqlServer + QuestDB.
2026-10-12 17:52:34 -04:00
Joseph Doherty 88141d6ed5 Merge fix/689-reverse-walk-sizing: page-sized reverse walk, skip proven-empty history, reverse events on Gateway (#689 #691.4 #691.5) 2026-10-12 17:50:56 -04:00
Joseph Doherty 9bb1363cd6 fix(historian): #691.4 re-review — never narrow a walk segment below the 1 ms request granularity
A tie cluster's truncated read left the width at one tick; once the gap
above it was under two milliseconds nothing floored it, so segHi
stepped down one tick per read: 30 rows at one ms 0.3 s below hi with a
carried 1 s width took 10,006 identical reads (S8). (a) The top-anchored
width is floored at the granularity, Min kept; (b) a segment no wider
than the granularity is the unit that escalates to
MaxTieClusterOverfetch and then refuses - per millisecond, exact for the
ms-resolution Gateway (documented). Both are needed: the floor alone
loops on crowded reads with a small tie budget. S8 is a repo test
(4 shapes, reads <= 30, 0 repeated requests); random-data tests now
measure the refusal unit as rows in any 1 ms window. Reviewer harness,
5,000 seeds x default/MSONLY/CROWD: worst page 70 reads (default/MSONLY),
63 (CROWD) at M=65536, p99.9 27-37; no page over 500 reads at any tie
budget.
2026-10-12 17:50:39 -04:00
Joseph Doherty 574c3aa1b0 fix(historian): #691.4 review — Null reports SupportsReverseEvents = true, like raw
The null source has no history in either direction; reporting false
made the server walk an empty reverse events read back through three
empty forward reads. It now answers the reverse overload with the empty
result itself (guard test unchanged: it still declares both members).
2026-10-12 17:37:32 -04:00
Joseph Doherty d28e3e64f1 fix(historian): #691.4 review — search the gap above a truncated read by gallop then bisect, never below 1 ms
Bisecting the gap above a truncated read halved below the 1 ms request
granularity (requests are ms-rounded, so it re-sent identical requests)
and bisected the WHOLE gap even when the burst ended right above the
row it saw: a 1,026-row burst ending 100 days ago took 46 reads. After
an empty top-anchored segment the walk now gallops up from the known
row ((known + d, top], d x4, following each truncated read), then
bisects the bracket, with halving floored at the granularity (the Min
kept; a crowded gallop falls back to top-anchored halving). Burst: 18
reads, 0 repeated requests; dense tag stopped 2 days ago: 26 (gallop
alone crawled: 94). Walk fuzzed at 4,000 seeds locally (400 in-tree).
Docs: measured counts; CLAUDE.md gateway prerequisites list
RuntimeDb:EventReadMaxRows.
2026-10-12 17:37:32 -04:00
Joseph Doherty ccda27d220 fix(historian): #691.4 review — uncapped Gateway events reads no longer pre-size to int.MaxValue
GatewayHistorianDataSource.ReadEventsAsync sized its result list to
maxEvents, which is int.MaxValue for NumValuesPerNode = 0: every
uncapped Gateway events read (forward on master, and the new walked
reverse one) threw 'Array dimensions exceeded' and counted as a backend
failure. Capacity is bounded at 1024. The events walk delegate moved to
ReverseEventsReader so it is tested over the real Gateway source
(forward uncapped, reverse uncapped, reverse capped).
2026-10-12 17:20:41 -04:00
Joseph Doherty d277f34a96 fix(historian): #691.4 walk reverse events reads back on forward-only backends (Gateway)
A reverse or end-only capped events read on a backend without
SupportsReverseEvents (Gateway, Null) was refused
BadHistoryOperationUnsupported (explicit reverse) or answered with the
OLDEST N ascending (end-only, #675). It is now served newest-first by
the same ReverseWalk as raw reads: page-sized ms-aligned segments,
never answered from a read the source flagged truncated, ties sorted by
EventId (the gateway orders by event time only - the fake shuffles
them per call), one extra event walked to signal 'more', uncapped = one
forward read reversed, and the width + proven-empty stretch carried in
the events cursor. The oversized-cluster over-fetch keeps exactly the
events at T in the same order. A tie cluster over the budget is
BadTooManyOperations. Docs note the gateway's EventReadMaxRows must be
unlimited (it truncates silently).
2026-10-12 17:19:05 -04:00
Joseph Doherty 186b7d5ea5 fix(historian): #691.5 a reverse walk proves the rest empty once and carries it page to page
The last pages of a deep reverse page-through on a forward-only backend
each walked down to 1970 one x4 growth step at a time (~14-16 reads,
~2.5 s live). After two empty segments the walk now asks once per page
for the whole remainder at the page budget (a truncated answer is
discarded and the walk carries on), and the stretch the page's last
complete read proved empty rides the continuation point
(HistoryContinuationState.EmptyFromTicks/EmptyToTicks, session-held, so
no versioning) so the next page steps over it. Fuzz now pages with the
walk's own hints and with true random empty stretches.
2026-10-12 17:19:05 -04:00
Joseph Doherty d33992034d fix(historian): #689 size reverse-walk segments from the page, not the tie budget
ReverseRawReader aimed every segment at half of MaxTieClusterOverfetch
(~32k rows by default) and carried that width page to page, so a paged
'latest N' read on a dense Gateway tag fetched ~32,770 rows per page
(2-5 s live) to return N. The walk now lives in a generic ReverseWalk
core: segments are aimed at ~N+8 rows (page budget 2N+16), a discovery
segment (first without a hint / after an empty one) may read 1,024, and
the tie budget is only the ceiling a single-timestamp cluster escalates
to. A truncated read never re-enters the stretch below its newest row
until the gap above it is bisected. Tests: per-page row counts on a
dense tag (node manager + walk), randomized exactness fuzz (walk and
node-manager paging), dense-tag-below-a-long-gap read bound.
2026-10-12 17:19:05 -04:00
Joseph Doherty 71d031fcd0 docs(historian): #691.2 forward native reads now share the 100k-interval BucketPlan cap 2026-10-12 17:00:44 -04:00
Joseph Doherty f0d064a68c Merge fix/690-691-gateway-mapping: Bad Gateway rows carry no value, learned gateway ceiling, native TimeAverage/Total grid both directions (#690 #691.2 #691.3) 2026-10-12 17:00:37 -04:00
Joseph Doherty 830b47f624 fix(historian): #691.2 review — forward native TimeAverage/Total on the SQL grid: clipped latest interval read over its exact span, Partial, interval clamped to the window 2026-10-12 16:55:01 -04:00
Joseph Doherty 7b86059daf fix(historian): #691.3 review — remember a strict gateway's ceiling per data source; re-probe once when a bounded read comes back full 2026-10-12 16:50:32 -04:00
Joseph Doherty ab5030174a fix(historian): #691.3 review — a ceiling-bound per-sample read names the gateway ceiling, not the cap 2026-10-12 16:47:57 -04:00
Joseph Doherty 0e47e887de fix(historian): #691.2 serve reverse native TimeAverage/Total on Gateway via forward reads on the later-anchored #677 grid 2026-10-12 16:17:05 -04:00
Joseph Doherty dbffb6bdb4 fix(historian): #691.3 strict-ceiling gateway: re-ask once at the named ceiling, never answer a truncated read 2026-10-12 16:13:14 -04:00
Joseph Doherty 39552e15b9 fix(historian): #690 Gateway raw/AtTime Bad rows carry no value (a stored NULL was value 0) 2026-10-12 16:08:54 -04:00
42 changed files with 3834 additions and 380 deletions
+8 -3
View File
@@ -1353,8 +1353,9 @@ historian tagname, which is the tag's driver `FullName`.
- **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; per-sample aggregates (10, simple definitions) are pushed down to T-SQL with bucket math in **100 ns ticks**; the Part 13 time-weighted seven (TimeAverage/TimeAverage2/Total/Total2/Interpolative, #671 — ⚠️ `Total` is now the time-weighted integral, no longer `SUM` — plus the **duration-based** PercentGood/PercentBad, #680) stream window + 4 bound rows into the shared `TimeWeightedAggregator` (both SQL backends, identical results; reverse `Interpolative` anchored at the later time; window capped by `ServerHistorian:MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations`, percentages included). ⚠️ **#680 status rules (client-visible):** TimeAverage/Total are Uncertain if any part of the interval uses an Uncertain value, is bridged across a Bad sample, is extrapolated past the last sample (⇒ a live tag's latest interval is Uncertain) or is not covered; TimeAverage2/Total2 keep the SDK time-based rule (Uncertain time counts as Good) but never extrapolate past the last sample — that interval is Partial, later ones `BadNoData` (a report-by-exception tag with nothing after its last sample reads `BadNoData` until its next sample arrives). `docs/Historian.md` §"Time-weighted aggregates". ⚠️ **#680 per-sample (client-visible):** `Count` counts **Good** samples only (value-based status; an all-Bad interval is generic Bad, value null) and an empty interval **inside the tag's data range** is `0`/Good (outside it still `BadNoData`); `Start`/`End` return the **raw** first/last sample of any quality — its own timestamp and status, value null when Bad; `Delta` is Uncertain only when the interval's first or last sample is unusable; ⚠️ #684: `Average`/`Minimum`/`Maximum`/`Range`/`*ActualTime` use **Good samples only** (was Good + Uncertain; an Uncertain-only interval is `BadNoData`), Min/Max-family status Uncertain on a Bad sample, an Uncertain value beyond the Good extreme (Part 13 — the SDK ignores that) or an Uncertain non-numeric sample, and only a **Good** String/DateTime sample refuses them (`BadAggregateInvalidInputs`; Delta still refuses any usable one); and every per-sample aggregate but `Average` sets **end-of-data Partial** on the interval holding the last sample (forward incl. its earlier edge, reverse strictly inside), never on a Bad result. The data-range lookup is a key seek on SQL Server and a `tag` symbol-index seek on QuestDB (#683 — `{p}sample.tag` is `SYMBOL INDEX`; the bootstrapper ADDs it to a pre-#683 table, idempotent), read BEFORE the bucket statement so a racing append can never yield a Good `Count` of 0 (a concurrent HistoryUpdate delete is the mirror case and still can, transiently). `docs/Historian.md` §"Aggregates". Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. **Retention (#667):** opt-in `SqlServer:RetentionDays` (+ `PurgeInterval` 1h, `PurgeBatchSize` 4000) -> `HistorianRetentionService` purges old `hist.Sample` (per tag) / `hist.Event` in `DELETE TOP(n)` batches, Primary-gated on `IHistorizationRoleView`, one purger per DB via `sp_getapplock`; counter `otopcua.historian.retention.purged{table}`. Optional `IHistorianBackendProvider.CreateRetention` (default null). `docs/Historian.md` §Retention.
- **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend".
- ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); on forward reads Partial also marks the interval the data starts inside (not Average — the SDK's rule, forward-only); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates.
- ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` serves it from FORWARD reads: uncapped ⇒ ONE forward read reversed in memory; capped ⇒ a bounded walk back in segments (ms-aligned request bounds, width aimed from observed density and carried page to page in the continuation point, ≤ `MaxTieClusterOverfetch + 2` rows per read — a full read is re-read, never answered truncated; only one timestamp holding more than max(1, `MaxTieClusterOverfetch`) samples ⇒ `BadTooManyOperations`; walks no later than now + 1 day and never before the Unix epoch). The raw/events tie-path oversized-cluster refusal is now `BadTooManyOperations` too (was `BadHistoryOperationUnsupported`). Uncapped Gateway raw reads (forward too) used to send `max_values = 0`, which the client/gateway reject — now unbounded (`int.MaxValue`; a ceiling refusal ⇒ `BadTooManyOperations`), and a cancelled read is no longer a backend failure. Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events still differ (end-only forward / explicit reverse refused on Gateway, #675). `docs/Historian.md` §"Open-ended and reverse raw reads".
- ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` serves it from FORWARD reads: uncapped ⇒ ONE forward read reversed in memory; capped ⇒ a bounded walk back in segments (ms-aligned request bounds, width aimed from observed density at ≈ N + 8 rows and carried page to page in the continuation point — #689: it used to aim at half `MaxTieClusterOverfetch`, ~32k rows per page on a dense tag; a read asks for `2N + 18` rows (a discovery segment ≤ 1,026), `MaxTieClusterOverfetch` is only the tie-cluster ceiling — a full read is re-read, never answered truncated; only more than max(1, `MaxTieClusterOverfetch`) samples within ONE MILLISECOND ⇒ `BadTooManyOperations` (per ms, not per timestamp — requests are ms-rounded, and letting a segment narrow below 1 ms stepped a page through one ms a tick per read, ~10k reads); walks no later than now + 1 day and never before the Unix epoch; #691: after two empty segments it asks once for the whole remainder, and the stretch a page proved empty rides the continuation point — the last pages of a deep page-through no longer each walk ~14 reads to 1970). The raw/events tie-path oversized-cluster refusal is now `BadTooManyOperations` too (was `BadHistoryOperationUnsupported`). Uncapped Gateway raw reads (forward too) used to send `max_values = 0`, which the client/gateway reject — now unbounded (`int.MaxValue`; a ceiling refusal ⇒ `BadTooManyOperations`), and a cancelled read is no longer a backend failure. Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events follow since #691.4: a reverse/end-only capped events read on Gateway (`SupportsReverseEvents` false; Null now reports `true` and answers empty) is walked back the same way (ties sorted by EventId — the gateway orders by time only; one extra event found ⇒ continuation point) — it used to be refused `BadHistoryOperationUnsupported` (explicit reverse) or answered with the OLDEST N ascending (end-only); needs the gateway's `RuntimeDb:EventReadMaxRows` 0/unlimited (it truncates silently). Uncapped Gateway events reads (forward too) used to throw "Array dimensions exceeded" (result list pre-sized to `int.MaxValue`) — fixed. Above a truncated read the walk gallops up from the newest row seen, then bisects (never below 1 ms): a burst that ended 100 days ago ≈ 18 reads, a dense tag that stopped days ago ≈ 26. `docs/Historian.md` §"Open-ended and reverse raw reads".
- **Advertised aggregates (#682):** `IHistorianDataSource.SupportedAggregates` (Gateway 4 via `GatewayHistorianDataSource.ServedAggregates`, SqlServer/QuestDB 17 via `AggregateFamilies.Served`, Null **none**; a null set ⇒ all) is the one authority for `HistoryServerCapabilities/AggregateFunctions` (i=11201 — only its folder references are trimmed) and the up-front `BadAggregateNotSupported` in HistoryRead-Processed. ⚠️ `ServerCapabilities/AggregateFunctions` (i=2997) is deliberately NOT trimmed: it keeps the SDK's 37 because live `AggregateFilter` subscriptions still compute them all. ⚠️ Historian disabled ⇒ Processed is now `BadAggregateNotSupported`, not `GoodNoData`. `docs/Historian.md` §Advertised aggregates.
- **Raw `ReturnBounds` is honoured on every backend (#691, client-visible):** Part 11 bounding values (§4.6 Table 1 is the test oracle) are composed in the server by `HistoryBounds` from each backend's raw reads — "at or before" = capped reverse read (native on SQL backends, the #686 `ReverseRawReader` walk-back on Gateway, never AVEVA's synthetic boundary row), "at or after" = capped forward read. Bounds are stored samples of any quality, count toward `NumValuesPerNode`, start bound on the first page / end bound on the last (pending end bound rides the continuation point); missing ⇒ `Bad_BoundNotFound` (null, at the bound time; a one-sided read's open side ±1 s from the previous value). ⚠️ **Breaking for SDK-default clients:** the .NET SDK's `new ReadRawModifiedDetails()` defaults BOTH `ReturnBounds = true` AND `IsReadModified = true` — a client that cleared only `IsReadModified` now gets bounds, and one that set neither is a modified read with bounds ⇒ `Bad_InvalidArgument` (§6.5.3.3); set both explicitly (Client.Shared does; tests building details must too). No data AND no bound found ⇒ `GoodNoData`, no values (the pre-bounds answer). On Gateway the at-or-after lookup ends at now + 1 day (never a year-9999 end to AVEVA). Kill switch `ServerHistorian:HonorReturnBounds` (default `true`; `false` = pre-#691). `docs/Historian.md` §Bounding values.
- **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good.
- **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node.
- **HistoryUpdate** (SqlServer + QuestDb; Gateway ⇒ `BadHistoryOperationUnsupported`): Insert/Replace/Update/DeleteRaw/DeleteAtTime, **Primary-only** since #674 (off-Primary ⇒ `BadInvalidState`, after authorization AND after planning — an unservable request (Remove / `IsDeleteModified` / not historized) stays `BadHistoryOperationUnsupported` off-Primary; the verdict is `IHistoryUpdateRoleView` = the data-plane verdict but **closed until this node's first redundancy snapshot**, so a pair cannot both accept in the boot window; admin-only nodes register no view ⇒ no gate; reads ungated), items with a `SourceTimestamp` past now + `ServerHistorian:HistoryUpdateMaxFutureSkew` (default 1 day, ≤0 disables) ⇒ `BadOutOfRange` in the node manager (every backend), gated in both ACL modes — flag off needs the `HistoryUpdate` role from a **Cluster-scoped** grant (+ key re-check), flag on uses `DataPlaneAuthorizer`'s HistoryUpdate operation; `WriteOperate` never implies it. The `HistoryWrite` AccessLevel bit is set only when an updater is wired at materialise time. `docs/Historian.md` §HistoryUpdate, `docs/security.md`. ⚠️ **SqlServer schema v1 is accepted (#674)** with `AutoCreateSchema=false`: writes/reads/events/retention work; modified reads + HistoryUpdate ⇒ node-level `BadHistoryOperationUnsupported` on every arm (the updater throws `NotSupportedException`, nothing sent; the `HistoryWrite` bit is still advertised — not knowable at materialise time), retention skips `SampleModified`, one Warning per boot (`SqlServerSchemaBootstrapper.SupportsModifiedTrail`; `RequiredVersion` = 1, `ModifiedTrailVersion` = 2). QuestDB: keep `cairo.ttl.use.wall.clock=true`; a QuestDB trail INSERT retried after a client timeout duplicates (no DEDUP key).
@@ -1364,8 +1365,8 @@ historian tagname, which is the tag's driver `FullName`.
The server dispatches all OPC UA HistoryRead to the registered `IHistorianDataSource` — the
`GatewayHistorianDataSource` read client when enabled, else the `NullHistorianDataSource` default
(historized nodes return `GoodNoData` for Raw/AtTime — Processed is `BadAggregateNotSupported` since #682). Supported variants: Raw (⚠️ #687, client-visible: Gateway raw reads no longer return AVEVA's synthetic start-boundary row — asked from 1 ms early (epoch-floored) with `NumValuesPerNode + 16` rows (unclamped; 0 ⇒ `int.MaxValue`) and filtered by timestamp — so an empty window is `GoodNoData`, not a fabricated value at StartTime; the forward end is now exclusive (a point read inclusive); pre-window rows that crowd the budget trigger ONE re-read with room for them, else `BadTooManyOperations`; `ReturnBounds` is ignored on every backend), Processed
(backend-dependent: SqlServer/QuestDb serve seventeen; Gateway twelve = `GatewayHistorianDataSource.ServedAggregates` — ⚠️ #681, client-visible: the ten per-sample ones (Average, Count, Min/Max, Range, Start/End/Delta, Min/MaxActualTime) are evaluated from RAW gateway reads by the shared `Sql.Common` evaluator — identical to the SQL backends, Calculated bits, interval-start stamps, #677 reverse; Min/Max are no longer AVEVA modes, the read moves every raw sample of the window (a month of 1 Hz ⇒ `BadTooManyOperations` at the default `MaxTimeWeightedRawRows` cap), and every gateway stream is consumed to completion (an abandoned one makes the gateway evict its shared historian session) — plus TimeAverage/Total via AVEVA modes (interval-start stamped + Calculated; ⚠️ #685: AVEVA's extra pre-window row is dropped ⇒ one row per interval in `[Start, End)`, and an Uncertain (opc 64) interval keeps its value as `UncertainDataSubNormal` — AVEVA's TWA excludes NULL-gap time where the SQL backends bridge it; per-interval values still need gateway fix historiangw#27); the other time-weighted five stay `BadAggregateNotSupported` because the gateway has no prior-sample read for Part 13 bounds; `docs/Historian.md` §Aggregate set), and AtTime over historized variable nodes; Events over
(historized nodes return `GoodNoData` for Raw/AtTime — Processed is `BadAggregateNotSupported` since #682). Supported variants: Raw (⚠️ #687, client-visible: Gateway raw reads no longer return AVEVA's synthetic start-boundary row — asked from 1 ms early (epoch-floored) with `NumValuesPerNode + 16` rows (unclamped; 0 ⇒ `int.MaxValue`) and filtered by timestamp — so an empty window is `GoodNoData`, not a fabricated value at StartTime; the forward end is now exclusive (a point read inclusive); pre-window rows that crowd the budget trigger ONE re-read with room for them, else `BadTooManyOperations`; `ReturnBounds` is ignored on every backend; ⚠️ #690: a Gateway Bad row — a stored NULL, which histsdk sends as value 0 — now reads value **null** + Bad on Raw and AtTime, as on the SQL backends; ⚠️ #691.3: a strict gateway (`AllowStreamingBeyondCeiling=false`) that refuses a raw budget is re-asked ONCE at the ceiling its refusal names — raw and per-sample reads alike; the ceiling is then remembered per data source (`GatewayRowCeiling`: refusals count as gateway errors), re-learned from a refusal when lowered, and re-probed once at the full budget when a ceiling-bounded read comes back full (raised / streaming) — so an uncapped (`NumValuesPerNode = 0`) read of a small window is served; a ceiling-bounded stream that is not provably complete is `BadTooManyOperations`, never a truncated answer), Processed
(backend-dependent: SqlServer/QuestDb serve seventeen; Gateway twelve = `GatewayHistorianDataSource.ServedAggregates` — ⚠️ #681, client-visible: the ten per-sample ones (Average, Count, Min/Max, Range, Start/End/Delta, Min/MaxActualTime) are evaluated from RAW gateway reads by the shared `Sql.Common` evaluator — identical to the SQL backends, Calculated bits, interval-start stamps, #677 reverse; Min/Max are no longer AVEVA modes, the read moves every raw sample of the window (a month of 1 Hz ⇒ `BadTooManyOperations` at the default `MaxTimeWeightedRawRows` cap), and every gateway stream is consumed to completion (an abandoned one makes the gateway evict its shared historian session) — plus TimeAverage/Total via AVEVA modes (interval-start stamped + Calculated; ⚠️ #685: AVEVA's extra pre-window row is dropped ⇒ one row per interval in `[Start, End)`, and an Uncertain (opc 64) interval keeps its value as `UncertainDataSubNormal` — AVEVA's TWA excludes NULL-gap time where the SQL backends bridge it; per-interval values still need gateway fix historiangw#27; ⚠️ #691.2: native reads use the SQL backends' grid in both directions via forward requests only — a REVERSE read (refused by the gateway before) is anchored at its later time (#677), and the clipped interval (earliest reverse, LATEST forward — which used to be averaged past EndTime) is its own exact-span read marked Partial unless Bad; an interval wider than the window is sent as the window, interval 0 as one interval; more than 100,000 intervals is refused up front (`BadHistoryOperationUnsupported`, as on the SQL backends — forward used to be served up to the gateway's 1,000,000); until historiangw#27 lands both directions return Good-status wrong values); the other time-weighted five stay `BadAggregateNotSupported` because the gateway has no prior-sample read for Part 13 bounds; `docs/Historian.md` §Aggregate set), and AtTime over historized variable nodes; Events over
alarm-owning equipment-folder event-notifier nodes. Reads are ungated (served from any redundancy node);
authorization uses the standard `AccessLevels.HistoryRead` bit set at materialization.
@@ -1381,6 +1382,7 @@ authorization uses the standard `AccessLevels.HistoryRead` bit set at materializ
| `CaCertificatePath` | `null` | PEM CA file pinning the gateway TLS chain; null/empty uses the OS trust store |
| `CallTimeout` | `00:00:30` | Per-call deadline for each unary gateway read |
| `MaxTieClusterOverfetch` | `65536` | Bounded over-fetch the HistoryRead-Raw paging uses to page within an oversized same-timestamp tie cluster (retained from the prior backend) |
| `HonorReturnBounds` | `true` | #691 kill switch: `false` ignores HistoryRead-Raw `ReturnBounds` as before (no bounding values, no modified-read `Bad_InvalidArgument`) |
| `HistoryReadBatchParallelism` | `4` | Max nodes served concurrently within one HistoryRead batch (03/S3); `⌈N/P⌉ × CallTimeout` worst case instead of sequential |
| `MaxConcurrentHistoryReadBatches` | `16` | Process-wide concurrent HistoryRead batch cap (03/S3); flood degrades to `BadTooManyOperations`. Total in-flight reads capped at 16 × 4 = 64 |
| `MaxTimeWeightedRawRows` | `1000000` | `SqlServer`/`QuestDb` (#671): raw in-window rows one time-weighted aggregate read (incl. `Total` and, since #680, PercentGood/PercentBad) may stream into the C# evaluator — and, since #681, every `Gateway` per-sample read (raw-evaluated); window query capped at this + 1, past it ⇒ `BadTooManyOperations` for that node. ≤ 0 = unbounded. Gateway (#688): the `cap + 18` budget is clamped at the gateway's default `MaxValuesCeiling` (1,000,000) while the cap is ≤ it, so the default no longer trips a strict (`AllowStreamingBeyondCeiling=false`) gateway |
@@ -1455,6 +1457,9 @@ The target HistorianGateway OtOpcUa points at **must** run with:
- `RuntimeDb:Enabled=true` — enables the `WriteLiveValues` SQL live path (continuous historization).
- `RuntimeDb:EventReadsEnabled=true` — enables `ReadEvents` from `Runtime.dbo.Events` (alarm-history reads).
- `RuntimeDb:EventReadMaxRows=0` (its default, unlimited) — or larger than the reverse-events walk's biggest read
(`2·MaxTieClusterOverfetch + 4`). The cap truncates **silently**, so a walked-back reverse events read (#691.4) would
lose the newest events of a segment it cut.
- An API key carrying scopes **`historian:read`**, **`historian:write`**, **`historian:tags:write`**.
### Migration note (deployments upgrading from the Wonderware backend)
+187 -46
View File
@@ -111,6 +111,7 @@ Processed read is `BadAggregateNotSupported` and no history aggregate functions
"CaCertificatePath": null,
"CallTimeout": "00:00:30",
"MaxTieClusterOverfetch": 65536,
"HonorReturnBounds": true,
"HistoryReadBatchParallelism": 4,
"MaxConcurrentHistoryReadBatches": 16,
"HistoryReadDeadline": "00:01:00",
@@ -144,9 +145,10 @@ Processed read is `BadAggregateNotSupported` and no history aggregate functions
| `CaCertificatePath` | string\|null | `null` | **Gateway backend only.** PEM CA file pinning the gateway's TLS chain. Null/empty uses the OS trust store. |
| `CallTimeout` | TimeSpan | `00:00:30` | **Gateway backend only.** Per-call deadline applied to each unary gateway read. A read for a tag the historian does not yet know about — e.g. a freshly-historized tag that has **not** been provisioned (see [Tag auto-provisioning](#tag-auto-provisioning-ensuretags)) — can run to this **full** deadline before the gateway errors, so an unprovisioned-tag HistoryRead can block for ~30 s. Lower this to fail faster, at the cost of truncating legitimately long reads. |
| `MaxTieClusterOverfetch` | int | `65536` | (Node-manager paging; all backends — a `SqlServer` PK rules out tie clusters, so it is moot there.) Maximum samples the server will fetch in one shot to page through a tie cluster (multiple samples sharing one `SourceTimestamp`). A cluster larger than this ceiling fails `BadTooManyOperations` (#686; was `BadHistoryOperationUnsupported`). Also the per-read row budget of a Gateway reverse raw read (walked back from forward reads, #686): one timestamp holding more samples is refused `BadTooManyOperations`. Values below 1 count as 1. Raise to handle abnormally large tie clusters; the default covers all normal-data cases. |
| `HonorReturnBounds` | bool | `true` | (#691, all backends.) Whether HistoryRead-Raw honours `ReturnBounds` (Part 11 bounding values, [Bounding values](#bounding-values-returnbounds-691)). `false` is the kill switch back to the pre-#691 behaviour: the flag is ignored (stored samples only) and a modified read carrying it is not refused `Bad_InvalidArgument`. Note the .NET SDK defaults both `ReturnBounds` and `IsReadModified` to `true`. |
| `HistoryReadBatchParallelism` | int | `4` | Max historized nodes served **concurrently** within one HistoryRead batch (arch-review 03/S3). The SDK's `HistoryRead*` override surface is synchronous, so each arm bridges once at its boundary; this turns per-node serving from sequential (`N × CallTimeout` worst case) into `⌈N / P⌉ × CallTimeout`. Also caps the gateway-load multiplication a single client can command. `≤ 1` serves sequentially. |
| `MaxConcurrentHistoryReadBatches` | int | `16` | Process-wide limit on HistoryRead (and HistoryUpdate) batches served **concurrently** (arch-review 03/S3). A flood of history clients degrades to `BadTooManyOperations` rather than exhausting the SDK request-thread pool. With `HistoryReadBatchParallelism` this caps total in-flight gateway reads at `MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism` (default 16 × 4 = 64). A batch that cannot get a slot within `min(HistoryReadDeadline, 5 s)` fails every handle `BadTooManyOperations`. |
| `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` (#671), and the `Gateway` per-sample aggregates (#681 — every per-sample read there streams raw samples, so the cap applies to the window's raw rows; see [Aggregate set](#aggregate-set-and-badaggregatenotsupported-285)). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative, and — since #680 — **PercentGood/PercentBad**) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped on the SQL backends (pushed down). 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). `Gateway` (#688): the row budget it asks the gateway for (`cap + 18`) is clamped at the gateway's default `MaxValuesCeiling` (1,000,000) while the cap is at or under it, so the default works against a gateway with `AllowStreamingBeyondCeiling=false`; against a gateway with a *lower* ceiling, set this to `ceiling − 18`. |
| `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` (#671), and the `Gateway` per-sample aggregates (#681 — every per-sample read there streams raw samples, so the cap applies to the window's raw rows; see [Aggregate set](#aggregate-set-and-badaggregatenotsupported-285)). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative, and — since #680 — **PercentGood/PercentBad**) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped on the SQL backends (pushed down). 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). `Gateway` (#688): the row budget it asks the gateway for (`cap + 18`) is clamped at the gateway's default `MaxValuesCeiling` (1,000,000) while the cap is at or under it, so the default works against a gateway with `AllowStreamingBeyondCeiling=false`; against a gateway with a *lower* ceiling the refused budget is retried once at the ceiling the gateway names (#691.3), so no change is needed (`ceiling − 18` only saves that round-trip). |
| `HistoryReadDeadline` | TimeSpan | `00:01:00` | Per-request wall-clock deadline for a HistoryRead (and HistoryUpdate) batch (arch-review 03/S3). A node still in flight at the deadline surfaces `BadTimeout`, so one request can never hold an SDK request thread longer than this regardless of node count. Non-positive = unbounded (a `Validate()` warning). |
| `HistoryUpdateMaxFutureSkew` | TimeSpan | `1.00:00:00` | #674. How far past the server clock a HistoryUpdate (Insert / Replace / Update) item's `SourceTimestamp` may lie; an item beyond `now + skew` is refused **`BadOutOfRange`** by the node manager before any backend sees it (every backend). Guards against a far-future typo (a 2062 timestamp). The live recorder is not bounded. Non-positive disables the bound (no warning — a deliberate choice). See [HistoryUpdate](#historyupdate). |
@@ -639,33 +641,59 @@ backend:
`[⌊segLo⌋, ⌈segHi + 1 tick⌉]`, both rounded **outward to the millisecond**, and filtered on the samples' own
timestamps — exact for an inclusive or exclusive forward end, and the row AVEVA fabricates at a query's (possibly
rounded-up) start can never land inside the segment. The first segment is **1 minute** (or the width the previous
page ended on — it rides the continuation point); each next width is aimed at about half the row budget from the
density just observed, growing at most ×4. So "the last N values" of a live tag is normally one read, a dense tag
costs about one read per page, and sparse history is found in a few dozen tiny ones.
- **Never answered from a truncated read.** The row budget is `M = max(1, MaxTieClusterOverfetch)`; a segment read asks
for `M + 2` rows (room for the fabricated row and one to see past the segment). Rows stream ascending, so a read is
complete when it came back short or its last row is past the segment; otherwise it is discarded — re-read once with
room for the rows before the segment that crowded it (a neighbouring tie cluster, at most `2M + 4` rows), or narrowed
to the density it showed. Only **one timestamp holding more than `M` samples** is refused, **`BadTooManyOperations`**
(a request limit, not a backend fault); a cluster of exactly `M`, or neighbouring clusters each within `M`, is served.
page ended on — it rides the continuation point); each next width is aimed from the density just observed at about
half the **page budget** `2N + 16` (≈ N + 8 rows), growing at most ×4. So "the last N values" of a live tag is
normally one read of about N rows, a dense tag costs about one N-sized read per page, and sparse history is found in
a few dozen tiny ones.
- **Read sizes (#689).** A segment read asks for its budget + 2 rows (room for the fabricated row and one to see past
the segment). The budget is the page budget `2N + 16`, except a **discovery** segment — the first of a page with no
carried width, or one after an empty segment, whose ×4 growth may overshoot into dense data — which may read up to
**1,024** rows. The tie budget `M = max(1, MaxTieClusterOverfetch)` is only a **ceiling**: a single timestamp holding
more rows than the segment budget escalates to `M`.
⚠️ Before #689 every segment was aimed at half of `M` (≈ 32k rows by default) and that width was carried page to
page, so on a 1 Hz tag from about page 5 every page of a "latest N" read fetched ~32,770 rows and took 2–5 s to
return N.
- **Never answered from a truncated read.** Rows stream ascending, so a read is complete when it came back short or its
last row is past the segment; otherwise it is discarded — re-read once with room for the rows before the segment
that crowded it (a neighbouring tie cluster, at most twice the budget + 4 rows), or narrowed: to the density it
showed, and never again below the newest row it saw until what lies above that row is read. The gap above it is
searched: a segment at that density ending on the top (data running right up to it); if empty, a **gallop** up from
that row (`(row + d, top]`, `d` ×4 per step, following each truncated read) until a read comes back empty, then a
**bisection** of the bracket — never narrower than the 1 ms request granularity, which only re-sent the same request.
Measured: a dense tag that stopped two days ago ≈ **26** reads, a burst of just over 1,024 rows that ended 100 days ago
≈ **18** (was 46 bisecting the whole gap) — rather than re-reading the oldest rows of a burst over and over. Only **more than `M` samples within one millisecond** are refused,
**`BadTooManyOperations`** (a request limit, not a backend fault); a cluster of exactly `M`, or neighbouring
clusters each within `M` and a millisecond or more apart, is served. ⚠️ The unit is the **millisecond**, not the
timestamp (#691.4 re-review): requests are rounded to the millisecond, so a narrower segment is the same request —
letting the width fall below it once stepped a page through one millisecond a tick at a time (~10,000 reads). Exact
for the Gateway, which stores milliseconds; a sub-millisecond backend walked this way would refuse more than `M`
samples inside one millisecond even when no single timestamp holds that many.
- **Proving the rest empty (#691).** Walking down through empty history to the floor used to cost one read per ×4
growth step — ~14–16 reads (~2.5 s live) on **every** last page of a deep page-through. Now, after two consecutive
empty segments, the walk asks **once per page** for the whole remainder down to the floor at the page budget: a
complete answer ends the walk in one read; a truncated one (deep history) is discarded and the walk carries on. And
the stretch the page's last complete read proved empty below its oldest row (often all the way to the floor) rides
the continuation point, so the next page steps straight over it. Data backfilled into such a stretch between pages
is not seen by that read — paging was never a snapshot (the forward path misses data inserted behind its cursor too).
- **Bounds.** The walk starts no later than **now + 1 day** (a `StartTime = DateTime.MaxValue` read does not send
year-9999 query starts) and never reads before the **Unix epoch** (AVEVA holds nothing earlier): a walked-back read
returns only samples strictly after 1970-01-01T00:00:00Z and no later than now + 1 day.
Paging is unchanged — a reverse cursor resumes as a fresh walk from its resume point.
Paging is otherwise unchanged — a reverse cursor resumes as a fresh walk from its resume point, carrying the width and
the proven-empty stretch (session-held cursor state, never serialized, so the new fields need no versioning).
- **Null** reports `true` (empty in either direction).
> ⚠️ **Client-visible (#686).** Since #669 an end-only capped raw read, and any explicit reverse raw read, on the Gateway
> backend failed the node (`BadHistoryOperationUnsupported`) and was counted as a historian backend failure (the gateway
> refused `start > end`). Before #669 an end-only read on Gateway returned the **oldest** N values ascending. Both now return
> the Part 11 answer — the latest values at or before `EndTime`, newest first. **Events differ:** an end-only events read on
> Gateway still keeps its forward meaning and an explicit reverse events read is refused (see below).
> the Part 11 answer — the latest values at or before `EndTime`, newest first. **Events** follow since #691.4 (see below).
>
> ⚠️ **Also fixed with #686: uncapped raw reads on Gateway.** `NumValuesPerNode = 0` ("all values") used to be forwarded as
> `max_values = 0`, which the client package and the gateway reject — every uncapped raw read (forward too) failed the node
> and counted as a backend failure. It is now sent unbounded (`int.MaxValue`); a gateway configured with
> `AllowStreamingBeyondCeiling=false` refuses that over its `MaxValuesCeiling`, which is answered **`BadTooManyOperations`**
> (not a backend fault). A cancelled read (client cancel / `HistoryReadDeadline`) is no longer counted as a backend failure.
> `AllowStreamingBeyondCeiling=false` refuses that over its `MaxValuesCeiling`; since #691.3 the read is retried once at the
> ceiling the refusal names and answered when provably complete, else **`BadTooManyOperations`** (not a backend fault) — see
> [#687](#gateway-raw-reads-return-stored-samples-only-687) "Strict gateway". A cancelled read (client cancel / `HistoryReadDeadline`) is no longer counted as a backend failure.
Continuation points are bound to the OPC UA session (the SDK's
`ServerConfiguration.MaxHistoryContinuationPoints` cap, default 100, with oldest-eviction; points
@@ -686,7 +714,12 @@ data.
the gateway is asked from **1 ms before** the window (never before the Unix epoch, the start of the historian's time encoding),
so the boundary row lands before it, and only rows `RawAggregateReader.IsWindowSample` are returned. The decision is by
**timestamp**, not by value or quality — a stored NULL sample looks exactly like the no-data boundary row on the wire (value 0,
Bad), and a stored NULL inside the window is still returned (Bad), as it is a stored sample. A row stamped **at or before the
Bad), and a stored NULL inside the window is still returned (Bad), as it is a stored sample. ⚠️ **Since #690 it is returned
with a null value** (client-visible): histsdk puts a value on every row, so the NULL used to come back as value `0` + Bad (`""`
on a string tag), a fabricated 0 for any client that reads the value without the status. `SampleMapper.ToSnapshot` drops the
value of every row whose mapped status is **Bad** (AVEVA stores a Bad-quality sample as NULL, so a Bad row never carries a real
value) — raw reads and AtTime alike, matching SqlServer/QuestDB (an Empty sample reads null + Bad). Uncertain rows keep their
value; the raw-evaluated aggregates were already unaffected (a Bad sample is never usable, and Start/End null a Bad value). A row stamped **at or before the
epoch** is never a stored sample (AVEVA stores nothing there — the #686 walk-back's floor too): a window starting at or before
the epoch gets no margin, so AVEVA's no-data row lands exactly at its start; raw reads and the per-sample aggregates both drop it.
- **The forward end is exclusive** (`[StartTime, EndTime)`, the `IHistorianDataSource` contract and what the SQL backends return;
@@ -696,8 +729,23 @@ data.
- **Row budget.** `NumValuesPerNode = 0` ("all values") asks the gateway for an unbounded `max_values` (`int.MaxValue` — the client
package and the gateway reject 0, #686). A cap asks for `NumValuesPerNode + 16` rows, **not** clamped at the gateway's default
ceiling (#688 clamps only the per-sample aggregate budget): AVEVA always emits the boundary row, so a clamped budget would make
every page size from 999,985 to 1,000,000 come back short. A gateway with `AllowStreamingBeyondCeiling=false` refuses an
over-ceiling budget instead ⇒ `BadTooManyOperations` (not a backend fault); a default gateway serves it.
every page size from 999,985 to 1,000,000 come back short. A default gateway serves any budget.
- ⚠️ **Strict gateway (#691.3, client-visible).** A gateway with `AllowStreamingBeyondCeiling=false` refuses a budget over its
`MaxValuesCeiling` (before streaming) and names the ceiling in the refusal; the gateway has no RPC that reports it. The read is
then **re-issued once with the ceiling as the budget**, and the ceiling is **remembered per data source** (`GatewayRowCeiling`,
shared by raw and per-sample reads): every refused attempt counts as an error in the gateway's own outcome metrics, so after the
first refusal later reads ask for at most the learned ceiling and are not refused. A **lowered** ceiling is re-learned from the
refusal of the remembered one. A **raised** ceiling (or a switch to streaming) is noticed by the only reads it matters to: when a
read bounded by the remembered ceiling comes back full without proving completeness, it is re-probed **once** at its full budget
— served ⇒ the ceiling is forgotten; refused naming a larger ceiling ⇒ that one is learned and the read re-issued at it; refused
naming the same ceiling ⇒ the bounded result stands. No timer: a TTL would only add refusals for reads that already fit; the cost
of the rule is one refused re-probe per genuinely over-ceiling read (which fails either way). So an uncapped read (`NumValuesPerNode = 0`, which used to be refused outright on a strict gateway, even for a 10-row
window) and a page size within 16 of the ceiling are now served when the window fits. A ceiling-bounded read is answered only
when it is **provably complete** — the stream came back short of the ceiling, or reached `EndTime` (or, capped, filled the page);
a stream that delivered the whole ceiling short of `EndTime` may be truncated and fails **`BadTooManyOperations`** (not a backend
fault) — set `NumValuesPerNode` to page, or narrow the window. A refusal whose text carries no number falls back to the gateway's
default ceiling (1,000,000). Every stream is still drained. The #686 reverse walk-back (segment budget `MaxTieClusterOverfetch + 2`)
inherits this: against a ceiling below that budget a dense segment fails `BadTooManyOperations` instead of being narrowed.
- **Paging (#669) is unaffected.** A full page still holds exactly `NumValuesPerNode` stored samples and carries a continuation
point, and a resumed page (which re-reads from the last returned timestamp inclusive, and the tie-cluster over-fetch
`start == end` read) no longer gets a fabricated row at its resume point. A read is complete when the stream came back short,
@@ -706,10 +754,63 @@ data.
asking for `NumValuesPerNode + m + 16` (m = the pre-window rows the first read saw, unclamped). Only if that re-read is still
short because of pre-window rows does the node fail `BadTooManyOperations` (not a backend fault) rather than answer a short page
that would end the client's paging early. Every gateway stream is still drained to completion.
- **`ReturnBounds` is not honoured** — on any backend: the server does not pass it to the data source, so no bounding values are
returned (see [Known limitations](#known-limitations)). The AVEVA boundary row is not used as a bound either: it is stamped at the
query start rather than carrying the time of the real prior sample, so it could only ever be a Part 11 *interpolated* bound, which
this backend does not compute.
- **`ReturnBounds` is served by the server, not by this row** (#691, [Bounding values](#bounding-values-returnbounds-691)). The AVEVA
boundary row is never used as a bound: it is stamped at the query start rather than carrying the time of the real prior sample, so
it could only ever be a Part 11 *interpolated* bound. The start bound on the Gateway is the real prior sample, found by the
#686 walk-back.
### Bounding values (`ReturnBounds`, #691)
⚠️ **Client-visible — a breaking change for .NET clients that rely on SDK defaults.** Until #691
`ReadRawModifiedDetails.ReturnBounds` was ignored on every backend. It is now honoured on **every** backend (SqlServer, QuestDB,
Gateway, and none), following OPC UA Part 11 §4.6 / §6.5.3.2 and checked against every row of Part 11 Table 1. **The OPC Foundation
.NET SDK's `new ReadRawModifiedDetails()` defaults BOTH `ReturnBounds = true` AND `IsReadModified = true`** (verified on
1.5.378.106), so:
- a client that clears `IsReadModified` but never sets `ReturnBounds` used to get stored values only and **now gets bounding values**;
- a client that sets neither (a bare `new ReadRawModifiedDetails { StartTime, EndTime }`) is a *modified* read with bounds, which
used to be served (or `BadHistoryOperationUnsupported` on a backend with no trail) and is **now `Bad_InvalidArgument`** (§6.5.3.3).
Set both explicitly. `Client.Shared` (Client.CLI / Client.UI) already does, and ScadaBridge does not issue HistoryRead. The kill
switch `ServerHistorian:HonorReturnBounds = false` restores the pre-#691 behaviour (both changes) fleet-wide.
- **What a bound is.** Forward (`StartTime < EndTime`): the start bound is the stored sample **at or immediately before**
StartTime, the end bound the one **at or immediately after** EndTime. Reverse mirrors it (start bound at or after StartTime, end
bound at or before EndTime), newest first. A sample exactly **on** StartTime is the bound and is already the data's first value
(nothing extra is added); a sample exactly on EndTime is added as the end bound (the data window excludes EndTime).
`StartTime == EndTime` reads forward: the data is the samples at that instant, the end bound the first sample strictly after it.
- **A bound is a stored sample of any quality** — a Bad sample or a comms-loss outage marker comes back as stored. Inside a tie
cluster the bound is the cluster's first sample in the read's direction.
- **Nothing at all ⇒ `GoodNoData`.** A read with no data in the window **and** neither bound found returns no values and
`GoodNoData` — the answer it gave before bounds (§6.5.3.2: bounds make the result a success only "if one or both exist"). This is
judged over the whole read (it only arises on a single, final first page); once any bound or value exists, a missing bound is
returned as its placeholder.
- **Missing bound:** `Bad_BoundNotFound`, null value, stamped with StartTime / EndTime. A one-sided read's open side (only a
StartTime, or only an EndTime) is never looked up; it ends the read as `Bad_BoundNotFound` stamped one second after (forward) or
before (reverse) the previous returned value (Table 1 footnotes a/b). `Bad_BoundNotSupported` is never returned — every backend
can find a bound.
- **Counting and paging.** Bounds **count toward `NumValuesPerNode`** (§6.5.3.2: "any Bounding Values returned are included in the
numValuesPerNode count"; `NumValuesPerNode = 1` returns the start bound alone). The start bound leads the **first** page and the
end bound ends the **last**: the pending end bound rides the continuation point, and when a page is exactly full the end bound is
the whole next page. Paging a bounded read at any page size concatenates to the unpaged answer.
- **How it is served — one authority, `HistoryBounds` in the OPC UA server.** "At or before T" is a newest-first read of `(0, T]`
capped at one — native on SqlServer/QuestDB, the #686 walk-back of bounded forward reads on the Gateway (a tag with no sample
before StartTime is proven empty down to 1970 by the #689 walk — after two empty segments it asks once for the whole remainder —
so a missing start bound costs a few small reads, not one per ×4 step). "At or after T" is a forward read
of `[T, MaxValue)` capped at one — on the **Gateway** it ends at **now + 1 day** instead (the walk-back's horizon; the
HistorianGateway only checks `start ≤ end`, so a year-9999 end would reach AVEVA), and an end bound beyond that horizon is
`Bad_BoundNotFound` without a read. SqlServer/QuestDB keep `MaxValue` (they bind it safely and may hold samples further ahead).
"Strictly after" starts one storage resolution later. On QuestDB, whose window bounds truncate to
the microsecond, "on the bound" compares at that resolution, so a sample at ⌊StartTime⌋ is never returned twice. A first page
therefore costs up to three backend reads (start bound, data, end bound). The `BoundingValueConformance` suite pins those exact
lookup reads on SqlServer and QuestDB.
⚠️ On the **Gateway** the start-bound walk-back is subject to the #689 per-millisecond tie limit: if the millisecond holding the
bound has more than `MaxTieClusterOverfetch` samples, the lookup cannot pick the cluster's last sample without reading it all, so
the **whole node** read fails `BadTooManyOperations` (even though the data read alone would succeed) rather than return a wrong bound.
- **Modified reads:** `IsReadModified = true` with `ReturnBounds = true` is `Bad_InvalidArgument` per node (Part 11 §6.5.3.3).
- **Kill switch:** `ServerHistorian:HonorReturnBounds` (default `true`). `false` ignores `ReturnBounds` exactly as before #691 —
stored samples only, and no modified-read rejection.
- **Not affected:** Processed, AtTime and Events reads (they have no `ReturnBounds`).
### Continuation-point paging (Events) (#669)
@@ -723,30 +824,46 @@ says is complete carries no continuation point. `NumValuesPerNode == 0` stays un
that many more rows than the cap, so every page short of the last is a FULL `NumValuesPerNode` page (#675; resumed pages used
to come back short by the skip).
- A tie cluster larger than the cap is paged within by over-fetching the half-open window `[T, T + resolution)` up to
`MaxTieClusterOverfetch`; beyond that bound the node answers `BadHistoryOperationUnsupported` (raise the bound).
`MaxTieClusterOverfetch`; beyond that bound the node answers `BadTooManyOperations` (raise the bound).
- **Tie exactness depends on the backend.** SqlServer and QuestDB order by `(EventTimeUtc, EventId)`, so the skip count is exact.
The Gateway backend has no stable intra-timestamp order (and only a truncation flag), so ties at a page boundary are
best-effort there.
The Gateway backend has no stable intra-timestamp order (and only a truncation flag), so ties at a page boundary of a
**forward** read are best-effort there; a walked-back **reverse** read (below) sorts them by `EventId` itself and is exact.
- **Reverse event reads page newest-first (#675).** `StartTime > EndTime` reads the Part 11 reverse window `(EndTime, StartTime]`
with events in DESCENDING `(EventTimeUtc, EventId)` order, so a cap keeps the **latest** N and a continuation point is issued
when the source flags more. The direction is stored **in the cursor**: a reverse point resumes reverse (and a forward one
forward) whatever the resume request's details say. Reverse tie clusters are over-fetched forward and sliced from the newest
end, exactly as Raw does. Before #675 a capped reverse read silently returned the EARLIEST N, ascending, Good, with no
continuation point.
- **End-only events reads: backward on SqlServer/QuestDB, forward on Gateway (#675).** `StartTime` unset
(`DateTime.MinValue`) with an `EndTime` and `NumValuesPerNode > 0` pages backward from `EndTime` over `(MinValue, EndTime]` —
the same rule as Raw — **only on a backend that reports `SupportsReverseEvents`** (SqlServer, QuestDB). On Gateway (the
default backend) and Null it keeps its pre-#675 forward `[MinValue, EndTime)` meaning, paged oldest-first, rather than
becoming a refused reverse read. Start-only reads page forward to an open upper bound (`DateTime.MaxValue`, UTC) everywhere.
Without a cap an end-only read is forward `[MinValue, EndTime)` everywhere.
- **Backends.** The server checks `IHistorianDataSource.SupportsReverseEvents` and then calls
`ReadEventsAsync(…, bool reverse, …)`. SqlServer and QuestDB serve reverse natively (`… > lo AND … <= hi ORDER BY time DESC,
id DESC`) and report `true`. **Gateway and Null report `false`**: an explicit reverse read (`StartTime > EndTime`) is answered
`BadHistoryOperationUnsupported` **without calling the backend** — the gateway streams ascending only and rejects `end < start`,
and draining the whole window to keep its tail would be exactly the unbounded read the cap prevents. Because the refusal is a
capability check, not a caught exception, a `NotSupportedException` thrown from inside a backend (a driver library) is logged
as a backend fault like any other. Every implementation (and any future wrapper) must declare both members itself — the
interface defaults (`false` / throw) would silently disable reverse reads; `HistorianDataSourceReverseEventsGuardTests` pins it.
- **End-only events reads page backward on every backend (#675, #691.4).** `StartTime` unset (`DateTime.MinValue`) with an
`EndTime` and `NumValuesPerNode > 0` pages backward from `EndTime` over `(MinValue, EndTime]` — the same rule as Raw.
Start-only reads page forward to an open upper bound (`DateTime.MaxValue`, UTC) everywhere. Without a cap an end-only read
is forward `[MinValue, EndTime)` everywhere.
- **Backends.** The server checks `IHistorianDataSource.SupportsReverseEvents`. SqlServer and QuestDB serve reverse natively
(`… > lo AND … <= hi ORDER BY time DESC, id DESC`, the `ReadEventsAsync(…, bool reverse, …)` overload) and report `true`.
**Gateway reports `false`** — the gateway streams ascending only and rejects `end < start` — and is never sent a reverse
read (Null reports `true` and answers empty either way, as for raw): since **#691.4** the server walks it back from bounded **forward** reads with the raw walk (#686/#689/#691:
page-sized ms-aligned segments filtered to `(segLo, segHi]`, a read the source flagged truncated is never answered from,
the now + 1 day / 1970 bounds, the remainder probe and the proven-empty stretch carried in the cursor). Event specifics:
- **Ties** are sorted by `EventId` (ordinal) on every read — the gateway orders by `EventTimeUtc` only, in any order within a
timestamp — so the page is DESCENDING `(EventTimeUtc, EventId)` and a resumed page's boundary skip lines up; the
oversized-cluster over-fetch keeps exactly the events at `T` (the gateway's end is inclusive) in the same order.
- **"More exists"** is found by walking for one event past the page (the walk has no source truncation flag to hand on).
- **Uncapped** reverse reads are one forward read of the window, sorted and reversed.
- ⚠️ **Also fixed (#691.4 review): uncapped events reads on Gateway.** `NumValuesPerNode = 0` reaches the source as
`int.MaxValue`, and `GatewayHistorianDataSource` pre-sized its result list to it ("Array dimensions exceeded") — every
uncapped Gateway events read, forward too, failed the node and counted as a backend failure. The capacity is now bounded.
- A timestamp holding more than `MaxTieClusterOverfetch` events ⇒ **`BadTooManyOperations`** for that node.
- ⚠️ **Gateway prerequisite:** the gateway's `RuntimeDb:EventReadMaxRows` must be `0` (its default, unlimited) or larger than
the walk's biggest read (`2N + 18` rows, a discovery read 1,026, a tie-cluster read up to `2·MaxTieClusterOverfetch + 4`).
That cap truncates **silently** (the stream just ends), so a segment it cut would look complete and drop its newest events —
the same blindness a forward read has to it, but on the end a reverse read is asking for.
⚠️ **Client-visible (#691.4).** On Gateway (the default backend) and Null an explicit reverse events read used to be refused
`BadHistoryOperationUnsupported` (#675), and an end-only capped events read returned the **oldest** N events ascending. Both
now return the latest events newest-first. Because the capability is checked, not a caught exception, a
`NotSupportedException` thrown from inside a backend (a driver library) is still logged as a backend fault. Every
implementation (and any future wrapper) must declare both members itself — the interface defaults (`false` / throw) would
silently send a native-reverse backend through the walk; `HistorianDataSourceReverseEventsGuardTests` pins it.
- **Gateway open upper bound.** The `DateTime.MaxValue` (UTC) bound of a start-only forward read is pinned offline through the
published client's proto `Timestamp` conversion (captured by an in-process gRPC server) and the FILETIME encoding
(`GatewayEventBoundsTests`). What a live AVEVA Historian does with a year-9999 bound is still unverified (needs the VPN).
@@ -865,8 +982,30 @@ fixed:
`HistorianTagDefinition.IntegralDivisor` is `0`, so `GatewayTagProvisioner` now sends **`IntegralDivisor = 1`** explicitly on every
`EnsureTags` — the value·seconds unit the SQL backends' `Total` uses. A tag provisioned earlier keeps its divisor until its next
`EnsureTags` (an upsert). The live magnitude check against AVEVA (a constant 1 integrated over 1 s reads 1) passed on 2026-10-12 at the AVEVA
level; end to end it waits on historiangw#27 (above). Reverse processed reads of the AVEVA-native `TimeAverage`/`Total` are not re-anchored by #677 (the gateway refuses `end < start`);
the raw-evaluated per-sample aggregates (#681) do follow #677. Since #681 every native bucket is stamped with its interval **start**
level; end to end it waits on historiangw#27 (above). ⚠️ **Native reads on the SQL backends' interval grid (#691.2, client-visible).** Both directions now use `BucketPlan`, served by
**forward** native requests only (`NativeAggregatePlan` — AVEVA anchors a native read's interval grid at the request start, and the
gateway refuses `start > end` and `interval <= 0`):
- **Reverse** (`StartTime > EndTime`) used to reach the gateway as `start > end` and fail the node (`InvalidArgument`, counted as a backend
failure). It now follows #677 like the SQL backends and the raw-evaluated per-sample aggregates (#681): window `(EndTime, StartTime]`,
intervals stepping back from `StartTime`, the clipped interval the **earliest**, each stamped with its **later** edge, newest first —
one read `[O + I·f, StartTime)` with the interval `I` for the whole intervals (`O = StartTime − n·I`, `f` = 1 when clipped) and a
second read over exactly the clipped span `[EndTime, O + I)` with its own width as the interval. A time-weighted value over `[a, b)` and
over `(a, b]` is the same integral (two instants apart), so each forward row is the reverse interval's value.
- **Forward** windows that are not a whole number of intervals used to send one request whose last AVEVA interval ran **past EndTime**;
now the whole intervals `[StartTime, StartTime + (n−1)·I)` are one read and the clipped **latest** interval `[StartTime + (n−1)·I,
EndTime)` is its own read over exactly that span. An interval wider than the window is sent as the window, and `interval = 0` as one
interval over the window (it used to reach the gateway as 0 and be refused). An empty window (`StartTime == EndTime`) sends nothing.
- Stamps match SqlServer/QuestDB (forward: earlier edge, ascending; reverse: later edge, descending), and the clipped interval carries
the **Partial** bit unless Bad (Part 13 §5.3.3.2 Case 3, as the SQL backends' time-weighted aggregates — which also mark the intervals
the tag's data starts/ends inside; AVEVA's rows carry no data range, so that is not reproduced). Rows are otherwise mapped as before
(#685). An interval AVEVA answers no row for is absent (the SQL backends answer it `BadNoData`).
- ⚠️ **Interval cap (client-visible):** forward native reads now go through `BucketPlan` too, so a window of more than 100,000
intervals (e.g. 1 s over 2 days) is refused up front with `BadHistoryOperationUnsupported`, as on the SQL backends — it used to be
served up to the gateway's own `MaxAggregateIntervals` (1,000,000). Not counted as a backend failure. A clipped interval shorter than
1 ms (a sub-ms window end) is sent as a sub-ms interval request; re-check it in the post-historiangw#27 live leg.
- ⚠️ **Until historiangw#27 is fixed, native values are wrong in BOTH directions with a Good status:** AVEVA answers one row per request
(the whole span's value), which lands on that request's first interval as a Good, Calculated value while the other intervals are
absent. Pinned against a native-mode fake only — re-run the forward and reverse native legs live once the gateway is fixed. Since #681 every native bucket is stamped with its interval **start**
(Part 13; it was the interval end) and carries the Calculated info bits (`Good` ⇒ `0x00000401`, no data ⇒ `BadNoData | 0x401`).
The live AVEVA gate (2026-10-12) confirmed `start_time` is the interval start (and the `value·s` unit). Previously noted as
pending: it reads the aggregate row's `start_time`, which histsdk decodes from a row
@@ -912,9 +1051,11 @@ and evicts its shared historian session (see the HistorianGateway issue text in
streaming gateway and keeps its `+ 18`. Past the cap the rest of the (bounded) stream is drained, then the node fails
`BadTooManyOperations`; a stream that delivers the whole budget cannot be proven complete and fails the same way (so with the
default cap a window of more than ~999,982 rows fails, as it would one row later). The gateway's own refusal of the budget
(`InvalidArgument` "max_values exceeds the server ceiling" — `Historian:ReadLimits:MaxValuesCeiling` lowered below the default,
with `AllowStreamingBeyondCeiling=false`) is also `BadTooManyOperations`, not a backend fault — lower `MaxTimeWeightedRawRows` to
`ceiling − 18` on such a gateway. A raw read's budget is **not** clamped (see [#687](#gateway-raw-reads-return-stored-samples-only-687)). A caller cancellation (client cancel, `HistoryReadDeadline`) is not counted as a backend fault.
(`InvalidArgument` "max_values exceeds the server ceiling of N" — `Historian:ReadLimits:MaxValuesCeiling` lowered below the
default, or an unbounded `MaxTimeWeightedRawRows ≤ 0`, with `AllowStreamingBeyondCeiling=false`) is **retried once with N as the
budget** (#691.3): a window that fits is served, one that delivers all N rows fails `BadTooManyOperations` as above (not a backend
fault). Lowering `MaxTimeWeightedRawRows` to `ceiling − 18` on such a gateway is no longer needed — it only saves the refused
round-trip. A raw read's budget is **not** clamped (see [#687](#gateway-raw-reads-return-stored-samples-only-687)). A caller cancellation (client cancel, `HistoryReadDeadline`) is not counted as a backend fault.
- **Bucket statistics** are `RawBucketAccumulator`, the C# statement of the SQL backends' per-sample columns — incl. #684:
Average / Minimum / Maximum / Range / *ActualTime over **Good** samples only, the Uncertain extremes feeding only the Min/Max
status rule, and an Uncertain String/DateTime sample not refusing the Good-value aggregates. A change to those SQL columns
@@ -1009,8 +1150,8 @@ whatever the SDK sets — #682 touches only the i=11201 folder (follow-up candid
sample per requested timestamp, so the single-shot backend returns the complete result in one
read and there is no "full page ⇒ maybe more" signal to page on. Returning the full result with
no continuation point is spec-conformant.
- **`ReturnBounds` is ignored** (every backend): Raw reads return stored values only; no bounding values are added at the window
edges. The node manager does not pass the flag to `IHistorianDataSource`.
- **Bounds are raw only:** `ReturnBounds` (#691, [Bounding values](#bounding-values-returnbounds-691)) never interpolates a bound —
it is always a stored sample or `Bad_BoundNotFound`.
- **No modified-value history on the `Gateway` backend.** Modified reads (`IsReadModified = true`) are served by
`SqlServer` and `QuestDb` ([Modified-value history](#modified-value-history-part-11), #668); the HistorianGateway exposes no
modified/edited-history surface, so on `Gateway` (and with no historian) they return `BadHistoryOperationUnsupported`.
@@ -120,9 +120,9 @@ public interface IHistorianDataSource : IDisposable
/// <summary>
/// Whether this backend can serve a reverse (newest-first) events read — the reverse overload of
/// <see cref="ReadEventsAsync(string?, DateTime, DateTime, int, bool, CancellationToken)"/> (Gitea #675). The server
/// checks it BEFORE calling the source: an explicit reverse read (<c>StartTime &gt; EndTime</c>) on a backend that
/// cannot is answered <c>BadHistoryOperationUnsupported</c> without a call, and an end-only capped read, which Part 11
/// reads backward, keeps its forward <c>[MinValue, EndTime)</c> meaning there. Default <c>false</c>; an
/// checks it BEFORE calling the source: a reverse read (explicit <c>StartTime &gt; EndTime</c>, or an end-only capped
/// read, which Part 11 reads backward) on a backend that cannot is never sent the reverse overload — since Gitea
/// #691.4 the server walks it back from bounded FORWARD reads instead (#675 refused it). Default <c>false</c>; an
/// implementation (or a wrapper) declares it itself — a guard test pins that.
/// </summary>
bool SupportsReverseEvents => false;
@@ -143,10 +143,10 @@ public interface IHistorianDataSource : IDisposable
/// </para>
/// <para>
/// The default forwards a forward read to the five-argument overload and throws
/// <see cref="NotSupportedException"/> for a reverse one — a backend that cannot read newest-first (Gateway,
/// Null). The server never relies on that throw: it consults <see cref="SupportsReverseEvents"/> first and
/// answers <c>BadHistoryOperationUnsupported</c> without calling, rather than returning the earliest events in
/// ascending order as if they were the answer. A backend that serves reverse must override BOTH members.
/// <see cref="NotSupportedException"/> for a reverse one — a backend that cannot read newest-first (the
/// Gateway). The server never relies on that throw: it consults <see cref="SupportsReverseEvents"/> first and walks
/// such a read back from forward reads (#691.4), never returning the earliest events in ascending order as if
/// they were the answer. A backend that serves reverse natively must override BOTH members.
/// </para>
/// </remarks>
/// <param name="sourceName">The source name to filter events, or null to return events from all sources.</param>
@@ -61,13 +61,12 @@ public sealed class NullHistorianDataSource : IHistorianDataSource
public bool SupportsReverseRaw => true;
/// <inheritdoc />
/// <remarks><c>false</c> (Gitea #675): there is no history to read in either direction, and an explicit reverse read
/// is answered <c>BadHistoryOperationUnsupported</c> like any backend that cannot read newest-first.</remarks>
public bool SupportsReverseEvents => false;
/// <remarks><c>true</c>, like <see cref="SupportsReverseRaw"/> (Gitea #691.4 review; <c>false</c> under #675): there is no
/// history to read in either direction — no need for the server to walk back through empty forward reads.</remarks>
public bool SupportsReverseEvents => true;
/// <inheritdoc />
/// <remarks>Forward ⇒ the empty result; reverse ⇒ <see cref="NotSupportedException"/>, the interface contract for a
/// backend without <see cref="SupportsReverseEvents"/>.</remarks>
/// <remarks>The empty result in either direction.</remarks>
public Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName,
DateTime startUtc,
@@ -75,9 +74,7 @@ public sealed class NullHistorianDataSource : IHistorianDataSource
int maxEvents,
bool reverse,
CancellationToken cancellationToken) =>
reverse
? throw new NotSupportedException("The null historian cannot read events newest-first (reverse).")
: Task.FromResult(EmptyEvents);
Task.FromResult(EmptyEvents);
/// <inheritdoc />
/// <remarks>
@@ -47,6 +47,9 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
private readonly int _maxRawRows;
private readonly TimeProvider _time;
/// <summary>A strict gateway's row ceiling, learned from its first refusal (#691.3).</summary>
private readonly GatewayRowCeiling _ceiling = new();
private readonly object _healthLock = new();
private DateTime? _lastSuccessUtc;
private DateTime? _lastFailureUtc;
@@ -144,8 +147,12 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
/// gateway reject <c>max_values &lt;= 0</c>, #686). A cap asks for <c>cap + </c>
/// <see cref="RawAggregateReader.PreWindowAllowance"/> rows — NOT clamped at the gateway's default ceiling (#688
/// clamps only the per-sample aggregate budget): AVEVA always emits the boundary row, so a clamped budget would make
/// every page size within the allowance of the ceiling come back short. A strict gateway refuses an over-ceiling
/// budget instead (⇒ <see cref="HistoryReadLimitExceededException"/>, <c>BadTooManyOperations</c>).
/// every page size within the allowance of the ceiling come back short. A strict gateway
/// (<c>AllowStreamingBeyondCeiling=false</c>) refuses an over-ceiling budget and names its ceiling; the read is
/// then re-issued ONCE with the ceiling as budget (#691.3, <see cref="GatewayReadLimits"/>) — so an uncapped read
/// of a small window is served — and a ceiling-bounded read that is not provably complete (it delivered the
/// whole ceiling without reaching the window end, or a page short of its cap) fails
/// <see cref="HistoryReadLimitExceededException"/> (<c>BadTooManyOperations</c>), never a truncated answer.
/// </para>
/// <para>
/// <b>Completeness.</b> A full page must hold exactly the cap of stored samples — a short page ends the client's
@@ -166,22 +173,43 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
// The gateway seam caps with an int; OPC UA hands us a uint, so clamp to int range.
var cap = (int)Math.Min(maxValuesPerNode, int.MaxValue);
if (cap == 0)
return (await ReadRawPassAsync(fullReference, startUtc, endUtc, cap, int.MaxValue, cancellationToken)
.ConfigureAwait(false)).Samples;
{
var all = await PassAsync(int.MaxValue).ConfigureAwait(false);
if (all.Result.Complete) return all.Result.Samples;
// Only reachable on a strict gateway (#691.3): the ceiling re-read delivered the whole ceiling without reaching
// the window end — the window may hold more, and "all values" is never answered from a truncated read.
throw new HistoryReadLimitExceededException(cap,
$"The raw read of '{fullReference}' holds more than the HistorianGateway's row ceiling ({all.Budget}, its " +
"Historian:ReadLimits:MaxValuesCeiling with AllowStreamingBeyondCeiling=false); set NumValuesPerNode to page, " +
"or narrow the window.");
}
var first = await ReadRawPassAsync(fullReference, startUtc, endUtc, cap,
(int)Math.Min(int.MaxValue, (long)cap + RawAggregateReader.PreWindowAllowance), cancellationToken).ConfigureAwait(false);
if (first.Complete) return first.Samples;
var asked = (int)Math.Min(int.MaxValue, (long)cap + RawAggregateReader.PreWindowAllowance);
var first = await PassAsync(asked).ConfigureAwait(false);
if (first.Result.Complete) return first.Result.Samples;
// A strict gateway's ceiling already bounded that read (#691.3): a larger budget would only be refused again.
if (first.Budget < asked)
throw new HistoryReadLimitExceededException(cap,
$"The raw read of '{fullReference}' could not be read completely within the HistorianGateway's row ceiling " +
$"({first.Budget}, its Historian:ReadLimits:MaxValuesCeiling with AllowStreamingBeyondCeiling=false); lower " +
"NumValuesPerNode or narrow the window.");
// The margin crowded the window out: one bounded re-read with room for the pre-window rows already seen.
var second = await ReadRawPassAsync(fullReference, startUtc, endUtc, cap,
(int)Math.Min(int.MaxValue, (long)cap + first.PreWindowRows + RawAggregateReader.PreWindowAllowance),
cancellationToken).ConfigureAwait(false);
if (second.Complete) return second.Samples;
var second = await PassAsync(
(int)Math.Min(int.MaxValue, (long)cap + first.Result.PreWindowRows + RawAggregateReader.PreWindowAllowance))
.ConfigureAwait(false);
if (second.Result.Complete) return second.Result.Samples;
throw new HistoryReadLimitExceededException(cap,
$"The raw read of '{fullReference}' could not be read completely: more than {second.PreWindowRows} rows lie in " +
$"The raw read of '{fullReference}' could not be read completely: more than {second.Result.PreWindowRows} rows lie in " +
$"the {RawAggregateReader.BoundaryMargin.TotalMilliseconds} ms before the window and exhausted the row budget.");
// One drained pass; a strict gateway's refusal of the budget is retried once at the ceiling it names (#691.3).
Task<((List<HistorianSample> Samples, int PreWindowRows, bool Complete) Result, int Budget)> PassAsync(int budget) =>
_ceiling.ReadAsync(budget,
b => ReadRawPassAsync(fullReference, startUtc, endUtc, cap, b, cancellationToken),
(r, _) => !r.Complete);
}
/// <summary>One forward raw read of a window, drained, with its pre-window row count and whether it is complete.</summary>
@@ -206,7 +234,9 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
else if (sample.Timestamp is not null) reachedEnd = true;
}
var complete = received < budget || samples.Count >= cap || reachedEnd || cap == 0;
// cap == 0 ("all values") is complete only when the stream proves it: short of the budget (always, unless a strict
// gateway's ceiling became the budget — #691.3) or past the window end.
var complete = received < budget || reachedEnd || (cap > 0 && samples.Count >= cap);
return (samples, preWindow, complete);
}
@@ -223,7 +253,9 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
/// <para>
/// <b>TimeAverage / Total</b> stay native AVEVA retrieval modes (<see cref="AggregateModeMapper"/>), stamped with
/// the interval start and the Calculated bit; AVEVA's extra row for the interval before the window is dropped and
/// an Uncertain interval keeps its value (#685, <see cref="SampleMapper.ToAggregateSnapshots"/>). Everything else is <see cref="HistoryAggregateNotSupportedException"/>
/// an Uncertain interval keeps its value (#685). Both directions use the SQL backends' interval grid with forward
/// requests only — a reverse read (which the gateway refuses) anchored at its later time (#677), the clipped interval
/// read over exactly its span and marked Partial (#691.2, <see cref="NativeAggregatePlan"/>). Everything else is <see cref="HistoryAggregateNotSupportedException"/>
/// (see <see cref="ServedAggregates"/>).
/// </para>
/// </remarks>
@@ -240,22 +272,36 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
// Mapped OUTSIDE the failure-recording try: an aggregate the gateway has no mode for is a client
// request the server answers BadAggregateNotSupported, not a backend fault to count (#285).
var mode = AggregateModeMapper.ToRetrievalMode(aggregate);
// Planned OUTSIDE the failure-recording try: too many intervals is a client request limit, not a backend fault.
return await ReadNativeAsync(fullReference, BucketPlan.Create(startUtc, endUtc, interval), mode, cancellationToken)
.ConfigureAwait(false);
}
/// <summary>
/// A native TimeAverage / Total read on the SQL backends' interval grid, either direction (#691.2): forward native
/// requests only, the clipped interval read over exactly its span. See <see cref="NativeAggregatePlan"/>.
/// </summary>
private async Task<HistoryReadResult> ReadNativeAsync(string fullReference, BucketPlan plan, RetrievalMode mode,
CancellationToken cancellationToken)
{
if (plan.BucketCount == 0) return new HistoryReadResult([], ContinuationPoint: null);
try
{
// TimeAverage / Total are native gateway retrieval modes (TimeWeightedAverage / Integral) — no
// client-side scaling.
var buckets = new List<HistorianAggregateSample>();
await foreach (var bucket in _client
.ReadAggregateAsync(fullReference, startUtc, endUtc, mode, interval, cancellationToken)
.ConfigureAwait(false))
var byBucket = new Dictionary<long, DataValueSnapshot>();
foreach (var request in NativeAggregatePlan.Requests(plan))
{
buckets.Add(bucket);
var buckets = new List<HistorianAggregateSample>();
await foreach (var bucket in _client
.ReadAggregateAsync(fullReference, request.StartUtc, request.EndUtc, mode, request.Interval, cancellationToken)
.ConfigureAwait(false))
{
buckets.Add(bucket); // drained to completion, mapped afterwards
}
NativeAggregatePlan.Place(plan, request, buckets, byBucket);
}
// Filtered while mapping, after the stream was drained (#685: AVEVA's extra pre-window row).
var snapshots = SampleMapper.ToAggregateSnapshots(buckets, startUtc, endUtc);
RecordOutcome(success: true, error: null);
return new HistoryReadResult(snapshots, ContinuationPoint: null);
return new HistoryReadResult(NativeAggregatePlan.Assemble(plan, byBucket), ContinuationPoint: null);
}
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
{
@@ -280,7 +326,7 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
{
var now = _time.GetUtcNow().UtcDateTime;
var values = await RawAggregateReader
.ReadAsync(_client, fullReference, aggregate, plan, _maxRawRows, now, cancellationToken)
.ReadAsync(_client, _ceiling, fullReference, aggregate, plan, _maxRawRows, now, cancellationToken)
.ConfigureAwait(false);
RecordOutcome(success: true, error: null);
return new HistoryReadResult(values, ContinuationPoint: null);
@@ -315,14 +361,7 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
/// True when <paramref name="ex"/> (or an inner exception — the client package wraps the <see cref="RpcException"/>)
/// is the gateway's <c>InvalidArgument</c> "max_values exceeds the server ceiling" rejection.
/// </summary>
internal static bool IsMaxValuesCeilingRejection(Exception ex)
{
for (var e = ex; e is not null; e = e.InnerException)
if (e is RpcException { StatusCode: StatusCode.InvalidArgument } rpc
&& rpc.Status.Detail.Contains("max_values exceeds the server ceiling", StringComparison.Ordinal))
return true;
return false;
}
internal static bool IsMaxValuesCeilingRejection(Exception ex) => GatewayReadLimits.IsCeilingRejection(ex);
/// <inheritdoc />
public async Task<HistoryReadResult> ReadAtTimeAsync(
@@ -354,8 +393,9 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
/// <inheritdoc />
/// <remarks>
/// <c>false</c>: the gateway's <c>ReadEvents</c> streams ascending only (Gitea #675). The server therefore refuses an
/// explicit reverse events read without calling, and keeps an end-only capped read forward.
/// <c>false</c>: the gateway's <c>ReadEvents</c> streams ascending only (Gitea #675). The server never sends it a
/// reverse read; since #691.4 it walks one back from bounded forward reads (ties sorted by EventId — the gateway
/// orders by event time only).
/// </remarks>
public bool SupportsReverseEvents => false;
@@ -365,10 +405,9 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
/// <inheritdoc />
/// <remarks>
/// A reverse (newest-first) read is refused with <see cref="NotSupportedException"/> — the server reports
/// <c>BadHistoryOperationUnsupported</c> (Gitea #675). The gateway's <c>ReadEvents</c> streams ascending only and
/// rejects <c>end &lt; start</c>, so the only way to serve "the latest N" would be to drain the whole window and
/// keep its tail, an unbounded read the cap exists to prevent. Nothing is sent.
/// A reverse (newest-first) read is refused with <see cref="NotSupportedException"/> (Gitea #675) — the gateway's
/// <c>ReadEvents</c> streams ascending only and rejects <c>end &lt; start</c>. The server never calls this with
/// <c>reverse</c>: it walks a reverse read back from bounded forward reads (#691.4). Nothing is sent.
/// </remarks>
public Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, bool reverse,
@@ -386,7 +425,8 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
try
{
var hasCap = maxEvents > 0;
var collected = new List<HistoricalEvent>(hasCap ? maxEvents : 0);
// Bounded initial capacity: an uncapped read arrives as int.MaxValue ("Array dimensions exceeded", #691.4 review).
var collected = new List<HistoricalEvent>(hasCap ? Math.Min(maxEvents, 1024) : 0);
var truncated = false;
await foreach (var wireEvent in _client
@@ -0,0 +1,135 @@
using System.Globalization;
using System.Text.RegularExpressions;
using Grpc.Core;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway;
/// <summary>
/// The HistorianGateway's raw-read row ceiling (<c>Historian:ReadLimits:MaxValuesCeiling</c>, default 1,000,000) as seen
/// from this side of the wire. With <c>AllowStreamingBeyondCeiling=false</c> (a "strict" gateway) the gateway refuses
/// any <c>ReadRaw</c> whose <c>max_values</c> exceeds the ceiling with <c>InvalidArgument</c> — BEFORE it streams a row
/// — and names the ceiling in the refusal (<c>"max_values exceeds the server ceiling of {n}; …"</c>). The gateway has
/// no RPC that reports the ceiling, so the refusal is how it is learned (#691.3).
/// </summary>
/// <remarks>
/// A read whose budget the gateway refuses is re-issued ONCE with the ceiling as its budget, and the ceiling is
/// remembered per data source (<see cref="GatewayRowCeiling"/>); the caller treats a stream that delivered the whole
/// ceiling as possibly truncated (never answered from).
/// </remarks>
internal static class GatewayReadLimits
{
private const string CeilingRejectionText = "max_values exceeds the server ceiling";
private static readonly Regex CeilingNumber = new(@"max_values exceeds the server ceiling of (\d+)",
RegexOptions.CultureInvariant | RegexOptions.Compiled);
/// <summary>
/// True when <paramref name="ex"/> (or an inner exception — the client package wraps the <see cref="RpcException"/>)
/// is the gateway's <c>InvalidArgument</c> "max_values exceeds the server ceiling" rejection.
/// </summary>
/// <param name="ex">The exception.</param>
/// <returns>Whether it is the ceiling rejection.</returns>
public static bool IsCeilingRejection(Exception ex) => Rejection(ex) is not null;
/// <summary>
/// The ceiling a ceiling rejection names, or — when <paramref name="ex"/> is a ceiling rejection that does not
/// carry a parseable number — the gateway's default (<see cref="RawAggregateReader.GatewayDefaultMaxValuesCeiling"/>).
/// </summary>
/// <param name="ex">The exception.</param>
/// <param name="ceiling">The ceiling (&gt; 0) when this returns true.</param>
/// <returns>Whether <paramref name="ex"/> is a ceiling rejection.</returns>
public static bool TryGetCeiling(Exception ex, out int ceiling)
{
ceiling = 0;
if (Rejection(ex) is not { } rpc) return false;
var m = CeilingNumber.Match(rpc.Status.Detail);
ceiling = m.Success && int.TryParse(m.Groups[1].Value, NumberStyles.None, CultureInfo.InvariantCulture, out var n) && n > 0
? n
: RawAggregateReader.GatewayDefaultMaxValuesCeiling;
return true;
}
private static RpcException? Rejection(Exception ex)
{
for (var e = ex; e is not null; e = e.InnerException)
if (e is RpcException { StatusCode: StatusCode.InvalidArgument } rpc
&& rpc.Status.Detail.Contains(CeilingRejectionText, StringComparison.Ordinal))
return rpc;
return null;
}
}
/// <summary>
/// A strict gateway's row ceiling as learned by ONE data source (#691.3 review): every refused attempt counts as an
/// error in the gateway's own outcome metrics, so the ceiling a refusal names is remembered and later reads ask for at
/// most it — no refusal per read in the steady state. Thread-safe (one <see cref="int"/>, read/written atomically; a
/// race costs at most one extra refusal or re-probe, never a wrong answer).
/// </summary>
/// <remarks>
/// <para>
/// <b>Rules.</b> (1) Unknown ceiling: ask for the full budget; a refusal naming <c>N</c> &lt; budget learns <c>N</c> and
/// the read is re-issued once at <c>N</c>. (2) Learned ceiling <c>C</c> below the budget: ask for <c>C</c> — a refusal
/// there (the gateway was LOWERED) re-learns its <c>N</c> as in (1). (3) If that <c>C</c>-bounded read came back
/// full but not provably complete, the gateway may have been RAISED (or switched to streaming) since <c>C</c> was
/// learned, and answering <c>BadTooManyOperations</c> from a stale ceiling would be wrong: the read is re-probed ONCE
/// at the full budget — served ⇒ the ceiling is forgotten; refused naming a larger <c>N</c> ⇒ <c>N</c> is learned and
/// the read is re-issued at <c>N</c>; refused naming <c>C</c> again ⇒ the <c>C</c>-bounded result stands (no repeat).
/// </para>
/// <para>
/// <b>Why no TTL.</b> A raise only matters to a read that needs more than <c>C</c>, and such a read re-probes anyway
/// (rule 3), so a timer would only add refusals for reads that fit. The cost of the rule is one refused re-probe per
/// genuinely over-ceiling read — a read that fails <c>BadTooManyOperations</c> whatever is done.
/// </para>
/// </remarks>
internal sealed class GatewayRowCeiling
{
private int _learned; // 0 = unknown / not strict
/// <summary>The learned ceiling (0 = none).</summary>
public int Learned => Volatile.Read(ref _learned);
/// <summary>Runs a drained read under the rules above.</summary>
/// <typeparam name="T">The pass result.</typeparam>
/// <param name="budget">The <c>max_values</c> the read wants.</param>
/// <param name="pass">One drained read with the given budget.</param>
/// <param name="needsMore">True when a pass with the given budget delivered it all without proving completeness.</param>
/// <returns>The result and the budget it was read with.</returns>
public async Task<(T Result, int Budget)> ReadAsync<T>(int budget, Func<int, Task<T>> pass, Func<T, int, bool> needsMore)
{
var learned = Learned;
if (learned <= 0 || learned >= budget) return await AttemptAsync(budget, pass).ConfigureAwait(false);
var bounded = await AttemptAsync(learned, pass).ConfigureAwait(false);
if (bounded.Budget < learned || !needsMore(bounded.Result, bounded.Budget)) return bounded;
// Rule 3: full at a possibly stale ceiling — re-probe once at the full budget.
int fresh;
try
{
var full = await pass(budget).ConfigureAwait(false);
Interlocked.CompareExchange(ref _learned, 0, learned);
return (full, budget);
}
catch (Exception ex) when (GatewayReadLimits.TryGetCeiling(ex, out fresh) && fresh < budget)
{
// fall through (outside the catch)
}
Volatile.Write(ref _learned, fresh);
return fresh > learned ? (await pass(fresh).ConfigureAwait(false), fresh) : bounded;
}
private async Task<(T Result, int Budget)> AttemptAsync<T>(int budget, Func<int, Task<T>> pass)
{
int ceiling;
try
{
return (await pass(budget).ConfigureAwait(false), budget);
}
catch (Exception ex) when (GatewayReadLimits.TryGetCeiling(ex, out ceiling) && ceiling < budget)
{
// fall through to the retry (outside the catch, so a second refusal is not nested in this one)
}
Volatile.Write(ref _learned, ceiling);
return (await pass(ceiling).ConfigureAwait(false), ceiling);
}
}
@@ -23,11 +23,21 @@ internal static class SampleMapper
/// <summary>Maps a single raw sample to a value snapshot.</summary>
/// <param name="sample">The gateway raw sample.</param>
/// <returns>The driver-agnostic snapshot.</returns>
/// <remarks>
/// #690: a Bad row carries no value. histsdk puts a value on EVERY row (<c>NumericValue = 0</c> on a numeric row,
/// <c>StringValue = ""</c> on a string one), so a stored NULL arrives as <c>0</c> / <c>""</c> with a Bad OPC quality
/// (live: <c>q=1 opc=0</c>) — that 0 is not a value, and AVEVA stores a Bad-quality sample as NULL, so a Bad row never
/// carries a real one. The value is therefore dropped whenever the mapped status is Bad, matching the SqlServer/QuestDB
/// backends (an Empty sample is null + Bad) and the aggregate mapper's no-data rows. Uncertain rows keep their value.
/// </remarks>
public static DataValueSnapshot ToSnapshot(HistorianSample sample)
{
var status = ToStatus(sample);
// proto3 explicit presence: prefer the numeric value, else the string value, else null.
object? value;
if (sample.HasNumericValue)
if (StatusBits.IsBad(status))
value = null;
else if (sample.HasNumericValue)
value = sample.NumericValue; // boxes as System.Double
else if (sample.HasStringValue)
value = sample.StringValue;
@@ -36,7 +46,7 @@ internal static class SampleMapper
return new DataValueSnapshot(
Value: value,
StatusCode: ToStatus(sample),
StatusCode: status,
SourceTimestampUtc: sample.Timestamp?.ToDateTime(), // Utc kind
ServerTimestampUtc: DateTime.UtcNow);
}
@@ -0,0 +1,112 @@
using ZB.MOM.WW.HistorianGateway.Contracts.Grpc;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Mapping;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway;
/// <summary>
/// Serves a processed read of the AVEVA-native <c>TimeAverage</c> / <c>Total</c> on the SQL backends' interval grid
/// (<see cref="BucketPlan"/>), in either direction: forward — window <c>[start, end)</c>, intervals stepping forward
/// from <c>start</c>, the clipped interval the LATEST, each stamped with its earlier edge, oldest first; reverse (#677,
/// #691.2) — window <c>(end, start]</c>, intervals stepping back from <c>start</c>, the clipped interval the EARLIEST,
/// each stamped with its later edge, newest first. The gateway refuses <c>start &gt; end</c> and <c>interval &lt;= 0</c>,
/// so every request it sees is forward with a positive interval.
/// </summary>
/// <remarks>
/// <para>
/// <b>Forward requests on the plan's grid.</b> AVEVA anchors a native read's interval grid at the request start (live,
/// 2026-10-12: rows at <c>start + j·interval</c>, plus one extra row for the interval before it, #685). So the plan's
/// whole intervals are requested as ONE forward read spanning exactly them with the interval <c>I</c>, and the
/// clipped interval (when the window is not a whole number of intervals) as a second forward read over exactly its
/// span with its own width as the interval — AVEVA never averages time outside the window (before it on a reverse
/// read, past <c>EndTime</c> on a forward one). An interval wider than the window is sent as the window, and
/// <c>interval &lt;= 0</c> as one interval over the whole window (it used to reach the gateway as 0 and be refused).
/// </para>
/// <para>
/// <b>Soundness of reverse.</b> A time-weighted average / integral over <c>[a, b)</c> and over <c>(a, b]</c> integrate
/// the same function over sets differing by two instants, so a forward native row IS the value of the reverse
/// interval with the same edges. The one place the half-open side could matter is AVEVA's status for an interval
/// whose only non-good sample lies exactly on an edge — accepted, as on every native row.
/// </para>
/// <para>
/// <b>Mapping.</b> Each row is mapped as before (<see cref="SampleMapper.ToAggregateSnapshot"/>: Uncertain keeps its
/// value, no data ⇒ <c>BadNoData</c>, Calculated bits), placed by its interval start (else end), restamped with
/// <see cref="BucketPlan.BucketTimestamp"/>, and the clipped interval carries the Partial bit unless Bad (Part 13
/// §5.3.3.2 Case 3, as the SQL backends' time-weighted aggregates — which additionally mark the intervals the data
/// starts/ends inside; AVEVA's rows carry no data range, so those are not reproduced). An interval AVEVA answers no
/// row for is absent (not a fabricated <c>BadNoData</c>, unlike the SQL backends).
/// </para>
/// <para>
/// ⚠️ Today historiangw#27 (the interval reaches AVEVA in ticks where it reads milliseconds) makes AVEVA answer ONE
/// Good row per request — the whole-intervals read lands on its earliest interval with a whole-span value and the rest
/// are absent — in both directions. The mapping needs no change once the gateway is fixed.
/// </para>
/// </remarks>
internal static class NativeAggregatePlan
{
/// <summary>One forward native request and the ascending bucket index of its first interval.</summary>
/// <param name="StartUtc">Forward request start (inclusive).</param>
/// <param name="EndUtc">Forward request end.</param>
/// <param name="Interval">The request's interval (&gt; 0).</param>
/// <param name="FirstBucket">Ascending plan bucket of the request's first interval.</param>
/// <param name="BucketCount">Number of plan buckets the request covers.</param>
internal readonly record struct Request(DateTime StartUtc, DateTime EndUtc, TimeSpan Interval, long FirstBucket, long BucketCount);
/// <summary>The forward requests serving a plan: the whole intervals first, then the clipped one (if any).</summary>
/// <param name="plan">A plan with at least one bucket.</param>
/// <returns>One or two forward requests.</returns>
public static IReadOnlyList<Request> Requests(BucketPlan plan)
{
var interval = TimeSpan.FromTicks(plan.IntervalTicks);
var n = plan.BucketCount;
var clipped = plan.IsTruncated(0) ? 0L : plan.IsTruncated(n - 1) ? n - 1 : -1L;
if (clipped < 0) return [new Request(plan.StartUtc, plan.EndUtc, interval, 0, n)];
var clipLo = new DateTime(plan.LowerTicks(clipped), DateTimeKind.Utc);
var clipHi = new DateTime(plan.UpperTicks(clipped), DateTimeKind.Utc);
var requests = new List<Request>(2);
if (n > 1)
requests.Add(clipped == 0
? new Request(clipHi, plan.EndUtc, interval, 1, n - 1) // reverse: the clipped one is the earliest
: new Request(plan.StartUtc, clipLo, interval, 0, n - 1)); // forward: the clipped one is the latest
requests.Add(new Request(clipLo, clipHi, clipHi - clipLo, clipped, 1));
return requests;
}
/// <summary>
/// Maps a request's rows onto the plan's buckets: rows outside the request window (AVEVA's pre-window row, #685)
/// are dropped; each in-window row is placed by its interval start (or, without one, its end).
/// </summary>
/// <param name="plan">The plan.</param>
/// <param name="request">The forward request the rows answer.</param>
/// <param name="rows">The drained rows.</param>
/// <param name="byBucket">Results by ascending bucket index (first row per bucket wins).</param>
public static void Place(BucketPlan plan, Request request, IEnumerable<HistorianAggregateSample> rows,
Dictionary<long, DataValueSnapshot> byBucket)
{
var interval = request.Interval.Ticks;
foreach (var row in rows)
{
if (!SampleMapper.IsInWindow(row, request.StartUtc, request.EndUtc)) continue;
long offset = row.StartTime is { } s
? (s.ToDateTime().Ticks - request.StartUtc.Ticks) / interval
: (row.EndTime.ToDateTime().Ticks - request.StartUtc.Ticks - 1) / interval; // IsInWindow ⇒ end in (lo, hi]
if (offset < 0 || offset >= request.BucketCount) continue;
var index = request.FirstBucket + offset;
if (byBucket.ContainsKey(index)) continue;
var snapshot = SampleMapper.ToAggregateSnapshot(row);
var status = snapshot.StatusCode;
if (plan.IsTruncated(index) && !StatusBits.IsBad(status)) status |= HistorianInfoBits.HistorianPartial;
byBucket[index] = snapshot with { StatusCode = status, SourceTimestampUtc = plan.BucketTimestamp(index) };
}
}
/// <summary>The placed results in the read's order (ascending forward, descending reverse).</summary>
/// <param name="plan">The plan.</param>
/// <param name="byBucket">Results by ascending bucket index.</param>
/// <returns>The results.</returns>
public static IReadOnlyList<DataValueSnapshot> Assemble(BucketPlan plan, IReadOnlyDictionary<long, DataValueSnapshot> byBucket) =>
(plan.Reversed ? byBucket.OrderByDescending(kv => kv.Key) : byBucket.OrderBy(kv => kv.Key)).Select(kv => kv.Value).ToList();
}
@@ -49,7 +49,10 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway;
/// requested row budget — has been drained. The gateway is asked for at most <c>cap + 2 + </c>
/// <see cref="PreWindowAllowance"/> rows, clamped at <see cref="GatewayDefaultMaxValuesCeiling"/> while the cap is at
/// or under it (#688, <see cref="ClampToGatewayCeiling"/>); a stream that delivers all of them cannot be proven
/// complete and fails the same way rather than answering from a truncated window.
/// complete and fails the same way rather than answering from a truncated window. A strict gateway whose ceiling is
/// below that budget (a ceiling under 1,000,000, or an unbounded cap) refuses it naming its ceiling; the window is
/// then re-read once with the ceiling as budget (#691.3, <see cref="GatewayReadLimits"/>), and the same
/// "delivered the whole budget ⇒ cannot be proven complete" rule applies.
/// </para>
/// <para>
/// <b>Values and status.</b> A sample's value is its numeric value (Double), else its string value, else Empty; its
@@ -163,6 +166,7 @@ internal static class RawAggregateReader
/// <summary>Reads and evaluates one per-sample processed read.</summary>
/// <param name="client">The gateway client.</param>
/// <param name="ceiling">The data source's learned gateway row ceiling (#691.3).</param>
/// <param name="tag">Historian tag name.</param>
/// <param name="aggregate">A per-sample aggregate (<see cref="Serves"/>).</param>
/// <param name="plan">The bucket plan.</param>
@@ -172,14 +176,51 @@ internal static class RawAggregateReader
/// <returns>One value per interval, in the plan's order.</returns>
/// <exception cref="HistoryReadLimitExceededException">The window holds more than <paramref name="maxWindowRows"/> rows,
/// or the stream could not be read to completion within the requested row budget.</exception>
public static async Task<IReadOnlyList<DataValueSnapshot>> ReadAsync(IHistorianGatewayClient client, string tag,
public static async Task<IReadOnlyList<DataValueSnapshot>> ReadAsync(IHistorianGatewayClient client, GatewayRowCeiling ceiling, string tag,
HistoryAggregateType aggregate, BucketPlan plan, int maxWindowRows, DateTime now, CancellationToken ct)
{
if (plan.BucketCount == 0) return [];
var needsRange = AggregateEvaluator.NeedsDataRange(aggregate, plan);
var requestStart = RequestStartFor(plan.StartUtc);
var maxValues = MaxValuesFor(maxWindowRows);
// A strict gateway that refuses the budget is re-asked once at the ceiling it names (#691.3; e.g. a ceiling below
// the 1,000,000 #688 clamps at, or an unbounded cap) — and a stream that fills that ceiling is never evaluated.
var requested = MaxValuesFor(maxWindowRows);
var (pass, maxValues) = await ceiling.ReadAsync(requested,
budget => ReadWindowAsync(client, tag, aggregate, plan, requestStart, budget, maxWindowRows, ct),
(r, budget) => !r.OverCap && r.Received >= budget).ConfigureAwait(false);
var (buckets, earliest, latest, received, overCap) = pass;
if (overCap) throw LimitExceeded(tag, maxWindowRows);
if (received >= maxValues) throw BudgetExhausted(tag, maxWindowRows, requested, maxValues);
if (needsRange)
{
// Earlier data: no non-Bad pre-window row ⇒ ask explicitly (normally only when the tag's history really starts
// inside the window).
var earlierSeen = earliest is { } e && (e < plan.StartUtc || (plan.Reversed && e == plan.StartUtc));
if (!earlierSeen && plan.StartUtc.AddTicks(-1) > ProbeFloor)
if (await FirstSampleAsync(client, tag, ProbeFloor, plan.StartUtc.AddTicks(-1), ct).ConfigureAwait(false) is { } before)
earliest = before;
// Later data: the main read covers a sample exactly at the window end (forward: that is already later data);
// otherwise a small separate read from the end to now + horizon.
var laterSeen = latest is { } l && (plan.Reversed ? l > plan.EndUtc : l >= plan.EndUtc);
var horizon = DateTime.MaxValue - now > LookAheadHorizon
? DateTime.SpecifyKind(now + LookAheadHorizon, DateTimeKind.Utc)
: DateTime.MaxValue;
if (!laterSeen && horizon > plan.EndUtc)
if (await FirstSampleAsync(client, tag, plan.EndUtc, horizon, ct).ConfigureAwait(false) is { } after)
latest = after;
}
return AggregateEvaluator.EvaluateAll(aggregate, plan, buckets.Rows(), earliest, latest, now);
}
/// <summary>The window read of <see cref="ReadAsync"/> with one row budget, drained.</summary>
private static async Task<(RawBucketAccumulator Buckets, DateTime? Earliest, DateTime? Latest, long Received, bool OverCap)>
ReadWindowAsync(IHistorianGatewayClient client, string tag, HistoryAggregateType aggregate, BucketPlan plan,
DateTime requestStart, int maxValues, int maxWindowRows, CancellationToken ct)
{
var buckets = new RawBucketAccumulator(aggregate == HistoryAggregateType.MaximumActualTime);
DateTime? earliest = null, latest = null;
long windowRows = 0, received = 0;
@@ -214,30 +255,7 @@ internal static class RawAggregateReader
if (t >= plan.StartUtc && (latest is null || t > latest)) latest = t;
}
}
if (overCap || (maxWindowRows > 0 && received >= maxValues))
throw LimitExceeded(tag, maxWindowRows);
if (needsRange)
{
// Earlier data: no non-Bad pre-window row ⇒ ask explicitly (normally only when the tag's history really starts
// inside the window).
var earlierSeen = earliest is { } e && (e < plan.StartUtc || (plan.Reversed && e == plan.StartUtc));
if (!earlierSeen && plan.StartUtc.AddTicks(-1) > ProbeFloor)
if (await FirstSampleAsync(client, tag, ProbeFloor, plan.StartUtc.AddTicks(-1), ct).ConfigureAwait(false) is { } before)
earliest = before;
// Later data: the main read covers a sample exactly at the window end (forward: that is already later data);
// otherwise a small separate read from the end to now + horizon.
var laterSeen = latest is { } l && (plan.Reversed ? l > plan.EndUtc : l >= plan.EndUtc);
var horizon = DateTime.MaxValue - now > LookAheadHorizon
? DateTime.SpecifyKind(now + LookAheadHorizon, DateTimeKind.Utc)
: DateTime.MaxValue;
if (!laterSeen && horizon > plan.EndUtc)
if (await FirstSampleAsync(client, tag, plan.EndUtc, horizon, ct).ConfigureAwait(false) is { } after)
latest = after;
}
return AggregateEvaluator.EvaluateAll(aggregate, plan, buckets.Rows(), earliest, latest, now);
return (buckets, earliest, latest, received, overCap);
}
/// <summary>
@@ -266,6 +284,26 @@ internal static class RawAggregateReader
/// </summary>
private static bool IsNotBadOnTheWire(HistorianSample raw) => (byte)raw.OpcQuality >= 64; // low byte, as SampleMapper.ToStatus reads it
/// <summary>
/// The stream delivered its whole row budget, so the window cannot be proven complete. Names the limit that actually
/// bound the budget: the gateway's ceiling (a strict gateway refused the requested budget and named it, #691.3), the
/// #688 clamp at the gateway's default ceiling, or the configured cap.
/// </summary>
private static HistoryReadLimitExceededException BudgetExhausted(string tag, int maxWindowRows, int requested, int budget)
{
if (budget < requested)
return new HistoryReadLimitExceededException(budget,
$"The processed read of '{tag}' covers more raw samples than the HistorianGateway's row ceiling ({budget}, its " +
"Historian:ReadLimits:MaxValuesCeiling with AllowStreamingBeyondCeiling=false); the Gateway backend evaluates " +
"per-sample aggregates from raw reads — narrow the window.");
if (maxWindowRows > 0 && budget < (long)maxWindowRows + 2 + PreWindowAllowance)
return new HistoryReadLimitExceededException(budget,
$"The processed read of '{tag}' covers more raw samples than the HistorianGateway's default row ceiling " +
$"({budget}) the request is clamped at; the Gateway backend evaluates per-sample aggregates from raw reads — " +
"narrow the window.");
return LimitExceeded(tag, maxWindowRows);
}
private static HistoryReadLimitExceededException LimitExceeded(string tag, int maxWindowRows) =>
new(maxWindowRows,
$"The processed read of '{tag}' covers more than {maxWindowRows} raw samples " +
@@ -317,6 +317,9 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
// section is absent; NodeManager is non-null here (guarded above).
_server.NodeManager.MaxTieClusterOverfetch = _serverHistorianOptions.MaxTieClusterOverfetch;
// Gitea #691: the ReturnBounds kill switch (default on).
_server.NodeManager.HonorReturnBounds = HonorReturnBounds;
// Propagate the S3 HistoryRead concurrency bounds (arch-review 03/S3): bounded per-node fan-out
// within a batch, the process-wide concurrent-batch limiter, and the per-request deadline. Defaults
// (4 / 16 / 60 s) survive when the ServerHistorian section is absent.
@@ -347,6 +350,12 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
_logger.LogInformation("OtOpcUaServerHostedService: SDK started, address-space + ServiceLevel sinks bound");
}
/// <summary>
/// Gitea #691 — the <c>ServerHistorian:HonorReturnBounds</c> kill switch <c>StartAsync</c> applies to the node manager
/// (default <c>true</c> when the section or key is absent). Exposed so the wiring is testable without booting the SDK.
/// </summary>
internal bool HonorReturnBounds => _serverHistorianOptions.HonorReturnBounds;
/// <summary>
/// Gitea #674 — what <c>StartAsync</c> wires onto the node manager for HistoryUpdate: the Primary gate (read from
/// the <see cref="IHistoryUpdateRoleView"/> on every call; null when none is registered, i.e. allow), the
@@ -0,0 +1,219 @@
using Opc.Ua;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
/// <summary>
/// The end bound still owed to a HistoryRead-Raw read with <c>ReturnBounds = true</c> (Gitea #691). It rides the
/// continuation point, because Part 11 returns the end bound with the LAST page only.
/// </summary>
/// <param name="AtUtc">The request's end time (the later bound of a forward read, the earlier of a reverse one).</param>
/// <param name="Reverse">The read is newest-first (<c>StartTime &gt; EndTime</c>, including an end-only read).</param>
/// <param name="Inclusive">
/// <c>false</c> only for <c>StartTime == EndTime</c>: the samples AT that instant are the data, so the end bound is the
/// first sample strictly after it (Part 11 Table 1 footnote c). Otherwise the end bound is the sample at or beyond the
/// end time, and the data window excludes the end time.
/// </param>
/// <param name="Open">The end time was not given (a one-sided read's sentinel): there is no end bound to look up.</param>
/// <param name="DataExhausted">The data is drained and the end bound did not fit on the previous page.</param>
/// <param name="LastReturnedUtc">The last value's time returned on an earlier page (stamps an open bound's placeholder).</param>
internal sealed record HistoryEndBound(
DateTime AtUtc, bool Reverse, bool Inclusive, bool Open, bool DataExhausted = false, DateTime? LastReturnedUtc = null);
/// <summary>
/// Gitea #691: OPC UA Part 11 bounding values for HistoryRead-Raw (<c>ReadRawModifiedDetails.ReturnBounds</c>), composed
/// in the server from every backend's existing raw read — the ONE authority, so SqlServer, QuestDB, Gateway and Null
/// answer identically.
/// </summary>
/// <remarks>
/// <para>
/// <b>What a bound is</b> (Part 11 §4.6, Table 1). Forward: the start bound is the stored sample AT or immediately
/// BEFORE StartTime, the end bound the one AT or immediately AFTER EndTime. Reverse mirrors it: the start bound is at
/// or after StartTime, the end bound at or before EndTime. A sample exactly on a bound IS the bound: on the start it
/// is already the first sample of the data (the window includes StartTime), so nothing extra is added; on the end it is
/// added (the window excludes EndTime). <c>StartTime == EndTime</c> reads forward, and its end bound is the first
/// sample strictly after the instant. A bound is a stored sample of any quality — a Bad sample is returned as stored.
/// </para>
/// <para>
/// <b>Missing bound.</b> <c>Bad_BoundNotFound</c>, a null value, stamped with the bound's time. A one-sided read's
/// missing side (no EndTime / no StartTime) is never looked up; it is <c>Bad_BoundNotFound</c> stamped one second after
/// (forward) or before (reverse) the previous returned value (Table 1 footnotes a and b).
/// </para>
/// <para>
/// <b>Lookups.</b> "At or before T" is a newest-first read of <c>(0, T]</c> capped at one — native on a backend that
/// declares <see cref="IHistorianDataSource.SupportsReverseRaw"/>, otherwise <see cref="ReverseRawReader"/>'s walk-back
/// from bounded forward reads (the Gateway: NOT AVEVA's synthetic boundary row, which is stamped at the query start
/// rather than at the prior sample's own time). "At or after T" is a forward read of <c>[T, MaxValue)</c> capped at one (on a
/// forward-only backend ending at now + <see cref="ReverseRawReader.LookAhead"/> instead — #691 review);
/// "strictly after T" starts one storage resolution later (lossless: nothing is stored in between). Inside a tie
/// cluster the bound is the cluster's first sample in the read's direction — the forward read's first, the reverse
/// read's first (the forward order's last).
/// </para>
/// <para>
/// <b>Resolution.</b> "On the bound" compares at the backend's storage resolution, because a microsecond backend
/// (QuestDB) truncates the window bounds it is sent: a sample at <c>⌊T⌋</c> is inside a window starting at <c>T</c>
/// there, so it must not also be returned as the start bound.
/// </para>
/// </remarks>
internal static class HistoryBounds
{
private static readonly TimeSpan OpenBoundStep = TimeSpan.FromSeconds(1);
/// <summary>The end bound owed by a fresh read over the normalised window.</summary>
/// <param name="startUtc">The (normalised) start: the later time of a reverse read.</param>
/// <param name="endUtc">The (normalised) end; a one-sided read's sentinel (tick 0 / MaxValue) is an open bound.</param>
/// <returns>The pending end bound.</returns>
public static HistoryEndBound For(DateTime startUtc, DateTime endUtc) =>
new(endUtc, Reverse: startUtc > endUtc, Inclusive: startUtc != endUtc, Open: IsOpen(endUtc));
/// <summary>Whether a returned value is a <see cref="NotFound"/> placeholder rather than a stored sample.</summary>
/// <param name="value">The value.</param>
/// <returns><c>true</c> for a <c>Bad_BoundNotFound</c> placeholder.</returns>
public static bool IsPlaceholder(DataValueSnapshot value) =>
value.StatusCode == StatusCodes.BadBoundNotFound && value.Value is null;
/// <summary>The pending end bound to store behind a continuation point, remembering the page's last returned time.</summary>
/// <param name="bound">The pending end bound (null = bounds not requested).</param>
/// <param name="page">The page being returned.</param>
/// <returns>The bound to carry, or null.</returns>
public static HistoryEndBound? Carry(HistoryEndBound? bound, IReadOnlyList<DataValueSnapshot> page) =>
bound is null || page.Count == 0 || page[^1].SourceTimestampUtc is not { } last
? bound
: bound with { LastReturnedUtc = last };
/// <summary>The <c>Bad_BoundNotFound</c> placeholder (null value) stamped at <paramref name="atUtc"/>.</summary>
/// <param name="atUtc">The bound's time.</param>
/// <returns>The placeholder.</returns>
public static DataValueSnapshot NotFound(DateTime atUtc) =>
new(null, StatusCodes.BadBoundNotFound, atUtc, atUtc);
/// <summary>
/// The start bound to return ahead of the data, or <c>null</c> when a stored sample sits on StartTime (that sample is
/// the bound, and it is already the data's first sample).
/// </summary>
public static async Task<DataValueSnapshot?> StartBoundAsync(
IHistorianDataSource source, string tagname, DateTime startUtc, bool reverse, int maxTieCluster, DateTime nowUtc,
CancellationToken ct)
{
if (IsOpen(startUtc)) return NotFound(startUtc);
var found = reverse
? await EarliestAtOrAfterAsync(source, tagname, startUtc, nowUtc, ct).ConfigureAwait(false)
: await LatestAtOrBeforeAsync(source, tagname, startUtc, maxTieCluster, nowUtc, ct).ConfigureAwait(false);
if (found is null) return NotFound(startUtc);
return SameInstant(Ticks(found), startUtc.Ticks, Resolution(source)) ? null : found;
}
/// <summary>The end bound: the stored sample on or beyond the end time, else the placeholder.</summary>
/// <param name="source">The backend.</param>
/// <param name="tagname">Historian tag.</param>
/// <param name="bound">The pending end bound.</param>
/// <param name="previousReturnedUtc">The last value's time already returned (stamps an open bound's placeholder).</param>
/// <param name="maxTieCluster">The walk-back's row budget (forward-only backends).</param>
/// <param name="nowUtc">Now (the walk-back's look-ahead clamp).</param>
/// <param name="ct">Cancellation.</param>
public static async Task<DataValueSnapshot> EndBoundAsync(
IHistorianDataSource source, string tagname, HistoryEndBound bound, DateTime previousReturnedUtc,
int maxTieCluster, DateTime nowUtc, CancellationToken ct)
{
if (bound.Open)
{
var ticks = bound.Reverse
? Math.Max(0L, previousReturnedUtc.Ticks - OpenBoundStep.Ticks)
: Math.Min(DateTime.MaxValue.Ticks, previousReturnedUtc.Ticks + OpenBoundStep.Ticks);
return NotFound(new DateTime(ticks, DateTimeKind.Utc));
}
DataValueSnapshot? found;
if (bound.Reverse)
found = await LatestAtOrBeforeAsync(source, tagname, bound.AtUtc, maxTieCluster, nowUtc, ct).ConfigureAwait(false);
else if (bound.Inclusive)
found = await EarliestAtOrAfterAsync(source, tagname, bound.AtUtc, nowUtc, ct).ConfigureAwait(false);
else
{
var resolution = Resolution(source);
found = bound.AtUtc.Ticks > DateTime.MaxValue.Ticks - resolution
? null
: await EarliestAtOrAfterAsync(source, tagname, bound.AtUtc.AddTicks(resolution), nowUtc, ct).ConfigureAwait(false);
}
return found ?? NotFound(bound.AtUtc);
}
/// <summary>
/// Whether a data sample lies inside the data window of a bounded read — a forward window excludes the end time (a
/// sample there is the end bound), compared at the storage resolution. A backend whose forward end is inclusive, or a
/// resumed read that steps onto the end time, would otherwise return the end bound twice.
/// </summary>
public static bool InDataWindow(DataValueSnapshot sample, HistoryEndBound bound, long resolution)
{
if (!bound.Inclusive || bound.Open) return true;
var ts = Align(Ticks(sample), resolution);
var end = Align(bound.AtUtc.Ticks, resolution);
return bound.Reverse ? ts > end : ts < end;
}
/// <summary>Whether a resumed read's next start is already past the data window (only the end bound remains).</summary>
public static bool DataWindowDrained(DateTime nextStartUtc, HistoryEndBound bound, long resolution)
{
if (!bound.Inclusive || bound.Open) return false;
var next = Align(nextStartUtc.Ticks, resolution);
var end = Align(bound.AtUtc.Ticks, resolution);
return bound.Reverse ? next <= end : next >= end;
}
/// <summary>The latest stored sample at or before <paramref name="atUtc"/>, or null.</summary>
internal static async Task<DataValueSnapshot?> LatestAtOrBeforeAsync(
IHistorianDataSource source, string tagname, DateTime atUtc, int maxTieCluster, DateTime nowUtc, CancellationToken ct)
{
if (atUtc.Ticks == 0) return null;
var floor = new DateTime(0, DateTimeKind.Utc);
IReadOnlyList<DataValueSnapshot> rows = source.SupportsReverseRaw
? (await source.ReadRawAsync(tagname, atUtc, floor, 1, ct).ConfigureAwait(false)).Samples
: (await ReverseRawReader.ReadAsync(source, tagname, atUtc, floor, 1, maxTieCluster, nowUtc, 0, ct)
.ConfigureAwait(false)).Samples;
var limit = Align(atUtc.Ticks, Resolution(source));
foreach (var row in rows)
if (Align(Ticks(row), Resolution(source)) <= limit)
return row;
return null;
}
/// <summary>The earliest stored sample at or after <paramref name="atUtc"/>, or null.</summary>
/// <remarks>
/// #691 review: on a backend that reads forward only (the Gateway) the read's end is clamped to now +
/// <see cref="ReverseRawReader.LookAhead"/> — the same horizon the walk-back uses — and a time beyond it is "not found"
/// without a read. The HistorianGateway only checks <c>start &lt;= end</c>, so a <c>DateTime.MaxValue</c> end would
/// reach AVEVA as a year-9999 query end (unverified: a rejection, or a fabricated row). The clamp is NOT applied to a
/// backend that reads newest-first natively (SqlServer / QuestDB): they bind <c>MaxValue</c> safely (pinned by the
/// <c>Extreme_bounds</c> conformance test) and can legitimately hold samples further ahead (HistoryUpdate's
/// <c>HistoryUpdateMaxFutureSkew</c> is configurable, and driver-stamped live values carry device time) — clamping
/// there would turn a real end bound into <c>Bad_BoundNotFound</c>.
/// </remarks>
internal static async Task<DataValueSnapshot?> EarliestAtOrAfterAsync(
IHistorianDataSource source, string tagname, DateTime atUtc, DateTime nowUtc, CancellationToken ct)
{
var endTicks = DateTime.MaxValue.Ticks;
if (!source.SupportsReverseRaw)
endTicks = nowUtc.Ticks > DateTime.MaxValue.Ticks - ReverseRawReader.LookAhead.Ticks
? DateTime.MaxValue.Ticks
: nowUtc.Ticks + ReverseRawReader.LookAhead.Ticks;
if (atUtc.Ticks >= endTicks) return null;
var rows = (await source.ReadRawAsync(tagname, atUtc, new DateTime(endTicks, DateTimeKind.Utc), 1, ct)
.ConfigureAwait(false)).Samples;
var limit = Align(atUtc.Ticks, Resolution(source));
foreach (var row in rows)
if (Align(Ticks(row), Resolution(source)) >= limit)
return row;
return null;
}
private static bool IsOpen(DateTime t) => t.Ticks == 0 || t.Ticks == DateTime.MaxValue.Ticks;
private static bool SameInstant(long a, long b, long resolution) => Align(a, resolution) == Align(b, resolution);
private static long Resolution(IHistorianDataSource source) => Math.Max(1L, source.TimestampResolutionTicks);
private static long Ticks(DataValueSnapshot s) => (s.SourceTimestampUtc ?? DateTime.MinValue).Ticks;
private static long Align(long ticks, long resolution) => ticks - ticks % resolution;
}
@@ -55,6 +55,17 @@ namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
/// <see cref="HistoryCursorKind.Raw"/> reverse reads on a forward-only backend only (Gitea #686): the segment width the
/// walk-back ended on, so the next page starts at the observed density instead of re-discovering it. 0 = none.
/// </param>
/// <param name="EmptyFromTicks">
/// Walked reverse reads only (Gitea #691): with <paramref name="EmptyToTicks"/>, a stretch <c>(from, to)</c> the previous
/// page's walk proved holds no row, so the next page steps over it rather than re-walking it (to the 1970 floor, ~14
/// reads, at the end of a deep page-through). 0/0 = none. The cursor lives in the session's in-memory store (never
/// serialized), so a new field with a "none" default is all the versioning it needs.
/// </param>
/// <param name="EmptyToTicks">The exclusive upper end of that stretch.</param>
/// <param name="PendingEndBound">
/// <see cref="HistoryCursorKind.Raw"/> with <c>ReturnBounds = true</c> only (Gitea #691): the end bound still owed —
/// Part 11 returns it with the last page. <c>null</c> = bounds not requested.
/// </param>
internal sealed record HistoryContinuationState(
string Tagname,
DateTime NextStartUtc,
@@ -65,7 +76,14 @@ internal sealed record HistoryContinuationState(
long LastModifiedId = 0,
DateTime WindowStartUtc = default,
bool Reverse = false,
long SegmentWidthTicks = 0);
long SegmentWidthTicks = 0,
long EmptyFromTicks = 0,
long EmptyToTicks = 0,
HistoryEndBound? PendingEndBound = null)
{
/// <summary>The walk-back hint this cursor carries (#686/#689/#691).</summary>
public ReverseWalkHint WalkHint => new(SegmentWidthTicks, EmptyFromTicks, EmptyToTicks);
}
/// <summary>The HistoryRead kind that issued a <see cref="HistoryContinuationState"/>.</summary>
internal enum HistoryCursorKind
@@ -537,13 +537,20 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
/// paging over-fetches up to <c>MaxTieClusterOverfetch + 1</c> ties at that single timestamp (a
/// <c>start == end</c> read) and slices through them via <see cref="HistoryPaging.SliceTieCluster"/>.
/// A cluster strictly larger than this still fails that node <c>BadTooManyOperations</c> (the absurd-burst
/// backstop; <c>BadHistoryOperationUnsupported</c> before #686). Also the per-read row budget of a reverse raw read
/// backstop; <c>BadHistoryOperationUnsupported</c> before #686). Also the tie-cluster ceiling (not the segment size, #689) of a reverse read
/// walked back from forward reads (<see cref="ReverseRawReader"/>). Mirrors the configured
/// <c>ServerHistorianOptions.MaxTieClusterOverfetch</c>; the Host sets it at <c>StartAsync</c>. The
/// default (65536) survives when the historian section is absent.
/// </summary>
public int MaxTieClusterOverfetch { get; set; } = 65536;
/// <summary>
/// Gitea #691: whether HistoryRead-Raw honours <c>ReturnBounds</c> (Part 11 bounding values, <see cref="HistoryBounds"/>)
/// — <c>ServerHistorian:HonorReturnBounds</c>, default <c>true</c>. <c>false</c> ignores the flag, as before #691,
/// including the modified-read <c>Bad_InvalidArgument</c>.
/// </summary>
public bool HonorReturnBounds { get; set; } = true;
/// <summary>
/// Maximum historized nodes served concurrently within one HistoryRead batch (arch-review 03/S3).
/// Mirrors <c>ServerHistorianOptions.HistoryReadBatchParallelism</c>; the Host sets it at
@@ -5056,6 +5063,15 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
var work = new List<Func<Task>>(nodesToProcess.Count);
foreach (var handle in nodesToProcess)
{
if (details.IsReadModified && details.ReturnBounds && HonorReturnBounds)
{
// Part 11 §6.5.3.3: returnBounds "shall be set to FALSE" for a modified read, "otherwise the Server returns
// a Bad_InvalidArgument" (#691). Off with the ServerHistorian:HonorReturnBounds kill switch.
errors[handle.Index] = StatusCodes.BadInvalidArgument;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadInvalidArgument };
continue;
}
if (details.IsReadModified)
{
// Part 11 modified-value history (#668): the prior values every HistoryUpdate recorded. Gated by the
@@ -5070,7 +5086,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
work.Add(() => ServeRawPagedAsync(
handle, session, nodesToRead, results, errors,
details.StartTime, details.EndTime, details.NumValuesPerNode,
ct));
ct, details.ReturnBounds && HonorReturnBounds));
}
RunBounded(work, HistoryReadBatchParallelism);
@@ -5248,6 +5264,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
var endUtc = details.EndTime;
var numValuesPerNode = details.NumValuesPerNode;
var boundarySkip = 0;
var walkHint = default(ReverseWalkHint); // #691.4: a walked-back read's width + proven-empty stretch
bool reverse;
if (resumed)
@@ -5271,35 +5288,30 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
// The direction is the CURSOR's (Gitea #675), never re-derived from the resume request's details: a
// reverse read resumes reverse, a forward one forward.
reverse = state.Reverse;
walkHint = state.WalkHint;
}
else
{
// Part 11: with one bound unset, an unset EndTime (MinValue) with a StartTime means "page FORWARD from
// StartTime" — normalise it to an open upper bound, or the cursor would store MinValue and the read would
// look reversed. An unset StartTime with an EndTime and a count cap means "page BACKWARD from EndTime"
// (#675, mirroring raw): the reverse window (MinValue, EndTime] — but only on a backend that can read
// newest-first. Elsewhere (Gateway, the default backend) and without a count cap an end-only read keeps its
// historical forward [MinValue, EndTime) meaning rather than becoming a refused reverse read.
// (#675, mirroring raw): the reverse window (MinValue, EndTime] — on every backend since #691.4 (a backend
// that cannot read newest-first is walked back below). Without a count cap an end-only read keeps its
// forward [MinValue, EndTime) meaning.
if (endUtc == DateTime.MinValue && startUtc != DateTime.MinValue)
endUtc = new DateTime(DateTime.MaxValue.Ticks, DateTimeKind.Utc); // Utc kind: a proto Timestamp conversion rejects Unspecified
else if (numValuesPerNode > 0 && startUtc == DateTime.MinValue && endUtc != DateTime.MinValue
&& source.SupportsReverseEvents)
else if (numValuesPerNode > 0 && startUtc == DateTime.MinValue && endUtc != DateTime.MinValue)
(startUtc, endUtc) = (endUtc, new DateTime(0, DateTimeKind.Utc));
// Part 11: StartTime later than EndTime is a reverse read — events newest-first over (end, start].
reverse = startUtc > endUtc;
}
// An explicit reverse read (or a reverse cursor resumed after the source was swapped) on a backend that cannot
// read newest-first: refuse it BEFORE calling the source — never the earliest events in ascending order as if
// they were the answer. Checked here rather than by catching an exception, so a NotSupportedException thrown
// from inside a backend (a driver library, the projection) stays a backend fault.
if (reverse && !source.SupportsReverseEvents)
{
errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadHistoryOperationUnsupported };
return;
}
// A reverse read on a backend that cannot read newest-first (the Gateway) is never sent there (#675 refused it
// here); since #691.4 it is walked back from bounded forward reads — the raw walk (#686), with ties ordered by
// EventId because the gateway orders by event time only. Decided by the capability, not by catching an
// exception, so a NotSupportedException thrown from inside a backend stays a backend fault.
var walked = reverse && !source.SupportsReverseEvents;
// NOT under the node-manager Lock — awaiting the async source is safe here.
// NumValuesPerNode==0 means "no limit — return ALL values" per OPC UA Part 4/11, so it is translated
@@ -5309,8 +5321,26 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
var fetch = EventMaxEvents(numValuesPerNode);
if (numValuesPerNode > 0 && boundarySkip > 0)
fetch = (int)Math.Min(int.MaxValue - 1L, (long)fetch + boundarySkip);
var sourceResult = await source.ReadEventsAsync(
sourceName, startUtc, endUtc, fetch, reverse, ct).ConfigureAwait(false);
HistoricalEventsResult sourceResult;
if (walked)
{
// Walk for one more than the fetch: finding it is the "more events exist" signal a source gives with its
// continuation point (the walk itself cannot flag truncation).
var want = numValuesPerNode == 0 ? 0u : (uint)Math.Min(uint.MaxValue - 1L, (long)fetch + 1);
var (rows, next) = await ReverseEventsReader.ReadAsync(
source, sourceName, startUtc, endUtc, want, MaxTieClusterOverfetch, Clock.GetUtcNow().UtcDateTime,
walkHint, ct)
.ConfigureAwait(false);
walkHint = next;
var more = want > 0 && rows.Count > fetch;
if (more) rows.RemoveRange(fetch, rows.Count - fetch);
sourceResult = new HistoricalEventsResult(rows, more ? [0x01] : null);
}
else
{
sourceResult = await source.ReadEventsAsync(
sourceName, startUtc, endUtc, fetch, reverse, ct).ConfigureAwait(false);
}
// Truncation is the SOURCE's word (non-null continuation point) — never inferred from the count, so a
// page that happens to be exactly cap-sized but complete never emits a point.
@@ -5350,6 +5380,12 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
sourceName, startUtc, startUtc.AddTicks(resolution), overfetchCap, reverse: false, ct)
.ConfigureAwait(false)).Events;
// A walked read (#691.4) emitted its ties sorted by EventId; the gateway returns them in any order and its
// end is inclusive — keep exactly the cluster, in the walk's order, so the skip indexes the same events.
if (walked)
cluster = cluster.Where(e => e.EventTimeUtc == startUtc)
.OrderBy(e => e, EventIdTieOrder.Instance).ToList();
// The over-fetch reads the cluster FORWARD (ascending EventId), but a reverse read emitted its ties — and
// counted boundarySkip — in descending order. Slice in the read's own direction, as raw does.
if (reverse) cluster = cluster.Reverse().ToList();
@@ -5378,7 +5414,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
{
pendingCursor = new HistoryContinuationState(
sourceName ?? string.Empty, resumeAt, endUtc, nextSkip, numValuesPerNode, HistoryCursorKind.Events,
Reverse: reverse);
Reverse: reverse, SegmentWidthTicks: walkHint.WidthTicks, EmptyFromTicks: walkHint.EmptyFromTicks,
EmptyToTicks: walkHint.EmptyToTicks);
}
}
else if (truncated && events.Count > 0)
@@ -5390,7 +5427,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
for (var i = events.Count - 1; i >= 0 && events[i].EventTimeUtc == boundary; i--) skip++;
pendingCursor = new HistoryContinuationState(
sourceName ?? string.Empty, boundary, endUtc, skip, numValuesPerNode, HistoryCursorKind.Events,
Reverse: reverse);
Reverse: reverse, SegmentWidthTicks: walkHint.WidthTicks, EmptyFromTicks: walkHint.EmptyFromTicks,
EmptyToTicks: walkHint.EmptyToTicks);
}
var historyEvent = ProjectEvents(events, selectClauses);
@@ -5411,6 +5449,14 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
errors[handle.Index] = StatusCodes.BadTimeout;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadTimeout };
}
catch (HistoryReadLimitExceededException ex)
{
// A request limit (#691.4: a walked-back reverse read met a tie cluster over the budget) — BadTooManyOperations
// for THIS node, as on the raw path; not a backend fault.
_logger.LogWarning("OtOpcUaNodeManager: HistoryReadEvents limit exceeded for node {NodeId}: {Message}", handle.NodeId, ex.Message);
errors[handle.Index] = StatusCodes.BadTooManyOperations;
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadTooManyOperations };
}
catch (Exception ex)
{
// One node's backend failure must not throw out of the batch — surface Bad for THIS node
@@ -6167,6 +6213,11 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
/// <param name="startTimeUtc">The request window's (inclusive) lower bound, used for a fresh read.</param>
/// <param name="endUtc">The (inclusive) upper bound of the read window; unchanged across pages.</param>
/// <param name="numValuesPerNode">The client's per-page cap; <c>0</c> means "all values, no paging".</param>
/// <param name="ct">Cancellation (the per-request deadline).</param>
/// <param name="returnBounds">
/// Part 11 bounding values (#691, <see cref="HistoryBounds"/>): the start bound leads the first page, the end bound
/// ends the last, and both count toward <paramref name="numValuesPerNode"/>. A resumed read takes it from the cursor.
/// </param>
private async Task ServeRawPagedAsync(
NodeHandle handle,
ISession? session,
@@ -6176,7 +6227,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
DateTime startTimeUtc,
DateTime endUtc,
uint numValuesPerNode,
CancellationToken ct)
CancellationToken ct,
bool returnBounds = false)
{
var inboundCp = nodesToRead[handle.Index].ContinuationPoint;
@@ -6184,7 +6236,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
{
DateTime startUtc;
var boundarySkip = 0;
long segmentWidth = 0; // #686: the walk-back's width, carried page to page
var walkHint = default(ReverseWalkHint); // #686/#689/#691: the walk-back's width + proven-empty stretch, page to page
HistoryEndBound? endBound = null; // #691: the end bound still owed (null = bounds not requested)
string tagname;
if (inboundCp is { Length: > 0 })
@@ -6205,7 +6258,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
boundarySkip = state.BoundarySkipCount;
endUtc = state.EndUtc;
numValuesPerNode = state.NumValuesPerNode;
segmentWidth = state.SegmentWidthTicks;
walkHint = state.WalkHint;
endBound = state.PendingEndBound;
}
else
{
@@ -6231,6 +6285,107 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
else if (startUtc == DateTime.MinValue && endUtc != DateTime.MinValue)
(startUtc, endUtc) = (endUtc, new DateTime(0, DateTimeKind.Utc));
}
if (returnBounds) endBound = HistoryBounds.For(startUtc, endUtc);
}
var nowUtc = Clock.GetUtcNow().UtcDateTime;
var resolution = Math.Max(1L, HistorianDataSource.TimestampResolutionTicks);
// #691: the data is drained — the end bound ends this page when it has room (Part 11: bounds count toward
// NumValuesPerNode), else it is the whole next page. Returns the continuation point to emit, if any.
async Task<byte[]?> AppendEndBoundAsync(List<DataValueSnapshot> page, DateTime resumedAtUtc)
{
// The previous returned value stamps an open bound's placeholder (Part 11 Table 1 footnotes a/b): this
// page's last, else the last of an earlier page (carried in the cursor), else where the read resumed.
var previous = (page.Count > 0 ? page[^1].SourceTimestampUtc : null)
?? endBound!.LastReturnedUtc ?? resumedAtUtc;
if (numValuesPerNode == 0 || page.Count < numValuesPerNode)
{
page.Add(await HistoryBounds.EndBoundAsync(
HistorianDataSource, tagname, endBound!, previous, MaxTieClusterOverfetch, nowUtc, ct).ConfigureAwait(false));
return null;
}
return _historyContinuationStore.Save(session, new HistoryContinuationState(
tagname, previous, endUtc, 0, numValuesPerNode, SegmentWidthTicks: walkHint.WidthTicks,
EmptyFromTicks: walkHint.EmptyFromTicks, EmptyToTicks: walkHint.EmptyToTicks,
PendingEndBound: HistoryBounds.Carry(endBound!, page)! with { DataExhausted = true }));
}
// #691 review: whether a bounded read whose start bound was NOT found has any data or a found end bound — only
// asked when a one-value page would otherwise be the start-bound placeholder alone.
async Task<bool> BoundedReadHasAnythingAsync()
{
var src = HistorianDataSource;
IReadOnlyList<DataValueSnapshot> first = startUtc > endUtc && !src.SupportsReverseRaw
? (await ReverseRawReader.ReadAsync(src, tagname, startUtc, endUtc, 1, MaxTieClusterOverfetch, nowUtc, 0, ct)
.ConfigureAwait(false)).Samples
: (await src.ReadRawAsync(tagname, startUtc, endUtc, 1, ct).ConfigureAwait(false)).Samples;
if (first.Any(x => HistoryBounds.InDataWindow(x, endBound!, resolution))) return true;
var end = await HistoryBounds.EndBoundAsync(
src, tagname, endBound!, startUtc, MaxTieClusterOverfetch, nowUtc, ct).ConfigureAwait(false);
return !HistoryBounds.IsPlaceholder(end);
}
// #691: a resumed bounded read whose data is drained (the end bound did not fit on the previous page, or the
// resume point already sits on the end time) owes only the end bound.
if (endBound is not null && inboundCp is { Length: > 0 }
&& (endBound.DataExhausted || HistoryBounds.DataWindowDrained(startUtc, endBound, resolution)))
{
var onlyBound = new List<DataValueSnapshot>(1);
var afterCp = await AppendEndBoundAsync(onlyBound, startUtc).ConfigureAwait(false);
results[handle.Index] = new SdkHistoryReadResult
{
StatusCode = StatusCodes.Good,
HistoryData = new ExtensionObject(ToHistoryDataFromSamples(onlyBound)),
ContinuationPoint = afterCp,
};
errors[handle.Index] = ServiceResult.Good;
return;
}
// #691: the start bound leads the first page and counts toward the cap; null when a sample sits on StartTime
// (that sample is the bound, and it is the data's first).
DataValueSnapshot? startBound = null;
var readCap = numValuesPerNode;
if (endBound is not null && inboundCp is not { Length: > 0 })
{
startBound = await HistoryBounds.StartBoundAsync(
HistorianDataSource, tagname, startUtc, endBound.Reverse, MaxTieClusterOverfetch, nowUtc, ct)
.ConfigureAwait(false);
if (startBound is not null && numValuesPerNode == 1
&& (!HistoryBounds.IsPlaceholder(startBound) || await BoundedReadHasAnythingAsync().ConfigureAwait(false)))
{
// The page is the start bound alone; the data starts at StartTime on the next page.
var boundOnlyCp = _historyContinuationStore.Save(session, new HistoryContinuationState(
tagname, startUtc, endUtc, 0, numValuesPerNode, SegmentWidthTicks: walkHint.WidthTicks,
EmptyFromTicks: walkHint.EmptyFromTicks, EmptyToTicks: walkHint.EmptyToTicks,
PendingEndBound: HistoryBounds.Carry(endBound, [startBound])));
results[handle.Index] = new SdkHistoryReadResult
{
StatusCode = StatusCodes.Good,
HistoryData = new ExtensionObject(ToHistoryDataFromSamples([startBound])),
ContinuationPoint = boundOnlyCp,
};
errors[handle.Index] = ServiceResult.Good;
return;
}
if (startBound is not null && numValuesPerNode == 1)
{
// Nothing at all — no data and no bound found (review of #691): the pre-bounds GoodNoData answer.
results[handle.Index] = new SdkHistoryReadResult
{
StatusCode = StatusCodes.GoodNoData,
HistoryData = new ExtensionObject(ToHistoryDataFromSamples([])),
ContinuationPoint = null,
};
errors[handle.Index] = ServiceResult.Good;
return;
}
if (startBound is not null && numValuesPerNode > 0) readCap = numValuesPerNode - 1;
}
// HistoryRead is NOT under the node-manager Lock — awaiting the async source is safe.
@@ -6242,26 +6397,38 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
if (startUtc > endUtc && !source.SupportsReverseRaw)
{
var page = await ReverseRawReader.ReadAsync(
source, tagname, startUtc, endUtc, numValuesPerNode, MaxTieClusterOverfetch,
Clock.GetUtcNow().UtcDateTime, segmentWidth, ct)
source, tagname, startUtc, endUtc, readCap, MaxTieClusterOverfetch,
nowUtc, walkHint, ct)
.ConfigureAwait(false);
segmentWidth = page.WidthHintTicks;
walkHint = page.Hint;
sourceResult = new HistorianRead(page.Samples, ContinuationPoint: null);
}
else
{
sourceResult = await source
.ReadRawAsync(tagname, startUtc, endUtc, numValuesPerNode, ct)
.ReadRawAsync(tagname, startUtc, endUtc, readCap, ct)
.ConfigureAwait(false);
}
var backendFull = HistoryPaging.IsFullPage(sourceResult.Samples.Count, numValuesPerNode);
var backendFull = HistoryPaging.IsFullPage(sourceResult.Samples.Count, readCap);
// On a resume read, drop the boundary ties already returned on the prior page.
var samples = inboundCp is { Length: > 0 }
? HistoryPaging.TrimBoundaryDuplicates(sourceResult.Samples, startUtc, boundarySkip)
: sourceResult.Samples;
// #691: a bounded read's data window excludes the end time (a sample there is the end bound). A backend whose
// forward end is inclusive (the gateway's own) returns it as data; dropping it also means the data is drained.
if (endBound is not null)
{
var inside = samples.Where(x => HistoryBounds.InDataWindow(x, endBound, resolution)).ToList();
if (inside.Count != samples.Count)
{
samples = inside;
backendFull = false;
}
}
// Oversized tie cluster: a resume read whose FULL backend page does not advance past the resume
// boundary timestamp `startUtc`. That happens when more samples share `startUtc` than the page
// cap — a tie cluster larger than NumValuesPerNode. The fixed-(start,end,cap) backend can only
@@ -6324,9 +6491,16 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
if (nextStartUtc is { } resumeAt)
{
var clusterState = new HistoryContinuationState(
tagname, resumeAt, endUtc, nextSkip, numValuesPerNode, SegmentWidthTicks: segmentWidth);
tagname, resumeAt, endUtc, nextSkip, numValuesPerNode, SegmentWidthTicks: walkHint.WidthTicks,
EmptyFromTicks: walkHint.EmptyFromTicks, EmptyToTicks: walkHint.EmptyToTicks,
PendingEndBound: HistoryBounds.Carry(endBound, slice));
clusterCp = _historyContinuationStore.Save(session, clusterState);
}
else if (endBound is not null)
{
// #691: the data is drained — the end bound ends this page, or is the next one when this one is full.
clusterCp = await AppendEndBoundAsync(slice, startUtc).ConfigureAwait(false);
}
results[handle.Index] = new SdkHistoryReadResult
{
@@ -6338,6 +6512,14 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
return;
}
// #691: the start bound (first page only) leads the data.
if (startBound is not null)
{
var led = new List<DataValueSnapshot>(samples.Count + 2) { startBound };
led.AddRange(samples);
samples = led;
}
// The "full page" test is against the RAW backend count (before trimming): the backend honoured
// the cap, so a full backend page ⇒ there may be more even if we trimmed some boundary ties.
var historyData = ToHistoryDataFromSamples(samples);
@@ -6347,11 +6529,31 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
{
HistoryPaging.ComputeResumeCursor(samples, out var nextStart, out var skip);
var nextState = new HistoryContinuationState(
tagname, nextStart, endUtc, skip, numValuesPerNode, SegmentWidthTicks: segmentWidth);
tagname, nextStart, endUtc, skip, numValuesPerNode, SegmentWidthTicks: walkHint.WidthTicks,
EmptyFromTicks: walkHint.EmptyFromTicks, EmptyToTicks: walkHint.EmptyToTicks,
PendingEndBound: HistoryBounds.Carry(endBound, samples));
// Save may return null (no session on this request) ⇒ degrade to single-shot for this node.
// Built AFTER historyData so a failure projecting samples can never orphan a stored cursor.
outboundCp = _historyContinuationStore.Save(session, nextState);
}
else if (endBound is not null)
{
// #691: the data is drained — the end bound ends this page, or is the next one when this one is full.
var page = new List<DataValueSnapshot>(samples.Count + 1);
page.AddRange(samples);
outboundCp = await AppendEndBoundAsync(page, startUtc).ConfigureAwait(false);
samples = page;
historyData = ToHistoryDataFromSamples(samples);
}
// #691 review: a whole bounded read with no data and no bound found (one page, every value a placeholder) is the
// pre-bounds answer — GoodNoData, no values (§6.5.3.2: bounds make it a success only "if one or both exist").
if (endBound is not null && inboundCp is not { Length: > 0 } && outboundCp is null
&& samples.All(HistoryBounds.IsPlaceholder))
{
samples = [];
historyData = ToHistoryDataFromSamples(samples);
}
results[handle.Index] = new SdkHistoryReadResult
{
@@ -0,0 +1,54 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
/// <summary>One walked-back reverse events page (<see cref="ReverseEventsReader"/>).</summary>
/// <param name="Events">At most the requested count, newest first by <c>(EventTimeUtc, EventId)</c>.</param>
/// <param name="Hint">What the next page of the same read resumes with.</param>
internal readonly record struct ReverseEventsPage(List<HistoricalEvent> Events, ReverseWalkHint Hint);
/// <summary>
/// Gitea #691.4: serves a Part 11 reverse events read — <c>(lo, hi]</c> newest-first — on a backend that reads events
/// forward only (<see cref="IHistorianDataSource.SupportsReverseEvents"/> = <c>false</c>: the Gateway) with the
/// <see cref="ReverseWalk"/>. A segment read is cut short when the source flags truncation (its continuation point); ties
/// are ordered by <see cref="EventIdTieOrder"/> because the gateway orders by event time only.
/// </summary>
internal static class ReverseEventsReader
{
/// <summary>Reads <c>(lo, hi]</c> newest-first from forward event reads.</summary>
/// <param name="source">A backend that reads events forward.</param>
/// <param name="sourceName">Event-source filter (null = all).</param>
/// <param name="hi">The later (inclusive) bound.</param>
/// <param name="lo">The earlier (exclusive) bound.</param>
/// <param name="want">How many events; <c>0</c> = all in the window (one forward read).</param>
/// <param name="maxTieCluster">The tie budget (<c>MaxTieClusterOverfetch</c>) — a ceiling only.</param>
/// <param name="nowUtc">Now, for the look-ahead clamp.</param>
/// <param name="hint">What the previous page carried (default = none).</param>
/// <param name="ct">Cancellation.</param>
/// <returns>The events, newest first, and the hint for the next page.</returns>
/// <exception cref="HistoryReadLimitExceededException">One timestamp holds more than the tie budget.</exception>
public static async Task<ReverseEventsPage> ReadAsync(
IHistorianDataSource source, string? sourceName, DateTime hi, DateTime lo, uint want, int maxTieCluster,
DateTime nowUtc, ReverseWalkHint hint, CancellationToken ct)
{
var budget = Math.Max(1, maxTieCluster);
var (rows, next) = await ReverseWalk.WalkAsync<HistoricalEvent>(
async (start, end, cap, token) =>
{
// cap 0 = the whole window: int.MaxValue is the events contract's "no cap" (a non-positive value would
// mean the backend's DEFAULT cap).
var r = await source.ReadEventsAsync(sourceName, start, end, cap == 0 ? int.MaxValue : cap, reverse: false, token)
.ConfigureAwait(false);
return new ForwardSegmentRead<HistoricalEvent>(r.Events, r.ContinuationPoint is not null);
},
ev => ev.EventTimeUtc.Ticks, EventIdTieOrder.Instance, hi, lo, want, budget, source.TimestampResolutionTicks,
nowUtc, hint,
at => new HistoryReadLimitExceededException(budget,
$"A reverse events read of '{sourceName}' at {at:O} found more than {budget} events within one millisecond; the " +
"backend reads forward only, so the latest of them cannot be selected. Increase " +
"ServerHistorian:MaxTieClusterOverfetch."),
ct)
.ConfigureAwait(false);
return new ReverseEventsPage(rows, next);
}
}
@@ -5,70 +5,29 @@ namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
/// <summary>One walked-back reverse raw page (<see cref="ReverseRawReader"/>).</summary>
/// <param name="Samples">At most the requested cap, newest first.</param>
/// <param name="WidthHintTicks">The segment width the walk ended on — the next page's starting width (0 = none).</param>
internal readonly record struct ReverseRawPage(IReadOnlyList<DataValueSnapshot> Samples, long WidthHintTicks);
internal readonly record struct ReverseRawPage(IReadOnlyList<DataValueSnapshot> Samples, long WidthHintTicks)
{
/// <summary>Everything the next page of the same read should resume with (its width is <see cref="WidthHintTicks"/>).</summary>
public ReverseWalkHint Hint { get; init; } = new(WidthHintTicks);
}
/// <summary>
/// Gitea #686: serves a Part 11 reverse raw read — <c>(lo, hi]</c> newest-first, capped — on a historian backend that can
/// only read forward (<see cref="IHistorianDataSource.SupportsReverseRaw"/> = <c>false</c>: the Gateway, whose
/// <c>ReadRaw</c> streams ascending and rejects <c>start &gt; end</c>).
/// <c>ReadRaw</c> streams ascending and rejects <c>start &gt; end</c>). The walk itself — segment sizing (#689), the
/// truncation rules, edges, ties and bounds — is <see cref="ReverseWalk"/>; this adapter supplies the raw read (a read is
/// cut short when it returned its full cap) and keeps the backend's tie order.
/// </summary>
/// <remarks>
/// <para>
/// <b>Uncapped</b> (<c>maxValues = 0</c>): ONE forward read of the whole window, filtered and reversed in memory — the
/// same rows, round trips and memory as an uncapped forward read (and as a native backend's uncapped reverse read).
/// </para>
/// <para>
/// <b>Capped: walk back in segments.</b> Starting at <c>hi</c>, read one segment <c>(segLo, segHi]</c> FORWARD, keep
/// it newest-first, and step to the next older segment until the cap is met or <c>lo</c> is reached. The first
/// segment is <see cref="InitialSegment"/> wide, or the width the previous page of the same read ended on (carried in
/// the continuation point). After each read the next width is re-aimed at about half the row budget from the density
/// just observed, growing at most ×4 per step — so a live tag's "latest N" is usually one read, a dense tag does not
/// throw away a full read per page, and sparse history is found in a few dozen tiny reads.
/// </para>
/// <para>
/// <b>Never answer from a truncated read.</b> The row budget is <c>M = max(1, MaxTieClusterOverfetch)</c>; a segment
/// read asks for <c>M + 2</c> rows (room for the row an AVEVA read fabricates at its query start, and one more to see
/// past the segment). Rows stream ascending, so a read is complete for its segment when it came back short OR its last
/// row is already past <c>segHi</c>. An incomplete read is discarded: when rows BEFORE the segment (a neighbouring
/// tie cluster inside the alignment slack, the fabricated row) crowded it out it is re-read once with room for them
/// (at most <c>2M + 4</c> rows); otherwise the segment is narrowed (to about half the budget at the density the
/// truncated read showed) and re-read. Only a single timestamp holding more than <c>M</c> samples — a segment already
/// one timestamp-resolution wide that still does not fit — is refused: <see cref="HistoryReadLimitExceededException"/>
/// (⇒ <c>BadTooManyOperations</c>, a request limit, never a backend fault).
/// </para>
/// <para>
/// <b>Edges.</b> A segment is requested as <c>[⌊segLo⌋, ⌈segHi + 1 tick⌉]</c>, both rounded outward to the millisecond
/// (the gateway's / AVEVA's resolution), and filtered to <c>(segLo, segHi]</c> on the samples' own timestamps. That is
/// exact whether the backend's forward end is exclusive (the interface contract) or inclusive (the gateway) and
/// whether it truncates its bounds (QuestDB-style); and because the request start is already on a millisecond, a row
/// an AVEVA read fabricates at its (possibly rounded-up) query start can never land inside the segment (#687).
/// Adjacent segments share no timestamp, so nothing is returned twice.
/// </para>
/// <para>
/// <b>Ties</b> come back in the backend's forward order reversed — the same order the node manager's tie-cluster
/// over-fetch uses for a reverse read — so a resumed page's boundary-tie skip lines up.
/// </para>
/// <para>
/// <b>Bounds.</b> The walk never asks for data after now + <see cref="LookAhead"/> (one day — the historian stores
/// nothing further ahead; a <c>StartTime = DateTime.MaxValue</c> read would otherwise send centuries-ahead query
/// starts), nor before <see cref="Floor"/> (the Unix epoch — AVEVA holds nothing earlier, and an end-only read's year-1
/// lower bound is outside the historian's time encoding). A walked-back read therefore returns only samples strictly
/// after the epoch and no later than now + one day.
/// </para>
/// </remarks>
internal static class ReverseRawReader
{
/// <summary>The first segment's width when no hint is carried.</summary>
internal static readonly TimeSpan InitialSegment = TimeSpan.FromMinutes(1);
internal static readonly TimeSpan InitialSegment = ReverseWalk.InitialSegment;
/// <summary>The walk's exclusive lower limit (see the remarks).</summary>
internal static readonly DateTime Floor = DateTime.UnixEpoch;
/// <summary>The walk's exclusive lower limit.</summary>
internal static readonly DateTime Floor = ReverseWalk.Floor;
/// <summary>How far past "now" the walk starts at most (the same horizon as the Gateway's raw-aggregate look-ahead).</summary>
internal static readonly TimeSpan LookAhead = TimeSpan.FromDays(1);
/// <summary>Request bounds are rounded outward to this (the gateway / AVEVA timestamp resolution).</summary>
private const long Alignment = TimeSpan.TicksPerMillisecond;
/// <summary>How far past "now" the walk starts at most.</summary>
internal static readonly TimeSpan LookAhead = ReverseWalk.LookAhead;
/// <summary>Reads <c>(lo, hi]</c> newest-first from forward reads.</summary>
/// <param name="source">A backend that reads forward.</param>
@@ -76,129 +35,49 @@ internal static class ReverseRawReader
/// <param name="hi">The later (inclusive) bound — the request's StartTime or a reverse resume point.</param>
/// <param name="lo">The earlier (exclusive) bound.</param>
/// <param name="maxValues">Cap; <c>0</c> = every sample in the window (one forward read).</param>
/// <param name="maxTieCluster">Row budget per segment read (<c>MaxTieClusterOverfetch</c>; values below 1 count as 1).</param>
/// <param name="maxTieCluster">The tie budget (<c>MaxTieClusterOverfetch</c>; values below 1 count as 1) — a ceiling only.</param>
/// <param name="nowUtc">Now, for the look-ahead clamp.</param>
/// <param name="widthHintTicks">Starting segment width (the previous page's <see cref="ReverseRawPage.WidthHintTicks"/>), 0 = none.</param>
/// <param name="ct">Cancellation.</param>
/// <returns>The page, newest first, and the width to start the next page with.</returns>
/// <exception cref="HistoryReadLimitExceededException">One timestamp holds more than the row budget.</exception>
/// <exception cref="HistoryReadLimitExceededException">One timestamp holds more than the tie budget.</exception>
public static Task<ReverseRawPage> ReadAsync(
IHistorianDataSource source, string tagname, DateTime hi, DateTime lo, uint maxValues, int maxTieCluster,
DateTime nowUtc, long widthHintTicks, CancellationToken ct) =>
ReadAsync(source, tagname, hi, lo, maxValues, maxTieCluster, nowUtc, new ReverseWalkHint(widthHintTicks), ct);
/// <summary>Reads <c>(lo, hi]</c> newest-first from forward reads, resuming with the previous page's full hint.</summary>
/// <param name="source">A backend that reads forward.</param>
/// <param name="tagname">Historian tag.</param>
/// <param name="hi">The later (inclusive) bound.</param>
/// <param name="lo">The earlier (exclusive) bound.</param>
/// <param name="maxValues">Cap; <c>0</c> = every sample in the window (one forward read).</param>
/// <param name="maxTieCluster">The tie budget — a ceiling only.</param>
/// <param name="nowUtc">Now, for the look-ahead clamp.</param>
/// <param name="hint">What the previous page carried (<see cref="ReverseRawPage.Hint"/>), default = none.</param>
/// <param name="ct">Cancellation.</param>
/// <returns>The page, newest first, and the hint for the next page.</returns>
/// <exception cref="HistoryReadLimitExceededException">One timestamp holds more than the tie budget.</exception>
public static async Task<ReverseRawPage> ReadAsync(
IHistorianDataSource source, string tagname, DateTime hi, DateTime lo, uint maxValues, int maxTieCluster,
DateTime nowUtc, long widthHintTicks, CancellationToken ct)
DateTime nowUtc, ReverseWalkHint hint, CancellationToken ct)
{
var resolution = Math.Max(1L, source.TimestampResolutionTicks);
var granularity = Math.Max(Alignment, resolution);
var budget = Math.Max(1, maxTieCluster);
var result = new List<DataValueSnapshot>();
var horizon = nowUtc.Ticks > DateTime.MaxValue.Ticks - LookAhead.Ticks ? DateTime.MaxValue.Ticks : nowUtc.Ticks + LookAhead.Ticks;
var hiTicks = Math.Min(hi.Ticks, horizon);
var stopTicks = Math.Max(lo.Ticks, Floor.Ticks);
if (hiTicks <= stopTicks) return new ReverseRawPage(result, widthHintTicks);
if (maxValues == 0)
{
// One forward read of the whole window, reversed — the uncapped forward read's own cost and bound.
var all = (await source.ReadRawAsync(tagname, Utc(AlignDown(stopTicks, granularity)),
Utc(AlignUp(hiTicks + 1, granularity)), 0, ct).ConfigureAwait(false)).Samples;
for (var i = all.Count - 1; i >= 0; i--)
{
var ts = Ticks(all[i]);
if (ts > stopTicks && ts <= hiTicks) result.Add(all[i]);
}
return new ReverseRawPage(result, widthHintTicks);
}
var maxWidth = DateTime.MaxValue.Ticks;
var width = widthHintTicks > 0 ? Math.Clamp(widthHintTicks, resolution, maxWidth) : Math.Max(resolution, InitialSegment.Ticks);
var baseCap = budget + 2;
var maxCap = 2 * budget + 4;
var segHi = hiTicks;
while (segHi > stopTicks && result.Count < maxValues)
{
var segLo = segHi - stopTicks <= width ? stopTicks : segHi - width;
var start = Utc(AlignDown(segLo, granularity));
var end = Utc(AlignUp(segHi + 1, granularity));
var cap = baseCap;
IReadOnlyList<DataValueSnapshot> rows;
var narrowed = false;
while (true)
{
ct.ThrowIfCancellationRequested();
rows = (await source.ReadRawAsync(tagname, start, end, (uint)cap, ct).ConfigureAwait(false)).Samples;
if (rows.Count < cap || Ticks(rows[^1]) > segHi) break; // complete for (segLo, segHi]
var leading = 0;
var inner = 0;
var firstInner = 0L;
foreach (var r in rows)
var (rows, next) = await ReverseWalk.WalkAsync(
async (start, end, cap, token) =>
{
var ts = Ticks(r);
if (ts <= segLo) leading++;
else if (ts <= segHi && inner++ == 0) firstInner = ts;
}
// Crowded out by rows before the segment, with the segment itself possibly within budget: re-read once
// with room for them.
var roomier = leading + budget + 2;
if (leading > 0 && inner <= budget && roomier > cap && roomier <= maxCap)
{
cap = roomier;
continue;
}
if (width <= resolution)
throw new HistoryReadLimitExceededException(budget,
$"A reverse raw read of '{tagname}' at {Utc(segHi):O} found more than {budget} samples at one " +
"timestamp; the backend reads forward only, so the latest of them cannot be selected. Increase " +
"ServerHistorian:MaxTieClusterOverfetch.");
// Narrow to about half the budget at the density the truncated read's own samples showed (their span, not
// the segment's — the segment may open on an empty stretch), at least halving.
var lastTs = Ticks(rows[^1]);
var aimed = inner > 1 && lastTs > firstInner
? (long)Math.Min((double)(lastTs - firstInner) * (budget / 2.0) / (inner - 1), width / 2.0)
: width / 4;
width = Math.Max(resolution, aimed);
narrowed = true;
break;
}
if (narrowed) continue; // re-read the same segHi, narrower
var count = 0;
for (var i = rows.Count - 1; i >= 0; i--)
{
var ts = Ticks(rows[i]);
if (ts <= segLo || ts > segHi) continue;
count++;
if (result.Count < maxValues) result.Add(rows[i]);
}
// Re-aim at about half the budget from the density just seen; grow at most x4.
var grown = width > maxWidth / 4 ? maxWidth : width * 4;
width = count == 0
? grown
: (long)Math.Clamp((double)width * Math.Max(1.0, budget / 2.0) / count, resolution, grown);
segHi = segLo;
}
return new ReverseRawPage(result, width);
var samples = (await source.ReadRawAsync(tagname, start, end, (uint)cap, token).ConfigureAwait(false)).Samples;
return new ForwardSegmentRead<DataValueSnapshot>(samples, cap > 0 && samples.Count >= cap);
},
Ticks, tieOrder: null, hi, lo, maxValues, budget, source.TimestampResolutionTicks, nowUtc, hint,
at => new HistoryReadLimitExceededException(budget,
$"A reverse raw read of '{tagname}' at {at:O} found more than {budget} samples within one millisecond; the " +
"backend reads forward only, so the latest of them cannot be selected. Increase " +
"ServerHistorian:MaxTieClusterOverfetch."),
ct)
.ConfigureAwait(false);
return new ReverseRawPage(rows, next.WidthTicks) { Hint = next };
}
private static long Ticks(DataValueSnapshot s) => (s.SourceTimestampUtc ?? DateTime.MinValue).Ticks;
private static DateTime Utc(long ticks) => new(ticks, DateTimeKind.Utc);
private static long AlignDown(long ticks, long step) => ticks - ticks % step;
private static long AlignUp(long ticks, long step)
{
var down = AlignDown(ticks, step);
if (down == ticks) return ticks;
return down > DateTime.MaxValue.Ticks - step ? DateTime.MaxValue.Ticks : down + step;
}
}
@@ -0,0 +1,448 @@
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
/// <summary>What one walked-back page tells the next page of the same read (carried in the continuation point).</summary>
/// <param name="WidthTicks">The segment width the walk ended on, aimed at the page's density — the next page's first width (0 = none).</param>
/// <param name="EmptyFromTicks">
/// With <paramref name="EmptyToTicks"/>: a stretch <c>(EmptyFrom, EmptyTo)</c> (both exclusive) a complete segment read
/// proved holds no row (#691) — typically from the oldest row the page saw down to the walk's floor. A resumed page
/// steps straight over it instead of re-walking it. Both 0 = none.
/// </param>
/// <param name="EmptyToTicks">The exclusive upper end of the empty stretch.</param>
internal readonly record struct ReverseWalkHint(long WidthTicks, long EmptyFromTicks = 0, long EmptyToTicks = 0)
{
/// <summary>Whether an empty stretch is carried.</summary>
public bool HasEmptyStretch => EmptyToTicks > EmptyFromTicks && EmptyToTicks > 0;
}
/// <summary>One forward read of a segment: the rows (ascending by time) and whether a cap cut the read short.</summary>
/// <typeparam name="T">Row type.</typeparam>
/// <param name="Rows">Rows in ascending time order (ties in any order).</param>
/// <param name="Truncated">A cap stopped the read — more rows may follow the last one.</param>
internal readonly record struct ForwardSegmentRead<T>(IReadOnlyList<T> Rows, bool Truncated);
/// <summary>
/// Gitea #686/#689: the walk-back that serves a Part 11 reverse read — <c>(lo, hi]</c> newest-first, capped — from a
/// backend that can only read FORWARD. Shared by raw values (<see cref="ReverseRawReader"/>) and events; the row type
/// only has to expose a timestamp.
/// </summary>
/// <remarks>
/// <para>
/// <b>Uncapped</b> (<c>want = 0</c>): ONE forward read of the whole window, filtered and reversed in memory.
/// </para>
/// <para>
/// <b>Capped: walk back in segments.</b> Starting at <c>hi</c>, read one segment <c>(segLo, segHi]</c> FORWARD, keep
/// it newest-first, and step to the next older segment until the cap is met or <c>lo</c> is reached. The first
/// segment is <see cref="InitialSegment"/> wide, or the width the previous page of the same read ended on (carried in
/// the continuation point). After each read the next width is re-aimed from the density just observed at about half
/// the PAGE budget <c>2N + 16</c> (≈ N + 8 rows), growing at most ×4 per step (#689 — it used to aim at half the tie
/// budget, ~32k rows, whatever N was) — so a live tag's "latest N" is usually one read of about N rows, and sparse
/// history is found in a few dozen tiny reads.
/// </para>
/// <para>
/// <b>Read sizes.</b> A segment read asks for its budget + 2 rows (room for a row an AVEVA read fabricates at its query
/// start, and one to see past the segment). The budget is the page budget, except a <b>discovery</b> segment — the
/// first of a page with no carried width, or one after an empty segment (whose ×4 growth may overshoot into dense
/// data) — which may read up to <see cref="DiscoveryRows"/>. The tie budget <c>M = max(1, MaxTieClusterOverfetch)</c>
/// is only a CEILING: a single timestamp holding more rows than the segment budget escalates to <c>M</c>.
/// </para>
/// <para>
/// <b>Never answer from a truncated read.</b> Rows stream ascending, so a read is complete for its segment when it was
/// not cut short OR its last row is already past <c>segHi</c>. An incomplete read is discarded: when rows BEFORE the
/// segment (a neighbouring tie cluster inside the alignment slack, a fabricated row) crowded it out it is re-read once
/// with room for them (at most twice the budget + 4); otherwise the segment is narrowed — to the density the truncated
/// read showed, and never again below the newest row it saw (<c>known</c>: rows exist there, so walking past it before
/// finding what lies above it would re-read the same dense stretch over and over; the gap above it is bisected
/// instead). Only more than <c>M</c> rows within one millisecond — a segment no wider than the 1 ms request
/// granularity that still does not fit at the tie budget — is refused (per millisecond, not per timestamp: narrower
/// requests are the same request, and the Gateway stores milliseconds).
/// </para>
/// <para>
/// <b>Proving the rest empty (#691).</b> Walking down to the floor through empty history used to take one read per ×4
/// growth step (~14 reads from a minute to the epoch) — on every page that ran out of rows. After two consecutive empty
/// segments the walk asks once per page for the WHOLE remainder <c>(lo, segHi]</c> at the page budget: a complete
/// answer ends the walk in one read; a truncated one (deep history) is discarded and the walk carries on as before. And
/// the stretch a page's last complete segment proved empty below its oldest row rides the continuation point
/// (<see cref="ReverseWalkHint.EmptyFromTicks"/>), so the next page steps over it — the rows of a page are often the
/// last ones before a stretch the previous page already read.
/// </para>
/// <para>
/// <b>Edges.</b> A segment is requested as <c>[⌊segLo⌋, ⌈segHi + 1 tick⌉]</c>, both rounded outward to the millisecond
/// (the gateway's / AVEVA's resolution), and filtered to <c>(segLo, segHi]</c> on the rows' own timestamps — exact
/// whether the backend's forward end is exclusive or inclusive and whether it truncates its bounds; a row fabricated at
/// the (millisecond) query start can never land inside the segment (#687). Adjacent segments share no timestamp.
/// </para>
/// <para>
/// <b>Ties</b> come back in the backend's forward order reversed, or — with a tie order given (events, whose gateway
/// order within a timestamp is arbitrary) — sorted by it and reversed, so a resumed page's boundary-tie skip lines up.
/// </para>
/// <para>
/// <b>Bounds.</b> The walk never asks for data after now + <see cref="LookAhead"/> nor before <see cref="Floor"/> (the
/// Unix epoch — AVEVA holds nothing earlier). A walked-back read returns only rows strictly after the epoch and no
/// later than now + one day.
/// </para>
/// </remarks>
internal static class ReverseWalk
{
/// <summary>The first segment's width when no hint is carried.</summary>
internal static readonly TimeSpan InitialSegment = TimeSpan.FromMinutes(1);
/// <summary>The walk's exclusive lower limit.</summary>
internal static readonly DateTime Floor = DateTime.UnixEpoch;
/// <summary>How far past "now" the walk starts at most.</summary>
internal static readonly TimeSpan LookAhead = TimeSpan.FromDays(1);
/// <summary>Slack added to a page's row budget (#689): room for density variation between segments.</summary>
internal const int PageSlack = 16;
/// <summary>The most rows a discovery segment (see the remarks) may read before it counts as truncated (#689).</summary>
internal const int DiscoveryRows = 1024;
/// <summary>Request bounds are rounded outward to this (the gateway / AVEVA timestamp resolution).</summary>
private const long Alignment = TimeSpan.TicksPerMillisecond;
/// <summary>The rows a segment is sized for: <c>2N + </c><see cref="PageSlack"/>, never above the tie budget.</summary>
/// <param name="want">The page cap N (&gt; 0).</param>
/// <param name="tieBudget">The tie budget M (&gt;= 1).</param>
/// <returns>The page budget.</returns>
internal static int PageBudget(uint want, int tieBudget) => (int)Math.Clamp(2L * want + PageSlack, 1L, Math.Max(1, tieBudget));
/// <summary>Reads <c>(lo, hi]</c> newest-first from forward reads.</summary>
/// <typeparam name="T">Row type.</typeparam>
/// <param name="read">One forward read <c>[start, end]</c> asking for at most <c>cap</c> rows (<c>0</c> = all).</param>
/// <param name="ticksOf">A row's timestamp, in ticks.</param>
/// <param name="tieOrder">Order within one timestamp (null = the backend's forward order).</param>
/// <param name="hi">The later (inclusive) bound.</param>
/// <param name="lo">The earlier (exclusive) bound.</param>
/// <param name="want">Cap; <c>0</c> = every row in the window (one forward read).</param>
/// <param name="tieBudget">The tie budget M (values below 1 count as 1).</param>
/// <param name="resolutionTicks">The backend's timestamp resolution (&gt;= 1).</param>
/// <param name="nowUtc">Now, for the look-ahead clamp.</param>
/// <param name="hint">What the previous page of the same read carried (default = none).</param>
/// <param name="tooManyAtOneTimestamp">Builds the exception for a timestamp holding more than M rows.</param>
/// <param name="ct">Cancellation.</param>
/// <returns>At most <paramref name="want"/> rows, newest first, and the hint for the next page.</returns>
public static async Task<(List<T> Rows, ReverseWalkHint Next)> WalkAsync<T>(
Func<DateTime, DateTime, int, CancellationToken, Task<ForwardSegmentRead<T>>> read,
Func<T, long> ticksOf,
IComparer<T>? tieOrder,
DateTime hi, DateTime lo, uint want, int tieBudget, long resolutionTicks, DateTime nowUtc,
ReverseWalkHint hint, Func<DateTime, Exception> tooManyAtOneTimestamp, CancellationToken ct)
{
var resolution = Math.Max(1L, resolutionTicks);
var granularity = Math.Max(Alignment, resolution);
var budget = Math.Max(1, tieBudget);
var result = new List<T>();
var horizon = nowUtc.Ticks > DateTime.MaxValue.Ticks - LookAhead.Ticks ? DateTime.MaxValue.Ticks : nowUtc.Ticks + LookAhead.Ticks;
var hiTicks = Math.Min(hi.Ticks, horizon);
var stopTicks = Math.Max(lo.Ticks, Floor.Ticks);
if (hiTicks <= stopTicks) return (result, hint);
if (want == 0)
{
// One forward read of the whole window, reversed — the uncapped forward read's own cost and bound.
var all = (await read(Utc(AlignDown(stopTicks, granularity)), Utc(AlignUp(hiTicks + 1, granularity)), 0, ct)
.ConfigureAwait(false)).Rows;
AppendNewestFirst(all, ticksOf, tieOrder, stopTicks, hiTicks, result, int.MaxValue);
return (result, hint);
}
var maxWidth = DateTime.MaxValue.Ticks;
var discovering = hint.WidthTicks <= 0;
var width = discovering ? Math.Max(resolution, InitialSegment.Ticks) : Math.Clamp(hint.WidthTicks, resolution, maxWidth);
var pageBudget = PageBudget(want, budget);
var discoveryBudget = Math.Min(budget, Math.Max(pageBudget, DiscoveryRows));
// `known` > 0: a truncated read saw rows up to this timestamp (below segHi), and nothing is known of (known, segHi].
// Segments stay above it until that stretch is read; then `knownWidth` (the density the truncated read showed)
// sizes the segment that ends on it. Searching the gap: first a top-anchored segment at that density (data running
// right up to segHi); if it is empty, the data stops somewhere above `known`, so GALLOP up from it — read
// (known + lowerBy, segHi] with lowerBy growing ×4, `known` following what each truncated read saw — until a read
// comes back empty (the top is now bracketed below its segLo), then BISECT that bracket. Any complete read holding
// rows ends the search. A burst that ended long before segHi is found in a few reads instead of bisecting the whole
// gap down to the millisecond, and a long dense stretch is crossed geometrically, not a budget at a time (#691.4
// review). `noGallop` falls back to plain top-anchored halving if a galloping read is crowded by rows before it.
var known = 0L;
var knownWidth = 0L;
var lowerBy = 0L;
var bisecting = false;
var noGallop = false;
var segHi = hiTicks;
// #691: the remainder probe (once per page, after two empty segments) and the empty stretches.
var emptyRun = 0;
var probed = false;
var carried = hint.HasEmptyStretch ? (hint.EmptyFromTicks, hint.EmptyToTicks) : (0L, 0L);
(long From, long To) proven = (0L, 0L);
while (segHi > stopTicks && result.Count < want)
{
if (carried.Item2 > carried.Item1 && segHi > carried.Item1 && segHi < carried.Item2)
{
segHi = carried.Item1; // (from, segHi] lies inside a stretch a previous page proved empty
continue;
}
var probe = !probed && known == 0 && emptyRun >= 2 && segHi - stopTicks > width;
var w = probe ? segHi - stopTicks : width;
var lowered = false;
var bisected = false;
if (known > 0)
{
var gap = segHi - known;
lowered = lowerBy > 0 && lowerBy < gap;
bisected = !lowered && (bisecting || lowerBy > 0) && gap > 2 * Math.Max(granularity, knownWidth);
// Never narrower than the request granularity: requests are rounded outward to it, so a narrower segment
// re-sends the same request — and a width left at one tick (a tie cluster's) stepped segHi down a tick per
// read, ~10,000 identical reads (#691.4 re-review). Keep the Min: a truncated read crowded by rows before
// the segment must still shrink the segment.
w = lowered ? gap - lowerBy
: bisected ? gap / 2
: Math.Min(Math.Max(w, granularity), gap <= 2 * granularity ? gap : gap / 2);
}
var segLo = segHi - stopTicks <= w ? stopTicks : segHi - w;
var start = Utc(AlignDown(segLo, granularity));
var end = Utc(AlignUp(segHi + 1, granularity));
var segBudget = discovering && !probe ? discoveryBudget : pageBudget;
var cap = segBudget + 2;
IReadOnlyList<T> rows;
var narrowed = false;
while (true)
{
ct.ThrowIfCancellationRequested();
var segment = await read(start, end, cap, ct).ConfigureAwait(false);
rows = segment.Rows;
if (!segment.Truncated || rows.Count == 0 || ticksOf(rows[^1]) > segHi) break; // complete for (segLo, segHi]
if (probe)
{
// Deep history below: the probe saw only its oldest rows. Carry on walking from the same segHi.
probed = true;
narrowed = true;
break;
}
var leading = 0;
var inner = 0;
var firstInner = 0L;
var lastInner = 0L;
foreach (var r in rows)
{
var ts = ticksOf(r);
if (ts <= segLo) leading++;
else if (ts <= segHi)
{
if (inner++ == 0) firstInner = ts;
lastInner = ts;
}
}
// Crowded out by rows before the segment, with the segment itself possibly within budget: re-read once
// with room for them.
var roomier = leading + segBudget + 2;
if (leading > 0 && inner <= segBudget && roomier > cap && roomier <= 2 * segBudget + 4)
{
cap = roomier;
continue;
}
if (segHi - segLo <= granularity)
{
// A segment no wider than the request granularity (1 ms — narrower requests are the same request) still
// over its budget: a tie cluster, or a burst inside one millisecond. Escalate to the tie budget M before
// refusing — up to M rows within one millisecond are served (#686); more are refused, per millisecond
// (#691.4 re-review — exact for the millisecond-resolution Gateway).
if (segBudget < budget)
{
segBudget = budget;
cap = segBudget + 2;
continue;
}
throw tooManyAtOneTimestamp(Utc(segHi));
}
// Narrow — the segment shrinks every time, so this terminates.
if (inner > 0 && lastInner == segHi)
{
width = resolution; // a tie cluster at the top of the segment: read that one timestamp
lowerBy = 0;
bisecting = false;
}
else if (inner > 0 && (lowered || bisected))
{
// Data reaches past this segment's bottom: move `known` up to what this read saw; a gallop goes further.
known = lastInner;
if (lowered) lowerBy = lowerBy > maxWidth / 4 ? maxWidth : lowerBy * 4;
}
else if (inner > 0)
{
// Rows exist at lastInner; what lies above it is unknown. Aim at about half the page budget at the
// density the truncated rows showed (their span, not the segment's — it may open on an empty stretch).
var aimed = inner > 1 && lastInner > firstInner
? (long)Math.Min((double)(lastInner - firstInner) * (pageBudget / 2.0) / (inner - 1), maxWidth)
: resolution;
known = lastInner;
knownWidth = Math.Max(resolution, aimed);
width = Math.Max(resolution, Math.Min(aimed, segHi - segLo - 1));
lowerBy = 0;
bisecting = false;
}
else
{
// Only rows before the segment: shrink it — top-anchored from here on (a gallop would re-send the same
// crowded read).
width = Math.Max(resolution, Math.Min(width, segHi - segLo) / 4);
if (lowered || bisected) noGallop = true;
lowerBy = 0;
bisecting = false;
}
discovering = false;
narrowed = true;
break;
}
if (narrowed) continue; // re-read the same segHi, narrower
var count = AppendNewestFirst(rows, ticksOf, tieOrder, segLo, segHi, result, (int)Math.Min(int.MaxValue, want),
out var oldest);
// This complete read proves (segLo, oldest) empty — or the whole segment when it held nothing.
proven = count > 0 ? (segLo, oldest) : (segLo, segHi + 1);
if (probe)
{
probed = true;
segHi = segLo; // the whole remainder is read
continue;
}
emptyRun = count == 0 ? emptyRun + 1 : 0;
var grown = width > maxWidth / 4 ? maxWidth : width * 4;
if (count == 0)
{
discovering = true; // ×4 growth may overshoot into dense data
if (known > 0)
{
// An empty top-anchored segment: gallop up from `known`. An empty galloping read: the top is bracketed
// in (known, segLo] — bisect it (an empty bisecting read keeps bisecting).
if (lowered || bisected)
{
lowerBy = 0;
bisecting = true;
}
else if (noGallop)
{
width = maxWidth; // halve the gap, top-anchored
}
else
{
lowerBy = Math.Max(knownWidth, granularity);
}
}
else
{
width = grown;
}
}
else
{
// Re-aim at about half the page budget from the density just seen.
width = (long)Math.Clamp((double)(segHi - segLo) * Math.Max(1.0, pageBudget / 2.0) / count, resolution, grown);
discovering = false;
known = 0;
lowerBy = 0;
bisecting = false;
noGallop = false;
}
segHi = segLo;
if (known > 0 && segHi <= known)
{
// Everything above the newest row the truncated read saw is read: the next segment ends on it, at its density.
known = 0;
lowerBy = 0;
bisecting = false;
noGallop = false;
width = knownWidth;
discovering = false;
}
}
// Carry the stretch the last complete read proved empty, joined to the one carried in when they touch; when they
// do not, keep the one reaching the floor (it saves a whole walk), else the newer.
var next = carried;
if (proven.To > proven.From)
{
if (carried.Item2 > carried.Item1 && proven.From < carried.Item2 && carried.Item1 < proven.To)
next = (Math.Min(proven.From, carried.Item1), Math.Max(proven.To, carried.Item2));
else if (!(carried.Item2 > carried.Item1 && carried.Item1 <= stopTicks) || proven.From <= stopTicks)
next = proven;
}
return (result, new ReverseWalkHint(width, next.Item1, next.Item2));
}
/// <summary>Appends the rows in <c>(segLo, segHi]</c> newest-first (until <paramref name="limit"/>) and counts them.</summary>
private static int AppendNewestFirst<T>(
IReadOnlyList<T> rows, Func<T, long> ticksOf, IComparer<T>? tieOrder, long segLo, long segHi, List<T> result, int limit) =>
AppendNewestFirst(rows, ticksOf, tieOrder, segLo, segHi, result, limit, out _);
/// <summary>As above, also returning the oldest in-range timestamp (meaningless when none).</summary>
private static int AppendNewestFirst<T>(
IReadOnlyList<T> rows, Func<T, long> ticksOf, IComparer<T>? tieOrder, long segLo, long segHi, List<T> result, int limit,
out long oldest)
{
oldest = long.MaxValue;
IReadOnlyList<T> ordered = rows;
if (tieOrder is not null)
{
var copy = rows.ToList();
copy.Sort((a, b) =>
{
var c = ticksOf(a).CompareTo(ticksOf(b));
return c != 0 ? c : tieOrder.Compare(a, b);
});
ordered = copy;
}
var count = 0;
for (var i = ordered.Count - 1; i >= 0; i--)
{
var ts = ticksOf(ordered[i]);
if (ts <= segLo || ts > segHi) continue;
count++;
if (ts < oldest) oldest = ts;
if (result.Count < limit) result.Add(ordered[i]);
}
return count;
}
private static DateTime Utc(long ticks) => new(ticks, DateTimeKind.Utc);
private static long AlignDown(long ticks, long step) => ticks - ticks % step;
private static long AlignUp(long ticks, long step)
{
var down = AlignDown(ticks, step);
if (down == ticks) return ticks;
return down > DateTime.MaxValue.Ticks - step ? DateTime.MaxValue.Ticks : down + step;
}
}
/// <summary>
/// Orders events within one timestamp by <see cref="HistoricalEvent.EventId"/> (ordinal) — the tie order a walked-back
/// events read (#691.4) and its oversized-cluster over-fetch share, because the Gateway orders by event time only.
/// </summary>
internal sealed class EventIdTieOrder : IComparer<HistoricalEvent>
{
/// <summary>The shared instance.</summary>
public static readonly EventIdTieOrder Instance = new();
/// <inheritdoc />
public int Compare(HistoricalEvent? x, HistoricalEvent? y) => string.CompareOrdinal(x?.EventId, y?.EventId);
}
@@ -82,13 +82,21 @@ public sealed class ServerHistorianOptions
/// <c>MaxTieClusterOverfetch + 1</c> ties at that single timestamp (a <c>start == end</c> read) and
/// pages through them; this bounds how large a single-timestamp burst the server will buffer in
/// memory before it gives up and surfaces <c>BadTooManyOperations</c> for that node (#686; was
/// <c>BadHistoryOperationUnsupported</c>). It is also the per-read row budget of a reverse raw read walked
/// back from forward reads on a backend that cannot read newest-first (the Gateway): one timestamp holding
/// <c>BadHistoryOperationUnsupported</c>). It is also the tie-cluster CEILING of a reverse read walked
/// back from forward reads on a backend that cannot read newest-first (the Gateway; segments are sized from the page, #689): one timestamp holding
/// more samples than this is refused <c>BadTooManyOperations</c>. The default (65536) comfortably covers
/// any realistic same-millisecond burst; a value below 1 is treated as 1.
/// </summary>
public int MaxTieClusterOverfetch { get; init; } = 65536;
/// <summary>
/// Gitea #691: whether HistoryRead-Raw honours <c>ReadRawModifiedDetails.ReturnBounds</c> (OPC UA Part 11 bounding
/// values). Default <c>true</c>. <c>false</c> is the kill switch back to the pre-#691 answer: the flag is ignored
/// (stored samples only) and a modified read carrying it is not refused <c>Bad_InvalidArgument</c>. Note the .NET
/// SDK's <c>new ReadRawModifiedDetails()</c> defaults <c>ReturnBounds</c> (and <c>IsReadModified</c>) to <c>true</c>.
/// </summary>
public bool HonorReturnBounds { get; init; } = true;
/// <summary>
/// Maximum number of historized nodes served CONCURRENTLY within one HistoryRead batch (arch-review
/// 03/S3). The SDK's <c>HistoryRead*</c> override surface is synchronous, so each arm block-bridges
@@ -68,17 +68,19 @@ public sealed class NullHistorianDataSourceTests
}
/// <summary>
/// #675: the direction-aware overload's default — forward is the five-argument read, reverse is refused
/// (the server answers <c>BadHistoryOperationUnsupported</c>).
/// #691.4 review: like raw, the null source reads events in either direction (empty) and says so — the server sends
/// it the reverse read directly instead of walking back through empty forward reads (it refused reverse under #675).
/// </summary>
[Fact]
public async Task Direction_aware_ReadEventsAsync_forwards_forward_and_refuses_reverse()
public async Task Direction_aware_ReadEventsAsync_is_empty_in_both_directions()
{
IHistorianDataSource source = Source;
source.SupportsReverseEvents.ShouldBeTrue();
(await source.ReadEventsAsync(null, DateTime.UtcNow.AddHours(-1), DateTime.UtcNow, 0, reverse: false, Ct))
.Events.ShouldBeEmpty();
await Should.ThrowAsync<NotSupportedException>(() =>
source.ReadEventsAsync(null, DateTime.UtcNow, DateTime.UtcNow.AddHours(-1), 0, reverse: true, Ct));
var reverse = await source.ReadEventsAsync(null, DateTime.UtcNow, DateTime.UtcNow.AddHours(-1), 0, reverse: true, Ct);
reverse.Events.ShouldBeEmpty();
reverse.ContinuationPoint.ShouldBeNull();
}
/// <summary>GetHealthSnapshot reports a fully-disabled (zeroed, disconnected, no-node) shape.</summary>
@@ -0,0 +1,100 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
/// <summary>
/// Gitea #691: the server composes Part 11 bounding values (HistoryRead-Raw <c>ReturnBounds</c>) from exactly these raw
/// reads on a backend that reads newest-first natively (<c>HistoryBounds</c> in the OPC UA server) — so every backend must
/// answer them precisely: "at or before T" = a reverse read of <c>(0, T]</c> capped at one, "at or after T" = a forward read
/// of <c>[T, MaxValue)</c> capped at one, "strictly after T" = the same from <c>T + resolution</c>. A bound is a stored
/// sample of any quality, so a Bad row or an outage marker comes back as the bound.
/// </summary>
public abstract class BoundingValueConformance : ConformanceBase
{
private static readonly CancellationToken Ct = CancellationToken.None;
private static readonly DateTime Floor = new(0, DateTimeKind.Utc);
private static readonly DateTime Ceiling = new(DateTime.MaxValue.Ticks, DateTimeKind.Utc);
private async Task<DataValueSnapshot?> AtOrBeforeAsync(IHistorianDataSource src, string tag, DateTime t) =>
(await src.ReadRawAsync(tag, t, Floor, 1, Ct)).Samples.SingleOrDefault();
private async Task<DataValueSnapshot?> AtOrAfterAsync(IHistorianDataSource src, string tag, DateTime t) =>
(await src.ReadRawAsync(tag, t, Ceiling, 1, Ct)).Samples.SingleOrDefault();
[Fact]
public async Task The_sample_at_or_before_a_time_is_the_latest_one_on_or_before_it()
{
await ConformanceData.SeedAAsync(B);
var src = B.CreateDataSource();
(await AtOrBeforeAsync(src, "A", At(10)))!.Value.ShouldBe(20d); // exactly on it
(await AtOrBeforeAsync(src, "A", At(15)))!.Value.ShouldBe(20d); // between samples
(await AtOrBeforeAsync(src, "A", At(1000)))!.Value.ShouldBe(50d); // past the last
(await AtOrBeforeAsync(src, "A", At(-1))).ShouldBeNull(); // before the first: no bound
(await AtOrBeforeAsync(src, "A", At(15)))!.SourceTimestampUtc!.Value.Ticks.ShouldBe(B.Round(At(10)).Ticks);
}
[Fact]
public async Task The_sample_at_or_after_a_time_is_the_earliest_one_on_or_after_it()
{
await ConformanceData.SeedAAsync(B);
var src = B.CreateDataSource();
(await AtOrAfterAsync(src, "A", At(10)))!.Value.ShouldBe(20d); // exactly on it
(await AtOrAfterAsync(src, "A", At(15)))!.Value.ShouldBe(30d); // between samples
(await AtOrAfterAsync(src, "A", At(-100)))!.Value.ShouldBe(10d); // before the first
(await AtOrAfterAsync(src, "A", At(41))).ShouldBeNull(); // past the last: no bound
}
[Fact]
public async Task The_sample_strictly_after_a_time_starts_one_storage_resolution_later()
{
await ConformanceData.SeedAAsync(B);
var src = B.CreateDataSource();
(await AtOrAfterAsync(src, "A", At(10).AddTicks(B.ResolutionTicks)))!.Value.ShouldBe(30d);
(await AtOrAfterAsync(src, "A", At(40).AddTicks(B.ResolutionTicks))).ShouldBeNull();
}
[Fact]
public async Task A_time_between_storage_ticks_finds_the_sample_the_backend_stores_at_its_truncation()
{
// The server compares "on the bound" at the storage resolution, so the lookups must agree with the window reads:
// a time a fraction of a resolution step after a stored sample is that sample's instant on a coarse backend.
await ConformanceData.SeedAAsync(B);
var src = B.CreateDataSource();
var t = At(10).AddTicks(B.ResolutionTicks - 1); // == At(10) on a 1-tick backend
(await AtOrBeforeAsync(src, "A", t))!.Value.ShouldBe(20d);
var after = await AtOrAfterAsync(src, "A", t);
after!.SourceTimestampUtc!.Value.Ticks.ShouldBeGreaterThanOrEqualTo(B.Round(t).Ticks);
after.Value.ShouldBe(B.Round(t) == B.Round(At(10)) ? 20d : 30d);
}
[Fact]
public async Task A_bad_sample_or_an_outage_marker_is_a_bound_as_stored()
{
await B.SeedAsync("M",
(At(0), HistorianValue.FromDouble(10), Good),
(At(5), HistorianValue.Empty, BadNoCommunication),
(At(7), HistorianValue.FromDouble(99), Bad));
var src = B.CreateDataSource();
var before = await AtOrBeforeAsync(src, "M", At(6));
before!.Value.ShouldBeNull();
before.StatusCode.ShouldBe(0x80310000u);
var after = await AtOrAfterAsync(src, "M", At(6));
after!.Value.ShouldBe(99d);
after.StatusCode.ShouldBe(0x80000000u);
}
[Fact]
public async Task An_unknown_tag_has_no_bounds()
{
var src = B.CreateDataSource();
(await AtOrBeforeAsync(src, "nope", At(10))).ShouldBeNull();
(await AtOrAfterAsync(src, "nope", At(10))).ShouldBeNull();
}
}
@@ -54,6 +54,9 @@ public sealed class FakeHistorianGatewayClient : IHistorianGatewayClient
public int LastReadEventsMaxEvents;
public int ReadEventsCallCount;
/// <summary>When set, ReadEvents serves <c>EventsSource(start, end)</c> instead of <see cref="Events"/>.</summary>
public Func<DateTime, DateTime, IEnumerable<HistorianEvent>>? EventsSource;
/// <summary>Thrown (deferred to first enumeration) by every read method when set.</summary>
public Exception? ThrowOnRead;
@@ -100,12 +103,35 @@ public sealed class FakeHistorianGatewayClient : IHistorianGatewayClient
LastReadRawStartUtc = startUtc;
LastReadRawEndUtc = endUtc;
LastReadRawMaxValues = maxValues;
ReadRawCallCount++;
RawCalls.Add((startUtc, endUtc, maxValues));
lock (RawCalls)
{
ReadRawCallCount++;
RawCalls.Add((startUtc, endUtc, maxValues));
}
if (MaxValuesCeiling is { } ceiling && maxValues > ceiling)
{
// The real gateway's refusal (HistorianReadService.ReadRaw), raised on the first MoveNext before any row, as
// the client package surfaces it: a HistorianGatewayException wrapping the RpcException.
var rpc = new Grpc.Core.RpcException(new Grpc.Core.Status(Grpc.Core.StatusCode.InvalidArgument,
CeilingRejectionDetail ?? $"max_values exceeds the server ceiling of {ceiling}; read large ranges by time-slicing " +
"(split the range and read the slices sequentially), or enable " +
"Historian:ReadLimits:AllowStreamingBeyondCeiling to page through server-side."));
return ToAsyncStream(Array.Empty<HistorianSample>(),
new ZB.MOM.WW.HistorianGateway.Client.HistorianGatewayException(rpc.Status.Detail, rpc), ct);
}
var items = RawSource is null ? RawSamples : RawSource(startUtc, endUtc, maxValues).ToList();
return CountingStream(ToAsyncStream(items, ThrowOnRead, ct));
}
/// <summary>
/// A strict gateway (<c>Historian:ReadLimits:AllowStreamingBeyondCeiling=false</c>) with this
/// <c>MaxValuesCeiling</c>: a ReadRaw asking for more is refused <c>InvalidArgument</c> (#691.3).
/// </summary>
public int? MaxValuesCeiling;
/// <summary>Overrides the refusal's detail text (an older/other gateway wording).</summary>
public string? CeilingRejectionDetail;
public IAsyncEnumerable<HistorianAggregateSample> ReadAggregateAsync(
string tag,
DateTime startUtc,
@@ -120,9 +146,25 @@ public sealed class FakeHistorianGatewayClient : IHistorianGatewayClient
LastAggregateMode = mode;
LastAggregateInterval = interval;
ReadAggregateCallCount++;
return ToAsyncStream(AggregateSamples, ThrowOnRead, ct);
AggregateCalls.Add((startUtc, endUtc, mode, interval));
if (AggregateSource is null) return ToAsyncStream(AggregateSamples, ThrowOnRead, ct);
if (startUtc > endUtc || interval <= TimeSpan.Zero)
{
// The real gateway's request validation (ValidateRange; interval must be > 0), raised before any row.
var rpc = new Grpc.Core.RpcException(new Grpc.Core.Status(Grpc.Core.StatusCode.InvalidArgument,
startUtc > endUtc ? "start_time must be less than or equal to end_time." : "interval must be greater than zero."));
return ToAsyncStream(Array.Empty<HistorianAggregateSample>(),
new ZB.MOM.WW.HistorianGateway.Client.HistorianGatewayException(rpc.Status.Detail, rpc), ct);
}
return ToAsyncStream(AggregateSource(startUtc, endUtc, mode, interval).ToList(), ThrowOnRead, ct);
}
/// <summary>When set, ReadAggregate serves <c>AggregateSource(start, end, mode, interval)</c> (and refuses start &gt; end).</summary>
public Func<DateTime, DateTime, RetrievalMode, TimeSpan, IEnumerable<HistorianAggregateSample>>? AggregateSource;
/// <summary>Every ReadAggregate call's (start, end, mode, interval), in order.</summary>
public readonly List<(DateTime Start, DateTime End, RetrievalMode Mode, TimeSpan Interval)> AggregateCalls = new();
public Task<IReadOnlyList<HistorianSample>> ReadAtTimeAsync(
string tag,
IReadOnlyList<DateTime> timestampsUtc,
@@ -148,7 +190,7 @@ public sealed class FakeHistorianGatewayClient : IHistorianGatewayClient
LastReadEventsEndUtc = endUtc;
LastReadEventsMaxEvents = maxEvents;
ReadEventsCallCount++;
return ToAsyncStream(Events, ThrowOnRead, ct);
return ToAsyncStream(EventsSource is null ? Events : EventsSource(startUtc, endUtc).ToList(), ThrowOnRead, ct);
}
public Task<WriteAck> WriteLiveValuesAsync(
@@ -0,0 +1,174 @@
using Google.Protobuf.WellKnownTypes;
using Microsoft.Extensions.Logging.Abstractions;
using Shouldly;
using Xunit;
using ZB.MOM.WW.HistorianGateway.Contracts.Grpc;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests;
/// <summary>
/// Gitea #691: Part 11 bounding values on the Gateway, composed by the server's <see cref="HistoryBounds"/> over the REAL
/// source and an AVEVA-shaped fake (the live gate's wire shape). The start bound of a forward read must be the stored
/// sample before StartTime, at ITS OWN time — never AVEVA's synthetic boundary row, which is stamped at the query start
/// and carries the value in effect there (<c>Quality = 133</c>). The Gateway reads forward only, so "at or before" is
/// served by the walk-back (<see cref="ReverseRawReader"/>), and no reverse window is ever sent.
/// </summary>
public sealed class GatewayBoundingValueTests
{
private static CancellationToken Ct => TestContext.Current.CancellationToken;
private static readonly DateTime T0 = new(2026, 10, 12, 18, 6, 0, DateTimeKind.Utc);
private static readonly DateTime Now = T0.AddHours(1);
/// <summary>The live gate's tag <c>HistGW.LiveTest.Gate681</c> (18:06:40 is a stored NULL).</summary>
private static readonly (double Sec, double Value, bool Null)[] Gate681 =
[
(0, 10, false), (5, 20, false), (20, 30, false), (40, 0, true), (50, 5, false), (60, 1, false), (90, 1, false),
(100, 7, false),
];
private static DateTime At(double seconds) => T0.AddTicks((long)(seconds * TimeSpan.TicksPerSecond));
private static HistorianSample Stored(DateTime t, double v, bool isNull) => new()
{
Tag = "T",
Timestamp = Timestamp.FromDateTime(t),
NumericValue = isNull ? 0 : v,
OpcQuality = isNull ? 0u : 192u,
Quality = isNull ? 1u : 0u,
};
/// <summary>The latest query end any ReadRaw was sent (a far-future end is the #691 review's risk).</summary>
private sealed class EndProbe
{
public DateTime MaxEnd = DateTime.MinValue;
}
/// <summary>
/// ReadRaw over <c>[start, end]</c> ascending, capped, led by AVEVA's boundary row at the query start. A query end more
/// than two days past <see cref="Now"/> is REFUSED (<c>InvalidArgument</c>), modelling the unverified AVEVA behaviour
/// for a year-9999 end (the HistorianGateway only checks <c>start &lt;= end</c>, so it reaches AVEVA unclamped).
/// </summary>
private static FakeHistorianGatewayClient Aveva(EndProbe? probe = null)
{
var sorted = Gate681.Select(x => (T: At(x.Sec), V: x.Value, x.Null)).ToList();
return new FakeHistorianGatewayClient
{
RawSource = (start, end, max) =>
{
if (probe is not null && end > probe.MaxEnd) probe.MaxEnd = end;
if (end > Now.AddDays(2))
throw new Grpc.Core.RpcException(new Grpc.Core.Status(Grpc.Core.StatusCode.InvalidArgument,
"end_time is outside the historian's time range"));
var rows = new List<HistorianSample>();
if (sorted.All(d => d.T != start))
{
var prior = sorted.Where(d => d.T < start).Select(d => ((DateTime T, double V, bool Null)?)d).LastOrDefault();
HistorianSample boundary;
if (prior is { } p)
{
boundary = Stored(start, p.V, p.Null);
if (!p.Null) boundary.Quality = 133;
}
else
{
boundary = new HistorianSample { Tag = "T", Timestamp = Timestamp.FromDateTime(start), NumericValue = 0, OpcQuality = 0, Quality = 1 };
}
rows.Add(boundary);
}
rows.AddRange(sorted.Where(d => d.T >= start && d.T <= end).Select(d => Stored(d.T, d.V, d.Null)));
return max > 0 ? rows.Take(max) : rows;
},
};
}
private static GatewayHistorianDataSource Source(FakeHistorianGatewayClient fake) =>
new(fake, NullLogger<GatewayHistorianDataSource>.Instance);
[Fact]
public async Task The_start_bound_is_the_stored_prior_sample_at_its_own_time_not_the_synthetic_row()
{
var bound = await HistoryBounds.StartBoundAsync(Source(Aveva()), "T", At(21), reverse: false, 65536, Now, Ct);
bound.ShouldNotBeNull();
bound.SourceTimestampUtc.ShouldBe(At(20)); // the synthetic row would be stamped At(21)
bound.Value.ShouldBe(30d);
bound.StatusCode.ShouldBe(0u);
}
[Fact]
public async Task A_stored_NULL_prior_sample_is_the_start_bound_as_the_Bad_sample_it_is()
{
var bound = await HistoryBounds.StartBoundAsync(Source(Aveva()), "T", At(45), reverse: false, 65536, Now, Ct);
bound!.SourceTimestampUtc.ShouldBe(At(40));
bound.StatusCode.ShouldBe(0x80000000u);
}
[Fact]
public async Task Before_the_first_sample_the_start_bound_is_not_found_at_the_start_time()
{
var bound = await HistoryBounds.StartBoundAsync(Source(Aveva()), "T", At(-30), reverse: false, 65536, Now, Ct);
bound!.StatusCode.ShouldBe(Opc.Ua.StatusCodes.BadBoundNotFound);
bound.SourceTimestampUtc.ShouldBe(At(-30));
bound.Value.ShouldBeNull();
}
[Fact]
public async Task A_sample_exactly_on_the_start_time_is_the_bound_and_already_the_data()
{
(await HistoryBounds.StartBoundAsync(Source(Aveva()), "T", At(20), reverse: false, 65536, Now, Ct)).ShouldBeNull();
}
[Fact]
public async Task The_end_bound_is_the_first_stored_sample_at_or_after_the_end_time()
{
var source = Source(Aveva());
var after = await HistoryBounds.EndBoundAsync(source, "T", HistoryBounds.For(At(0), At(41)), At(40), 65536, Now, Ct);
after.SourceTimestampUtc.ShouldBe(At(50));
after.Value.ShouldBe(5d);
var missing = await HistoryBounds.EndBoundAsync(source, "T", HistoryBounds.For(At(0), At(101)), At(100), 65536, Now, Ct);
missing.StatusCode.ShouldBe(Opc.Ua.StatusCodes.BadBoundNotFound);
missing.SourceTimestampUtc.ShouldBe(At(101));
}
/// <summary>
/// #691 review: the at-or-after read never sends an end beyond now + <see cref="ReverseRawReader.LookAhead"/> (the
/// historian stores nothing further ahead), and an end bound wanted beyond that horizon is not found without a read.
/// </summary>
[Fact]
public async Task The_at_or_after_read_never_asks_past_now_plus_one_day()
{
var probe = new EndProbe();
var fake = Aveva(probe);
var source = Source(fake);
var found = await HistoryBounds.EndBoundAsync(source, "T", HistoryBounds.For(At(0), At(41)), At(40), 65536, Now, Ct);
found.SourceTimestampUtc.ShouldBe(At(50));
probe.MaxEnd.ShouldBeLessThanOrEqualTo(Now + ReverseRawReader.LookAhead.Add(TimeSpan.FromMilliseconds(1)));
var calls = fake.ReadRawCallCount;
var beyond = Now.AddDays(3);
var missing = await HistoryBounds.EndBoundAsync(source, "T", HistoryBounds.For(At(0), beyond), At(100), 65536, Now, Ct);
missing.StatusCode.ShouldBe(Opc.Ua.StatusCodes.BadBoundNotFound);
missing.SourceTimestampUtc.ShouldBe(beyond);
fake.ReadRawCallCount.ShouldBe(calls, "a bound beyond the horizon is not found without a read");
source.GetHealthSnapshot().TotalFailures.ShouldBe(0);
}
[Fact]
public async Task A_reverse_read_end_bound_walks_back_to_the_stored_sample_at_or_before_the_end_time()
{
var fake = Aveva();
var bound = await HistoryBounds.EndBoundAsync(Source(fake), "T", HistoryBounds.For(At(100), At(55)), At(60), 65536, Now, Ct);
bound.SourceTimestampUtc.ShouldBe(At(50));
bound.Value.ShouldBe(5d);
}
}
@@ -35,7 +35,7 @@ public sealed class GatewayEventBoundsTests
{
var fake = new FakeHistorianGatewayClient();
IHistorianDataSource ds = new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
ds.SupportsReverseEvents.ShouldBeFalse("the server must keep end-only reads forward on the default backend");
ds.SupportsReverseEvents.ShouldBeFalse("the server must walk reverse reads back from forward reads on the default backend (#691.4)");
await Should.ThrowAsync<NotSupportedException>(() =>
ds.ReadEventsAsync("Pump1", DateTime.UtcNow, DateTime.UtcNow.AddHours(-1), 10, reverse: true, Ct));
@@ -29,12 +29,14 @@ public sealed class GatewayHistorianDataSourceTests
Assert.Equal(100 + RawAggregateReader.PreWindowAllowance, fake.LastReadRawMaxValues);
}
private static readonly DateTime Hour0 = new(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
[Fact]
public async Task ReadProcessed_uses_aggregate_mode_mapping()
{
var fake = new FakeHistorianGatewayClient();
var ds = new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
await ds.ReadProcessedAsync("T", default, default, TimeSpan.FromSeconds(60), HistoryAggregateType.Total, TestContext.Current.CancellationToken);
await ds.ReadProcessedAsync("T", Hour0, Hour0.AddHours(1), TimeSpan.FromSeconds(60), HistoryAggregateType.Total, TestContext.Current.CancellationToken); // an empty window sends nothing
Assert.Equal(RetrievalMode.Integral, fake.LastAggregateMode);
Assert.Equal(TimeSpan.FromSeconds(60), fake.LastAggregateInterval);
}
@@ -62,7 +64,7 @@ public sealed class GatewayHistorianDataSourceTests
{
var fake = new FakeHistorianGatewayClient();
var ds = new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
await ds.ReadProcessedAsync("T", default, default, TimeSpan.FromSeconds(60), HistoryAggregateType.TimeAverage, TestContext.Current.CancellationToken);
await ds.ReadProcessedAsync("T", Hour0, Hour0.AddHours(1), TimeSpan.FromSeconds(60), HistoryAggregateType.TimeAverage, TestContext.Current.CancellationToken); // an empty window sends nothing
Assert.Equal(RetrievalMode.TimeWeightedAverage, fake.LastAggregateMode);
}
@@ -111,6 +113,21 @@ public sealed class GatewayHistorianDataSourceTests
Assert.Equal(0x80000000u, r.Samples[1].StatusCode); // gap → Bad at requested ts
}
[Fact] // #690: an at-time NULL (histsdk: value 0 + Bad) is null + Bad, like the SQL backends' AtTime of a Bad sample
public async Task ReadAtTime_of_a_NULL_row_carries_no_value()
{
var fake = new FakeHistorianGatewayClient();
var t0 = new DateTime(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
fake.AtTimeSamples = new[]
{
new HistorianSample { NumericValue = 0, OpcQuality = 0, Quality = 1, Timestamp = Timestamp.FromDateTime(t0) },
};
var ds = new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
var r = await ds.ReadAtTimeAsync("T", new[] { t0 }, TestContext.Current.CancellationToken);
Assert.Null(r.Samples[0].Value);
Assert.Equal(0x80000000u, r.Samples[0].StatusCode);
}
[Fact]
public async Task Empty_window_is_not_a_fault()
{
@@ -0,0 +1,216 @@
using Google.Protobuf.WellKnownTypes;
using Microsoft.Extensions.Logging.Abstractions;
using Shouldly;
using Xunit;
using ZB.MOM.WW.HistorianGateway.Contracts.Grpc;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests;
/// <summary>
/// #691.2: a forward or reverse processed read of the AVEVA-native <c>TimeAverage</c> / <c>Total</c> on the SQL
/// backends' interval grid. The gateway refuses <c>start &gt; end</c>, so a reverse read is served by FORWARD native
/// requests whose bucket grid is anchored at the request's later time (#677: window <c>(end, start]</c>, intervals
/// stepping back from <c>start</c>, the clipped interval the earliest, each stamped with its later edge, results newest
/// first); a forward read's clipped LATEST interval is read over exactly its span (review). The fake is a native-mode AVEVA
/// model: a step-held stored series; for a forward request <c>[s, e)</c> with interval <c>I</c> it answers one row per
/// interval of the grid anchored at <c>s</c> (plus AVEVA's extra row for the interval before <c>s</c>, #685), each the
/// time-weighted average / integral of exactly that interval — the per-interval behaviour the gateway will have once
/// historiangw#27 is fixed.
/// </summary>
public sealed class GatewayNativeAggregateGridTests
{
private static readonly CancellationToken Ct = CancellationToken.None;
private static readonly DateTime T0 = new(2026, 10, 12, 18, 0, 0, DateTimeKind.Utc);
private const uint GoodCalc = 0x00000401u;
private const uint Partial = 0x4u;
private const uint NoDataCalc = 0x809B0401u;
private static DateTime At(double seconds) => T0.AddTicks((long)(seconds * TimeSpan.TicksPerSecond));
/// <summary>Step-held series: value changes at these offsets (s); data starts at −100 s.</summary>
private static readonly (double T, double V)[] Series = [(-100, 1), (7, 5), (23, 2), (41, 10), (55, 4), (80, 6)];
/// <summary>∫ v dt (value·s) of the step-held series over [a, b); null when no data covers any of it.</summary>
private static double? Integral(DateTime a, DateTime b)
{
double sum = 0;
var covered = false;
for (var i = 0; i < Series.Length; i++)
{
var from = At(Series[i].T);
var to = i + 1 < Series.Length ? At(Series[i + 1].T) : DateTime.MaxValue;
var lo = from > a ? from : a;
var hi = to < b ? to : b;
if (hi <= lo) continue;
covered = true;
sum += Series[i].V * (hi - lo).TotalSeconds;
}
return covered ? sum : null;
}
private static double? Expected(RetrievalMode mode, DateTime a, DateTime b) =>
Integral(a, b) is { } i ? mode == RetrievalMode.Integral ? i : i / (b - a).TotalSeconds : null;
private static FakeHistorianGatewayClient Aveva() => new()
{
AggregateSource = (s, e, mode, interval) =>
{
var rows = new List<HistorianAggregateSample>();
for (var lo = s - interval; lo < e; lo += interval) // the pre-window interval first (#685)
{
var hi = lo + interval;
var v = Expected(mode, lo, hi);
rows.Add(new HistorianAggregateSample
{
Tag = "T",
StartTime = Timestamp.FromDateTime(lo),
EndTime = Timestamp.FromDateTime(hi),
Value = v ?? 0,
OpcQuality = v is null ? 0u : 192u,
RetrievalMode = mode,
});
}
return rows;
},
};
private static GatewayHistorianDataSource Source(FakeHistorianGatewayClient fake) =>
new(fake, NullLogger<GatewayHistorianDataSource>.Instance);
[Theory]
[InlineData(HistoryAggregateType.TimeAverage, RetrievalMode.TimeWeightedAverage)]
[InlineData(HistoryAggregateType.Total, RetrievalMode.Integral)]
public async Task A_reverse_read_over_whole_intervals_is_anchored_at_its_later_time(HistoryAggregateType agg, RetrievalMode mode)
{
var fake = Aveva();
var ds = Source(fake);
var s = (await ds.ReadProcessedAsync("T", At(60), At(0), TimeSpan.FromSeconds(20), agg, Ct)).Samples;
// (0, 60] in three intervals, newest first, each stamped with its LATER edge (#677).
s.Select(x => x.SourceTimestampUtc).ShouldBe(new DateTime?[] { At(60), At(40), At(20) });
s.Select(x => (double)x.Value!).ShouldBe(
[Expected(mode, At(40), At(60))!.Value, Expected(mode, At(20), At(40))!.Value, Expected(mode, At(0), At(20))!.Value],
tolerance: 1e-9);
s.ShouldAllBe(x => x.StatusCode == GoodCalc);
fake.AggregateCalls.ShouldBe([(At(0), At(60), mode, TimeSpan.FromSeconds(20))]);
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
}
[Fact]
public async Task A_reverse_read_with_a_partial_interval_clips_the_EARLIEST_one_and_marks_it_Partial()
{
var fake = Aveva();
var s = (await Source(fake).ReadProcessedAsync("T", At(70), At(5), TimeSpan.FromSeconds(20),
HistoryAggregateType.TimeAverage, Ct)).Samples;
// (5, 70]: (50,70], (30,50], (10,30], and the clipped (5,10] — the grid steps back from 70, not forward from 5.
s.Select(x => x.SourceTimestampUtc).ShouldBe(new DateTime?[] { At(70), At(50), At(30), At(10) });
var m = RetrievalMode.TimeWeightedAverage;
s.Select(x => (double)x.Value!).ShouldBe(
[Expected(m, At(50), At(70))!.Value, Expected(m, At(30), At(50))!.Value, Expected(m, At(10), At(30))!.Value,
Expected(m, At(5), At(10))!.Value],
tolerance: 1e-9);
s.Select(x => x.StatusCode).ShouldBe([GoodCalc, GoodCalc, GoodCalc, GoodCalc | Partial]);
// Two forward requests, neither reversed: the whole intervals on the later-anchored grid, and the clipped one alone
// (its own width as the interval, so AVEVA computes it over exactly (5, 10]).
fake.AggregateCalls.ShouldBe([
(At(10), At(70), m, TimeSpan.FromSeconds(20)),
(At(5), At(10), m, TimeSpan.FromSeconds(5)),
]);
}
[Fact]
public async Task A_reverse_read_narrower_than_one_interval_is_one_interval_stamped_at_its_later_time()
{
var fake = Aveva();
var s = (await Source(fake).ReadProcessedAsync("T", At(30), At(12), TimeSpan.FromSeconds(60),
HistoryAggregateType.Total, Ct)).Samples;
s.Single().SourceTimestampUtc.ShouldBe(At(30));
((double)s.Single().Value!).ShouldBe(Expected(RetrievalMode.Integral, At(12), At(30))!.Value, 1e-9);
s.Single().StatusCode.ShouldBe(GoodCalc); // a whole (clamped) interval, not a truncated one
fake.AggregateCalls.ShouldBe([(At(12), At(30), RetrievalMode.Integral, TimeSpan.FromSeconds(18))]);
}
[Fact]
public async Task A_zero_interval_reverse_read_is_one_interval_over_the_window()
{
var fake = Aveva();
var s = (await Source(fake).ReadProcessedAsync("T", At(30), At(0), TimeSpan.Zero,
HistoryAggregateType.TimeAverage, Ct)).Samples;
s.Single().SourceTimestampUtc.ShouldBe(At(30));
fake.AggregateCalls.Single().Interval.ShouldBe(TimeSpan.FromSeconds(30));
}
[Fact]
public async Task A_no_data_interval_is_BadNoData_without_the_Partial_bit()
{
// Data starts at −100 s: (−110, −90] holds some; (−130, −110] and the clipped earliest (−135, −130] hold none.
var s = (await Source(Aveva()).ReadProcessedAsync("T", At(-90), At(-135), TimeSpan.FromSeconds(20),
HistoryAggregateType.TimeAverage, Ct)).Samples;
s.Select(x => x.SourceTimestampUtc).ShouldBe(new DateTime?[] { At(-90), At(-110), At(-130) });
s[0].StatusCode.ShouldBe(GoodCalc);
s[1].StatusCode.ShouldBe(NoDataCalc);
s[2].StatusCode.ShouldBe(NoDataCalc); // the clipped interval, but Bad ⇒ no Partial bit
s[2].Value.ShouldBeNull();
}
// ---- forward reads on the same grid (#691.2 review): the clipped LATEST interval is its own exact-span read ----------
[Theory]
[InlineData(HistoryAggregateType.TimeAverage, RetrievalMode.TimeWeightedAverage)]
[InlineData(HistoryAggregateType.Total, RetrievalMode.Integral)]
public async Task A_forward_read_with_a_partial_interval_clips_the_LATEST_one_and_marks_it_Partial(HistoryAggregateType agg,
RetrievalMode mode)
{
var fake = Aveva();
var s = (await Source(fake).ReadProcessedAsync("T", At(5), At(70), TimeSpan.FromSeconds(20), agg, Ct)).Samples;
// [5, 70): [5,25), [25,45), [45,65) and the clipped [65,70) — stamped with each EARLIER edge, oldest first, as on
// SqlServer/QuestDB (BucketPlan forward; the truncated interval Partial unless Bad).
s.Select(x => x.SourceTimestampUtc).ShouldBe(new DateTime?[] { At(5), At(25), At(45), At(65) });
s.Select(x => (double)x.Value!).ShouldBe(
[Expected(mode, At(5), At(25))!.Value, Expected(mode, At(25), At(45))!.Value, Expected(mode, At(45), At(65))!.Value,
Expected(mode, At(65), At(70))!.Value],
tolerance: 1e-9);
s.Select(x => x.StatusCode).ShouldBe([GoodCalc, GoodCalc, GoodCalc, GoodCalc | Partial]);
// AVEVA never averages past EndTime: the whole intervals, then the clipped one over exactly its span.
fake.AggregateCalls.ShouldBe([
(At(5), At(65), mode, TimeSpan.FromSeconds(20)),
(At(65), At(70), mode, TimeSpan.FromSeconds(5)),
]);
}
[Fact]
public async Task A_forward_interval_wider_than_the_window_is_sent_as_the_window()
{
var fake = Aveva();
var s = (await Source(fake).ReadProcessedAsync("T", At(12), At(30), TimeSpan.FromSeconds(60),
HistoryAggregateType.Total, Ct)).Samples;
s.Single().SourceTimestampUtc.ShouldBe(At(12));
s.Single().StatusCode.ShouldBe(GoodCalc);
fake.AggregateCalls.ShouldBe([(At(12), At(30), RetrievalMode.Integral, TimeSpan.FromSeconds(18))]);
}
[Fact]
public async Task A_zero_interval_forward_read_is_one_interval_over_the_window_not_a_zero_interval_request()
{
var fake = Aveva();
var s = (await Source(fake).ReadProcessedAsync("T", At(0), At(30), TimeSpan.Zero,
HistoryAggregateType.TimeAverage, Ct)).Samples;
s.Single().SourceTimestampUtc.ShouldBe(At(0));
fake.AggregateCalls.Single().Interval.ShouldBe(TimeSpan.FromSeconds(30));
}
[Fact]
public async Task The_forward_read_is_unchanged()
{
var fake = Aveva();
var s = (await Source(fake).ReadProcessedAsync("T", At(0), At(60), TimeSpan.FromSeconds(20),
HistoryAggregateType.TimeAverage, Ct)).Samples;
s.Select(x => x.SourceTimestampUtc).ShouldBe(new DateTime?[] { At(0), At(20), At(40) });
fake.AggregateCalls.ShouldBe([(At(0), At(60), RetrievalMode.TimeWeightedAverage, TimeSpan.FromSeconds(20))]);
}
}
@@ -106,9 +106,11 @@ public sealed class GatewayNativeAggregateRowsTests
public async Task The_window_filter_is_direction_agnostic()
{
// A reverse request (start > end) covers the same span (end, start]; a row for the interval before the span is out.
// #691.2: served by a forward request on the later-anchored grid, newest first, stamped with each later edge.
var wire = new[] { Row(T0 - I20, T0, 9, 192), Row(T0, T0 + I20, 1, 192), Row(T0 + I20, T0 + 2 * I20, 2, 192) };
var s = await Read(wire, T0 + 2 * I20, T0);
s.Select(x => x.Value).ShouldBe(new object?[] { 1d, 2d });
s.Select(x => x.Value).ShouldBe(new object?[] { 2d, 1d });
s.Select(x => x.SourceTimestampUtc).ShouldBe(new DateTime?[] { T0 + 2 * I20, T0 + I20 });
}
// ---- D2: an Uncertain row keeps its value ------------------------------------------------------------------------
@@ -109,6 +109,15 @@ public sealed class GatewayRawBoundaryRowTests
r.Samples[0].StatusCode.ShouldBe(0x80000000u);
}
[Fact] // #690: the wire's value 0 on a NULL row is not a value — SqlServer/QuestDB return null + Bad for the same sample
public async Task A_stored_NULL_sample_carries_no_value_not_the_wires_zero()
{
var r = await Source(Gate()).ReadRawAsync("T", At(30), At(55), 0, Ct);
r.Samples[0].SourceTimestampUtc.ShouldBe(At(40));
r.Samples[0].Value.ShouldBeNull();
r.Samples[1].Value.ShouldBe(5d); // a Good neighbour keeps its value
}
[Fact]
public async Task A_stored_sample_exactly_at_the_window_start_is_kept()
{
@@ -0,0 +1,76 @@
using Google.Protobuf.WellKnownTypes;
using Microsoft.Extensions.Logging.Abstractions;
using Shouldly;
using Xunit;
using ZB.MOM.WW.HistorianGateway.Contracts.Grpc;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests;
/// <summary>
/// Gitea #691.4 review: events read uncapped (and walked back newest-first) through the REAL
/// <see cref="GatewayHistorianDataSource"/> over a gateway-shaped fake — inclusive <c>[start, end]</c>, ordered by event time.
/// </summary>
public sealed class GatewayReverseEventsTests
{
private static CancellationToken Ct => TestContext.Current.CancellationToken;
private static readonly DateTime T0 = new(2026, 10, 12, 18, 0, 0, DateTimeKind.Utc);
private static FakeHistorianGatewayClient Store(int count)
{
var stored = Enumerable.Range(0, count)
.Select(i => new HistorianEvent { Id = $"e{i:0000}", Type = "Alarm", EventTime = Timestamp.FromDateTime(T0.AddSeconds(i)) })
.ToList();
return new FakeHistorianGatewayClient
{
EventsSource = (start, end) => stored.Where(e => e.EventTime.ToDateTime() >= start && e.EventTime.ToDateTime() <= end),
};
}
/// <summary>
/// <c>NumValuesPerNode = 0</c> reaches the source as <c>int.MaxValue</c>; the result list used to be pre-sized to it
/// ("Array dimensions exceeded") — every uncapped Gateway events read failed and counted as a backend failure.
/// </summary>
[Fact]
public async Task An_uncapped_forward_events_read_is_served()
{
var fake = Store(30);
var ds = new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
var r = await ds.ReadEventsAsync(null, T0, T0.AddMinutes(1), int.MaxValue, Ct);
r.Events.Count.ShouldBe(30);
r.ContinuationPoint.ShouldBeNull();
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
}
[Fact]
public async Task An_uncapped_reverse_events_read_is_one_forward_read_newest_first()
{
var fake = Store(30);
var ds = new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
var page = await ReverseEventsReader.ReadAsync(ds, null, hi: T0.AddSeconds(25), lo: T0.AddSeconds(5), want: 0,
maxTieCluster: 65536, nowUtc: T0.AddHours(1), hint: default, Ct);
page.Events.Select(e => e.EventId).ShouldBe(Enumerable.Range(6, 20).Reverse().Select(i => $"e{i:0000}"));
fake.ReadEventsCallCount.ShouldBe(1);
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
}
[Fact]
public async Task A_capped_reverse_events_read_walked_back_over_the_gateway_returns_the_latest_newest_first()
{
var fake = Store(300);
var ds = new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
var page = await ReverseEventsReader.ReadAsync(ds, null, hi: T0.AddSeconds(200), lo: T0.AddSeconds(50), want: 10,
maxTieCluster: 65536, nowUtc: T0.AddHours(1), hint: default, Ct);
page.Events.Select(e => e.EventId).ShouldBe(Enumerable.Range(191, 10).Reverse().Select(i => $"e{i:0000}"));
fake.LastReadEventsStartUtc.ShouldBeLessThanOrEqualTo(fake.LastReadEventsEndUtc);
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
}
}
@@ -0,0 +1,290 @@
using Google.Protobuf.WellKnownTypes;
using Microsoft.Extensions.Logging.Abstractions;
using Shouldly;
using Xunit;
using ZB.MOM.WW.HistorianGateway.Contracts.Grpc;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests;
/// <summary>
/// #691.3: a gateway with <c>Historian:ReadLimits:AllowStreamingBeyondCeiling=false</c> refuses any <c>ReadRaw</c> whose
/// <c>max_values</c> exceeds its <c>MaxValuesCeiling</c> (before streaming, naming the ceiling). An uncapped raw read
/// (<c>NumValuesPerNode = 0</c>, sent as <see cref="int.MaxValue"/>) used to be refused outright — even a 10-row window.
/// The read is now re-issued once with the ceiling the refusal named, and a stream that delivered the whole ceiling
/// without reaching the window end is never answered from (<c>BadTooManyOperations</c>). The fake is AVEVA-shaped:
/// a boundary row at the query start, <c>[start, end]</c> inclusive, capped at <c>max</c>.
/// </summary>
public sealed class GatewayStrictCeilingTests
{
private static readonly CancellationToken Ct = CancellationToken.None;
private static readonly DateTime T0 = new(2026, 10, 12, 19, 0, 0, DateTimeKind.Utc);
private static readonly DateTime Now = T0.AddDays(1);
private static DateTime At(int seconds) => T0.AddSeconds(seconds);
private static HistorianSample Row(DateTime t, double v, uint quality = 0) =>
new() { Tag = "T", Timestamp = Timestamp.FromDateTime(t), NumericValue = v, OpcQuality = 192, Quality = quality };
/// <summary>One stored sample per second, value = seconds, at +0 … +(count − 1); strict ceiling when given.</summary>
private static FakeHistorianGatewayClient Store(int count, int? ceiling)
{
var stored = Enumerable.Range(0, count).Select(i => (T: At(i), V: (double)i)).ToList();
return new FakeHistorianGatewayClient
{
MaxValuesCeiling = ceiling,
RawSource = (start, end, max) =>
{
var rows = new List<HistorianSample>();
if (stored.All(s => s.T != start))
{
var prior = stored.LastOrDefault(s => s.T < start);
rows.Add(prior.T == default
? new HistorianSample { Tag = "T", Timestamp = Timestamp.FromDateTime(start), NumericValue = 0, OpcQuality = 0, Quality = 1 }
: Row(start, prior.V, 133));
}
rows.AddRange(stored.Where(s => s.T >= start && s.T <= end).Select(s => Row(s.T, s.V)));
return rows.Take(max);
},
};
}
private static GatewayHistorianDataSource Source(FakeHistorianGatewayClient fake, int maxRawRows = 1_000_000) =>
new(fake, NullLogger<GatewayHistorianDataSource>.Instance, maxRawRows, new FixedClock(Now));
private sealed class FixedClock(DateTime now) : TimeProvider
{
public override DateTimeOffset GetUtcNow() => new(now);
}
// ---- raw, uncapped ---------------------------------------------------------------------------------------------------
[Fact]
public async Task An_uncapped_read_of_a_small_window_on_a_strict_gateway_is_served_with_the_ceiling_as_budget()
{
var fake = Store(10, ceiling: 1000);
var ds = Source(fake);
var r = await ds.ReadRawAsync("T", At(0), At(60), 0, Ct);
r.Samples.Select(s => s.Value).ShouldBe(Enumerable.Range(0, 10).Select(i => (object?)(double)i));
fake.RawCalls.Select(c => c.MaxValues).ShouldBe([int.MaxValue, 1000]); // refused, then the ceiling it named
fake.RawStreamsCompleted.ShouldBe(1);
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
}
[Fact]
public async Task An_uncapped_read_on_a_default_gateway_still_sends_one_unbounded_request()
{
var fake = Store(10, ceiling: null);
await Source(fake).ReadRawAsync("T", At(0), At(60), 0, Ct);
fake.RawCalls.Select(c => c.MaxValues).ShouldBe([int.MaxValue]);
}
[Fact]
public async Task An_uncapped_window_holding_more_than_the_ceiling_is_a_request_limit_never_a_truncated_answer()
{
var fake = Store(100, ceiling: 50);
var ds = Source(fake);
await Should.ThrowAsync<HistoryReadLimitExceededException>(() => ds.ReadRawAsync("T", At(0), At(200), 0, Ct));
fake.RawStreamsCompleted.ShouldBe(1); // drained, never abandoned
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
}
[Fact]
public async Task A_ceiling_full_stream_that_reached_the_window_end_is_complete()
{
// 50 rows: the boundary row + +0 … +48; the window end is +48 (the gateway's end is inclusive), so the stream
// provably ended at the window end — complete although it delivered the whole ceiling.
var fake = Store(100, ceiling: 50);
var r = await Source(fake).ReadRawAsync("T", At(0), At(48), 0, Ct);
r.Samples.Count.ShouldBe(48); // [+0, +48): the end is exclusive
}
[Fact]
public async Task A_refusal_without_a_parseable_ceiling_retries_at_the_gateways_default_ceiling()
{
var fake = Store(10, ceiling: 1_000_000);
fake.CeilingRejectionDetail = "max_values exceeds the server ceiling; read large ranges by time-slicing.";
var r = await Source(fake).ReadRawAsync("T", At(0), At(60), 0, Ct);
r.Samples.Count.ShouldBe(10);
fake.RawCalls.Select(c => c.MaxValues).ShouldBe([int.MaxValue, RawAggregateReader.GatewayDefaultMaxValuesCeiling]);
}
// ---- the learned ceiling (review: a refused attempt counts as an error in the gateway's own metrics) -----------------
private static int Refusals(FakeHistorianGatewayClient fake) =>
fake.RawCalls.Count(c => fake.MaxValuesCeiling is { } n && c.MaxValues > n);
[Fact]
public async Task The_ceiling_is_learned_once_per_data_source_so_later_reads_are_not_refused()
{
var fake = Store(10, ceiling: 1000);
var ds = Source(fake);
await ds.ReadRawAsync("T", At(0), At(60), 0, Ct);
await ds.ReadRawAsync("T", At(0), At(60), 0, Ct);
await ds.ReadRawAsync("T", At(0), At(60), 995, Ct);
await ds.ReadProcessedAsync("T", At(0), At(10), TimeSpan.FromSeconds(10), HistoryAggregateType.Count, Ct);
fake.RawCalls.Select(c => c.MaxValues).Take(4).ShouldBe([int.MaxValue, 1000, 1000, 1000]);
fake.RawCalls[4].MaxValues.ShouldBe(1000); // the per-sample read shares it (its own budget 1,000,000)
Refusals(fake).ShouldBe(1);
}
[Fact]
public async Task A_gateway_that_now_streams_is_re_probed_when_a_learned_ceiling_read_comes_back_full()
{
var fake = Store(100, ceiling: 50);
var ds = Source(fake);
await ds.ReadRawAsync("T", At(0), At(5), 0, Ct); // learns 50
fake.MaxValuesCeiling = null; // the operator enabled AllowStreamingBeyondCeiling
var r = await ds.ReadRawAsync("T", At(0), At(200), 0, Ct);
r.Samples.Count.ShouldBe(100); // not BadTooManyOperations from a stale ceiling
fake.RawCalls.Skip(2).Select(c => c.MaxValues).ShouldBe([50, int.MaxValue]);
await ds.ReadRawAsync("T", At(0), At(5), 0, Ct);
fake.RawCalls[^1].MaxValues.ShouldBe(int.MaxValue); // forgotten
}
[Fact]
public async Task A_raised_but_still_strict_ceiling_is_re_learned_from_the_re_probe()
{
var fake = Store(100, ceiling: 50);
var ds = Source(fake);
await ds.ReadRawAsync("T", At(0), At(5), 0, Ct);
fake.MaxValuesCeiling = 500;
(await ds.ReadRawAsync("T", At(0), At(200), 0, Ct)).Samples.Count.ShouldBe(100);
fake.RawCalls.Skip(2).Select(c => c.MaxValues).ShouldBe([50, int.MaxValue, 500]);
await ds.ReadRawAsync("T", At(0), At(5), 0, Ct);
fake.RawCalls[^1].MaxValues.ShouldBe(500);
}
[Fact]
public async Task A_genuinely_over_ceiling_read_costs_one_re_probe_and_no_repeat_at_the_same_ceiling()
{
var fake = Store(100, ceiling: 50);
var ds = Source(fake);
await ds.ReadRawAsync("T", At(0), At(5), 0, Ct);
await Should.ThrowAsync<HistoryReadLimitExceededException>(() => ds.ReadRawAsync("T", At(0), At(200), 0, Ct));
fake.RawCalls.Skip(2).Select(c => c.MaxValues).ShouldBe([50, int.MaxValue]);
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
}
[Fact]
public async Task A_lowered_ceiling_is_re_learned_from_the_refusal_of_the_learned_one()
{
var fake = Store(10, ceiling: 1000);
var ds = Source(fake);
await ds.ReadRawAsync("T", At(0), At(60), 0, Ct);
fake.MaxValuesCeiling = 50;
(await ds.ReadRawAsync("T", At(0), At(60), 0, Ct)).Samples.Count.ShouldBe(10);
await ds.ReadRawAsync("T", At(0), At(60), 0, Ct);
fake.RawCalls.Skip(2).Select(c => c.MaxValues).ShouldBe([1000, 50, 50]);
}
[Fact]
public async Task Concurrent_reads_share_the_learned_ceiling_safely()
{
var fake = Store(10, ceiling: 1000);
var ds = Source(fake);
await ds.ReadRawAsync("T", At(0), At(60), 0, Ct);
var results = await Task.WhenAll(Enumerable.Range(0, 32).Select(_ => Task.Run(() => ds.ReadRawAsync("T", At(0), At(60), 0, Ct))));
results.ShouldAllBe(r => r.Samples.Count == 10);
Refusals(fake).ShouldBe(1);
}
// ---- raw, capped -----------------------------------------------------------------------------------------------------
[Fact]
public async Task A_cap_whose_allowance_exceeds_the_ceiling_is_served_with_the_ceiling_as_budget()
{
// cap 995 + 16 = 1011 > 1000: refused, re-asked with 1000 — a small window is served in full.
var fake = Store(10, ceiling: 1000);
var r = await Source(fake).ReadRawAsync("T", At(0), At(60), 995, Ct);
r.Samples.Count.ShouldBe(10);
fake.RawCalls.Select(c => c.MaxValues).ShouldBe([995 + RawAggregateReader.PreWindowAllowance, 1000]);
}
[Fact]
public async Task A_cap_above_the_ceiling_over_a_dense_window_is_a_request_limit_not_a_short_page()
{
// A short page would end the client's paging early (silent truncation), so it must fail instead.
var fake = Store(100, ceiling: 50);
var ds = Source(fake);
await Should.ThrowAsync<HistoryReadLimitExceededException>(() => ds.ReadRawAsync("T", At(0), At(200), 80, Ct));
fake.RawCalls.Count.ShouldBe(2); // the ceiling re-read is final: no third read at the same ceiling
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
}
[Fact]
public async Task A_cap_that_fits_the_ceiling_is_unaffected()
{
var fake = Store(100, ceiling: 50);
var r = await Source(fake).ReadRawAsync("T", At(0), At(200), 10, Ct);
r.Samples.Count.ShouldBe(10);
fake.RawCalls.Select(c => c.MaxValues).ShouldBe([10 + RawAggregateReader.PreWindowAllowance]);
}
// ---- per-sample aggregates (#688 clamp vs a ceiling below 1,000,000) ---------------------------------------------------
[Fact]
public async Task A_per_sample_aggregate_on_a_gateway_with_a_lower_ceiling_is_served_when_the_window_fits()
{
var fake = Store(30, ceiling: 1000); // default cap 1,000,000 ⇒ budget 1,000,000 > 1000
var s = (await Source(fake).ReadProcessedAsync("T", At(0), At(30), TimeSpan.FromSeconds(10),
HistoryAggregateType.Count, Ct)).Samples;
s.Select(x => x.Value).ShouldBe([10, 10, 10]);
fake.RawCalls[0].MaxValues.ShouldBe(RawAggregateReader.GatewayDefaultMaxValuesCeiling);
fake.RawCalls[1].MaxValues.ShouldBe(1000);
}
[Fact]
public async Task A_per_sample_aggregate_whose_window_fills_the_lower_ceiling_is_a_request_limit()
{
var fake = Store(100, ceiling: 50);
var ds = Source(fake);
await Should.ThrowAsync<HistoryReadLimitExceededException>(() => ds.ReadProcessedAsync("T", At(0), At(100),
TimeSpan.FromSeconds(10), HistoryAggregateType.Count, Ct));
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
}
[Fact]
public async Task An_unbounded_per_sample_aggregate_on_a_strict_gateway_is_served_with_the_ceiling()
{
var fake = Store(30, ceiling: 1000);
var s = (await Source(fake, maxRawRows: 0).ReadProcessedAsync("T", At(0), At(30), TimeSpan.FromSeconds(10),
HistoryAggregateType.Count, Ct)).Samples;
s.Select(x => x.Value).ShouldBe([10, 10, 10]);
fake.RawCalls.Take(2).Select(c => c.MaxValues).ShouldBe([int.MaxValue, 1000]);
}
[Fact]
public async Task An_unbounded_per_sample_aggregate_filling_the_ceiling_is_a_request_limit()
{
var fake = Store(100, ceiling: 50);
await Should.ThrowAsync<HistoryReadLimitExceededException>(() => Source(fake, maxRawRows: 0).ReadProcessedAsync("T",
At(0), At(100), TimeSpan.FromSeconds(10), HistoryAggregateType.Count, Ct));
}
[Theory] // review: the message names the limit that actually bound the read — the gateway's ceiling, not the cap
[InlineData(0)] // unbounded cap: used to say "more than 0 raw samples (MaxTimeWeightedRawRows)"
[InlineData(1_000_000)] // default cap: used to cite 1,000,000
public async Task A_ceiling_bound_per_sample_read_names_the_gateway_ceiling(int maxRawRows)
{
var ex = await Should.ThrowAsync<HistoryReadLimitExceededException>(() => Source(Store(100, ceiling: 50), maxRawRows)
.ReadProcessedAsync("T", At(0), At(100), TimeSpan.FromSeconds(10), HistoryAggregateType.Count, Ct));
ex.Message.ShouldContain("row ceiling (50,");
ex.Message.ShouldNotContain("MaxTimeWeightedRawRows); the Gateway");
}
[Fact]
public async Task A_cap_bound_per_sample_read_still_names_the_cap()
{
var ex = await Should.ThrowAsync<HistoryReadLimitExceededException>(() => Source(Store(100, ceiling: null), 20)
.ReadProcessedAsync("T", At(0), At(100), TimeSpan.FromSeconds(10), HistoryAggregateType.Count, Ct));
ex.Message.ShouldContain("more than 20 raw samples");
}
}
@@ -32,6 +32,31 @@ public sealed class SampleMapperTests
Assert.Equal(0x80000000u, SampleMapper.ToSnapshot(s).StatusCode);
}
[Theory] // #690: histsdk puts a value on every row, so a NULL row arrives as 0 / "" with a Bad quality
[InlineData(0u, 1u)] // the live NULL row: opc_quality 0, AVEVA quality 1
[InlineData(24u, 0u)] // any Bad OPC quality (out of service)
public void A_Bad_numeric_row_carries_no_value(uint opc, uint quality)
{
var s = new HistorianSample { Tag = "T", NumericValue = 0, OpcQuality = opc, Quality = quality, Timestamp = Ts(2026, 1, 1, 0, 0, 0) };
var snap = SampleMapper.ToSnapshot(s);
Assert.True(snap.StatusCode >= 0x80000000u);
Assert.Null(snap.Value);
}
[Fact] // #690: a string tag's NULL row arrives as "" + Bad
public void A_Bad_string_row_carries_no_value()
{
var s = new HistorianSample { Tag = "T", StringValue = "", OpcQuality = 0, Quality = 1, Timestamp = Ts(2026, 1, 1, 0, 0, 0) };
Assert.Null(SampleMapper.ToSnapshot(s).Value);
}
[Fact] // #690: an Uncertain row keeps its value (only Bad means "no value")
public void An_Uncertain_row_keeps_its_value()
{
var s = new HistorianSample { Tag = "T", NumericValue = 3, OpcQuality = 64, Timestamp = Ts(2026, 1, 1, 0, 0, 0) };
Assert.Equal(3d, SampleMapper.ToSnapshot(s).Value);
}
[Fact]
public void Aggregate_null_value_is_BadNoData()
{
@@ -67,3 +67,10 @@ public sealed class QuestDbEventReadConformance(QuestDbFixture fx) : EventReadCo
/// <inheritdoc/>
protected override Task<IConformanceBackend> CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx);
}
[Collection(QuestDbCollection.Name)]
public sealed class QuestDbBoundingValueConformance(QuestDbFixture fx) : BoundingValueConformance
{
/// <inheritdoc/>
protected override Task<IConformanceBackend> CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx);
}
@@ -67,3 +67,10 @@ public sealed class SqlServerEventReadConformance(SqlHistorianFixture fx) : Even
/// <inheritdoc/>
protected override Task<IConformanceBackend> CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx);
}
[Collection(SqlHistorianCollection.Name)]
public sealed class SqlServerBoundingValueConformance(SqlHistorianFixture fx) : BoundingValueConformance
{
/// <inheritdoc/>
protected override Task<IConformanceBackend> CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx);
}
@@ -72,6 +72,18 @@ public sealed class OtOpcUaServerHostedServiceHistoryUpdateTests
Service(view: null).BuildHistoryUpdatePosture().MaxFutureSkew.ShouldBe(TimeSpan.FromDays(1));
}
/// <summary>#691: the ReturnBounds kill switch is bound from <c>ServerHistorian:HonorReturnBounds</c>, default on.</summary>
[Fact]
public void The_return_bounds_kill_switch_comes_from_options_and_defaults_on()
{
Service(view: null).HonorReturnBounds.ShouldBeTrue();
var config = new ConfigurationBuilder().AddInMemoryCollection(new Dictionary<string, string?>
{
["ServerHistorian:HonorReturnBounds"] = "false",
}).Build();
Service(view: null, config: config).HonorReturnBounds.ShouldBeFalse();
}
private static OtOpcUaServerHostedService Service(
IHistoryUpdateRoleView? view, IDriverPrimaryRoleSource? primary = null, IConfiguration? config = null,
TimeProvider? clock = null) => new(
@@ -97,6 +97,7 @@ public sealed class HistoryUpdateOverTheWireTests
var details = new ReadRawModifiedDetails
{
ReturnBounds = false,
StartTime = T1.AddHours(-1), EndTime = T1.AddHours(1), NumValuesPerNode = 100, IsReadModified = true,
};
var response = await session.HistoryReadAsync(
@@ -384,6 +384,7 @@ public sealed class NodeManagerAclEnforcementTests : IDisposable
context,
new ReadRawModifiedDetails
{
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow,
NumValuesPerNode = 10,
@@ -0,0 +1,523 @@
using Microsoft.Extensions.Logging.Abstractions;
using Opc.Ua;
using Opc.Ua.Server;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using HistorianRead = ZB.MOM.WW.OtOpcUa.Core.Abstractions.HistoryReadResult;
using SdkHistoryReadResult = Opc.Ua.HistoryReadResult;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests;
/// <summary>Boots ONE real SDK server for the whole bounds suite (the cases swap the historian source per read).</summary>
public sealed class BoundsServerFixture : IAsyncLifetime
{
private readonly string _pkiRoot = Path.Combine(Path.GetTempPath(), $"otopcua-bounds-{Guid.NewGuid():N}");
private OpcUaApplicationHost? _host;
/// <summary>The node manager under test.</summary>
public OtOpcUaNodeManager Nm { get; private set; } = null!;
/// <summary>The historized variable every case reads.</summary>
public NodeId NodeId { get; private set; } = null!;
/// <summary>Serialises the cases (they share the node manager's source).</summary>
public SemaphoreSlim Gate { get; } = new(1, 1);
public async ValueTask InitializeAsync()
{
var started = await OpcUaHostStart.StartAsync(
port => new OpcUaApplicationHost(
new OpcUaApplicationHostOptions
{
ApplicationName = "OtOpcUa.BoundsTest",
ApplicationUri = $"urn:OtOpcUa.BoundsTest:{Guid.NewGuid():N}",
OpcUaPort = port,
PublicHostname = "localhost",
PkiStoreRoot = _pkiRoot,
},
NullLogger<OpcUaApplicationHost>.Instance),
() => new OtOpcUaSdkServer(), CancellationToken.None);
_host = started.Host;
Nm = started.Server.NodeManager!;
Nm.HistoryContinuationStore = new InMemoryHistoryContinuationStore();
Nm.EnsureVariable("eq-1/bounds", parentFolderNodeId: null, displayName: "Bounds", dataType: "Double",
writable: false, historianTagname: "WW.Bounds", realm: AddressSpaceRealm.Uns);
NodeId = Nm.TryGetVariable("eq-1/bounds")!.NodeId;
}
public async ValueTask DisposeAsync()
{
if (_host is not null) await _host.DisposeAsync();
if (Directory.Exists(_pkiRoot))
{
try { Directory.Delete(_pkiRoot, recursive: true); }
catch { /* best-effort */ }
}
}
}
/// <summary>
/// Gitea #691: HistoryRead-Raw <c>ReturnBounds</c> (OPC UA Part 11 §4.6 bounding values, §6.5.3.2). The oracle is Part 11
/// Table 1 itself — every row, over its data set (samples at 5:00, 5:02, 5:03, 5:05, 5:06) — run against three backend
/// shapes: one that reads newest-first natively (SqlServer-like), one that reads forward only (Gateway-like, walked back
/// by <see cref="ReverseRawReader"/>) and a microsecond one (QuestDB-like,
/// truncating its window bounds).
/// </summary>
public sealed class NodeManagerHistoryReadBoundsTests(BoundsServerFixture fx) : IClassFixture<BoundsServerFixture>
{
private static readonly DateTime Day = new(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
private static readonly DateTime Unspecified = DateTime.MinValue;
private static DateTime T(string hhmm)
{
var parts = hhmm.Split(':');
return Day.AddHours(int.Parse(parts[0])).AddMinutes(int.Parse(parts[1]));
}
/// <summary>Part 11 Table 1's data: value = minutes past 5:00.</summary>
private static DataValueSnapshot[] TableData() =>
new[] { "5:00", "5:02", "5:03", "5:05", "5:06" }
.Select(s => new DataValueSnapshot((double)(T(s) - T("5:00")).TotalMinutes, StatusCodes.Good, T(s), T(s)))
.ToArray();
public enum Shape { Reverse, ForwardOnly, Microsecond }
private static IHistorianDataSource Source(Shape shape, IReadOnlyList<DataValueSnapshot> series) => shape switch
{
Shape.Reverse => new SeriesSource(series, reverse: true, inclusiveForwardEnd: false, resolution: 1),
Shape.ForwardOnly => new SeriesSource(series, reverse: false, inclusiveForwardEnd: false, resolution: 1),
_ => new SeriesSource(series, reverse: true, inclusiveForwardEnd: false, resolution: 10),
};
/// <summary>
/// Part 11 Table 1, every row, verbatim. "FIRST"/"LAST" are <c>Bad_BoundNotFound</c> placeholders (null value) at the
/// StartTime/EndTime; a/b footnotes: one second after/before the previous returned value when that side is unspecified.
/// </summary>
public static TheoryData<string, string, uint, bool, string> Table1 => new()
{
{ "5:00", "5:05", 0, true, "5:00, 5:02, 5:03, 5:05" },
{ "5:00", "5:05", 0, false, "5:00, 5:02, 5:03" },
{ "5:01", "5:04", 0, true, "5:00, 5:02, 5:03, 5:05" },
{ "5:01", "5:04", 0, false, "5:02, 5:03" },
{ "5:05", "5:00", 0, true, "5:05, 5:03, 5:02, 5:00" },
{ "5:05", "5:00", 0, false, "5:05, 5:03, 5:02" },
{ "5:04", "5:01", 0, true, "5:05, 5:03, 5:02, 5:00" },
{ "5:04", "5:01", 0, false, "5:03, 5:02" },
{ "4:59", "5:05", 0, true, "FIRST, 5:00, 5:02, 5:03, 5:05" },
{ "4:59", "5:05", 0, false, "5:00, 5:02, 5:03" },
{ "5:01", "5:07", 0, true, "5:00, 5:02, 5:03, 5:05, 5:06, LAST" },
{ "5:01", "5:07", 0, false, "5:02, 5:03, 5:05, 5:06" },
{ "5:00", "5:05", 3, true, "5:00, 5:02, 5:03" },
{ "5:00", "5:05", 3, false, "5:00, 5:02, 5:03" },
{ "5:01", "5:04", 3, true, "5:00, 5:02, 5:03" },
{ "5:01", "5:04", 3, false, "5:02, 5:03" },
{ "5:05", "5:00", 3, true, "5:05, 5:03, 5:02" },
{ "5:05", "5:00", 3, false, "5:05, 5:03, 5:02" },
{ "5:04", "5:01", 3, true, "5:05, 5:03, 5:02" },
{ "5:04", "5:01", 3, false, "5:03, 5:02" },
{ "4:59", "5:05", 3, true, "FIRST, 5:00, 5:02" },
{ "4:59", "5:05", 3, false, "5:00, 5:02, 5:03" },
{ "5:01", "5:07", 3, true, "5:00, 5:02, 5:03" },
{ "5:01", "5:07", 3, false, "5:02, 5:03, 5:05" },
{ "5:00", "", 3, true, "5:00, 5:02, 5:03" },
{ "5:00", "", 3, false, "5:00, 5:02, 5:03" },
{ "5:00", "", 6, true, "5:00, 5:02, 5:03, 5:05, 5:06, LAST+" },
{ "5:00", "", 6, false, "5:00, 5:02, 5:03, 5:05, 5:06" },
{ "5:07", "", 6, true, "5:06, LAST+" },
{ "5:07", "", 6, false, "" },
{ "", "5:06", 3, true, "5:06, 5:05, 5:03" },
{ "", "5:06", 3, false, "5:06, 5:05, 5:03" },
{ "", "5:06", 6, true, "5:06, 5:05, 5:03, 5:02, 5:00, FIRST-" },
{ "", "5:06", 6, false, "5:06, 5:05, 5:03, 5:02, 5:00" },
{ "", "4:48", 6, true, "5:00, FIRST-" },
{ "", "4:48", 6, false, "" },
{ "4:48", "4:48", 0, true, "FIRST, 5:00" },
{ "4:48", "4:48", 0, false, "" },
{ "4:48", "4:48", 1, true, "FIRST" },
{ "4:48", "4:48", 1, false, "" },
{ "4:48", "4:48", 2, true, "FIRST, 5:00" },
{ "5:00", "5:00", 0, true, "5:00, 5:02" },
{ "5:00", "5:00", 0, false, "5:00" },
{ "5:00", "5:00", 1, true, "5:00" },
{ "5:00", "5:00", 1, false, "5:00" },
{ "5:01", "5:01", 0, true, "5:00, 5:02" },
{ "5:01", "5:01", 0, false, "" },
{ "5:01", "5:01", 1, true, "5:00" },
{ "5:01", "5:01", 1, false, "" },
};
[Theory]
[MemberData(nameof(Table1))]
public async Task Part11_table1_first_page_on_a_reverse_capable_backend(string start, string end, uint max, bool bounds, string expected) =>
await AssertFirstPage(Shape.Reverse, start, end, max, bounds, expected);
[Theory]
[MemberData(nameof(Table1))]
public async Task Part11_table1_first_page_on_a_forward_only_backend(string start, string end, uint max, bool bounds, string expected) =>
await AssertFirstPage(Shape.ForwardOnly, start, end, max, bounds, expected);
[Theory]
[MemberData(nameof(Table1))]
public async Task Part11_table1_first_page_on_a_microsecond_backend(string start, string end, uint max, bool bounds, string expected) =>
await AssertFirstPage(Shape.Microsecond, start, end, max, bounds, expected);
private async Task AssertFirstPage(Shape shape, string start, string end, uint max, bool bounds, string expected)
{
var s = start.Length == 0 ? Unspecified : T(start);
var e = end.Length == 0 ? Unspecified : T(end);
var source = Source(shape, TableData());
var (values, error, _) = await ReadAsync(source, s, e, max, bounds, cp: null);
error.StatusCode.Code.ShouldBe(StatusCodes.Good);
Render(values, s, e).ShouldBe(expected);
(source as SeriesSource)?.ReverseWindowsOnForwardOnly.ShouldBe(0);
}
/// <summary>
/// Part 11: bounds count toward NumValuesPerNode, the start bound comes with the first page and the end bound with the
/// last — so paging a bounded two-sided read at any page size concatenates to the unpaged (N = 0) answer.
/// </summary>
[Theory]
[InlineData(Shape.Reverse)]
[InlineData(Shape.ForwardOnly)]
[InlineData(Shape.Microsecond)]
public async Task Paging_a_bounded_read_concatenates_to_the_unpaged_answer(Shape shape)
{
var windows = new[]
{
("5:00", "5:05"), ("5:01", "5:04"), ("5:05", "5:00"), ("5:04", "5:01"), ("4:59", "5:05"), ("5:01", "5:07"),
("4:48", "4:48"), ("5:00", "5:00"), ("5:01", "5:01"), ("5:07", "4:00"), ("5:06", "5:06"),
};
foreach (var (start, end) in windows)
{
var source = Source(shape, TableData());
var all = await ReadAsync(source, T(start), T(end), 0, true, cp: null);
all.Cp.ShouldBeNull();
var expected = Render(all.Values, T(start), T(end));
for (uint n = 1; n <= 6; n++)
{
var paged = await DrainAsync(source, T(start), T(end), n, bounds: true);
Render(paged, T(start), T(end)).ShouldBe(expected, $"{start}->{end} paged by {n}");
}
}
}
/// <summary>One-sided reads page to the open placeholder exactly once (footnotes a/b), never past it.</summary>
[Theory]
[InlineData(Shape.Reverse)]
[InlineData(Shape.ForwardOnly)]
[InlineData(Shape.Microsecond)]
public async Task Paging_a_one_sided_bounded_read_ends_with_the_open_placeholder(Shape shape)
{
for (uint n = 1; n <= 7; n++)
{
var fwd = await DrainAsync(Source(shape, TableData()), T("5:01"), Unspecified, n, bounds: true);
Render(fwd, T("5:01"), Unspecified).ShouldBe("5:00, 5:02, 5:03, 5:05, 5:06, LAST+", $"forward by {n}");
var rev = await DrainAsync(Source(shape, TableData()), Unspecified, T("5:04"), n, bounds: true);
Render(rev, Unspecified, T("5:04")).ShouldBe("5:05, 5:03, 5:02, 5:00, FIRST-", $"reverse by {n}");
}
}
/// <summary>A bound is a stored sample of any quality: a Bad sample before StartTime is the start bound, as stored.</summary>
[Theory]
[InlineData(Shape.Reverse)]
[InlineData(Shape.ForwardOnly)]
[InlineData(Shape.Microsecond)]
public async Task A_bad_sample_is_returned_as_the_bound_as_stored(Shape shape)
{
var series = TableData();
series[0] = new DataValueSnapshot(null, StatusCodes.BadNoCommunication, T("5:00"), T("5:00"));
var (values, _, _) = await ReadAsync(Source(shape, series), T("5:01"), T("5:04"), 0, true, cp: null);
values[0].StatusCode.Code.ShouldBe(StatusCodes.BadNoCommunication);
values[0].SourceTimestamp.ShouldBe(T("5:00"));
values.Count.ShouldBe(4);
}
/// <summary>
/// Ties on a bound: the start bound before a forward window is the LAST of a tie cluster (the sample immediately
/// before StartTime in the read's order); the end bound after it is the FIRST of one; and ties ON StartTime are all
/// data (the bound is the first of them, so nothing extra is added).
/// </summary>
[Theory]
[InlineData(Shape.Reverse)]
[InlineData(Shape.ForwardOnly)]
[InlineData(Shape.Microsecond)]
public async Task Tie_clusters_on_either_side_yield_one_bound_each(Shape shape)
{
var series = new List<DataValueSnapshot>();
void Add(string t, double v) => series.Add(new DataValueSnapshot(v, StatusCodes.Good, T(t), T(t)));
Add("5:00", 1); Add("5:00", 2); Add("5:00", 3);
Add("5:02", 4);
Add("5:05", 5); Add("5:05", 6);
var (fwd, _, _) = await ReadAsync(Source(shape, series), T("5:01"), T("5:04"), 0, true, cp: null);
fwd.Select(v => (double)v.Value).ShouldBe([3d, 4d, 5d]);
var (rev, _, _) = await ReadAsync(Source(shape, series), T("5:04"), T("5:01"), 0, true, cp: null);
rev.Select(v => (double)v.Value).ShouldBe([5d, 4d, 3d]);
var (onStart, _, _) = await ReadAsync(Source(shape, series), T("5:00"), T("5:03"), 0, true, cp: null);
onStart.Select(v => (double)v.Value).ShouldBe([1d, 2d, 3d, 4d, 5d]);
}
/// <summary>
/// QuestDB truncates the window bounds it is sent to the microsecond: a sample at ⌊StartTime⌋ is inside a window that
/// starts a few ticks later, so it is data — never returned a second time as the start bound.
/// </summary>
[Fact]
public async Task A_microsecond_backend_does_not_return_a_sample_at_the_truncated_start_twice()
{
var series = TableData();
var (values, _, _) = await ReadAsync(Source(Shape.Microsecond, series), T("5:00").AddTicks(7), T("5:04"), 0, true, cp: null);
values.Select(v => (double)v.Value).ShouldBe([0d, 2d, 3d, 5d]);
}
/// <summary>
/// A backend whose forward end is inclusive (the gateway's own <c>ReadRaw</c>), or a resumed read that steps onto
/// EndTime, returns the sample AT EndTime as data — it must come back once, as the end bound.
/// </summary>
[Theory]
[InlineData(0u)]
[InlineData(1u)]
[InlineData(2u)]
[InlineData(3u)]
public async Task An_inclusive_forward_end_does_not_return_the_end_bound_twice(uint max)
{
var source = new SeriesSource(TableData(), reverse: false, inclusiveForwardEnd: true, resolution: 1);
var values = await DrainAsync(source, T("5:00"), T("5:05"), max, bounds: true);
Render(values, T("5:00"), T("5:05")).ShouldBe("5:00, 5:02, 5:03, 5:05");
}
/// <summary>
/// #691 review: no data AND no bound found ⇒ <c>GoodNoData</c> with no values — the answer a tag with no history (and a
/// disabled historian) gave before bounds (§6.5.3.2: bounds make the result Good only "if one or both exist"). Only
/// when the WHOLE read is empty: a single bound found keeps the placeholders (Table 1: "4:48 4:48 1 ⇒ FIRST").
/// </summary>
[Theory]
[InlineData(Shape.Reverse)]
[InlineData(Shape.ForwardOnly)]
[InlineData(Shape.Microsecond)]
public async Task No_data_and_no_bound_found_is_GoodNoData_with_no_values(Shape shape)
{
var windows = new (string S, string E)[] { ("5:00", "5:05"), ("5:05", "5:00"), ("5:00", "5:00"), ("5:00", ""), ("", "5:05") };
foreach (var (start, end) in windows)
{
var s = start.Length == 0 ? Unspecified : T(start);
var e = end.Length == 0 ? Unspecified : T(end);
foreach (var n in new uint[] { 0, 1, 2, 3 })
{
if (n == 0 && (s == Unspecified || e == Unspecified)) continue; // a one-sided read needs a cap
var (values, error, cp) = await ReadResultAsync(Source(shape, []), s, e, n);
values.ShouldBeEmpty($"{start}->{end} by {n}");
cp.ShouldBeNull();
error.StatusCode.Code.ShouldBe(StatusCodes.Good);
LastNodeStatus.Code.ShouldBe(StatusCodes.GoodNoData, $"{start}->{end} by {n}");
}
}
// Positive control: one bound found keeps the missing one's placeholder.
var single = await DrainAsync(Source(shape, TableData()), T("4:48"), T("4:48"), 1, bounds: true);
Render(single, T("4:48"), T("4:48")).ShouldBe("FIRST, 5:00");
}
/// <summary>
/// #691 review: <c>ServerHistorian:HonorReturnBounds = false</c> restores the pre-#691 answer — ReturnBounds is ignored
/// (Table 1's "No" rows for every "Yes" row) and a modified read with ReturnBounds is not refused for it.
/// </summary>
[Fact]
public async Task The_kill_switch_ignores_return_bounds()
{
await fx.Gate.WaitAsync(TestContext.Current.CancellationToken);
try
{
fx.Nm.HonorReturnBounds = false;
fx.Nm.HistorianDataSource = Source(Shape.Reverse, TableData());
var (values, _, _) = Read(T("5:01"), T("5:07"), 0, bounds: true, cp: null, modified: false);
Render(values, T("5:01"), T("5:07")).ShouldBe("5:02, 5:03, 5:05, 5:06");
var (_, modErr, _) = Read(T("5:00"), T("5:05"), 0, bounds: true, cp: null, modified: true);
modErr.StatusCode.Code.ShouldBe(StatusCodes.BadHistoryOperationUnsupported, "no trail on this fake — not InvalidArgument");
}
finally
{
fx.Nm.HonorReturnBounds = true;
fx.Gate.Release();
}
}
[Fact]
public async Task Bounds_are_honoured_by_default()
{
await fx.Gate.WaitAsync(TestContext.Current.CancellationToken);
try { fx.Nm.HonorReturnBounds.ShouldBeTrue(); }
finally { fx.Gate.Release(); }
}
/// <summary>Part 11 §6.5.3.3: a modified read with ReturnBounds is <c>Bad_InvalidArgument</c>.</summary>
[Fact]
public async Task A_modified_read_with_return_bounds_is_an_invalid_argument()
{
await fx.Gate.WaitAsync(TestContext.Current.CancellationToken);
try
{
fx.Nm.HistorianDataSource = Source(Shape.Reverse, TableData());
var (_, error, cp) = Read(T("5:00"), T("5:05"), 0, bounds: true, cp: null, modified: true);
error.StatusCode.Code.ShouldBe(StatusCodes.BadInvalidArgument);
cp.ShouldBeNull();
}
finally { fx.Gate.Release(); }
}
// --- helpers --------------------------------------------------------------------------------
private static string Render(IReadOnlyList<DataValue> values, DateTime start, DateTime end)
{
var parts = new List<string>();
for (var i = 0; i < values.Count; i++)
{
var v = values[i];
if (v.StatusCode.Code != StatusCodes.BadBoundNotFound)
{
v.Value.ShouldNotBeNull();
parts.Add($"{v.SourceTimestamp.Hour}:{v.SourceTimestamp.Minute:00}");
continue;
}
v.Value.ShouldBeNull("a missing bound carries a null value");
var isFirst = i == 0 && start != Unspecified && v.SourceTimestamp == start;
var isLast = end != Unspecified && v.SourceTimestamp == end;
if (isFirst) { parts.Add("FIRST"); continue; }
if (isLast) { parts.Add("LAST"); continue; }
// Open side (footnotes a/b): one second from the previous returned value.
i.ShouldBeGreaterThan(0);
var prev = values[i - 1].SourceTimestamp;
if (v.SourceTimestamp == prev.AddSeconds(1)) parts.Add("LAST+");
else if (v.SourceTimestamp == prev.AddSeconds(-1)) parts.Add("FIRST-");
else parts.Add($"?{v.SourceTimestamp:O}");
}
return string.Join(", ", parts);
}
private async Task<List<DataValue>> DrainAsync(IHistorianDataSource source, DateTime start, DateTime end, uint max, bool bounds)
{
var all = new List<DataValue>();
byte[]? cp = null;
var pages = 0;
do
{
var (values, error, next) = await ReadAsync(source, start, end, max, bounds, cp);
error.StatusCode.Code.ShouldBe(StatusCodes.Good);
if (max > 0) values.Count.ShouldBeLessThanOrEqualTo((int)max, "bounds count toward NumValuesPerNode");
all.AddRange(values);
cp = next;
(++pages).ShouldBeLessThan(50, "paging must terminate");
}
while (cp is not null);
return all;
}
/// <summary>The node-level StatusCode of the last <see cref="Read"/> (GoodNoData vs Good).</summary>
private StatusCode LastNodeStatus;
private async Task<(IReadOnlyList<DataValue> Values, ServiceResult Error, byte[]? Cp)> ReadResultAsync(
IHistorianDataSource source, DateTime start, DateTime end, uint max) =>
await ReadAsync(source, start, end, max, bounds: true, cp: null);
private async Task<(IReadOnlyList<DataValue> Values, ServiceResult Error, byte[]? Cp)> ReadAsync(
IHistorianDataSource source, DateTime start, DateTime end, uint max, bool bounds, byte[]? cp)
{
await fx.Gate.WaitAsync(TestContext.Current.CancellationToken);
try
{
fx.Nm.HistorianDataSource = source;
return Read(start, end, max, bounds, cp, modified: false);
}
finally { fx.Gate.Release(); }
}
private (IReadOnlyList<DataValue> Values, ServiceResult Error, byte[]? Cp) Read(
DateTime start, DateTime end, uint max, bool bounds, byte[]? cp, bool modified)
{
var context = new OperationContext(new RequestHeader(), secureChannelContext: null, RequestType.HistoryRead, identity: null);
var details = new ReadRawModifiedDetails
{
StartTime = start, EndTime = end, NumValuesPerNode = max, ReturnBounds = bounds, IsReadModified = modified,
};
var nodesToRead = new List<HistoryReadValueId> { new() { NodeId = fx.NodeId, ContinuationPoint = cp } };
var results = new List<SdkHistoryReadResult> { null! };
var errors = new List<ServiceResult> { null! };
fx.Nm.HistoryRead(context, details, TimestampsToReturn.Both, releaseContinuationPoints: false, nodesToRead, results, errors);
LastNodeStatus = results[0]?.StatusCode ?? StatusCodes.Bad;
var values = new List<DataValue>();
if (results[0]?.HistoryData is { } ho && !ExtensionObject.IsNull(ho))
values.AddRange(((HistoryData)ExtensionObject.ToEncodeable(ho)).DataValues);
return (values, errors[0], results[0]?.ContinuationPoint);
}
/// <summary>
/// A series-backed historian. <paramref name="reverse"/>: serves <c>start &gt; end</c> as <c>(end, start]</c>
/// newest-first (SqlServer/QuestDB); otherwise rejects it (the Gateway) and counts that. The forward read is
/// <c>[start, end)</c>, or <c>[start, end]</c> when <paramref name="inclusiveForwardEnd"/> (the gateway's own end);
/// a point read is the instant. <paramref name="resolution"/> &gt; 1 truncates every window bound to it first (QuestDB
/// µs); the series must already be stored at that resolution.
/// </summary>
private sealed class SeriesSource(IReadOnlyList<DataValueSnapshot> series, bool reverse, bool inclusiveForwardEnd, long resolution)
: IHistorianDataSource
{
public int ReverseWindowsOnForwardOnly;
public bool SupportsReverseRaw => reverse;
public long TimestampResolutionTicks => resolution;
private DateTime Trunc(DateTime t) => new(t.Ticks - t.Ticks % resolution, DateTimeKind.Utc);
public Task<HistorianRead> ReadRawAsync(
string fullReference, DateTime startUtc, DateTime endUtc, uint maxValuesPerNode, CancellationToken cancellationToken)
{
static DateTime Ts(DataValueSnapshot s) => s.SourceTimestampUtc ?? DateTime.MinValue;
startUtc = Trunc(startUtc);
endUtc = Trunc(endUtc);
List<DataValueSnapshot> window;
if (startUtc > endUtc)
{
if (!reverse)
{
Interlocked.Increment(ref ReverseWindowsOnForwardOnly);
throw new InvalidOperationException("start_time must be less than or equal to end_time. (InvalidArgument)");
}
window = series.Where(s => Ts(s) > endUtc && Ts(s) <= startUtc).Reverse().ToList();
}
else if (startUtc == endUtc) window = series.Where(s => Ts(s) == startUtc).ToList();
else window = series.Where(s => Ts(s) >= startUtc && (inclusiveForwardEnd ? Ts(s) <= endUtc : Ts(s) < endUtc)).ToList();
var page = maxValuesPerNode == 0 ? window : window.Take((int)maxValuesPerNode).ToList();
return Task.FromResult(new HistorianRead(page, null));
}
public Task<HistorianRead> ReadProcessedAsync(
string fullReference, DateTime startUtc, DateTime endUtc, TimeSpan interval,
HistoryAggregateType aggregate, CancellationToken cancellationToken) =>
Task.FromResult(new HistorianRead(Array.Empty<DataValueSnapshot>(), null));
public Task<HistorianRead> ReadAtTimeAsync(
string fullReference, IReadOnlyList<DateTime> timestampsUtc, CancellationToken cancellationToken) =>
Task.FromResult(new HistorianRead(Array.Empty<DataValueSnapshot>(), null));
public Task<HistoricalEventsResult> ReadEventsAsync(
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, CancellationToken cancellationToken) =>
Task.FromResult(new HistoricalEventsResult(Array.Empty<HistoricalEvent>(), null));
public HistorianHealthSnapshot GetHealthSnapshot() => NullHistorianDataSource.Instance.GetHealthSnapshot();
public void Dispose() { }
}
}
@@ -45,6 +45,7 @@ public sealed class NodeManagerHistoryReadConcurrencyTests : IDisposable
var details = new ReadRawModifiedDetails
{
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow,
NumValuesPerNode = 10,
@@ -78,6 +79,7 @@ public sealed class NodeManagerHistoryReadConcurrencyTests : IDisposable
var nodeIds = MaterializeHistorizedNodes(nm, count: 12);
var details = new ReadRawModifiedDetails
{
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow,
NumValuesPerNode = 10,
@@ -115,6 +117,7 @@ public sealed class NodeManagerHistoryReadConcurrencyTests : IDisposable
var nodeIds = MaterializeHistorizedNodes(nm, count: 8);
var details = new ReadRawModifiedDetails
{
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow,
NumValuesPerNode = 10,
@@ -157,6 +160,7 @@ public sealed class NodeManagerHistoryReadConcurrencyTests : IDisposable
var nodeIds = MaterializeHistorizedNodes(nm, count: 4);
var details = new ReadRawModifiedDetails
{
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow,
NumValuesPerNode = 10,
@@ -190,6 +194,7 @@ public sealed class NodeManagerHistoryReadConcurrencyTests : IDisposable
var ids = MaterializeHistorizedNodes(nm, count: 2);
var details = new ReadRawModifiedDetails
{
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow,
NumValuesPerNode = 10,
@@ -160,7 +160,7 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable
var ctx = new OperationContext(new RequestHeader(), secureChannelContext: null, RequestType.HistoryRead, identity: null);
var res = new List<SdkHistoryReadResult> { null! };
var errs = new List<ServiceResult> { null! };
nm.HistoryRead(ctx, new ReadRawModifiedDetails { StartTime = Epoch, EndTime = Epoch.AddHours(1), NumValuesPerNode = 2, IsReadModified = false },
nm.HistoryRead(ctx, new ReadRawModifiedDetails { ReturnBounds = false, StartTime = Epoch, EndTime = Epoch.AddHours(1), NumValuesPerNode = 2, IsReadModified = false },
TimestampsToReturn.Both, false,
new List<HistoryReadValueId> { new() { NodeId = n, ContinuationPoint = cp } }, res, errs);
err = errs[0];
@@ -333,12 +333,16 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable
}
[Fact]
public async Task A_backend_without_reverse_support_answers_BadHistoryOperationUnsupported_not_the_earliest_events()
public async Task A_backend_without_reverse_support_walks_a_reverse_read_back_never_answering_the_earliest_events()
{
// #691.4 (was refused BadHistoryOperationUnsupported since #675): served newest-first from forward reads.
var (host, nm, node, src) = await SetupAsync(Distinct(10));
nm.HistorianDataSource = new ForwardOnlyEventSource(src);
ReadEvents(nm, node, max: 3, inboundCp: null, end: Early, start: Late).Error.StatusCode.Code
.ShouldBe(StatusCodes.BadHistoryOperationUnsupported);
var fwd = new ForwardOnlyEventSource(src);
nm.HistorianDataSource = fwd;
var r = ReadEvents(nm, node, max: 3, inboundCp: null, end: Early, start: Late);
r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good);
r.Ids.ShouldBe(["e0009", "e0008", "e0007"]);
r.Cp.ShouldNotBeNull();
// Forward reads on the same backend are unaffected.
DrainAsync(nm, node, max: 4).ShouldBe(src.All.Select(e => e.EventId));
await host.DisposeAsync();
@@ -349,30 +353,133 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable
// --- #675 review follow-ups: the reverse-events capability ------------------------------------------------
// --- #691.4: reverse events on a backend that reads forward only (the Gateway) are walked back ------------------
[Fact]
public async Task End_only_capped_read_on_a_backend_without_reverse_support_pages_forward_as_before()
public async Task End_only_capped_read_on_a_forward_only_backend_pages_backward_from_EndTime()
{
// Gateway (the default backend) cannot read newest-first: an end-only read keeps its pre-#675 forward
// [MinValue, EndTime) meaning there rather than turning into a refused reverse read.
var (host, nm, node, src) = await SetupAsync(Distinct(25));
var fwd = new NonReverseSource(src);
nm.HistorianDataSource = fwd;
// Part 11 (as raw, #686): the latest events at or before EndTime, newest first — on Gateway this used to keep its
// pre-#675 forward [MinValue, EndTime) meaning, i.e. the OLDEST events. The backend is never sent a reverse read.
var (host, nm, node, _) = await SetupAsync([]);
var gw = new GatewayShapedEventSource(Distinct(25));
nm.HistorianDataSource = gw;
var ids = DrainAsync(nm, node, max: 10, start: DateTime.MinValue, end: Epoch.AddSeconds(20));
ids.ShouldBe(Enumerable.Range(0, 20).Select(i => $"e{i:0000}"));
fwd.ReverseCalls.ShouldBe(0);
ids.ShouldBe(Enumerable.Range(0, 21).Reverse().Select(i => $"e{i:0000}"));
gw.ReverseCalls.ShouldBe(0);
gw.BadRequests.ShouldBe(0);
await host.DisposeAsync();
}
[Fact]
public async Task Explicit_reverse_read_on_a_backend_without_reverse_support_is_refused_without_calling_it()
public async Task Explicit_reverse_read_on_a_forward_only_backend_pages_the_reverse_window_newest_first()
{
var (host, nm, node, src) = await SetupAsync(Distinct(10));
var fwd = new NonReverseSource(src);
nm.HistorianDataSource = fwd;
ReadEvents(nm, node, max: 3, inboundCp: null, end: Early, start: Late).Error.StatusCode.Code
.ShouldBe(StatusCodes.BadHistoryOperationUnsupported);
fwd.ReverseCalls.ShouldBe(0, "the capability is checked BEFORE the source is called");
fwd.ForwardCalls.ShouldBe(0);
// (EndTime, StartTime]: StartTime inclusive, EndTime exclusive.
var (host, nm, node, _) = await SetupAsync([]);
var gw = new GatewayShapedEventSource(Distinct(30));
nm.HistorianDataSource = gw;
var ids = DrainAsync(nm, node, max: 4, start: Epoch.AddSeconds(25), end: Epoch.AddSeconds(5));
ids.ShouldBe(Enumerable.Range(6, 20).Reverse().Select(i => $"e{i:0000}"));
gw.ReverseCalls.ShouldBe(0);
await host.DisposeAsync();
}
[Fact]
public async Task Reverse_ties_on_a_forward_only_backend_page_in_a_stable_order_although_the_backend_shuffles_them()
{
// The gateway orders events by EventTimeUtc only, so ties come back in any order — this fake shuffles them on every
// call. The walk sorts ties by EventId (descending, newest-first) on every read, and the oversized-cluster over-fetch
// does the same, so a boundary-tie skip always lines up. Cap 3 cuts through a 7-way cluster.
var events = new List<HistoricalEvent> { Ev("a0", 0) };
for (var i = 0; i < 7; i++) events.Add(Ev($"t{i}", 5));
events.Add(Ev("z0", 9));
var (host, nm, node, _) = await SetupAsync([]);
nm.HistorianDataSource = new GatewayShapedEventSource(events, shuffleSeed: 7);
DrainAsync(nm, node, max: 3, start: DateTime.MinValue, end: Epoch.AddSeconds(30)).ShouldBe(
["z0", "t6", "t5", "t4", "t3", "t2", "t1", "t0", "a0"]);
await host.DisposeAsync();
}
[Fact]
public async Task Uncapped_reverse_read_on_a_forward_only_backend_is_one_forward_read()
{
var (host, nm, node, _) = await SetupAsync([]);
var gw = new GatewayShapedEventSource(Distinct(30));
nm.HistorianDataSource = gw;
var r = ReadEvents(nm, node, max: 0, inboundCp: null, end: Epoch.AddSeconds(5), start: Epoch.AddSeconds(25));
r.Ids.ShouldBe(Enumerable.Range(6, 20).Reverse().Select(i => $"e{i:0000}"));
r.Cp.ShouldBeNull();
gw.ForwardCalls.ShouldBe(1);
await host.DisposeAsync();
}
[Fact]
public async Task Reverse_paging_on_a_forward_only_backend_is_exact_on_random_data()
{
var (host, nm, node, _) = await SetupAsync([]);
for (var seed = 0; seed < 80; seed++)
{
var rng = new Random(seed);
var events = new List<HistoricalEvent>();
var t = Epoch.Ticks;
var n = rng.Next(1, 120);
while (events.Count < n)
{
t += rng.Next(4) switch
{
0 => rng.Next(1, 10),
1 => rng.Next(1, 5) * TimeSpan.TicksPerMillisecond,
2 => rng.Next(1, 90) * TimeSpan.TicksPerMinute,
_ => rng.Next(1, 2000) * TimeSpan.TicksPerMillisecond,
};
var ties = rng.Next(5) == 0 ? rng.Next(2, 8) : 1;
for (var k = 0; k < ties; k++)
{
var at = new DateTime(t, DateTimeKind.Utc);
events.Add(new HistoricalEvent($"{Guid.NewGuid():N}", "src", at, at, "m", 500));
}
}
nm.HistorianDataSource = new GatewayShapedEventSource(events, shuffleSeed: seed);
nm.MaxTieClusterOverfetch = rng.Next(2) == 0 ? 8 : 65536;
var max = (uint)rng.Next(1, 9);
var end = new DateTime(t + rng.Next(0, 3) * TimeSpan.TicksPerSecond, DateTimeKind.Utc);
var expected = events.OrderByDescending(e => e.EventTimeUtc).ThenByDescending(e => e.EventId, StringComparer.Ordinal)
.Select(e => e.EventId).ToList();
// The walk refuses more than MaxTieClusterOverfetch events within ONE MILLISECOND (#691.4 re-review): with the
// small budget a dense sub-millisecond burst is a legitimate BadTooManyOperations — and nothing else is.
var times = events.Select(e => e.EventTimeUtc.Ticks).Order().ToList();
var msMax = 0;
for (int i = 0, j = 0; i < times.Count; i++)
{
while (times[i] - times[j] >= TimeSpan.TicksPerMillisecond) j++;
msMax = Math.Max(msMax, i - j + 1);
}
var all = new List<string>();
byte[]? cp = null;
var pages = 0;
var refused = false;
do
{
var r = ReadEvents(nm, node, max, cp, end, DateTime.MinValue);
if (r.Error.StatusCode.Code == StatusCodes.BadTooManyOperations && msMax > nm.MaxTieClusterOverfetch)
{
refused = true;
break;
}
r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good, $"seed {seed}, ms max {msMax}");
all.AddRange(r.Ids);
cp = r.Cp;
(++pages).ShouldBeLessThan(2000);
}
while (cp is not null);
if (refused) all.ShouldBe(expected.Take(all.Count), $"seed {seed}: what came before the refusal");
else all.ShouldBe(expected, $"seed {seed}, N = {max}");
}
await host.DisposeAsync();
}
@@ -391,29 +498,39 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable
await host.DisposeAsync();
}
/// <summary>Declares the reverse overload (so a call would be visible) but reports no reverse capability.</summary>
private sealed class NonReverseSource(IHistorianDataSource inner) : IHistorianDataSource
/// <summary>
/// Shaped like the Gateway's SQL event read (#691.4): forward only (<see cref="SupportsReverseEvents"/> false — a
/// reverse call is counted and refused), <c>[start, end]</c> INCLUSIVE, ordered by event time only with ties in an
/// arbitrary order (shuffled on every call when a seed is given), capped with a truncation flag.
/// </summary>
private sealed class GatewayShapedEventSource(List<HistoricalEvent> events, int? shuffleSeed = null) : IHistorianDataSource
{
private readonly Random? _rng = shuffleSeed is { } s ? new Random(s) : null;
public int ReverseCalls { get; private set; }
public int ForwardCalls { get; private set; }
public int BadRequests { get; private set; }
public bool SupportsReverseEvents => false;
public Task<HistoricalEventsResult> ReadEventsAsync(string? n, DateTime s, DateTime e, int m, CancellationToken ct)
{
ForwardCalls++;
return inner.ReadEventsAsync(n, s, e, m, ct);
if (s > e || s < DateTime.UnixEpoch) BadRequests++;
var window = events.Where(x => x.EventTimeUtc >= s && x.EventTimeUtc <= e)
.OrderBy(x => x.EventTimeUtc).ThenBy(_ => _rng?.Next() ?? 0).ToList();
var cap = m <= 0 ? int.MaxValue : m;
return Task.FromResult(new HistoricalEventsResult(window.Take(cap).ToList(), window.Count > cap ? new byte[] { 1 } : null));
}
public Task<HistoricalEventsResult> ReadEventsAsync(string? n, DateTime s, DateTime e, int m, bool reverse, CancellationToken ct)
{
if (reverse) { ReverseCalls++; return inner.ReadEventsAsync(n, s, e, m, true, ct); }
if (reverse) { ReverseCalls++; throw new NotSupportedException("forward only"); }
return ReadEventsAsync(n, s, e, m, ct);
}
public Task<HistorianRead> ReadRawAsync(string r, DateTime s, DateTime e, uint m, CancellationToken ct) =>
inner.ReadRawAsync(r, s, e, m, ct);
NullHistorianDataSource.Instance.ReadRawAsync(r, s, e, m, ct);
public Task<HistorianRead> ReadProcessedAsync(string r, DateTime s, DateTime e, TimeSpan i, HistoryAggregateType a, CancellationToken ct) =>
inner.ReadProcessedAsync(r, s, e, i, a, ct);
NullHistorianDataSource.Instance.ReadProcessedAsync(r, s, e, i, a, ct);
public Task<HistorianRead> ReadAtTimeAsync(string r, IReadOnlyList<DateTime> t, CancellationToken ct) =>
inner.ReadAtTimeAsync(r, t, ct);
public HistorianHealthSnapshot GetHealthSnapshot() => inner.GetHealthSnapshot();
NullHistorianDataSource.Instance.ReadAtTimeAsync(r, t, ct);
public HistorianHealthSnapshot GetHealthSnapshot() => NullHistorianDataSource.Instance.GetHealthSnapshot();
public void Dispose() { }
}
@@ -271,7 +271,7 @@ public sealed class NodeManagerHistoryReadModifiedTests : IDisposable
var results = new List<SdkHistoryReadResult> { null! };
var errors = new List<ServiceResult> { null! };
nm.HistoryRead(context,
new ReadRawModifiedDetails { StartTime = start, EndTime = end, NumValuesPerNode = max, IsReadModified = modified },
new ReadRawModifiedDetails { ReturnBounds = false, StartTime = start, EndTime = end, NumValuesPerNode = max, IsReadModified = modified },
TimestampsToReturn.Both, false,
new List<HistoryReadValueId> { new() { NodeId = node, ContinuationPoint = cp } }, results, errors);
@@ -650,6 +650,156 @@ public sealed class NodeManagerHistoryReadPagingTests : IDisposable
await host.DisposeAsync();
}
/// <summary>
/// #689: a paged end-only read on a dense tag sizes each segment from the PAGE (N), not from
/// <c>MaxTieClusterOverfetch</c>. Live (AVEVA, 1 Hz, N = 3): from about page 5 every page read 32,770 rows and took
/// 2–5 s to return 2 rows, because the walk re-aimed its width at half the 65,536-row tie budget and carried it.
/// </summary>
[Fact]
public async Task Raw_reverse_paging_on_a_dense_forward_only_tag_reads_about_the_page_size_per_page()
{
// 100,000 samples 100 ms apart (~2.8 h): far more than the default tie budget would ever let a page read.
var series = Enumerable.Range(0, 100_000)
.Select(i => { var t = Epoch.AddMilliseconds(100L * i); return new DataValueSnapshot((double)i, StatusCodes.Good, t, t); })
.ToArray();
var source = new ForwardOnlyHistorianDataSource(series);
var (host, nm, nodeId) = await BootForwardOnlyAsync(source, "FwdDense689");
nm.MaxTieClusterOverfetch.ShouldBe(65536, "the default budget — the one the live gate ran with");
var end = Epoch.AddMilliseconds(100L * 99_999);
byte[]? cp = null;
var expected = 99_999;
for (var page = 1; page <= 20; page++)
{
source.ResetCounts();
var (values, error, next) = ReadRaw(nm, nodeId, DateTime.MinValue, end, max: 3, inboundCp: cp);
error.StatusCode.Code.ShouldBe(StatusCodes.Good);
values.ShouldNotBeEmpty();
values[^1].ShouldBe((double)(expected - 2), $"page {page} continues exactly");
expected = (int)values[^1];
next.ShouldNotBeNull();
cp = next;
// The discriminating counts: rows read and the largest cap asked for, per page. Page 1 has no density yet and
// may read one discovery segment (a minute here, 600 rows); every page after it is sized from N.
var rowsBound = page == 1 ? ReverseWalk.DiscoveryRows + 2 : 64;
source.RowsReturned.ShouldBeLessThanOrEqualTo(rowsBound, $"page {page} read {source.RowsReturned} rows to return {values.Count}");
source.MaxCapRequested.ShouldBeLessThanOrEqualTo((uint)rowsBound, $"page {page} asked for {source.MaxCapRequested} rows");
source.ReadCount.ShouldBeLessThanOrEqualTo(3, $"page {page}");
}
await host.DisposeAsync();
}
/// <summary>
/// #689/#691: randomized end-to-end paging of end-only reads through the node manager on a forward-only backend —
/// bursts, gaps, sub-millisecond stamps and tie clusters (some larger than the page) — the union of the pages is
/// every sample exactly once, newest first, whatever N and the tie budget.
/// </summary>
[Fact]
public async Task Raw_reverse_paging_on_a_forward_only_backend_is_exact_on_random_data()
{
var (host, nm, nodeId) = await BootForwardOnlyAsync(new ForwardOnlyHistorianDataSource([]), "FwdFuzz");
for (var seed = 0; seed < 150; seed++)
{
var rng = new Random(seed);
var series = new List<DataValueSnapshot>();
var t = Epoch.Ticks;
var v = 0;
var count = rng.Next(1, 300);
while (series.Count < count)
{
t += rng.Next(5) switch
{
0 => rng.Next(1, 10),
1 => rng.Next(1, 5) * TimeSpan.TicksPerMillisecond,
2 => rng.Next(1, 90) * TimeSpan.TicksPerMinute,
_ => rng.Next(1, 2000) * TimeSpan.TicksPerMillisecond,
};
var ties = rng.Next(6) == 0 ? rng.Next(2, 7) : 1;
for (var k = 0; k < ties; k++)
{
var ts = new DateTime(t, DateTimeKind.Utc);
series.Add(new DataValueSnapshot((double)v++, StatusCodes.Good, ts, ts));
}
}
// No fabricated start row: the Gateway source drops it (#687) — the node manager's tie-cluster point read relies on
// that. The walk itself is fuzzed WITH the row in ReverseWalkTests.
nm.HistorianDataSource = new ForwardOnlyHistorianDataSource(series, emitBoundaryRow: false);
nm.MaxTieClusterOverfetch = rng.Next(2) == 0 ? 8 : 65536;
var max = (uint)rng.Next(1, 12);
var end = new DateTime(t + rng.Next(0, 3) * TimeSpan.TicksPerSecond, DateTimeKind.Utc);
// The walk refuses more than MaxTieClusterOverfetch samples within ONE MILLISECOND (#691.4 re-review), so with
// the small budget a dense sub-millisecond burst is a legitimate BadTooManyOperations — and nothing else is.
var msMax = 0;
for (int i = 0, j = 0; i < series.Count; i++)
{
while (series[i].SourceTimestampUtc!.Value.Ticks - series[j].SourceTimestampUtc!.Value.Ticks >= TimeSpan.TicksPerMillisecond) j++;
msMax = Math.Max(msMax, i - j + 1);
}
var collected = new List<double>();
byte[]? cp = null;
var pages = 0;
var refused = false;
do
{
var (values, error, next) = ReadRaw(nm, nodeId, DateTime.MinValue, end, max, inboundCp: cp);
if (error.StatusCode.Code == StatusCodes.BadTooManyOperations && msMax > nm.MaxTieClusterOverfetch)
{
refused = true;
break;
}
error.StatusCode.Code.ShouldBe(StatusCodes.Good, $"seed {seed}, N = {max}, ms max {msMax}");
collected.AddRange(values);
cp = next;
(++pages).ShouldBeLessThan(2000, "paging must terminate, not loop");
}
while (cp is not null);
var expected = Enumerable.Range(0, series.Count).Reverse().Select(i => (double)i).ToList();
if (refused) collected.ShouldBe(expected.Take(collected.Count), $"seed {seed}: what came before the refusal");
else collected.ShouldBe(expected, $"seed {seed}, N = {max}");
}
await host.DisposeAsync();
}
/// <summary>
/// #691: the last pages of a deep reverse page-through no longer each walk down to 1970 (live: ~14–16 reads, ~2.5 s
/// per page). The stretch a page proved empty below its oldest row rides the continuation point, and a page that runs
/// out of rows asks once for the whole remainder — so every page here is a couple of reads.
/// </summary>
[Fact]
public async Task Raw_reverse_page_through_on_a_forward_only_backend_does_not_rewalk_to_the_epoch_on_each_last_page()
{
var source = new ForwardOnlyHistorianDataSource(MakeSeries(count: 25, stepSeconds: 1));
var (host, nm, nodeId) = await BootForwardOnlyAsync(source, "FwdDeep691");
nm.Clock = new FixedClock(Epoch.AddHours(1));
var collected = new List<double>();
byte[]? cp = null;
var page = 0;
do
{
source.ResetCounts();
var (values, error, next) = ReadRaw(nm, nodeId, DateTime.MinValue, Epoch.AddHours(1), max: 10, inboundCp: cp);
error.StatusCode.Code.ShouldBe(StatusCodes.Good);
collected.AddRange(values);
cp = next;
page++;
source.ReadCount.ShouldBeLessThanOrEqualTo(page == 1 ? 4 : 1, $"page {page}");
}
while (cp is not null && page < 10);
collected.ShouldBe(Enumerable.Range(0, 25).Reverse().Select(i => (double)i));
source.MinRequestedStart.ShouldBeGreaterThanOrEqualTo(DateTime.UnixEpoch);
await host.DisposeAsync();
}
/// <summary>#686 review: a reverse read from <c>StartTime = DateTime.MaxValue</c> walks back from now + 1 day, not from
/// year 9999.</summary>
[Fact]
@@ -817,6 +967,7 @@ public sealed class NodeManagerHistoryReadPagingTests : IDisposable
var details = new ReadRawModifiedDetails
{
ReturnBounds = false,
StartTime = start,
EndTime = end,
NumValuesPerNode = max,
@@ -850,6 +1001,7 @@ public sealed class NodeManagerHistoryReadPagingTests : IDisposable
new RequestHeader(), secureChannelContext: null, RequestType.HistoryRead, identity: null);
var details = new ReadRawModifiedDetails
{
ReturnBounds = false,
StartTime = Epoch, EndTime = Epoch.AddHours(1), NumValuesPerNode = 100, IsReadModified = false,
};
var nodesToRead = new List<HistoryReadValueId> { new() { NodeId = nodeId, ContinuationPoint = cp } };
@@ -978,9 +1130,15 @@ public sealed class NodeManagerHistoryReadPagingTests : IDisposable
/// <summary>Reads whose row count equalled the cap (a full, possibly truncated, read).</summary>
public int FullReads { get; private set; }
/// <summary>Rows returned across every read since the last <see cref="ResetCounts"/> (#689).</summary>
public long RowsReturned { get; private set; }
/// <summary>The largest <c>maxValuesPerNode</c> asked for since the last <see cref="ResetCounts"/> (#689).</summary>
public uint MaxCapRequested { get; private set; }
public void ResetCounts()
{
lock (_gate) FullReads = 0;
lock (_gate) { FullReads = 0; RowsReturned = 0; MaxCapRequested = 0; }
Interlocked.Exchange(ref _readCount, 0);
}
@@ -1016,8 +1174,12 @@ public sealed class NodeManagerHistoryReadPagingTests : IDisposable
: new DataValueSnapshot(prior.Value, prior.StatusCode, stamp, stamp));
rows.AddRange(series.Where(s => Ts(s) >= startUtc && Ts(s) <= endUtc));
var page = maxValuesPerNode == 0 ? rows : rows.Take((int)maxValuesPerNode).ToList();
if (maxValuesPerNode > 0 && page.Count == maxValuesPerNode)
lock (_gate) FullReads++;
lock (_gate)
{
if (maxValuesPerNode > 0 && page.Count == maxValuesPerNode) FullReads++;
RowsReturned += page.Count;
if (maxValuesPerNode > MaxCapRequested) MaxCapRequested = maxValuesPerNode;
}
int seen;
while (page.Count > (seen = Volatile.Read(ref _maxRowsReturned))
&& Interlocked.CompareExchange(ref _maxRowsReturned, page.Count, seen) != seen) { }
@@ -52,6 +52,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
var end = DateTime.UtcNow;
var details = new ReadRawModifiedDetails
{
ReturnBounds = false,
StartTime = start,
EndTime = end,
NumValuesPerNode = 100,
@@ -97,6 +98,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
var details = new ReadRawModifiedDetails
{
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow,
NumValuesPerNode = 10,
@@ -129,6 +131,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
var details = new ReadRawModifiedDetails
{
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow,
NumValuesPerNode = 10,
@@ -162,6 +165,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
var details = new ReadRawModifiedDetails
{
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow,
NumValuesPerNode = 10,
@@ -195,6 +199,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
var details = new ReadRawModifiedDetails
{
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow,
NumValuesPerNode = 10,
@@ -613,6 +618,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
var details = new ReadRawModifiedDetails
{
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow,
NumValuesPerNode = 10,
@@ -658,6 +664,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
var details = new ReadRawModifiedDetails
{
ReturnBounds = false,
StartTime = DateTime.UtcNow.AddHours(-1),
EndTime = DateTime.UtcNow,
NumValuesPerNode = 10,
@@ -0,0 +1,329 @@
using Opc.Ua;
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using HistorianRead = ZB.MOM.WW.OtOpcUa.Core.Abstractions.HistoryReadResult;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests;
/// <summary>
/// Gitea #689 / #691: the reverse walk-back (<see cref="ReverseWalk"/> through <see cref="ReverseRawReader"/>) driven
/// directly, without a server: a randomized exactness check against a brute-force answer, and the sizing properties.
/// </summary>
public sealed class ReverseWalkTests
{
private static CancellationToken Ct => TestContext.Current.CancellationToken;
private static readonly DateTime T0 = new(2026, 10, 12, 18, 0, 0, DateTimeKind.Utc);
private static DataValueSnapshot At(long ticks, double v) =>
new(v, StatusCodes.Good, new DateTime(ticks, DateTimeKind.Utc), new DateTime(ticks, DateTimeKind.Utc));
/// <summary>
/// A randomized walk over AVEVA-shaped forward reads (inclusive end, a fabricated row at the query start — optionally
/// stamped at the start rounded UP to the millisecond — counted against the cap): every page is exactly the brute-force
/// newest-first answer over <c>(lo, hi]</c>, ties in the backend's forward order reversed, whatever the carried width
/// — and a timestamp over the tie budget is the only refusal.
/// </summary>
[Fact]
public async Task A_walked_page_is_exactly_the_newest_N_over_the_window_on_random_data()
{
var failures = new List<string>();
for (var seed = 0; seed < 400; seed++)
{
var rng = new Random(seed);
var series = RandomSeries(rng, out _);
// The refusal unit is one millisecond (#691.4 re-review): the most rows inside any 1 ms window.
var maxCluster = 0;
for (int i = 0, j = 0; i < series.Count; i++)
{
while (Ts(series[i]) - Ts(series[j]) >= TimeSpan.TicksPerMillisecond) j++;
maxCluster = Math.Max(maxCluster, i - j + 1);
}
var source = new AvevaShapedSource(series, roundUp: rng.Next(2) == 0);
var tieBudget = rng.Next(4) switch { 0 => 1, 1 => 3, 2 => 8, _ => 65536 };
var want = (uint)rng.Next(1, 40);
var span = Ts(series[^1]) - Ts(series[0]);
var hi = Ts(series[0]) + (long)(rng.NextDouble() * 1.2 * span) + rng.Next(0, 3);
var lo = Ts(series[0]) - TimeSpan.TicksPerSecond + (long)(rng.NextDouble() * (hi - Ts(series[0])) * 0.8);
if (rng.Next(5) == 0) lo = 0; // an end-only read
var hint = rng.Next(3) switch
{
0 => default,
1 => new ReverseWalkHint(rng.Next(1, 10)),
_ => new ReverseWalkHint((long)(rng.NextDouble() * TimeSpan.TicksPerHour)),
};
if (rng.Next(2) == 0)
{
// A TRUE empty stretch (#691): between two adjacent distinct timestamps, or from the floor to the first row.
var k = rng.Next(series.Count);
var from = k == 0 ? (rng.Next(2) == 0 ? 0 : DateTime.UnixEpoch.Ticks) : Ts(series[k - 1]);
var to = Ts(series[k]);
if (to > from) hint = hint with { EmptyFromTicks = from, EmptyToTicks = to };
}
// Page on: each next page resumes just below the previous page's oldest timestamp with the hint it returned —
// so the hints the walk itself produces (width, proven-empty stretch) are exercised too.
for (var pageNo = 0; pageNo < 6 && hi > Math.Max(lo, DateTime.UnixEpoch.Ticks); pageNo++)
{
var expected = series.Where(s => Ts(s) > Math.Max(lo, DateTime.UnixEpoch.Ticks) && Ts(s) <= hi)
.Select((s, i) => (s, i)).OrderByDescending(x => Ts(x.s)).ThenByDescending(x => x.i)
.Take((int)want).Select(x => (double)x.s.Value!).ToList();
try
{
var page = await ReverseRawReader.ReadAsync(source, "T", new DateTime(hi, DateTimeKind.Utc),
new DateTime(lo, DateTimeKind.Utc), want, tieBudget, T0.AddDays(1), hint, Ct);
var got = page.Samples.Select(s => (double)s.Value!).ToList();
if (!got.SequenceEqual(expected))
{
failures.Add($"seed {seed} page {pageNo}: got [{string.Join(",", got)}] expected [{string.Join(",", expected)}]");
break;
}
if (page.Hint.WidthTicks <= 0) failures.Add($"seed {seed}: no width carried");
if (got.Count < want) break;
hint = page.Hint;
hi = Ts(page.Samples[^1]) - 1;
}
catch (HistoryReadLimitExceededException)
{
if (maxCluster <= tieBudget) failures.Add($"seed {seed}: refused with no millisecond over {tieBudget} rows (max {maxCluster})");
break;
}
}
if (source.MaxCap > 2L * Math.Max(Math.Max(tieBudget, ReverseWalk.DiscoveryRows), 2 * want + 16) + 4)
failures.Add($"seed {seed}: asked for {source.MaxCap} rows");
if (source.BadRequests > 0) failures.Add($"seed {seed}: {source.BadRequests} invalid requests");
}
failures.ShouldBeEmpty();
}
/// <summary>
/// #689: the carried width sizes a resumed page from N, not from the tie budget — the live symptom was 32,770 rows
/// per page for N = 3 on a 1 Hz tag once the width had grown over a few pages.
/// </summary>
[Fact]
public async Task A_resumed_page_on_a_dense_tag_reads_about_N_rows_with_the_default_tie_budget()
{
var series = Enumerable.Range(0, 200_000).Select(i => At(T0.Ticks + i * TimeSpan.TicksPerSecond, i)).ToList();
var source = new AvevaShapedSource(series);
var hi = new DateTime(Ts(series[^1]), DateTimeKind.Utc);
var hint = default(ReverseWalkHint);
for (var page = 1; page <= 30; page++)
{
source.Reset();
var result = await ReverseRawReader.ReadAsync(source, "T", hi, DateTime.MinValue, 3, 65536, hi.AddHours(1), hint, Ct);
result.Samples.Count.ShouldBe(3);
hint = result.Hint;
hi = result.Samples[^1].SourceTimestampUtc!.Value.AddTicks(-1);
if (page > 1) source.RowsReturned.ShouldBeLessThanOrEqualTo(30, $"page {page}");
}
}
/// <summary>
/// #689: a dense tag that stopped long ago. The walk's ×4 growth across the empty stretch overshoots into far more
/// rows than one read takes; the truncated read only shows the OLDEST of them, so the walk must not keep re-entering
/// that stretch — it searches the gap above the newest row it saw (gallop up, then bisect). Measured: 26 reads.
/// </summary>
[Fact]
public async Task A_dense_tag_below_a_long_empty_stretch_is_found_in_a_bounded_number_of_reads()
{
// 10 Hz for 3 hours, ending two days before "now".
var start = T0.AddDays(-2).AddHours(-3);
var series = Enumerable.Range(0, 108_000).Select(i => At(start.Ticks + i * TimeSpan.TicksPerMillisecond * 100, i)).ToList();
var source = new AvevaShapedSource(series);
var page = await ReverseRawReader.ReadAsync(source, "T", T0, DateTime.MinValue, 3, 65536, T0, default(ReverseWalkHint), Ct);
page.Samples.Select(s => (double)s.Value!).ShouldBe([107_999d, 107_998d, 107_997d]);
source.Reads.ShouldBeLessThanOrEqualTo(30);
source.RowsReturned.ShouldBeLessThanOrEqualTo(30 * (ReverseWalk.DiscoveryRows + 2));
}
/// <summary>
/// #691: the end of a walk proves the rest of the window empty in about one read — after two empty segments the walk
/// asks once for the whole remainder down to the floor — instead of one read per ×4 growth step to 1970 (~14 reads).
/// </summary>
[Fact]
public async Task Running_out_of_rows_proves_the_rest_of_the_window_empty_in_a_few_reads()
{
var series = Enumerable.Range(0, 5).Select(i => At(T0.Ticks + i * TimeSpan.TicksPerSecond, i)).ToList();
var source = new AvevaShapedSource(series);
var page = await ReverseRawReader.ReadAsync(source, "T", T0.AddSeconds(4), DateTime.MinValue, 10, 65536, T0,
default(ReverseWalkHint), Ct);
page.Samples.Select(s => (double)s.Value!).ShouldBe([4d, 3d, 2d, 1d, 0d]);
source.Reads.ShouldBeLessThanOrEqualTo(4, "one segment holding the rows, two empty ones, then the remainder at once");
}
/// <summary>
/// #691: a page's last complete read proves the stretch below its oldest row empty, and the continuation carries it:
/// the next page steps straight over it to the floor instead of walking it again.
/// </summary>
[Fact]
public async Task A_resumed_page_steps_over_the_stretch_the_previous_page_proved_empty()
{
var series = Enumerable.Range(0, 25).Select(i => At(T0.Ticks + i * TimeSpan.TicksPerSecond, i)).ToList();
var source = new AvevaShapedSource(series);
var now = T0.AddHours(1);
// Page 1: an hour of nothing above the rows; the remainder probe reads all 25 and proves (1970, row 0) empty.
var p1 = await ReverseRawReader.ReadAsync(source, "T", now, DateTime.MinValue, 10, 65536, now, default(ReverseWalkHint), Ct);
p1.Samples.Select(s => (double)s.Value!).ShouldBe(Enumerable.Range(15, 10).Reverse().Select(i => (double)i));
p1.Hint.HasEmptyStretch.ShouldBeTrue();
p1.Hint.EmptyFromTicks.ShouldBeLessThanOrEqualTo(DateTime.UnixEpoch.Ticks);
p1.Hint.EmptyToTicks.ShouldBe(Ts(series[0]));
var hint = p1.Hint;
var hi = p1.Samples[^1].SourceTimestampUtc!.Value.AddTicks(-1);
for (var page = 2; page <= 3; page++)
{
source.Reset();
var p = await ReverseRawReader.ReadAsync(source, "T", hi, DateTime.MinValue, 10, 65536, now, hint, Ct);
p.Samples.Count.ShouldBe(page == 2 ? 10 : 5);
source.Reads.ShouldBe(1, $"page {page} reads its rows and steps over the proven-empty stretch to the floor");
hint = p.Hint;
hi = p.Samples[^1].SourceTimestampUtc!.Value.AddTicks(-1);
}
}
/// <summary>
/// #691.4 review: a burst just over the discovery read, sub-millisecond spaced, ending 100 days ago. The search for
/// the top of the burst above the truncated read never re-sends an identical request (halving below the 1 ms request
/// granularity did), and gallops up from the truncated read's newest row instead of bisecting the whole 100-day gap
/// down to the millisecond (was 46 reads).
/// </summary>
[Fact]
public async Task Bisecting_above_a_truncated_read_never_resends_an_identical_request()
{
var end = T0.AddDays(-100);
var count = ReverseWalk.DiscoveryRows + 2;
var series = Enumerable.Range(0, count).Select(i => At(end.Ticks - (count - 1 - i) * 1_000, i)).ToList(); // 0.1 ms apart
var source = new AvevaShapedSource(series);
var page = await ReverseRawReader.ReadAsync(source, "T", T0, DateTime.MinValue, 3, 65536, T0, default(ReverseWalkHint), Ct);
page.Samples.Select(s => (double)s.Value!).ShouldBe([count - 1d, count - 2d, count - 3d]);
source.RepeatedRequests.ShouldBe(0, string.Join("\n", source.Log));
source.Reads.ShouldBeLessThanOrEqualTo(22, string.Join("\n", source.Log));
}
/// <summary>
/// #691.4 re-review (S8): a tie cluster under a carried width. 30 rows at one whole-millisecond timestamp 0.3 s below
/// <c>hi</c>, nothing above, N = 1, a carried 1 s width. The truncated read of the cluster left the width at one tick;
/// once the gap above it was under two milliseconds nothing floored it, so segHi stepped down one tick per read —
/// 10,006 reads of the same millisecond (28 before the gallop).
/// </summary>
[Theory]
[InlineData(30, 1, 0.3, 1.0)]
[InlineData(34, 1, 0.52, 0.48)]
[InlineData(25, 1, 1.2, 1.0)]
[InlineData(40, 3, 3.5, 2.0)]
public async Task A_tie_cluster_under_a_carried_width_is_read_in_a_bounded_number_of_reads(
int ties, int want, double secondsBelow, double hintSeconds)
{
var at = T0.Ticks - (long)(secondsBelow * TimeSpan.TicksPerSecond);
at -= at % TimeSpan.TicksPerMillisecond;
var series = Enumerable.Range(0, ties).Select(i => At(at, i)).ToList();
var source = new AvevaShapedSource(series);
var page = await ReverseRawReader.ReadAsync(source, "T", T0, DateTime.MinValue, (uint)want, 65536, T0,
new ReverseWalkHint((long)(hintSeconds * TimeSpan.TicksPerSecond)), Ct);
page.Samples.Select(s => (double)s.Value!).ShouldBe(Enumerable.Range(ties - want, want).Reverse().Select(i => (double)i));
source.Reads.ShouldBeLessThanOrEqualTo(30, string.Join("\n", source.Log.Take(40)));
source.RepeatedRequests.ShouldBe(0);
}
private static long Ts(DataValueSnapshot s) => s.SourceTimestampUtc!.Value.Ticks;
/// <summary>Random ascending series: bursts, gaps, sub-millisecond stamps and tie clusters.</summary>
private static List<DataValueSnapshot> RandomSeries(Random rng, out int maxCluster)
{
var list = new List<DataValueSnapshot>();
var t = T0.Ticks - rng.Next(0, 3) * TimeSpan.TicksPerDay;
var count = rng.Next(1, 400);
maxCluster = 1;
var v = 0;
while (list.Count < count)
{
t += rng.Next(6) switch
{
0 => rng.Next(1, 10), // sub-millisecond
1 => rng.Next(1, 5) * TimeSpan.TicksPerMillisecond, // dense
2 => rng.Next(1, 3) * TimeSpan.TicksPerSecond,
3 => rng.Next(1, 120) * TimeSpan.TicksPerMinute, // a gap
_ => rng.Next(1, 1000) * TimeSpan.TicksPerMillisecond,
};
var ties = rng.Next(8) == 0 ? rng.Next(2, 12) : 1;
maxCluster = Math.Max(maxCluster, ties);
for (var k = 0; k < ties; k++) list.Add(At(t, v++));
}
return list;
}
/// <summary>AVEVA-shaped forward-only raw source (see the fuzz test's summary).</summary>
private sealed class AvevaShapedSource(IReadOnlyList<DataValueSnapshot> series, bool roundUp = false) : IHistorianDataSource
{
public long MaxCap { get; private set; }
public long RowsReturned { get; private set; }
public int Reads { get; private set; }
public int BadRequests { get; private set; }
/// <summary>Requests that repeated an earlier request exactly (start, end, cap) — read-only, so pure waste.</summary>
public int RepeatedRequests { get; private set; }
private readonly HashSet<(DateTime, DateTime, uint)> _seen = [];
/// <summary>Every request, for failure messages.</summary>
public List<string> Log { get; } = [];
public void Reset() { MaxCap = 0; RowsReturned = 0; Reads = 0; RepeatedRequests = 0; _seen.Clear(); Log.Clear(); }
public bool SupportsReverseRaw => false;
public Task<HistorianRead> ReadRawAsync(string fullReference, DateTime startUtc, DateTime endUtc, uint maxValuesPerNode,
CancellationToken cancellationToken)
{
Reads++;
if (!_seen.Add((startUtc, endUtc, maxValuesPerNode))) RepeatedRequests++;
Log.Add($"[{startUtc:O} .. {endUtc:O}] cap {maxValuesPerNode}");
if (startUtc > endUtc || maxValuesPerNode == 0 || startUtc < DateTime.UnixEpoch) BadRequests++;
MaxCap = Math.Max(MaxCap, maxValuesPerNode);
var stamp = roundUp && startUtc.Ticks % TimeSpan.TicksPerMillisecond != 0
? new DateTime(startUtc.Ticks - startUtc.Ticks % TimeSpan.TicksPerMillisecond + TimeSpan.TicksPerMillisecond, DateTimeKind.Utc)
: startUtc;
var prior = series.LastOrDefault(s => s.SourceTimestampUtc <= startUtc);
var rows = new List<DataValueSnapshot>
{
prior is null
? new DataValueSnapshot(-1d, StatusCodes.BadNoData, stamp, stamp)
: new DataValueSnapshot(-2d, prior.StatusCode, stamp, stamp),
};
rows.AddRange(series.Where(s => s.SourceTimestampUtc >= startUtc && s.SourceTimestampUtc <= endUtc));
var page = rows.Take((int)Math.Min(int.MaxValue, maxValuesPerNode)).ToList();
RowsReturned += page.Count;
return Task.FromResult(new HistorianRead(page, null));
}
public Task<HistorianRead> ReadProcessedAsync(string fullReference, DateTime startUtc, DateTime endUtc, TimeSpan interval,
HistoryAggregateType aggregate, CancellationToken cancellationToken) => throw new NotSupportedException();
public Task<HistorianRead> ReadAtTimeAsync(string fullReference, IReadOnlyList<DateTime> timestampsUtc,
CancellationToken cancellationToken) => throw new NotSupportedException();
public Task<HistoricalEventsResult> ReadEventsAsync(string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents,
CancellationToken cancellationToken) => throw new NotSupportedException();
public HistorianHealthSnapshot GetHealthSnapshot() => NullHistorianDataSource.Instance.GetHealthSnapshot();
public void Dispose() { }
}
}
@@ -104,6 +104,21 @@ public sealed class ServerHistorianOptionsTests
/// #674: the HistoryUpdate future-skew bound defaults to one day and binds from configuration as a TimeSpan;
/// a non-positive value is a deliberate "disabled", not a misconfiguration, so it does not warn.
/// </summary>
/// <summary>#691: the ReturnBounds kill switch defaults on and binds off; neither value warns.</summary>
[Fact]
public void HonorReturnBounds_defaults_to_true_and_binds_false_silently()
{
Assert.True(new ServerHistorianOptions().HonorReturnBounds);
var bound = new ConfigurationBuilder()
.AddInMemoryCollection(new Dictionary<string, string?> { ["ServerHistorian:HonorReturnBounds"] = "false" })
.Build()
.GetSection(ServerHistorianOptions.SectionName)
.Get<ServerHistorianOptions>();
Assert.False(bound!.HonorReturnBounds);
Assert.Empty(new ServerHistorianOptions
{ Enabled = true, Endpoint = "https://h:5222", ApiKey = "histgw_x_y", HonorReturnBounds = false }.Validate());
}
[Fact]
public void HistoryUpdateMaxFutureSkew_defaults_to_one_day_binds_and_zero_is_silent()
{