Compare commits
25 Commits
1f5d310071
...
15ccd9b722
| Author | SHA1 | Date | |
|---|---|---|---|
| 15ccd9b722 | |||
| 942ca7bc14 | |||
| df0c1f3513 | |||
| ba46f4f47a | |||
| 934909b430 | |||
| ebc6b8d4d9 | |||
| f6681544e7 | |||
| 92aab7282c | |||
| 60febbf2dc | |||
| 88141d6ed5 | |||
| 9bb1363cd6 | |||
| 574c3aa1b0 | |||
| d28e3e64f1 | |||
| ccda27d220 | |||
| d277f34a96 | |||
| 186b7d5ea5 | |||
| d33992034d | |||
| 71d031fcd0 | |||
| f0d064a68c | |||
| 830b47f624 | |||
| 7b86059daf | |||
| ab5030174a | |||
| 0e47e887de | |||
| dbffb6bdb4 | |||
| 39552e15b9 |
@@ -1353,8 +1353,9 @@ historian tagname, which is the tag's driver `FullName`.
|
||||
- **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; per-sample aggregates (10, simple definitions) are pushed down to T-SQL with bucket math in **100 ns ticks**; the Part 13 time-weighted seven (TimeAverage/TimeAverage2/Total/Total2/Interpolative, #671 — ⚠️ `Total` is now the time-weighted integral, no longer `SUM` — plus the **duration-based** PercentGood/PercentBad, #680) stream window + 4 bound rows into the shared `TimeWeightedAggregator` (both SQL backends, identical results; reverse `Interpolative` anchored at the later time; window capped by `ServerHistorian:MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations`, percentages included). ⚠️ **#680 status rules (client-visible):** TimeAverage/Total are Uncertain if any part of the interval uses an Uncertain value, is bridged across a Bad sample, is extrapolated past the last sample (⇒ a live tag's latest interval is Uncertain) or is not covered; TimeAverage2/Total2 keep the SDK time-based rule (Uncertain time counts as Good) but never extrapolate past the last sample — that interval is Partial, later ones `BadNoData` (a report-by-exception tag with nothing after its last sample reads `BadNoData` until its next sample arrives). `docs/Historian.md` §"Time-weighted aggregates". ⚠️ **#680 per-sample (client-visible):** `Count` counts **Good** samples only (value-based status; an all-Bad interval is generic Bad, value null) and an empty interval **inside the tag's data range** is `0`/Good (outside it still `BadNoData`); `Start`/`End` return the **raw** first/last sample of any quality — its own timestamp and status, value null when Bad; `Delta` is Uncertain only when the interval's first or last sample is unusable; ⚠️ #684: `Average`/`Minimum`/`Maximum`/`Range`/`*ActualTime` use **Good samples only** (was Good + Uncertain; an Uncertain-only interval is `BadNoData`), Min/Max-family status Uncertain on a Bad sample, an Uncertain value beyond the Good extreme (Part 13 — the SDK ignores that) or an Uncertain non-numeric sample, and only a **Good** String/DateTime sample refuses them (`BadAggregateInvalidInputs`; Delta still refuses any usable one); and every per-sample aggregate but `Average` sets **end-of-data Partial** on the interval holding the last sample (forward incl. its earlier edge, reverse strictly inside), never on a Bad result. The data-range lookup is a key seek on SQL Server and a `tag` symbol-index seek on QuestDB (#683 — `{p}sample.tag` is `SYMBOL INDEX`; the bootstrapper ADDs it to a pre-#683 table, idempotent), read BEFORE the bucket statement so a racing append can never yield a Good `Count` of 0 (a concurrent HistoryUpdate delete is the mirror case and still can, transiently). `docs/Historian.md` §"Aggregates". Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. **Retention (#667):** opt-in `SqlServer:RetentionDays` (+ `PurgeInterval` 1h, `PurgeBatchSize` 4000) -> `HistorianRetentionService` purges old `hist.Sample` (per tag) / `hist.Event` in `DELETE TOP(n)` batches, Primary-gated on `IHistorizationRoleView`, one purger per DB via `sp_getapplock`; counter `otopcua.historian.retention.purged{table}`. Optional `IHistorianBackendProvider.CreateRetention` (default null). `docs/Historian.md` §Retention.
|
||||
- **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend".
|
||||
- ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); on forward reads Partial also marks the interval the data starts inside (not Average — the SDK's rule, forward-only); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates.
|
||||
- ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` serves it from FORWARD reads: uncapped ⇒ ONE forward read reversed in memory; capped ⇒ a bounded walk back in segments (ms-aligned request bounds, width aimed from observed density and carried page to page in the continuation point, ≤ `MaxTieClusterOverfetch + 2` rows per read — a full read is re-read, never answered truncated; only one timestamp holding more than max(1, `MaxTieClusterOverfetch`) samples ⇒ `BadTooManyOperations`; walks no later than now + 1 day and never before the Unix epoch). The raw/events tie-path oversized-cluster refusal is now `BadTooManyOperations` too (was `BadHistoryOperationUnsupported`). Uncapped Gateway raw reads (forward too) used to send `max_values = 0`, which the client/gateway reject — now unbounded (`int.MaxValue`; a ceiling refusal ⇒ `BadTooManyOperations`), and a cancelled read is no longer a backend failure. Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events still differ (end-only forward / explicit reverse refused on Gateway, #675). `docs/Historian.md` §"Open-ended and reverse raw reads".
|
||||
- ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` serves it from FORWARD reads: uncapped ⇒ ONE forward read reversed in memory; capped ⇒ a bounded walk back in segments (ms-aligned request bounds, width aimed from observed density at ≈ N + 8 rows and carried page to page in the continuation point — #689: it used to aim at half `MaxTieClusterOverfetch`, ~32k rows per page on a dense tag; a read asks for `2N + 18` rows (a discovery segment ≤ 1,026), `MaxTieClusterOverfetch` is only the tie-cluster ceiling — a full read is re-read, never answered truncated; only more than max(1, `MaxTieClusterOverfetch`) samples within ONE MILLISECOND ⇒ `BadTooManyOperations` (per ms, not per timestamp — requests are ms-rounded, and letting a segment narrow below 1 ms stepped a page through one ms a tick per read, ~10k reads); walks no later than now + 1 day and never before the Unix epoch; #691: after two empty segments it asks once for the whole remainder, and the stretch a page proved empty rides the continuation point — the last pages of a deep page-through no longer each walk ~14 reads to 1970). The raw/events tie-path oversized-cluster refusal is now `BadTooManyOperations` too (was `BadHistoryOperationUnsupported`). Uncapped Gateway raw reads (forward too) used to send `max_values = 0`, which the client/gateway reject — now unbounded (`int.MaxValue`; a ceiling refusal ⇒ `BadTooManyOperations`), and a cancelled read is no longer a backend failure. Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events follow since #691.4: a reverse/end-only capped events read on Gateway (`SupportsReverseEvents` false; Null now reports `true` and answers empty) is walked back the same way (ties sorted by EventId — the gateway orders by time only; one extra event found ⇒ continuation point) — it used to be refused `BadHistoryOperationUnsupported` (explicit reverse) or answered with the OLDEST N ascending (end-only); needs the gateway's `RuntimeDb:EventReadMaxRows` 0/unlimited (it truncates silently). Uncapped Gateway events reads (forward too) used to throw "Array dimensions exceeded" (result list pre-sized to `int.MaxValue`) — fixed. Above a truncated read the walk gallops up from the newest row seen, then bisects (never below 1 ms): a burst that ended 100 days ago ≈ 18 reads, a dense tag that stopped days ago ≈ 26. `docs/Historian.md` §"Open-ended and reverse raw reads".
|
||||
- **Advertised aggregates (#682):** `IHistorianDataSource.SupportedAggregates` (Gateway 4 via `GatewayHistorianDataSource.ServedAggregates`, SqlServer/QuestDB 17 via `AggregateFamilies.Served`, Null **none**; a null set ⇒ all) is the one authority for `HistoryServerCapabilities/AggregateFunctions` (i=11201 — only its folder references are trimmed) and the up-front `BadAggregateNotSupported` in HistoryRead-Processed. ⚠️ `ServerCapabilities/AggregateFunctions` (i=2997) is deliberately NOT trimmed: it keeps the SDK's 37 because live `AggregateFilter` subscriptions still compute them all. ⚠️ Historian disabled ⇒ Processed is now `BadAggregateNotSupported`, not `GoodNoData`. `docs/Historian.md` §Advertised aggregates.
|
||||
- **Raw `ReturnBounds` is honoured on every backend (#691, client-visible):** Part 11 bounding values (§4.6 Table 1 is the test oracle) are composed in the server by `HistoryBounds` from each backend's raw reads — "at or before" = capped reverse read (native on SQL backends, the #686 `ReverseRawReader` walk-back on Gateway, never AVEVA's synthetic boundary row), "at or after" = capped forward read. Bounds are stored samples of any quality, count toward `NumValuesPerNode`, start bound on the first page / end bound on the last (pending end bound rides the continuation point); missing ⇒ `Bad_BoundNotFound` (null, at the bound time; a one-sided read's open side ±1 s from the previous value). ⚠️ **Breaking for SDK-default clients:** the .NET SDK's `new ReadRawModifiedDetails()` defaults BOTH `ReturnBounds = true` AND `IsReadModified = true` — a client that cleared only `IsReadModified` now gets bounds, and one that set neither is a modified read with bounds ⇒ `Bad_InvalidArgument` (§6.5.3.3); set both explicitly (Client.Shared does; tests building details must too). No data AND no bound found ⇒ `GoodNoData`, no values (the pre-bounds answer). On Gateway the at-or-after lookup ends at now + 1 day (never a year-9999 end to AVEVA). Kill switch `ServerHistorian:HonorReturnBounds` (default `true`; `false` = pre-#691). `docs/Historian.md` §Bounding values.
|
||||
- **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good.
|
||||
- **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node.
|
||||
- **HistoryUpdate** (SqlServer + QuestDb; Gateway ⇒ `BadHistoryOperationUnsupported`): Insert/Replace/Update/DeleteRaw/DeleteAtTime, **Primary-only** since #674 (off-Primary ⇒ `BadInvalidState`, after authorization AND after planning — an unservable request (Remove / `IsDeleteModified` / not historized) stays `BadHistoryOperationUnsupported` off-Primary; the verdict is `IHistoryUpdateRoleView` = the data-plane verdict but **closed until this node's first redundancy snapshot**, so a pair cannot both accept in the boot window; admin-only nodes register no view ⇒ no gate; reads ungated), items with a `SourceTimestamp` past now + `ServerHistorian:HistoryUpdateMaxFutureSkew` (default 1 day, ≤0 disables) ⇒ `BadOutOfRange` in the node manager (every backend), gated in both ACL modes — flag off needs the `HistoryUpdate` role from a **Cluster-scoped** grant (+ key re-check), flag on uses `DataPlaneAuthorizer`'s HistoryUpdate operation; `WriteOperate` never implies it. The `HistoryWrite` AccessLevel bit is set only when an updater is wired at materialise time. `docs/Historian.md` §HistoryUpdate, `docs/security.md`. ⚠️ **SqlServer schema v1 is accepted (#674)** with `AutoCreateSchema=false`: writes/reads/events/retention work; modified reads + HistoryUpdate ⇒ node-level `BadHistoryOperationUnsupported` on every arm (the updater throws `NotSupportedException`, nothing sent; the `HistoryWrite` bit is still advertised — not knowable at materialise time), retention skips `SampleModified`, one Warning per boot (`SqlServerSchemaBootstrapper.SupportsModifiedTrail`; `RequiredVersion` = 1, `ModifiedTrailVersion` = 2). QuestDB: keep `cairo.ttl.use.wall.clock=true`; a QuestDB trail INSERT retried after a client timeout duplicates (no DEDUP key).
|
||||
@@ -1364,8 +1365,8 @@ historian tagname, which is the tag's driver `FullName`.
|
||||
|
||||
The server dispatches all OPC UA HistoryRead to the registered `IHistorianDataSource` — the
|
||||
`GatewayHistorianDataSource` read client when enabled, else the `NullHistorianDataSource` default
|
||||
(historized nodes return `GoodNoData` for Raw/AtTime — Processed is `BadAggregateNotSupported` since #682). Supported variants: Raw (⚠️ #687, client-visible: Gateway raw reads no longer return AVEVA's synthetic start-boundary row — asked from 1 ms early (epoch-floored) with `NumValuesPerNode + 16` rows (unclamped; 0 ⇒ `int.MaxValue`) and filtered by timestamp — so an empty window is `GoodNoData`, not a fabricated value at StartTime; the forward end is now exclusive (a point read inclusive); pre-window rows that crowd the budget trigger ONE re-read with room for them, else `BadTooManyOperations`; `ReturnBounds` is ignored on every backend), Processed
|
||||
(backend-dependent: SqlServer/QuestDb serve seventeen; Gateway twelve = `GatewayHistorianDataSource.ServedAggregates` — ⚠️ #681, client-visible: the ten per-sample ones (Average, Count, Min/Max, Range, Start/End/Delta, Min/MaxActualTime) are evaluated from RAW gateway reads by the shared `Sql.Common` evaluator — identical to the SQL backends, Calculated bits, interval-start stamps, #677 reverse; Min/Max are no longer AVEVA modes, the read moves every raw sample of the window (a month of 1 Hz ⇒ `BadTooManyOperations` at the default `MaxTimeWeightedRawRows` cap), and every gateway stream is consumed to completion (an abandoned one makes the gateway evict its shared historian session) — plus TimeAverage/Total via AVEVA modes (interval-start stamped + Calculated; ⚠️ #685: AVEVA's extra pre-window row is dropped ⇒ one row per interval in `[Start, End)`, and an Uncertain (opc 64) interval keeps its value as `UncertainDataSubNormal` — AVEVA's TWA excludes NULL-gap time where the SQL backends bridge it; per-interval values still need gateway fix historiangw#27); the other time-weighted five stay `BadAggregateNotSupported` because the gateway has no prior-sample read for Part 13 bounds; `docs/Historian.md` §Aggregate set), and AtTime over historized variable nodes; Events over
|
||||
(historized nodes return `GoodNoData` for Raw/AtTime — Processed is `BadAggregateNotSupported` since #682). Supported variants: Raw (⚠️ #687, client-visible: Gateway raw reads no longer return AVEVA's synthetic start-boundary row — asked from 1 ms early (epoch-floored) with `NumValuesPerNode + 16` rows (unclamped; 0 ⇒ `int.MaxValue`) and filtered by timestamp — so an empty window is `GoodNoData`, not a fabricated value at StartTime; the forward end is now exclusive (a point read inclusive); pre-window rows that crowd the budget trigger ONE re-read with room for them, else `BadTooManyOperations`; `ReturnBounds` is ignored on every backend; ⚠️ #690: a Gateway Bad row — a stored NULL, which histsdk sends as value 0 — now reads value **null** + Bad on Raw and AtTime, as on the SQL backends; ⚠️ #691.3: a strict gateway (`AllowStreamingBeyondCeiling=false`) that refuses a raw budget is re-asked ONCE at the ceiling its refusal names — raw and per-sample reads alike; the ceiling is then remembered per data source (`GatewayRowCeiling`: refusals count as gateway errors), re-learned from a refusal when lowered, and re-probed once at the full budget when a ceiling-bounded read comes back full (raised / streaming) — so an uncapped (`NumValuesPerNode = 0`) read of a small window is served; a ceiling-bounded stream that is not provably complete is `BadTooManyOperations`, never a truncated answer), Processed
|
||||
(backend-dependent: SqlServer/QuestDb serve seventeen; Gateway twelve = `GatewayHistorianDataSource.ServedAggregates` — ⚠️ #681, client-visible: the ten per-sample ones (Average, Count, Min/Max, Range, Start/End/Delta, Min/MaxActualTime) are evaluated from RAW gateway reads by the shared `Sql.Common` evaluator — identical to the SQL backends, Calculated bits, interval-start stamps, #677 reverse; Min/Max are no longer AVEVA modes, the read moves every raw sample of the window (a month of 1 Hz ⇒ `BadTooManyOperations` at the default `MaxTimeWeightedRawRows` cap), and every gateway stream is consumed to completion (an abandoned one makes the gateway evict its shared historian session) — plus TimeAverage/Total via AVEVA modes (interval-start stamped + Calculated; ⚠️ #685: AVEVA's extra pre-window row is dropped ⇒ one row per interval in `[Start, End)`, and an Uncertain (opc 64) interval keeps its value as `UncertainDataSubNormal` — AVEVA's TWA excludes NULL-gap time where the SQL backends bridge it; per-interval values still need gateway fix historiangw#27; ⚠️ #691.2: native reads use the SQL backends' grid in both directions via forward requests only — a REVERSE read (refused by the gateway before) is anchored at its later time (#677), and the clipped interval (earliest reverse, LATEST forward — which used to be averaged past EndTime) is its own exact-span read marked Partial unless Bad; an interval wider than the window is sent as the window, interval 0 as one interval; more than 100,000 intervals is refused up front (`BadHistoryOperationUnsupported`, as on the SQL backends — forward used to be served up to the gateway's 1,000,000); until historiangw#27 lands both directions return Good-status wrong values); the other time-weighted five stay `BadAggregateNotSupported` because the gateway has no prior-sample read for Part 13 bounds; `docs/Historian.md` §Aggregate set), and AtTime over historized variable nodes; Events over
|
||||
alarm-owning equipment-folder event-notifier nodes. Reads are ungated (served from any redundancy node);
|
||||
authorization uses the standard `AccessLevels.HistoryRead` bit set at materialization.
|
||||
|
||||
@@ -1381,6 +1382,7 @@ authorization uses the standard `AccessLevels.HistoryRead` bit set at materializ
|
||||
| `CaCertificatePath` | `null` | PEM CA file pinning the gateway TLS chain; null/empty uses the OS trust store |
|
||||
| `CallTimeout` | `00:00:30` | Per-call deadline for each unary gateway read |
|
||||
| `MaxTieClusterOverfetch` | `65536` | Bounded over-fetch the HistoryRead-Raw paging uses to page within an oversized same-timestamp tie cluster (retained from the prior backend) |
|
||||
| `HonorReturnBounds` | `true` | #691 kill switch: `false` ignores HistoryRead-Raw `ReturnBounds` as before (no bounding values, no modified-read `Bad_InvalidArgument`) |
|
||||
| `HistoryReadBatchParallelism` | `4` | Max nodes served concurrently within one HistoryRead batch (03/S3); `⌈N/P⌉ × CallTimeout` worst case instead of sequential |
|
||||
| `MaxConcurrentHistoryReadBatches` | `16` | Process-wide concurrent HistoryRead batch cap (03/S3); flood degrades to `BadTooManyOperations`. Total in-flight reads capped at 16 × 4 = 64 |
|
||||
| `MaxTimeWeightedRawRows` | `1000000` | `SqlServer`/`QuestDb` (#671): raw in-window rows one time-weighted aggregate read (incl. `Total` and, since #680, PercentGood/PercentBad) may stream into the C# evaluator — and, since #681, every `Gateway` per-sample read (raw-evaluated); window query capped at this + 1, past it ⇒ `BadTooManyOperations` for that node. ≤ 0 = unbounded. Gateway (#688): the `cap + 18` budget is clamped at the gateway's default `MaxValuesCeiling` (1,000,000) while the cap is ≤ it, so the default no longer trips a strict (`AllowStreamingBeyondCeiling=false`) gateway |
|
||||
@@ -1455,6 +1457,9 @@ The target HistorianGateway OtOpcUa points at **must** run with:
|
||||
|
||||
- `RuntimeDb:Enabled=true` — enables the `WriteLiveValues` SQL live path (continuous historization).
|
||||
- `RuntimeDb:EventReadsEnabled=true` — enables `ReadEvents` from `Runtime.dbo.Events` (alarm-history reads).
|
||||
- `RuntimeDb:EventReadMaxRows=0` (its default, unlimited) — or larger than the reverse-events walk's biggest read
|
||||
(`2·MaxTieClusterOverfetch + 4`). The cap truncates **silently**, so a walked-back reverse events read (#691.4) would
|
||||
lose the newest events of a segment it cut.
|
||||
- An API key carrying scopes **`historian:read`**, **`historian:write`**, **`historian:tags:write`**.
|
||||
|
||||
### Migration note (deployments upgrading from the Wonderware backend)
|
||||
|
||||
+187
-46
@@ -111,6 +111,7 @@ Processed read is `BadAggregateNotSupported` and no history aggregate functions
|
||||
"CaCertificatePath": null,
|
||||
"CallTimeout": "00:00:30",
|
||||
"MaxTieClusterOverfetch": 65536,
|
||||
"HonorReturnBounds": true,
|
||||
"HistoryReadBatchParallelism": 4,
|
||||
"MaxConcurrentHistoryReadBatches": 16,
|
||||
"HistoryReadDeadline": "00:01:00",
|
||||
@@ -144,9 +145,10 @@ Processed read is `BadAggregateNotSupported` and no history aggregate functions
|
||||
| `CaCertificatePath` | string\|null | `null` | **Gateway backend only.** PEM CA file pinning the gateway's TLS chain. Null/empty uses the OS trust store. |
|
||||
| `CallTimeout` | TimeSpan | `00:00:30` | **Gateway backend only.** Per-call deadline applied to each unary gateway read. A read for a tag the historian does not yet know about — e.g. a freshly-historized tag that has **not** been provisioned (see [Tag auto-provisioning](#tag-auto-provisioning-ensuretags)) — can run to this **full** deadline before the gateway errors, so an unprovisioned-tag HistoryRead can block for ~30 s. Lower this to fail faster, at the cost of truncating legitimately long reads. |
|
||||
| `MaxTieClusterOverfetch` | int | `65536` | (Node-manager paging; all backends — a `SqlServer` PK rules out tie clusters, so it is moot there.) Maximum samples the server will fetch in one shot to page through a tie cluster (multiple samples sharing one `SourceTimestamp`). A cluster larger than this ceiling fails `BadTooManyOperations` (#686; was `BadHistoryOperationUnsupported`). Also the per-read row budget of a Gateway reverse raw read (walked back from forward reads, #686): one timestamp holding more samples is refused `BadTooManyOperations`. Values below 1 count as 1. Raise to handle abnormally large tie clusters; the default covers all normal-data cases. |
|
||||
| `HonorReturnBounds` | bool | `true` | (#691, all backends.) Whether HistoryRead-Raw honours `ReturnBounds` (Part 11 bounding values, [Bounding values](#bounding-values-returnbounds-691)). `false` is the kill switch back to the pre-#691 behaviour: the flag is ignored (stored samples only) and a modified read carrying it is not refused `Bad_InvalidArgument`. Note the .NET SDK defaults both `ReturnBounds` and `IsReadModified` to `true`. |
|
||||
| `HistoryReadBatchParallelism` | int | `4` | Max historized nodes served **concurrently** within one HistoryRead batch (arch-review 03/S3). The SDK's `HistoryRead*` override surface is synchronous, so each arm bridges once at its boundary; this turns per-node serving from sequential (`N × CallTimeout` worst case) into `⌈N / P⌉ × CallTimeout`. Also caps the gateway-load multiplication a single client can command. `≤ 1` serves sequentially. |
|
||||
| `MaxConcurrentHistoryReadBatches` | int | `16` | Process-wide limit on HistoryRead (and HistoryUpdate) batches served **concurrently** (arch-review 03/S3). A flood of history clients degrades to `BadTooManyOperations` rather than exhausting the SDK request-thread pool. With `HistoryReadBatchParallelism` this caps total in-flight gateway reads at `MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism` (default 16 × 4 = 64). A batch that cannot get a slot within `min(HistoryReadDeadline, 5 s)` fails every handle `BadTooManyOperations`. |
|
||||
| `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` (#671), and the `Gateway` per-sample aggregates (#681 — every per-sample read there streams raw samples, so the cap applies to the window's raw rows; see [Aggregate set](#aggregate-set-and-badaggregatenotsupported-285)). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative, and — since #680 — **PercentGood/PercentBad**) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped on the SQL backends (pushed down). 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). `Gateway` (#688): the row budget it asks the gateway for (`cap + 18`) is clamped at the gateway's default `MaxValuesCeiling` (1,000,000) while the cap is at or under it, so the default works against a gateway with `AllowStreamingBeyondCeiling=false`; against a gateway with a *lower* ceiling, set this to `ceiling − 18`. |
|
||||
| `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` (#671), and the `Gateway` per-sample aggregates (#681 — every per-sample read there streams raw samples, so the cap applies to the window's raw rows; see [Aggregate set](#aggregate-set-and-badaggregatenotsupported-285)). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative, and — since #680 — **PercentGood/PercentBad**) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped on the SQL backends (pushed down). 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). `Gateway` (#688): the row budget it asks the gateway for (`cap + 18`) is clamped at the gateway's default `MaxValuesCeiling` (1,000,000) while the cap is at or under it, so the default works against a gateway with `AllowStreamingBeyondCeiling=false`; against a gateway with a *lower* ceiling the refused budget is retried once at the ceiling the gateway names (#691.3), so no change is needed (`ceiling − 18` only saves that round-trip). |
|
||||
| `HistoryReadDeadline` | TimeSpan | `00:01:00` | Per-request wall-clock deadline for a HistoryRead (and HistoryUpdate) batch (arch-review 03/S3). A node still in flight at the deadline surfaces `BadTimeout`, so one request can never hold an SDK request thread longer than this regardless of node count. Non-positive = unbounded (a `Validate()` warning). |
|
||||
| `HistoryUpdateMaxFutureSkew` | TimeSpan | `1.00:00:00` | #674. How far past the server clock a HistoryUpdate (Insert / Replace / Update) item's `SourceTimestamp` may lie; an item beyond `now + skew` is refused **`BadOutOfRange`** by the node manager before any backend sees it (every backend). Guards against a far-future typo (a 2062 timestamp). The live recorder is not bounded. Non-positive disables the bound (no warning — a deliberate choice). See [HistoryUpdate](#historyupdate). |
|
||||
|
||||
@@ -639,33 +641,59 @@ backend:
|
||||
`[⌊segLo⌋, ⌈segHi + 1 tick⌉]`, both rounded **outward to the millisecond**, and filtered on the samples' own
|
||||
timestamps — exact for an inclusive or exclusive forward end, and the row AVEVA fabricates at a query's (possibly
|
||||
rounded-up) start can never land inside the segment. The first segment is **1 minute** (or the width the previous
|
||||
page ended on — it rides the continuation point); each next width is aimed at about half the row budget from the
|
||||
density just observed, growing at most ×4. So "the last N values" of a live tag is normally one read, a dense tag
|
||||
costs about one read per page, and sparse history is found in a few dozen tiny ones.
|
||||
- **Never answered from a truncated read.** The row budget is `M = max(1, MaxTieClusterOverfetch)`; a segment read asks
|
||||
for `M + 2` rows (room for the fabricated row and one to see past the segment). Rows stream ascending, so a read is
|
||||
complete when it came back short or its last row is past the segment; otherwise it is discarded — re-read once with
|
||||
room for the rows before the segment that crowded it (a neighbouring tie cluster, at most `2M + 4` rows), or narrowed
|
||||
to the density it showed. Only **one timestamp holding more than `M` samples** is refused, **`BadTooManyOperations`**
|
||||
(a request limit, not a backend fault); a cluster of exactly `M`, or neighbouring clusters each within `M`, is served.
|
||||
page ended on — it rides the continuation point); each next width is aimed from the density just observed at about
|
||||
half the **page budget** `2N + 16` (≈ N + 8 rows), growing at most ×4. So "the last N values" of a live tag is
|
||||
normally one read of about N rows, a dense tag costs about one N-sized read per page, and sparse history is found in
|
||||
a few dozen tiny ones.
|
||||
- **Read sizes (#689).** A segment read asks for its budget + 2 rows (room for the fabricated row and one to see past
|
||||
the segment). The budget is the page budget `2N + 16`, except a **discovery** segment — the first of a page with no
|
||||
carried width, or one after an empty segment, whose ×4 growth may overshoot into dense data — which may read up to
|
||||
**1,024** rows. The tie budget `M = max(1, MaxTieClusterOverfetch)` is only a **ceiling**: a single timestamp holding
|
||||
more rows than the segment budget escalates to `M`.
|
||||
⚠️ Before #689 every segment was aimed at half of `M` (≈ 32k rows by default) and that width was carried page to
|
||||
page, so on a 1 Hz tag from about page 5 every page of a "latest N" read fetched ~32,770 rows and took 2–5 s to
|
||||
return N.
|
||||
- **Never answered from a truncated read.** Rows stream ascending, so a read is complete when it came back short or its
|
||||
last row is past the segment; otherwise it is discarded — re-read once with room for the rows before the segment
|
||||
that crowded it (a neighbouring tie cluster, at most twice the budget + 4 rows), or narrowed: to the density it
|
||||
showed, and never again below the newest row it saw until what lies above that row is read. The gap above it is
|
||||
searched: a segment at that density ending on the top (data running right up to it); if empty, a **gallop** up from
|
||||
that row (`(row + d, top]`, `d` ×4 per step, following each truncated read) until a read comes back empty, then a
|
||||
**bisection** of the bracket — never narrower than the 1 ms request granularity, which only re-sent the same request.
|
||||
Measured: a dense tag that stopped two days ago ≈ **26** reads, a burst of just over 1,024 rows that ended 100 days ago
|
||||
≈ **18** (was 46 bisecting the whole gap) — rather than re-reading the oldest rows of a burst over and over. Only **more than `M` samples within one millisecond** are refused,
|
||||
**`BadTooManyOperations`** (a request limit, not a backend fault); a cluster of exactly `M`, or neighbouring
|
||||
clusters each within `M` and a millisecond or more apart, is served. ⚠️ The unit is the **millisecond**, not the
|
||||
timestamp (#691.4 re-review): requests are rounded to the millisecond, so a narrower segment is the same request —
|
||||
letting the width fall below it once stepped a page through one millisecond a tick at a time (~10,000 reads). Exact
|
||||
for the Gateway, which stores milliseconds; a sub-millisecond backend walked this way would refuse more than `M`
|
||||
samples inside one millisecond even when no single timestamp holds that many.
|
||||
- **Proving the rest empty (#691).** Walking down through empty history to the floor used to cost one read per ×4
|
||||
growth step — ~14–16 reads (~2.5 s live) on **every** last page of a deep page-through. Now, after two consecutive
|
||||
empty segments, the walk asks **once per page** for the whole remainder down to the floor at the page budget: a
|
||||
complete answer ends the walk in one read; a truncated one (deep history) is discarded and the walk carries on. And
|
||||
the stretch the page's last complete read proved empty below its oldest row (often all the way to the floor) rides
|
||||
the continuation point, so the next page steps straight over it. Data backfilled into such a stretch between pages
|
||||
is not seen by that read — paging was never a snapshot (the forward path misses data inserted behind its cursor too).
|
||||
- **Bounds.** The walk starts no later than **now + 1 day** (a `StartTime = DateTime.MaxValue` read does not send
|
||||
year-9999 query starts) and never reads before the **Unix epoch** (AVEVA holds nothing earlier): a walked-back read
|
||||
returns only samples strictly after 1970-01-01T00:00:00Z and no later than now + 1 day.
|
||||
|
||||
Paging is unchanged — a reverse cursor resumes as a fresh walk from its resume point.
|
||||
Paging is otherwise unchanged — a reverse cursor resumes as a fresh walk from its resume point, carrying the width and
|
||||
the proven-empty stretch (session-held cursor state, never serialized, so the new fields need no versioning).
|
||||
- **Null** reports `true` (empty in either direction).
|
||||
|
||||
> ⚠️ **Client-visible (#686).** Since #669 an end-only capped raw read, and any explicit reverse raw read, on the Gateway
|
||||
> backend failed the node (`BadHistoryOperationUnsupported`) and was counted as a historian backend failure (the gateway
|
||||
> refused `start > end`). Before #669 an end-only read on Gateway returned the **oldest** N values ascending. Both now return
|
||||
> the Part 11 answer — the latest values at or before `EndTime`, newest first. **Events differ:** an end-only events read on
|
||||
> Gateway still keeps its forward meaning and an explicit reverse events read is refused (see below).
|
||||
> the Part 11 answer — the latest values at or before `EndTime`, newest first. **Events** follow since #691.4 (see below).
|
||||
>
|
||||
> ⚠️ **Also fixed with #686: uncapped raw reads on Gateway.** `NumValuesPerNode = 0` ("all values") used to be forwarded as
|
||||
> `max_values = 0`, which the client package and the gateway reject — every uncapped raw read (forward too) failed the node
|
||||
> and counted as a backend failure. It is now sent unbounded (`int.MaxValue`); a gateway configured with
|
||||
> `AllowStreamingBeyondCeiling=false` refuses that over its `MaxValuesCeiling`, which is answered **`BadTooManyOperations`**
|
||||
> (not a backend fault). A cancelled read (client cancel / `HistoryReadDeadline`) is no longer counted as a backend failure.
|
||||
> `AllowStreamingBeyondCeiling=false` refuses that over its `MaxValuesCeiling`; since #691.3 the read is retried once at the
|
||||
> ceiling the refusal names and answered when provably complete, else **`BadTooManyOperations`** (not a backend fault) — see
|
||||
> [#687](#gateway-raw-reads-return-stored-samples-only-687) "Strict gateway". A cancelled read (client cancel / `HistoryReadDeadline`) is no longer counted as a backend failure.
|
||||
|
||||
Continuation points are bound to the OPC UA session (the SDK's
|
||||
`ServerConfiguration.MaxHistoryContinuationPoints` cap, default 100, with oldest-eviction; points
|
||||
@@ -686,7 +714,12 @@ data.
|
||||
the gateway is asked from **1 ms before** the window (never before the Unix epoch, the start of the historian's time encoding),
|
||||
so the boundary row lands before it, and only rows `RawAggregateReader.IsWindowSample` are returned. The decision is by
|
||||
**timestamp**, not by value or quality — a stored NULL sample looks exactly like the no-data boundary row on the wire (value 0,
|
||||
Bad), and a stored NULL inside the window is still returned (Bad), as it is a stored sample. A row stamped **at or before the
|
||||
Bad), and a stored NULL inside the window is still returned (Bad), as it is a stored sample. ⚠️ **Since #690 it is returned
|
||||
with a null value** (client-visible): histsdk puts a value on every row, so the NULL used to come back as value `0` + Bad (`""`
|
||||
on a string tag), a fabricated 0 for any client that reads the value without the status. `SampleMapper.ToSnapshot` drops the
|
||||
value of every row whose mapped status is **Bad** (AVEVA stores a Bad-quality sample as NULL, so a Bad row never carries a real
|
||||
value) — raw reads and AtTime alike, matching SqlServer/QuestDB (an Empty sample reads null + Bad). Uncertain rows keep their
|
||||
value; the raw-evaluated aggregates were already unaffected (a Bad sample is never usable, and Start/End null a Bad value). A row stamped **at or before the
|
||||
epoch** is never a stored sample (AVEVA stores nothing there — the #686 walk-back's floor too): a window starting at or before
|
||||
the epoch gets no margin, so AVEVA's no-data row lands exactly at its start; raw reads and the per-sample aggregates both drop it.
|
||||
- **The forward end is exclusive** (`[StartTime, EndTime)`, the `IHistorianDataSource` contract and what the SQL backends return;
|
||||
@@ -696,8 +729,23 @@ data.
|
||||
- **Row budget.** `NumValuesPerNode = 0` ("all values") asks the gateway for an unbounded `max_values` (`int.MaxValue` — the client
|
||||
package and the gateway reject 0, #686). A cap asks for `NumValuesPerNode + 16` rows, **not** clamped at the gateway's default
|
||||
ceiling (#688 clamps only the per-sample aggregate budget): AVEVA always emits the boundary row, so a clamped budget would make
|
||||
every page size from 999,985 to 1,000,000 come back short. A gateway with `AllowStreamingBeyondCeiling=false` refuses an
|
||||
over-ceiling budget instead ⇒ `BadTooManyOperations` (not a backend fault); a default gateway serves it.
|
||||
every page size from 999,985 to 1,000,000 come back short. A default gateway serves any budget.
|
||||
- ⚠️ **Strict gateway (#691.3, client-visible).** A gateway with `AllowStreamingBeyondCeiling=false` refuses a budget over its
|
||||
`MaxValuesCeiling` (before streaming) and names the ceiling in the refusal; the gateway has no RPC that reports it. The read is
|
||||
then **re-issued once with the ceiling as the budget**, and the ceiling is **remembered per data source** (`GatewayRowCeiling`,
|
||||
shared by raw and per-sample reads): every refused attempt counts as an error in the gateway's own outcome metrics, so after the
|
||||
first refusal later reads ask for at most the learned ceiling and are not refused. A **lowered** ceiling is re-learned from the
|
||||
refusal of the remembered one. A **raised** ceiling (or a switch to streaming) is noticed by the only reads it matters to: when a
|
||||
read bounded by the remembered ceiling comes back full without proving completeness, it is re-probed **once** at its full budget
|
||||
— served ⇒ the ceiling is forgotten; refused naming a larger ceiling ⇒ that one is learned and the read re-issued at it; refused
|
||||
naming the same ceiling ⇒ the bounded result stands. No timer: a TTL would only add refusals for reads that already fit; the cost
|
||||
of the rule is one refused re-probe per genuinely over-ceiling read (which fails either way). So an uncapped read (`NumValuesPerNode = 0`, which used to be refused outright on a strict gateway, even for a 10-row
|
||||
window) and a page size within 16 of the ceiling are now served when the window fits. A ceiling-bounded read is answered only
|
||||
when it is **provably complete** — the stream came back short of the ceiling, or reached `EndTime` (or, capped, filled the page);
|
||||
a stream that delivered the whole ceiling short of `EndTime` may be truncated and fails **`BadTooManyOperations`** (not a backend
|
||||
fault) — set `NumValuesPerNode` to page, or narrow the window. A refusal whose text carries no number falls back to the gateway's
|
||||
default ceiling (1,000,000). Every stream is still drained. The #686 reverse walk-back (segment budget `MaxTieClusterOverfetch + 2`)
|
||||
inherits this: against a ceiling below that budget a dense segment fails `BadTooManyOperations` instead of being narrowed.
|
||||
- **Paging (#669) is unaffected.** A full page still holds exactly `NumValuesPerNode` stored samples and carries a continuation
|
||||
point, and a resumed page (which re-reads from the last returned timestamp inclusive, and the tie-cluster over-fetch
|
||||
`start == end` read) no longer gets a fabricated row at its resume point. A read is complete when the stream came back short,
|
||||
@@ -706,10 +754,63 @@ data.
|
||||
asking for `NumValuesPerNode + m + 16` (m = the pre-window rows the first read saw, unclamped). Only if that re-read is still
|
||||
short because of pre-window rows does the node fail `BadTooManyOperations` (not a backend fault) rather than answer a short page
|
||||
that would end the client's paging early. Every gateway stream is still drained to completion.
|
||||
- **`ReturnBounds` is not honoured** — on any backend: the server does not pass it to the data source, so no bounding values are
|
||||
returned (see [Known limitations](#known-limitations)). The AVEVA boundary row is not used as a bound either: it is stamped at the
|
||||
query start rather than carrying the time of the real prior sample, so it could only ever be a Part 11 *interpolated* bound, which
|
||||
this backend does not compute.
|
||||
- **`ReturnBounds` is served by the server, not by this row** (#691, [Bounding values](#bounding-values-returnbounds-691)). The AVEVA
|
||||
boundary row is never used as a bound: it is stamped at the query start rather than carrying the time of the real prior sample, so
|
||||
it could only ever be a Part 11 *interpolated* bound. The start bound on the Gateway is the real prior sample, found by the
|
||||
#686 walk-back.
|
||||
|
||||
### Bounding values (`ReturnBounds`, #691)
|
||||
|
||||
⚠️ **Client-visible — a breaking change for .NET clients that rely on SDK defaults.** Until #691
|
||||
`ReadRawModifiedDetails.ReturnBounds` was ignored on every backend. It is now honoured on **every** backend (SqlServer, QuestDB,
|
||||
Gateway, and none), following OPC UA Part 11 §4.6 / §6.5.3.2 and checked against every row of Part 11 Table 1. **The OPC Foundation
|
||||
.NET SDK's `new ReadRawModifiedDetails()` defaults BOTH `ReturnBounds = true` AND `IsReadModified = true`** (verified on
|
||||
1.5.378.106), so:
|
||||
|
||||
- a client that clears `IsReadModified` but never sets `ReturnBounds` used to get stored values only and **now gets bounding values**;
|
||||
- a client that sets neither (a bare `new ReadRawModifiedDetails { StartTime, EndTime }`) is a *modified* read with bounds, which
|
||||
used to be served (or `BadHistoryOperationUnsupported` on a backend with no trail) and is **now `Bad_InvalidArgument`** (§6.5.3.3).
|
||||
|
||||
Set both explicitly. `Client.Shared` (Client.CLI / Client.UI) already does, and ScadaBridge does not issue HistoryRead. The kill
|
||||
switch `ServerHistorian:HonorReturnBounds = false` restores the pre-#691 behaviour (both changes) fleet-wide.
|
||||
|
||||
- **What a bound is.** Forward (`StartTime < EndTime`): the start bound is the stored sample **at or immediately before**
|
||||
StartTime, the end bound the one **at or immediately after** EndTime. Reverse mirrors it (start bound at or after StartTime, end
|
||||
bound at or before EndTime), newest first. A sample exactly **on** StartTime is the bound and is already the data's first value
|
||||
(nothing extra is added); a sample exactly on EndTime is added as the end bound (the data window excludes EndTime).
|
||||
`StartTime == EndTime` reads forward: the data is the samples at that instant, the end bound the first sample strictly after it.
|
||||
- **A bound is a stored sample of any quality** — a Bad sample or a comms-loss outage marker comes back as stored. Inside a tie
|
||||
cluster the bound is the cluster's first sample in the read's direction.
|
||||
- **Nothing at all ⇒ `GoodNoData`.** A read with no data in the window **and** neither bound found returns no values and
|
||||
`GoodNoData` — the answer it gave before bounds (§6.5.3.2: bounds make the result a success only "if one or both exist"). This is
|
||||
judged over the whole read (it only arises on a single, final first page); once any bound or value exists, a missing bound is
|
||||
returned as its placeholder.
|
||||
- **Missing bound:** `Bad_BoundNotFound`, null value, stamped with StartTime / EndTime. A one-sided read's open side (only a
|
||||
StartTime, or only an EndTime) is never looked up; it ends the read as `Bad_BoundNotFound` stamped one second after (forward) or
|
||||
before (reverse) the previous returned value (Table 1 footnotes a/b). `Bad_BoundNotSupported` is never returned — every backend
|
||||
can find a bound.
|
||||
- **Counting and paging.** Bounds **count toward `NumValuesPerNode`** (§6.5.3.2: "any Bounding Values returned are included in the
|
||||
numValuesPerNode count"; `NumValuesPerNode = 1` returns the start bound alone). The start bound leads the **first** page and the
|
||||
end bound ends the **last**: the pending end bound rides the continuation point, and when a page is exactly full the end bound is
|
||||
the whole next page. Paging a bounded read at any page size concatenates to the unpaged answer.
|
||||
- **How it is served — one authority, `HistoryBounds` in the OPC UA server.** "At or before T" is a newest-first read of `(0, T]`
|
||||
capped at one — native on SqlServer/QuestDB, the #686 walk-back of bounded forward reads on the Gateway (a tag with no sample
|
||||
before StartTime is proven empty down to 1970 by the #689 walk — after two empty segments it asks once for the whole remainder —
|
||||
so a missing start bound costs a few small reads, not one per ×4 step). "At or after T" is a forward read
|
||||
of `[T, MaxValue)` capped at one — on the **Gateway** it ends at **now + 1 day** instead (the walk-back's horizon; the
|
||||
HistorianGateway only checks `start ≤ end`, so a year-9999 end would reach AVEVA), and an end bound beyond that horizon is
|
||||
`Bad_BoundNotFound` without a read. SqlServer/QuestDB keep `MaxValue` (they bind it safely and may hold samples further ahead).
|
||||
"Strictly after" starts one storage resolution later. On QuestDB, whose window bounds truncate to
|
||||
the microsecond, "on the bound" compares at that resolution, so a sample at ⌊StartTime⌋ is never returned twice. A first page
|
||||
therefore costs up to three backend reads (start bound, data, end bound). The `BoundingValueConformance` suite pins those exact
|
||||
lookup reads on SqlServer and QuestDB.
|
||||
⚠️ On the **Gateway** the start-bound walk-back is subject to the #689 per-millisecond tie limit: if the millisecond holding the
|
||||
bound has more than `MaxTieClusterOverfetch` samples, the lookup cannot pick the cluster's last sample without reading it all, so
|
||||
the **whole node** read fails `BadTooManyOperations` (even though the data read alone would succeed) rather than return a wrong bound.
|
||||
- **Modified reads:** `IsReadModified = true` with `ReturnBounds = true` is `Bad_InvalidArgument` per node (Part 11 §6.5.3.3).
|
||||
- **Kill switch:** `ServerHistorian:HonorReturnBounds` (default `true`). `false` ignores `ReturnBounds` exactly as before #691 —
|
||||
stored samples only, and no modified-read rejection.
|
||||
- **Not affected:** Processed, AtTime and Events reads (they have no `ReturnBounds`).
|
||||
|
||||
### Continuation-point paging (Events) (#669)
|
||||
|
||||
@@ -723,30 +824,46 @@ says is complete carries no continuation point. `NumValuesPerNode == 0` stays un
|
||||
that many more rows than the cap, so every page short of the last is a FULL `NumValuesPerNode` page (#675; resumed pages used
|
||||
to come back short by the skip).
|
||||
- A tie cluster larger than the cap is paged within by over-fetching the half-open window `[T, T + resolution)` up to
|
||||
`MaxTieClusterOverfetch`; beyond that bound the node answers `BadHistoryOperationUnsupported` (raise the bound).
|
||||
`MaxTieClusterOverfetch`; beyond that bound the node answers `BadTooManyOperations` (raise the bound).
|
||||
- **Tie exactness depends on the backend.** SqlServer and QuestDB order by `(EventTimeUtc, EventId)`, so the skip count is exact.
|
||||
The Gateway backend has no stable intra-timestamp order (and only a truncation flag), so ties at a page boundary are
|
||||
best-effort there.
|
||||
The Gateway backend has no stable intra-timestamp order (and only a truncation flag), so ties at a page boundary of a
|
||||
**forward** read are best-effort there; a walked-back **reverse** read (below) sorts them by `EventId` itself and is exact.
|
||||
- **Reverse event reads page newest-first (#675).** `StartTime > EndTime` reads the Part 11 reverse window `(EndTime, StartTime]`
|
||||
with events in DESCENDING `(EventTimeUtc, EventId)` order, so a cap keeps the **latest** N and a continuation point is issued
|
||||
when the source flags more. The direction is stored **in the cursor**: a reverse point resumes reverse (and a forward one
|
||||
forward) whatever the resume request's details say. Reverse tie clusters are over-fetched forward and sliced from the newest
|
||||
end, exactly as Raw does. Before #675 a capped reverse read silently returned the EARLIEST N, ascending, Good, with no
|
||||
continuation point.
|
||||
- **End-only events reads: backward on SqlServer/QuestDB, forward on Gateway (#675).** `StartTime` unset
|
||||
(`DateTime.MinValue`) with an `EndTime` and `NumValuesPerNode > 0` pages backward from `EndTime` over `(MinValue, EndTime]` —
|
||||
the same rule as Raw — **only on a backend that reports `SupportsReverseEvents`** (SqlServer, QuestDB). On Gateway (the
|
||||
default backend) and Null it keeps its pre-#675 forward `[MinValue, EndTime)` meaning, paged oldest-first, rather than
|
||||
becoming a refused reverse read. Start-only reads page forward to an open upper bound (`DateTime.MaxValue`, UTC) everywhere.
|
||||
Without a cap an end-only read is forward `[MinValue, EndTime)` everywhere.
|
||||
- **Backends.** The server checks `IHistorianDataSource.SupportsReverseEvents` and then calls
|
||||
`ReadEventsAsync(…, bool reverse, …)`. SqlServer and QuestDB serve reverse natively (`… > lo AND … <= hi ORDER BY time DESC,
|
||||
id DESC`) and report `true`. **Gateway and Null report `false`**: an explicit reverse read (`StartTime > EndTime`) is answered
|
||||
`BadHistoryOperationUnsupported` **without calling the backend** — the gateway streams ascending only and rejects `end < start`,
|
||||
and draining the whole window to keep its tail would be exactly the unbounded read the cap prevents. Because the refusal is a
|
||||
capability check, not a caught exception, a `NotSupportedException` thrown from inside a backend (a driver library) is logged
|
||||
as a backend fault like any other. Every implementation (and any future wrapper) must declare both members itself — the
|
||||
interface defaults (`false` / throw) would silently disable reverse reads; `HistorianDataSourceReverseEventsGuardTests` pins it.
|
||||
- **End-only events reads page backward on every backend (#675, #691.4).** `StartTime` unset (`DateTime.MinValue`) with an
|
||||
`EndTime` and `NumValuesPerNode > 0` pages backward from `EndTime` over `(MinValue, EndTime]` — the same rule as Raw.
|
||||
Start-only reads page forward to an open upper bound (`DateTime.MaxValue`, UTC) everywhere. Without a cap an end-only read
|
||||
is forward `[MinValue, EndTime)` everywhere.
|
||||
- **Backends.** The server checks `IHistorianDataSource.SupportsReverseEvents`. SqlServer and QuestDB serve reverse natively
|
||||
(`… > lo AND … <= hi ORDER BY time DESC, id DESC`, the `ReadEventsAsync(…, bool reverse, …)` overload) and report `true`.
|
||||
**Gateway reports `false`** — the gateway streams ascending only and rejects `end < start` — and is never sent a reverse
|
||||
read (Null reports `true` and answers empty either way, as for raw): since **#691.4** the server walks it back from bounded **forward** reads with the raw walk (#686/#689/#691:
|
||||
page-sized ms-aligned segments filtered to `(segLo, segHi]`, a read the source flagged truncated is never answered from,
|
||||
the now + 1 day / 1970 bounds, the remainder probe and the proven-empty stretch carried in the cursor). Event specifics:
|
||||
- **Ties** are sorted by `EventId` (ordinal) on every read — the gateway orders by `EventTimeUtc` only, in any order within a
|
||||
timestamp — so the page is DESCENDING `(EventTimeUtc, EventId)` and a resumed page's boundary skip lines up; the
|
||||
oversized-cluster over-fetch keeps exactly the events at `T` (the gateway's end is inclusive) in the same order.
|
||||
- **"More exists"** is found by walking for one event past the page (the walk has no source truncation flag to hand on).
|
||||
- **Uncapped** reverse reads are one forward read of the window, sorted and reversed.
|
||||
- ⚠️ **Also fixed (#691.4 review): uncapped events reads on Gateway.** `NumValuesPerNode = 0` reaches the source as
|
||||
`int.MaxValue`, and `GatewayHistorianDataSource` pre-sized its result list to it ("Array dimensions exceeded") — every
|
||||
uncapped Gateway events read, forward too, failed the node and counted as a backend failure. The capacity is now bounded.
|
||||
- A timestamp holding more than `MaxTieClusterOverfetch` events ⇒ **`BadTooManyOperations`** for that node.
|
||||
- ⚠️ **Gateway prerequisite:** the gateway's `RuntimeDb:EventReadMaxRows` must be `0` (its default, unlimited) or larger than
|
||||
the walk's biggest read (`2N + 18` rows, a discovery read 1,026, a tie-cluster read up to `2·MaxTieClusterOverfetch + 4`).
|
||||
That cap truncates **silently** (the stream just ends), so a segment it cut would look complete and drop its newest events —
|
||||
the same blindness a forward read has to it, but on the end a reverse read is asking for.
|
||||
|
||||
⚠️ **Client-visible (#691.4).** On Gateway (the default backend) and Null an explicit reverse events read used to be refused
|
||||
`BadHistoryOperationUnsupported` (#675), and an end-only capped events read returned the **oldest** N events ascending. Both
|
||||
now return the latest events newest-first. Because the capability is checked, not a caught exception, a
|
||||
`NotSupportedException` thrown from inside a backend (a driver library) is still logged as a backend fault. Every
|
||||
implementation (and any future wrapper) must declare both members itself — the interface defaults (`false` / throw) would
|
||||
silently send a native-reverse backend through the walk; `HistorianDataSourceReverseEventsGuardTests` pins it.
|
||||
- **Gateway open upper bound.** The `DateTime.MaxValue` (UTC) bound of a start-only forward read is pinned offline through the
|
||||
published client's proto `Timestamp` conversion (captured by an in-process gRPC server) and the FILETIME encoding
|
||||
(`GatewayEventBoundsTests`). What a live AVEVA Historian does with a year-9999 bound is still unverified (needs the VPN).
|
||||
@@ -865,8 +982,30 @@ fixed:
|
||||
`HistorianTagDefinition.IntegralDivisor` is `0`, so `GatewayTagProvisioner` now sends **`IntegralDivisor = 1`** explicitly on every
|
||||
`EnsureTags` — the value·seconds unit the SQL backends' `Total` uses. A tag provisioned earlier keeps its divisor until its next
|
||||
`EnsureTags` (an upsert). The live magnitude check against AVEVA (a constant 1 integrated over 1 s reads 1) passed on 2026-10-12 at the AVEVA
|
||||
level; end to end it waits on historiangw#27 (above). Reverse processed reads of the AVEVA-native `TimeAverage`/`Total` are not re-anchored by #677 (the gateway refuses `end < start`);
|
||||
the raw-evaluated per-sample aggregates (#681) do follow #677. Since #681 every native bucket is stamped with its interval **start**
|
||||
level; end to end it waits on historiangw#27 (above). ⚠️ **Native reads on the SQL backends' interval grid (#691.2, client-visible).** Both directions now use `BucketPlan`, served by
|
||||
**forward** native requests only (`NativeAggregatePlan` — AVEVA anchors a native read's interval grid at the request start, and the
|
||||
gateway refuses `start > end` and `interval <= 0`):
|
||||
- **Reverse** (`StartTime > EndTime`) used to reach the gateway as `start > end` and fail the node (`InvalidArgument`, counted as a backend
|
||||
failure). It now follows #677 like the SQL backends and the raw-evaluated per-sample aggregates (#681): window `(EndTime, StartTime]`,
|
||||
intervals stepping back from `StartTime`, the clipped interval the **earliest**, each stamped with its **later** edge, newest first —
|
||||
one read `[O + I·f, StartTime)` with the interval `I` for the whole intervals (`O = StartTime − n·I`, `f` = 1 when clipped) and a
|
||||
second read over exactly the clipped span `[EndTime, O + I)` with its own width as the interval. A time-weighted value over `[a, b)` and
|
||||
over `(a, b]` is the same integral (two instants apart), so each forward row is the reverse interval's value.
|
||||
- **Forward** windows that are not a whole number of intervals used to send one request whose last AVEVA interval ran **past EndTime**;
|
||||
now the whole intervals `[StartTime, StartTime + (n−1)·I)` are one read and the clipped **latest** interval `[StartTime + (n−1)·I,
|
||||
EndTime)` is its own read over exactly that span. An interval wider than the window is sent as the window, and `interval = 0` as one
|
||||
interval over the window (it used to reach the gateway as 0 and be refused). An empty window (`StartTime == EndTime`) sends nothing.
|
||||
- Stamps match SqlServer/QuestDB (forward: earlier edge, ascending; reverse: later edge, descending), and the clipped interval carries
|
||||
the **Partial** bit unless Bad (Part 13 §5.3.3.2 Case 3, as the SQL backends' time-weighted aggregates — which also mark the intervals
|
||||
the tag's data starts/ends inside; AVEVA's rows carry no data range, so that is not reproduced). Rows are otherwise mapped as before
|
||||
(#685). An interval AVEVA answers no row for is absent (the SQL backends answer it `BadNoData`).
|
||||
- ⚠️ **Interval cap (client-visible):** forward native reads now go through `BucketPlan` too, so a window of more than 100,000
|
||||
intervals (e.g. 1 s over 2 days) is refused up front with `BadHistoryOperationUnsupported`, as on the SQL backends — it used to be
|
||||
served up to the gateway's own `MaxAggregateIntervals` (1,000,000). Not counted as a backend failure. A clipped interval shorter than
|
||||
1 ms (a sub-ms window end) is sent as a sub-ms interval request; re-check it in the post-historiangw#27 live leg.
|
||||
- ⚠️ **Until historiangw#27 is fixed, native values are wrong in BOTH directions with a Good status:** AVEVA answers one row per request
|
||||
(the whole span's value), which lands on that request's first interval as a Good, Calculated value while the other intervals are
|
||||
absent. Pinned against a native-mode fake only — re-run the forward and reverse native legs live once the gateway is fixed. Since #681 every native bucket is stamped with its interval **start**
|
||||
(Part 13; it was the interval end) and carries the Calculated info bits (`Good` ⇒ `0x00000401`, no data ⇒ `BadNoData | 0x401`).
|
||||
The live AVEVA gate (2026-10-12) confirmed `start_time` is the interval start (and the `value·s` unit). Previously noted as
|
||||
pending: it reads the aggregate row's `start_time`, which histsdk decodes from a row
|
||||
@@ -912,9 +1051,11 @@ and evicts its shared historian session (see the HistorianGateway issue text in
|
||||
streaming gateway and keeps its `+ 18`. Past the cap the rest of the (bounded) stream is drained, then the node fails
|
||||
`BadTooManyOperations`; a stream that delivers the whole budget cannot be proven complete and fails the same way (so with the
|
||||
default cap a window of more than ~999,982 rows fails, as it would one row later). The gateway's own refusal of the budget
|
||||
(`InvalidArgument` "max_values exceeds the server ceiling" — `Historian:ReadLimits:MaxValuesCeiling` lowered below the default,
|
||||
with `AllowStreamingBeyondCeiling=false`) is also `BadTooManyOperations`, not a backend fault — lower `MaxTimeWeightedRawRows` to
|
||||
`ceiling − 18` on such a gateway. A raw read's budget is **not** clamped (see [#687](#gateway-raw-reads-return-stored-samples-only-687)). A caller cancellation (client cancel, `HistoryReadDeadline`) is not counted as a backend fault.
|
||||
(`InvalidArgument` "max_values exceeds the server ceiling of N" — `Historian:ReadLimits:MaxValuesCeiling` lowered below the
|
||||
default, or an unbounded `MaxTimeWeightedRawRows ≤ 0`, with `AllowStreamingBeyondCeiling=false`) is **retried once with N as the
|
||||
budget** (#691.3): a window that fits is served, one that delivers all N rows fails `BadTooManyOperations` as above (not a backend
|
||||
fault). Lowering `MaxTimeWeightedRawRows` to `ceiling − 18` on such a gateway is no longer needed — it only saves the refused
|
||||
round-trip. A raw read's budget is **not** clamped (see [#687](#gateway-raw-reads-return-stored-samples-only-687)). A caller cancellation (client cancel, `HistoryReadDeadline`) is not counted as a backend fault.
|
||||
- **Bucket statistics** are `RawBucketAccumulator`, the C# statement of the SQL backends' per-sample columns — incl. #684:
|
||||
Average / Minimum / Maximum / Range / *ActualTime over **Good** samples only, the Uncertain extremes feeding only the Min/Max
|
||||
status rule, and an Uncertain String/DateTime sample not refusing the Good-value aggregates. A change to those SQL columns
|
||||
@@ -1009,8 +1150,8 @@ whatever the SDK sets — #682 touches only the i=11201 folder (follow-up candid
|
||||
sample per requested timestamp, so the single-shot backend returns the complete result in one
|
||||
read and there is no "full page ⇒ maybe more" signal to page on. Returning the full result with
|
||||
no continuation point is spec-conformant.
|
||||
- **`ReturnBounds` is ignored** (every backend): Raw reads return stored values only; no bounding values are added at the window
|
||||
edges. The node manager does not pass the flag to `IHistorianDataSource`.
|
||||
- **Bounds are raw only:** `ReturnBounds` (#691, [Bounding values](#bounding-values-returnbounds-691)) never interpolates a bound —
|
||||
it is always a stored sample or `Bad_BoundNotFound`.
|
||||
- **No modified-value history on the `Gateway` backend.** Modified reads (`IsReadModified = true`) are served by
|
||||
`SqlServer` and `QuestDb` ([Modified-value history](#modified-value-history-part-11), #668); the HistorianGateway exposes no
|
||||
modified/edited-history surface, so on `Gateway` (and with no historian) they return `BadHistoryOperationUnsupported`.
|
||||
|
||||
@@ -120,9 +120,9 @@ public interface IHistorianDataSource : IDisposable
|
||||
/// <summary>
|
||||
/// Whether this backend can serve a reverse (newest-first) events read — the reverse overload of
|
||||
/// <see cref="ReadEventsAsync(string?, DateTime, DateTime, int, bool, CancellationToken)"/> (Gitea #675). The server
|
||||
/// checks it BEFORE calling the source: an explicit reverse read (<c>StartTime > EndTime</c>) on a backend that
|
||||
/// cannot is answered <c>BadHistoryOperationUnsupported</c> without a call, and an end-only capped read, which Part 11
|
||||
/// reads backward, keeps its forward <c>[MinValue, EndTime)</c> meaning there. Default <c>false</c>; an
|
||||
/// checks it BEFORE calling the source: a reverse read (explicit <c>StartTime > EndTime</c>, or an end-only capped
|
||||
/// read, which Part 11 reads backward) on a backend that cannot is never sent the reverse overload — since Gitea
|
||||
/// #691.4 the server walks it back from bounded FORWARD reads instead (#675 refused it). Default <c>false</c>; an
|
||||
/// implementation (or a wrapper) declares it itself — a guard test pins that.
|
||||
/// </summary>
|
||||
bool SupportsReverseEvents => false;
|
||||
@@ -143,10 +143,10 @@ public interface IHistorianDataSource : IDisposable
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// The default forwards a forward read to the five-argument overload and throws
|
||||
/// <see cref="NotSupportedException"/> for a reverse one — a backend that cannot read newest-first (Gateway,
|
||||
/// Null). The server never relies on that throw: it consults <see cref="SupportsReverseEvents"/> first and
|
||||
/// answers <c>BadHistoryOperationUnsupported</c> without calling, rather than returning the earliest events in
|
||||
/// ascending order as if they were the answer. A backend that serves reverse must override BOTH members.
|
||||
/// <see cref="NotSupportedException"/> for a reverse one — a backend that cannot read newest-first (the
|
||||
/// Gateway). The server never relies on that throw: it consults <see cref="SupportsReverseEvents"/> first and walks
|
||||
/// such a read back from forward reads (#691.4), never returning the earliest events in ascending order as if
|
||||
/// they were the answer. A backend that serves reverse natively must override BOTH members.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
/// <param name="sourceName">The source name to filter events, or null to return events from all sources.</param>
|
||||
|
||||
@@ -61,13 +61,12 @@ public sealed class NullHistorianDataSource : IHistorianDataSource
|
||||
public bool SupportsReverseRaw => true;
|
||||
|
||||
/// <inheritdoc />
|
||||
/// <remarks><c>false</c> (Gitea #675): there is no history to read in either direction, and an explicit reverse read
|
||||
/// is answered <c>BadHistoryOperationUnsupported</c> like any backend that cannot read newest-first.</remarks>
|
||||
public bool SupportsReverseEvents => false;
|
||||
/// <remarks><c>true</c>, like <see cref="SupportsReverseRaw"/> (Gitea #691.4 review; <c>false</c> under #675): there is no
|
||||
/// history to read in either direction — no need for the server to walk back through empty forward reads.</remarks>
|
||||
public bool SupportsReverseEvents => true;
|
||||
|
||||
/// <inheritdoc />
|
||||
/// <remarks>Forward ⇒ the empty result; reverse ⇒ <see cref="NotSupportedException"/>, the interface contract for a
|
||||
/// backend without <see cref="SupportsReverseEvents"/>.</remarks>
|
||||
/// <remarks>The empty result in either direction.</remarks>
|
||||
public Task<HistoricalEventsResult> ReadEventsAsync(
|
||||
string? sourceName,
|
||||
DateTime startUtc,
|
||||
@@ -75,9 +74,7 @@ public sealed class NullHistorianDataSource : IHistorianDataSource
|
||||
int maxEvents,
|
||||
bool reverse,
|
||||
CancellationToken cancellationToken) =>
|
||||
reverse
|
||||
? throw new NotSupportedException("The null historian cannot read events newest-first (reverse).")
|
||||
: Task.FromResult(EmptyEvents);
|
||||
Task.FromResult(EmptyEvents);
|
||||
|
||||
/// <inheritdoc />
|
||||
/// <remarks>
|
||||
|
||||
+80
-40
@@ -47,6 +47,9 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
|
||||
private readonly int _maxRawRows;
|
||||
private readonly TimeProvider _time;
|
||||
|
||||
/// <summary>A strict gateway's row ceiling, learned from its first refusal (#691.3).</summary>
|
||||
private readonly GatewayRowCeiling _ceiling = new();
|
||||
|
||||
private readonly object _healthLock = new();
|
||||
private DateTime? _lastSuccessUtc;
|
||||
private DateTime? _lastFailureUtc;
|
||||
@@ -144,8 +147,12 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
|
||||
/// gateway reject <c>max_values <= 0</c>, #686). A cap asks for <c>cap + </c>
|
||||
/// <see cref="RawAggregateReader.PreWindowAllowance"/> rows — NOT clamped at the gateway's default ceiling (#688
|
||||
/// clamps only the per-sample aggregate budget): AVEVA always emits the boundary row, so a clamped budget would make
|
||||
/// every page size within the allowance of the ceiling come back short. A strict gateway refuses an over-ceiling
|
||||
/// budget instead (⇒ <see cref="HistoryReadLimitExceededException"/>, <c>BadTooManyOperations</c>).
|
||||
/// every page size within the allowance of the ceiling come back short. A strict gateway
|
||||
/// (<c>AllowStreamingBeyondCeiling=false</c>) refuses an over-ceiling budget and names its ceiling; the read is
|
||||
/// then re-issued ONCE with the ceiling as budget (#691.3, <see cref="GatewayReadLimits"/>) — so an uncapped read
|
||||
/// of a small window is served — and a ceiling-bounded read that is not provably complete (it delivered the
|
||||
/// whole ceiling without reaching the window end, or a page short of its cap) fails
|
||||
/// <see cref="HistoryReadLimitExceededException"/> (<c>BadTooManyOperations</c>), never a truncated answer.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Completeness.</b> A full page must hold exactly the cap of stored samples — a short page ends the client's
|
||||
@@ -166,22 +173,43 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
|
||||
// The gateway seam caps with an int; OPC UA hands us a uint, so clamp to int range.
|
||||
var cap = (int)Math.Min(maxValuesPerNode, int.MaxValue);
|
||||
if (cap == 0)
|
||||
return (await ReadRawPassAsync(fullReference, startUtc, endUtc, cap, int.MaxValue, cancellationToken)
|
||||
.ConfigureAwait(false)).Samples;
|
||||
{
|
||||
var all = await PassAsync(int.MaxValue).ConfigureAwait(false);
|
||||
if (all.Result.Complete) return all.Result.Samples;
|
||||
// Only reachable on a strict gateway (#691.3): the ceiling re-read delivered the whole ceiling without reaching
|
||||
// the window end — the window may hold more, and "all values" is never answered from a truncated read.
|
||||
throw new HistoryReadLimitExceededException(cap,
|
||||
$"The raw read of '{fullReference}' holds more than the HistorianGateway's row ceiling ({all.Budget}, its " +
|
||||
"Historian:ReadLimits:MaxValuesCeiling with AllowStreamingBeyondCeiling=false); set NumValuesPerNode to page, " +
|
||||
"or narrow the window.");
|
||||
}
|
||||
|
||||
var first = await ReadRawPassAsync(fullReference, startUtc, endUtc, cap,
|
||||
(int)Math.Min(int.MaxValue, (long)cap + RawAggregateReader.PreWindowAllowance), cancellationToken).ConfigureAwait(false);
|
||||
if (first.Complete) return first.Samples;
|
||||
var asked = (int)Math.Min(int.MaxValue, (long)cap + RawAggregateReader.PreWindowAllowance);
|
||||
var first = await PassAsync(asked).ConfigureAwait(false);
|
||||
if (first.Result.Complete) return first.Result.Samples;
|
||||
|
||||
// A strict gateway's ceiling already bounded that read (#691.3): a larger budget would only be refused again.
|
||||
if (first.Budget < asked)
|
||||
throw new HistoryReadLimitExceededException(cap,
|
||||
$"The raw read of '{fullReference}' could not be read completely within the HistorianGateway's row ceiling " +
|
||||
$"({first.Budget}, its Historian:ReadLimits:MaxValuesCeiling with AllowStreamingBeyondCeiling=false); lower " +
|
||||
"NumValuesPerNode or narrow the window.");
|
||||
|
||||
// The margin crowded the window out: one bounded re-read with room for the pre-window rows already seen.
|
||||
var second = await ReadRawPassAsync(fullReference, startUtc, endUtc, cap,
|
||||
(int)Math.Min(int.MaxValue, (long)cap + first.PreWindowRows + RawAggregateReader.PreWindowAllowance),
|
||||
cancellationToken).ConfigureAwait(false);
|
||||
if (second.Complete) return second.Samples;
|
||||
var second = await PassAsync(
|
||||
(int)Math.Min(int.MaxValue, (long)cap + first.Result.PreWindowRows + RawAggregateReader.PreWindowAllowance))
|
||||
.ConfigureAwait(false);
|
||||
if (second.Result.Complete) return second.Result.Samples;
|
||||
|
||||
throw new HistoryReadLimitExceededException(cap,
|
||||
$"The raw read of '{fullReference}' could not be read completely: more than {second.PreWindowRows} rows lie in " +
|
||||
$"The raw read of '{fullReference}' could not be read completely: more than {second.Result.PreWindowRows} rows lie in " +
|
||||
$"the {RawAggregateReader.BoundaryMargin.TotalMilliseconds} ms before the window and exhausted the row budget.");
|
||||
|
||||
// One drained pass; a strict gateway's refusal of the budget is retried once at the ceiling it names (#691.3).
|
||||
Task<((List<HistorianSample> Samples, int PreWindowRows, bool Complete) Result, int Budget)> PassAsync(int budget) =>
|
||||
_ceiling.ReadAsync(budget,
|
||||
b => ReadRawPassAsync(fullReference, startUtc, endUtc, cap, b, cancellationToken),
|
||||
(r, _) => !r.Complete);
|
||||
}
|
||||
|
||||
/// <summary>One forward raw read of a window, drained, with its pre-window row count and whether it is complete.</summary>
|
||||
@@ -206,7 +234,9 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
|
||||
else if (sample.Timestamp is not null) reachedEnd = true;
|
||||
}
|
||||
|
||||
var complete = received < budget || samples.Count >= cap || reachedEnd || cap == 0;
|
||||
// cap == 0 ("all values") is complete only when the stream proves it: short of the budget (always, unless a strict
|
||||
// gateway's ceiling became the budget — #691.3) or past the window end.
|
||||
var complete = received < budget || reachedEnd || (cap > 0 && samples.Count >= cap);
|
||||
return (samples, preWindow, complete);
|
||||
}
|
||||
|
||||
@@ -223,7 +253,9 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
|
||||
/// <para>
|
||||
/// <b>TimeAverage / Total</b> stay native AVEVA retrieval modes (<see cref="AggregateModeMapper"/>), stamped with
|
||||
/// the interval start and the Calculated bit; AVEVA's extra row for the interval before the window is dropped and
|
||||
/// an Uncertain interval keeps its value (#685, <see cref="SampleMapper.ToAggregateSnapshots"/>). Everything else is <see cref="HistoryAggregateNotSupportedException"/>
|
||||
/// an Uncertain interval keeps its value (#685). Both directions use the SQL backends' interval grid with forward
|
||||
/// requests only — a reverse read (which the gateway refuses) anchored at its later time (#677), the clipped interval
|
||||
/// read over exactly its span and marked Partial (#691.2, <see cref="NativeAggregatePlan"/>). Everything else is <see cref="HistoryAggregateNotSupportedException"/>
|
||||
/// (see <see cref="ServedAggregates"/>).
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
@@ -240,22 +272,36 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
|
||||
// Mapped OUTSIDE the failure-recording try: an aggregate the gateway has no mode for is a client
|
||||
// request the server answers BadAggregateNotSupported, not a backend fault to count (#285).
|
||||
var mode = AggregateModeMapper.ToRetrievalMode(aggregate);
|
||||
// Planned OUTSIDE the failure-recording try: too many intervals is a client request limit, not a backend fault.
|
||||
return await ReadNativeAsync(fullReference, BucketPlan.Create(startUtc, endUtc, interval), mode, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A native TimeAverage / Total read on the SQL backends' interval grid, either direction (#691.2): forward native
|
||||
/// requests only, the clipped interval read over exactly its span. See <see cref="NativeAggregatePlan"/>.
|
||||
/// </summary>
|
||||
private async Task<HistoryReadResult> ReadNativeAsync(string fullReference, BucketPlan plan, RetrievalMode mode,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
if (plan.BucketCount == 0) return new HistoryReadResult([], ContinuationPoint: null);
|
||||
try
|
||||
{
|
||||
// TimeAverage / Total are native gateway retrieval modes (TimeWeightedAverage / Integral) — no
|
||||
// client-side scaling.
|
||||
var buckets = new List<HistorianAggregateSample>();
|
||||
await foreach (var bucket in _client
|
||||
.ReadAggregateAsync(fullReference, startUtc, endUtc, mode, interval, cancellationToken)
|
||||
.ConfigureAwait(false))
|
||||
var byBucket = new Dictionary<long, DataValueSnapshot>();
|
||||
foreach (var request in NativeAggregatePlan.Requests(plan))
|
||||
{
|
||||
buckets.Add(bucket);
|
||||
var buckets = new List<HistorianAggregateSample>();
|
||||
await foreach (var bucket in _client
|
||||
.ReadAggregateAsync(fullReference, request.StartUtc, request.EndUtc, mode, request.Interval, cancellationToken)
|
||||
.ConfigureAwait(false))
|
||||
{
|
||||
buckets.Add(bucket); // drained to completion, mapped afterwards
|
||||
}
|
||||
NativeAggregatePlan.Place(plan, request, buckets, byBucket);
|
||||
}
|
||||
|
||||
// Filtered while mapping, after the stream was drained (#685: AVEVA's extra pre-window row).
|
||||
var snapshots = SampleMapper.ToAggregateSnapshots(buckets, startUtc, endUtc);
|
||||
RecordOutcome(success: true, error: null);
|
||||
return new HistoryReadResult(snapshots, ContinuationPoint: null);
|
||||
return new HistoryReadResult(NativeAggregatePlan.Assemble(plan, byBucket), ContinuationPoint: null);
|
||||
}
|
||||
catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested)
|
||||
{
|
||||
@@ -280,7 +326,7 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
|
||||
{
|
||||
var now = _time.GetUtcNow().UtcDateTime;
|
||||
var values = await RawAggregateReader
|
||||
.ReadAsync(_client, fullReference, aggregate, plan, _maxRawRows, now, cancellationToken)
|
||||
.ReadAsync(_client, _ceiling, fullReference, aggregate, plan, _maxRawRows, now, cancellationToken)
|
||||
.ConfigureAwait(false);
|
||||
RecordOutcome(success: true, error: null);
|
||||
return new HistoryReadResult(values, ContinuationPoint: null);
|
||||
@@ -315,14 +361,7 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
|
||||
/// True when <paramref name="ex"/> (or an inner exception — the client package wraps the <see cref="RpcException"/>)
|
||||
/// is the gateway's <c>InvalidArgument</c> "max_values exceeds the server ceiling" rejection.
|
||||
/// </summary>
|
||||
internal static bool IsMaxValuesCeilingRejection(Exception ex)
|
||||
{
|
||||
for (var e = ex; e is not null; e = e.InnerException)
|
||||
if (e is RpcException { StatusCode: StatusCode.InvalidArgument } rpc
|
||||
&& rpc.Status.Detail.Contains("max_values exceeds the server ceiling", StringComparison.Ordinal))
|
||||
return true;
|
||||
return false;
|
||||
}
|
||||
internal static bool IsMaxValuesCeilingRejection(Exception ex) => GatewayReadLimits.IsCeilingRejection(ex);
|
||||
|
||||
/// <inheritdoc />
|
||||
public async Task<HistoryReadResult> ReadAtTimeAsync(
|
||||
@@ -354,8 +393,9 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
|
||||
|
||||
/// <inheritdoc />
|
||||
/// <remarks>
|
||||
/// <c>false</c>: the gateway's <c>ReadEvents</c> streams ascending only (Gitea #675). The server therefore refuses an
|
||||
/// explicit reverse events read without calling, and keeps an end-only capped read forward.
|
||||
/// <c>false</c>: the gateway's <c>ReadEvents</c> streams ascending only (Gitea #675). The server never sends it a
|
||||
/// reverse read; since #691.4 it walks one back from bounded forward reads (ties sorted by EventId — the gateway
|
||||
/// orders by event time only).
|
||||
/// </remarks>
|
||||
public bool SupportsReverseEvents => false;
|
||||
|
||||
@@ -365,10 +405,9 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
|
||||
|
||||
/// <inheritdoc />
|
||||
/// <remarks>
|
||||
/// A reverse (newest-first) read is refused with <see cref="NotSupportedException"/> — the server reports
|
||||
/// <c>BadHistoryOperationUnsupported</c> (Gitea #675). The gateway's <c>ReadEvents</c> streams ascending only and
|
||||
/// rejects <c>end < start</c>, so the only way to serve "the latest N" would be to drain the whole window and
|
||||
/// keep its tail, an unbounded read the cap exists to prevent. Nothing is sent.
|
||||
/// A reverse (newest-first) read is refused with <see cref="NotSupportedException"/> (Gitea #675) — the gateway's
|
||||
/// <c>ReadEvents</c> streams ascending only and rejects <c>end < start</c>. The server never calls this with
|
||||
/// <c>reverse</c>: it walks a reverse read back from bounded forward reads (#691.4). Nothing is sent.
|
||||
/// </remarks>
|
||||
public Task<HistoricalEventsResult> ReadEventsAsync(
|
||||
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, bool reverse,
|
||||
@@ -386,7 +425,8 @@ public sealed class GatewayHistorianDataSource : IHistorianDataSource, IAsyncDis
|
||||
try
|
||||
{
|
||||
var hasCap = maxEvents > 0;
|
||||
var collected = new List<HistoricalEvent>(hasCap ? maxEvents : 0);
|
||||
// Bounded initial capacity: an uncapped read arrives as int.MaxValue ("Array dimensions exceeded", #691.4 review).
|
||||
var collected = new List<HistoricalEvent>(hasCap ? Math.Min(maxEvents, 1024) : 0);
|
||||
var truncated = false;
|
||||
|
||||
await foreach (var wireEvent in _client
|
||||
|
||||
@@ -0,0 +1,135 @@
|
||||
using System.Globalization;
|
||||
using System.Text.RegularExpressions;
|
||||
using Grpc.Core;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway;
|
||||
|
||||
/// <summary>
|
||||
/// The HistorianGateway's raw-read row ceiling (<c>Historian:ReadLimits:MaxValuesCeiling</c>, default 1,000,000) as seen
|
||||
/// from this side of the wire. With <c>AllowStreamingBeyondCeiling=false</c> (a "strict" gateway) the gateway refuses
|
||||
/// any <c>ReadRaw</c> whose <c>max_values</c> exceeds the ceiling with <c>InvalidArgument</c> — BEFORE it streams a row
|
||||
/// — and names the ceiling in the refusal (<c>"max_values exceeds the server ceiling of {n}; …"</c>). The gateway has
|
||||
/// no RPC that reports the ceiling, so the refusal is how it is learned (#691.3).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// A read whose budget the gateway refuses is re-issued ONCE with the ceiling as its budget, and the ceiling is
|
||||
/// remembered per data source (<see cref="GatewayRowCeiling"/>); the caller treats a stream that delivered the whole
|
||||
/// ceiling as possibly truncated (never answered from).
|
||||
/// </remarks>
|
||||
internal static class GatewayReadLimits
|
||||
{
|
||||
private const string CeilingRejectionText = "max_values exceeds the server ceiling";
|
||||
|
||||
private static readonly Regex CeilingNumber = new(@"max_values exceeds the server ceiling of (\d+)",
|
||||
RegexOptions.CultureInvariant | RegexOptions.Compiled);
|
||||
|
||||
/// <summary>
|
||||
/// True when <paramref name="ex"/> (or an inner exception — the client package wraps the <see cref="RpcException"/>)
|
||||
/// is the gateway's <c>InvalidArgument</c> "max_values exceeds the server ceiling" rejection.
|
||||
/// </summary>
|
||||
/// <param name="ex">The exception.</param>
|
||||
/// <returns>Whether it is the ceiling rejection.</returns>
|
||||
public static bool IsCeilingRejection(Exception ex) => Rejection(ex) is not null;
|
||||
|
||||
/// <summary>
|
||||
/// The ceiling a ceiling rejection names, or — when <paramref name="ex"/> is a ceiling rejection that does not
|
||||
/// carry a parseable number — the gateway's default (<see cref="RawAggregateReader.GatewayDefaultMaxValuesCeiling"/>).
|
||||
/// </summary>
|
||||
/// <param name="ex">The exception.</param>
|
||||
/// <param name="ceiling">The ceiling (> 0) when this returns true.</param>
|
||||
/// <returns>Whether <paramref name="ex"/> is a ceiling rejection.</returns>
|
||||
public static bool TryGetCeiling(Exception ex, out int ceiling)
|
||||
{
|
||||
ceiling = 0;
|
||||
if (Rejection(ex) is not { } rpc) return false;
|
||||
var m = CeilingNumber.Match(rpc.Status.Detail);
|
||||
ceiling = m.Success && int.TryParse(m.Groups[1].Value, NumberStyles.None, CultureInfo.InvariantCulture, out var n) && n > 0
|
||||
? n
|
||||
: RawAggregateReader.GatewayDefaultMaxValuesCeiling;
|
||||
return true;
|
||||
}
|
||||
|
||||
private static RpcException? Rejection(Exception ex)
|
||||
{
|
||||
for (var e = ex; e is not null; e = e.InnerException)
|
||||
if (e is RpcException { StatusCode: StatusCode.InvalidArgument } rpc
|
||||
&& rpc.Status.Detail.Contains(CeilingRejectionText, StringComparison.Ordinal))
|
||||
return rpc;
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A strict gateway's row ceiling as learned by ONE data source (#691.3 review): every refused attempt counts as an
|
||||
/// error in the gateway's own outcome metrics, so the ceiling a refusal names is remembered and later reads ask for at
|
||||
/// most it — no refusal per read in the steady state. Thread-safe (one <see cref="int"/>, read/written atomically; a
|
||||
/// race costs at most one extra refusal or re-probe, never a wrong answer).
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Rules.</b> (1) Unknown ceiling: ask for the full budget; a refusal naming <c>N</c> < budget learns <c>N</c> and
|
||||
/// the read is re-issued once at <c>N</c>. (2) Learned ceiling <c>C</c> below the budget: ask for <c>C</c> — a refusal
|
||||
/// there (the gateway was LOWERED) re-learns its <c>N</c> as in (1). (3) If that <c>C</c>-bounded read came back
|
||||
/// full but not provably complete, the gateway may have been RAISED (or switched to streaming) since <c>C</c> was
|
||||
/// learned, and answering <c>BadTooManyOperations</c> from a stale ceiling would be wrong: the read is re-probed ONCE
|
||||
/// at the full budget — served ⇒ the ceiling is forgotten; refused naming a larger <c>N</c> ⇒ <c>N</c> is learned and
|
||||
/// the read is re-issued at <c>N</c>; refused naming <c>C</c> again ⇒ the <c>C</c>-bounded result stands (no repeat).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Why no TTL.</b> A raise only matters to a read that needs more than <c>C</c>, and such a read re-probes anyway
|
||||
/// (rule 3), so a timer would only add refusals for reads that fit. The cost of the rule is one refused re-probe per
|
||||
/// genuinely over-ceiling read — a read that fails <c>BadTooManyOperations</c> whatever is done.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
internal sealed class GatewayRowCeiling
|
||||
{
|
||||
private int _learned; // 0 = unknown / not strict
|
||||
|
||||
/// <summary>The learned ceiling (0 = none).</summary>
|
||||
public int Learned => Volatile.Read(ref _learned);
|
||||
|
||||
/// <summary>Runs a drained read under the rules above.</summary>
|
||||
/// <typeparam name="T">The pass result.</typeparam>
|
||||
/// <param name="budget">The <c>max_values</c> the read wants.</param>
|
||||
/// <param name="pass">One drained read with the given budget.</param>
|
||||
/// <param name="needsMore">True when a pass with the given budget delivered it all without proving completeness.</param>
|
||||
/// <returns>The result and the budget it was read with.</returns>
|
||||
public async Task<(T Result, int Budget)> ReadAsync<T>(int budget, Func<int, Task<T>> pass, Func<T, int, bool> needsMore)
|
||||
{
|
||||
var learned = Learned;
|
||||
if (learned <= 0 || learned >= budget) return await AttemptAsync(budget, pass).ConfigureAwait(false);
|
||||
|
||||
var bounded = await AttemptAsync(learned, pass).ConfigureAwait(false);
|
||||
if (bounded.Budget < learned || !needsMore(bounded.Result, bounded.Budget)) return bounded;
|
||||
|
||||
// Rule 3: full at a possibly stale ceiling — re-probe once at the full budget.
|
||||
int fresh;
|
||||
try
|
||||
{
|
||||
var full = await pass(budget).ConfigureAwait(false);
|
||||
Interlocked.CompareExchange(ref _learned, 0, learned);
|
||||
return (full, budget);
|
||||
}
|
||||
catch (Exception ex) when (GatewayReadLimits.TryGetCeiling(ex, out fresh) && fresh < budget)
|
||||
{
|
||||
// fall through (outside the catch)
|
||||
}
|
||||
Volatile.Write(ref _learned, fresh);
|
||||
return fresh > learned ? (await pass(fresh).ConfigureAwait(false), fresh) : bounded;
|
||||
}
|
||||
|
||||
private async Task<(T Result, int Budget)> AttemptAsync<T>(int budget, Func<int, Task<T>> pass)
|
||||
{
|
||||
int ceiling;
|
||||
try
|
||||
{
|
||||
return (await pass(budget).ConfigureAwait(false), budget);
|
||||
}
|
||||
catch (Exception ex) when (GatewayReadLimits.TryGetCeiling(ex, out ceiling) && ceiling < budget)
|
||||
{
|
||||
// fall through to the retry (outside the catch, so a second refusal is not nested in this one)
|
||||
}
|
||||
Volatile.Write(ref _learned, ceiling);
|
||||
return (await pass(ceiling).ConfigureAwait(false), ceiling);
|
||||
}
|
||||
}
|
||||
@@ -23,11 +23,21 @@ internal static class SampleMapper
|
||||
/// <summary>Maps a single raw sample to a value snapshot.</summary>
|
||||
/// <param name="sample">The gateway raw sample.</param>
|
||||
/// <returns>The driver-agnostic snapshot.</returns>
|
||||
/// <remarks>
|
||||
/// #690: a Bad row carries no value. histsdk puts a value on EVERY row (<c>NumericValue = 0</c> on a numeric row,
|
||||
/// <c>StringValue = ""</c> on a string one), so a stored NULL arrives as <c>0</c> / <c>""</c> with a Bad OPC quality
|
||||
/// (live: <c>q=1 opc=0</c>) — that 0 is not a value, and AVEVA stores a Bad-quality sample as NULL, so a Bad row never
|
||||
/// carries a real one. The value is therefore dropped whenever the mapped status is Bad, matching the SqlServer/QuestDB
|
||||
/// backends (an Empty sample is null + Bad) and the aggregate mapper's no-data rows. Uncertain rows keep their value.
|
||||
/// </remarks>
|
||||
public static DataValueSnapshot ToSnapshot(HistorianSample sample)
|
||||
{
|
||||
var status = ToStatus(sample);
|
||||
// proto3 explicit presence: prefer the numeric value, else the string value, else null.
|
||||
object? value;
|
||||
if (sample.HasNumericValue)
|
||||
if (StatusBits.IsBad(status))
|
||||
value = null;
|
||||
else if (sample.HasNumericValue)
|
||||
value = sample.NumericValue; // boxes as System.Double
|
||||
else if (sample.HasStringValue)
|
||||
value = sample.StringValue;
|
||||
@@ -36,7 +46,7 @@ internal static class SampleMapper
|
||||
|
||||
return new DataValueSnapshot(
|
||||
Value: value,
|
||||
StatusCode: ToStatus(sample),
|
||||
StatusCode: status,
|
||||
SourceTimestampUtc: sample.Timestamp?.ToDateTime(), // Utc kind
|
||||
ServerTimestampUtc: DateTime.UtcNow);
|
||||
}
|
||||
|
||||
@@ -0,0 +1,112 @@
|
||||
using ZB.MOM.WW.HistorianGateway.Contracts.Grpc;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Mapping;
|
||||
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway;
|
||||
|
||||
/// <summary>
|
||||
/// Serves a processed read of the AVEVA-native <c>TimeAverage</c> / <c>Total</c> on the SQL backends' interval grid
|
||||
/// (<see cref="BucketPlan"/>), in either direction: forward — window <c>[start, end)</c>, intervals stepping forward
|
||||
/// from <c>start</c>, the clipped interval the LATEST, each stamped with its earlier edge, oldest first; reverse (#677,
|
||||
/// #691.2) — window <c>(end, start]</c>, intervals stepping back from <c>start</c>, the clipped interval the EARLIEST,
|
||||
/// each stamped with its later edge, newest first. The gateway refuses <c>start > end</c> and <c>interval <= 0</c>,
|
||||
/// so every request it sees is forward with a positive interval.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Forward requests on the plan's grid.</b> AVEVA anchors a native read's interval grid at the request start (live,
|
||||
/// 2026-10-12: rows at <c>start + j·interval</c>, plus one extra row for the interval before it, #685). So the plan's
|
||||
/// whole intervals are requested as ONE forward read spanning exactly them with the interval <c>I</c>, and the
|
||||
/// clipped interval (when the window is not a whole number of intervals) as a second forward read over exactly its
|
||||
/// span with its own width as the interval — AVEVA never averages time outside the window (before it on a reverse
|
||||
/// read, past <c>EndTime</c> on a forward one). An interval wider than the window is sent as the window, and
|
||||
/// <c>interval <= 0</c> as one interval over the whole window (it used to reach the gateway as 0 and be refused).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Soundness of reverse.</b> A time-weighted average / integral over <c>[a, b)</c> and over <c>(a, b]</c> integrate
|
||||
/// the same function over sets differing by two instants, so a forward native row IS the value of the reverse
|
||||
/// interval with the same edges. The one place the half-open side could matter is AVEVA's status for an interval
|
||||
/// whose only non-good sample lies exactly on an edge — accepted, as on every native row.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Mapping.</b> Each row is mapped as before (<see cref="SampleMapper.ToAggregateSnapshot"/>: Uncertain keeps its
|
||||
/// value, no data ⇒ <c>BadNoData</c>, Calculated bits), placed by its interval start (else end), restamped with
|
||||
/// <see cref="BucketPlan.BucketTimestamp"/>, and the clipped interval carries the Partial bit unless Bad (Part 13
|
||||
/// §5.3.3.2 Case 3, as the SQL backends' time-weighted aggregates — which additionally mark the intervals the data
|
||||
/// starts/ends inside; AVEVA's rows carry no data range, so those are not reproduced). An interval AVEVA answers no
|
||||
/// row for is absent (not a fabricated <c>BadNoData</c>, unlike the SQL backends).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// ⚠️ Today historiangw#27 (the interval reaches AVEVA in ticks where it reads milliseconds) makes AVEVA answer ONE
|
||||
/// Good row per request — the whole-intervals read lands on its earliest interval with a whole-span value and the rest
|
||||
/// are absent — in both directions. The mapping needs no change once the gateway is fixed.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
internal static class NativeAggregatePlan
|
||||
{
|
||||
/// <summary>One forward native request and the ascending bucket index of its first interval.</summary>
|
||||
/// <param name="StartUtc">Forward request start (inclusive).</param>
|
||||
/// <param name="EndUtc">Forward request end.</param>
|
||||
/// <param name="Interval">The request's interval (> 0).</param>
|
||||
/// <param name="FirstBucket">Ascending plan bucket of the request's first interval.</param>
|
||||
/// <param name="BucketCount">Number of plan buckets the request covers.</param>
|
||||
internal readonly record struct Request(DateTime StartUtc, DateTime EndUtc, TimeSpan Interval, long FirstBucket, long BucketCount);
|
||||
|
||||
/// <summary>The forward requests serving a plan: the whole intervals first, then the clipped one (if any).</summary>
|
||||
/// <param name="plan">A plan with at least one bucket.</param>
|
||||
/// <returns>One or two forward requests.</returns>
|
||||
public static IReadOnlyList<Request> Requests(BucketPlan plan)
|
||||
{
|
||||
var interval = TimeSpan.FromTicks(plan.IntervalTicks);
|
||||
var n = plan.BucketCount;
|
||||
var clipped = plan.IsTruncated(0) ? 0L : plan.IsTruncated(n - 1) ? n - 1 : -1L;
|
||||
if (clipped < 0) return [new Request(plan.StartUtc, plan.EndUtc, interval, 0, n)];
|
||||
|
||||
var clipLo = new DateTime(plan.LowerTicks(clipped), DateTimeKind.Utc);
|
||||
var clipHi = new DateTime(plan.UpperTicks(clipped), DateTimeKind.Utc);
|
||||
var requests = new List<Request>(2);
|
||||
if (n > 1)
|
||||
requests.Add(clipped == 0
|
||||
? new Request(clipHi, plan.EndUtc, interval, 1, n - 1) // reverse: the clipped one is the earliest
|
||||
: new Request(plan.StartUtc, clipLo, interval, 0, n - 1)); // forward: the clipped one is the latest
|
||||
requests.Add(new Request(clipLo, clipHi, clipHi - clipLo, clipped, 1));
|
||||
return requests;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Maps a request's rows onto the plan's buckets: rows outside the request window (AVEVA's pre-window row, #685)
|
||||
/// are dropped; each in-window row is placed by its interval start (or, without one, its end).
|
||||
/// </summary>
|
||||
/// <param name="plan">The plan.</param>
|
||||
/// <param name="request">The forward request the rows answer.</param>
|
||||
/// <param name="rows">The drained rows.</param>
|
||||
/// <param name="byBucket">Results by ascending bucket index (first row per bucket wins).</param>
|
||||
public static void Place(BucketPlan plan, Request request, IEnumerable<HistorianAggregateSample> rows,
|
||||
Dictionary<long, DataValueSnapshot> byBucket)
|
||||
{
|
||||
var interval = request.Interval.Ticks;
|
||||
foreach (var row in rows)
|
||||
{
|
||||
if (!SampleMapper.IsInWindow(row, request.StartUtc, request.EndUtc)) continue;
|
||||
long offset = row.StartTime is { } s
|
||||
? (s.ToDateTime().Ticks - request.StartUtc.Ticks) / interval
|
||||
: (row.EndTime.ToDateTime().Ticks - request.StartUtc.Ticks - 1) / interval; // IsInWindow ⇒ end in (lo, hi]
|
||||
if (offset < 0 || offset >= request.BucketCount) continue;
|
||||
var index = request.FirstBucket + offset;
|
||||
if (byBucket.ContainsKey(index)) continue;
|
||||
|
||||
var snapshot = SampleMapper.ToAggregateSnapshot(row);
|
||||
var status = snapshot.StatusCode;
|
||||
if (plan.IsTruncated(index) && !StatusBits.IsBad(status)) status |= HistorianInfoBits.HistorianPartial;
|
||||
byBucket[index] = snapshot with { StatusCode = status, SourceTimestampUtc = plan.BucketTimestamp(index) };
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>The placed results in the read's order (ascending forward, descending reverse).</summary>
|
||||
/// <param name="plan">The plan.</param>
|
||||
/// <param name="byBucket">Results by ascending bucket index.</param>
|
||||
/// <returns>The results.</returns>
|
||||
public static IReadOnlyList<DataValueSnapshot> Assemble(BucketPlan plan, IReadOnlyDictionary<long, DataValueSnapshot> byBucket) =>
|
||||
(plan.Reversed ? byBucket.OrderByDescending(kv => kv.Key) : byBucket.OrderBy(kv => kv.Key)).Select(kv => kv.Value).ToList();
|
||||
}
|
||||
@@ -49,7 +49,10 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway;
|
||||
/// requested row budget — has been drained. The gateway is asked for at most <c>cap + 2 + </c>
|
||||
/// <see cref="PreWindowAllowance"/> rows, clamped at <see cref="GatewayDefaultMaxValuesCeiling"/> while the cap is at
|
||||
/// or under it (#688, <see cref="ClampToGatewayCeiling"/>); a stream that delivers all of them cannot be proven
|
||||
/// complete and fails the same way rather than answering from a truncated window.
|
||||
/// complete and fails the same way rather than answering from a truncated window. A strict gateway whose ceiling is
|
||||
/// below that budget (a ceiling under 1,000,000, or an unbounded cap) refuses it naming its ceiling; the window is
|
||||
/// then re-read once with the ceiling as budget (#691.3, <see cref="GatewayReadLimits"/>), and the same
|
||||
/// "delivered the whole budget ⇒ cannot be proven complete" rule applies.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Values and status.</b> A sample's value is its numeric value (Double), else its string value, else Empty; its
|
||||
@@ -163,6 +166,7 @@ internal static class RawAggregateReader
|
||||
|
||||
/// <summary>Reads and evaluates one per-sample processed read.</summary>
|
||||
/// <param name="client">The gateway client.</param>
|
||||
/// <param name="ceiling">The data source's learned gateway row ceiling (#691.3).</param>
|
||||
/// <param name="tag">Historian tag name.</param>
|
||||
/// <param name="aggregate">A per-sample aggregate (<see cref="Serves"/>).</param>
|
||||
/// <param name="plan">The bucket plan.</param>
|
||||
@@ -172,14 +176,51 @@ internal static class RawAggregateReader
|
||||
/// <returns>One value per interval, in the plan's order.</returns>
|
||||
/// <exception cref="HistoryReadLimitExceededException">The window holds more than <paramref name="maxWindowRows"/> rows,
|
||||
/// or the stream could not be read to completion within the requested row budget.</exception>
|
||||
public static async Task<IReadOnlyList<DataValueSnapshot>> ReadAsync(IHistorianGatewayClient client, string tag,
|
||||
public static async Task<IReadOnlyList<DataValueSnapshot>> ReadAsync(IHistorianGatewayClient client, GatewayRowCeiling ceiling, string tag,
|
||||
HistoryAggregateType aggregate, BucketPlan plan, int maxWindowRows, DateTime now, CancellationToken ct)
|
||||
{
|
||||
if (plan.BucketCount == 0) return [];
|
||||
var needsRange = AggregateEvaluator.NeedsDataRange(aggregate, plan);
|
||||
var requestStart = RequestStartFor(plan.StartUtc);
|
||||
var maxValues = MaxValuesFor(maxWindowRows);
|
||||
|
||||
// A strict gateway that refuses the budget is re-asked once at the ceiling it names (#691.3; e.g. a ceiling below
|
||||
// the 1,000,000 #688 clamps at, or an unbounded cap) — and a stream that fills that ceiling is never evaluated.
|
||||
var requested = MaxValuesFor(maxWindowRows);
|
||||
var (pass, maxValues) = await ceiling.ReadAsync(requested,
|
||||
budget => ReadWindowAsync(client, tag, aggregate, plan, requestStart, budget, maxWindowRows, ct),
|
||||
(r, budget) => !r.OverCap && r.Received >= budget).ConfigureAwait(false);
|
||||
var (buckets, earliest, latest, received, overCap) = pass;
|
||||
if (overCap) throw LimitExceeded(tag, maxWindowRows);
|
||||
if (received >= maxValues) throw BudgetExhausted(tag, maxWindowRows, requested, maxValues);
|
||||
|
||||
if (needsRange)
|
||||
{
|
||||
// Earlier data: no non-Bad pre-window row ⇒ ask explicitly (normally only when the tag's history really starts
|
||||
// inside the window).
|
||||
var earlierSeen = earliest is { } e && (e < plan.StartUtc || (plan.Reversed && e == plan.StartUtc));
|
||||
if (!earlierSeen && plan.StartUtc.AddTicks(-1) > ProbeFloor)
|
||||
if (await FirstSampleAsync(client, tag, ProbeFloor, plan.StartUtc.AddTicks(-1), ct).ConfigureAwait(false) is { } before)
|
||||
earliest = before;
|
||||
|
||||
// Later data: the main read covers a sample exactly at the window end (forward: that is already later data);
|
||||
// otherwise a small separate read from the end to now + horizon.
|
||||
var laterSeen = latest is { } l && (plan.Reversed ? l > plan.EndUtc : l >= plan.EndUtc);
|
||||
var horizon = DateTime.MaxValue - now > LookAheadHorizon
|
||||
? DateTime.SpecifyKind(now + LookAheadHorizon, DateTimeKind.Utc)
|
||||
: DateTime.MaxValue;
|
||||
if (!laterSeen && horizon > plan.EndUtc)
|
||||
if (await FirstSampleAsync(client, tag, plan.EndUtc, horizon, ct).ConfigureAwait(false) is { } after)
|
||||
latest = after;
|
||||
}
|
||||
|
||||
return AggregateEvaluator.EvaluateAll(aggregate, plan, buckets.Rows(), earliest, latest, now);
|
||||
}
|
||||
|
||||
/// <summary>The window read of <see cref="ReadAsync"/> with one row budget, drained.</summary>
|
||||
private static async Task<(RawBucketAccumulator Buckets, DateTime? Earliest, DateTime? Latest, long Received, bool OverCap)>
|
||||
ReadWindowAsync(IHistorianGatewayClient client, string tag, HistoryAggregateType aggregate, BucketPlan plan,
|
||||
DateTime requestStart, int maxValues, int maxWindowRows, CancellationToken ct)
|
||||
{
|
||||
var buckets = new RawBucketAccumulator(aggregate == HistoryAggregateType.MaximumActualTime);
|
||||
DateTime? earliest = null, latest = null;
|
||||
long windowRows = 0, received = 0;
|
||||
@@ -214,30 +255,7 @@ internal static class RawAggregateReader
|
||||
if (t >= plan.StartUtc && (latest is null || t > latest)) latest = t;
|
||||
}
|
||||
}
|
||||
if (overCap || (maxWindowRows > 0 && received >= maxValues))
|
||||
throw LimitExceeded(tag, maxWindowRows);
|
||||
|
||||
if (needsRange)
|
||||
{
|
||||
// Earlier data: no non-Bad pre-window row ⇒ ask explicitly (normally only when the tag's history really starts
|
||||
// inside the window).
|
||||
var earlierSeen = earliest is { } e && (e < plan.StartUtc || (plan.Reversed && e == plan.StartUtc));
|
||||
if (!earlierSeen && plan.StartUtc.AddTicks(-1) > ProbeFloor)
|
||||
if (await FirstSampleAsync(client, tag, ProbeFloor, plan.StartUtc.AddTicks(-1), ct).ConfigureAwait(false) is { } before)
|
||||
earliest = before;
|
||||
|
||||
// Later data: the main read covers a sample exactly at the window end (forward: that is already later data);
|
||||
// otherwise a small separate read from the end to now + horizon.
|
||||
var laterSeen = latest is { } l && (plan.Reversed ? l > plan.EndUtc : l >= plan.EndUtc);
|
||||
var horizon = DateTime.MaxValue - now > LookAheadHorizon
|
||||
? DateTime.SpecifyKind(now + LookAheadHorizon, DateTimeKind.Utc)
|
||||
: DateTime.MaxValue;
|
||||
if (!laterSeen && horizon > plan.EndUtc)
|
||||
if (await FirstSampleAsync(client, tag, plan.EndUtc, horizon, ct).ConfigureAwait(false) is { } after)
|
||||
latest = after;
|
||||
}
|
||||
|
||||
return AggregateEvaluator.EvaluateAll(aggregate, plan, buckets.Rows(), earliest, latest, now);
|
||||
return (buckets, earliest, latest, received, overCap);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
@@ -266,6 +284,26 @@ internal static class RawAggregateReader
|
||||
/// </summary>
|
||||
private static bool IsNotBadOnTheWire(HistorianSample raw) => (byte)raw.OpcQuality >= 64; // low byte, as SampleMapper.ToStatus reads it
|
||||
|
||||
/// <summary>
|
||||
/// The stream delivered its whole row budget, so the window cannot be proven complete. Names the limit that actually
|
||||
/// bound the budget: the gateway's ceiling (a strict gateway refused the requested budget and named it, #691.3), the
|
||||
/// #688 clamp at the gateway's default ceiling, or the configured cap.
|
||||
/// </summary>
|
||||
private static HistoryReadLimitExceededException BudgetExhausted(string tag, int maxWindowRows, int requested, int budget)
|
||||
{
|
||||
if (budget < requested)
|
||||
return new HistoryReadLimitExceededException(budget,
|
||||
$"The processed read of '{tag}' covers more raw samples than the HistorianGateway's row ceiling ({budget}, its " +
|
||||
"Historian:ReadLimits:MaxValuesCeiling with AllowStreamingBeyondCeiling=false); the Gateway backend evaluates " +
|
||||
"per-sample aggregates from raw reads — narrow the window.");
|
||||
if (maxWindowRows > 0 && budget < (long)maxWindowRows + 2 + PreWindowAllowance)
|
||||
return new HistoryReadLimitExceededException(budget,
|
||||
$"The processed read of '{tag}' covers more raw samples than the HistorianGateway's default row ceiling " +
|
||||
$"({budget}) the request is clamped at; the Gateway backend evaluates per-sample aggregates from raw reads — " +
|
||||
"narrow the window.");
|
||||
return LimitExceeded(tag, maxWindowRows);
|
||||
}
|
||||
|
||||
private static HistoryReadLimitExceededException LimitExceeded(string tag, int maxWindowRows) =>
|
||||
new(maxWindowRows,
|
||||
$"The processed read of '{tag}' covers more than {maxWindowRows} raw samples " +
|
||||
|
||||
@@ -317,6 +317,9 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
|
||||
// section is absent; NodeManager is non-null here (guarded above).
|
||||
_server.NodeManager.MaxTieClusterOverfetch = _serverHistorianOptions.MaxTieClusterOverfetch;
|
||||
|
||||
// Gitea #691: the ReturnBounds kill switch (default on).
|
||||
_server.NodeManager.HonorReturnBounds = HonorReturnBounds;
|
||||
|
||||
// Propagate the S3 HistoryRead concurrency bounds (arch-review 03/S3): bounded per-node fan-out
|
||||
// within a batch, the process-wide concurrent-batch limiter, and the per-request deadline. Defaults
|
||||
// (4 / 16 / 60 s) survive when the ServerHistorian section is absent.
|
||||
@@ -347,6 +350,12 @@ public sealed class OtOpcUaServerHostedService : IHostedService, IAsyncDisposabl
|
||||
_logger.LogInformation("OtOpcUaServerHostedService: SDK started, address-space + ServiceLevel sinks bound");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Gitea #691 — the <c>ServerHistorian:HonorReturnBounds</c> kill switch <c>StartAsync</c> applies to the node manager
|
||||
/// (default <c>true</c> when the section or key is absent). Exposed so the wiring is testable without booting the SDK.
|
||||
/// </summary>
|
||||
internal bool HonorReturnBounds => _serverHistorianOptions.HonorReturnBounds;
|
||||
|
||||
/// <summary>
|
||||
/// Gitea #674 — what <c>StartAsync</c> wires onto the node manager for HistoryUpdate: the Primary gate (read from
|
||||
/// the <see cref="IHistoryUpdateRoleView"/> on every call; null when none is registered, i.e. allow), the
|
||||
|
||||
@@ -0,0 +1,219 @@
|
||||
using Opc.Ua;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
|
||||
|
||||
/// <summary>
|
||||
/// The end bound still owed to a HistoryRead-Raw read with <c>ReturnBounds = true</c> (Gitea #691). It rides the
|
||||
/// continuation point, because Part 11 returns the end bound with the LAST page only.
|
||||
/// </summary>
|
||||
/// <param name="AtUtc">The request's end time (the later bound of a forward read, the earlier of a reverse one).</param>
|
||||
/// <param name="Reverse">The read is newest-first (<c>StartTime > EndTime</c>, including an end-only read).</param>
|
||||
/// <param name="Inclusive">
|
||||
/// <c>false</c> only for <c>StartTime == EndTime</c>: the samples AT that instant are the data, so the end bound is the
|
||||
/// first sample strictly after it (Part 11 Table 1 footnote c). Otherwise the end bound is the sample at or beyond the
|
||||
/// end time, and the data window excludes the end time.
|
||||
/// </param>
|
||||
/// <param name="Open">The end time was not given (a one-sided read's sentinel): there is no end bound to look up.</param>
|
||||
/// <param name="DataExhausted">The data is drained and the end bound did not fit on the previous page.</param>
|
||||
/// <param name="LastReturnedUtc">The last value's time returned on an earlier page (stamps an open bound's placeholder).</param>
|
||||
internal sealed record HistoryEndBound(
|
||||
DateTime AtUtc, bool Reverse, bool Inclusive, bool Open, bool DataExhausted = false, DateTime? LastReturnedUtc = null);
|
||||
|
||||
/// <summary>
|
||||
/// Gitea #691: OPC UA Part 11 bounding values for HistoryRead-Raw (<c>ReadRawModifiedDetails.ReturnBounds</c>), composed
|
||||
/// in the server from every backend's existing raw read — the ONE authority, so SqlServer, QuestDB, Gateway and Null
|
||||
/// answer identically.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>What a bound is</b> (Part 11 §4.6, Table 1). Forward: the start bound is the stored sample AT or immediately
|
||||
/// BEFORE StartTime, the end bound the one AT or immediately AFTER EndTime. Reverse mirrors it: the start bound is at
|
||||
/// or after StartTime, the end bound at or before EndTime. A sample exactly on a bound IS the bound: on the start it
|
||||
/// is already the first sample of the data (the window includes StartTime), so nothing extra is added; on the end it is
|
||||
/// added (the window excludes EndTime). <c>StartTime == EndTime</c> reads forward, and its end bound is the first
|
||||
/// sample strictly after the instant. A bound is a stored sample of any quality — a Bad sample is returned as stored.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Missing bound.</b> <c>Bad_BoundNotFound</c>, a null value, stamped with the bound's time. A one-sided read's
|
||||
/// missing side (no EndTime / no StartTime) is never looked up; it is <c>Bad_BoundNotFound</c> stamped one second after
|
||||
/// (forward) or before (reverse) the previous returned value (Table 1 footnotes a and b).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Lookups.</b> "At or before T" is a newest-first read of <c>(0, T]</c> capped at one — native on a backend that
|
||||
/// declares <see cref="IHistorianDataSource.SupportsReverseRaw"/>, otherwise <see cref="ReverseRawReader"/>'s walk-back
|
||||
/// from bounded forward reads (the Gateway: NOT AVEVA's synthetic boundary row, which is stamped at the query start
|
||||
/// rather than at the prior sample's own time). "At or after T" is a forward read of <c>[T, MaxValue)</c> capped at one (on a
|
||||
/// forward-only backend ending at now + <see cref="ReverseRawReader.LookAhead"/> instead — #691 review);
|
||||
/// "strictly after T" starts one storage resolution later (lossless: nothing is stored in between). Inside a tie
|
||||
/// cluster the bound is the cluster's first sample in the read's direction — the forward read's first, the reverse
|
||||
/// read's first (the forward order's last).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Resolution.</b> "On the bound" compares at the backend's storage resolution, because a microsecond backend
|
||||
/// (QuestDB) truncates the window bounds it is sent: a sample at <c>⌊T⌋</c> is inside a window starting at <c>T</c>
|
||||
/// there, so it must not also be returned as the start bound.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
internal static class HistoryBounds
|
||||
{
|
||||
private static readonly TimeSpan OpenBoundStep = TimeSpan.FromSeconds(1);
|
||||
|
||||
/// <summary>The end bound owed by a fresh read over the normalised window.</summary>
|
||||
/// <param name="startUtc">The (normalised) start: the later time of a reverse read.</param>
|
||||
/// <param name="endUtc">The (normalised) end; a one-sided read's sentinel (tick 0 / MaxValue) is an open bound.</param>
|
||||
/// <returns>The pending end bound.</returns>
|
||||
public static HistoryEndBound For(DateTime startUtc, DateTime endUtc) =>
|
||||
new(endUtc, Reverse: startUtc > endUtc, Inclusive: startUtc != endUtc, Open: IsOpen(endUtc));
|
||||
|
||||
/// <summary>Whether a returned value is a <see cref="NotFound"/> placeholder rather than a stored sample.</summary>
|
||||
/// <param name="value">The value.</param>
|
||||
/// <returns><c>true</c> for a <c>Bad_BoundNotFound</c> placeholder.</returns>
|
||||
public static bool IsPlaceholder(DataValueSnapshot value) =>
|
||||
value.StatusCode == StatusCodes.BadBoundNotFound && value.Value is null;
|
||||
|
||||
/// <summary>The pending end bound to store behind a continuation point, remembering the page's last returned time.</summary>
|
||||
/// <param name="bound">The pending end bound (null = bounds not requested).</param>
|
||||
/// <param name="page">The page being returned.</param>
|
||||
/// <returns>The bound to carry, or null.</returns>
|
||||
public static HistoryEndBound? Carry(HistoryEndBound? bound, IReadOnlyList<DataValueSnapshot> page) =>
|
||||
bound is null || page.Count == 0 || page[^1].SourceTimestampUtc is not { } last
|
||||
? bound
|
||||
: bound with { LastReturnedUtc = last };
|
||||
|
||||
/// <summary>The <c>Bad_BoundNotFound</c> placeholder (null value) stamped at <paramref name="atUtc"/>.</summary>
|
||||
/// <param name="atUtc">The bound's time.</param>
|
||||
/// <returns>The placeholder.</returns>
|
||||
public static DataValueSnapshot NotFound(DateTime atUtc) =>
|
||||
new(null, StatusCodes.BadBoundNotFound, atUtc, atUtc);
|
||||
|
||||
/// <summary>
|
||||
/// The start bound to return ahead of the data, or <c>null</c> when a stored sample sits on StartTime (that sample is
|
||||
/// the bound, and it is already the data's first sample).
|
||||
/// </summary>
|
||||
public static async Task<DataValueSnapshot?> StartBoundAsync(
|
||||
IHistorianDataSource source, string tagname, DateTime startUtc, bool reverse, int maxTieCluster, DateTime nowUtc,
|
||||
CancellationToken ct)
|
||||
{
|
||||
if (IsOpen(startUtc)) return NotFound(startUtc);
|
||||
var found = reverse
|
||||
? await EarliestAtOrAfterAsync(source, tagname, startUtc, nowUtc, ct).ConfigureAwait(false)
|
||||
: await LatestAtOrBeforeAsync(source, tagname, startUtc, maxTieCluster, nowUtc, ct).ConfigureAwait(false);
|
||||
if (found is null) return NotFound(startUtc);
|
||||
return SameInstant(Ticks(found), startUtc.Ticks, Resolution(source)) ? null : found;
|
||||
}
|
||||
|
||||
/// <summary>The end bound: the stored sample on or beyond the end time, else the placeholder.</summary>
|
||||
/// <param name="source">The backend.</param>
|
||||
/// <param name="tagname">Historian tag.</param>
|
||||
/// <param name="bound">The pending end bound.</param>
|
||||
/// <param name="previousReturnedUtc">The last value's time already returned (stamps an open bound's placeholder).</param>
|
||||
/// <param name="maxTieCluster">The walk-back's row budget (forward-only backends).</param>
|
||||
/// <param name="nowUtc">Now (the walk-back's look-ahead clamp).</param>
|
||||
/// <param name="ct">Cancellation.</param>
|
||||
public static async Task<DataValueSnapshot> EndBoundAsync(
|
||||
IHistorianDataSource source, string tagname, HistoryEndBound bound, DateTime previousReturnedUtc,
|
||||
int maxTieCluster, DateTime nowUtc, CancellationToken ct)
|
||||
{
|
||||
if (bound.Open)
|
||||
{
|
||||
var ticks = bound.Reverse
|
||||
? Math.Max(0L, previousReturnedUtc.Ticks - OpenBoundStep.Ticks)
|
||||
: Math.Min(DateTime.MaxValue.Ticks, previousReturnedUtc.Ticks + OpenBoundStep.Ticks);
|
||||
return NotFound(new DateTime(ticks, DateTimeKind.Utc));
|
||||
}
|
||||
|
||||
DataValueSnapshot? found;
|
||||
if (bound.Reverse)
|
||||
found = await LatestAtOrBeforeAsync(source, tagname, bound.AtUtc, maxTieCluster, nowUtc, ct).ConfigureAwait(false);
|
||||
else if (bound.Inclusive)
|
||||
found = await EarliestAtOrAfterAsync(source, tagname, bound.AtUtc, nowUtc, ct).ConfigureAwait(false);
|
||||
else
|
||||
{
|
||||
var resolution = Resolution(source);
|
||||
found = bound.AtUtc.Ticks > DateTime.MaxValue.Ticks - resolution
|
||||
? null
|
||||
: await EarliestAtOrAfterAsync(source, tagname, bound.AtUtc.AddTicks(resolution), nowUtc, ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
return found ?? NotFound(bound.AtUtc);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Whether a data sample lies inside the data window of a bounded read — a forward window excludes the end time (a
|
||||
/// sample there is the end bound), compared at the storage resolution. A backend whose forward end is inclusive, or a
|
||||
/// resumed read that steps onto the end time, would otherwise return the end bound twice.
|
||||
/// </summary>
|
||||
public static bool InDataWindow(DataValueSnapshot sample, HistoryEndBound bound, long resolution)
|
||||
{
|
||||
if (!bound.Inclusive || bound.Open) return true;
|
||||
var ts = Align(Ticks(sample), resolution);
|
||||
var end = Align(bound.AtUtc.Ticks, resolution);
|
||||
return bound.Reverse ? ts > end : ts < end;
|
||||
}
|
||||
|
||||
/// <summary>Whether a resumed read's next start is already past the data window (only the end bound remains).</summary>
|
||||
public static bool DataWindowDrained(DateTime nextStartUtc, HistoryEndBound bound, long resolution)
|
||||
{
|
||||
if (!bound.Inclusive || bound.Open) return false;
|
||||
var next = Align(nextStartUtc.Ticks, resolution);
|
||||
var end = Align(bound.AtUtc.Ticks, resolution);
|
||||
return bound.Reverse ? next <= end : next >= end;
|
||||
}
|
||||
|
||||
/// <summary>The latest stored sample at or before <paramref name="atUtc"/>, or null.</summary>
|
||||
internal static async Task<DataValueSnapshot?> LatestAtOrBeforeAsync(
|
||||
IHistorianDataSource source, string tagname, DateTime atUtc, int maxTieCluster, DateTime nowUtc, CancellationToken ct)
|
||||
{
|
||||
if (atUtc.Ticks == 0) return null;
|
||||
var floor = new DateTime(0, DateTimeKind.Utc);
|
||||
IReadOnlyList<DataValueSnapshot> rows = source.SupportsReverseRaw
|
||||
? (await source.ReadRawAsync(tagname, atUtc, floor, 1, ct).ConfigureAwait(false)).Samples
|
||||
: (await ReverseRawReader.ReadAsync(source, tagname, atUtc, floor, 1, maxTieCluster, nowUtc, 0, ct)
|
||||
.ConfigureAwait(false)).Samples;
|
||||
var limit = Align(atUtc.Ticks, Resolution(source));
|
||||
foreach (var row in rows)
|
||||
if (Align(Ticks(row), Resolution(source)) <= limit)
|
||||
return row;
|
||||
return null;
|
||||
}
|
||||
|
||||
/// <summary>The earliest stored sample at or after <paramref name="atUtc"/>, or null.</summary>
|
||||
/// <remarks>
|
||||
/// #691 review: on a backend that reads forward only (the Gateway) the read's end is clamped to now +
|
||||
/// <see cref="ReverseRawReader.LookAhead"/> — the same horizon the walk-back uses — and a time beyond it is "not found"
|
||||
/// without a read. The HistorianGateway only checks <c>start <= end</c>, so a <c>DateTime.MaxValue</c> end would
|
||||
/// reach AVEVA as a year-9999 query end (unverified: a rejection, or a fabricated row). The clamp is NOT applied to a
|
||||
/// backend that reads newest-first natively (SqlServer / QuestDB): they bind <c>MaxValue</c> safely (pinned by the
|
||||
/// <c>Extreme_bounds</c> conformance test) and can legitimately hold samples further ahead (HistoryUpdate's
|
||||
/// <c>HistoryUpdateMaxFutureSkew</c> is configurable, and driver-stamped live values carry device time) — clamping
|
||||
/// there would turn a real end bound into <c>Bad_BoundNotFound</c>.
|
||||
/// </remarks>
|
||||
internal static async Task<DataValueSnapshot?> EarliestAtOrAfterAsync(
|
||||
IHistorianDataSource source, string tagname, DateTime atUtc, DateTime nowUtc, CancellationToken ct)
|
||||
{
|
||||
var endTicks = DateTime.MaxValue.Ticks;
|
||||
if (!source.SupportsReverseRaw)
|
||||
endTicks = nowUtc.Ticks > DateTime.MaxValue.Ticks - ReverseRawReader.LookAhead.Ticks
|
||||
? DateTime.MaxValue.Ticks
|
||||
: nowUtc.Ticks + ReverseRawReader.LookAhead.Ticks;
|
||||
if (atUtc.Ticks >= endTicks) return null;
|
||||
var rows = (await source.ReadRawAsync(tagname, atUtc, new DateTime(endTicks, DateTimeKind.Utc), 1, ct)
|
||||
.ConfigureAwait(false)).Samples;
|
||||
var limit = Align(atUtc.Ticks, Resolution(source));
|
||||
foreach (var row in rows)
|
||||
if (Align(Ticks(row), Resolution(source)) >= limit)
|
||||
return row;
|
||||
return null;
|
||||
}
|
||||
|
||||
private static bool IsOpen(DateTime t) => t.Ticks == 0 || t.Ticks == DateTime.MaxValue.Ticks;
|
||||
|
||||
private static bool SameInstant(long a, long b, long resolution) => Align(a, resolution) == Align(b, resolution);
|
||||
|
||||
private static long Resolution(IHistorianDataSource source) => Math.Max(1L, source.TimestampResolutionTicks);
|
||||
|
||||
private static long Ticks(DataValueSnapshot s) => (s.SourceTimestampUtc ?? DateTime.MinValue).Ticks;
|
||||
|
||||
private static long Align(long ticks, long resolution) => ticks - ticks % resolution;
|
||||
}
|
||||
@@ -55,6 +55,17 @@ namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
|
||||
/// <see cref="HistoryCursorKind.Raw"/> reverse reads on a forward-only backend only (Gitea #686): the segment width the
|
||||
/// walk-back ended on, so the next page starts at the observed density instead of re-discovering it. 0 = none.
|
||||
/// </param>
|
||||
/// <param name="EmptyFromTicks">
|
||||
/// Walked reverse reads only (Gitea #691): with <paramref name="EmptyToTicks"/>, a stretch <c>(from, to)</c> the previous
|
||||
/// page's walk proved holds no row, so the next page steps over it rather than re-walking it (to the 1970 floor, ~14
|
||||
/// reads, at the end of a deep page-through). 0/0 = none. The cursor lives in the session's in-memory store (never
|
||||
/// serialized), so a new field with a "none" default is all the versioning it needs.
|
||||
/// </param>
|
||||
/// <param name="EmptyToTicks">The exclusive upper end of that stretch.</param>
|
||||
/// <param name="PendingEndBound">
|
||||
/// <see cref="HistoryCursorKind.Raw"/> with <c>ReturnBounds = true</c> only (Gitea #691): the end bound still owed —
|
||||
/// Part 11 returns it with the last page. <c>null</c> = bounds not requested.
|
||||
/// </param>
|
||||
internal sealed record HistoryContinuationState(
|
||||
string Tagname,
|
||||
DateTime NextStartUtc,
|
||||
@@ -65,7 +76,14 @@ internal sealed record HistoryContinuationState(
|
||||
long LastModifiedId = 0,
|
||||
DateTime WindowStartUtc = default,
|
||||
bool Reverse = false,
|
||||
long SegmentWidthTicks = 0);
|
||||
long SegmentWidthTicks = 0,
|
||||
long EmptyFromTicks = 0,
|
||||
long EmptyToTicks = 0,
|
||||
HistoryEndBound? PendingEndBound = null)
|
||||
{
|
||||
/// <summary>The walk-back hint this cursor carries (#686/#689/#691).</summary>
|
||||
public ReverseWalkHint WalkHint => new(SegmentWidthTicks, EmptyFromTicks, EmptyToTicks);
|
||||
}
|
||||
|
||||
/// <summary>The HistoryRead kind that issued a <see cref="HistoryContinuationState"/>.</summary>
|
||||
internal enum HistoryCursorKind
|
||||
|
||||
@@ -537,13 +537,20 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
/// paging over-fetches up to <c>MaxTieClusterOverfetch + 1</c> ties at that single timestamp (a
|
||||
/// <c>start == end</c> read) and slices through them via <see cref="HistoryPaging.SliceTieCluster"/>.
|
||||
/// A cluster strictly larger than this still fails that node <c>BadTooManyOperations</c> (the absurd-burst
|
||||
/// backstop; <c>BadHistoryOperationUnsupported</c> before #686). Also the per-read row budget of a reverse raw read
|
||||
/// backstop; <c>BadHistoryOperationUnsupported</c> before #686). Also the tie-cluster ceiling (not the segment size, #689) of a reverse read
|
||||
/// walked back from forward reads (<see cref="ReverseRawReader"/>). Mirrors the configured
|
||||
/// <c>ServerHistorianOptions.MaxTieClusterOverfetch</c>; the Host sets it at <c>StartAsync</c>. The
|
||||
/// default (65536) survives when the historian section is absent.
|
||||
/// </summary>
|
||||
public int MaxTieClusterOverfetch { get; set; } = 65536;
|
||||
|
||||
/// <summary>
|
||||
/// Gitea #691: whether HistoryRead-Raw honours <c>ReturnBounds</c> (Part 11 bounding values, <see cref="HistoryBounds"/>)
|
||||
/// — <c>ServerHistorian:HonorReturnBounds</c>, default <c>true</c>. <c>false</c> ignores the flag, as before #691,
|
||||
/// including the modified-read <c>Bad_InvalidArgument</c>.
|
||||
/// </summary>
|
||||
public bool HonorReturnBounds { get; set; } = true;
|
||||
|
||||
/// <summary>
|
||||
/// Maximum historized nodes served concurrently within one HistoryRead batch (arch-review 03/S3).
|
||||
/// Mirrors <c>ServerHistorianOptions.HistoryReadBatchParallelism</c>; the Host sets it at
|
||||
@@ -5056,6 +5063,15 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
var work = new List<Func<Task>>(nodesToProcess.Count);
|
||||
foreach (var handle in nodesToProcess)
|
||||
{
|
||||
if (details.IsReadModified && details.ReturnBounds && HonorReturnBounds)
|
||||
{
|
||||
// Part 11 §6.5.3.3: returnBounds "shall be set to FALSE" for a modified read, "otherwise the Server returns
|
||||
// a Bad_InvalidArgument" (#691). Off with the ServerHistorian:HonorReturnBounds kill switch.
|
||||
errors[handle.Index] = StatusCodes.BadInvalidArgument;
|
||||
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadInvalidArgument };
|
||||
continue;
|
||||
}
|
||||
|
||||
if (details.IsReadModified)
|
||||
{
|
||||
// Part 11 modified-value history (#668): the prior values every HistoryUpdate recorded. Gated by the
|
||||
@@ -5070,7 +5086,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
work.Add(() => ServeRawPagedAsync(
|
||||
handle, session, nodesToRead, results, errors,
|
||||
details.StartTime, details.EndTime, details.NumValuesPerNode,
|
||||
ct));
|
||||
ct, details.ReturnBounds && HonorReturnBounds));
|
||||
}
|
||||
|
||||
RunBounded(work, HistoryReadBatchParallelism);
|
||||
@@ -5248,6 +5264,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
var endUtc = details.EndTime;
|
||||
var numValuesPerNode = details.NumValuesPerNode;
|
||||
var boundarySkip = 0;
|
||||
var walkHint = default(ReverseWalkHint); // #691.4: a walked-back read's width + proven-empty stretch
|
||||
bool reverse;
|
||||
|
||||
if (resumed)
|
||||
@@ -5271,35 +5288,30 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
// The direction is the CURSOR's (Gitea #675), never re-derived from the resume request's details: a
|
||||
// reverse read resumes reverse, a forward one forward.
|
||||
reverse = state.Reverse;
|
||||
walkHint = state.WalkHint;
|
||||
}
|
||||
else
|
||||
{
|
||||
// Part 11: with one bound unset, an unset EndTime (MinValue) with a StartTime means "page FORWARD from
|
||||
// StartTime" — normalise it to an open upper bound, or the cursor would store MinValue and the read would
|
||||
// look reversed. An unset StartTime with an EndTime and a count cap means "page BACKWARD from EndTime"
|
||||
// (#675, mirroring raw): the reverse window (MinValue, EndTime] — but only on a backend that can read
|
||||
// newest-first. Elsewhere (Gateway, the default backend) and without a count cap an end-only read keeps its
|
||||
// historical forward [MinValue, EndTime) meaning rather than becoming a refused reverse read.
|
||||
// (#675, mirroring raw): the reverse window (MinValue, EndTime] — on every backend since #691.4 (a backend
|
||||
// that cannot read newest-first is walked back below). Without a count cap an end-only read keeps its
|
||||
// forward [MinValue, EndTime) meaning.
|
||||
if (endUtc == DateTime.MinValue && startUtc != DateTime.MinValue)
|
||||
endUtc = new DateTime(DateTime.MaxValue.Ticks, DateTimeKind.Utc); // Utc kind: a proto Timestamp conversion rejects Unspecified
|
||||
else if (numValuesPerNode > 0 && startUtc == DateTime.MinValue && endUtc != DateTime.MinValue
|
||||
&& source.SupportsReverseEvents)
|
||||
else if (numValuesPerNode > 0 && startUtc == DateTime.MinValue && endUtc != DateTime.MinValue)
|
||||
(startUtc, endUtc) = (endUtc, new DateTime(0, DateTimeKind.Utc));
|
||||
|
||||
// Part 11: StartTime later than EndTime is a reverse read — events newest-first over (end, start].
|
||||
reverse = startUtc > endUtc;
|
||||
}
|
||||
|
||||
// An explicit reverse read (or a reverse cursor resumed after the source was swapped) on a backend that cannot
|
||||
// read newest-first: refuse it BEFORE calling the source — never the earliest events in ascending order as if
|
||||
// they were the answer. Checked here rather than by catching an exception, so a NotSupportedException thrown
|
||||
// from inside a backend (a driver library, the projection) stays a backend fault.
|
||||
if (reverse && !source.SupportsReverseEvents)
|
||||
{
|
||||
errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported;
|
||||
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadHistoryOperationUnsupported };
|
||||
return;
|
||||
}
|
||||
// A reverse read on a backend that cannot read newest-first (the Gateway) is never sent there (#675 refused it
|
||||
// here); since #691.4 it is walked back from bounded forward reads — the raw walk (#686), with ties ordered by
|
||||
// EventId because the gateway orders by event time only. Decided by the capability, not by catching an
|
||||
// exception, so a NotSupportedException thrown from inside a backend stays a backend fault.
|
||||
var walked = reverse && !source.SupportsReverseEvents;
|
||||
|
||||
// NOT under the node-manager Lock — awaiting the async source is safe here.
|
||||
// NumValuesPerNode==0 means "no limit — return ALL values" per OPC UA Part 4/11, so it is translated
|
||||
@@ -5309,8 +5321,26 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
var fetch = EventMaxEvents(numValuesPerNode);
|
||||
if (numValuesPerNode > 0 && boundarySkip > 0)
|
||||
fetch = (int)Math.Min(int.MaxValue - 1L, (long)fetch + boundarySkip);
|
||||
var sourceResult = await source.ReadEventsAsync(
|
||||
sourceName, startUtc, endUtc, fetch, reverse, ct).ConfigureAwait(false);
|
||||
HistoricalEventsResult sourceResult;
|
||||
if (walked)
|
||||
{
|
||||
// Walk for one more than the fetch: finding it is the "more events exist" signal a source gives with its
|
||||
// continuation point (the walk itself cannot flag truncation).
|
||||
var want = numValuesPerNode == 0 ? 0u : (uint)Math.Min(uint.MaxValue - 1L, (long)fetch + 1);
|
||||
var (rows, next) = await ReverseEventsReader.ReadAsync(
|
||||
source, sourceName, startUtc, endUtc, want, MaxTieClusterOverfetch, Clock.GetUtcNow().UtcDateTime,
|
||||
walkHint, ct)
|
||||
.ConfigureAwait(false);
|
||||
walkHint = next;
|
||||
var more = want > 0 && rows.Count > fetch;
|
||||
if (more) rows.RemoveRange(fetch, rows.Count - fetch);
|
||||
sourceResult = new HistoricalEventsResult(rows, more ? [0x01] : null);
|
||||
}
|
||||
else
|
||||
{
|
||||
sourceResult = await source.ReadEventsAsync(
|
||||
sourceName, startUtc, endUtc, fetch, reverse, ct).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
// Truncation is the SOURCE's word (non-null continuation point) — never inferred from the count, so a
|
||||
// page that happens to be exactly cap-sized but complete never emits a point.
|
||||
@@ -5350,6 +5380,12 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
sourceName, startUtc, startUtc.AddTicks(resolution), overfetchCap, reverse: false, ct)
|
||||
.ConfigureAwait(false)).Events;
|
||||
|
||||
// A walked read (#691.4) emitted its ties sorted by EventId; the gateway returns them in any order and its
|
||||
// end is inclusive — keep exactly the cluster, in the walk's order, so the skip indexes the same events.
|
||||
if (walked)
|
||||
cluster = cluster.Where(e => e.EventTimeUtc == startUtc)
|
||||
.OrderBy(e => e, EventIdTieOrder.Instance).ToList();
|
||||
|
||||
// The over-fetch reads the cluster FORWARD (ascending EventId), but a reverse read emitted its ties — and
|
||||
// counted boundarySkip — in descending order. Slice in the read's own direction, as raw does.
|
||||
if (reverse) cluster = cluster.Reverse().ToList();
|
||||
@@ -5378,7 +5414,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
{
|
||||
pendingCursor = new HistoryContinuationState(
|
||||
sourceName ?? string.Empty, resumeAt, endUtc, nextSkip, numValuesPerNode, HistoryCursorKind.Events,
|
||||
Reverse: reverse);
|
||||
Reverse: reverse, SegmentWidthTicks: walkHint.WidthTicks, EmptyFromTicks: walkHint.EmptyFromTicks,
|
||||
EmptyToTicks: walkHint.EmptyToTicks);
|
||||
}
|
||||
}
|
||||
else if (truncated && events.Count > 0)
|
||||
@@ -5390,7 +5427,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
for (var i = events.Count - 1; i >= 0 && events[i].EventTimeUtc == boundary; i--) skip++;
|
||||
pendingCursor = new HistoryContinuationState(
|
||||
sourceName ?? string.Empty, boundary, endUtc, skip, numValuesPerNode, HistoryCursorKind.Events,
|
||||
Reverse: reverse);
|
||||
Reverse: reverse, SegmentWidthTicks: walkHint.WidthTicks, EmptyFromTicks: walkHint.EmptyFromTicks,
|
||||
EmptyToTicks: walkHint.EmptyToTicks);
|
||||
}
|
||||
|
||||
var historyEvent = ProjectEvents(events, selectClauses);
|
||||
@@ -5411,6 +5449,14 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
errors[handle.Index] = StatusCodes.BadTimeout;
|
||||
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadTimeout };
|
||||
}
|
||||
catch (HistoryReadLimitExceededException ex)
|
||||
{
|
||||
// A request limit (#691.4: a walked-back reverse read met a tie cluster over the budget) — BadTooManyOperations
|
||||
// for THIS node, as on the raw path; not a backend fault.
|
||||
_logger.LogWarning("OtOpcUaNodeManager: HistoryReadEvents limit exceeded for node {NodeId}: {Message}", handle.NodeId, ex.Message);
|
||||
errors[handle.Index] = StatusCodes.BadTooManyOperations;
|
||||
results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadTooManyOperations };
|
||||
}
|
||||
catch (Exception ex)
|
||||
{
|
||||
// One node's backend failure must not throw out of the batch — surface Bad for THIS node
|
||||
@@ -6167,6 +6213,11 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
/// <param name="startTimeUtc">The request window's (inclusive) lower bound, used for a fresh read.</param>
|
||||
/// <param name="endUtc">The (inclusive) upper bound of the read window; unchanged across pages.</param>
|
||||
/// <param name="numValuesPerNode">The client's per-page cap; <c>0</c> means "all values, no paging".</param>
|
||||
/// <param name="ct">Cancellation (the per-request deadline).</param>
|
||||
/// <param name="returnBounds">
|
||||
/// Part 11 bounding values (#691, <see cref="HistoryBounds"/>): the start bound leads the first page, the end bound
|
||||
/// ends the last, and both count toward <paramref name="numValuesPerNode"/>. A resumed read takes it from the cursor.
|
||||
/// </param>
|
||||
private async Task ServeRawPagedAsync(
|
||||
NodeHandle handle,
|
||||
ISession? session,
|
||||
@@ -6176,7 +6227,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
DateTime startTimeUtc,
|
||||
DateTime endUtc,
|
||||
uint numValuesPerNode,
|
||||
CancellationToken ct)
|
||||
CancellationToken ct,
|
||||
bool returnBounds = false)
|
||||
{
|
||||
var inboundCp = nodesToRead[handle.Index].ContinuationPoint;
|
||||
|
||||
@@ -6184,7 +6236,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
{
|
||||
DateTime startUtc;
|
||||
var boundarySkip = 0;
|
||||
long segmentWidth = 0; // #686: the walk-back's width, carried page to page
|
||||
var walkHint = default(ReverseWalkHint); // #686/#689/#691: the walk-back's width + proven-empty stretch, page to page
|
||||
HistoryEndBound? endBound = null; // #691: the end bound still owed (null = bounds not requested)
|
||||
|
||||
string tagname;
|
||||
if (inboundCp is { Length: > 0 })
|
||||
@@ -6205,7 +6258,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
boundarySkip = state.BoundarySkipCount;
|
||||
endUtc = state.EndUtc;
|
||||
numValuesPerNode = state.NumValuesPerNode;
|
||||
segmentWidth = state.SegmentWidthTicks;
|
||||
walkHint = state.WalkHint;
|
||||
endBound = state.PendingEndBound;
|
||||
}
|
||||
else
|
||||
{
|
||||
@@ -6231,6 +6285,107 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
else if (startUtc == DateTime.MinValue && endUtc != DateTime.MinValue)
|
||||
(startUtc, endUtc) = (endUtc, new DateTime(0, DateTimeKind.Utc));
|
||||
}
|
||||
|
||||
if (returnBounds) endBound = HistoryBounds.For(startUtc, endUtc);
|
||||
}
|
||||
|
||||
var nowUtc = Clock.GetUtcNow().UtcDateTime;
|
||||
var resolution = Math.Max(1L, HistorianDataSource.TimestampResolutionTicks);
|
||||
|
||||
// #691: the data is drained — the end bound ends this page when it has room (Part 11: bounds count toward
|
||||
// NumValuesPerNode), else it is the whole next page. Returns the continuation point to emit, if any.
|
||||
async Task<byte[]?> AppendEndBoundAsync(List<DataValueSnapshot> page, DateTime resumedAtUtc)
|
||||
{
|
||||
// The previous returned value stamps an open bound's placeholder (Part 11 Table 1 footnotes a/b): this
|
||||
// page's last, else the last of an earlier page (carried in the cursor), else where the read resumed.
|
||||
var previous = (page.Count > 0 ? page[^1].SourceTimestampUtc : null)
|
||||
?? endBound!.LastReturnedUtc ?? resumedAtUtc;
|
||||
if (numValuesPerNode == 0 || page.Count < numValuesPerNode)
|
||||
{
|
||||
page.Add(await HistoryBounds.EndBoundAsync(
|
||||
HistorianDataSource, tagname, endBound!, previous, MaxTieClusterOverfetch, nowUtc, ct).ConfigureAwait(false));
|
||||
return null;
|
||||
}
|
||||
|
||||
return _historyContinuationStore.Save(session, new HistoryContinuationState(
|
||||
tagname, previous, endUtc, 0, numValuesPerNode, SegmentWidthTicks: walkHint.WidthTicks,
|
||||
EmptyFromTicks: walkHint.EmptyFromTicks, EmptyToTicks: walkHint.EmptyToTicks,
|
||||
PendingEndBound: HistoryBounds.Carry(endBound!, page)! with { DataExhausted = true }));
|
||||
}
|
||||
|
||||
// #691 review: whether a bounded read whose start bound was NOT found has any data or a found end bound — only
|
||||
// asked when a one-value page would otherwise be the start-bound placeholder alone.
|
||||
async Task<bool> BoundedReadHasAnythingAsync()
|
||||
{
|
||||
var src = HistorianDataSource;
|
||||
IReadOnlyList<DataValueSnapshot> first = startUtc > endUtc && !src.SupportsReverseRaw
|
||||
? (await ReverseRawReader.ReadAsync(src, tagname, startUtc, endUtc, 1, MaxTieClusterOverfetch, nowUtc, 0, ct)
|
||||
.ConfigureAwait(false)).Samples
|
||||
: (await src.ReadRawAsync(tagname, startUtc, endUtc, 1, ct).ConfigureAwait(false)).Samples;
|
||||
if (first.Any(x => HistoryBounds.InDataWindow(x, endBound!, resolution))) return true;
|
||||
var end = await HistoryBounds.EndBoundAsync(
|
||||
src, tagname, endBound!, startUtc, MaxTieClusterOverfetch, nowUtc, ct).ConfigureAwait(false);
|
||||
return !HistoryBounds.IsPlaceholder(end);
|
||||
}
|
||||
|
||||
// #691: a resumed bounded read whose data is drained (the end bound did not fit on the previous page, or the
|
||||
// resume point already sits on the end time) owes only the end bound.
|
||||
if (endBound is not null && inboundCp is { Length: > 0 }
|
||||
&& (endBound.DataExhausted || HistoryBounds.DataWindowDrained(startUtc, endBound, resolution)))
|
||||
{
|
||||
var onlyBound = new List<DataValueSnapshot>(1);
|
||||
var afterCp = await AppendEndBoundAsync(onlyBound, startUtc).ConfigureAwait(false);
|
||||
results[handle.Index] = new SdkHistoryReadResult
|
||||
{
|
||||
StatusCode = StatusCodes.Good,
|
||||
HistoryData = new ExtensionObject(ToHistoryDataFromSamples(onlyBound)),
|
||||
ContinuationPoint = afterCp,
|
||||
};
|
||||
errors[handle.Index] = ServiceResult.Good;
|
||||
return;
|
||||
}
|
||||
|
||||
// #691: the start bound leads the first page and counts toward the cap; null when a sample sits on StartTime
|
||||
// (that sample is the bound, and it is the data's first).
|
||||
DataValueSnapshot? startBound = null;
|
||||
var readCap = numValuesPerNode;
|
||||
if (endBound is not null && inboundCp is not { Length: > 0 })
|
||||
{
|
||||
startBound = await HistoryBounds.StartBoundAsync(
|
||||
HistorianDataSource, tagname, startUtc, endBound.Reverse, MaxTieClusterOverfetch, nowUtc, ct)
|
||||
.ConfigureAwait(false);
|
||||
if (startBound is not null && numValuesPerNode == 1
|
||||
&& (!HistoryBounds.IsPlaceholder(startBound) || await BoundedReadHasAnythingAsync().ConfigureAwait(false)))
|
||||
{
|
||||
// The page is the start bound alone; the data starts at StartTime on the next page.
|
||||
var boundOnlyCp = _historyContinuationStore.Save(session, new HistoryContinuationState(
|
||||
tagname, startUtc, endUtc, 0, numValuesPerNode, SegmentWidthTicks: walkHint.WidthTicks,
|
||||
EmptyFromTicks: walkHint.EmptyFromTicks, EmptyToTicks: walkHint.EmptyToTicks,
|
||||
PendingEndBound: HistoryBounds.Carry(endBound, [startBound])));
|
||||
results[handle.Index] = new SdkHistoryReadResult
|
||||
{
|
||||
StatusCode = StatusCodes.Good,
|
||||
HistoryData = new ExtensionObject(ToHistoryDataFromSamples([startBound])),
|
||||
ContinuationPoint = boundOnlyCp,
|
||||
};
|
||||
errors[handle.Index] = ServiceResult.Good;
|
||||
return;
|
||||
}
|
||||
|
||||
if (startBound is not null && numValuesPerNode == 1)
|
||||
{
|
||||
// Nothing at all — no data and no bound found (review of #691): the pre-bounds GoodNoData answer.
|
||||
results[handle.Index] = new SdkHistoryReadResult
|
||||
{
|
||||
StatusCode = StatusCodes.GoodNoData,
|
||||
HistoryData = new ExtensionObject(ToHistoryDataFromSamples([])),
|
||||
ContinuationPoint = null,
|
||||
};
|
||||
errors[handle.Index] = ServiceResult.Good;
|
||||
return;
|
||||
}
|
||||
|
||||
if (startBound is not null && numValuesPerNode > 0) readCap = numValuesPerNode - 1;
|
||||
}
|
||||
|
||||
// HistoryRead is NOT under the node-manager Lock — awaiting the async source is safe.
|
||||
@@ -6242,26 +6397,38 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
if (startUtc > endUtc && !source.SupportsReverseRaw)
|
||||
{
|
||||
var page = await ReverseRawReader.ReadAsync(
|
||||
source, tagname, startUtc, endUtc, numValuesPerNode, MaxTieClusterOverfetch,
|
||||
Clock.GetUtcNow().UtcDateTime, segmentWidth, ct)
|
||||
source, tagname, startUtc, endUtc, readCap, MaxTieClusterOverfetch,
|
||||
nowUtc, walkHint, ct)
|
||||
.ConfigureAwait(false);
|
||||
segmentWidth = page.WidthHintTicks;
|
||||
walkHint = page.Hint;
|
||||
sourceResult = new HistorianRead(page.Samples, ContinuationPoint: null);
|
||||
}
|
||||
else
|
||||
{
|
||||
sourceResult = await source
|
||||
.ReadRawAsync(tagname, startUtc, endUtc, numValuesPerNode, ct)
|
||||
.ReadRawAsync(tagname, startUtc, endUtc, readCap, ct)
|
||||
.ConfigureAwait(false);
|
||||
}
|
||||
|
||||
var backendFull = HistoryPaging.IsFullPage(sourceResult.Samples.Count, numValuesPerNode);
|
||||
var backendFull = HistoryPaging.IsFullPage(sourceResult.Samples.Count, readCap);
|
||||
|
||||
// On a resume read, drop the boundary ties already returned on the prior page.
|
||||
var samples = inboundCp is { Length: > 0 }
|
||||
? HistoryPaging.TrimBoundaryDuplicates(sourceResult.Samples, startUtc, boundarySkip)
|
||||
: sourceResult.Samples;
|
||||
|
||||
// #691: a bounded read's data window excludes the end time (a sample there is the end bound). A backend whose
|
||||
// forward end is inclusive (the gateway's own) returns it as data; dropping it also means the data is drained.
|
||||
if (endBound is not null)
|
||||
{
|
||||
var inside = samples.Where(x => HistoryBounds.InDataWindow(x, endBound, resolution)).ToList();
|
||||
if (inside.Count != samples.Count)
|
||||
{
|
||||
samples = inside;
|
||||
backendFull = false;
|
||||
}
|
||||
}
|
||||
|
||||
// Oversized tie cluster: a resume read whose FULL backend page does not advance past the resume
|
||||
// boundary timestamp `startUtc`. That happens when more samples share `startUtc` than the page
|
||||
// cap — a tie cluster larger than NumValuesPerNode. The fixed-(start,end,cap) backend can only
|
||||
@@ -6324,9 +6491,16 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
if (nextStartUtc is { } resumeAt)
|
||||
{
|
||||
var clusterState = new HistoryContinuationState(
|
||||
tagname, resumeAt, endUtc, nextSkip, numValuesPerNode, SegmentWidthTicks: segmentWidth);
|
||||
tagname, resumeAt, endUtc, nextSkip, numValuesPerNode, SegmentWidthTicks: walkHint.WidthTicks,
|
||||
EmptyFromTicks: walkHint.EmptyFromTicks, EmptyToTicks: walkHint.EmptyToTicks,
|
||||
PendingEndBound: HistoryBounds.Carry(endBound, slice));
|
||||
clusterCp = _historyContinuationStore.Save(session, clusterState);
|
||||
}
|
||||
else if (endBound is not null)
|
||||
{
|
||||
// #691: the data is drained — the end bound ends this page, or is the next one when this one is full.
|
||||
clusterCp = await AppendEndBoundAsync(slice, startUtc).ConfigureAwait(false);
|
||||
}
|
||||
|
||||
results[handle.Index] = new SdkHistoryReadResult
|
||||
{
|
||||
@@ -6338,6 +6512,14 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
return;
|
||||
}
|
||||
|
||||
// #691: the start bound (first page only) leads the data.
|
||||
if (startBound is not null)
|
||||
{
|
||||
var led = new List<DataValueSnapshot>(samples.Count + 2) { startBound };
|
||||
led.AddRange(samples);
|
||||
samples = led;
|
||||
}
|
||||
|
||||
// The "full page" test is against the RAW backend count (before trimming): the backend honoured
|
||||
// the cap, so a full backend page ⇒ there may be more even if we trimmed some boundary ties.
|
||||
var historyData = ToHistoryDataFromSamples(samples);
|
||||
@@ -6347,11 +6529,31 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
|
||||
{
|
||||
HistoryPaging.ComputeResumeCursor(samples, out var nextStart, out var skip);
|
||||
var nextState = new HistoryContinuationState(
|
||||
tagname, nextStart, endUtc, skip, numValuesPerNode, SegmentWidthTicks: segmentWidth);
|
||||
tagname, nextStart, endUtc, skip, numValuesPerNode, SegmentWidthTicks: walkHint.WidthTicks,
|
||||
EmptyFromTicks: walkHint.EmptyFromTicks, EmptyToTicks: walkHint.EmptyToTicks,
|
||||
PendingEndBound: HistoryBounds.Carry(endBound, samples));
|
||||
// Save may return null (no session on this request) ⇒ degrade to single-shot for this node.
|
||||
// Built AFTER historyData so a failure projecting samples can never orphan a stored cursor.
|
||||
outboundCp = _historyContinuationStore.Save(session, nextState);
|
||||
}
|
||||
else if (endBound is not null)
|
||||
{
|
||||
// #691: the data is drained — the end bound ends this page, or is the next one when this one is full.
|
||||
var page = new List<DataValueSnapshot>(samples.Count + 1);
|
||||
page.AddRange(samples);
|
||||
outboundCp = await AppendEndBoundAsync(page, startUtc).ConfigureAwait(false);
|
||||
samples = page;
|
||||
historyData = ToHistoryDataFromSamples(samples);
|
||||
}
|
||||
|
||||
// #691 review: a whole bounded read with no data and no bound found (one page, every value a placeholder) is the
|
||||
// pre-bounds answer — GoodNoData, no values (§6.5.3.2: bounds make it a success only "if one or both exist").
|
||||
if (endBound is not null && inboundCp is not { Length: > 0 } && outboundCp is null
|
||||
&& samples.All(HistoryBounds.IsPlaceholder))
|
||||
{
|
||||
samples = [];
|
||||
historyData = ToHistoryDataFromSamples(samples);
|
||||
}
|
||||
|
||||
results[handle.Index] = new SdkHistoryReadResult
|
||||
{
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
|
||||
|
||||
/// <summary>One walked-back reverse events page (<see cref="ReverseEventsReader"/>).</summary>
|
||||
/// <param name="Events">At most the requested count, newest first by <c>(EventTimeUtc, EventId)</c>.</param>
|
||||
/// <param name="Hint">What the next page of the same read resumes with.</param>
|
||||
internal readonly record struct ReverseEventsPage(List<HistoricalEvent> Events, ReverseWalkHint Hint);
|
||||
|
||||
/// <summary>
|
||||
/// Gitea #691.4: serves a Part 11 reverse events read — <c>(lo, hi]</c> newest-first — on a backend that reads events
|
||||
/// forward only (<see cref="IHistorianDataSource.SupportsReverseEvents"/> = <c>false</c>: the Gateway) with the
|
||||
/// <see cref="ReverseWalk"/>. A segment read is cut short when the source flags truncation (its continuation point); ties
|
||||
/// are ordered by <see cref="EventIdTieOrder"/> because the gateway orders by event time only.
|
||||
/// </summary>
|
||||
internal static class ReverseEventsReader
|
||||
{
|
||||
/// <summary>Reads <c>(lo, hi]</c> newest-first from forward event reads.</summary>
|
||||
/// <param name="source">A backend that reads events forward.</param>
|
||||
/// <param name="sourceName">Event-source filter (null = all).</param>
|
||||
/// <param name="hi">The later (inclusive) bound.</param>
|
||||
/// <param name="lo">The earlier (exclusive) bound.</param>
|
||||
/// <param name="want">How many events; <c>0</c> = all in the window (one forward read).</param>
|
||||
/// <param name="maxTieCluster">The tie budget (<c>MaxTieClusterOverfetch</c>) — a ceiling only.</param>
|
||||
/// <param name="nowUtc">Now, for the look-ahead clamp.</param>
|
||||
/// <param name="hint">What the previous page carried (default = none).</param>
|
||||
/// <param name="ct">Cancellation.</param>
|
||||
/// <returns>The events, newest first, and the hint for the next page.</returns>
|
||||
/// <exception cref="HistoryReadLimitExceededException">One timestamp holds more than the tie budget.</exception>
|
||||
public static async Task<ReverseEventsPage> ReadAsync(
|
||||
IHistorianDataSource source, string? sourceName, DateTime hi, DateTime lo, uint want, int maxTieCluster,
|
||||
DateTime nowUtc, ReverseWalkHint hint, CancellationToken ct)
|
||||
{
|
||||
var budget = Math.Max(1, maxTieCluster);
|
||||
var (rows, next) = await ReverseWalk.WalkAsync<HistoricalEvent>(
|
||||
async (start, end, cap, token) =>
|
||||
{
|
||||
// cap 0 = the whole window: int.MaxValue is the events contract's "no cap" (a non-positive value would
|
||||
// mean the backend's DEFAULT cap).
|
||||
var r = await source.ReadEventsAsync(sourceName, start, end, cap == 0 ? int.MaxValue : cap, reverse: false, token)
|
||||
.ConfigureAwait(false);
|
||||
return new ForwardSegmentRead<HistoricalEvent>(r.Events, r.ContinuationPoint is not null);
|
||||
},
|
||||
ev => ev.EventTimeUtc.Ticks, EventIdTieOrder.Instance, hi, lo, want, budget, source.TimestampResolutionTicks,
|
||||
nowUtc, hint,
|
||||
at => new HistoryReadLimitExceededException(budget,
|
||||
$"A reverse events read of '{sourceName}' at {at:O} found more than {budget} events within one millisecond; the " +
|
||||
"backend reads forward only, so the latest of them cannot be selected. Increase " +
|
||||
"ServerHistorian:MaxTieClusterOverfetch."),
|
||||
ct)
|
||||
.ConfigureAwait(false);
|
||||
return new ReverseEventsPage(rows, next);
|
||||
}
|
||||
}
|
||||
@@ -5,70 +5,29 @@ namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
|
||||
/// <summary>One walked-back reverse raw page (<see cref="ReverseRawReader"/>).</summary>
|
||||
/// <param name="Samples">At most the requested cap, newest first.</param>
|
||||
/// <param name="WidthHintTicks">The segment width the walk ended on — the next page's starting width (0 = none).</param>
|
||||
internal readonly record struct ReverseRawPage(IReadOnlyList<DataValueSnapshot> Samples, long WidthHintTicks);
|
||||
internal readonly record struct ReverseRawPage(IReadOnlyList<DataValueSnapshot> Samples, long WidthHintTicks)
|
||||
{
|
||||
/// <summary>Everything the next page of the same read should resume with (its width is <see cref="WidthHintTicks"/>).</summary>
|
||||
public ReverseWalkHint Hint { get; init; } = new(WidthHintTicks);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Gitea #686: serves a Part 11 reverse raw read — <c>(lo, hi]</c> newest-first, capped — on a historian backend that can
|
||||
/// only read forward (<see cref="IHistorianDataSource.SupportsReverseRaw"/> = <c>false</c>: the Gateway, whose
|
||||
/// <c>ReadRaw</c> streams ascending and rejects <c>start > end</c>).
|
||||
/// <c>ReadRaw</c> streams ascending and rejects <c>start > end</c>). The walk itself — segment sizing (#689), the
|
||||
/// truncation rules, edges, ties and bounds — is <see cref="ReverseWalk"/>; this adapter supplies the raw read (a read is
|
||||
/// cut short when it returned its full cap) and keeps the backend's tie order.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Uncapped</b> (<c>maxValues = 0</c>): ONE forward read of the whole window, filtered and reversed in memory — the
|
||||
/// same rows, round trips and memory as an uncapped forward read (and as a native backend's uncapped reverse read).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Capped: walk back in segments.</b> Starting at <c>hi</c>, read one segment <c>(segLo, segHi]</c> FORWARD, keep
|
||||
/// it newest-first, and step to the next older segment until the cap is met or <c>lo</c> is reached. The first
|
||||
/// segment is <see cref="InitialSegment"/> wide, or the width the previous page of the same read ended on (carried in
|
||||
/// the continuation point). After each read the next width is re-aimed at about half the row budget from the density
|
||||
/// just observed, growing at most ×4 per step — so a live tag's "latest N" is usually one read, a dense tag does not
|
||||
/// throw away a full read per page, and sparse history is found in a few dozen tiny reads.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Never answer from a truncated read.</b> The row budget is <c>M = max(1, MaxTieClusterOverfetch)</c>; a segment
|
||||
/// read asks for <c>M + 2</c> rows (room for the row an AVEVA read fabricates at its query start, and one more to see
|
||||
/// past the segment). Rows stream ascending, so a read is complete for its segment when it came back short OR its last
|
||||
/// row is already past <c>segHi</c>. An incomplete read is discarded: when rows BEFORE the segment (a neighbouring
|
||||
/// tie cluster inside the alignment slack, the fabricated row) crowded it out it is re-read once with room for them
|
||||
/// (at most <c>2M + 4</c> rows); otherwise the segment is narrowed (to about half the budget at the density the
|
||||
/// truncated read showed) and re-read. Only a single timestamp holding more than <c>M</c> samples — a segment already
|
||||
/// one timestamp-resolution wide that still does not fit — is refused: <see cref="HistoryReadLimitExceededException"/>
|
||||
/// (⇒ <c>BadTooManyOperations</c>, a request limit, never a backend fault).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Edges.</b> A segment is requested as <c>[⌊segLo⌋, ⌈segHi + 1 tick⌉]</c>, both rounded outward to the millisecond
|
||||
/// (the gateway's / AVEVA's resolution), and filtered to <c>(segLo, segHi]</c> on the samples' own timestamps. That is
|
||||
/// exact whether the backend's forward end is exclusive (the interface contract) or inclusive (the gateway) and
|
||||
/// whether it truncates its bounds (QuestDB-style); and because the request start is already on a millisecond, a row
|
||||
/// an AVEVA read fabricates at its (possibly rounded-up) query start can never land inside the segment (#687).
|
||||
/// Adjacent segments share no timestamp, so nothing is returned twice.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Ties</b> come back in the backend's forward order reversed — the same order the node manager's tie-cluster
|
||||
/// over-fetch uses for a reverse read — so a resumed page's boundary-tie skip lines up.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Bounds.</b> The walk never asks for data after now + <see cref="LookAhead"/> (one day — the historian stores
|
||||
/// nothing further ahead; a <c>StartTime = DateTime.MaxValue</c> read would otherwise send centuries-ahead query
|
||||
/// starts), nor before <see cref="Floor"/> (the Unix epoch — AVEVA holds nothing earlier, and an end-only read's year-1
|
||||
/// lower bound is outside the historian's time encoding). A walked-back read therefore returns only samples strictly
|
||||
/// after the epoch and no later than now + one day.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
internal static class ReverseRawReader
|
||||
{
|
||||
/// <summary>The first segment's width when no hint is carried.</summary>
|
||||
internal static readonly TimeSpan InitialSegment = TimeSpan.FromMinutes(1);
|
||||
internal static readonly TimeSpan InitialSegment = ReverseWalk.InitialSegment;
|
||||
|
||||
/// <summary>The walk's exclusive lower limit (see the remarks).</summary>
|
||||
internal static readonly DateTime Floor = DateTime.UnixEpoch;
|
||||
/// <summary>The walk's exclusive lower limit.</summary>
|
||||
internal static readonly DateTime Floor = ReverseWalk.Floor;
|
||||
|
||||
/// <summary>How far past "now" the walk starts at most (the same horizon as the Gateway's raw-aggregate look-ahead).</summary>
|
||||
internal static readonly TimeSpan LookAhead = TimeSpan.FromDays(1);
|
||||
|
||||
/// <summary>Request bounds are rounded outward to this (the gateway / AVEVA timestamp resolution).</summary>
|
||||
private const long Alignment = TimeSpan.TicksPerMillisecond;
|
||||
/// <summary>How far past "now" the walk starts at most.</summary>
|
||||
internal static readonly TimeSpan LookAhead = ReverseWalk.LookAhead;
|
||||
|
||||
/// <summary>Reads <c>(lo, hi]</c> newest-first from forward reads.</summary>
|
||||
/// <param name="source">A backend that reads forward.</param>
|
||||
@@ -76,129 +35,49 @@ internal static class ReverseRawReader
|
||||
/// <param name="hi">The later (inclusive) bound — the request's StartTime or a reverse resume point.</param>
|
||||
/// <param name="lo">The earlier (exclusive) bound.</param>
|
||||
/// <param name="maxValues">Cap; <c>0</c> = every sample in the window (one forward read).</param>
|
||||
/// <param name="maxTieCluster">Row budget per segment read (<c>MaxTieClusterOverfetch</c>; values below 1 count as 1).</param>
|
||||
/// <param name="maxTieCluster">The tie budget (<c>MaxTieClusterOverfetch</c>; values below 1 count as 1) — a ceiling only.</param>
|
||||
/// <param name="nowUtc">Now, for the look-ahead clamp.</param>
|
||||
/// <param name="widthHintTicks">Starting segment width (the previous page's <see cref="ReverseRawPage.WidthHintTicks"/>), 0 = none.</param>
|
||||
/// <param name="ct">Cancellation.</param>
|
||||
/// <returns>The page, newest first, and the width to start the next page with.</returns>
|
||||
/// <exception cref="HistoryReadLimitExceededException">One timestamp holds more than the row budget.</exception>
|
||||
/// <exception cref="HistoryReadLimitExceededException">One timestamp holds more than the tie budget.</exception>
|
||||
public static Task<ReverseRawPage> ReadAsync(
|
||||
IHistorianDataSource source, string tagname, DateTime hi, DateTime lo, uint maxValues, int maxTieCluster,
|
||||
DateTime nowUtc, long widthHintTicks, CancellationToken ct) =>
|
||||
ReadAsync(source, tagname, hi, lo, maxValues, maxTieCluster, nowUtc, new ReverseWalkHint(widthHintTicks), ct);
|
||||
|
||||
/// <summary>Reads <c>(lo, hi]</c> newest-first from forward reads, resuming with the previous page's full hint.</summary>
|
||||
/// <param name="source">A backend that reads forward.</param>
|
||||
/// <param name="tagname">Historian tag.</param>
|
||||
/// <param name="hi">The later (inclusive) bound.</param>
|
||||
/// <param name="lo">The earlier (exclusive) bound.</param>
|
||||
/// <param name="maxValues">Cap; <c>0</c> = every sample in the window (one forward read).</param>
|
||||
/// <param name="maxTieCluster">The tie budget — a ceiling only.</param>
|
||||
/// <param name="nowUtc">Now, for the look-ahead clamp.</param>
|
||||
/// <param name="hint">What the previous page carried (<see cref="ReverseRawPage.Hint"/>), default = none.</param>
|
||||
/// <param name="ct">Cancellation.</param>
|
||||
/// <returns>The page, newest first, and the hint for the next page.</returns>
|
||||
/// <exception cref="HistoryReadLimitExceededException">One timestamp holds more than the tie budget.</exception>
|
||||
public static async Task<ReverseRawPage> ReadAsync(
|
||||
IHistorianDataSource source, string tagname, DateTime hi, DateTime lo, uint maxValues, int maxTieCluster,
|
||||
DateTime nowUtc, long widthHintTicks, CancellationToken ct)
|
||||
DateTime nowUtc, ReverseWalkHint hint, CancellationToken ct)
|
||||
{
|
||||
var resolution = Math.Max(1L, source.TimestampResolutionTicks);
|
||||
var granularity = Math.Max(Alignment, resolution);
|
||||
var budget = Math.Max(1, maxTieCluster);
|
||||
var result = new List<DataValueSnapshot>();
|
||||
|
||||
var horizon = nowUtc.Ticks > DateTime.MaxValue.Ticks - LookAhead.Ticks ? DateTime.MaxValue.Ticks : nowUtc.Ticks + LookAhead.Ticks;
|
||||
var hiTicks = Math.Min(hi.Ticks, horizon);
|
||||
var stopTicks = Math.Max(lo.Ticks, Floor.Ticks);
|
||||
if (hiTicks <= stopTicks) return new ReverseRawPage(result, widthHintTicks);
|
||||
|
||||
if (maxValues == 0)
|
||||
{
|
||||
// One forward read of the whole window, reversed — the uncapped forward read's own cost and bound.
|
||||
var all = (await source.ReadRawAsync(tagname, Utc(AlignDown(stopTicks, granularity)),
|
||||
Utc(AlignUp(hiTicks + 1, granularity)), 0, ct).ConfigureAwait(false)).Samples;
|
||||
for (var i = all.Count - 1; i >= 0; i--)
|
||||
{
|
||||
var ts = Ticks(all[i]);
|
||||
if (ts > stopTicks && ts <= hiTicks) result.Add(all[i]);
|
||||
}
|
||||
|
||||
return new ReverseRawPage(result, widthHintTicks);
|
||||
}
|
||||
|
||||
var maxWidth = DateTime.MaxValue.Ticks;
|
||||
var width = widthHintTicks > 0 ? Math.Clamp(widthHintTicks, resolution, maxWidth) : Math.Max(resolution, InitialSegment.Ticks);
|
||||
var baseCap = budget + 2;
|
||||
var maxCap = 2 * budget + 4;
|
||||
var segHi = hiTicks;
|
||||
|
||||
while (segHi > stopTicks && result.Count < maxValues)
|
||||
{
|
||||
var segLo = segHi - stopTicks <= width ? stopTicks : segHi - width;
|
||||
var start = Utc(AlignDown(segLo, granularity));
|
||||
var end = Utc(AlignUp(segHi + 1, granularity));
|
||||
var cap = baseCap;
|
||||
IReadOnlyList<DataValueSnapshot> rows;
|
||||
var narrowed = false;
|
||||
|
||||
while (true)
|
||||
{
|
||||
ct.ThrowIfCancellationRequested();
|
||||
rows = (await source.ReadRawAsync(tagname, start, end, (uint)cap, ct).ConfigureAwait(false)).Samples;
|
||||
if (rows.Count < cap || Ticks(rows[^1]) > segHi) break; // complete for (segLo, segHi]
|
||||
|
||||
var leading = 0;
|
||||
var inner = 0;
|
||||
var firstInner = 0L;
|
||||
foreach (var r in rows)
|
||||
var (rows, next) = await ReverseWalk.WalkAsync(
|
||||
async (start, end, cap, token) =>
|
||||
{
|
||||
var ts = Ticks(r);
|
||||
if (ts <= segLo) leading++;
|
||||
else if (ts <= segHi && inner++ == 0) firstInner = ts;
|
||||
}
|
||||
|
||||
// Crowded out by rows before the segment, with the segment itself possibly within budget: re-read once
|
||||
// with room for them.
|
||||
var roomier = leading + budget + 2;
|
||||
if (leading > 0 && inner <= budget && roomier > cap && roomier <= maxCap)
|
||||
{
|
||||
cap = roomier;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (width <= resolution)
|
||||
throw new HistoryReadLimitExceededException(budget,
|
||||
$"A reverse raw read of '{tagname}' at {Utc(segHi):O} found more than {budget} samples at one " +
|
||||
"timestamp; the backend reads forward only, so the latest of them cannot be selected. Increase " +
|
||||
"ServerHistorian:MaxTieClusterOverfetch.");
|
||||
|
||||
// Narrow to about half the budget at the density the truncated read's own samples showed (their span, not
|
||||
// the segment's — the segment may open on an empty stretch), at least halving.
|
||||
var lastTs = Ticks(rows[^1]);
|
||||
var aimed = inner > 1 && lastTs > firstInner
|
||||
? (long)Math.Min((double)(lastTs - firstInner) * (budget / 2.0) / (inner - 1), width / 2.0)
|
||||
: width / 4;
|
||||
width = Math.Max(resolution, aimed);
|
||||
narrowed = true;
|
||||
break;
|
||||
}
|
||||
|
||||
if (narrowed) continue; // re-read the same segHi, narrower
|
||||
|
||||
var count = 0;
|
||||
for (var i = rows.Count - 1; i >= 0; i--)
|
||||
{
|
||||
var ts = Ticks(rows[i]);
|
||||
if (ts <= segLo || ts > segHi) continue;
|
||||
count++;
|
||||
if (result.Count < maxValues) result.Add(rows[i]);
|
||||
}
|
||||
|
||||
// Re-aim at about half the budget from the density just seen; grow at most x4.
|
||||
var grown = width > maxWidth / 4 ? maxWidth : width * 4;
|
||||
width = count == 0
|
||||
? grown
|
||||
: (long)Math.Clamp((double)width * Math.Max(1.0, budget / 2.0) / count, resolution, grown);
|
||||
segHi = segLo;
|
||||
}
|
||||
|
||||
return new ReverseRawPage(result, width);
|
||||
var samples = (await source.ReadRawAsync(tagname, start, end, (uint)cap, token).ConfigureAwait(false)).Samples;
|
||||
return new ForwardSegmentRead<DataValueSnapshot>(samples, cap > 0 && samples.Count >= cap);
|
||||
},
|
||||
Ticks, tieOrder: null, hi, lo, maxValues, budget, source.TimestampResolutionTicks, nowUtc, hint,
|
||||
at => new HistoryReadLimitExceededException(budget,
|
||||
$"A reverse raw read of '{tagname}' at {at:O} found more than {budget} samples within one millisecond; the " +
|
||||
"backend reads forward only, so the latest of them cannot be selected. Increase " +
|
||||
"ServerHistorian:MaxTieClusterOverfetch."),
|
||||
ct)
|
||||
.ConfigureAwait(false);
|
||||
return new ReverseRawPage(rows, next.WidthTicks) { Hint = next };
|
||||
}
|
||||
|
||||
private static long Ticks(DataValueSnapshot s) => (s.SourceTimestampUtc ?? DateTime.MinValue).Ticks;
|
||||
|
||||
private static DateTime Utc(long ticks) => new(ticks, DateTimeKind.Utc);
|
||||
|
||||
private static long AlignDown(long ticks, long step) => ticks - ticks % step;
|
||||
|
||||
private static long AlignUp(long ticks, long step)
|
||||
{
|
||||
var down = AlignDown(ticks, step);
|
||||
if (down == ticks) return ticks;
|
||||
return down > DateTime.MaxValue.Ticks - step ? DateTime.MaxValue.Ticks : down + step;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -0,0 +1,448 @@
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
|
||||
|
||||
/// <summary>What one walked-back page tells the next page of the same read (carried in the continuation point).</summary>
|
||||
/// <param name="WidthTicks">The segment width the walk ended on, aimed at the page's density — the next page's first width (0 = none).</param>
|
||||
/// <param name="EmptyFromTicks">
|
||||
/// With <paramref name="EmptyToTicks"/>: a stretch <c>(EmptyFrom, EmptyTo)</c> (both exclusive) a complete segment read
|
||||
/// proved holds no row (#691) — typically from the oldest row the page saw down to the walk's floor. A resumed page
|
||||
/// steps straight over it instead of re-walking it. Both 0 = none.
|
||||
/// </param>
|
||||
/// <param name="EmptyToTicks">The exclusive upper end of the empty stretch.</param>
|
||||
internal readonly record struct ReverseWalkHint(long WidthTicks, long EmptyFromTicks = 0, long EmptyToTicks = 0)
|
||||
{
|
||||
/// <summary>Whether an empty stretch is carried.</summary>
|
||||
public bool HasEmptyStretch => EmptyToTicks > EmptyFromTicks && EmptyToTicks > 0;
|
||||
}
|
||||
|
||||
/// <summary>One forward read of a segment: the rows (ascending by time) and whether a cap cut the read short.</summary>
|
||||
/// <typeparam name="T">Row type.</typeparam>
|
||||
/// <param name="Rows">Rows in ascending time order (ties in any order).</param>
|
||||
/// <param name="Truncated">A cap stopped the read — more rows may follow the last one.</param>
|
||||
internal readonly record struct ForwardSegmentRead<T>(IReadOnlyList<T> Rows, bool Truncated);
|
||||
|
||||
/// <summary>
|
||||
/// Gitea #686/#689: the walk-back that serves a Part 11 reverse read — <c>(lo, hi]</c> newest-first, capped — from a
|
||||
/// backend that can only read FORWARD. Shared by raw values (<see cref="ReverseRawReader"/>) and events; the row type
|
||||
/// only has to expose a timestamp.
|
||||
/// </summary>
|
||||
/// <remarks>
|
||||
/// <para>
|
||||
/// <b>Uncapped</b> (<c>want = 0</c>): ONE forward read of the whole window, filtered and reversed in memory.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Capped: walk back in segments.</b> Starting at <c>hi</c>, read one segment <c>(segLo, segHi]</c> FORWARD, keep
|
||||
/// it newest-first, and step to the next older segment until the cap is met or <c>lo</c> is reached. The first
|
||||
/// segment is <see cref="InitialSegment"/> wide, or the width the previous page of the same read ended on (carried in
|
||||
/// the continuation point). After each read the next width is re-aimed from the density just observed at about half
|
||||
/// the PAGE budget <c>2N + 16</c> (≈ N + 8 rows), growing at most ×4 per step (#689 — it used to aim at half the tie
|
||||
/// budget, ~32k rows, whatever N was) — so a live tag's "latest N" is usually one read of about N rows, and sparse
|
||||
/// history is found in a few dozen tiny reads.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Read sizes.</b> A segment read asks for its budget + 2 rows (room for a row an AVEVA read fabricates at its query
|
||||
/// start, and one to see past the segment). The budget is the page budget, except a <b>discovery</b> segment — the
|
||||
/// first of a page with no carried width, or one after an empty segment (whose ×4 growth may overshoot into dense
|
||||
/// data) — which may read up to <see cref="DiscoveryRows"/>. The tie budget <c>M = max(1, MaxTieClusterOverfetch)</c>
|
||||
/// is only a CEILING: a single timestamp holding more rows than the segment budget escalates to <c>M</c>.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Never answer from a truncated read.</b> Rows stream ascending, so a read is complete for its segment when it was
|
||||
/// not cut short OR its last row is already past <c>segHi</c>. An incomplete read is discarded: when rows BEFORE the
|
||||
/// segment (a neighbouring tie cluster inside the alignment slack, a fabricated row) crowded it out it is re-read once
|
||||
/// with room for them (at most twice the budget + 4); otherwise the segment is narrowed — to the density the truncated
|
||||
/// read showed, and never again below the newest row it saw (<c>known</c>: rows exist there, so walking past it before
|
||||
/// finding what lies above it would re-read the same dense stretch over and over; the gap above it is bisected
|
||||
/// instead). Only more than <c>M</c> rows within one millisecond — a segment no wider than the 1 ms request
|
||||
/// granularity that still does not fit at the tie budget — is refused (per millisecond, not per timestamp: narrower
|
||||
/// requests are the same request, and the Gateway stores milliseconds).
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Proving the rest empty (#691).</b> Walking down to the floor through empty history used to take one read per ×4
|
||||
/// growth step (~14 reads from a minute to the epoch) — on every page that ran out of rows. After two consecutive empty
|
||||
/// segments the walk asks once per page for the WHOLE remainder <c>(lo, segHi]</c> at the page budget: a complete
|
||||
/// answer ends the walk in one read; a truncated one (deep history) is discarded and the walk carries on as before. And
|
||||
/// the stretch a page's last complete segment proved empty below its oldest row rides the continuation point
|
||||
/// (<see cref="ReverseWalkHint.EmptyFromTicks"/>), so the next page steps over it — the rows of a page are often the
|
||||
/// last ones before a stretch the previous page already read.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Edges.</b> A segment is requested as <c>[⌊segLo⌋, ⌈segHi + 1 tick⌉]</c>, both rounded outward to the millisecond
|
||||
/// (the gateway's / AVEVA's resolution), and filtered to <c>(segLo, segHi]</c> on the rows' own timestamps — exact
|
||||
/// whether the backend's forward end is exclusive or inclusive and whether it truncates its bounds; a row fabricated at
|
||||
/// the (millisecond) query start can never land inside the segment (#687). Adjacent segments share no timestamp.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Ties</b> come back in the backend's forward order reversed, or — with a tie order given (events, whose gateway
|
||||
/// order within a timestamp is arbitrary) — sorted by it and reversed, so a resumed page's boundary-tie skip lines up.
|
||||
/// </para>
|
||||
/// <para>
|
||||
/// <b>Bounds.</b> The walk never asks for data after now + <see cref="LookAhead"/> nor before <see cref="Floor"/> (the
|
||||
/// Unix epoch — AVEVA holds nothing earlier). A walked-back read returns only rows strictly after the epoch and no
|
||||
/// later than now + one day.
|
||||
/// </para>
|
||||
/// </remarks>
|
||||
internal static class ReverseWalk
|
||||
{
|
||||
/// <summary>The first segment's width when no hint is carried.</summary>
|
||||
internal static readonly TimeSpan InitialSegment = TimeSpan.FromMinutes(1);
|
||||
|
||||
/// <summary>The walk's exclusive lower limit.</summary>
|
||||
internal static readonly DateTime Floor = DateTime.UnixEpoch;
|
||||
|
||||
/// <summary>How far past "now" the walk starts at most.</summary>
|
||||
internal static readonly TimeSpan LookAhead = TimeSpan.FromDays(1);
|
||||
|
||||
/// <summary>Slack added to a page's row budget (#689): room for density variation between segments.</summary>
|
||||
internal const int PageSlack = 16;
|
||||
|
||||
/// <summary>The most rows a discovery segment (see the remarks) may read before it counts as truncated (#689).</summary>
|
||||
internal const int DiscoveryRows = 1024;
|
||||
|
||||
/// <summary>Request bounds are rounded outward to this (the gateway / AVEVA timestamp resolution).</summary>
|
||||
private const long Alignment = TimeSpan.TicksPerMillisecond;
|
||||
|
||||
/// <summary>The rows a segment is sized for: <c>2N + </c><see cref="PageSlack"/>, never above the tie budget.</summary>
|
||||
/// <param name="want">The page cap N (> 0).</param>
|
||||
/// <param name="tieBudget">The tie budget M (>= 1).</param>
|
||||
/// <returns>The page budget.</returns>
|
||||
internal static int PageBudget(uint want, int tieBudget) => (int)Math.Clamp(2L * want + PageSlack, 1L, Math.Max(1, tieBudget));
|
||||
|
||||
/// <summary>Reads <c>(lo, hi]</c> newest-first from forward reads.</summary>
|
||||
/// <typeparam name="T">Row type.</typeparam>
|
||||
/// <param name="read">One forward read <c>[start, end]</c> asking for at most <c>cap</c> rows (<c>0</c> = all).</param>
|
||||
/// <param name="ticksOf">A row's timestamp, in ticks.</param>
|
||||
/// <param name="tieOrder">Order within one timestamp (null = the backend's forward order).</param>
|
||||
/// <param name="hi">The later (inclusive) bound.</param>
|
||||
/// <param name="lo">The earlier (exclusive) bound.</param>
|
||||
/// <param name="want">Cap; <c>0</c> = every row in the window (one forward read).</param>
|
||||
/// <param name="tieBudget">The tie budget M (values below 1 count as 1).</param>
|
||||
/// <param name="resolutionTicks">The backend's timestamp resolution (>= 1).</param>
|
||||
/// <param name="nowUtc">Now, for the look-ahead clamp.</param>
|
||||
/// <param name="hint">What the previous page of the same read carried (default = none).</param>
|
||||
/// <param name="tooManyAtOneTimestamp">Builds the exception for a timestamp holding more than M rows.</param>
|
||||
/// <param name="ct">Cancellation.</param>
|
||||
/// <returns>At most <paramref name="want"/> rows, newest first, and the hint for the next page.</returns>
|
||||
public static async Task<(List<T> Rows, ReverseWalkHint Next)> WalkAsync<T>(
|
||||
Func<DateTime, DateTime, int, CancellationToken, Task<ForwardSegmentRead<T>>> read,
|
||||
Func<T, long> ticksOf,
|
||||
IComparer<T>? tieOrder,
|
||||
DateTime hi, DateTime lo, uint want, int tieBudget, long resolutionTicks, DateTime nowUtc,
|
||||
ReverseWalkHint hint, Func<DateTime, Exception> tooManyAtOneTimestamp, CancellationToken ct)
|
||||
{
|
||||
var resolution = Math.Max(1L, resolutionTicks);
|
||||
var granularity = Math.Max(Alignment, resolution);
|
||||
var budget = Math.Max(1, tieBudget);
|
||||
var result = new List<T>();
|
||||
|
||||
var horizon = nowUtc.Ticks > DateTime.MaxValue.Ticks - LookAhead.Ticks ? DateTime.MaxValue.Ticks : nowUtc.Ticks + LookAhead.Ticks;
|
||||
var hiTicks = Math.Min(hi.Ticks, horizon);
|
||||
var stopTicks = Math.Max(lo.Ticks, Floor.Ticks);
|
||||
if (hiTicks <= stopTicks) return (result, hint);
|
||||
|
||||
if (want == 0)
|
||||
{
|
||||
// One forward read of the whole window, reversed — the uncapped forward read's own cost and bound.
|
||||
var all = (await read(Utc(AlignDown(stopTicks, granularity)), Utc(AlignUp(hiTicks + 1, granularity)), 0, ct)
|
||||
.ConfigureAwait(false)).Rows;
|
||||
AppendNewestFirst(all, ticksOf, tieOrder, stopTicks, hiTicks, result, int.MaxValue);
|
||||
return (result, hint);
|
||||
}
|
||||
|
||||
var maxWidth = DateTime.MaxValue.Ticks;
|
||||
var discovering = hint.WidthTicks <= 0;
|
||||
var width = discovering ? Math.Max(resolution, InitialSegment.Ticks) : Math.Clamp(hint.WidthTicks, resolution, maxWidth);
|
||||
var pageBudget = PageBudget(want, budget);
|
||||
var discoveryBudget = Math.Min(budget, Math.Max(pageBudget, DiscoveryRows));
|
||||
|
||||
// `known` > 0: a truncated read saw rows up to this timestamp (below segHi), and nothing is known of (known, segHi].
|
||||
// Segments stay above it until that stretch is read; then `knownWidth` (the density the truncated read showed)
|
||||
// sizes the segment that ends on it. Searching the gap: first a top-anchored segment at that density (data running
|
||||
// right up to segHi); if it is empty, the data stops somewhere above `known`, so GALLOP up from it — read
|
||||
// (known + lowerBy, segHi] with lowerBy growing ×4, `known` following what each truncated read saw — until a read
|
||||
// comes back empty (the top is now bracketed below its segLo), then BISECT that bracket. Any complete read holding
|
||||
// rows ends the search. A burst that ended long before segHi is found in a few reads instead of bisecting the whole
|
||||
// gap down to the millisecond, and a long dense stretch is crossed geometrically, not a budget at a time (#691.4
|
||||
// review). `noGallop` falls back to plain top-anchored halving if a galloping read is crowded by rows before it.
|
||||
var known = 0L;
|
||||
var knownWidth = 0L;
|
||||
var lowerBy = 0L;
|
||||
var bisecting = false;
|
||||
var noGallop = false;
|
||||
var segHi = hiTicks;
|
||||
|
||||
// #691: the remainder probe (once per page, after two empty segments) and the empty stretches.
|
||||
var emptyRun = 0;
|
||||
var probed = false;
|
||||
var carried = hint.HasEmptyStretch ? (hint.EmptyFromTicks, hint.EmptyToTicks) : (0L, 0L);
|
||||
(long From, long To) proven = (0L, 0L);
|
||||
|
||||
while (segHi > stopTicks && result.Count < want)
|
||||
{
|
||||
if (carried.Item2 > carried.Item1 && segHi > carried.Item1 && segHi < carried.Item2)
|
||||
{
|
||||
segHi = carried.Item1; // (from, segHi] lies inside a stretch a previous page proved empty
|
||||
continue;
|
||||
}
|
||||
|
||||
var probe = !probed && known == 0 && emptyRun >= 2 && segHi - stopTicks > width;
|
||||
var w = probe ? segHi - stopTicks : width;
|
||||
var lowered = false;
|
||||
var bisected = false;
|
||||
if (known > 0)
|
||||
{
|
||||
var gap = segHi - known;
|
||||
lowered = lowerBy > 0 && lowerBy < gap;
|
||||
bisected = !lowered && (bisecting || lowerBy > 0) && gap > 2 * Math.Max(granularity, knownWidth);
|
||||
// Never narrower than the request granularity: requests are rounded outward to it, so a narrower segment
|
||||
// re-sends the same request — and a width left at one tick (a tie cluster's) stepped segHi down a tick per
|
||||
// read, ~10,000 identical reads (#691.4 re-review). Keep the Min: a truncated read crowded by rows before
|
||||
// the segment must still shrink the segment.
|
||||
w = lowered ? gap - lowerBy
|
||||
: bisected ? gap / 2
|
||||
: Math.Min(Math.Max(w, granularity), gap <= 2 * granularity ? gap : gap / 2);
|
||||
}
|
||||
var segLo = segHi - stopTicks <= w ? stopTicks : segHi - w;
|
||||
var start = Utc(AlignDown(segLo, granularity));
|
||||
var end = Utc(AlignUp(segHi + 1, granularity));
|
||||
var segBudget = discovering && !probe ? discoveryBudget : pageBudget;
|
||||
var cap = segBudget + 2;
|
||||
IReadOnlyList<T> rows;
|
||||
var narrowed = false;
|
||||
|
||||
while (true)
|
||||
{
|
||||
ct.ThrowIfCancellationRequested();
|
||||
var segment = await read(start, end, cap, ct).ConfigureAwait(false);
|
||||
rows = segment.Rows;
|
||||
if (!segment.Truncated || rows.Count == 0 || ticksOf(rows[^1]) > segHi) break; // complete for (segLo, segHi]
|
||||
|
||||
if (probe)
|
||||
{
|
||||
// Deep history below: the probe saw only its oldest rows. Carry on walking from the same segHi.
|
||||
probed = true;
|
||||
narrowed = true;
|
||||
break;
|
||||
}
|
||||
|
||||
var leading = 0;
|
||||
var inner = 0;
|
||||
var firstInner = 0L;
|
||||
var lastInner = 0L;
|
||||
foreach (var r in rows)
|
||||
{
|
||||
var ts = ticksOf(r);
|
||||
if (ts <= segLo) leading++;
|
||||
else if (ts <= segHi)
|
||||
{
|
||||
if (inner++ == 0) firstInner = ts;
|
||||
lastInner = ts;
|
||||
}
|
||||
}
|
||||
|
||||
// Crowded out by rows before the segment, with the segment itself possibly within budget: re-read once
|
||||
// with room for them.
|
||||
var roomier = leading + segBudget + 2;
|
||||
if (leading > 0 && inner <= segBudget && roomier > cap && roomier <= 2 * segBudget + 4)
|
||||
{
|
||||
cap = roomier;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (segHi - segLo <= granularity)
|
||||
{
|
||||
// A segment no wider than the request granularity (1 ms — narrower requests are the same request) still
|
||||
// over its budget: a tie cluster, or a burst inside one millisecond. Escalate to the tie budget M before
|
||||
// refusing — up to M rows within one millisecond are served (#686); more are refused, per millisecond
|
||||
// (#691.4 re-review — exact for the millisecond-resolution Gateway).
|
||||
if (segBudget < budget)
|
||||
{
|
||||
segBudget = budget;
|
||||
cap = segBudget + 2;
|
||||
continue;
|
||||
}
|
||||
|
||||
throw tooManyAtOneTimestamp(Utc(segHi));
|
||||
}
|
||||
|
||||
// Narrow — the segment shrinks every time, so this terminates.
|
||||
if (inner > 0 && lastInner == segHi)
|
||||
{
|
||||
width = resolution; // a tie cluster at the top of the segment: read that one timestamp
|
||||
lowerBy = 0;
|
||||
bisecting = false;
|
||||
}
|
||||
else if (inner > 0 && (lowered || bisected))
|
||||
{
|
||||
// Data reaches past this segment's bottom: move `known` up to what this read saw; a gallop goes further.
|
||||
known = lastInner;
|
||||
if (lowered) lowerBy = lowerBy > maxWidth / 4 ? maxWidth : lowerBy * 4;
|
||||
}
|
||||
else if (inner > 0)
|
||||
{
|
||||
// Rows exist at lastInner; what lies above it is unknown. Aim at about half the page budget at the
|
||||
// density the truncated rows showed (their span, not the segment's — it may open on an empty stretch).
|
||||
var aimed = inner > 1 && lastInner > firstInner
|
||||
? (long)Math.Min((double)(lastInner - firstInner) * (pageBudget / 2.0) / (inner - 1), maxWidth)
|
||||
: resolution;
|
||||
known = lastInner;
|
||||
knownWidth = Math.Max(resolution, aimed);
|
||||
width = Math.Max(resolution, Math.Min(aimed, segHi - segLo - 1));
|
||||
lowerBy = 0;
|
||||
bisecting = false;
|
||||
}
|
||||
else
|
||||
{
|
||||
// Only rows before the segment: shrink it — top-anchored from here on (a gallop would re-send the same
|
||||
// crowded read).
|
||||
width = Math.Max(resolution, Math.Min(width, segHi - segLo) / 4);
|
||||
if (lowered || bisected) noGallop = true;
|
||||
lowerBy = 0;
|
||||
bisecting = false;
|
||||
}
|
||||
|
||||
discovering = false;
|
||||
narrowed = true;
|
||||
break;
|
||||
}
|
||||
|
||||
if (narrowed) continue; // re-read the same segHi, narrower
|
||||
|
||||
var count = AppendNewestFirst(rows, ticksOf, tieOrder, segLo, segHi, result, (int)Math.Min(int.MaxValue, want),
|
||||
out var oldest);
|
||||
// This complete read proves (segLo, oldest) empty — or the whole segment when it held nothing.
|
||||
proven = count > 0 ? (segLo, oldest) : (segLo, segHi + 1);
|
||||
if (probe)
|
||||
{
|
||||
probed = true;
|
||||
segHi = segLo; // the whole remainder is read
|
||||
continue;
|
||||
}
|
||||
|
||||
emptyRun = count == 0 ? emptyRun + 1 : 0;
|
||||
var grown = width > maxWidth / 4 ? maxWidth : width * 4;
|
||||
if (count == 0)
|
||||
{
|
||||
discovering = true; // ×4 growth may overshoot into dense data
|
||||
if (known > 0)
|
||||
{
|
||||
// An empty top-anchored segment: gallop up from `known`. An empty galloping read: the top is bracketed
|
||||
// in (known, segLo] — bisect it (an empty bisecting read keeps bisecting).
|
||||
if (lowered || bisected)
|
||||
{
|
||||
lowerBy = 0;
|
||||
bisecting = true;
|
||||
}
|
||||
else if (noGallop)
|
||||
{
|
||||
width = maxWidth; // halve the gap, top-anchored
|
||||
}
|
||||
else
|
||||
{
|
||||
lowerBy = Math.Max(knownWidth, granularity);
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
width = grown;
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
// Re-aim at about half the page budget from the density just seen.
|
||||
width = (long)Math.Clamp((double)(segHi - segLo) * Math.Max(1.0, pageBudget / 2.0) / count, resolution, grown);
|
||||
discovering = false;
|
||||
known = 0;
|
||||
lowerBy = 0;
|
||||
bisecting = false;
|
||||
noGallop = false;
|
||||
}
|
||||
|
||||
segHi = segLo;
|
||||
if (known > 0 && segHi <= known)
|
||||
{
|
||||
// Everything above the newest row the truncated read saw is read: the next segment ends on it, at its density.
|
||||
known = 0;
|
||||
lowerBy = 0;
|
||||
bisecting = false;
|
||||
noGallop = false;
|
||||
width = knownWidth;
|
||||
discovering = false;
|
||||
}
|
||||
}
|
||||
|
||||
// Carry the stretch the last complete read proved empty, joined to the one carried in when they touch; when they
|
||||
// do not, keep the one reaching the floor (it saves a whole walk), else the newer.
|
||||
var next = carried;
|
||||
if (proven.To > proven.From)
|
||||
{
|
||||
if (carried.Item2 > carried.Item1 && proven.From < carried.Item2 && carried.Item1 < proven.To)
|
||||
next = (Math.Min(proven.From, carried.Item1), Math.Max(proven.To, carried.Item2));
|
||||
else if (!(carried.Item2 > carried.Item1 && carried.Item1 <= stopTicks) || proven.From <= stopTicks)
|
||||
next = proven;
|
||||
}
|
||||
|
||||
return (result, new ReverseWalkHint(width, next.Item1, next.Item2));
|
||||
}
|
||||
|
||||
/// <summary>Appends the rows in <c>(segLo, segHi]</c> newest-first (until <paramref name="limit"/>) and counts them.</summary>
|
||||
private static int AppendNewestFirst<T>(
|
||||
IReadOnlyList<T> rows, Func<T, long> ticksOf, IComparer<T>? tieOrder, long segLo, long segHi, List<T> result, int limit) =>
|
||||
AppendNewestFirst(rows, ticksOf, tieOrder, segLo, segHi, result, limit, out _);
|
||||
|
||||
/// <summary>As above, also returning the oldest in-range timestamp (meaningless when none).</summary>
|
||||
private static int AppendNewestFirst<T>(
|
||||
IReadOnlyList<T> rows, Func<T, long> ticksOf, IComparer<T>? tieOrder, long segLo, long segHi, List<T> result, int limit,
|
||||
out long oldest)
|
||||
{
|
||||
oldest = long.MaxValue;
|
||||
IReadOnlyList<T> ordered = rows;
|
||||
if (tieOrder is not null)
|
||||
{
|
||||
var copy = rows.ToList();
|
||||
copy.Sort((a, b) =>
|
||||
{
|
||||
var c = ticksOf(a).CompareTo(ticksOf(b));
|
||||
return c != 0 ? c : tieOrder.Compare(a, b);
|
||||
});
|
||||
ordered = copy;
|
||||
}
|
||||
|
||||
var count = 0;
|
||||
for (var i = ordered.Count - 1; i >= 0; i--)
|
||||
{
|
||||
var ts = ticksOf(ordered[i]);
|
||||
if (ts <= segLo || ts > segHi) continue;
|
||||
count++;
|
||||
if (ts < oldest) oldest = ts;
|
||||
if (result.Count < limit) result.Add(ordered[i]);
|
||||
}
|
||||
|
||||
return count;
|
||||
}
|
||||
|
||||
private static DateTime Utc(long ticks) => new(ticks, DateTimeKind.Utc);
|
||||
|
||||
private static long AlignDown(long ticks, long step) => ticks - ticks % step;
|
||||
|
||||
private static long AlignUp(long ticks, long step)
|
||||
{
|
||||
var down = AlignDown(ticks, step);
|
||||
if (down == ticks) return ticks;
|
||||
return down > DateTime.MaxValue.Ticks - step ? DateTime.MaxValue.Ticks : down + step;
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Orders events within one timestamp by <see cref="HistoricalEvent.EventId"/> (ordinal) — the tie order a walked-back
|
||||
/// events read (#691.4) and its oversized-cluster over-fetch share, because the Gateway orders by event time only.
|
||||
/// </summary>
|
||||
internal sealed class EventIdTieOrder : IComparer<HistoricalEvent>
|
||||
{
|
||||
/// <summary>The shared instance.</summary>
|
||||
public static readonly EventIdTieOrder Instance = new();
|
||||
|
||||
/// <inheritdoc />
|
||||
public int Compare(HistoricalEvent? x, HistoricalEvent? y) => string.CompareOrdinal(x?.EventId, y?.EventId);
|
||||
}
|
||||
@@ -82,13 +82,21 @@ public sealed class ServerHistorianOptions
|
||||
/// <c>MaxTieClusterOverfetch + 1</c> ties at that single timestamp (a <c>start == end</c> read) and
|
||||
/// pages through them; this bounds how large a single-timestamp burst the server will buffer in
|
||||
/// memory before it gives up and surfaces <c>BadTooManyOperations</c> for that node (#686; was
|
||||
/// <c>BadHistoryOperationUnsupported</c>). It is also the per-read row budget of a reverse raw read walked
|
||||
/// back from forward reads on a backend that cannot read newest-first (the Gateway): one timestamp holding
|
||||
/// <c>BadHistoryOperationUnsupported</c>). It is also the tie-cluster CEILING of a reverse read walked
|
||||
/// back from forward reads on a backend that cannot read newest-first (the Gateway; segments are sized from the page, #689): one timestamp holding
|
||||
/// more samples than this is refused <c>BadTooManyOperations</c>. The default (65536) comfortably covers
|
||||
/// any realistic same-millisecond burst; a value below 1 is treated as 1.
|
||||
/// </summary>
|
||||
public int MaxTieClusterOverfetch { get; init; } = 65536;
|
||||
|
||||
/// <summary>
|
||||
/// Gitea #691: whether HistoryRead-Raw honours <c>ReadRawModifiedDetails.ReturnBounds</c> (OPC UA Part 11 bounding
|
||||
/// values). Default <c>true</c>. <c>false</c> is the kill switch back to the pre-#691 answer: the flag is ignored
|
||||
/// (stored samples only) and a modified read carrying it is not refused <c>Bad_InvalidArgument</c>. Note the .NET
|
||||
/// SDK's <c>new ReadRawModifiedDetails()</c> defaults <c>ReturnBounds</c> (and <c>IsReadModified</c>) to <c>true</c>.
|
||||
/// </summary>
|
||||
public bool HonorReturnBounds { get; init; } = true;
|
||||
|
||||
/// <summary>
|
||||
/// Maximum number of historized nodes served CONCURRENTLY within one HistoryRead batch (arch-review
|
||||
/// 03/S3). The SDK's <c>HistoryRead*</c> override surface is synchronous, so each arm block-bridges
|
||||
|
||||
+7
-5
@@ -68,17 +68,19 @@ public sealed class NullHistorianDataSourceTests
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #675: the direction-aware overload's default — forward is the five-argument read, reverse is refused
|
||||
/// (the server answers <c>BadHistoryOperationUnsupported</c>).
|
||||
/// #691.4 review: like raw, the null source reads events in either direction (empty) and says so — the server sends
|
||||
/// it the reverse read directly instead of walking back through empty forward reads (it refused reverse under #675).
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Direction_aware_ReadEventsAsync_forwards_forward_and_refuses_reverse()
|
||||
public async Task Direction_aware_ReadEventsAsync_is_empty_in_both_directions()
|
||||
{
|
||||
IHistorianDataSource source = Source;
|
||||
source.SupportsReverseEvents.ShouldBeTrue();
|
||||
(await source.ReadEventsAsync(null, DateTime.UtcNow.AddHours(-1), DateTime.UtcNow, 0, reverse: false, Ct))
|
||||
.Events.ShouldBeEmpty();
|
||||
await Should.ThrowAsync<NotSupportedException>(() =>
|
||||
source.ReadEventsAsync(null, DateTime.UtcNow, DateTime.UtcNow.AddHours(-1), 0, reverse: true, Ct));
|
||||
var reverse = await source.ReadEventsAsync(null, DateTime.UtcNow, DateTime.UtcNow.AddHours(-1), 0, reverse: true, Ct);
|
||||
reverse.Events.ShouldBeEmpty();
|
||||
reverse.ContinuationPoint.ShouldBeNull();
|
||||
}
|
||||
|
||||
/// <summary>GetHealthSnapshot reports a fully-disabled (zeroed, disconnected, no-node) shape.</summary>
|
||||
|
||||
@@ -0,0 +1,100 @@
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
|
||||
|
||||
/// <summary>
|
||||
/// Gitea #691: the server composes Part 11 bounding values (HistoryRead-Raw <c>ReturnBounds</c>) from exactly these raw
|
||||
/// reads on a backend that reads newest-first natively (<c>HistoryBounds</c> in the OPC UA server) — so every backend must
|
||||
/// answer them precisely: "at or before T" = a reverse read of <c>(0, T]</c> capped at one, "at or after T" = a forward read
|
||||
/// of <c>[T, MaxValue)</c> capped at one, "strictly after T" = the same from <c>T + resolution</c>. A bound is a stored
|
||||
/// sample of any quality, so a Bad row or an outage marker comes back as the bound.
|
||||
/// </summary>
|
||||
public abstract class BoundingValueConformance : ConformanceBase
|
||||
{
|
||||
private static readonly CancellationToken Ct = CancellationToken.None;
|
||||
private static readonly DateTime Floor = new(0, DateTimeKind.Utc);
|
||||
private static readonly DateTime Ceiling = new(DateTime.MaxValue.Ticks, DateTimeKind.Utc);
|
||||
|
||||
private async Task<DataValueSnapshot?> AtOrBeforeAsync(IHistorianDataSource src, string tag, DateTime t) =>
|
||||
(await src.ReadRawAsync(tag, t, Floor, 1, Ct)).Samples.SingleOrDefault();
|
||||
|
||||
private async Task<DataValueSnapshot?> AtOrAfterAsync(IHistorianDataSource src, string tag, DateTime t) =>
|
||||
(await src.ReadRawAsync(tag, t, Ceiling, 1, Ct)).Samples.SingleOrDefault();
|
||||
|
||||
[Fact]
|
||||
public async Task The_sample_at_or_before_a_time_is_the_latest_one_on_or_before_it()
|
||||
{
|
||||
await ConformanceData.SeedAAsync(B);
|
||||
var src = B.CreateDataSource();
|
||||
|
||||
(await AtOrBeforeAsync(src, "A", At(10)))!.Value.ShouldBe(20d); // exactly on it
|
||||
(await AtOrBeforeAsync(src, "A", At(15)))!.Value.ShouldBe(20d); // between samples
|
||||
(await AtOrBeforeAsync(src, "A", At(1000)))!.Value.ShouldBe(50d); // past the last
|
||||
(await AtOrBeforeAsync(src, "A", At(-1))).ShouldBeNull(); // before the first: no bound
|
||||
(await AtOrBeforeAsync(src, "A", At(15)))!.SourceTimestampUtc!.Value.Ticks.ShouldBe(B.Round(At(10)).Ticks);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task The_sample_at_or_after_a_time_is_the_earliest_one_on_or_after_it()
|
||||
{
|
||||
await ConformanceData.SeedAAsync(B);
|
||||
var src = B.CreateDataSource();
|
||||
|
||||
(await AtOrAfterAsync(src, "A", At(10)))!.Value.ShouldBe(20d); // exactly on it
|
||||
(await AtOrAfterAsync(src, "A", At(15)))!.Value.ShouldBe(30d); // between samples
|
||||
(await AtOrAfterAsync(src, "A", At(-100)))!.Value.ShouldBe(10d); // before the first
|
||||
(await AtOrAfterAsync(src, "A", At(41))).ShouldBeNull(); // past the last: no bound
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task The_sample_strictly_after_a_time_starts_one_storage_resolution_later()
|
||||
{
|
||||
await ConformanceData.SeedAAsync(B);
|
||||
var src = B.CreateDataSource();
|
||||
|
||||
(await AtOrAfterAsync(src, "A", At(10).AddTicks(B.ResolutionTicks)))!.Value.ShouldBe(30d);
|
||||
(await AtOrAfterAsync(src, "A", At(40).AddTicks(B.ResolutionTicks))).ShouldBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_time_between_storage_ticks_finds_the_sample_the_backend_stores_at_its_truncation()
|
||||
{
|
||||
// The server compares "on the bound" at the storage resolution, so the lookups must agree with the window reads:
|
||||
// a time a fraction of a resolution step after a stored sample is that sample's instant on a coarse backend.
|
||||
await ConformanceData.SeedAAsync(B);
|
||||
var src = B.CreateDataSource();
|
||||
var t = At(10).AddTicks(B.ResolutionTicks - 1); // == At(10) on a 1-tick backend
|
||||
|
||||
(await AtOrBeforeAsync(src, "A", t))!.Value.ShouldBe(20d);
|
||||
var after = await AtOrAfterAsync(src, "A", t);
|
||||
after!.SourceTimestampUtc!.Value.Ticks.ShouldBeGreaterThanOrEqualTo(B.Round(t).Ticks);
|
||||
after.Value.ShouldBe(B.Round(t) == B.Round(At(10)) ? 20d : 30d);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_bad_sample_or_an_outage_marker_is_a_bound_as_stored()
|
||||
{
|
||||
await B.SeedAsync("M",
|
||||
(At(0), HistorianValue.FromDouble(10), Good),
|
||||
(At(5), HistorianValue.Empty, BadNoCommunication),
|
||||
(At(7), HistorianValue.FromDouble(99), Bad));
|
||||
var src = B.CreateDataSource();
|
||||
|
||||
var before = await AtOrBeforeAsync(src, "M", At(6));
|
||||
before!.Value.ShouldBeNull();
|
||||
before.StatusCode.ShouldBe(0x80310000u);
|
||||
var after = await AtOrAfterAsync(src, "M", At(6));
|
||||
after!.Value.ShouldBe(99d);
|
||||
after.StatusCode.ShouldBe(0x80000000u);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task An_unknown_tag_has_no_bounds()
|
||||
{
|
||||
var src = B.CreateDataSource();
|
||||
(await AtOrBeforeAsync(src, "nope", At(10))).ShouldBeNull();
|
||||
(await AtOrAfterAsync(src, "nope", At(10))).ShouldBeNull();
|
||||
}
|
||||
}
|
||||
+46
-4
@@ -54,6 +54,9 @@ public sealed class FakeHistorianGatewayClient : IHistorianGatewayClient
|
||||
public int LastReadEventsMaxEvents;
|
||||
public int ReadEventsCallCount;
|
||||
|
||||
/// <summary>When set, ReadEvents serves <c>EventsSource(start, end)</c> instead of <see cref="Events"/>.</summary>
|
||||
public Func<DateTime, DateTime, IEnumerable<HistorianEvent>>? EventsSource;
|
||||
|
||||
/// <summary>Thrown (deferred to first enumeration) by every read method when set.</summary>
|
||||
public Exception? ThrowOnRead;
|
||||
|
||||
@@ -100,12 +103,35 @@ public sealed class FakeHistorianGatewayClient : IHistorianGatewayClient
|
||||
LastReadRawStartUtc = startUtc;
|
||||
LastReadRawEndUtc = endUtc;
|
||||
LastReadRawMaxValues = maxValues;
|
||||
ReadRawCallCount++;
|
||||
RawCalls.Add((startUtc, endUtc, maxValues));
|
||||
lock (RawCalls)
|
||||
{
|
||||
ReadRawCallCount++;
|
||||
RawCalls.Add((startUtc, endUtc, maxValues));
|
||||
}
|
||||
if (MaxValuesCeiling is { } ceiling && maxValues > ceiling)
|
||||
{
|
||||
// The real gateway's refusal (HistorianReadService.ReadRaw), raised on the first MoveNext before any row, as
|
||||
// the client package surfaces it: a HistorianGatewayException wrapping the RpcException.
|
||||
var rpc = new Grpc.Core.RpcException(new Grpc.Core.Status(Grpc.Core.StatusCode.InvalidArgument,
|
||||
CeilingRejectionDetail ?? $"max_values exceeds the server ceiling of {ceiling}; read large ranges by time-slicing " +
|
||||
"(split the range and read the slices sequentially), or enable " +
|
||||
"Historian:ReadLimits:AllowStreamingBeyondCeiling to page through server-side."));
|
||||
return ToAsyncStream(Array.Empty<HistorianSample>(),
|
||||
new ZB.MOM.WW.HistorianGateway.Client.HistorianGatewayException(rpc.Status.Detail, rpc), ct);
|
||||
}
|
||||
var items = RawSource is null ? RawSamples : RawSource(startUtc, endUtc, maxValues).ToList();
|
||||
return CountingStream(ToAsyncStream(items, ThrowOnRead, ct));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A strict gateway (<c>Historian:ReadLimits:AllowStreamingBeyondCeiling=false</c>) with this
|
||||
/// <c>MaxValuesCeiling</c>: a ReadRaw asking for more is refused <c>InvalidArgument</c> (#691.3).
|
||||
/// </summary>
|
||||
public int? MaxValuesCeiling;
|
||||
|
||||
/// <summary>Overrides the refusal's detail text (an older/other gateway wording).</summary>
|
||||
public string? CeilingRejectionDetail;
|
||||
|
||||
public IAsyncEnumerable<HistorianAggregateSample> ReadAggregateAsync(
|
||||
string tag,
|
||||
DateTime startUtc,
|
||||
@@ -120,9 +146,25 @@ public sealed class FakeHistorianGatewayClient : IHistorianGatewayClient
|
||||
LastAggregateMode = mode;
|
||||
LastAggregateInterval = interval;
|
||||
ReadAggregateCallCount++;
|
||||
return ToAsyncStream(AggregateSamples, ThrowOnRead, ct);
|
||||
AggregateCalls.Add((startUtc, endUtc, mode, interval));
|
||||
if (AggregateSource is null) return ToAsyncStream(AggregateSamples, ThrowOnRead, ct);
|
||||
if (startUtc > endUtc || interval <= TimeSpan.Zero)
|
||||
{
|
||||
// The real gateway's request validation (ValidateRange; interval must be > 0), raised before any row.
|
||||
var rpc = new Grpc.Core.RpcException(new Grpc.Core.Status(Grpc.Core.StatusCode.InvalidArgument,
|
||||
startUtc > endUtc ? "start_time must be less than or equal to end_time." : "interval must be greater than zero."));
|
||||
return ToAsyncStream(Array.Empty<HistorianAggregateSample>(),
|
||||
new ZB.MOM.WW.HistorianGateway.Client.HistorianGatewayException(rpc.Status.Detail, rpc), ct);
|
||||
}
|
||||
return ToAsyncStream(AggregateSource(startUtc, endUtc, mode, interval).ToList(), ThrowOnRead, ct);
|
||||
}
|
||||
|
||||
/// <summary>When set, ReadAggregate serves <c>AggregateSource(start, end, mode, interval)</c> (and refuses start > end).</summary>
|
||||
public Func<DateTime, DateTime, RetrievalMode, TimeSpan, IEnumerable<HistorianAggregateSample>>? AggregateSource;
|
||||
|
||||
/// <summary>Every ReadAggregate call's (start, end, mode, interval), in order.</summary>
|
||||
public readonly List<(DateTime Start, DateTime End, RetrievalMode Mode, TimeSpan Interval)> AggregateCalls = new();
|
||||
|
||||
public Task<IReadOnlyList<HistorianSample>> ReadAtTimeAsync(
|
||||
string tag,
|
||||
IReadOnlyList<DateTime> timestampsUtc,
|
||||
@@ -148,7 +190,7 @@ public sealed class FakeHistorianGatewayClient : IHistorianGatewayClient
|
||||
LastReadEventsEndUtc = endUtc;
|
||||
LastReadEventsMaxEvents = maxEvents;
|
||||
ReadEventsCallCount++;
|
||||
return ToAsyncStream(Events, ThrowOnRead, ct);
|
||||
return ToAsyncStream(EventsSource is null ? Events : EventsSource(startUtc, endUtc).ToList(), ThrowOnRead, ct);
|
||||
}
|
||||
|
||||
public Task<WriteAck> WriteLiveValuesAsync(
|
||||
|
||||
+174
@@ -0,0 +1,174 @@
|
||||
using Google.Protobuf.WellKnownTypes;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.HistorianGateway.Contracts.Grpc;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Gitea #691: Part 11 bounding values on the Gateway, composed by the server's <see cref="HistoryBounds"/> over the REAL
|
||||
/// source and an AVEVA-shaped fake (the live gate's wire shape). The start bound of a forward read must be the stored
|
||||
/// sample before StartTime, at ITS OWN time — never AVEVA's synthetic boundary row, which is stamped at the query start
|
||||
/// and carries the value in effect there (<c>Quality = 133</c>). The Gateway reads forward only, so "at or before" is
|
||||
/// served by the walk-back (<see cref="ReverseRawReader"/>), and no reverse window is ever sent.
|
||||
/// </summary>
|
||||
public sealed class GatewayBoundingValueTests
|
||||
{
|
||||
private static CancellationToken Ct => TestContext.Current.CancellationToken;
|
||||
|
||||
private static readonly DateTime T0 = new(2026, 10, 12, 18, 6, 0, DateTimeKind.Utc);
|
||||
private static readonly DateTime Now = T0.AddHours(1);
|
||||
|
||||
/// <summary>The live gate's tag <c>HistGW.LiveTest.Gate681</c> (18:06:40 is a stored NULL).</summary>
|
||||
private static readonly (double Sec, double Value, bool Null)[] Gate681 =
|
||||
[
|
||||
(0, 10, false), (5, 20, false), (20, 30, false), (40, 0, true), (50, 5, false), (60, 1, false), (90, 1, false),
|
||||
(100, 7, false),
|
||||
];
|
||||
|
||||
private static DateTime At(double seconds) => T0.AddTicks((long)(seconds * TimeSpan.TicksPerSecond));
|
||||
|
||||
private static HistorianSample Stored(DateTime t, double v, bool isNull) => new()
|
||||
{
|
||||
Tag = "T",
|
||||
Timestamp = Timestamp.FromDateTime(t),
|
||||
NumericValue = isNull ? 0 : v,
|
||||
OpcQuality = isNull ? 0u : 192u,
|
||||
Quality = isNull ? 1u : 0u,
|
||||
};
|
||||
|
||||
/// <summary>The latest query end any ReadRaw was sent (a far-future end is the #691 review's risk).</summary>
|
||||
private sealed class EndProbe
|
||||
{
|
||||
public DateTime MaxEnd = DateTime.MinValue;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// ReadRaw over <c>[start, end]</c> ascending, capped, led by AVEVA's boundary row at the query start. A query end more
|
||||
/// than two days past <see cref="Now"/> is REFUSED (<c>InvalidArgument</c>), modelling the unverified AVEVA behaviour
|
||||
/// for a year-9999 end (the HistorianGateway only checks <c>start <= end</c>, so it reaches AVEVA unclamped).
|
||||
/// </summary>
|
||||
private static FakeHistorianGatewayClient Aveva(EndProbe? probe = null)
|
||||
{
|
||||
var sorted = Gate681.Select(x => (T: At(x.Sec), V: x.Value, x.Null)).ToList();
|
||||
return new FakeHistorianGatewayClient
|
||||
{
|
||||
RawSource = (start, end, max) =>
|
||||
{
|
||||
if (probe is not null && end > probe.MaxEnd) probe.MaxEnd = end;
|
||||
if (end > Now.AddDays(2))
|
||||
throw new Grpc.Core.RpcException(new Grpc.Core.Status(Grpc.Core.StatusCode.InvalidArgument,
|
||||
"end_time is outside the historian's time range"));
|
||||
var rows = new List<HistorianSample>();
|
||||
if (sorted.All(d => d.T != start))
|
||||
{
|
||||
var prior = sorted.Where(d => d.T < start).Select(d => ((DateTime T, double V, bool Null)?)d).LastOrDefault();
|
||||
HistorianSample boundary;
|
||||
if (prior is { } p)
|
||||
{
|
||||
boundary = Stored(start, p.V, p.Null);
|
||||
if (!p.Null) boundary.Quality = 133;
|
||||
}
|
||||
else
|
||||
{
|
||||
boundary = new HistorianSample { Tag = "T", Timestamp = Timestamp.FromDateTime(start), NumericValue = 0, OpcQuality = 0, Quality = 1 };
|
||||
}
|
||||
|
||||
rows.Add(boundary);
|
||||
}
|
||||
|
||||
rows.AddRange(sorted.Where(d => d.T >= start && d.T <= end).Select(d => Stored(d.T, d.V, d.Null)));
|
||||
return max > 0 ? rows.Take(max) : rows;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
private static GatewayHistorianDataSource Source(FakeHistorianGatewayClient fake) =>
|
||||
new(fake, NullLogger<GatewayHistorianDataSource>.Instance);
|
||||
|
||||
[Fact]
|
||||
public async Task The_start_bound_is_the_stored_prior_sample_at_its_own_time_not_the_synthetic_row()
|
||||
{
|
||||
var bound = await HistoryBounds.StartBoundAsync(Source(Aveva()), "T", At(21), reverse: false, 65536, Now, Ct);
|
||||
|
||||
bound.ShouldNotBeNull();
|
||||
bound.SourceTimestampUtc.ShouldBe(At(20)); // the synthetic row would be stamped At(21)
|
||||
bound.Value.ShouldBe(30d);
|
||||
bound.StatusCode.ShouldBe(0u);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_stored_NULL_prior_sample_is_the_start_bound_as_the_Bad_sample_it_is()
|
||||
{
|
||||
var bound = await HistoryBounds.StartBoundAsync(Source(Aveva()), "T", At(45), reverse: false, 65536, Now, Ct);
|
||||
|
||||
bound!.SourceTimestampUtc.ShouldBe(At(40));
|
||||
bound.StatusCode.ShouldBe(0x80000000u);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Before_the_first_sample_the_start_bound_is_not_found_at_the_start_time()
|
||||
{
|
||||
var bound = await HistoryBounds.StartBoundAsync(Source(Aveva()), "T", At(-30), reverse: false, 65536, Now, Ct);
|
||||
|
||||
bound!.StatusCode.ShouldBe(Opc.Ua.StatusCodes.BadBoundNotFound);
|
||||
bound.SourceTimestampUtc.ShouldBe(At(-30));
|
||||
bound.Value.ShouldBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_sample_exactly_on_the_start_time_is_the_bound_and_already_the_data()
|
||||
{
|
||||
(await HistoryBounds.StartBoundAsync(Source(Aveva()), "T", At(20), reverse: false, 65536, Now, Ct)).ShouldBeNull();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task The_end_bound_is_the_first_stored_sample_at_or_after_the_end_time()
|
||||
{
|
||||
var source = Source(Aveva());
|
||||
var after = await HistoryBounds.EndBoundAsync(source, "T", HistoryBounds.For(At(0), At(41)), At(40), 65536, Now, Ct);
|
||||
after.SourceTimestampUtc.ShouldBe(At(50));
|
||||
after.Value.ShouldBe(5d);
|
||||
|
||||
var missing = await HistoryBounds.EndBoundAsync(source, "T", HistoryBounds.For(At(0), At(101)), At(100), 65536, Now, Ct);
|
||||
missing.StatusCode.ShouldBe(Opc.Ua.StatusCodes.BadBoundNotFound);
|
||||
missing.SourceTimestampUtc.ShouldBe(At(101));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #691 review: the at-or-after read never sends an end beyond now + <see cref="ReverseRawReader.LookAhead"/> (the
|
||||
/// historian stores nothing further ahead), and an end bound wanted beyond that horizon is not found without a read.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task The_at_or_after_read_never_asks_past_now_plus_one_day()
|
||||
{
|
||||
var probe = new EndProbe();
|
||||
var fake = Aveva(probe);
|
||||
var source = Source(fake);
|
||||
|
||||
var found = await HistoryBounds.EndBoundAsync(source, "T", HistoryBounds.For(At(0), At(41)), At(40), 65536, Now, Ct);
|
||||
found.SourceTimestampUtc.ShouldBe(At(50));
|
||||
probe.MaxEnd.ShouldBeLessThanOrEqualTo(Now + ReverseRawReader.LookAhead.Add(TimeSpan.FromMilliseconds(1)));
|
||||
|
||||
var calls = fake.ReadRawCallCount;
|
||||
var beyond = Now.AddDays(3);
|
||||
var missing = await HistoryBounds.EndBoundAsync(source, "T", HistoryBounds.For(At(0), beyond), At(100), 65536, Now, Ct);
|
||||
missing.StatusCode.ShouldBe(Opc.Ua.StatusCodes.BadBoundNotFound);
|
||||
missing.SourceTimestampUtc.ShouldBe(beyond);
|
||||
fake.ReadRawCallCount.ShouldBe(calls, "a bound beyond the horizon is not found without a read");
|
||||
source.GetHealthSnapshot().TotalFailures.ShouldBe(0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_reverse_read_end_bound_walks_back_to_the_stored_sample_at_or_before_the_end_time()
|
||||
{
|
||||
var fake = Aveva();
|
||||
var bound = await HistoryBounds.EndBoundAsync(Source(fake), "T", HistoryBounds.For(At(100), At(55)), At(60), 65536, Now, Ct);
|
||||
|
||||
bound.SourceTimestampUtc.ShouldBe(At(50));
|
||||
bound.Value.ShouldBe(5d);
|
||||
}
|
||||
}
|
||||
+1
-1
@@ -35,7 +35,7 @@ public sealed class GatewayEventBoundsTests
|
||||
{
|
||||
var fake = new FakeHistorianGatewayClient();
|
||||
IHistorianDataSource ds = new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
|
||||
ds.SupportsReverseEvents.ShouldBeFalse("the server must keep end-only reads forward on the default backend");
|
||||
ds.SupportsReverseEvents.ShouldBeFalse("the server must walk reverse reads back from forward reads on the default backend (#691.4)");
|
||||
|
||||
await Should.ThrowAsync<NotSupportedException>(() =>
|
||||
ds.ReadEventsAsync("Pump1", DateTime.UtcNow, DateTime.UtcNow.AddHours(-1), 10, reverse: true, Ct));
|
||||
|
||||
+19
-2
@@ -29,12 +29,14 @@ public sealed class GatewayHistorianDataSourceTests
|
||||
Assert.Equal(100 + RawAggregateReader.PreWindowAllowance, fake.LastReadRawMaxValues);
|
||||
}
|
||||
|
||||
private static readonly DateTime Hour0 = new(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
[Fact]
|
||||
public async Task ReadProcessed_uses_aggregate_mode_mapping()
|
||||
{
|
||||
var fake = new FakeHistorianGatewayClient();
|
||||
var ds = new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
|
||||
await ds.ReadProcessedAsync("T", default, default, TimeSpan.FromSeconds(60), HistoryAggregateType.Total, TestContext.Current.CancellationToken);
|
||||
await ds.ReadProcessedAsync("T", Hour0, Hour0.AddHours(1), TimeSpan.FromSeconds(60), HistoryAggregateType.Total, TestContext.Current.CancellationToken); // an empty window sends nothing
|
||||
Assert.Equal(RetrievalMode.Integral, fake.LastAggregateMode);
|
||||
Assert.Equal(TimeSpan.FromSeconds(60), fake.LastAggregateInterval);
|
||||
}
|
||||
@@ -62,7 +64,7 @@ public sealed class GatewayHistorianDataSourceTests
|
||||
{
|
||||
var fake = new FakeHistorianGatewayClient();
|
||||
var ds = new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
|
||||
await ds.ReadProcessedAsync("T", default, default, TimeSpan.FromSeconds(60), HistoryAggregateType.TimeAverage, TestContext.Current.CancellationToken);
|
||||
await ds.ReadProcessedAsync("T", Hour0, Hour0.AddHours(1), TimeSpan.FromSeconds(60), HistoryAggregateType.TimeAverage, TestContext.Current.CancellationToken); // an empty window sends nothing
|
||||
Assert.Equal(RetrievalMode.TimeWeightedAverage, fake.LastAggregateMode);
|
||||
}
|
||||
|
||||
@@ -111,6 +113,21 @@ public sealed class GatewayHistorianDataSourceTests
|
||||
Assert.Equal(0x80000000u, r.Samples[1].StatusCode); // gap → Bad at requested ts
|
||||
}
|
||||
|
||||
[Fact] // #690: an at-time NULL (histsdk: value 0 + Bad) is null + Bad, like the SQL backends' AtTime of a Bad sample
|
||||
public async Task ReadAtTime_of_a_NULL_row_carries_no_value()
|
||||
{
|
||||
var fake = new FakeHistorianGatewayClient();
|
||||
var t0 = new DateTime(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
fake.AtTimeSamples = new[]
|
||||
{
|
||||
new HistorianSample { NumericValue = 0, OpcQuality = 0, Quality = 1, Timestamp = Timestamp.FromDateTime(t0) },
|
||||
};
|
||||
var ds = new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
|
||||
var r = await ds.ReadAtTimeAsync("T", new[] { t0 }, TestContext.Current.CancellationToken);
|
||||
Assert.Null(r.Samples[0].Value);
|
||||
Assert.Equal(0x80000000u, r.Samples[0].StatusCode);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Empty_window_is_not_a_fault()
|
||||
{
|
||||
|
||||
+216
@@ -0,0 +1,216 @@
|
||||
using Google.Protobuf.WellKnownTypes;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.HistorianGateway.Contracts.Grpc;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// #691.2: a forward or reverse processed read of the AVEVA-native <c>TimeAverage</c> / <c>Total</c> on the SQL
|
||||
/// backends' interval grid. The gateway refuses <c>start > end</c>, so a reverse read is served by FORWARD native
|
||||
/// requests whose bucket grid is anchored at the request's later time (#677: window <c>(end, start]</c>, intervals
|
||||
/// stepping back from <c>start</c>, the clipped interval the earliest, each stamped with its later edge, results newest
|
||||
/// first); a forward read's clipped LATEST interval is read over exactly its span (review). The fake is a native-mode AVEVA
|
||||
/// model: a step-held stored series; for a forward request <c>[s, e)</c> with interval <c>I</c> it answers one row per
|
||||
/// interval of the grid anchored at <c>s</c> (plus AVEVA's extra row for the interval before <c>s</c>, #685), each the
|
||||
/// time-weighted average / integral of exactly that interval — the per-interval behaviour the gateway will have once
|
||||
/// historiangw#27 is fixed.
|
||||
/// </summary>
|
||||
public sealed class GatewayNativeAggregateGridTests
|
||||
{
|
||||
private static readonly CancellationToken Ct = CancellationToken.None;
|
||||
private static readonly DateTime T0 = new(2026, 10, 12, 18, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
private const uint GoodCalc = 0x00000401u;
|
||||
private const uint Partial = 0x4u;
|
||||
private const uint NoDataCalc = 0x809B0401u;
|
||||
|
||||
private static DateTime At(double seconds) => T0.AddTicks((long)(seconds * TimeSpan.TicksPerSecond));
|
||||
|
||||
/// <summary>Step-held series: value changes at these offsets (s); data starts at −100 s.</summary>
|
||||
private static readonly (double T, double V)[] Series = [(-100, 1), (7, 5), (23, 2), (41, 10), (55, 4), (80, 6)];
|
||||
|
||||
/// <summary>∫ v dt (value·s) of the step-held series over [a, b); null when no data covers any of it.</summary>
|
||||
private static double? Integral(DateTime a, DateTime b)
|
||||
{
|
||||
double sum = 0;
|
||||
var covered = false;
|
||||
for (var i = 0; i < Series.Length; i++)
|
||||
{
|
||||
var from = At(Series[i].T);
|
||||
var to = i + 1 < Series.Length ? At(Series[i + 1].T) : DateTime.MaxValue;
|
||||
var lo = from > a ? from : a;
|
||||
var hi = to < b ? to : b;
|
||||
if (hi <= lo) continue;
|
||||
covered = true;
|
||||
sum += Series[i].V * (hi - lo).TotalSeconds;
|
||||
}
|
||||
return covered ? sum : null;
|
||||
}
|
||||
|
||||
private static double? Expected(RetrievalMode mode, DateTime a, DateTime b) =>
|
||||
Integral(a, b) is { } i ? mode == RetrievalMode.Integral ? i : i / (b - a).TotalSeconds : null;
|
||||
|
||||
private static FakeHistorianGatewayClient Aveva() => new()
|
||||
{
|
||||
AggregateSource = (s, e, mode, interval) =>
|
||||
{
|
||||
var rows = new List<HistorianAggregateSample>();
|
||||
for (var lo = s - interval; lo < e; lo += interval) // the pre-window interval first (#685)
|
||||
{
|
||||
var hi = lo + interval;
|
||||
var v = Expected(mode, lo, hi);
|
||||
rows.Add(new HistorianAggregateSample
|
||||
{
|
||||
Tag = "T",
|
||||
StartTime = Timestamp.FromDateTime(lo),
|
||||
EndTime = Timestamp.FromDateTime(hi),
|
||||
Value = v ?? 0,
|
||||
OpcQuality = v is null ? 0u : 192u,
|
||||
RetrievalMode = mode,
|
||||
});
|
||||
}
|
||||
return rows;
|
||||
},
|
||||
};
|
||||
|
||||
private static GatewayHistorianDataSource Source(FakeHistorianGatewayClient fake) =>
|
||||
new(fake, NullLogger<GatewayHistorianDataSource>.Instance);
|
||||
|
||||
[Theory]
|
||||
[InlineData(HistoryAggregateType.TimeAverage, RetrievalMode.TimeWeightedAverage)]
|
||||
[InlineData(HistoryAggregateType.Total, RetrievalMode.Integral)]
|
||||
public async Task A_reverse_read_over_whole_intervals_is_anchored_at_its_later_time(HistoryAggregateType agg, RetrievalMode mode)
|
||||
{
|
||||
var fake = Aveva();
|
||||
var ds = Source(fake);
|
||||
var s = (await ds.ReadProcessedAsync("T", At(60), At(0), TimeSpan.FromSeconds(20), agg, Ct)).Samples;
|
||||
|
||||
// (0, 60] in three intervals, newest first, each stamped with its LATER edge (#677).
|
||||
s.Select(x => x.SourceTimestampUtc).ShouldBe(new DateTime?[] { At(60), At(40), At(20) });
|
||||
s.Select(x => (double)x.Value!).ShouldBe(
|
||||
[Expected(mode, At(40), At(60))!.Value, Expected(mode, At(20), At(40))!.Value, Expected(mode, At(0), At(20))!.Value],
|
||||
tolerance: 1e-9);
|
||||
s.ShouldAllBe(x => x.StatusCode == GoodCalc);
|
||||
fake.AggregateCalls.ShouldBe([(At(0), At(60), mode, TimeSpan.FromSeconds(20))]);
|
||||
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_reverse_read_with_a_partial_interval_clips_the_EARLIEST_one_and_marks_it_Partial()
|
||||
{
|
||||
var fake = Aveva();
|
||||
var s = (await Source(fake).ReadProcessedAsync("T", At(70), At(5), TimeSpan.FromSeconds(20),
|
||||
HistoryAggregateType.TimeAverage, Ct)).Samples;
|
||||
|
||||
// (5, 70]: (50,70], (30,50], (10,30], and the clipped (5,10] — the grid steps back from 70, not forward from 5.
|
||||
s.Select(x => x.SourceTimestampUtc).ShouldBe(new DateTime?[] { At(70), At(50), At(30), At(10) });
|
||||
var m = RetrievalMode.TimeWeightedAverage;
|
||||
s.Select(x => (double)x.Value!).ShouldBe(
|
||||
[Expected(m, At(50), At(70))!.Value, Expected(m, At(30), At(50))!.Value, Expected(m, At(10), At(30))!.Value,
|
||||
Expected(m, At(5), At(10))!.Value],
|
||||
tolerance: 1e-9);
|
||||
s.Select(x => x.StatusCode).ShouldBe([GoodCalc, GoodCalc, GoodCalc, GoodCalc | Partial]);
|
||||
|
||||
// Two forward requests, neither reversed: the whole intervals on the later-anchored grid, and the clipped one alone
|
||||
// (its own width as the interval, so AVEVA computes it over exactly (5, 10]).
|
||||
fake.AggregateCalls.ShouldBe([
|
||||
(At(10), At(70), m, TimeSpan.FromSeconds(20)),
|
||||
(At(5), At(10), m, TimeSpan.FromSeconds(5)),
|
||||
]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_reverse_read_narrower_than_one_interval_is_one_interval_stamped_at_its_later_time()
|
||||
{
|
||||
var fake = Aveva();
|
||||
var s = (await Source(fake).ReadProcessedAsync("T", At(30), At(12), TimeSpan.FromSeconds(60),
|
||||
HistoryAggregateType.Total, Ct)).Samples;
|
||||
s.Single().SourceTimestampUtc.ShouldBe(At(30));
|
||||
((double)s.Single().Value!).ShouldBe(Expected(RetrievalMode.Integral, At(12), At(30))!.Value, 1e-9);
|
||||
s.Single().StatusCode.ShouldBe(GoodCalc); // a whole (clamped) interval, not a truncated one
|
||||
fake.AggregateCalls.ShouldBe([(At(12), At(30), RetrievalMode.Integral, TimeSpan.FromSeconds(18))]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_zero_interval_reverse_read_is_one_interval_over_the_window()
|
||||
{
|
||||
var fake = Aveva();
|
||||
var s = (await Source(fake).ReadProcessedAsync("T", At(30), At(0), TimeSpan.Zero,
|
||||
HistoryAggregateType.TimeAverage, Ct)).Samples;
|
||||
s.Single().SourceTimestampUtc.ShouldBe(At(30));
|
||||
fake.AggregateCalls.Single().Interval.ShouldBe(TimeSpan.FromSeconds(30));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_no_data_interval_is_BadNoData_without_the_Partial_bit()
|
||||
{
|
||||
// Data starts at −100 s: (−110, −90] holds some; (−130, −110] and the clipped earliest (−135, −130] hold none.
|
||||
var s = (await Source(Aveva()).ReadProcessedAsync("T", At(-90), At(-135), TimeSpan.FromSeconds(20),
|
||||
HistoryAggregateType.TimeAverage, Ct)).Samples;
|
||||
s.Select(x => x.SourceTimestampUtc).ShouldBe(new DateTime?[] { At(-90), At(-110), At(-130) });
|
||||
s[0].StatusCode.ShouldBe(GoodCalc);
|
||||
s[1].StatusCode.ShouldBe(NoDataCalc);
|
||||
s[2].StatusCode.ShouldBe(NoDataCalc); // the clipped interval, but Bad ⇒ no Partial bit
|
||||
s[2].Value.ShouldBeNull();
|
||||
}
|
||||
|
||||
// ---- forward reads on the same grid (#691.2 review): the clipped LATEST interval is its own exact-span read ----------
|
||||
|
||||
[Theory]
|
||||
[InlineData(HistoryAggregateType.TimeAverage, RetrievalMode.TimeWeightedAverage)]
|
||||
[InlineData(HistoryAggregateType.Total, RetrievalMode.Integral)]
|
||||
public async Task A_forward_read_with_a_partial_interval_clips_the_LATEST_one_and_marks_it_Partial(HistoryAggregateType agg,
|
||||
RetrievalMode mode)
|
||||
{
|
||||
var fake = Aveva();
|
||||
var s = (await Source(fake).ReadProcessedAsync("T", At(5), At(70), TimeSpan.FromSeconds(20), agg, Ct)).Samples;
|
||||
|
||||
// [5, 70): [5,25), [25,45), [45,65) and the clipped [65,70) — stamped with each EARLIER edge, oldest first, as on
|
||||
// SqlServer/QuestDB (BucketPlan forward; the truncated interval Partial unless Bad).
|
||||
s.Select(x => x.SourceTimestampUtc).ShouldBe(new DateTime?[] { At(5), At(25), At(45), At(65) });
|
||||
s.Select(x => (double)x.Value!).ShouldBe(
|
||||
[Expected(mode, At(5), At(25))!.Value, Expected(mode, At(25), At(45))!.Value, Expected(mode, At(45), At(65))!.Value,
|
||||
Expected(mode, At(65), At(70))!.Value],
|
||||
tolerance: 1e-9);
|
||||
s.Select(x => x.StatusCode).ShouldBe([GoodCalc, GoodCalc, GoodCalc, GoodCalc | Partial]);
|
||||
// AVEVA never averages past EndTime: the whole intervals, then the clipped one over exactly its span.
|
||||
fake.AggregateCalls.ShouldBe([
|
||||
(At(5), At(65), mode, TimeSpan.FromSeconds(20)),
|
||||
(At(65), At(70), mode, TimeSpan.FromSeconds(5)),
|
||||
]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_forward_interval_wider_than_the_window_is_sent_as_the_window()
|
||||
{
|
||||
var fake = Aveva();
|
||||
var s = (await Source(fake).ReadProcessedAsync("T", At(12), At(30), TimeSpan.FromSeconds(60),
|
||||
HistoryAggregateType.Total, Ct)).Samples;
|
||||
s.Single().SourceTimestampUtc.ShouldBe(At(12));
|
||||
s.Single().StatusCode.ShouldBe(GoodCalc);
|
||||
fake.AggregateCalls.ShouldBe([(At(12), At(30), RetrievalMode.Integral, TimeSpan.FromSeconds(18))]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_zero_interval_forward_read_is_one_interval_over_the_window_not_a_zero_interval_request()
|
||||
{
|
||||
var fake = Aveva();
|
||||
var s = (await Source(fake).ReadProcessedAsync("T", At(0), At(30), TimeSpan.Zero,
|
||||
HistoryAggregateType.TimeAverage, Ct)).Samples;
|
||||
s.Single().SourceTimestampUtc.ShouldBe(At(0));
|
||||
fake.AggregateCalls.Single().Interval.ShouldBe(TimeSpan.FromSeconds(30));
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task The_forward_read_is_unchanged()
|
||||
{
|
||||
var fake = Aveva();
|
||||
var s = (await Source(fake).ReadProcessedAsync("T", At(0), At(60), TimeSpan.FromSeconds(20),
|
||||
HistoryAggregateType.TimeAverage, Ct)).Samples;
|
||||
s.Select(x => x.SourceTimestampUtc).ShouldBe(new DateTime?[] { At(0), At(20), At(40) });
|
||||
fake.AggregateCalls.ShouldBe([(At(0), At(60), RetrievalMode.TimeWeightedAverage, TimeSpan.FromSeconds(20))]);
|
||||
}
|
||||
}
|
||||
+3
-1
@@ -106,9 +106,11 @@ public sealed class GatewayNativeAggregateRowsTests
|
||||
public async Task The_window_filter_is_direction_agnostic()
|
||||
{
|
||||
// A reverse request (start > end) covers the same span (end, start]; a row for the interval before the span is out.
|
||||
// #691.2: served by a forward request on the later-anchored grid, newest first, stamped with each later edge.
|
||||
var wire = new[] { Row(T0 - I20, T0, 9, 192), Row(T0, T0 + I20, 1, 192), Row(T0 + I20, T0 + 2 * I20, 2, 192) };
|
||||
var s = await Read(wire, T0 + 2 * I20, T0);
|
||||
s.Select(x => x.Value).ShouldBe(new object?[] { 1d, 2d });
|
||||
s.Select(x => x.Value).ShouldBe(new object?[] { 2d, 1d });
|
||||
s.Select(x => x.SourceTimestampUtc).ShouldBe(new DateTime?[] { T0 + 2 * I20, T0 + I20 });
|
||||
}
|
||||
|
||||
// ---- D2: an Uncertain row keeps its value ------------------------------------------------------------------------
|
||||
|
||||
+9
@@ -109,6 +109,15 @@ public sealed class GatewayRawBoundaryRowTests
|
||||
r.Samples[0].StatusCode.ShouldBe(0x80000000u);
|
||||
}
|
||||
|
||||
[Fact] // #690: the wire's value 0 on a NULL row is not a value — SqlServer/QuestDB return null + Bad for the same sample
|
||||
public async Task A_stored_NULL_sample_carries_no_value_not_the_wires_zero()
|
||||
{
|
||||
var r = await Source(Gate()).ReadRawAsync("T", At(30), At(55), 0, Ct);
|
||||
r.Samples[0].SourceTimestampUtc.ShouldBe(At(40));
|
||||
r.Samples[0].Value.ShouldBeNull();
|
||||
r.Samples[1].Value.ShouldBe(5d); // a Good neighbour keeps its value
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_stored_sample_exactly_at_the_window_start_is_kept()
|
||||
{
|
||||
|
||||
+76
@@ -0,0 +1,76 @@
|
||||
using Google.Protobuf.WellKnownTypes;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.HistorianGateway.Contracts.Grpc;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using ZB.MOM.WW.OtOpcUa.OpcUaServer;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Gitea #691.4 review: events read uncapped (and walked back newest-first) through the REAL
|
||||
/// <see cref="GatewayHistorianDataSource"/> over a gateway-shaped fake — inclusive <c>[start, end]</c>, ordered by event time.
|
||||
/// </summary>
|
||||
public sealed class GatewayReverseEventsTests
|
||||
{
|
||||
private static CancellationToken Ct => TestContext.Current.CancellationToken;
|
||||
|
||||
private static readonly DateTime T0 = new(2026, 10, 12, 18, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
private static FakeHistorianGatewayClient Store(int count)
|
||||
{
|
||||
var stored = Enumerable.Range(0, count)
|
||||
.Select(i => new HistorianEvent { Id = $"e{i:0000}", Type = "Alarm", EventTime = Timestamp.FromDateTime(T0.AddSeconds(i)) })
|
||||
.ToList();
|
||||
return new FakeHistorianGatewayClient
|
||||
{
|
||||
EventsSource = (start, end) => stored.Where(e => e.EventTime.ToDateTime() >= start && e.EventTime.ToDateTime() <= end),
|
||||
};
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// <c>NumValuesPerNode = 0</c> reaches the source as <c>int.MaxValue</c>; the result list used to be pre-sized to it
|
||||
/// ("Array dimensions exceeded") — every uncapped Gateway events read failed and counted as a backend failure.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task An_uncapped_forward_events_read_is_served()
|
||||
{
|
||||
var fake = Store(30);
|
||||
var ds = new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
|
||||
|
||||
var r = await ds.ReadEventsAsync(null, T0, T0.AddMinutes(1), int.MaxValue, Ct);
|
||||
|
||||
r.Events.Count.ShouldBe(30);
|
||||
r.ContinuationPoint.ShouldBeNull();
|
||||
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task An_uncapped_reverse_events_read_is_one_forward_read_newest_first()
|
||||
{
|
||||
var fake = Store(30);
|
||||
var ds = new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
|
||||
|
||||
var page = await ReverseEventsReader.ReadAsync(ds, null, hi: T0.AddSeconds(25), lo: T0.AddSeconds(5), want: 0,
|
||||
maxTieCluster: 65536, nowUtc: T0.AddHours(1), hint: default, Ct);
|
||||
|
||||
page.Events.Select(e => e.EventId).ShouldBe(Enumerable.Range(6, 20).Reverse().Select(i => $"e{i:0000}"));
|
||||
fake.ReadEventsCallCount.ShouldBe(1);
|
||||
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_capped_reverse_events_read_walked_back_over_the_gateway_returns_the_latest_newest_first()
|
||||
{
|
||||
var fake = Store(300);
|
||||
var ds = new GatewayHistorianDataSource(fake, NullLogger<GatewayHistorianDataSource>.Instance);
|
||||
|
||||
var page = await ReverseEventsReader.ReadAsync(ds, null, hi: T0.AddSeconds(200), lo: T0.AddSeconds(50), want: 10,
|
||||
maxTieCluster: 65536, nowUtc: T0.AddHours(1), hint: default, Ct);
|
||||
|
||||
page.Events.Select(e => e.EventId).ShouldBe(Enumerable.Range(191, 10).Reverse().Select(i => $"e{i:0000}"));
|
||||
fake.LastReadEventsStartUtc.ShouldBeLessThanOrEqualTo(fake.LastReadEventsEndUtc);
|
||||
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
|
||||
}
|
||||
}
|
||||
+290
@@ -0,0 +1,290 @@
|
||||
using Google.Protobuf.WellKnownTypes;
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.HistorianGateway.Contracts.Grpc;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// #691.3: a gateway with <c>Historian:ReadLimits:AllowStreamingBeyondCeiling=false</c> refuses any <c>ReadRaw</c> whose
|
||||
/// <c>max_values</c> exceeds its <c>MaxValuesCeiling</c> (before streaming, naming the ceiling). An uncapped raw read
|
||||
/// (<c>NumValuesPerNode = 0</c>, sent as <see cref="int.MaxValue"/>) used to be refused outright — even a 10-row window.
|
||||
/// The read is now re-issued once with the ceiling the refusal named, and a stream that delivered the whole ceiling
|
||||
/// without reaching the window end is never answered from (<c>BadTooManyOperations</c>). The fake is AVEVA-shaped:
|
||||
/// a boundary row at the query start, <c>[start, end]</c> inclusive, capped at <c>max</c>.
|
||||
/// </summary>
|
||||
public sealed class GatewayStrictCeilingTests
|
||||
{
|
||||
private static readonly CancellationToken Ct = CancellationToken.None;
|
||||
private static readonly DateTime T0 = new(2026, 10, 12, 19, 0, 0, DateTimeKind.Utc);
|
||||
private static readonly DateTime Now = T0.AddDays(1);
|
||||
|
||||
private static DateTime At(int seconds) => T0.AddSeconds(seconds);
|
||||
|
||||
private static HistorianSample Row(DateTime t, double v, uint quality = 0) =>
|
||||
new() { Tag = "T", Timestamp = Timestamp.FromDateTime(t), NumericValue = v, OpcQuality = 192, Quality = quality };
|
||||
|
||||
/// <summary>One stored sample per second, value = seconds, at +0 … +(count − 1); strict ceiling when given.</summary>
|
||||
private static FakeHistorianGatewayClient Store(int count, int? ceiling)
|
||||
{
|
||||
var stored = Enumerable.Range(0, count).Select(i => (T: At(i), V: (double)i)).ToList();
|
||||
return new FakeHistorianGatewayClient
|
||||
{
|
||||
MaxValuesCeiling = ceiling,
|
||||
RawSource = (start, end, max) =>
|
||||
{
|
||||
var rows = new List<HistorianSample>();
|
||||
if (stored.All(s => s.T != start))
|
||||
{
|
||||
var prior = stored.LastOrDefault(s => s.T < start);
|
||||
rows.Add(prior.T == default
|
||||
? new HistorianSample { Tag = "T", Timestamp = Timestamp.FromDateTime(start), NumericValue = 0, OpcQuality = 0, Quality = 1 }
|
||||
: Row(start, prior.V, 133));
|
||||
}
|
||||
rows.AddRange(stored.Where(s => s.T >= start && s.T <= end).Select(s => Row(s.T, s.V)));
|
||||
return rows.Take(max);
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
private static GatewayHistorianDataSource Source(FakeHistorianGatewayClient fake, int maxRawRows = 1_000_000) =>
|
||||
new(fake, NullLogger<GatewayHistorianDataSource>.Instance, maxRawRows, new FixedClock(Now));
|
||||
|
||||
private sealed class FixedClock(DateTime now) : TimeProvider
|
||||
{
|
||||
public override DateTimeOffset GetUtcNow() => new(now);
|
||||
}
|
||||
|
||||
// ---- raw, uncapped ---------------------------------------------------------------------------------------------------
|
||||
|
||||
[Fact]
|
||||
public async Task An_uncapped_read_of_a_small_window_on_a_strict_gateway_is_served_with_the_ceiling_as_budget()
|
||||
{
|
||||
var fake = Store(10, ceiling: 1000);
|
||||
var ds = Source(fake);
|
||||
var r = await ds.ReadRawAsync("T", At(0), At(60), 0, Ct);
|
||||
|
||||
r.Samples.Select(s => s.Value).ShouldBe(Enumerable.Range(0, 10).Select(i => (object?)(double)i));
|
||||
fake.RawCalls.Select(c => c.MaxValues).ShouldBe([int.MaxValue, 1000]); // refused, then the ceiling it named
|
||||
fake.RawStreamsCompleted.ShouldBe(1);
|
||||
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task An_uncapped_read_on_a_default_gateway_still_sends_one_unbounded_request()
|
||||
{
|
||||
var fake = Store(10, ceiling: null);
|
||||
await Source(fake).ReadRawAsync("T", At(0), At(60), 0, Ct);
|
||||
fake.RawCalls.Select(c => c.MaxValues).ShouldBe([int.MaxValue]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task An_uncapped_window_holding_more_than_the_ceiling_is_a_request_limit_never_a_truncated_answer()
|
||||
{
|
||||
var fake = Store(100, ceiling: 50);
|
||||
var ds = Source(fake);
|
||||
await Should.ThrowAsync<HistoryReadLimitExceededException>(() => ds.ReadRawAsync("T", At(0), At(200), 0, Ct));
|
||||
fake.RawStreamsCompleted.ShouldBe(1); // drained, never abandoned
|
||||
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_ceiling_full_stream_that_reached_the_window_end_is_complete()
|
||||
{
|
||||
// 50 rows: the boundary row + +0 … +48; the window end is +48 (the gateway's end is inclusive), so the stream
|
||||
// provably ended at the window end — complete although it delivered the whole ceiling.
|
||||
var fake = Store(100, ceiling: 50);
|
||||
var r = await Source(fake).ReadRawAsync("T", At(0), At(48), 0, Ct);
|
||||
r.Samples.Count.ShouldBe(48); // [+0, +48): the end is exclusive
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_refusal_without_a_parseable_ceiling_retries_at_the_gateways_default_ceiling()
|
||||
{
|
||||
var fake = Store(10, ceiling: 1_000_000);
|
||||
fake.CeilingRejectionDetail = "max_values exceeds the server ceiling; read large ranges by time-slicing.";
|
||||
var r = await Source(fake).ReadRawAsync("T", At(0), At(60), 0, Ct);
|
||||
r.Samples.Count.ShouldBe(10);
|
||||
fake.RawCalls.Select(c => c.MaxValues).ShouldBe([int.MaxValue, RawAggregateReader.GatewayDefaultMaxValuesCeiling]);
|
||||
}
|
||||
|
||||
// ---- the learned ceiling (review: a refused attempt counts as an error in the gateway's own metrics) -----------------
|
||||
|
||||
private static int Refusals(FakeHistorianGatewayClient fake) =>
|
||||
fake.RawCalls.Count(c => fake.MaxValuesCeiling is { } n && c.MaxValues > n);
|
||||
|
||||
[Fact]
|
||||
public async Task The_ceiling_is_learned_once_per_data_source_so_later_reads_are_not_refused()
|
||||
{
|
||||
var fake = Store(10, ceiling: 1000);
|
||||
var ds = Source(fake);
|
||||
await ds.ReadRawAsync("T", At(0), At(60), 0, Ct);
|
||||
await ds.ReadRawAsync("T", At(0), At(60), 0, Ct);
|
||||
await ds.ReadRawAsync("T", At(0), At(60), 995, Ct);
|
||||
await ds.ReadProcessedAsync("T", At(0), At(10), TimeSpan.FromSeconds(10), HistoryAggregateType.Count, Ct);
|
||||
|
||||
fake.RawCalls.Select(c => c.MaxValues).Take(4).ShouldBe([int.MaxValue, 1000, 1000, 1000]);
|
||||
fake.RawCalls[4].MaxValues.ShouldBe(1000); // the per-sample read shares it (its own budget 1,000,000)
|
||||
Refusals(fake).ShouldBe(1);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_gateway_that_now_streams_is_re_probed_when_a_learned_ceiling_read_comes_back_full()
|
||||
{
|
||||
var fake = Store(100, ceiling: 50);
|
||||
var ds = Source(fake);
|
||||
await ds.ReadRawAsync("T", At(0), At(5), 0, Ct); // learns 50
|
||||
fake.MaxValuesCeiling = null; // the operator enabled AllowStreamingBeyondCeiling
|
||||
|
||||
var r = await ds.ReadRawAsync("T", At(0), At(200), 0, Ct);
|
||||
r.Samples.Count.ShouldBe(100); // not BadTooManyOperations from a stale ceiling
|
||||
fake.RawCalls.Skip(2).Select(c => c.MaxValues).ShouldBe([50, int.MaxValue]);
|
||||
|
||||
await ds.ReadRawAsync("T", At(0), At(5), 0, Ct);
|
||||
fake.RawCalls[^1].MaxValues.ShouldBe(int.MaxValue); // forgotten
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_raised_but_still_strict_ceiling_is_re_learned_from_the_re_probe()
|
||||
{
|
||||
var fake = Store(100, ceiling: 50);
|
||||
var ds = Source(fake);
|
||||
await ds.ReadRawAsync("T", At(0), At(5), 0, Ct);
|
||||
fake.MaxValuesCeiling = 500;
|
||||
|
||||
(await ds.ReadRawAsync("T", At(0), At(200), 0, Ct)).Samples.Count.ShouldBe(100);
|
||||
fake.RawCalls.Skip(2).Select(c => c.MaxValues).ShouldBe([50, int.MaxValue, 500]);
|
||||
|
||||
await ds.ReadRawAsync("T", At(0), At(5), 0, Ct);
|
||||
fake.RawCalls[^1].MaxValues.ShouldBe(500);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_genuinely_over_ceiling_read_costs_one_re_probe_and_no_repeat_at_the_same_ceiling()
|
||||
{
|
||||
var fake = Store(100, ceiling: 50);
|
||||
var ds = Source(fake);
|
||||
await ds.ReadRawAsync("T", At(0), At(5), 0, Ct);
|
||||
await Should.ThrowAsync<HistoryReadLimitExceededException>(() => ds.ReadRawAsync("T", At(0), At(200), 0, Ct));
|
||||
fake.RawCalls.Skip(2).Select(c => c.MaxValues).ShouldBe([50, int.MaxValue]);
|
||||
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_lowered_ceiling_is_re_learned_from_the_refusal_of_the_learned_one()
|
||||
{
|
||||
var fake = Store(10, ceiling: 1000);
|
||||
var ds = Source(fake);
|
||||
await ds.ReadRawAsync("T", At(0), At(60), 0, Ct);
|
||||
fake.MaxValuesCeiling = 50;
|
||||
|
||||
(await ds.ReadRawAsync("T", At(0), At(60), 0, Ct)).Samples.Count.ShouldBe(10);
|
||||
await ds.ReadRawAsync("T", At(0), At(60), 0, Ct);
|
||||
fake.RawCalls.Skip(2).Select(c => c.MaxValues).ShouldBe([1000, 50, 50]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Concurrent_reads_share_the_learned_ceiling_safely()
|
||||
{
|
||||
var fake = Store(10, ceiling: 1000);
|
||||
var ds = Source(fake);
|
||||
await ds.ReadRawAsync("T", At(0), At(60), 0, Ct);
|
||||
var results = await Task.WhenAll(Enumerable.Range(0, 32).Select(_ => Task.Run(() => ds.ReadRawAsync("T", At(0), At(60), 0, Ct))));
|
||||
results.ShouldAllBe(r => r.Samples.Count == 10);
|
||||
Refusals(fake).ShouldBe(1);
|
||||
}
|
||||
|
||||
// ---- raw, capped -----------------------------------------------------------------------------------------------------
|
||||
|
||||
[Fact]
|
||||
public async Task A_cap_whose_allowance_exceeds_the_ceiling_is_served_with_the_ceiling_as_budget()
|
||||
{
|
||||
// cap 995 + 16 = 1011 > 1000: refused, re-asked with 1000 — a small window is served in full.
|
||||
var fake = Store(10, ceiling: 1000);
|
||||
var r = await Source(fake).ReadRawAsync("T", At(0), At(60), 995, Ct);
|
||||
r.Samples.Count.ShouldBe(10);
|
||||
fake.RawCalls.Select(c => c.MaxValues).ShouldBe([995 + RawAggregateReader.PreWindowAllowance, 1000]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_cap_above_the_ceiling_over_a_dense_window_is_a_request_limit_not_a_short_page()
|
||||
{
|
||||
// A short page would end the client's paging early (silent truncation), so it must fail instead.
|
||||
var fake = Store(100, ceiling: 50);
|
||||
var ds = Source(fake);
|
||||
await Should.ThrowAsync<HistoryReadLimitExceededException>(() => ds.ReadRawAsync("T", At(0), At(200), 80, Ct));
|
||||
fake.RawCalls.Count.ShouldBe(2); // the ceiling re-read is final: no third read at the same ceiling
|
||||
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_cap_that_fits_the_ceiling_is_unaffected()
|
||||
{
|
||||
var fake = Store(100, ceiling: 50);
|
||||
var r = await Source(fake).ReadRawAsync("T", At(0), At(200), 10, Ct);
|
||||
r.Samples.Count.ShouldBe(10);
|
||||
fake.RawCalls.Select(c => c.MaxValues).ShouldBe([10 + RawAggregateReader.PreWindowAllowance]);
|
||||
}
|
||||
|
||||
// ---- per-sample aggregates (#688 clamp vs a ceiling below 1,000,000) ---------------------------------------------------
|
||||
|
||||
[Fact]
|
||||
public async Task A_per_sample_aggregate_on_a_gateway_with_a_lower_ceiling_is_served_when_the_window_fits()
|
||||
{
|
||||
var fake = Store(30, ceiling: 1000); // default cap 1,000,000 ⇒ budget 1,000,000 > 1000
|
||||
var s = (await Source(fake).ReadProcessedAsync("T", At(0), At(30), TimeSpan.FromSeconds(10),
|
||||
HistoryAggregateType.Count, Ct)).Samples;
|
||||
s.Select(x => x.Value).ShouldBe([10, 10, 10]);
|
||||
fake.RawCalls[0].MaxValues.ShouldBe(RawAggregateReader.GatewayDefaultMaxValuesCeiling);
|
||||
fake.RawCalls[1].MaxValues.ShouldBe(1000);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_per_sample_aggregate_whose_window_fills_the_lower_ceiling_is_a_request_limit()
|
||||
{
|
||||
var fake = Store(100, ceiling: 50);
|
||||
var ds = Source(fake);
|
||||
await Should.ThrowAsync<HistoryReadLimitExceededException>(() => ds.ReadProcessedAsync("T", At(0), At(100),
|
||||
TimeSpan.FromSeconds(10), HistoryAggregateType.Count, Ct));
|
||||
ds.GetHealthSnapshot().TotalFailures.ShouldBe(0);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task An_unbounded_per_sample_aggregate_on_a_strict_gateway_is_served_with_the_ceiling()
|
||||
{
|
||||
var fake = Store(30, ceiling: 1000);
|
||||
var s = (await Source(fake, maxRawRows: 0).ReadProcessedAsync("T", At(0), At(30), TimeSpan.FromSeconds(10),
|
||||
HistoryAggregateType.Count, Ct)).Samples;
|
||||
s.Select(x => x.Value).ShouldBe([10, 10, 10]);
|
||||
fake.RawCalls.Take(2).Select(c => c.MaxValues).ShouldBe([int.MaxValue, 1000]);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task An_unbounded_per_sample_aggregate_filling_the_ceiling_is_a_request_limit()
|
||||
{
|
||||
var fake = Store(100, ceiling: 50);
|
||||
await Should.ThrowAsync<HistoryReadLimitExceededException>(() => Source(fake, maxRawRows: 0).ReadProcessedAsync("T",
|
||||
At(0), At(100), TimeSpan.FromSeconds(10), HistoryAggregateType.Count, Ct));
|
||||
}
|
||||
|
||||
[Theory] // review: the message names the limit that actually bound the read — the gateway's ceiling, not the cap
|
||||
[InlineData(0)] // unbounded cap: used to say "more than 0 raw samples (MaxTimeWeightedRawRows)"
|
||||
[InlineData(1_000_000)] // default cap: used to cite 1,000,000
|
||||
public async Task A_ceiling_bound_per_sample_read_names_the_gateway_ceiling(int maxRawRows)
|
||||
{
|
||||
var ex = await Should.ThrowAsync<HistoryReadLimitExceededException>(() => Source(Store(100, ceiling: 50), maxRawRows)
|
||||
.ReadProcessedAsync("T", At(0), At(100), TimeSpan.FromSeconds(10), HistoryAggregateType.Count, Ct));
|
||||
ex.Message.ShouldContain("row ceiling (50,");
|
||||
ex.Message.ShouldNotContain("MaxTimeWeightedRawRows); the Gateway");
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_cap_bound_per_sample_read_still_names_the_cap()
|
||||
{
|
||||
var ex = await Should.ThrowAsync<HistoryReadLimitExceededException>(() => Source(Store(100, ceiling: null), 20)
|
||||
.ReadProcessedAsync("T", At(0), At(100), TimeSpan.FromSeconds(10), HistoryAggregateType.Count, Ct));
|
||||
ex.Message.ShouldContain("more than 20 raw samples");
|
||||
}
|
||||
}
|
||||
+25
@@ -32,6 +32,31 @@ public sealed class SampleMapperTests
|
||||
Assert.Equal(0x80000000u, SampleMapper.ToSnapshot(s).StatusCode);
|
||||
}
|
||||
|
||||
[Theory] // #690: histsdk puts a value on every row, so a NULL row arrives as 0 / "" with a Bad quality
|
||||
[InlineData(0u, 1u)] // the live NULL row: opc_quality 0, AVEVA quality 1
|
||||
[InlineData(24u, 0u)] // any Bad OPC quality (out of service)
|
||||
public void A_Bad_numeric_row_carries_no_value(uint opc, uint quality)
|
||||
{
|
||||
var s = new HistorianSample { Tag = "T", NumericValue = 0, OpcQuality = opc, Quality = quality, Timestamp = Ts(2026, 1, 1, 0, 0, 0) };
|
||||
var snap = SampleMapper.ToSnapshot(s);
|
||||
Assert.True(snap.StatusCode >= 0x80000000u);
|
||||
Assert.Null(snap.Value);
|
||||
}
|
||||
|
||||
[Fact] // #690: a string tag's NULL row arrives as "" + Bad
|
||||
public void A_Bad_string_row_carries_no_value()
|
||||
{
|
||||
var s = new HistorianSample { Tag = "T", StringValue = "", OpcQuality = 0, Quality = 1, Timestamp = Ts(2026, 1, 1, 0, 0, 0) };
|
||||
Assert.Null(SampleMapper.ToSnapshot(s).Value);
|
||||
}
|
||||
|
||||
[Fact] // #690: an Uncertain row keeps its value (only Bad means "no value")
|
||||
public void An_Uncertain_row_keeps_its_value()
|
||||
{
|
||||
var s = new HistorianSample { Tag = "T", NumericValue = 3, OpcQuality = 64, Timestamp = Ts(2026, 1, 1, 0, 0, 0) };
|
||||
Assert.Equal(3d, SampleMapper.ToSnapshot(s).Value);
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void Aggregate_null_value_is_BadNoData()
|
||||
{
|
||||
|
||||
+7
@@ -67,3 +67,10 @@ public sealed class QuestDbEventReadConformance(QuestDbFixture fx) : EventReadCo
|
||||
/// <inheritdoc/>
|
||||
protected override Task<IConformanceBackend> CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx);
|
||||
}
|
||||
|
||||
[Collection(QuestDbCollection.Name)]
|
||||
public sealed class QuestDbBoundingValueConformance(QuestDbFixture fx) : BoundingValueConformance
|
||||
{
|
||||
/// <inheritdoc/>
|
||||
protected override Task<IConformanceBackend> CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx);
|
||||
}
|
||||
|
||||
+7
@@ -67,3 +67,10 @@ public sealed class SqlServerEventReadConformance(SqlHistorianFixture fx) : Even
|
||||
/// <inheritdoc/>
|
||||
protected override Task<IConformanceBackend> CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx);
|
||||
}
|
||||
|
||||
[Collection(SqlHistorianCollection.Name)]
|
||||
public sealed class SqlServerBoundingValueConformance(SqlHistorianFixture fx) : BoundingValueConformance
|
||||
{
|
||||
/// <inheritdoc/>
|
||||
protected override Task<IConformanceBackend> CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx);
|
||||
}
|
||||
|
||||
+12
@@ -72,6 +72,18 @@ public sealed class OtOpcUaServerHostedServiceHistoryUpdateTests
|
||||
Service(view: null).BuildHistoryUpdatePosture().MaxFutureSkew.ShouldBe(TimeSpan.FromDays(1));
|
||||
}
|
||||
|
||||
/// <summary>#691: the ReturnBounds kill switch is bound from <c>ServerHistorian:HonorReturnBounds</c>, default on.</summary>
|
||||
[Fact]
|
||||
public void The_return_bounds_kill_switch_comes_from_options_and_defaults_on()
|
||||
{
|
||||
Service(view: null).HonorReturnBounds.ShouldBeTrue();
|
||||
var config = new ConfigurationBuilder().AddInMemoryCollection(new Dictionary<string, string?>
|
||||
{
|
||||
["ServerHistorian:HonorReturnBounds"] = "false",
|
||||
}).Build();
|
||||
Service(view: null, config: config).HonorReturnBounds.ShouldBeFalse();
|
||||
}
|
||||
|
||||
private static OtOpcUaServerHostedService Service(
|
||||
IHistoryUpdateRoleView? view, IDriverPrimaryRoleSource? primary = null, IConfiguration? config = null,
|
||||
TimeProvider? clock = null) => new(
|
||||
|
||||
+1
@@ -97,6 +97,7 @@ public sealed class HistoryUpdateOverTheWireTests
|
||||
|
||||
var details = new ReadRawModifiedDetails
|
||||
{
|
||||
ReturnBounds = false,
|
||||
StartTime = T1.AddHours(-1), EndTime = T1.AddHours(1), NumValuesPerNode = 100, IsReadModified = true,
|
||||
};
|
||||
var response = await session.HistoryReadAsync(
|
||||
|
||||
@@ -384,6 +384,7 @@ public sealed class NodeManagerAclEnforcementTests : IDisposable
|
||||
context,
|
||||
new ReadRawModifiedDetails
|
||||
{
|
||||
ReturnBounds = false,
|
||||
StartTime = DateTime.UtcNow.AddHours(-1),
|
||||
EndTime = DateTime.UtcNow,
|
||||
NumValuesPerNode = 10,
|
||||
|
||||
+523
@@ -0,0 +1,523 @@
|
||||
using Microsoft.Extensions.Logging.Abstractions;
|
||||
using Opc.Ua;
|
||||
using Opc.Ua.Server;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Commons.OpcUa;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using HistorianRead = ZB.MOM.WW.OtOpcUa.Core.Abstractions.HistoryReadResult;
|
||||
using SdkHistoryReadResult = Opc.Ua.HistoryReadResult;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests;
|
||||
|
||||
/// <summary>Boots ONE real SDK server for the whole bounds suite (the cases swap the historian source per read).</summary>
|
||||
public sealed class BoundsServerFixture : IAsyncLifetime
|
||||
{
|
||||
private readonly string _pkiRoot = Path.Combine(Path.GetTempPath(), $"otopcua-bounds-{Guid.NewGuid():N}");
|
||||
private OpcUaApplicationHost? _host;
|
||||
|
||||
/// <summary>The node manager under test.</summary>
|
||||
public OtOpcUaNodeManager Nm { get; private set; } = null!;
|
||||
|
||||
/// <summary>The historized variable every case reads.</summary>
|
||||
public NodeId NodeId { get; private set; } = null!;
|
||||
|
||||
/// <summary>Serialises the cases (they share the node manager's source).</summary>
|
||||
public SemaphoreSlim Gate { get; } = new(1, 1);
|
||||
|
||||
public async ValueTask InitializeAsync()
|
||||
{
|
||||
var started = await OpcUaHostStart.StartAsync(
|
||||
port => new OpcUaApplicationHost(
|
||||
new OpcUaApplicationHostOptions
|
||||
{
|
||||
ApplicationName = "OtOpcUa.BoundsTest",
|
||||
ApplicationUri = $"urn:OtOpcUa.BoundsTest:{Guid.NewGuid():N}",
|
||||
OpcUaPort = port,
|
||||
PublicHostname = "localhost",
|
||||
PkiStoreRoot = _pkiRoot,
|
||||
},
|
||||
NullLogger<OpcUaApplicationHost>.Instance),
|
||||
() => new OtOpcUaSdkServer(), CancellationToken.None);
|
||||
_host = started.Host;
|
||||
Nm = started.Server.NodeManager!;
|
||||
Nm.HistoryContinuationStore = new InMemoryHistoryContinuationStore();
|
||||
Nm.EnsureVariable("eq-1/bounds", parentFolderNodeId: null, displayName: "Bounds", dataType: "Double",
|
||||
writable: false, historianTagname: "WW.Bounds", realm: AddressSpaceRealm.Uns);
|
||||
NodeId = Nm.TryGetVariable("eq-1/bounds")!.NodeId;
|
||||
}
|
||||
|
||||
public async ValueTask DisposeAsync()
|
||||
{
|
||||
if (_host is not null) await _host.DisposeAsync();
|
||||
if (Directory.Exists(_pkiRoot))
|
||||
{
|
||||
try { Directory.Delete(_pkiRoot, recursive: true); }
|
||||
catch { /* best-effort */ }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Gitea #691: HistoryRead-Raw <c>ReturnBounds</c> (OPC UA Part 11 §4.6 bounding values, §6.5.3.2). The oracle is Part 11
|
||||
/// Table 1 itself — every row, over its data set (samples at 5:00, 5:02, 5:03, 5:05, 5:06) — run against three backend
|
||||
/// shapes: one that reads newest-first natively (SqlServer-like), one that reads forward only (Gateway-like, walked back
|
||||
/// by <see cref="ReverseRawReader"/>) and a microsecond one (QuestDB-like,
|
||||
/// truncating its window bounds).
|
||||
/// </summary>
|
||||
public sealed class NodeManagerHistoryReadBoundsTests(BoundsServerFixture fx) : IClassFixture<BoundsServerFixture>
|
||||
{
|
||||
private static readonly DateTime Day = new(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc);
|
||||
private static readonly DateTime Unspecified = DateTime.MinValue;
|
||||
|
||||
private static DateTime T(string hhmm)
|
||||
{
|
||||
var parts = hhmm.Split(':');
|
||||
return Day.AddHours(int.Parse(parts[0])).AddMinutes(int.Parse(parts[1]));
|
||||
}
|
||||
|
||||
/// <summary>Part 11 Table 1's data: value = minutes past 5:00.</summary>
|
||||
private static DataValueSnapshot[] TableData() =>
|
||||
new[] { "5:00", "5:02", "5:03", "5:05", "5:06" }
|
||||
.Select(s => new DataValueSnapshot((double)(T(s) - T("5:00")).TotalMinutes, StatusCodes.Good, T(s), T(s)))
|
||||
.ToArray();
|
||||
|
||||
public enum Shape { Reverse, ForwardOnly, Microsecond }
|
||||
|
||||
private static IHistorianDataSource Source(Shape shape, IReadOnlyList<DataValueSnapshot> series) => shape switch
|
||||
{
|
||||
Shape.Reverse => new SeriesSource(series, reverse: true, inclusiveForwardEnd: false, resolution: 1),
|
||||
Shape.ForwardOnly => new SeriesSource(series, reverse: false, inclusiveForwardEnd: false, resolution: 1),
|
||||
_ => new SeriesSource(series, reverse: true, inclusiveForwardEnd: false, resolution: 10),
|
||||
};
|
||||
|
||||
/// <summary>
|
||||
/// Part 11 Table 1, every row, verbatim. "FIRST"/"LAST" are <c>Bad_BoundNotFound</c> placeholders (null value) at the
|
||||
/// StartTime/EndTime; a/b footnotes: one second after/before the previous returned value when that side is unspecified.
|
||||
/// </summary>
|
||||
public static TheoryData<string, string, uint, bool, string> Table1 => new()
|
||||
{
|
||||
{ "5:00", "5:05", 0, true, "5:00, 5:02, 5:03, 5:05" },
|
||||
{ "5:00", "5:05", 0, false, "5:00, 5:02, 5:03" },
|
||||
{ "5:01", "5:04", 0, true, "5:00, 5:02, 5:03, 5:05" },
|
||||
{ "5:01", "5:04", 0, false, "5:02, 5:03" },
|
||||
{ "5:05", "5:00", 0, true, "5:05, 5:03, 5:02, 5:00" },
|
||||
{ "5:05", "5:00", 0, false, "5:05, 5:03, 5:02" },
|
||||
{ "5:04", "5:01", 0, true, "5:05, 5:03, 5:02, 5:00" },
|
||||
{ "5:04", "5:01", 0, false, "5:03, 5:02" },
|
||||
{ "4:59", "5:05", 0, true, "FIRST, 5:00, 5:02, 5:03, 5:05" },
|
||||
{ "4:59", "5:05", 0, false, "5:00, 5:02, 5:03" },
|
||||
{ "5:01", "5:07", 0, true, "5:00, 5:02, 5:03, 5:05, 5:06, LAST" },
|
||||
{ "5:01", "5:07", 0, false, "5:02, 5:03, 5:05, 5:06" },
|
||||
{ "5:00", "5:05", 3, true, "5:00, 5:02, 5:03" },
|
||||
{ "5:00", "5:05", 3, false, "5:00, 5:02, 5:03" },
|
||||
{ "5:01", "5:04", 3, true, "5:00, 5:02, 5:03" },
|
||||
{ "5:01", "5:04", 3, false, "5:02, 5:03" },
|
||||
{ "5:05", "5:00", 3, true, "5:05, 5:03, 5:02" },
|
||||
{ "5:05", "5:00", 3, false, "5:05, 5:03, 5:02" },
|
||||
{ "5:04", "5:01", 3, true, "5:05, 5:03, 5:02" },
|
||||
{ "5:04", "5:01", 3, false, "5:03, 5:02" },
|
||||
{ "4:59", "5:05", 3, true, "FIRST, 5:00, 5:02" },
|
||||
{ "4:59", "5:05", 3, false, "5:00, 5:02, 5:03" },
|
||||
{ "5:01", "5:07", 3, true, "5:00, 5:02, 5:03" },
|
||||
{ "5:01", "5:07", 3, false, "5:02, 5:03, 5:05" },
|
||||
{ "5:00", "", 3, true, "5:00, 5:02, 5:03" },
|
||||
{ "5:00", "", 3, false, "5:00, 5:02, 5:03" },
|
||||
{ "5:00", "", 6, true, "5:00, 5:02, 5:03, 5:05, 5:06, LAST+" },
|
||||
{ "5:00", "", 6, false, "5:00, 5:02, 5:03, 5:05, 5:06" },
|
||||
{ "5:07", "", 6, true, "5:06, LAST+" },
|
||||
{ "5:07", "", 6, false, "" },
|
||||
{ "", "5:06", 3, true, "5:06, 5:05, 5:03" },
|
||||
{ "", "5:06", 3, false, "5:06, 5:05, 5:03" },
|
||||
{ "", "5:06", 6, true, "5:06, 5:05, 5:03, 5:02, 5:00, FIRST-" },
|
||||
{ "", "5:06", 6, false, "5:06, 5:05, 5:03, 5:02, 5:00" },
|
||||
{ "", "4:48", 6, true, "5:00, FIRST-" },
|
||||
{ "", "4:48", 6, false, "" },
|
||||
{ "4:48", "4:48", 0, true, "FIRST, 5:00" },
|
||||
{ "4:48", "4:48", 0, false, "" },
|
||||
{ "4:48", "4:48", 1, true, "FIRST" },
|
||||
{ "4:48", "4:48", 1, false, "" },
|
||||
{ "4:48", "4:48", 2, true, "FIRST, 5:00" },
|
||||
{ "5:00", "5:00", 0, true, "5:00, 5:02" },
|
||||
{ "5:00", "5:00", 0, false, "5:00" },
|
||||
{ "5:00", "5:00", 1, true, "5:00" },
|
||||
{ "5:00", "5:00", 1, false, "5:00" },
|
||||
{ "5:01", "5:01", 0, true, "5:00, 5:02" },
|
||||
{ "5:01", "5:01", 0, false, "" },
|
||||
{ "5:01", "5:01", 1, true, "5:00" },
|
||||
{ "5:01", "5:01", 1, false, "" },
|
||||
};
|
||||
|
||||
[Theory]
|
||||
[MemberData(nameof(Table1))]
|
||||
public async Task Part11_table1_first_page_on_a_reverse_capable_backend(string start, string end, uint max, bool bounds, string expected) =>
|
||||
await AssertFirstPage(Shape.Reverse, start, end, max, bounds, expected);
|
||||
|
||||
[Theory]
|
||||
[MemberData(nameof(Table1))]
|
||||
public async Task Part11_table1_first_page_on_a_forward_only_backend(string start, string end, uint max, bool bounds, string expected) =>
|
||||
await AssertFirstPage(Shape.ForwardOnly, start, end, max, bounds, expected);
|
||||
|
||||
[Theory]
|
||||
[MemberData(nameof(Table1))]
|
||||
public async Task Part11_table1_first_page_on_a_microsecond_backend(string start, string end, uint max, bool bounds, string expected) =>
|
||||
await AssertFirstPage(Shape.Microsecond, start, end, max, bounds, expected);
|
||||
|
||||
private async Task AssertFirstPage(Shape shape, string start, string end, uint max, bool bounds, string expected)
|
||||
{
|
||||
var s = start.Length == 0 ? Unspecified : T(start);
|
||||
var e = end.Length == 0 ? Unspecified : T(end);
|
||||
var source = Source(shape, TableData());
|
||||
var (values, error, _) = await ReadAsync(source, s, e, max, bounds, cp: null);
|
||||
|
||||
error.StatusCode.Code.ShouldBe(StatusCodes.Good);
|
||||
Render(values, s, e).ShouldBe(expected);
|
||||
(source as SeriesSource)?.ReverseWindowsOnForwardOnly.ShouldBe(0);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Part 11: bounds count toward NumValuesPerNode, the start bound comes with the first page and the end bound with the
|
||||
/// last — so paging a bounded two-sided read at any page size concatenates to the unpaged (N = 0) answer.
|
||||
/// </summary>
|
||||
[Theory]
|
||||
[InlineData(Shape.Reverse)]
|
||||
[InlineData(Shape.ForwardOnly)]
|
||||
[InlineData(Shape.Microsecond)]
|
||||
public async Task Paging_a_bounded_read_concatenates_to_the_unpaged_answer(Shape shape)
|
||||
{
|
||||
var windows = new[]
|
||||
{
|
||||
("5:00", "5:05"), ("5:01", "5:04"), ("5:05", "5:00"), ("5:04", "5:01"), ("4:59", "5:05"), ("5:01", "5:07"),
|
||||
("4:48", "4:48"), ("5:00", "5:00"), ("5:01", "5:01"), ("5:07", "4:00"), ("5:06", "5:06"),
|
||||
};
|
||||
foreach (var (start, end) in windows)
|
||||
{
|
||||
var source = Source(shape, TableData());
|
||||
var all = await ReadAsync(source, T(start), T(end), 0, true, cp: null);
|
||||
all.Cp.ShouldBeNull();
|
||||
var expected = Render(all.Values, T(start), T(end));
|
||||
for (uint n = 1; n <= 6; n++)
|
||||
{
|
||||
var paged = await DrainAsync(source, T(start), T(end), n, bounds: true);
|
||||
Render(paged, T(start), T(end)).ShouldBe(expected, $"{start}->{end} paged by {n}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>One-sided reads page to the open placeholder exactly once (footnotes a/b), never past it.</summary>
|
||||
[Theory]
|
||||
[InlineData(Shape.Reverse)]
|
||||
[InlineData(Shape.ForwardOnly)]
|
||||
[InlineData(Shape.Microsecond)]
|
||||
public async Task Paging_a_one_sided_bounded_read_ends_with_the_open_placeholder(Shape shape)
|
||||
{
|
||||
for (uint n = 1; n <= 7; n++)
|
||||
{
|
||||
var fwd = await DrainAsync(Source(shape, TableData()), T("5:01"), Unspecified, n, bounds: true);
|
||||
Render(fwd, T("5:01"), Unspecified).ShouldBe("5:00, 5:02, 5:03, 5:05, 5:06, LAST+", $"forward by {n}");
|
||||
var rev = await DrainAsync(Source(shape, TableData()), Unspecified, T("5:04"), n, bounds: true);
|
||||
Render(rev, Unspecified, T("5:04")).ShouldBe("5:05, 5:03, 5:02, 5:00, FIRST-", $"reverse by {n}");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>A bound is a stored sample of any quality: a Bad sample before StartTime is the start bound, as stored.</summary>
|
||||
[Theory]
|
||||
[InlineData(Shape.Reverse)]
|
||||
[InlineData(Shape.ForwardOnly)]
|
||||
[InlineData(Shape.Microsecond)]
|
||||
public async Task A_bad_sample_is_returned_as_the_bound_as_stored(Shape shape)
|
||||
{
|
||||
var series = TableData();
|
||||
series[0] = new DataValueSnapshot(null, StatusCodes.BadNoCommunication, T("5:00"), T("5:00"));
|
||||
var (values, _, _) = await ReadAsync(Source(shape, series), T("5:01"), T("5:04"), 0, true, cp: null);
|
||||
|
||||
values[0].StatusCode.Code.ShouldBe(StatusCodes.BadNoCommunication);
|
||||
values[0].SourceTimestamp.ShouldBe(T("5:00"));
|
||||
values.Count.ShouldBe(4);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Ties on a bound: the start bound before a forward window is the LAST of a tie cluster (the sample immediately
|
||||
/// before StartTime in the read's order); the end bound after it is the FIRST of one; and ties ON StartTime are all
|
||||
/// data (the bound is the first of them, so nothing extra is added).
|
||||
/// </summary>
|
||||
[Theory]
|
||||
[InlineData(Shape.Reverse)]
|
||||
[InlineData(Shape.ForwardOnly)]
|
||||
[InlineData(Shape.Microsecond)]
|
||||
public async Task Tie_clusters_on_either_side_yield_one_bound_each(Shape shape)
|
||||
{
|
||||
var series = new List<DataValueSnapshot>();
|
||||
void Add(string t, double v) => series.Add(new DataValueSnapshot(v, StatusCodes.Good, T(t), T(t)));
|
||||
Add("5:00", 1); Add("5:00", 2); Add("5:00", 3);
|
||||
Add("5:02", 4);
|
||||
Add("5:05", 5); Add("5:05", 6);
|
||||
|
||||
var (fwd, _, _) = await ReadAsync(Source(shape, series), T("5:01"), T("5:04"), 0, true, cp: null);
|
||||
fwd.Select(v => (double)v.Value).ShouldBe([3d, 4d, 5d]);
|
||||
|
||||
var (rev, _, _) = await ReadAsync(Source(shape, series), T("5:04"), T("5:01"), 0, true, cp: null);
|
||||
rev.Select(v => (double)v.Value).ShouldBe([5d, 4d, 3d]);
|
||||
|
||||
var (onStart, _, _) = await ReadAsync(Source(shape, series), T("5:00"), T("5:03"), 0, true, cp: null);
|
||||
onStart.Select(v => (double)v.Value).ShouldBe([1d, 2d, 3d, 4d, 5d]);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// QuestDB truncates the window bounds it is sent to the microsecond: a sample at ⌊StartTime⌋ is inside a window that
|
||||
/// starts a few ticks later, so it is data — never returned a second time as the start bound.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task A_microsecond_backend_does_not_return_a_sample_at_the_truncated_start_twice()
|
||||
{
|
||||
var series = TableData();
|
||||
var (values, _, _) = await ReadAsync(Source(Shape.Microsecond, series), T("5:00").AddTicks(7), T("5:04"), 0, true, cp: null);
|
||||
|
||||
values.Select(v => (double)v.Value).ShouldBe([0d, 2d, 3d, 5d]);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A backend whose forward end is inclusive (the gateway's own <c>ReadRaw</c>), or a resumed read that steps onto
|
||||
/// EndTime, returns the sample AT EndTime as data — it must come back once, as the end bound.
|
||||
/// </summary>
|
||||
[Theory]
|
||||
[InlineData(0u)]
|
||||
[InlineData(1u)]
|
||||
[InlineData(2u)]
|
||||
[InlineData(3u)]
|
||||
public async Task An_inclusive_forward_end_does_not_return_the_end_bound_twice(uint max)
|
||||
{
|
||||
var source = new SeriesSource(TableData(), reverse: false, inclusiveForwardEnd: true, resolution: 1);
|
||||
var values = await DrainAsync(source, T("5:00"), T("5:05"), max, bounds: true);
|
||||
Render(values, T("5:00"), T("5:05")).ShouldBe("5:00, 5:02, 5:03, 5:05");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #691 review: no data AND no bound found ⇒ <c>GoodNoData</c> with no values — the answer a tag with no history (and a
|
||||
/// disabled historian) gave before bounds (§6.5.3.2: bounds make the result Good only "if one or both exist"). Only
|
||||
/// when the WHOLE read is empty: a single bound found keeps the placeholders (Table 1: "4:48 4:48 1 ⇒ FIRST").
|
||||
/// </summary>
|
||||
[Theory]
|
||||
[InlineData(Shape.Reverse)]
|
||||
[InlineData(Shape.ForwardOnly)]
|
||||
[InlineData(Shape.Microsecond)]
|
||||
public async Task No_data_and_no_bound_found_is_GoodNoData_with_no_values(Shape shape)
|
||||
{
|
||||
var windows = new (string S, string E)[] { ("5:00", "5:05"), ("5:05", "5:00"), ("5:00", "5:00"), ("5:00", ""), ("", "5:05") };
|
||||
foreach (var (start, end) in windows)
|
||||
{
|
||||
var s = start.Length == 0 ? Unspecified : T(start);
|
||||
var e = end.Length == 0 ? Unspecified : T(end);
|
||||
foreach (var n in new uint[] { 0, 1, 2, 3 })
|
||||
{
|
||||
if (n == 0 && (s == Unspecified || e == Unspecified)) continue; // a one-sided read needs a cap
|
||||
var (values, error, cp) = await ReadResultAsync(Source(shape, []), s, e, n);
|
||||
values.ShouldBeEmpty($"{start}->{end} by {n}");
|
||||
cp.ShouldBeNull();
|
||||
error.StatusCode.Code.ShouldBe(StatusCodes.Good);
|
||||
LastNodeStatus.Code.ShouldBe(StatusCodes.GoodNoData, $"{start}->{end} by {n}");
|
||||
}
|
||||
}
|
||||
|
||||
// Positive control: one bound found keeps the missing one's placeholder.
|
||||
var single = await DrainAsync(Source(shape, TableData()), T("4:48"), T("4:48"), 1, bounds: true);
|
||||
Render(single, T("4:48"), T("4:48")).ShouldBe("FIRST, 5:00");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #691 review: <c>ServerHistorian:HonorReturnBounds = false</c> restores the pre-#691 answer — ReturnBounds is ignored
|
||||
/// (Table 1's "No" rows for every "Yes" row) and a modified read with ReturnBounds is not refused for it.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task The_kill_switch_ignores_return_bounds()
|
||||
{
|
||||
await fx.Gate.WaitAsync(TestContext.Current.CancellationToken);
|
||||
try
|
||||
{
|
||||
fx.Nm.HonorReturnBounds = false;
|
||||
fx.Nm.HistorianDataSource = Source(Shape.Reverse, TableData());
|
||||
var (values, _, _) = Read(T("5:01"), T("5:07"), 0, bounds: true, cp: null, modified: false);
|
||||
Render(values, T("5:01"), T("5:07")).ShouldBe("5:02, 5:03, 5:05, 5:06");
|
||||
|
||||
var (_, modErr, _) = Read(T("5:00"), T("5:05"), 0, bounds: true, cp: null, modified: true);
|
||||
modErr.StatusCode.Code.ShouldBe(StatusCodes.BadHistoryOperationUnsupported, "no trail on this fake — not InvalidArgument");
|
||||
}
|
||||
finally
|
||||
{
|
||||
fx.Nm.HonorReturnBounds = true;
|
||||
fx.Gate.Release();
|
||||
}
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Bounds_are_honoured_by_default()
|
||||
{
|
||||
await fx.Gate.WaitAsync(TestContext.Current.CancellationToken);
|
||||
try { fx.Nm.HonorReturnBounds.ShouldBeTrue(); }
|
||||
finally { fx.Gate.Release(); }
|
||||
}
|
||||
|
||||
/// <summary>Part 11 §6.5.3.3: a modified read with ReturnBounds is <c>Bad_InvalidArgument</c>.</summary>
|
||||
[Fact]
|
||||
public async Task A_modified_read_with_return_bounds_is_an_invalid_argument()
|
||||
{
|
||||
await fx.Gate.WaitAsync(TestContext.Current.CancellationToken);
|
||||
try
|
||||
{
|
||||
fx.Nm.HistorianDataSource = Source(Shape.Reverse, TableData());
|
||||
var (_, error, cp) = Read(T("5:00"), T("5:05"), 0, bounds: true, cp: null, modified: true);
|
||||
error.StatusCode.Code.ShouldBe(StatusCodes.BadInvalidArgument);
|
||||
cp.ShouldBeNull();
|
||||
}
|
||||
finally { fx.Gate.Release(); }
|
||||
}
|
||||
|
||||
// --- helpers --------------------------------------------------------------------------------
|
||||
|
||||
private static string Render(IReadOnlyList<DataValue> values, DateTime start, DateTime end)
|
||||
{
|
||||
var parts = new List<string>();
|
||||
for (var i = 0; i < values.Count; i++)
|
||||
{
|
||||
var v = values[i];
|
||||
if (v.StatusCode.Code != StatusCodes.BadBoundNotFound)
|
||||
{
|
||||
v.Value.ShouldNotBeNull();
|
||||
parts.Add($"{v.SourceTimestamp.Hour}:{v.SourceTimestamp.Minute:00}");
|
||||
continue;
|
||||
}
|
||||
|
||||
v.Value.ShouldBeNull("a missing bound carries a null value");
|
||||
var isFirst = i == 0 && start != Unspecified && v.SourceTimestamp == start;
|
||||
var isLast = end != Unspecified && v.SourceTimestamp == end;
|
||||
if (isFirst) { parts.Add("FIRST"); continue; }
|
||||
if (isLast) { parts.Add("LAST"); continue; }
|
||||
// Open side (footnotes a/b): one second from the previous returned value.
|
||||
i.ShouldBeGreaterThan(0);
|
||||
var prev = values[i - 1].SourceTimestamp;
|
||||
if (v.SourceTimestamp == prev.AddSeconds(1)) parts.Add("LAST+");
|
||||
else if (v.SourceTimestamp == prev.AddSeconds(-1)) parts.Add("FIRST-");
|
||||
else parts.Add($"?{v.SourceTimestamp:O}");
|
||||
}
|
||||
|
||||
return string.Join(", ", parts);
|
||||
}
|
||||
|
||||
private async Task<List<DataValue>> DrainAsync(IHistorianDataSource source, DateTime start, DateTime end, uint max, bool bounds)
|
||||
{
|
||||
var all = new List<DataValue>();
|
||||
byte[]? cp = null;
|
||||
var pages = 0;
|
||||
do
|
||||
{
|
||||
var (values, error, next) = await ReadAsync(source, start, end, max, bounds, cp);
|
||||
error.StatusCode.Code.ShouldBe(StatusCodes.Good);
|
||||
if (max > 0) values.Count.ShouldBeLessThanOrEqualTo((int)max, "bounds count toward NumValuesPerNode");
|
||||
all.AddRange(values);
|
||||
cp = next;
|
||||
(++pages).ShouldBeLessThan(50, "paging must terminate");
|
||||
}
|
||||
while (cp is not null);
|
||||
|
||||
return all;
|
||||
}
|
||||
|
||||
/// <summary>The node-level StatusCode of the last <see cref="Read"/> (GoodNoData vs Good).</summary>
|
||||
private StatusCode LastNodeStatus;
|
||||
|
||||
private async Task<(IReadOnlyList<DataValue> Values, ServiceResult Error, byte[]? Cp)> ReadResultAsync(
|
||||
IHistorianDataSource source, DateTime start, DateTime end, uint max) =>
|
||||
await ReadAsync(source, start, end, max, bounds: true, cp: null);
|
||||
|
||||
private async Task<(IReadOnlyList<DataValue> Values, ServiceResult Error, byte[]? Cp)> ReadAsync(
|
||||
IHistorianDataSource source, DateTime start, DateTime end, uint max, bool bounds, byte[]? cp)
|
||||
{
|
||||
await fx.Gate.WaitAsync(TestContext.Current.CancellationToken);
|
||||
try
|
||||
{
|
||||
fx.Nm.HistorianDataSource = source;
|
||||
return Read(start, end, max, bounds, cp, modified: false);
|
||||
}
|
||||
finally { fx.Gate.Release(); }
|
||||
}
|
||||
|
||||
private (IReadOnlyList<DataValue> Values, ServiceResult Error, byte[]? Cp) Read(
|
||||
DateTime start, DateTime end, uint max, bool bounds, byte[]? cp, bool modified)
|
||||
{
|
||||
var context = new OperationContext(new RequestHeader(), secureChannelContext: null, RequestType.HistoryRead, identity: null);
|
||||
var details = new ReadRawModifiedDetails
|
||||
{
|
||||
StartTime = start, EndTime = end, NumValuesPerNode = max, ReturnBounds = bounds, IsReadModified = modified,
|
||||
};
|
||||
var nodesToRead = new List<HistoryReadValueId> { new() { NodeId = fx.NodeId, ContinuationPoint = cp } };
|
||||
var results = new List<SdkHistoryReadResult> { null! };
|
||||
var errors = new List<ServiceResult> { null! };
|
||||
fx.Nm.HistoryRead(context, details, TimestampsToReturn.Both, releaseContinuationPoints: false, nodesToRead, results, errors);
|
||||
LastNodeStatus = results[0]?.StatusCode ?? StatusCodes.Bad;
|
||||
|
||||
var values = new List<DataValue>();
|
||||
if (results[0]?.HistoryData is { } ho && !ExtensionObject.IsNull(ho))
|
||||
values.AddRange(((HistoryData)ExtensionObject.ToEncodeable(ho)).DataValues);
|
||||
return (values, errors[0], results[0]?.ContinuationPoint);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// A series-backed historian. <paramref name="reverse"/>: serves <c>start > end</c> as <c>(end, start]</c>
|
||||
/// newest-first (SqlServer/QuestDB); otherwise rejects it (the Gateway) and counts that. The forward read is
|
||||
/// <c>[start, end)</c>, or <c>[start, end]</c> when <paramref name="inclusiveForwardEnd"/> (the gateway's own end);
|
||||
/// a point read is the instant. <paramref name="resolution"/> > 1 truncates every window bound to it first (QuestDB
|
||||
/// µs); the series must already be stored at that resolution.
|
||||
/// </summary>
|
||||
private sealed class SeriesSource(IReadOnlyList<DataValueSnapshot> series, bool reverse, bool inclusiveForwardEnd, long resolution)
|
||||
: IHistorianDataSource
|
||||
{
|
||||
public int ReverseWindowsOnForwardOnly;
|
||||
|
||||
public bool SupportsReverseRaw => reverse;
|
||||
|
||||
public long TimestampResolutionTicks => resolution;
|
||||
|
||||
private DateTime Trunc(DateTime t) => new(t.Ticks - t.Ticks % resolution, DateTimeKind.Utc);
|
||||
|
||||
public Task<HistorianRead> ReadRawAsync(
|
||||
string fullReference, DateTime startUtc, DateTime endUtc, uint maxValuesPerNode, CancellationToken cancellationToken)
|
||||
{
|
||||
static DateTime Ts(DataValueSnapshot s) => s.SourceTimestampUtc ?? DateTime.MinValue;
|
||||
startUtc = Trunc(startUtc);
|
||||
endUtc = Trunc(endUtc);
|
||||
List<DataValueSnapshot> window;
|
||||
if (startUtc > endUtc)
|
||||
{
|
||||
if (!reverse)
|
||||
{
|
||||
Interlocked.Increment(ref ReverseWindowsOnForwardOnly);
|
||||
throw new InvalidOperationException("start_time must be less than or equal to end_time. (InvalidArgument)");
|
||||
}
|
||||
|
||||
window = series.Where(s => Ts(s) > endUtc && Ts(s) <= startUtc).Reverse().ToList();
|
||||
}
|
||||
else if (startUtc == endUtc) window = series.Where(s => Ts(s) == startUtc).ToList();
|
||||
else window = series.Where(s => Ts(s) >= startUtc && (inclusiveForwardEnd ? Ts(s) <= endUtc : Ts(s) < endUtc)).ToList();
|
||||
|
||||
var page = maxValuesPerNode == 0 ? window : window.Take((int)maxValuesPerNode).ToList();
|
||||
return Task.FromResult(new HistorianRead(page, null));
|
||||
}
|
||||
|
||||
public Task<HistorianRead> ReadProcessedAsync(
|
||||
string fullReference, DateTime startUtc, DateTime endUtc, TimeSpan interval,
|
||||
HistoryAggregateType aggregate, CancellationToken cancellationToken) =>
|
||||
Task.FromResult(new HistorianRead(Array.Empty<DataValueSnapshot>(), null));
|
||||
|
||||
public Task<HistorianRead> ReadAtTimeAsync(
|
||||
string fullReference, IReadOnlyList<DateTime> timestampsUtc, CancellationToken cancellationToken) =>
|
||||
Task.FromResult(new HistorianRead(Array.Empty<DataValueSnapshot>(), null));
|
||||
|
||||
public Task<HistoricalEventsResult> ReadEventsAsync(
|
||||
string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents, CancellationToken cancellationToken) =>
|
||||
Task.FromResult(new HistoricalEventsResult(Array.Empty<HistoricalEvent>(), null));
|
||||
|
||||
public HistorianHealthSnapshot GetHealthSnapshot() => NullHistorianDataSource.Instance.GetHealthSnapshot();
|
||||
|
||||
public void Dispose() { }
|
||||
}
|
||||
}
|
||||
+5
@@ -45,6 +45,7 @@ public sealed class NodeManagerHistoryReadConcurrencyTests : IDisposable
|
||||
|
||||
var details = new ReadRawModifiedDetails
|
||||
{
|
||||
ReturnBounds = false,
|
||||
StartTime = DateTime.UtcNow.AddHours(-1),
|
||||
EndTime = DateTime.UtcNow,
|
||||
NumValuesPerNode = 10,
|
||||
@@ -78,6 +79,7 @@ public sealed class NodeManagerHistoryReadConcurrencyTests : IDisposable
|
||||
var nodeIds = MaterializeHistorizedNodes(nm, count: 12);
|
||||
var details = new ReadRawModifiedDetails
|
||||
{
|
||||
ReturnBounds = false,
|
||||
StartTime = DateTime.UtcNow.AddHours(-1),
|
||||
EndTime = DateTime.UtcNow,
|
||||
NumValuesPerNode = 10,
|
||||
@@ -115,6 +117,7 @@ public sealed class NodeManagerHistoryReadConcurrencyTests : IDisposable
|
||||
var nodeIds = MaterializeHistorizedNodes(nm, count: 8);
|
||||
var details = new ReadRawModifiedDetails
|
||||
{
|
||||
ReturnBounds = false,
|
||||
StartTime = DateTime.UtcNow.AddHours(-1),
|
||||
EndTime = DateTime.UtcNow,
|
||||
NumValuesPerNode = 10,
|
||||
@@ -157,6 +160,7 @@ public sealed class NodeManagerHistoryReadConcurrencyTests : IDisposable
|
||||
var nodeIds = MaterializeHistorizedNodes(nm, count: 4);
|
||||
var details = new ReadRawModifiedDetails
|
||||
{
|
||||
ReturnBounds = false,
|
||||
StartTime = DateTime.UtcNow.AddHours(-1),
|
||||
EndTime = DateTime.UtcNow,
|
||||
NumValuesPerNode = 10,
|
||||
@@ -190,6 +194,7 @@ public sealed class NodeManagerHistoryReadConcurrencyTests : IDisposable
|
||||
var ids = MaterializeHistorizedNodes(nm, count: 2);
|
||||
var details = new ReadRawModifiedDetails
|
||||
{
|
||||
ReturnBounds = false,
|
||||
StartTime = DateTime.UtcNow.AddHours(-1),
|
||||
EndTime = DateTime.UtcNow,
|
||||
NumValuesPerNode = 10,
|
||||
|
||||
+146
-29
@@ -160,7 +160,7 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable
|
||||
var ctx = new OperationContext(new RequestHeader(), secureChannelContext: null, RequestType.HistoryRead, identity: null);
|
||||
var res = new List<SdkHistoryReadResult> { null! };
|
||||
var errs = new List<ServiceResult> { null! };
|
||||
nm.HistoryRead(ctx, new ReadRawModifiedDetails { StartTime = Epoch, EndTime = Epoch.AddHours(1), NumValuesPerNode = 2, IsReadModified = false },
|
||||
nm.HistoryRead(ctx, new ReadRawModifiedDetails { ReturnBounds = false, StartTime = Epoch, EndTime = Epoch.AddHours(1), NumValuesPerNode = 2, IsReadModified = false },
|
||||
TimestampsToReturn.Both, false,
|
||||
new List<HistoryReadValueId> { new() { NodeId = n, ContinuationPoint = cp } }, res, errs);
|
||||
err = errs[0];
|
||||
@@ -333,12 +333,16 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task A_backend_without_reverse_support_answers_BadHistoryOperationUnsupported_not_the_earliest_events()
|
||||
public async Task A_backend_without_reverse_support_walks_a_reverse_read_back_never_answering_the_earliest_events()
|
||||
{
|
||||
// #691.4 (was refused BadHistoryOperationUnsupported since #675): served newest-first from forward reads.
|
||||
var (host, nm, node, src) = await SetupAsync(Distinct(10));
|
||||
nm.HistorianDataSource = new ForwardOnlyEventSource(src);
|
||||
ReadEvents(nm, node, max: 3, inboundCp: null, end: Early, start: Late).Error.StatusCode.Code
|
||||
.ShouldBe(StatusCodes.BadHistoryOperationUnsupported);
|
||||
var fwd = new ForwardOnlyEventSource(src);
|
||||
nm.HistorianDataSource = fwd;
|
||||
var r = ReadEvents(nm, node, max: 3, inboundCp: null, end: Early, start: Late);
|
||||
r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good);
|
||||
r.Ids.ShouldBe(["e0009", "e0008", "e0007"]);
|
||||
r.Cp.ShouldNotBeNull();
|
||||
// Forward reads on the same backend are unaffected.
|
||||
DrainAsync(nm, node, max: 4).ShouldBe(src.All.Select(e => e.EventId));
|
||||
await host.DisposeAsync();
|
||||
@@ -349,30 +353,133 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable
|
||||
|
||||
// --- #675 review follow-ups: the reverse-events capability ------------------------------------------------
|
||||
|
||||
// --- #691.4: reverse events on a backend that reads forward only (the Gateway) are walked back ------------------
|
||||
|
||||
[Fact]
|
||||
public async Task End_only_capped_read_on_a_backend_without_reverse_support_pages_forward_as_before()
|
||||
public async Task End_only_capped_read_on_a_forward_only_backend_pages_backward_from_EndTime()
|
||||
{
|
||||
// Gateway (the default backend) cannot read newest-first: an end-only read keeps its pre-#675 forward
|
||||
// [MinValue, EndTime) meaning there rather than turning into a refused reverse read.
|
||||
var (host, nm, node, src) = await SetupAsync(Distinct(25));
|
||||
var fwd = new NonReverseSource(src);
|
||||
nm.HistorianDataSource = fwd;
|
||||
// Part 11 (as raw, #686): the latest events at or before EndTime, newest first — on Gateway this used to keep its
|
||||
// pre-#675 forward [MinValue, EndTime) meaning, i.e. the OLDEST events. The backend is never sent a reverse read.
|
||||
var (host, nm, node, _) = await SetupAsync([]);
|
||||
var gw = new GatewayShapedEventSource(Distinct(25));
|
||||
nm.HistorianDataSource = gw;
|
||||
var ids = DrainAsync(nm, node, max: 10, start: DateTime.MinValue, end: Epoch.AddSeconds(20));
|
||||
ids.ShouldBe(Enumerable.Range(0, 20).Select(i => $"e{i:0000}"));
|
||||
fwd.ReverseCalls.ShouldBe(0);
|
||||
ids.ShouldBe(Enumerable.Range(0, 21).Reverse().Select(i => $"e{i:0000}"));
|
||||
gw.ReverseCalls.ShouldBe(0);
|
||||
gw.BadRequests.ShouldBe(0);
|
||||
await host.DisposeAsync();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Explicit_reverse_read_on_a_backend_without_reverse_support_is_refused_without_calling_it()
|
||||
public async Task Explicit_reverse_read_on_a_forward_only_backend_pages_the_reverse_window_newest_first()
|
||||
{
|
||||
var (host, nm, node, src) = await SetupAsync(Distinct(10));
|
||||
var fwd = new NonReverseSource(src);
|
||||
nm.HistorianDataSource = fwd;
|
||||
ReadEvents(nm, node, max: 3, inboundCp: null, end: Early, start: Late).Error.StatusCode.Code
|
||||
.ShouldBe(StatusCodes.BadHistoryOperationUnsupported);
|
||||
fwd.ReverseCalls.ShouldBe(0, "the capability is checked BEFORE the source is called");
|
||||
fwd.ForwardCalls.ShouldBe(0);
|
||||
// (EndTime, StartTime]: StartTime inclusive, EndTime exclusive.
|
||||
var (host, nm, node, _) = await SetupAsync([]);
|
||||
var gw = new GatewayShapedEventSource(Distinct(30));
|
||||
nm.HistorianDataSource = gw;
|
||||
var ids = DrainAsync(nm, node, max: 4, start: Epoch.AddSeconds(25), end: Epoch.AddSeconds(5));
|
||||
ids.ShouldBe(Enumerable.Range(6, 20).Reverse().Select(i => $"e{i:0000}"));
|
||||
gw.ReverseCalls.ShouldBe(0);
|
||||
await host.DisposeAsync();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Reverse_ties_on_a_forward_only_backend_page_in_a_stable_order_although_the_backend_shuffles_them()
|
||||
{
|
||||
// The gateway orders events by EventTimeUtc only, so ties come back in any order — this fake shuffles them on every
|
||||
// call. The walk sorts ties by EventId (descending, newest-first) on every read, and the oversized-cluster over-fetch
|
||||
// does the same, so a boundary-tie skip always lines up. Cap 3 cuts through a 7-way cluster.
|
||||
var events = new List<HistoricalEvent> { Ev("a0", 0) };
|
||||
for (var i = 0; i < 7; i++) events.Add(Ev($"t{i}", 5));
|
||||
events.Add(Ev("z0", 9));
|
||||
var (host, nm, node, _) = await SetupAsync([]);
|
||||
nm.HistorianDataSource = new GatewayShapedEventSource(events, shuffleSeed: 7);
|
||||
DrainAsync(nm, node, max: 3, start: DateTime.MinValue, end: Epoch.AddSeconds(30)).ShouldBe(
|
||||
["z0", "t6", "t5", "t4", "t3", "t2", "t1", "t0", "a0"]);
|
||||
await host.DisposeAsync();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Uncapped_reverse_read_on_a_forward_only_backend_is_one_forward_read()
|
||||
{
|
||||
var (host, nm, node, _) = await SetupAsync([]);
|
||||
var gw = new GatewayShapedEventSource(Distinct(30));
|
||||
nm.HistorianDataSource = gw;
|
||||
var r = ReadEvents(nm, node, max: 0, inboundCp: null, end: Epoch.AddSeconds(5), start: Epoch.AddSeconds(25));
|
||||
r.Ids.ShouldBe(Enumerable.Range(6, 20).Reverse().Select(i => $"e{i:0000}"));
|
||||
r.Cp.ShouldBeNull();
|
||||
gw.ForwardCalls.ShouldBe(1);
|
||||
await host.DisposeAsync();
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public async Task Reverse_paging_on_a_forward_only_backend_is_exact_on_random_data()
|
||||
{
|
||||
var (host, nm, node, _) = await SetupAsync([]);
|
||||
for (var seed = 0; seed < 80; seed++)
|
||||
{
|
||||
var rng = new Random(seed);
|
||||
var events = new List<HistoricalEvent>();
|
||||
var t = Epoch.Ticks;
|
||||
var n = rng.Next(1, 120);
|
||||
while (events.Count < n)
|
||||
{
|
||||
t += rng.Next(4) switch
|
||||
{
|
||||
0 => rng.Next(1, 10),
|
||||
1 => rng.Next(1, 5) * TimeSpan.TicksPerMillisecond,
|
||||
2 => rng.Next(1, 90) * TimeSpan.TicksPerMinute,
|
||||
_ => rng.Next(1, 2000) * TimeSpan.TicksPerMillisecond,
|
||||
};
|
||||
var ties = rng.Next(5) == 0 ? rng.Next(2, 8) : 1;
|
||||
for (var k = 0; k < ties; k++)
|
||||
{
|
||||
var at = new DateTime(t, DateTimeKind.Utc);
|
||||
events.Add(new HistoricalEvent($"{Guid.NewGuid():N}", "src", at, at, "m", 500));
|
||||
}
|
||||
}
|
||||
|
||||
nm.HistorianDataSource = new GatewayShapedEventSource(events, shuffleSeed: seed);
|
||||
nm.MaxTieClusterOverfetch = rng.Next(2) == 0 ? 8 : 65536;
|
||||
var max = (uint)rng.Next(1, 9);
|
||||
var end = new DateTime(t + rng.Next(0, 3) * TimeSpan.TicksPerSecond, DateTimeKind.Utc);
|
||||
var expected = events.OrderByDescending(e => e.EventTimeUtc).ThenByDescending(e => e.EventId, StringComparer.Ordinal)
|
||||
.Select(e => e.EventId).ToList();
|
||||
|
||||
// The walk refuses more than MaxTieClusterOverfetch events within ONE MILLISECOND (#691.4 re-review): with the
|
||||
// small budget a dense sub-millisecond burst is a legitimate BadTooManyOperations — and nothing else is.
|
||||
var times = events.Select(e => e.EventTimeUtc.Ticks).Order().ToList();
|
||||
var msMax = 0;
|
||||
for (int i = 0, j = 0; i < times.Count; i++)
|
||||
{
|
||||
while (times[i] - times[j] >= TimeSpan.TicksPerMillisecond) j++;
|
||||
msMax = Math.Max(msMax, i - j + 1);
|
||||
}
|
||||
|
||||
var all = new List<string>();
|
||||
byte[]? cp = null;
|
||||
var pages = 0;
|
||||
var refused = false;
|
||||
do
|
||||
{
|
||||
var r = ReadEvents(nm, node, max, cp, end, DateTime.MinValue);
|
||||
if (r.Error.StatusCode.Code == StatusCodes.BadTooManyOperations && msMax > nm.MaxTieClusterOverfetch)
|
||||
{
|
||||
refused = true;
|
||||
break;
|
||||
}
|
||||
|
||||
r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good, $"seed {seed}, ms max {msMax}");
|
||||
all.AddRange(r.Ids);
|
||||
cp = r.Cp;
|
||||
(++pages).ShouldBeLessThan(2000);
|
||||
}
|
||||
while (cp is not null);
|
||||
|
||||
if (refused) all.ShouldBe(expected.Take(all.Count), $"seed {seed}: what came before the refusal");
|
||||
else all.ShouldBe(expected, $"seed {seed}, N = {max}");
|
||||
}
|
||||
|
||||
await host.DisposeAsync();
|
||||
}
|
||||
|
||||
@@ -391,29 +498,39 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable
|
||||
await host.DisposeAsync();
|
||||
}
|
||||
|
||||
/// <summary>Declares the reverse overload (so a call would be visible) but reports no reverse capability.</summary>
|
||||
private sealed class NonReverseSource(IHistorianDataSource inner) : IHistorianDataSource
|
||||
/// <summary>
|
||||
/// Shaped like the Gateway's SQL event read (#691.4): forward only (<see cref="SupportsReverseEvents"/> false — a
|
||||
/// reverse call is counted and refused), <c>[start, end]</c> INCLUSIVE, ordered by event time only with ties in an
|
||||
/// arbitrary order (shuffled on every call when a seed is given), capped with a truncation flag.
|
||||
/// </summary>
|
||||
private sealed class GatewayShapedEventSource(List<HistoricalEvent> events, int? shuffleSeed = null) : IHistorianDataSource
|
||||
{
|
||||
private readonly Random? _rng = shuffleSeed is { } s ? new Random(s) : null;
|
||||
public int ReverseCalls { get; private set; }
|
||||
public int ForwardCalls { get; private set; }
|
||||
public int BadRequests { get; private set; }
|
||||
public bool SupportsReverseEvents => false;
|
||||
public Task<HistoricalEventsResult> ReadEventsAsync(string? n, DateTime s, DateTime e, int m, CancellationToken ct)
|
||||
{
|
||||
ForwardCalls++;
|
||||
return inner.ReadEventsAsync(n, s, e, m, ct);
|
||||
if (s > e || s < DateTime.UnixEpoch) BadRequests++;
|
||||
var window = events.Where(x => x.EventTimeUtc >= s && x.EventTimeUtc <= e)
|
||||
.OrderBy(x => x.EventTimeUtc).ThenBy(_ => _rng?.Next() ?? 0).ToList();
|
||||
var cap = m <= 0 ? int.MaxValue : m;
|
||||
return Task.FromResult(new HistoricalEventsResult(window.Take(cap).ToList(), window.Count > cap ? new byte[] { 1 } : null));
|
||||
}
|
||||
public Task<HistoricalEventsResult> ReadEventsAsync(string? n, DateTime s, DateTime e, int m, bool reverse, CancellationToken ct)
|
||||
{
|
||||
if (reverse) { ReverseCalls++; return inner.ReadEventsAsync(n, s, e, m, true, ct); }
|
||||
if (reverse) { ReverseCalls++; throw new NotSupportedException("forward only"); }
|
||||
return ReadEventsAsync(n, s, e, m, ct);
|
||||
}
|
||||
public Task<HistorianRead> ReadRawAsync(string r, DateTime s, DateTime e, uint m, CancellationToken ct) =>
|
||||
inner.ReadRawAsync(r, s, e, m, ct);
|
||||
NullHistorianDataSource.Instance.ReadRawAsync(r, s, e, m, ct);
|
||||
public Task<HistorianRead> ReadProcessedAsync(string r, DateTime s, DateTime e, TimeSpan i, HistoryAggregateType a, CancellationToken ct) =>
|
||||
inner.ReadProcessedAsync(r, s, e, i, a, ct);
|
||||
NullHistorianDataSource.Instance.ReadProcessedAsync(r, s, e, i, a, ct);
|
||||
public Task<HistorianRead> ReadAtTimeAsync(string r, IReadOnlyList<DateTime> t, CancellationToken ct) =>
|
||||
inner.ReadAtTimeAsync(r, t, ct);
|
||||
public HistorianHealthSnapshot GetHealthSnapshot() => inner.GetHealthSnapshot();
|
||||
NullHistorianDataSource.Instance.ReadAtTimeAsync(r, t, ct);
|
||||
public HistorianHealthSnapshot GetHealthSnapshot() => NullHistorianDataSource.Instance.GetHealthSnapshot();
|
||||
public void Dispose() { }
|
||||
}
|
||||
|
||||
|
||||
+1
-1
@@ -271,7 +271,7 @@ public sealed class NodeManagerHistoryReadModifiedTests : IDisposable
|
||||
var results = new List<SdkHistoryReadResult> { null! };
|
||||
var errors = new List<ServiceResult> { null! };
|
||||
nm.HistoryRead(context,
|
||||
new ReadRawModifiedDetails { StartTime = start, EndTime = end, NumValuesPerNode = max, IsReadModified = modified },
|
||||
new ReadRawModifiedDetails { ReturnBounds = false, StartTime = start, EndTime = end, NumValuesPerNode = max, IsReadModified = modified },
|
||||
TimestampsToReturn.Both, false,
|
||||
new List<HistoryReadValueId> { new() { NodeId = node, ContinuationPoint = cp } }, results, errors);
|
||||
|
||||
|
||||
+165
-3
@@ -650,6 +650,156 @@ public sealed class NodeManagerHistoryReadPagingTests : IDisposable
|
||||
await host.DisposeAsync();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #689: a paged end-only read on a dense tag sizes each segment from the PAGE (N), not from
|
||||
/// <c>MaxTieClusterOverfetch</c>. Live (AVEVA, 1 Hz, N = 3): from about page 5 every page read 32,770 rows and took
|
||||
/// 2–5 s to return 2 rows, because the walk re-aimed its width at half the 65,536-row tie budget and carried it.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Raw_reverse_paging_on_a_dense_forward_only_tag_reads_about_the_page_size_per_page()
|
||||
{
|
||||
// 100,000 samples 100 ms apart (~2.8 h): far more than the default tie budget would ever let a page read.
|
||||
var series = Enumerable.Range(0, 100_000)
|
||||
.Select(i => { var t = Epoch.AddMilliseconds(100L * i); return new DataValueSnapshot((double)i, StatusCodes.Good, t, t); })
|
||||
.ToArray();
|
||||
var source = new ForwardOnlyHistorianDataSource(series);
|
||||
var (host, nm, nodeId) = await BootForwardOnlyAsync(source, "FwdDense689");
|
||||
nm.MaxTieClusterOverfetch.ShouldBe(65536, "the default budget — the one the live gate ran with");
|
||||
var end = Epoch.AddMilliseconds(100L * 99_999);
|
||||
|
||||
byte[]? cp = null;
|
||||
var expected = 99_999;
|
||||
for (var page = 1; page <= 20; page++)
|
||||
{
|
||||
source.ResetCounts();
|
||||
var (values, error, next) = ReadRaw(nm, nodeId, DateTime.MinValue, end, max: 3, inboundCp: cp);
|
||||
error.StatusCode.Code.ShouldBe(StatusCodes.Good);
|
||||
values.ShouldNotBeEmpty();
|
||||
values[^1].ShouldBe((double)(expected - 2), $"page {page} continues exactly");
|
||||
expected = (int)values[^1];
|
||||
next.ShouldNotBeNull();
|
||||
cp = next;
|
||||
|
||||
// The discriminating counts: rows read and the largest cap asked for, per page. Page 1 has no density yet and
|
||||
// may read one discovery segment (a minute here, 600 rows); every page after it is sized from N.
|
||||
var rowsBound = page == 1 ? ReverseWalk.DiscoveryRows + 2 : 64;
|
||||
source.RowsReturned.ShouldBeLessThanOrEqualTo(rowsBound, $"page {page} read {source.RowsReturned} rows to return {values.Count}");
|
||||
source.MaxCapRequested.ShouldBeLessThanOrEqualTo((uint)rowsBound, $"page {page} asked for {source.MaxCapRequested} rows");
|
||||
source.ReadCount.ShouldBeLessThanOrEqualTo(3, $"page {page}");
|
||||
}
|
||||
|
||||
await host.DisposeAsync();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #689/#691: randomized end-to-end paging of end-only reads through the node manager on a forward-only backend —
|
||||
/// bursts, gaps, sub-millisecond stamps and tie clusters (some larger than the page) — the union of the pages is
|
||||
/// every sample exactly once, newest first, whatever N and the tie budget.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Raw_reverse_paging_on_a_forward_only_backend_is_exact_on_random_data()
|
||||
{
|
||||
var (host, nm, nodeId) = await BootForwardOnlyAsync(new ForwardOnlyHistorianDataSource([]), "FwdFuzz");
|
||||
for (var seed = 0; seed < 150; seed++)
|
||||
{
|
||||
var rng = new Random(seed);
|
||||
var series = new List<DataValueSnapshot>();
|
||||
var t = Epoch.Ticks;
|
||||
var v = 0;
|
||||
var count = rng.Next(1, 300);
|
||||
while (series.Count < count)
|
||||
{
|
||||
t += rng.Next(5) switch
|
||||
{
|
||||
0 => rng.Next(1, 10),
|
||||
1 => rng.Next(1, 5) * TimeSpan.TicksPerMillisecond,
|
||||
2 => rng.Next(1, 90) * TimeSpan.TicksPerMinute,
|
||||
_ => rng.Next(1, 2000) * TimeSpan.TicksPerMillisecond,
|
||||
};
|
||||
var ties = rng.Next(6) == 0 ? rng.Next(2, 7) : 1;
|
||||
for (var k = 0; k < ties; k++)
|
||||
{
|
||||
var ts = new DateTime(t, DateTimeKind.Utc);
|
||||
series.Add(new DataValueSnapshot((double)v++, StatusCodes.Good, ts, ts));
|
||||
}
|
||||
}
|
||||
|
||||
// No fabricated start row: the Gateway source drops it (#687) — the node manager's tie-cluster point read relies on
|
||||
// that. The walk itself is fuzzed WITH the row in ReverseWalkTests.
|
||||
nm.HistorianDataSource = new ForwardOnlyHistorianDataSource(series, emitBoundaryRow: false);
|
||||
nm.MaxTieClusterOverfetch = rng.Next(2) == 0 ? 8 : 65536;
|
||||
var max = (uint)rng.Next(1, 12);
|
||||
var end = new DateTime(t + rng.Next(0, 3) * TimeSpan.TicksPerSecond, DateTimeKind.Utc);
|
||||
|
||||
// The walk refuses more than MaxTieClusterOverfetch samples within ONE MILLISECOND (#691.4 re-review), so with
|
||||
// the small budget a dense sub-millisecond burst is a legitimate BadTooManyOperations — and nothing else is.
|
||||
var msMax = 0;
|
||||
for (int i = 0, j = 0; i < series.Count; i++)
|
||||
{
|
||||
while (series[i].SourceTimestampUtc!.Value.Ticks - series[j].SourceTimestampUtc!.Value.Ticks >= TimeSpan.TicksPerMillisecond) j++;
|
||||
msMax = Math.Max(msMax, i - j + 1);
|
||||
}
|
||||
|
||||
var collected = new List<double>();
|
||||
byte[]? cp = null;
|
||||
var pages = 0;
|
||||
var refused = false;
|
||||
do
|
||||
{
|
||||
var (values, error, next) = ReadRaw(nm, nodeId, DateTime.MinValue, end, max, inboundCp: cp);
|
||||
if (error.StatusCode.Code == StatusCodes.BadTooManyOperations && msMax > nm.MaxTieClusterOverfetch)
|
||||
{
|
||||
refused = true;
|
||||
break;
|
||||
}
|
||||
|
||||
error.StatusCode.Code.ShouldBe(StatusCodes.Good, $"seed {seed}, N = {max}, ms max {msMax}");
|
||||
collected.AddRange(values);
|
||||
cp = next;
|
||||
(++pages).ShouldBeLessThan(2000, "paging must terminate, not loop");
|
||||
}
|
||||
while (cp is not null);
|
||||
|
||||
var expected = Enumerable.Range(0, series.Count).Reverse().Select(i => (double)i).ToList();
|
||||
if (refused) collected.ShouldBe(expected.Take(collected.Count), $"seed {seed}: what came before the refusal");
|
||||
else collected.ShouldBe(expected, $"seed {seed}, N = {max}");
|
||||
}
|
||||
|
||||
await host.DisposeAsync();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #691: the last pages of a deep reverse page-through no longer each walk down to 1970 (live: ~14–16 reads, ~2.5 s
|
||||
/// per page). The stretch a page proved empty below its oldest row rides the continuation point, and a page that runs
|
||||
/// out of rows asks once for the whole remainder — so every page here is a couple of reads.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Raw_reverse_page_through_on_a_forward_only_backend_does_not_rewalk_to_the_epoch_on_each_last_page()
|
||||
{
|
||||
var source = new ForwardOnlyHistorianDataSource(MakeSeries(count: 25, stepSeconds: 1));
|
||||
var (host, nm, nodeId) = await BootForwardOnlyAsync(source, "FwdDeep691");
|
||||
nm.Clock = new FixedClock(Epoch.AddHours(1));
|
||||
|
||||
var collected = new List<double>();
|
||||
byte[]? cp = null;
|
||||
var page = 0;
|
||||
do
|
||||
{
|
||||
source.ResetCounts();
|
||||
var (values, error, next) = ReadRaw(nm, nodeId, DateTime.MinValue, Epoch.AddHours(1), max: 10, inboundCp: cp);
|
||||
error.StatusCode.Code.ShouldBe(StatusCodes.Good);
|
||||
collected.AddRange(values);
|
||||
cp = next;
|
||||
page++;
|
||||
source.ReadCount.ShouldBeLessThanOrEqualTo(page == 1 ? 4 : 1, $"page {page}");
|
||||
}
|
||||
while (cp is not null && page < 10);
|
||||
|
||||
collected.ShouldBe(Enumerable.Range(0, 25).Reverse().Select(i => (double)i));
|
||||
source.MinRequestedStart.ShouldBeGreaterThanOrEqualTo(DateTime.UnixEpoch);
|
||||
await host.DisposeAsync();
|
||||
}
|
||||
|
||||
/// <summary>#686 review: a reverse read from <c>StartTime = DateTime.MaxValue</c> walks back from now + 1 day, not from
|
||||
/// year 9999.</summary>
|
||||
[Fact]
|
||||
@@ -817,6 +967,7 @@ public sealed class NodeManagerHistoryReadPagingTests : IDisposable
|
||||
|
||||
var details = new ReadRawModifiedDetails
|
||||
{
|
||||
ReturnBounds = false,
|
||||
StartTime = start,
|
||||
EndTime = end,
|
||||
NumValuesPerNode = max,
|
||||
@@ -850,6 +1001,7 @@ public sealed class NodeManagerHistoryReadPagingTests : IDisposable
|
||||
new RequestHeader(), secureChannelContext: null, RequestType.HistoryRead, identity: null);
|
||||
var details = new ReadRawModifiedDetails
|
||||
{
|
||||
ReturnBounds = false,
|
||||
StartTime = Epoch, EndTime = Epoch.AddHours(1), NumValuesPerNode = 100, IsReadModified = false,
|
||||
};
|
||||
var nodesToRead = new List<HistoryReadValueId> { new() { NodeId = nodeId, ContinuationPoint = cp } };
|
||||
@@ -978,9 +1130,15 @@ public sealed class NodeManagerHistoryReadPagingTests : IDisposable
|
||||
/// <summary>Reads whose row count equalled the cap (a full, possibly truncated, read).</summary>
|
||||
public int FullReads { get; private set; }
|
||||
|
||||
/// <summary>Rows returned across every read since the last <see cref="ResetCounts"/> (#689).</summary>
|
||||
public long RowsReturned { get; private set; }
|
||||
|
||||
/// <summary>The largest <c>maxValuesPerNode</c> asked for since the last <see cref="ResetCounts"/> (#689).</summary>
|
||||
public uint MaxCapRequested { get; private set; }
|
||||
|
||||
public void ResetCounts()
|
||||
{
|
||||
lock (_gate) FullReads = 0;
|
||||
lock (_gate) { FullReads = 0; RowsReturned = 0; MaxCapRequested = 0; }
|
||||
Interlocked.Exchange(ref _readCount, 0);
|
||||
}
|
||||
|
||||
@@ -1016,8 +1174,12 @@ public sealed class NodeManagerHistoryReadPagingTests : IDisposable
|
||||
: new DataValueSnapshot(prior.Value, prior.StatusCode, stamp, stamp));
|
||||
rows.AddRange(series.Where(s => Ts(s) >= startUtc && Ts(s) <= endUtc));
|
||||
var page = maxValuesPerNode == 0 ? rows : rows.Take((int)maxValuesPerNode).ToList();
|
||||
if (maxValuesPerNode > 0 && page.Count == maxValuesPerNode)
|
||||
lock (_gate) FullReads++;
|
||||
lock (_gate)
|
||||
{
|
||||
if (maxValuesPerNode > 0 && page.Count == maxValuesPerNode) FullReads++;
|
||||
RowsReturned += page.Count;
|
||||
if (maxValuesPerNode > MaxCapRequested) MaxCapRequested = maxValuesPerNode;
|
||||
}
|
||||
int seen;
|
||||
while (page.Count > (seen = Volatile.Read(ref _maxRowsReturned))
|
||||
&& Interlocked.CompareExchange(ref _maxRowsReturned, page.Count, seen) != seen) { }
|
||||
|
||||
@@ -52,6 +52,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
|
||||
var end = DateTime.UtcNow;
|
||||
var details = new ReadRawModifiedDetails
|
||||
{
|
||||
ReturnBounds = false,
|
||||
StartTime = start,
|
||||
EndTime = end,
|
||||
NumValuesPerNode = 100,
|
||||
@@ -97,6 +98,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
|
||||
|
||||
var details = new ReadRawModifiedDetails
|
||||
{
|
||||
ReturnBounds = false,
|
||||
StartTime = DateTime.UtcNow.AddHours(-1),
|
||||
EndTime = DateTime.UtcNow,
|
||||
NumValuesPerNode = 10,
|
||||
@@ -129,6 +131,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
|
||||
|
||||
var details = new ReadRawModifiedDetails
|
||||
{
|
||||
ReturnBounds = false,
|
||||
StartTime = DateTime.UtcNow.AddHours(-1),
|
||||
EndTime = DateTime.UtcNow,
|
||||
NumValuesPerNode = 10,
|
||||
@@ -162,6 +165,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
|
||||
|
||||
var details = new ReadRawModifiedDetails
|
||||
{
|
||||
ReturnBounds = false,
|
||||
StartTime = DateTime.UtcNow.AddHours(-1),
|
||||
EndTime = DateTime.UtcNow,
|
||||
NumValuesPerNode = 10,
|
||||
@@ -195,6 +199,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
|
||||
|
||||
var details = new ReadRawModifiedDetails
|
||||
{
|
||||
ReturnBounds = false,
|
||||
StartTime = DateTime.UtcNow.AddHours(-1),
|
||||
EndTime = DateTime.UtcNow,
|
||||
NumValuesPerNode = 10,
|
||||
@@ -613,6 +618,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
|
||||
|
||||
var details = new ReadRawModifiedDetails
|
||||
{
|
||||
ReturnBounds = false,
|
||||
StartTime = DateTime.UtcNow.AddHours(-1),
|
||||
EndTime = DateTime.UtcNow,
|
||||
NumValuesPerNode = 10,
|
||||
@@ -658,6 +664,7 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
|
||||
|
||||
var details = new ReadRawModifiedDetails
|
||||
{
|
||||
ReturnBounds = false,
|
||||
StartTime = DateTime.UtcNow.AddHours(-1),
|
||||
EndTime = DateTime.UtcNow,
|
||||
NumValuesPerNode = 10,
|
||||
|
||||
@@ -0,0 +1,329 @@
|
||||
using Opc.Ua;
|
||||
using Shouldly;
|
||||
using Xunit;
|
||||
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
|
||||
using HistorianRead = ZB.MOM.WW.OtOpcUa.Core.Abstractions.HistoryReadResult;
|
||||
|
||||
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests;
|
||||
|
||||
/// <summary>
|
||||
/// Gitea #689 / #691: the reverse walk-back (<see cref="ReverseWalk"/> through <see cref="ReverseRawReader"/>) driven
|
||||
/// directly, without a server: a randomized exactness check against a brute-force answer, and the sizing properties.
|
||||
/// </summary>
|
||||
public sealed class ReverseWalkTests
|
||||
{
|
||||
private static CancellationToken Ct => TestContext.Current.CancellationToken;
|
||||
|
||||
private static readonly DateTime T0 = new(2026, 10, 12, 18, 0, 0, DateTimeKind.Utc);
|
||||
|
||||
private static DataValueSnapshot At(long ticks, double v) =>
|
||||
new(v, StatusCodes.Good, new DateTime(ticks, DateTimeKind.Utc), new DateTime(ticks, DateTimeKind.Utc));
|
||||
|
||||
/// <summary>
|
||||
/// A randomized walk over AVEVA-shaped forward reads (inclusive end, a fabricated row at the query start — optionally
|
||||
/// stamped at the start rounded UP to the millisecond — counted against the cap): every page is exactly the brute-force
|
||||
/// newest-first answer over <c>(lo, hi]</c>, ties in the backend's forward order reversed, whatever the carried width
|
||||
/// — and a timestamp over the tie budget is the only refusal.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task A_walked_page_is_exactly_the_newest_N_over_the_window_on_random_data()
|
||||
{
|
||||
var failures = new List<string>();
|
||||
for (var seed = 0; seed < 400; seed++)
|
||||
{
|
||||
var rng = new Random(seed);
|
||||
var series = RandomSeries(rng, out _);
|
||||
// The refusal unit is one millisecond (#691.4 re-review): the most rows inside any 1 ms window.
|
||||
var maxCluster = 0;
|
||||
for (int i = 0, j = 0; i < series.Count; i++)
|
||||
{
|
||||
while (Ts(series[i]) - Ts(series[j]) >= TimeSpan.TicksPerMillisecond) j++;
|
||||
maxCluster = Math.Max(maxCluster, i - j + 1);
|
||||
}
|
||||
var source = new AvevaShapedSource(series, roundUp: rng.Next(2) == 0);
|
||||
var tieBudget = rng.Next(4) switch { 0 => 1, 1 => 3, 2 => 8, _ => 65536 };
|
||||
var want = (uint)rng.Next(1, 40);
|
||||
var span = Ts(series[^1]) - Ts(series[0]);
|
||||
var hi = Ts(series[0]) + (long)(rng.NextDouble() * 1.2 * span) + rng.Next(0, 3);
|
||||
var lo = Ts(series[0]) - TimeSpan.TicksPerSecond + (long)(rng.NextDouble() * (hi - Ts(series[0])) * 0.8);
|
||||
if (rng.Next(5) == 0) lo = 0; // an end-only read
|
||||
var hint = rng.Next(3) switch
|
||||
{
|
||||
0 => default,
|
||||
1 => new ReverseWalkHint(rng.Next(1, 10)),
|
||||
_ => new ReverseWalkHint((long)(rng.NextDouble() * TimeSpan.TicksPerHour)),
|
||||
};
|
||||
if (rng.Next(2) == 0)
|
||||
{
|
||||
// A TRUE empty stretch (#691): between two adjacent distinct timestamps, or from the floor to the first row.
|
||||
var k = rng.Next(series.Count);
|
||||
var from = k == 0 ? (rng.Next(2) == 0 ? 0 : DateTime.UnixEpoch.Ticks) : Ts(series[k - 1]);
|
||||
var to = Ts(series[k]);
|
||||
if (to > from) hint = hint with { EmptyFromTicks = from, EmptyToTicks = to };
|
||||
}
|
||||
|
||||
// Page on: each next page resumes just below the previous page's oldest timestamp with the hint it returned —
|
||||
// so the hints the walk itself produces (width, proven-empty stretch) are exercised too.
|
||||
for (var pageNo = 0; pageNo < 6 && hi > Math.Max(lo, DateTime.UnixEpoch.Ticks); pageNo++)
|
||||
{
|
||||
var expected = series.Where(s => Ts(s) > Math.Max(lo, DateTime.UnixEpoch.Ticks) && Ts(s) <= hi)
|
||||
.Select((s, i) => (s, i)).OrderByDescending(x => Ts(x.s)).ThenByDescending(x => x.i)
|
||||
.Take((int)want).Select(x => (double)x.s.Value!).ToList();
|
||||
|
||||
try
|
||||
{
|
||||
var page = await ReverseRawReader.ReadAsync(source, "T", new DateTime(hi, DateTimeKind.Utc),
|
||||
new DateTime(lo, DateTimeKind.Utc), want, tieBudget, T0.AddDays(1), hint, Ct);
|
||||
var got = page.Samples.Select(s => (double)s.Value!).ToList();
|
||||
if (!got.SequenceEqual(expected))
|
||||
{
|
||||
failures.Add($"seed {seed} page {pageNo}: got [{string.Join(",", got)}] expected [{string.Join(",", expected)}]");
|
||||
break;
|
||||
}
|
||||
|
||||
if (page.Hint.WidthTicks <= 0) failures.Add($"seed {seed}: no width carried");
|
||||
if (got.Count < want) break;
|
||||
hint = page.Hint;
|
||||
hi = Ts(page.Samples[^1]) - 1;
|
||||
}
|
||||
catch (HistoryReadLimitExceededException)
|
||||
{
|
||||
if (maxCluster <= tieBudget) failures.Add($"seed {seed}: refused with no millisecond over {tieBudget} rows (max {maxCluster})");
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
if (source.MaxCap > 2L * Math.Max(Math.Max(tieBudget, ReverseWalk.DiscoveryRows), 2 * want + 16) + 4)
|
||||
failures.Add($"seed {seed}: asked for {source.MaxCap} rows");
|
||||
if (source.BadRequests > 0) failures.Add($"seed {seed}: {source.BadRequests} invalid requests");
|
||||
}
|
||||
|
||||
failures.ShouldBeEmpty();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #689: the carried width sizes a resumed page from N, not from the tie budget — the live symptom was 32,770 rows
|
||||
/// per page for N = 3 on a 1 Hz tag once the width had grown over a few pages.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task A_resumed_page_on_a_dense_tag_reads_about_N_rows_with_the_default_tie_budget()
|
||||
{
|
||||
var series = Enumerable.Range(0, 200_000).Select(i => At(T0.Ticks + i * TimeSpan.TicksPerSecond, i)).ToList();
|
||||
var source = new AvevaShapedSource(series);
|
||||
var hi = new DateTime(Ts(series[^1]), DateTimeKind.Utc);
|
||||
var hint = default(ReverseWalkHint);
|
||||
|
||||
for (var page = 1; page <= 30; page++)
|
||||
{
|
||||
source.Reset();
|
||||
var result = await ReverseRawReader.ReadAsync(source, "T", hi, DateTime.MinValue, 3, 65536, hi.AddHours(1), hint, Ct);
|
||||
result.Samples.Count.ShouldBe(3);
|
||||
hint = result.Hint;
|
||||
hi = result.Samples[^1].SourceTimestampUtc!.Value.AddTicks(-1);
|
||||
if (page > 1) source.RowsReturned.ShouldBeLessThanOrEqualTo(30, $"page {page}");
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #689: a dense tag that stopped long ago. The walk's ×4 growth across the empty stretch overshoots into far more
|
||||
/// rows than one read takes; the truncated read only shows the OLDEST of them, so the walk must not keep re-entering
|
||||
/// that stretch — it searches the gap above the newest row it saw (gallop up, then bisect). Measured: 26 reads.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task A_dense_tag_below_a_long_empty_stretch_is_found_in_a_bounded_number_of_reads()
|
||||
{
|
||||
// 10 Hz for 3 hours, ending two days before "now".
|
||||
var start = T0.AddDays(-2).AddHours(-3);
|
||||
var series = Enumerable.Range(0, 108_000).Select(i => At(start.Ticks + i * TimeSpan.TicksPerMillisecond * 100, i)).ToList();
|
||||
var source = new AvevaShapedSource(series);
|
||||
|
||||
var page = await ReverseRawReader.ReadAsync(source, "T", T0, DateTime.MinValue, 3, 65536, T0, default(ReverseWalkHint), Ct);
|
||||
|
||||
page.Samples.Select(s => (double)s.Value!).ShouldBe([107_999d, 107_998d, 107_997d]);
|
||||
source.Reads.ShouldBeLessThanOrEqualTo(30);
|
||||
source.RowsReturned.ShouldBeLessThanOrEqualTo(30 * (ReverseWalk.DiscoveryRows + 2));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #691: the end of a walk proves the rest of the window empty in about one read — after two empty segments the walk
|
||||
/// asks once for the whole remainder down to the floor — instead of one read per ×4 growth step to 1970 (~14 reads).
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Running_out_of_rows_proves_the_rest_of_the_window_empty_in_a_few_reads()
|
||||
{
|
||||
var series = Enumerable.Range(0, 5).Select(i => At(T0.Ticks + i * TimeSpan.TicksPerSecond, i)).ToList();
|
||||
var source = new AvevaShapedSource(series);
|
||||
|
||||
var page = await ReverseRawReader.ReadAsync(source, "T", T0.AddSeconds(4), DateTime.MinValue, 10, 65536, T0,
|
||||
default(ReverseWalkHint), Ct);
|
||||
|
||||
page.Samples.Select(s => (double)s.Value!).ShouldBe([4d, 3d, 2d, 1d, 0d]);
|
||||
source.Reads.ShouldBeLessThanOrEqualTo(4, "one segment holding the rows, two empty ones, then the remainder at once");
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #691: a page's last complete read proves the stretch below its oldest row empty, and the continuation carries it:
|
||||
/// the next page steps straight over it to the floor instead of walking it again.
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task A_resumed_page_steps_over_the_stretch_the_previous_page_proved_empty()
|
||||
{
|
||||
var series = Enumerable.Range(0, 25).Select(i => At(T0.Ticks + i * TimeSpan.TicksPerSecond, i)).ToList();
|
||||
var source = new AvevaShapedSource(series);
|
||||
var now = T0.AddHours(1);
|
||||
|
||||
// Page 1: an hour of nothing above the rows; the remainder probe reads all 25 and proves (1970, row 0) empty.
|
||||
var p1 = await ReverseRawReader.ReadAsync(source, "T", now, DateTime.MinValue, 10, 65536, now, default(ReverseWalkHint), Ct);
|
||||
p1.Samples.Select(s => (double)s.Value!).ShouldBe(Enumerable.Range(15, 10).Reverse().Select(i => (double)i));
|
||||
p1.Hint.HasEmptyStretch.ShouldBeTrue();
|
||||
p1.Hint.EmptyFromTicks.ShouldBeLessThanOrEqualTo(DateTime.UnixEpoch.Ticks);
|
||||
p1.Hint.EmptyToTicks.ShouldBe(Ts(series[0]));
|
||||
|
||||
var hint = p1.Hint;
|
||||
var hi = p1.Samples[^1].SourceTimestampUtc!.Value.AddTicks(-1);
|
||||
for (var page = 2; page <= 3; page++)
|
||||
{
|
||||
source.Reset();
|
||||
var p = await ReverseRawReader.ReadAsync(source, "T", hi, DateTime.MinValue, 10, 65536, now, hint, Ct);
|
||||
p.Samples.Count.ShouldBe(page == 2 ? 10 : 5);
|
||||
source.Reads.ShouldBe(1, $"page {page} reads its rows and steps over the proven-empty stretch to the floor");
|
||||
hint = p.Hint;
|
||||
hi = p.Samples[^1].SourceTimestampUtc!.Value.AddTicks(-1);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #691.4 review: a burst just over the discovery read, sub-millisecond spaced, ending 100 days ago. The search for
|
||||
/// the top of the burst above the truncated read never re-sends an identical request (halving below the 1 ms request
|
||||
/// granularity did), and gallops up from the truncated read's newest row instead of bisecting the whole 100-day gap
|
||||
/// down to the millisecond (was 46 reads).
|
||||
/// </summary>
|
||||
[Fact]
|
||||
public async Task Bisecting_above_a_truncated_read_never_resends_an_identical_request()
|
||||
{
|
||||
var end = T0.AddDays(-100);
|
||||
var count = ReverseWalk.DiscoveryRows + 2;
|
||||
var series = Enumerable.Range(0, count).Select(i => At(end.Ticks - (count - 1 - i) * 1_000, i)).ToList(); // 0.1 ms apart
|
||||
var source = new AvevaShapedSource(series);
|
||||
|
||||
var page = await ReverseRawReader.ReadAsync(source, "T", T0, DateTime.MinValue, 3, 65536, T0, default(ReverseWalkHint), Ct);
|
||||
|
||||
page.Samples.Select(s => (double)s.Value!).ShouldBe([count - 1d, count - 2d, count - 3d]);
|
||||
source.RepeatedRequests.ShouldBe(0, string.Join("\n", source.Log));
|
||||
source.Reads.ShouldBeLessThanOrEqualTo(22, string.Join("\n", source.Log));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// #691.4 re-review (S8): a tie cluster under a carried width. 30 rows at one whole-millisecond timestamp 0.3 s below
|
||||
/// <c>hi</c>, nothing above, N = 1, a carried 1 s width. The truncated read of the cluster left the width at one tick;
|
||||
/// once the gap above it was under two milliseconds nothing floored it, so segHi stepped down one tick per read —
|
||||
/// 10,006 reads of the same millisecond (28 before the gallop).
|
||||
/// </summary>
|
||||
[Theory]
|
||||
[InlineData(30, 1, 0.3, 1.0)]
|
||||
[InlineData(34, 1, 0.52, 0.48)]
|
||||
[InlineData(25, 1, 1.2, 1.0)]
|
||||
[InlineData(40, 3, 3.5, 2.0)]
|
||||
public async Task A_tie_cluster_under_a_carried_width_is_read_in_a_bounded_number_of_reads(
|
||||
int ties, int want, double secondsBelow, double hintSeconds)
|
||||
{
|
||||
var at = T0.Ticks - (long)(secondsBelow * TimeSpan.TicksPerSecond);
|
||||
at -= at % TimeSpan.TicksPerMillisecond;
|
||||
var series = Enumerable.Range(0, ties).Select(i => At(at, i)).ToList();
|
||||
var source = new AvevaShapedSource(series);
|
||||
|
||||
var page = await ReverseRawReader.ReadAsync(source, "T", T0, DateTime.MinValue, (uint)want, 65536, T0,
|
||||
new ReverseWalkHint((long)(hintSeconds * TimeSpan.TicksPerSecond)), Ct);
|
||||
|
||||
page.Samples.Select(s => (double)s.Value!).ShouldBe(Enumerable.Range(ties - want, want).Reverse().Select(i => (double)i));
|
||||
source.Reads.ShouldBeLessThanOrEqualTo(30, string.Join("\n", source.Log.Take(40)));
|
||||
source.RepeatedRequests.ShouldBe(0);
|
||||
}
|
||||
|
||||
private static long Ts(DataValueSnapshot s) => s.SourceTimestampUtc!.Value.Ticks;
|
||||
|
||||
/// <summary>Random ascending series: bursts, gaps, sub-millisecond stamps and tie clusters.</summary>
|
||||
private static List<DataValueSnapshot> RandomSeries(Random rng, out int maxCluster)
|
||||
{
|
||||
var list = new List<DataValueSnapshot>();
|
||||
var t = T0.Ticks - rng.Next(0, 3) * TimeSpan.TicksPerDay;
|
||||
var count = rng.Next(1, 400);
|
||||
maxCluster = 1;
|
||||
var v = 0;
|
||||
while (list.Count < count)
|
||||
{
|
||||
t += rng.Next(6) switch
|
||||
{
|
||||
0 => rng.Next(1, 10), // sub-millisecond
|
||||
1 => rng.Next(1, 5) * TimeSpan.TicksPerMillisecond, // dense
|
||||
2 => rng.Next(1, 3) * TimeSpan.TicksPerSecond,
|
||||
3 => rng.Next(1, 120) * TimeSpan.TicksPerMinute, // a gap
|
||||
_ => rng.Next(1, 1000) * TimeSpan.TicksPerMillisecond,
|
||||
};
|
||||
var ties = rng.Next(8) == 0 ? rng.Next(2, 12) : 1;
|
||||
maxCluster = Math.Max(maxCluster, ties);
|
||||
for (var k = 0; k < ties; k++) list.Add(At(t, v++));
|
||||
}
|
||||
|
||||
return list;
|
||||
}
|
||||
|
||||
/// <summary>AVEVA-shaped forward-only raw source (see the fuzz test's summary).</summary>
|
||||
private sealed class AvevaShapedSource(IReadOnlyList<DataValueSnapshot> series, bool roundUp = false) : IHistorianDataSource
|
||||
{
|
||||
public long MaxCap { get; private set; }
|
||||
public long RowsReturned { get; private set; }
|
||||
public int Reads { get; private set; }
|
||||
public int BadRequests { get; private set; }
|
||||
|
||||
/// <summary>Requests that repeated an earlier request exactly (start, end, cap) — read-only, so pure waste.</summary>
|
||||
public int RepeatedRequests { get; private set; }
|
||||
|
||||
private readonly HashSet<(DateTime, DateTime, uint)> _seen = [];
|
||||
|
||||
/// <summary>Every request, for failure messages.</summary>
|
||||
public List<string> Log { get; } = [];
|
||||
|
||||
public void Reset() { MaxCap = 0; RowsReturned = 0; Reads = 0; RepeatedRequests = 0; _seen.Clear(); Log.Clear(); }
|
||||
|
||||
public bool SupportsReverseRaw => false;
|
||||
|
||||
public Task<HistorianRead> ReadRawAsync(string fullReference, DateTime startUtc, DateTime endUtc, uint maxValuesPerNode,
|
||||
CancellationToken cancellationToken)
|
||||
{
|
||||
Reads++;
|
||||
if (!_seen.Add((startUtc, endUtc, maxValuesPerNode))) RepeatedRequests++;
|
||||
|
||||
Log.Add($"[{startUtc:O} .. {endUtc:O}] cap {maxValuesPerNode}");
|
||||
if (startUtc > endUtc || maxValuesPerNode == 0 || startUtc < DateTime.UnixEpoch) BadRequests++;
|
||||
MaxCap = Math.Max(MaxCap, maxValuesPerNode);
|
||||
var stamp = roundUp && startUtc.Ticks % TimeSpan.TicksPerMillisecond != 0
|
||||
? new DateTime(startUtc.Ticks - startUtc.Ticks % TimeSpan.TicksPerMillisecond + TimeSpan.TicksPerMillisecond, DateTimeKind.Utc)
|
||||
: startUtc;
|
||||
var prior = series.LastOrDefault(s => s.SourceTimestampUtc <= startUtc);
|
||||
var rows = new List<DataValueSnapshot>
|
||||
{
|
||||
prior is null
|
||||
? new DataValueSnapshot(-1d, StatusCodes.BadNoData, stamp, stamp)
|
||||
: new DataValueSnapshot(-2d, prior.StatusCode, stamp, stamp),
|
||||
};
|
||||
rows.AddRange(series.Where(s => s.SourceTimestampUtc >= startUtc && s.SourceTimestampUtc <= endUtc));
|
||||
var page = rows.Take((int)Math.Min(int.MaxValue, maxValuesPerNode)).ToList();
|
||||
RowsReturned += page.Count;
|
||||
return Task.FromResult(new HistorianRead(page, null));
|
||||
}
|
||||
|
||||
public Task<HistorianRead> ReadProcessedAsync(string fullReference, DateTime startUtc, DateTime endUtc, TimeSpan interval,
|
||||
HistoryAggregateType aggregate, CancellationToken cancellationToken) => throw new NotSupportedException();
|
||||
|
||||
public Task<HistorianRead> ReadAtTimeAsync(string fullReference, IReadOnlyList<DateTime> timestampsUtc,
|
||||
CancellationToken cancellationToken) => throw new NotSupportedException();
|
||||
|
||||
public Task<HistoricalEventsResult> ReadEventsAsync(string? sourceName, DateTime startUtc, DateTime endUtc, int maxEvents,
|
||||
CancellationToken cancellationToken) => throw new NotSupportedException();
|
||||
|
||||
public HistorianHealthSnapshot GetHealthSnapshot() => NullHistorianDataSource.Instance.GetHealthSnapshot();
|
||||
|
||||
public void Dispose() { }
|
||||
}
|
||||
}
|
||||
@@ -104,6 +104,21 @@ public sealed class ServerHistorianOptionsTests
|
||||
/// #674: the HistoryUpdate future-skew bound defaults to one day and binds from configuration as a TimeSpan;
|
||||
/// a non-positive value is a deliberate "disabled", not a misconfiguration, so it does not warn.
|
||||
/// </summary>
|
||||
/// <summary>#691: the ReturnBounds kill switch defaults on and binds off; neither value warns.</summary>
|
||||
[Fact]
|
||||
public void HonorReturnBounds_defaults_to_true_and_binds_false_silently()
|
||||
{
|
||||
Assert.True(new ServerHistorianOptions().HonorReturnBounds);
|
||||
var bound = new ConfigurationBuilder()
|
||||
.AddInMemoryCollection(new Dictionary<string, string?> { ["ServerHistorian:HonorReturnBounds"] = "false" })
|
||||
.Build()
|
||||
.GetSection(ServerHistorianOptions.SectionName)
|
||||
.Get<ServerHistorianOptions>();
|
||||
Assert.False(bound!.HonorReturnBounds);
|
||||
Assert.Empty(new ServerHistorianOptions
|
||||
{ Enabled = true, Endpoint = "https://h:5222", ApiKey = "histgw_x_y", HonorReturnBounds = false }.Validate());
|
||||
}
|
||||
|
||||
[Fact]
|
||||
public void HistoryUpdateMaxFutureSkew_defaults_to_one_day_binds_and_zero_is_silent()
|
||||
{
|
||||
|
||||
Reference in New Issue
Block a user