diff --git a/CLAUDE.md b/CLAUDE.md index 96dafeef..c9c902df 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1383,7 +1383,7 @@ authorization uses the standard `AccessLevels.HistoryRead` bit set at materializ | `MaxTieClusterOverfetch` | `65536` | Bounded over-fetch the HistoryRead-Raw paging uses to page within an oversized same-timestamp tie cluster (retained from the prior backend) | | `HistoryReadBatchParallelism` | `4` | Max nodes served concurrently within one HistoryRead batch (03/S3); `⌈N/P⌉ × CallTimeout` worst case instead of sequential | | `MaxConcurrentHistoryReadBatches` | `16` | Process-wide concurrent HistoryRead batch cap (03/S3); flood degrades to `BadTooManyOperations`. Total in-flight reads capped at 16 × 4 = 64 | -| `MaxTimeWeightedRawRows` | `1000000` | `SqlServer`/`QuestDb` (#671): raw in-window rows one time-weighted aggregate read (incl. `Total` and, since #680, PercentGood/PercentBad) may stream into the C# evaluator — and, since #681, every `Gateway` per-sample read (raw-evaluated); window query capped at this + 1, past it ⇒ `BadTooManyOperations` for that node. ≤ 0 = unbounded | +| `MaxTimeWeightedRawRows` | `1000000` | `SqlServer`/`QuestDb` (#671): raw in-window rows one time-weighted aggregate read (incl. `Total` and, since #680, PercentGood/PercentBad) may stream into the C# evaluator — and, since #681, every `Gateway` per-sample read (raw-evaluated); window query capped at this + 1, past it ⇒ `BadTooManyOperations` for that node. ≤ 0 = unbounded. Gateway (#688): the `cap + 18` budget is clamped at the gateway's default `MaxValuesCeiling` (1,000,000) while the cap is ≤ it, so the default no longer trips a strict (`AllowStreamingBeyondCeiling=false`) gateway | | `HistoryReadDeadline` | `00:01:00` | Per-request HistoryRead wall-clock deadline (03/S3); a node still in flight surfaces `BadTimeout`. Non-positive = unbounded | ### Alarm-history path (`AlarmHistorian` section) diff --git a/docs/Historian.md b/docs/Historian.md index 6353e9ce..cdbc450b 100644 --- a/docs/Historian.md +++ b/docs/Historian.md @@ -146,7 +146,7 @@ Processed read is `BadAggregateNotSupported` and no history aggregate functions | `MaxTieClusterOverfetch` | int | `65536` | (Node-manager paging; all backends — a `SqlServer` PK rules out tie clusters, so it is moot there.) Maximum samples the server will fetch in one shot to page through a tie cluster (multiple samples sharing one `SourceTimestamp`). A cluster larger than this ceiling fails `BadTooManyOperations` (#686; was `BadHistoryOperationUnsupported`). Also the per-read row budget of a Gateway reverse raw read (walked back from forward reads, #686): one timestamp holding more samples is refused `BadTooManyOperations`. Values below 1 count as 1. Raise to handle abnormally large tie clusters; the default covers all normal-data cases. | | `HistoryReadBatchParallelism` | int | `4` | Max historized nodes served **concurrently** within one HistoryRead batch (arch-review 03/S3). The SDK's `HistoryRead*` override surface is synchronous, so each arm bridges once at its boundary; this turns per-node serving from sequential (`N × CallTimeout` worst case) into `⌈N / P⌉ × CallTimeout`. Also caps the gateway-load multiplication a single client can command. `≤ 1` serves sequentially. | | `MaxConcurrentHistoryReadBatches` | int | `16` | Process-wide limit on HistoryRead (and HistoryUpdate) batches served **concurrently** (arch-review 03/S3). A flood of history clients degrades to `BadTooManyOperations` rather than exhausting the SDK request-thread pool. With `HistoryReadBatchParallelism` this caps total in-flight gateway reads at `MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism` (default 16 × 4 = 64). A batch that cannot get a slot within `min(HistoryReadDeadline, 5 s)` fails every handle `BadTooManyOperations`. | -| `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` (#671), and the `Gateway` per-sample aggregates (#681 — every per-sample read there streams raw samples, so the cap applies to the window's raw rows; see [Aggregate set](#aggregate-set-and-badaggregatenotsupported-285)). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative, and — since #680 — **PercentGood/PercentBad**) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped on the SQL backends (pushed down). 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). | +| `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` (#671), and the `Gateway` per-sample aggregates (#681 — every per-sample read there streams raw samples, so the cap applies to the window's raw rows; see [Aggregate set](#aggregate-set-and-badaggregatenotsupported-285)). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative, and — since #680 — **PercentGood/PercentBad**) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped on the SQL backends (pushed down). 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). `Gateway` (#688): the row budget it asks the gateway for (`cap + 18`) is clamped at the gateway's default `MaxValuesCeiling` (1,000,000) while the cap is at or under it, so the default works against a gateway with `AllowStreamingBeyondCeiling=false`; against a gateway with a *lower* ceiling, set this to `ceiling − 18`. | | `HistoryReadDeadline` | TimeSpan | `00:01:00` | Per-request wall-clock deadline for a HistoryRead (and HistoryUpdate) batch (arch-review 03/S3). A node still in flight at the deadline surfaces `BadTimeout`, so one request can never hold an SDK request thread longer than this regardless of node count. Non-positive = unbounded (a `Validate()` warning). | | `HistoryUpdateMaxFutureSkew` | TimeSpan | `1.00:00:00` | #674. How far past the server clock a HistoryUpdate (Insert / Replace / Update) item's `SourceTimestamp` may lie; an item beyond `now + skew` is refused **`BadOutOfRange`** by the node manager before any backend sees it (every backend). Guards against a far-future typo (a 2062 timestamp). The live recorder is not bounded. Non-positive disables the bound (no warning — a deliberate choice). See [HistoryUpdate](#historyupdate). | @@ -890,12 +890,16 @@ and evicts its shared historian session (see the HistorianGateway issue text in - **Look-ahead.** The end-of-data rules need to know whether data exists after the window: a sample exactly at the window end (forward) comes from the window read; otherwise a 2-row read `[end, now + 1 day]`. In both small reads a row stamped exactly at the query start (a boundary row) or the query end (a possible terminal row) is not a sample, whatever its value. -- **Bound.** In-window rows are capped by `MaxTimeWeightedRawRows`; the gateway is asked for `cap + 18` rows. Past the cap the rest - of the (bounded) stream is drained, then the node fails `BadTooManyOperations`; a stream that delivers the whole budget cannot be - proven complete and fails the same way. The gateway's own refusal of that budget (`InvalidArgument` "max_values exceeds the - server ceiling" — `Historian:ReadLimits:MaxValuesCeiling`, default 1,000,000, with `AllowStreamingBeyondCeiling=false`; the - default `1,000,000 + 18` exceeds it) is also `BadTooManyOperations`, not a backend fault — lower `MaxTimeWeightedRawRows` to - `ceiling − 18` on such a gateway. A caller cancellation (client cancel, `HistoryReadDeadline`) is not counted as a backend fault. +- **Bound.** In-window rows are capped by `MaxTimeWeightedRawRows`; the gateway is asked for `cap + 18` rows, **clamped at the + gateway's default ceiling of 1,000,000 when the cap itself is at or under it** (#688 — the default `1,000,000 + 18` used to exceed + that ceiling, so a gateway with `AllowStreamingBeyondCeiling=false` refused **every** per-sample read, even a 9-row window, until + the cap was lowered; the default config now works against the default gateway). A cap above 1,000,000 is a deliberate choice for a + streaming gateway and keeps its `+ 18`. Past the cap the rest of the (bounded) stream is drained, then the node fails + `BadTooManyOperations`; a stream that delivers the whole budget cannot be proven complete and fails the same way (so with the + default cap a window of more than ~999,982 rows fails, as it would one row later). The gateway's own refusal of the budget + (`InvalidArgument` "max_values exceeds the server ceiling" — `Historian:ReadLimits:MaxValuesCeiling` lowered below the default, + with `AllowStreamingBeyondCeiling=false`) is also `BadTooManyOperations`, not a backend fault — lower `MaxTimeWeightedRawRows` to + `ceiling − 18` on such a gateway. A raw read's budget is **not** clamped (see [#687](#gateway-raw-reads-return-stored-samples-only-687)). A caller cancellation (client cancel, `HistoryReadDeadline`) is not counted as a backend fault. - **Bucket statistics** are `RawBucketAccumulator`, the C# statement of the SQL backends' per-sample columns — incl. #684: Average / Minimum / Maximum / Range / *ActualTime over **Good** samples only, the Uncertain extremes feeding only the Min/Max status rule, and an Uncertain String/DateTime sample not refusing the Good-value aggregates. A change to those SQL columns diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/RawAggregateReader.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/RawAggregateReader.cs index ae75f362..484b56c4 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/RawAggregateReader.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/RawAggregateReader.cs @@ -47,8 +47,9 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway; /// SQL backends' time-weighted reads use): past it the node fails with /// (⇒ BadTooManyOperations) once the stream — bounded by the /// requested row budget — has been drained. The gateway is asked for at most cap + 2 + -/// rows; a stream that delivers all of them cannot be proven complete and fails the -/// same way rather than answering from a truncated window. +/// rows, clamped at while the cap is at +/// or under it (#688, ); a stream that delivers all of them cannot be proven +/// complete and fails the same way rather than answering from a truncated window. /// /// /// Values and status. A sample's value is its numeric value (Double), else its string value, else Empty; its @@ -109,7 +110,29 @@ internal static class RawAggregateReader /// The configured cap (<= 0 = unbounded). /// The row limit to request. public static int MaxValuesFor(int maxWindowRows) => - maxWindowRows <= 0 ? int.MaxValue : (int)Math.Min(int.MaxValue, (long)maxWindowRows + 2 + PreWindowAllowance); + maxWindowRows <= 0 ? int.MaxValue : ClampToGatewayCeiling((long)maxWindowRows + 2 + PreWindowAllowance, maxWindowRows); + + /// + /// The HistorianGateway's default Historian:ReadLimits:MaxValuesCeiling. With + /// AllowStreamingBeyondCeiling=false the gateway refuses any max_values above its ceiling. + /// + internal const int GatewayDefaultMaxValuesCeiling = 1_000_000; + + /// + /// #688: a ReadRaw row budget of plus this backend's own allowance + /// () clamped at — but never below + /// itself. The allowance (the boundary row, the margin, the completeness row) must not push a + /// budget that fits the gateway's default ceiling over it: the default MaxTimeWeightedRawRows (1,000,000) + 18 + /// used to exceed it, so a gateway with AllowStreamingBeyondCeiling=false refused EVERY per-sample read. A + /// count above the ceiling is the operator's (or client's) deliberate choice for a streaming gateway and keeps its + /// allowance. The cost of clamping is that a window holding within the allowance of the ceiling cannot be proven + /// complete and fails as over the cap — the same outcome it would get one row later. + /// + /// The budget including the allowance. + /// The caller's own row count (the cap, or NumValuesPerNode), > 0. + /// The max_values to request. + internal static int ClampToGatewayCeiling(long requested, long rows) => + (int)Math.Min(int.MaxValue, rows > GatewayDefaultMaxValuesCeiling ? requested : Math.Min(requested, GatewayDefaultMaxValuesCeiling)); /// Maps one gateway raw sample to a stored sample (value + mapped status, Empty clamped Bad). /// The gateway sample; its timestamp must be set. diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/GatewayRawAggregateTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/GatewayRawAggregateTests.cs index 5ce178df..b99f2b57 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/GatewayRawAggregateTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/GatewayRawAggregateTests.cs @@ -369,7 +369,7 @@ public sealed class GatewayRawAggregateTests await Source(count).ReadProcessedAsync("A", At(0), At(80), TimeSpan.FromSeconds(20), HistoryAggregateType.Count, Ct); // The window read ends AT the window end; probe + look-ahead are separate 2-row reads, all consumed to completion. count.RawCalls.ShouldBe([ - (At(0) - TimeSpan.FromMilliseconds(1), At(80), 1_000_000 + 2 + 16), + (At(0) - TimeSpan.FromMilliseconds(1), At(80), 1_000_000), // cap + 18, clamped at the gateway default ceiling (#688) (DateTime.UnixEpoch, At(0).AddTicks(-1), 2), (At(80), Now + TimeSpan.FromDays(1), 2), ]); @@ -473,6 +473,36 @@ public sealed class GatewayRawAggregateTests ds2.GetHealthSnapshot().TotalFailures.ShouldBe(1); } + [Fact] // #688: the default cap's budget fits the gateway's default ceiling, so a strict gateway serves the default config + public async Task The_default_caps_row_budget_does_not_exceed_the_gateways_default_ceiling() + { + var fake = Store([D(-5, 1), D(5, 2)], avevaBoundary: true); + await Source(fake).ReadProcessedAsync("T", At(0), At(10), TimeSpan.Zero, HistoryAggregateType.Average, Ct); + TimeWeightedAggregator.DefaultMaxWindowRows.ShouldBe(1_000_000); + fake.RawCalls[0].MaxValues.ShouldBe(RawAggregateReader.GatewayDefaultMaxValuesCeiling); + } + + [Theory] // #688: the + 18 allowance is clamped at the gateway's default ceiling, never below the cap itself + [InlineData(10, 28)] + [InlineData(999_982, 1_000_000)] + [InlineData(999_990, 1_000_000)] + [InlineData(1_000_000, 1_000_000)] + [InlineData(2_000_000, 2_000_018)] // a cap above the default ceiling is a deliberate choice for a streaming gateway + [InlineData(0, int.MaxValue)] + [InlineData(-1, int.MaxValue)] + public void The_window_row_budget_is_clamped_at_the_gateways_default_ceiling(int cap, int expected) => + RawAggregateReader.MaxValuesFor(cap).ShouldBe(expected); + + [Fact] // #688: near the ceiling the window is still served when it is not full + public async Task A_cap_at_the_ceiling_still_serves_a_window_below_it() + { + var data = Enumerable.Range(0, 10).Select(i => D(i, i)).ToArray(); + var fake = Store(data, avevaBoundary: true); + var r = await Source(fake, maxRawRows: 1_000_000) + .ReadProcessedAsync("T", At(0), At(10), TimeSpan.Zero, HistoryAggregateType.Average, Ct); + r.Samples.Single().Value.ShouldBe(4.5); + } + [Fact] public async Task A_caller_cancellation_is_not_counted_as_a_backend_failure() {