fix(historian): #688 clamp the Gateway per-sample row budget at the gateway's default ceiling

The default MaxTimeWeightedRawRows (1,000,000) + 18 exceeded the gateway's
default MaxValuesCeiling (1,000,000), so with AllowStreamingBeyondCeiling=false
every Gateway per-sample aggregate read was refused. The allowance is now clamped
at 1,000,000 while the cap is at or under it (RawAggregateReader.
ClampToGatewayCeiling); a cap above it keeps its allowance. Raw reads are not
clamped (rebased onto #686).

CLAUDE.md

@@ -1383,7 +1383,7 @@ authorization uses the standard `AccessLevels.HistoryRead` bit set at materializ
 | `MaxTieClusterOverfetch` | `65536` | Bounded over-fetch the HistoryRead-Raw paging uses to page within an oversized same-timestamp tie cluster (retained from the prior backend) |
 | `HistoryReadBatchParallelism` | `4` | Max nodes served concurrently within one HistoryRead batch (03/S3); `⌈N/P⌉ × CallTimeout` worst case instead of sequential |
 | `MaxConcurrentHistoryReadBatches` | `16` | Process-wide concurrent HistoryRead batch cap (03/S3); flood degrades to `BadTooManyOperations`. Total in-flight reads capped at 16 × 4 = 64 |
-| `MaxTimeWeightedRawRows` | `1000000` | `SqlServer`/`QuestDb` (#671): raw in-window rows one time-weighted aggregate read (incl. `Total` and, since #680, PercentGood/PercentBad) may stream into the C# evaluator — and, since #681, every `Gateway` per-sample read (raw-evaluated); window query capped at this + 1, past it ⇒ `BadTooManyOperations` for that node. ≤ 0 = unbounded |
+| `MaxTimeWeightedRawRows` | `1000000` | `SqlServer`/`QuestDb` (#671): raw in-window rows one time-weighted aggregate read (incl. `Total` and, since #680, PercentGood/PercentBad) may stream into the C# evaluator — and, since #681, every `Gateway` per-sample read (raw-evaluated); window query capped at this + 1, past it ⇒ `BadTooManyOperations` for that node. ≤ 0 = unbounded. Gateway (#688): the `cap + 18` budget is clamped at the gateway's default `MaxValuesCeiling` (1,000,000) while the cap is ≤ it, so the default no longer trips a strict (`AllowStreamingBeyondCeiling=false`) gateway |
 | `HistoryReadDeadline` | `00:01:00` | Per-request HistoryRead wall-clock deadline (03/S3); a node still in flight surfaces `BadTimeout`. Non-positive = unbounded |
 
 ### Alarm-history path (`AlarmHistorian` section)

docs/Historian.md

@@ -146,7 +146,7 @@ Processed read is `BadAggregateNotSupported` and no history aggregate functions
 | `MaxTieClusterOverfetch` | int | `65536` | (Node-manager paging; all backends — a `SqlServer` PK rules out tie clusters, so it is moot there.) Maximum samples the server will fetch in one shot to page through a tie cluster (multiple samples sharing one `SourceTimestamp`). A cluster larger than this ceiling fails `BadTooManyOperations` (#686; was `BadHistoryOperationUnsupported`). Also the per-read row budget of a Gateway reverse raw read (walked back from forward reads, #686): one timestamp holding more samples is refused `BadTooManyOperations`. Values below 1 count as 1. Raise to handle abnormally large tie clusters; the default covers all normal-data cases. |
 | `HistoryReadBatchParallelism` | int | `4` | Max historized nodes served **concurrently** within one HistoryRead batch (arch-review 03/S3). The SDK's `HistoryRead*` override surface is synchronous, so each arm bridges once at its boundary; this turns per-node serving from sequential (`N × CallTimeout` worst case) into `⌈N / P⌉ × CallTimeout`. Also caps the gateway-load multiplication a single client can command. `≤ 1` serves sequentially. |
 | `MaxConcurrentHistoryReadBatches` | int | `16` | Process-wide limit on HistoryRead (and HistoryUpdate) batches served **concurrently** (arch-review 03/S3). A flood of history clients degrades to `BadTooManyOperations` rather than exhausting the SDK request-thread pool. With `HistoryReadBatchParallelism` this caps total in-flight gateway reads at `MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism` (default 16 × 4 = 64). A batch that cannot get a slot within `min(HistoryReadDeadline, 5 s)` fails every handle `BadTooManyOperations`. |
-| `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` (#671), and the `Gateway` per-sample aggregates (#681 — every per-sample read there streams raw samples, so the cap applies to the window's raw rows; see [Aggregate set](#aggregate-set-and-badaggregatenotsupported-285)). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative, and — since #680 — **PercentGood/PercentBad**) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped on the SQL backends (pushed down). 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). |
+| `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` (#671), and the `Gateway` per-sample aggregates (#681 — every per-sample read there streams raw samples, so the cap applies to the window's raw rows; see [Aggregate set](#aggregate-set-and-badaggregatenotsupported-285)). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative, and — since #680 — **PercentGood/PercentBad**) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped on the SQL backends (pushed down). 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). `Gateway` (#688): the row budget it asks the gateway for (`cap + 18`) is clamped at the gateway's default `MaxValuesCeiling` (1,000,000) while the cap is at or under it, so the default works against a gateway with `AllowStreamingBeyondCeiling=false`; against a gateway with a *lower* ceiling, set this to `ceiling − 18`. |
 | `HistoryReadDeadline` | TimeSpan | `00:01:00` | Per-request wall-clock deadline for a HistoryRead (and HistoryUpdate) batch (arch-review 03/S3). A node still in flight at the deadline surfaces `BadTimeout`, so one request can never hold an SDK request thread longer than this regardless of node count. Non-positive = unbounded (a `Validate()` warning). |
 | `HistoryUpdateMaxFutureSkew` | TimeSpan | `1.00:00:00` | #674. How far past the server clock a HistoryUpdate (Insert / Replace / Update) item's `SourceTimestamp` may lie; an item beyond `now + skew` is refused **`BadOutOfRange`** by the node manager before any backend sees it (every backend). Guards against a far-future typo (a 2062 timestamp). The live recorder is not bounded. Non-positive disables the bound (no warning — a deliberate choice). See [HistoryUpdate](#historyupdate). |
 
@@ -890,12 +890,16 @@ and evicts its shared historian session (see the HistorianGateway issue text in
 - **Look-ahead.** The end-of-data rules need to know whether data exists after the window: a sample exactly at the window end
   (forward) comes from the window read; otherwise a 2-row read `[end, now + 1 day]`. In both small reads a row stamped exactly at
   the query start (a boundary row) or the query end (a possible terminal row) is not a sample, whatever its value.
-- **Bound.** In-window rows are capped by `MaxTimeWeightedRawRows`; the gateway is asked for `cap + 18` rows. Past the cap the rest
+- **Bound.** In-window rows are capped by `MaxTimeWeightedRawRows`; the gateway is asked for `cap + 18` rows, **clamped at the
-  of the (bounded) stream is drained, then the node fails `BadTooManyOperations`; a stream that delivers the whole budget cannot be
+  gateway's default ceiling of 1,000,000 when the cap itself is at or under it** (#688 — the default `1,000,000 + 18` used to exceed
-  proven complete and fails the same way. The gateway's own refusal of that budget (`InvalidArgument` "max_values exceeds the
+  that ceiling, so a gateway with `AllowStreamingBeyondCeiling=false` refused **every** per-sample read, even a 9-row window, until
-  server ceiling" — `Historian:ReadLimits:MaxValuesCeiling`, default 1,000,000, with `AllowStreamingBeyondCeiling=false`; the
+  the cap was lowered; the default config now works against the default gateway). A cap above 1,000,000 is a deliberate choice for a
-  default `1,000,000 + 18` exceeds it) is also `BadTooManyOperations`, not a backend fault — lower `MaxTimeWeightedRawRows` to
+  streaming gateway and keeps its `+ 18`. Past the cap the rest of the (bounded) stream is drained, then the node fails
-  `ceiling − 18` on such a gateway. A caller cancellation (client cancel, `HistoryReadDeadline`) is not counted as a backend fault.
+  `BadTooManyOperations`; a stream that delivers the whole budget cannot be proven complete and fails the same way (so with the
+  default cap a window of more than ~999,982 rows fails, as it would one row later). The gateway's own refusal of the budget
+  (`InvalidArgument` "max_values exceeds the server ceiling" — `Historian:ReadLimits:MaxValuesCeiling` lowered below the default,
+  with `AllowStreamingBeyondCeiling=false`) is also `BadTooManyOperations`, not a backend fault — lower `MaxTimeWeightedRawRows` to
+  `ceiling − 18` on such a gateway. A raw read's budget is **not** clamped (see [#687](#gateway-raw-reads-return-stored-samples-only-687)). A caller cancellation (client cancel, `HistoryReadDeadline`) is not counted as a backend fault.
 - **Bucket statistics** are `RawBucketAccumulator`, the C# statement of the SQL backends' per-sample columns — incl. #684:
   Average / Minimum / Maximum / Range / *ActualTime over **Good** samples only, the Uncertain extremes feeding only the Min/Max
   status rule, and an Uncertain String/DateTime sample not refusing the Good-value aggregates. A change to those SQL columns

src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway/RawAggregateReader.cs

@@ -47,8 +47,9 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway;
 ///         SQL backends' time-weighted reads use): past it the node fails with
 ///         <see cref="HistoryReadLimitExceededException"/> (⇒ <c>BadTooManyOperations</c>) once the stream — bounded by the
 ///         requested row budget — has been drained. The gateway is asked for at most <c>cap + 2 + </c>
-///         <see cref="PreWindowAllowance"/> rows; a stream that delivers all of them cannot be proven complete and fails the
+///         <see cref="PreWindowAllowance"/> rows, clamped at <see cref="GatewayDefaultMaxValuesCeiling"/> while the cap is at
-///         same way rather than answering from a truncated window.
+///         or under it (#688, <see cref="ClampToGatewayCeiling"/>); a stream that delivers all of them cannot be proven
+///         complete and fails the same way rather than answering from a truncated window.
 ///     </para>
 ///     <para>
 ///         <b>Values and status.</b> A sample's value is its numeric value (Double), else its string value, else Empty; its
@@ -109,7 +110,29 @@ internal static class RawAggregateReader
     /// <param name="maxWindowRows">The configured cap (&lt;= 0 = unbounded).</param>
     /// <returns>The row limit to request.</returns>
     public static int MaxValuesFor(int maxWindowRows) =>
-        maxWindowRows <= 0 ? int.MaxValue : (int)Math.Min(int.MaxValue, (long)maxWindowRows + 2 + PreWindowAllowance);
+        maxWindowRows <= 0 ? int.MaxValue : ClampToGatewayCeiling((long)maxWindowRows + 2 + PreWindowAllowance, maxWindowRows);
+
+    /// <summary>
+    ///     The HistorianGateway's default <c>Historian:ReadLimits:MaxValuesCeiling</c>. With
+    ///     <c>AllowStreamingBeyondCeiling=false</c> the gateway refuses any <c>max_values</c> above its ceiling.
+    /// </summary>
+    internal const int GatewayDefaultMaxValuesCeiling = 1_000_000;
+
+    /// <summary>
+    ///     #688: a <c>ReadRaw</c> row budget of <paramref name="rows"/> plus this backend's own allowance
+    ///     (<paramref name="requested"/>) clamped at <see cref="GatewayDefaultMaxValuesCeiling"/> — but never below
+    ///     <paramref name="rows"/> itself. The allowance (the boundary row, the margin, the completeness row) must not push a
+    ///     budget that fits the gateway's default ceiling over it: the default <c>MaxTimeWeightedRawRows</c> (1,000,000) + 18
+    ///     used to exceed it, so a gateway with <c>AllowStreamingBeyondCeiling=false</c> refused EVERY per-sample read. A
+    ///     count above the ceiling is the operator's (or client's) deliberate choice for a streaming gateway and keeps its
+    ///     allowance. The cost of clamping is that a window holding within the allowance of the ceiling cannot be proven
+    ///     complete and fails as over the cap — the same outcome it would get one row later.
+    /// </summary>
+    /// <param name="requested">The budget including the allowance.</param>
+    /// <param name="rows">The caller's own row count (the cap, or <c>NumValuesPerNode</c>), &gt; 0.</param>
+    /// <returns>The <c>max_values</c> to request.</returns>
+    internal static int ClampToGatewayCeiling(long requested, long rows) =>
+        (int)Math.Min(int.MaxValue, rows > GatewayDefaultMaxValuesCeiling ? requested : Math.Min(requested, GatewayDefaultMaxValuesCeiling));
 
     /// <summary>Maps one gateway raw sample to a stored sample (value + mapped status, Empty clamped Bad).</summary>
     /// <param name="sample">The gateway sample; its timestamp must be set.</param>

tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Gateway.Tests/GatewayRawAggregateTests.cs

@@ -369,7 +369,7 @@ public sealed class GatewayRawAggregateTests
         await Source(count).ReadProcessedAsync("A", At(0), At(80), TimeSpan.FromSeconds(20), HistoryAggregateType.Count, Ct);
         // The window read ends AT the window end; probe + look-ahead are separate 2-row reads, all consumed to completion.
         count.RawCalls.ShouldBe([
-            (At(0) - TimeSpan.FromMilliseconds(1), At(80), 1_000_000 + 2 + 16),
+            (At(0) - TimeSpan.FromMilliseconds(1), At(80), 1_000_000), // cap + 18, clamped at the gateway default ceiling (#688)
             (DateTime.UnixEpoch, At(0).AddTicks(-1), 2),
             (At(80), Now + TimeSpan.FromDays(1), 2),
         ]);
@@ -473,6 +473,36 @@ public sealed class GatewayRawAggregateTests
         ds2.GetHealthSnapshot().TotalFailures.ShouldBe(1);
     }
 
+    [Fact] // #688: the default cap's budget fits the gateway's default ceiling, so a strict gateway serves the default config
+    public async Task The_default_caps_row_budget_does_not_exceed_the_gateways_default_ceiling()
+    {
+        var fake = Store([D(-5, 1), D(5, 2)], avevaBoundary: true);
+        await Source(fake).ReadProcessedAsync("T", At(0), At(10), TimeSpan.Zero, HistoryAggregateType.Average, Ct);
+        TimeWeightedAggregator.DefaultMaxWindowRows.ShouldBe(1_000_000);
+        fake.RawCalls[0].MaxValues.ShouldBe(RawAggregateReader.GatewayDefaultMaxValuesCeiling);
+    }
+
+    [Theory] // #688: the + 18 allowance is clamped at the gateway's default ceiling, never below the cap itself
+    [InlineData(10, 28)]
+    [InlineData(999_982, 1_000_000)]
+    [InlineData(999_990, 1_000_000)]
+    [InlineData(1_000_000, 1_000_000)]
+    [InlineData(2_000_000, 2_000_018)] // a cap above the default ceiling is a deliberate choice for a streaming gateway
+    [InlineData(0, int.MaxValue)]
+    [InlineData(-1, int.MaxValue)]
+    public void The_window_row_budget_is_clamped_at_the_gateways_default_ceiling(int cap, int expected) =>
+        RawAggregateReader.MaxValuesFor(cap).ShouldBe(expected);
+
+    [Fact] // #688: near the ceiling the window is still served when it is not full
+    public async Task A_cap_at_the_ceiling_still_serves_a_window_below_it()
+    {
+        var data = Enumerable.Range(0, 10).Select(i => D(i, i)).ToArray();
+        var fake = Store(data, avevaBoundary: true);
+        var r = await Source(fake, maxRawRows: 1_000_000)
+            .ReadProcessedAsync("T", At(0), At(10), TimeSpan.Zero, HistoryAggregateType.Average, Ct);
+        r.Samples.Single().Value.ShouldBe(4.5);
+    }
+
     [Fact]
     public async Task A_caller_cancellation_is_not_counted_as_a_backend_failure()
     {