diff --git a/CLAUDE.md b/CLAUDE.md index 9dc3ac84..1b49cc92 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1353,7 +1353,7 @@ historian tagname, which is the tag's driver `FullName`. - **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; per-sample aggregates (10, simple definitions) are pushed down to T-SQL with bucket math in **100 ns ticks**; the Part 13 time-weighted seven (TimeAverage/TimeAverage2/Total/Total2/Interpolative, #671 — ⚠️ `Total` is now the time-weighted integral, no longer `SUM` — plus the **duration-based** PercentGood/PercentBad, #680) stream window + 4 bound rows into the shared `TimeWeightedAggregator` (both SQL backends, identical results; reverse `Interpolative` anchored at the later time; window capped by `ServerHistorian:MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations`, percentages included). ⚠️ **#680 status rules (client-visible):** TimeAverage/Total are Uncertain if any part of the interval uses an Uncertain value, is bridged across a Bad sample, is extrapolated past the last sample (⇒ a live tag's latest interval is Uncertain) or is not covered; TimeAverage2/Total2 keep the SDK time-based rule (Uncertain time counts as Good) but never extrapolate past the last sample — that interval is Partial, later ones `BadNoData` (a report-by-exception tag with nothing after its last sample reads `BadNoData` until its next sample arrives). `docs/Historian.md` §"Time-weighted aggregates". ⚠️ **#680 per-sample (client-visible):** `Count` counts **Good** samples only (value-based status; an all-Bad interval is generic Bad, value null) and an empty interval **inside the tag's data range** is `0`/Good (outside it still `BadNoData`); `Start`/`End` return the **raw** first/last sample of any quality — its own timestamp and status, value null when Bad; `Delta` is Uncertain only when the interval's first or last sample is unusable; ⚠️ #684: `Average`/`Minimum`/`Maximum`/`Range`/`*ActualTime` use **Good samples only** (was Good + Uncertain; an Uncertain-only interval is `BadNoData`), Min/Max-family status Uncertain on a Bad sample, an Uncertain value beyond the Good extreme (Part 13 — the SDK ignores that) or an Uncertain non-numeric sample, and only a **Good** String/DateTime sample refuses them (`BadAggregateInvalidInputs`; Delta still refuses any usable one); and every per-sample aggregate but `Average` sets **end-of-data Partial** on the interval holding the last sample (forward incl. its earlier edge, reverse strictly inside), never on a Bad result. The data-range lookup is a key seek on SQL Server and a `tag` symbol-index seek on QuestDB (#683 — `{p}sample.tag` is `SYMBOL INDEX`; the bootstrapper ADDs it to a pre-#683 table, idempotent), read BEFORE the bucket statement so a racing append can never yield a Good `Count` of 0 (a concurrent HistoryUpdate delete is the mirror case and still can, transiently). `docs/Historian.md` §"Aggregates". Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. **Retention (#667):** opt-in `SqlServer:RetentionDays` (+ `PurgeInterval` 1h, `PurgeBatchSize` 4000) -> `HistorianRetentionService` purges old `hist.Sample` (per tag) / `hist.Event` in `DELETE TOP(n)` batches, Primary-gated on `IHistorizationRoleView`, one purger per DB via `sp_getapplock`; counter `otopcua.historian.retention.purged{table}`. Optional `IHistorianBackendProvider.CreateRetention` (default null). `docs/Historian.md` §Retention. - **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend". - ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); on forward reads Partial also marks the interval the data starts inside (not Average — the SDK's rule, forward-only); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates. -- ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` serves it from FORWARD reads: uncapped ⇒ ONE forward read reversed in memory; capped ⇒ a bounded walk back in segments (ms-aligned request bounds, width aimed from observed density at ≈ N + 8 rows and carried page to page in the continuation point — #689: it used to aim at half `MaxTieClusterOverfetch`, ~32k rows per page on a dense tag; a read asks for `2N + 18` rows (a discovery segment ≤ 1,026), `MaxTieClusterOverfetch` is only the tie-cluster ceiling — a full read is re-read, never answered truncated; only one timestamp holding more than max(1, `MaxTieClusterOverfetch`) samples ⇒ `BadTooManyOperations`; walks no later than now + 1 day and never before the Unix epoch; #691: after two empty segments it asks once for the whole remainder, and the stretch a page proved empty rides the continuation point — the last pages of a deep page-through no longer each walk ~14 reads to 1970). The raw/events tie-path oversized-cluster refusal is now `BadTooManyOperations` too (was `BadHistoryOperationUnsupported`). Uncapped Gateway raw reads (forward too) used to send `max_values = 0`, which the client/gateway reject — now unbounded (`int.MaxValue`; a ceiling refusal ⇒ `BadTooManyOperations`), and a cancelled read is no longer a backend failure. Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events follow since #691.4: a reverse/end-only capped events read on Gateway (`SupportsReverseEvents` false; Null now reports `true` and answers empty) is walked back the same way (ties sorted by EventId — the gateway orders by time only; one extra event found ⇒ continuation point) — it used to be refused `BadHistoryOperationUnsupported` (explicit reverse) or answered with the OLDEST N ascending (end-only); needs the gateway's `RuntimeDb:EventReadMaxRows` 0/unlimited (it truncates silently). Uncapped Gateway events reads (forward too) used to throw "Array dimensions exceeded" (result list pre-sized to `int.MaxValue`) — fixed. Above a truncated read the walk gallops up from the newest row seen, then bisects (never below 1 ms): a burst that ended 100 days ago ≈ 18 reads, a dense tag that stopped days ago ≈ 26. `docs/Historian.md` §"Open-ended and reverse raw reads". +- ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` serves it from FORWARD reads: uncapped ⇒ ONE forward read reversed in memory; capped ⇒ a bounded walk back in segments (ms-aligned request bounds, width aimed from observed density at ≈ N + 8 rows and carried page to page in the continuation point — #689: it used to aim at half `MaxTieClusterOverfetch`, ~32k rows per page on a dense tag; a read asks for `2N + 18` rows (a discovery segment ≤ 1,026), `MaxTieClusterOverfetch` is only the tie-cluster ceiling — a full read is re-read, never answered truncated; only more than max(1, `MaxTieClusterOverfetch`) samples within ONE MILLISECOND ⇒ `BadTooManyOperations` (per ms, not per timestamp — requests are ms-rounded, and letting a segment narrow below 1 ms stepped a page through one ms a tick per read, ~10k reads); walks no later than now + 1 day and never before the Unix epoch; #691: after two empty segments it asks once for the whole remainder, and the stretch a page proved empty rides the continuation point — the last pages of a deep page-through no longer each walk ~14 reads to 1970). The raw/events tie-path oversized-cluster refusal is now `BadTooManyOperations` too (was `BadHistoryOperationUnsupported`). Uncapped Gateway raw reads (forward too) used to send `max_values = 0`, which the client/gateway reject — now unbounded (`int.MaxValue`; a ceiling refusal ⇒ `BadTooManyOperations`), and a cancelled read is no longer a backend failure. Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events follow since #691.4: a reverse/end-only capped events read on Gateway (`SupportsReverseEvents` false; Null now reports `true` and answers empty) is walked back the same way (ties sorted by EventId — the gateway orders by time only; one extra event found ⇒ continuation point) — it used to be refused `BadHistoryOperationUnsupported` (explicit reverse) or answered with the OLDEST N ascending (end-only); needs the gateway's `RuntimeDb:EventReadMaxRows` 0/unlimited (it truncates silently). Uncapped Gateway events reads (forward too) used to throw "Array dimensions exceeded" (result list pre-sized to `int.MaxValue`) — fixed. Above a truncated read the walk gallops up from the newest row seen, then bisects (never below 1 ms): a burst that ended 100 days ago ≈ 18 reads, a dense tag that stopped days ago ≈ 26. `docs/Historian.md` §"Open-ended and reverse raw reads". - **Advertised aggregates (#682):** `IHistorianDataSource.SupportedAggregates` (Gateway 4 via `GatewayHistorianDataSource.ServedAggregates`, SqlServer/QuestDB 17 via `AggregateFamilies.Served`, Null **none**; a null set ⇒ all) is the one authority for `HistoryServerCapabilities/AggregateFunctions` (i=11201 — only its folder references are trimmed) and the up-front `BadAggregateNotSupported` in HistoryRead-Processed. ⚠️ `ServerCapabilities/AggregateFunctions` (i=2997) is deliberately NOT trimmed: it keeps the SDK's 37 because live `AggregateFilter` subscriptions still compute them all. ⚠️ Historian disabled ⇒ Processed is now `BadAggregateNotSupported`, not `GoodNoData`. `docs/Historian.md` §Advertised aggregates. - **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good. - **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node. diff --git a/docs/Historian.md b/docs/Historian.md index 553768c6..40aae51a 100644 --- a/docs/Historian.md +++ b/docs/Historian.md @@ -659,9 +659,13 @@ backend: that row (`(row + d, top]`, `d` ×4 per step, following each truncated read) until a read comes back empty, then a **bisection** of the bracket — never narrower than the 1 ms request granularity, which only re-sent the same request. Measured: a dense tag that stopped two days ago ≈ **26** reads, a burst of just over 1,024 rows that ended 100 days ago - ≈ **18** (was 46 bisecting the whole gap) — rather than re-reading the oldest rows of a burst over and over. Only **one timestamp holding more than `M` samples** is refused, + ≈ **18** (was 46 bisecting the whole gap) — rather than re-reading the oldest rows of a burst over and over. Only **more than `M` samples within one millisecond** are refused, **`BadTooManyOperations`** (a request limit, not a backend fault); a cluster of exactly `M`, or neighbouring - clusters each within `M`, is served. + clusters each within `M` and a millisecond or more apart, is served. ⚠️ The unit is the **millisecond**, not the + timestamp (#691.4 re-review): requests are rounded to the millisecond, so a narrower segment is the same request — + letting the width fall below it once stepped a page through one millisecond a tick at a time (~10,000 reads). Exact + for the Gateway, which stores milliseconds; a sub-millisecond backend walked this way would refuse more than `M` + samples inside one millisecond even when no single timestamp holds that many. - **Proving the rest empty (#691).** Walking down through empty history to the floor used to cost one read per ×4 growth step — ~14–16 reads (~2.5 s live) on **every** last page of a deep page-through. Now, after two consecutive empty segments, the walk asks **once per page** for the whole remainder down to the floor at the page budget: a diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/ReverseEventsReader.cs b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/ReverseEventsReader.cs index f19648d3..2a7cc98c 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/ReverseEventsReader.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/ReverseEventsReader.cs @@ -44,7 +44,7 @@ internal static class ReverseEventsReader ev => ev.EventTimeUtc.Ticks, EventIdTieOrder.Instance, hi, lo, want, budget, source.TimestampResolutionTicks, nowUtc, hint, at => new HistoryReadLimitExceededException(budget, - $"A reverse events read of '{sourceName}' at {at:O} found more than {budget} events at one timestamp; the " + + $"A reverse events read of '{sourceName}' at {at:O} found more than {budget} events within one millisecond; the " + "backend reads forward only, so the latest of them cannot be selected. Increase " + "ServerHistorian:MaxTieClusterOverfetch."), ct) diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/ReverseRawReader.cs b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/ReverseRawReader.cs index 309e9e8b..6f3d3902 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/ReverseRawReader.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/ReverseRawReader.cs @@ -71,7 +71,7 @@ internal static class ReverseRawReader }, Ticks, tieOrder: null, hi, lo, maxValues, budget, source.TimestampResolutionTicks, nowUtc, hint, at => new HistoryReadLimitExceededException(budget, - $"A reverse raw read of '{tagname}' at {at:O} found more than {budget} samples at one timestamp; the " + + $"A reverse raw read of '{tagname}' at {at:O} found more than {budget} samples within one millisecond; the " + "backend reads forward only, so the latest of them cannot be selected. Increase " + "ServerHistorian:MaxTieClusterOverfetch."), ct) diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/ReverseWalk.cs b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/ReverseWalk.cs index 91e18537..014c760d 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/ReverseWalk.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/ReverseWalk.cs @@ -54,8 +54,9 @@ internal readonly record struct ForwardSegmentRead(IReadOnlyList Rows, boo /// with room for them (at most twice the budget + 4); otherwise the segment is narrowed — to the density the truncated /// read showed, and never again below the newest row it saw (known: rows exist there, so walking past it before /// finding what lies above it would re-read the same dense stretch over and over; the gap above it is bisected -/// instead). Only a single timestamp holding more than M rows — a one-resolution segment that still does not -/// fit at the tie budget — is refused. +/// instead). Only more than M rows within one millisecond — a segment no wider than the 1 ms request +/// granularity that still does not fit at the tie budget — is refused (per millisecond, not per timestamp: narrower +/// requests are the same request, and the Gateway stores milliseconds). /// /// /// Proving the rest empty (#691). Walking down to the floor through empty history used to take one read per ×4 @@ -194,12 +195,13 @@ internal static class ReverseWalk var gap = segHi - known; lowered = lowerBy > 0 && lowerBy < gap; bisected = !lowered && (bisecting || lowerBy > 0) && gap > 2 * Math.Max(granularity, knownWidth); - // Never narrower than the request granularity when halving: requests are rounded outward to it, so a - // narrower half re-sends the same request (#691.4 review). Keep the Min: a truncated read crowded by rows - // before the segment must still shrink the segment. + // Never narrower than the request granularity: requests are rounded outward to it, so a narrower segment + // re-sends the same request — and a width left at one tick (a tie cluster's) stepped segHi down a tick per + // read, ~10,000 identical reads (#691.4 re-review). Keep the Min: a truncated read crowded by rows before + // the segment must still shrink the segment. w = lowered ? gap - lowerBy : bisected ? gap / 2 - : Math.Min(w, gap <= 2 * granularity ? gap : gap / 2); + : Math.Min(Math.Max(w, granularity), gap <= 2 * granularity ? gap : gap / 2); } var segLo = segHi - stopTicks <= w ? stopTicks : segHi - w; var start = Utc(AlignDown(segLo, granularity)); @@ -248,10 +250,12 @@ internal static class ReverseWalk continue; } - if (segHi - segLo <= resolution) + if (segHi - segLo <= granularity) { - // A one-resolution segment still over its budget: one timestamp's tie cluster. Escalate to the tie - // budget M before refusing — a cluster of up to M rows is served (#686). + // A segment no wider than the request granularity (1 ms — narrower requests are the same request) still + // over its budget: a tie cluster, or a burst inside one millisecond. Escalate to the tie budget M before + // refusing — up to M rows within one millisecond are served (#686); more are refused, per millisecond + // (#691.4 re-review — exact for the millisecond-resolution Gateway). if (segBudget < budget) { segBudget = budget; diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadEventsPagingTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadEventsPagingTests.cs index 89be1c67..dc0ecf85 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadEventsPagingTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadEventsPagingTests.cs @@ -446,20 +446,38 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable var expected = events.OrderByDescending(e => e.EventTimeUtc).ThenByDescending(e => e.EventId, StringComparer.Ordinal) .Select(e => e.EventId).ToList(); + // The walk refuses more than MaxTieClusterOverfetch events within ONE MILLISECOND (#691.4 re-review): with the + // small budget a dense sub-millisecond burst is a legitimate BadTooManyOperations — and nothing else is. + var times = events.Select(e => e.EventTimeUtc.Ticks).Order().ToList(); + var msMax = 0; + for (int i = 0, j = 0; i < times.Count; i++) + { + while (times[i] - times[j] >= TimeSpan.TicksPerMillisecond) j++; + msMax = Math.Max(msMax, i - j + 1); + } + var all = new List(); byte[]? cp = null; var pages = 0; + var refused = false; do { var r = ReadEvents(nm, node, max, cp, end, DateTime.MinValue); - r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good, $"seed {seed}"); + if (r.Error.StatusCode.Code == StatusCodes.BadTooManyOperations && msMax > nm.MaxTieClusterOverfetch) + { + refused = true; + break; + } + + r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good, $"seed {seed}, ms max {msMax}"); all.AddRange(r.Ids); cp = r.Cp; (++pages).ShouldBeLessThan(2000); } while (cp is not null); - all.ShouldBe(expected, $"seed {seed}, N = {max}"); + if (refused) all.ShouldBe(expected.Take(all.Count), $"seed {seed}: what came before the refusal"); + else all.ShouldBe(expected, $"seed {seed}, N = {max}"); } await host.DisposeAsync(); diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadPagingTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadPagingTests.cs index b25afb52..747ac185 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadPagingTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadPagingTests.cs @@ -731,9 +731,38 @@ public sealed class NodeManagerHistoryReadPagingTests : IDisposable var max = (uint)rng.Next(1, 12); var end = new DateTime(t + rng.Next(0, 3) * TimeSpan.TicksPerSecond, DateTimeKind.Utc); - var collected = await DrainAsync(nm, nodeId, DateTime.MinValue, end, max, maxPages: 2000); + // The walk refuses more than MaxTieClusterOverfetch samples within ONE MILLISECOND (#691.4 re-review), so with + // the small budget a dense sub-millisecond burst is a legitimate BadTooManyOperations — and nothing else is. + var msMax = 0; + for (int i = 0, j = 0; i < series.Count; i++) + { + while (series[i].SourceTimestampUtc!.Value.Ticks - series[j].SourceTimestampUtc!.Value.Ticks >= TimeSpan.TicksPerMillisecond) j++; + msMax = Math.Max(msMax, i - j + 1); + } - collected.ShouldBe(Enumerable.Range(0, series.Count).Reverse().Select(i => (double)i), $"seed {seed}, N = {max}"); + var collected = new List(); + byte[]? cp = null; + var pages = 0; + var refused = false; + do + { + var (values, error, next) = ReadRaw(nm, nodeId, DateTime.MinValue, end, max, inboundCp: cp); + if (error.StatusCode.Code == StatusCodes.BadTooManyOperations && msMax > nm.MaxTieClusterOverfetch) + { + refused = true; + break; + } + + error.StatusCode.Code.ShouldBe(StatusCodes.Good, $"seed {seed}, N = {max}, ms max {msMax}"); + collected.AddRange(values); + cp = next; + (++pages).ShouldBeLessThan(2000, "paging must terminate, not loop"); + } + while (cp is not null); + + var expected = Enumerable.Range(0, series.Count).Reverse().Select(i => (double)i).ToList(); + if (refused) collected.ShouldBe(expected.Take(collected.Count), $"seed {seed}: what came before the refusal"); + else collected.ShouldBe(expected, $"seed {seed}, N = {max}"); } await host.DisposeAsync(); diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/ReverseWalkTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/ReverseWalkTests.cs index cfc3193e..12bf4b1a 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/ReverseWalkTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/ReverseWalkTests.cs @@ -32,7 +32,14 @@ public sealed class ReverseWalkTests for (var seed = 0; seed < 400; seed++) { var rng = new Random(seed); - var series = RandomSeries(rng, out var maxCluster); + var series = RandomSeries(rng, out _); + // The refusal unit is one millisecond (#691.4 re-review): the most rows inside any 1 ms window. + var maxCluster = 0; + for (int i = 0, j = 0; i < series.Count; i++) + { + while (Ts(series[i]) - Ts(series[j]) >= TimeSpan.TicksPerMillisecond) j++; + maxCluster = Math.Max(maxCluster, i - j + 1); + } var source = new AvevaShapedSource(series, roundUp: rng.Next(2) == 0); var tieBudget = rng.Next(4) switch { 0 => 1, 1 => 3, 2 => 8, _ => 65536 }; var want = (uint)rng.Next(1, 40); @@ -81,7 +88,7 @@ public sealed class ReverseWalkTests } catch (HistoryReadLimitExceededException) { - if (maxCluster <= tieBudget) failures.Add($"seed {seed}: refused with no cluster over {tieBudget} (max {maxCluster})"); + if (maxCluster <= tieBudget) failures.Add($"seed {seed}: refused with no millisecond over {tieBudget} rows (max {maxCluster})"); break; } } @@ -206,6 +213,33 @@ public sealed class ReverseWalkTests source.Reads.ShouldBeLessThanOrEqualTo(22, string.Join("\n", source.Log)); } + /// + /// #691.4 re-review (S8): a tie cluster under a carried width. 30 rows at one whole-millisecond timestamp 0.3 s below + /// hi, nothing above, N = 1, a carried 1 s width. The truncated read of the cluster left the width at one tick; + /// once the gap above it was under two milliseconds nothing floored it, so segHi stepped down one tick per read — + /// 10,006 reads of the same millisecond (28 before the gallop). + /// + [Theory] + [InlineData(30, 1, 0.3, 1.0)] + [InlineData(34, 1, 0.52, 0.48)] + [InlineData(25, 1, 1.2, 1.0)] + [InlineData(40, 3, 3.5, 2.0)] + public async Task A_tie_cluster_under_a_carried_width_is_read_in_a_bounded_number_of_reads( + int ties, int want, double secondsBelow, double hintSeconds) + { + var at = T0.Ticks - (long)(secondsBelow * TimeSpan.TicksPerSecond); + at -= at % TimeSpan.TicksPerMillisecond; + var series = Enumerable.Range(0, ties).Select(i => At(at, i)).ToList(); + var source = new AvevaShapedSource(series); + + var page = await ReverseRawReader.ReadAsync(source, "T", T0, DateTime.MinValue, (uint)want, 65536, T0, + new ReverseWalkHint((long)(hintSeconds * TimeSpan.TicksPerSecond)), Ct); + + page.Samples.Select(s => (double)s.Value!).ShouldBe(Enumerable.Range(ties - want, want).Reverse().Select(i => (double)i)); + source.Reads.ShouldBeLessThanOrEqualTo(30, string.Join("\n", source.Log.Take(40))); + source.RepeatedRequests.ShouldBe(0); + } + private static long Ts(DataValueSnapshot s) => s.SourceTimestampUtc!.Value.Ticks; /// Random ascending series: bursts, gaps, sub-millisecond stamps and tie clusters.