fix(historian): #691.4 re-review — never narrow a walk segment below the 1 ms request granularity

A tie cluster's truncated read left the width at one tick; once the gap
above it was under two milliseconds nothing floored it, so segHi
stepped down one tick per read: 30 rows at one ms 0.3 s below hi with a
carried 1 s width took 10,006 identical reads (S8). (a) The top-anchored
width is floored at the granularity, Min kept; (b) a segment no wider
than the granularity is the unit that escalates to
MaxTieClusterOverfetch and then refuses - per millisecond, exact for the
ms-resolution Gateway (documented). Both are needed: the floor alone
loops on crowded reads with a small tie budget. S8 is a repo test
(4 shapes, reads <= 30, 0 repeated requests); random-data tests now
measure the refusal unit as rows in any 1 ms window. Reviewer harness,
5,000 seeds x default/MSONLY/CROWD: worst page 70 reads (default/MSONLY),
63 (CROWD) at M=65536, p99.9 27-37; no page over 500 reads at any tie
budget.
This commit is contained in:
Joseph Doherty
2026-10-12 17:50:39 -04:00
parent 574c3aa1b0
commit 9bb1363cd6
8 changed files with 109 additions and 20 deletions
+1 -1
View File
@@ -1353,7 +1353,7 @@ historian tagname, which is the tag's driver `FullName`.
- **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; per-sample aggregates (10, simple definitions) are pushed down to T-SQL with bucket math in **100 ns ticks**; the Part 13 time-weighted seven (TimeAverage/TimeAverage2/Total/Total2/Interpolative, #671 — ⚠️ `Total` is now the time-weighted integral, no longer `SUM` — plus the **duration-based** PercentGood/PercentBad, #680) stream window + 4 bound rows into the shared `TimeWeightedAggregator` (both SQL backends, identical results; reverse `Interpolative` anchored at the later time; window capped by `ServerHistorian:MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations`, percentages included). ⚠️ **#680 status rules (client-visible):** TimeAverage/Total are Uncertain if any part of the interval uses an Uncertain value, is bridged across a Bad sample, is extrapolated past the last sample (⇒ a live tag's latest interval is Uncertain) or is not covered; TimeAverage2/Total2 keep the SDK time-based rule (Uncertain time counts as Good) but never extrapolate past the last sample — that interval is Partial, later ones `BadNoData` (a report-by-exception tag with nothing after its last sample reads `BadNoData` until its next sample arrives). `docs/Historian.md` §"Time-weighted aggregates". ⚠️ **#680 per-sample (client-visible):** `Count` counts **Good** samples only (value-based status; an all-Bad interval is generic Bad, value null) and an empty interval **inside the tag's data range** is `0`/Good (outside it still `BadNoData`); `Start`/`End` return the **raw** first/last sample of any quality — its own timestamp and status, value null when Bad; `Delta` is Uncertain only when the interval's first or last sample is unusable; ⚠️ #684: `Average`/`Minimum`/`Maximum`/`Range`/`*ActualTime` use **Good samples only** (was Good + Uncertain; an Uncertain-only interval is `BadNoData`), Min/Max-family status Uncertain on a Bad sample, an Uncertain value beyond the Good extreme (Part 13 — the SDK ignores that) or an Uncertain non-numeric sample, and only a **Good** String/DateTime sample refuses them (`BadAggregateInvalidInputs`; Delta still refuses any usable one); and every per-sample aggregate but `Average` sets **end-of-data Partial** on the interval holding the last sample (forward incl. its earlier edge, reverse strictly inside), never on a Bad result. The data-range lookup is a key seek on SQL Server and a `tag` symbol-index seek on QuestDB (#683 — `{p}sample.tag` is `SYMBOL INDEX`; the bootstrapper ADDs it to a pre-#683 table, idempotent), read BEFORE the bucket statement so a racing append can never yield a Good `Count` of 0 (a concurrent HistoryUpdate delete is the mirror case and still can, transiently). `docs/Historian.md` §"Aggregates". Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. **Retention (#667):** opt-in `SqlServer:RetentionDays` (+ `PurgeInterval` 1h, `PurgeBatchSize` 4000) -> `HistorianRetentionService` purges old `hist.Sample` (per tag) / `hist.Event` in `DELETE TOP(n)` batches, Primary-gated on `IHistorizationRoleView`, one purger per DB via `sp_getapplock`; counter `otopcua.historian.retention.purged{table}`. Optional `IHistorianBackendProvider.CreateRetention` (default null). `docs/Historian.md` §Retention. - **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; per-sample aggregates (10, simple definitions) are pushed down to T-SQL with bucket math in **100 ns ticks**; the Part 13 time-weighted seven (TimeAverage/TimeAverage2/Total/Total2/Interpolative, #671 — ⚠️ `Total` is now the time-weighted integral, no longer `SUM` — plus the **duration-based** PercentGood/PercentBad, #680) stream window + 4 bound rows into the shared `TimeWeightedAggregator` (both SQL backends, identical results; reverse `Interpolative` anchored at the later time; window capped by `ServerHistorian:MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations`, percentages included). ⚠️ **#680 status rules (client-visible):** TimeAverage/Total are Uncertain if any part of the interval uses an Uncertain value, is bridged across a Bad sample, is extrapolated past the last sample (⇒ a live tag's latest interval is Uncertain) or is not covered; TimeAverage2/Total2 keep the SDK time-based rule (Uncertain time counts as Good) but never extrapolate past the last sample — that interval is Partial, later ones `BadNoData` (a report-by-exception tag with nothing after its last sample reads `BadNoData` until its next sample arrives). `docs/Historian.md` §"Time-weighted aggregates". ⚠️ **#680 per-sample (client-visible):** `Count` counts **Good** samples only (value-based status; an all-Bad interval is generic Bad, value null) and an empty interval **inside the tag's data range** is `0`/Good (outside it still `BadNoData`); `Start`/`End` return the **raw** first/last sample of any quality — its own timestamp and status, value null when Bad; `Delta` is Uncertain only when the interval's first or last sample is unusable; ⚠️ #684: `Average`/`Minimum`/`Maximum`/`Range`/`*ActualTime` use **Good samples only** (was Good + Uncertain; an Uncertain-only interval is `BadNoData`), Min/Max-family status Uncertain on a Bad sample, an Uncertain value beyond the Good extreme (Part 13 — the SDK ignores that) or an Uncertain non-numeric sample, and only a **Good** String/DateTime sample refuses them (`BadAggregateInvalidInputs`; Delta still refuses any usable one); and every per-sample aggregate but `Average` sets **end-of-data Partial** on the interval holding the last sample (forward incl. its earlier edge, reverse strictly inside), never on a Bad result. The data-range lookup is a key seek on SQL Server and a `tag` symbol-index seek on QuestDB (#683 — `{p}sample.tag` is `SYMBOL INDEX`; the bootstrapper ADDs it to a pre-#683 table, idempotent), read BEFORE the bucket statement so a racing append can never yield a Good `Count` of 0 (a concurrent HistoryUpdate delete is the mirror case and still can, transiently). `docs/Historian.md` §"Aggregates". Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. **Retention (#667):** opt-in `SqlServer:RetentionDays` (+ `PurgeInterval` 1h, `PurgeBatchSize` 4000) -> `HistorianRetentionService` purges old `hist.Sample` (per tag) / `hist.Event` in `DELETE TOP(n)` batches, Primary-gated on `IHistorizationRoleView`, one purger per DB via `sp_getapplock`; counter `otopcua.historian.retention.purged{table}`. Optional `IHistorianBackendProvider.CreateRetention` (default null). `docs/Historian.md` §Retention.
- **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend". - **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend".
- ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); on forward reads Partial also marks the interval the data starts inside (not Average — the SDK's rule, forward-only); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates. - ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); on forward reads Partial also marks the interval the data starts inside (not Average — the SDK's rule, forward-only); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates.
- ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` serves it from FORWARD reads: uncapped ⇒ ONE forward read reversed in memory; capped ⇒ a bounded walk back in segments (ms-aligned request bounds, width aimed from observed density at ≈ N + 8 rows and carried page to page in the continuation point — #689: it used to aim at half `MaxTieClusterOverfetch`, ~32k rows per page on a dense tag; a read asks for `2N + 18` rows (a discovery segment ≤ 1,026), `MaxTieClusterOverfetch` is only the tie-cluster ceiling — a full read is re-read, never answered truncated; only one timestamp holding more than max(1, `MaxTieClusterOverfetch`) samples ⇒ `BadTooManyOperations`; walks no later than now + 1 day and never before the Unix epoch; #691: after two empty segments it asks once for the whole remainder, and the stretch a page proved empty rides the continuation point — the last pages of a deep page-through no longer each walk ~14 reads to 1970). The raw/events tie-path oversized-cluster refusal is now `BadTooManyOperations` too (was `BadHistoryOperationUnsupported`). Uncapped Gateway raw reads (forward too) used to send `max_values = 0`, which the client/gateway reject — now unbounded (`int.MaxValue`; a ceiling refusal ⇒ `BadTooManyOperations`), and a cancelled read is no longer a backend failure. Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events follow since #691.4: a reverse/end-only capped events read on Gateway (`SupportsReverseEvents` false; Null now reports `true` and answers empty) is walked back the same way (ties sorted by EventId — the gateway orders by time only; one extra event found ⇒ continuation point) — it used to be refused `BadHistoryOperationUnsupported` (explicit reverse) or answered with the OLDEST N ascending (end-only); needs the gateway's `RuntimeDb:EventReadMaxRows` 0/unlimited (it truncates silently). Uncapped Gateway events reads (forward too) used to throw "Array dimensions exceeded" (result list pre-sized to `int.MaxValue`) — fixed. Above a truncated read the walk gallops up from the newest row seen, then bisects (never below 1 ms): a burst that ended 100 days ago ≈ 18 reads, a dense tag that stopped days ago ≈ 26. `docs/Historian.md` §"Open-ended and reverse raw reads". - ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` serves it from FORWARD reads: uncapped ⇒ ONE forward read reversed in memory; capped ⇒ a bounded walk back in segments (ms-aligned request bounds, width aimed from observed density at ≈ N + 8 rows and carried page to page in the continuation point — #689: it used to aim at half `MaxTieClusterOverfetch`, ~32k rows per page on a dense tag; a read asks for `2N + 18` rows (a discovery segment ≤ 1,026), `MaxTieClusterOverfetch` is only the tie-cluster ceiling — a full read is re-read, never answered truncated; only more than max(1, `MaxTieClusterOverfetch`) samples within ONE MILLISECOND ⇒ `BadTooManyOperations` (per ms, not per timestamp — requests are ms-rounded, and letting a segment narrow below 1 ms stepped a page through one ms a tick per read, ~10k reads); walks no later than now + 1 day and never before the Unix epoch; #691: after two empty segments it asks once for the whole remainder, and the stretch a page proved empty rides the continuation point — the last pages of a deep page-through no longer each walk ~14 reads to 1970). The raw/events tie-path oversized-cluster refusal is now `BadTooManyOperations` too (was `BadHistoryOperationUnsupported`). Uncapped Gateway raw reads (forward too) used to send `max_values = 0`, which the client/gateway reject — now unbounded (`int.MaxValue`; a ceiling refusal ⇒ `BadTooManyOperations`), and a cancelled read is no longer a backend failure. Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events follow since #691.4: a reverse/end-only capped events read on Gateway (`SupportsReverseEvents` false; Null now reports `true` and answers empty) is walked back the same way (ties sorted by EventId — the gateway orders by time only; one extra event found ⇒ continuation point) — it used to be refused `BadHistoryOperationUnsupported` (explicit reverse) or answered with the OLDEST N ascending (end-only); needs the gateway's `RuntimeDb:EventReadMaxRows` 0/unlimited (it truncates silently). Uncapped Gateway events reads (forward too) used to throw "Array dimensions exceeded" (result list pre-sized to `int.MaxValue`) — fixed. Above a truncated read the walk gallops up from the newest row seen, then bisects (never below 1 ms): a burst that ended 100 days ago ≈ 18 reads, a dense tag that stopped days ago ≈ 26. `docs/Historian.md` §"Open-ended and reverse raw reads".
- **Advertised aggregates (#682):** `IHistorianDataSource.SupportedAggregates` (Gateway 4 via `GatewayHistorianDataSource.ServedAggregates`, SqlServer/QuestDB 17 via `AggregateFamilies.Served`, Null **none**; a null set ⇒ all) is the one authority for `HistoryServerCapabilities/AggregateFunctions` (i=11201 — only its folder references are trimmed) and the up-front `BadAggregateNotSupported` in HistoryRead-Processed. ⚠️ `ServerCapabilities/AggregateFunctions` (i=2997) is deliberately NOT trimmed: it keeps the SDK's 37 because live `AggregateFilter` subscriptions still compute them all. ⚠️ Historian disabled ⇒ Processed is now `BadAggregateNotSupported`, not `GoodNoData`. `docs/Historian.md` §Advertised aggregates. - **Advertised aggregates (#682):** `IHistorianDataSource.SupportedAggregates` (Gateway 4 via `GatewayHistorianDataSource.ServedAggregates`, SqlServer/QuestDB 17 via `AggregateFamilies.Served`, Null **none**; a null set ⇒ all) is the one authority for `HistoryServerCapabilities/AggregateFunctions` (i=11201 — only its folder references are trimmed) and the up-front `BadAggregateNotSupported` in HistoryRead-Processed. ⚠️ `ServerCapabilities/AggregateFunctions` (i=2997) is deliberately NOT trimmed: it keeps the SDK's 37 because live `AggregateFilter` subscriptions still compute them all. ⚠️ Historian disabled ⇒ Processed is now `BadAggregateNotSupported`, not `GoodNoData`. `docs/Historian.md` §Advertised aggregates.
- **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good. - **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good.
- **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node. - **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node.
+6 -2
View File
@@ -659,9 +659,13 @@ backend:
that row (`(row + d, top]`, `d` ×4 per step, following each truncated read) until a read comes back empty, then a that row (`(row + d, top]`, `d` ×4 per step, following each truncated read) until a read comes back empty, then a
**bisection** of the bracket — never narrower than the 1 ms request granularity, which only re-sent the same request. **bisection** of the bracket — never narrower than the 1 ms request granularity, which only re-sent the same request.
Measured: a dense tag that stopped two days ago ≈ **26** reads, a burst of just over 1,024 rows that ended 100 days ago Measured: a dense tag that stopped two days ago ≈ **26** reads, a burst of just over 1,024 rows that ended 100 days ago
≈ **18** (was 46 bisecting the whole gap) — rather than re-reading the oldest rows of a burst over and over. Only **one timestamp holding more than `M` samples** is refused, ≈ **18** (was 46 bisecting the whole gap) — rather than re-reading the oldest rows of a burst over and over. Only **more than `M` samples within one millisecond** are refused,
**`BadTooManyOperations`** (a request limit, not a backend fault); a cluster of exactly `M`, or neighbouring **`BadTooManyOperations`** (a request limit, not a backend fault); a cluster of exactly `M`, or neighbouring
clusters each within `M`, is served. clusters each within `M` and a millisecond or more apart, is served. ⚠️ The unit is the **millisecond**, not the
timestamp (#691.4 re-review): requests are rounded to the millisecond, so a narrower segment is the same request —
letting the width fall below it once stepped a page through one millisecond a tick at a time (~10,000 reads). Exact
for the Gateway, which stores milliseconds; a sub-millisecond backend walked this way would refuse more than `M`
samples inside one millisecond even when no single timestamp holds that many.
- **Proving the rest empty (#691).** Walking down through empty history to the floor used to cost one read per ×4 - **Proving the rest empty (#691).** Walking down through empty history to the floor used to cost one read per ×4
growth step — ~14–16 reads (~2.5 s live) on **every** last page of a deep page-through. Now, after two consecutive growth step — ~14–16 reads (~2.5 s live) on **every** last page of a deep page-through. Now, after two consecutive
empty segments, the walk asks **once per page** for the whole remainder down to the floor at the page budget: a empty segments, the walk asks **once per page** for the whole remainder down to the floor at the page budget: a
@@ -44,7 +44,7 @@ internal static class ReverseEventsReader
ev => ev.EventTimeUtc.Ticks, EventIdTieOrder.Instance, hi, lo, want, budget, source.TimestampResolutionTicks, ev => ev.EventTimeUtc.Ticks, EventIdTieOrder.Instance, hi, lo, want, budget, source.TimestampResolutionTicks,
nowUtc, hint, nowUtc, hint,
at => new HistoryReadLimitExceededException(budget, at => new HistoryReadLimitExceededException(budget,
$"A reverse events read of '{sourceName}' at {at:O} found more than {budget} events at one timestamp; the " + $"A reverse events read of '{sourceName}' at {at:O} found more than {budget} events within one millisecond; the " +
"backend reads forward only, so the latest of them cannot be selected. Increase " + "backend reads forward only, so the latest of them cannot be selected. Increase " +
"ServerHistorian:MaxTieClusterOverfetch."), "ServerHistorian:MaxTieClusterOverfetch."),
ct) ct)
@@ -71,7 +71,7 @@ internal static class ReverseRawReader
}, },
Ticks, tieOrder: null, hi, lo, maxValues, budget, source.TimestampResolutionTicks, nowUtc, hint, Ticks, tieOrder: null, hi, lo, maxValues, budget, source.TimestampResolutionTicks, nowUtc, hint,
at => new HistoryReadLimitExceededException(budget, at => new HistoryReadLimitExceededException(budget,
$"A reverse raw read of '{tagname}' at {at:O} found more than {budget} samples at one timestamp; the " + $"A reverse raw read of '{tagname}' at {at:O} found more than {budget} samples within one millisecond; the " +
"backend reads forward only, so the latest of them cannot be selected. Increase " + "backend reads forward only, so the latest of them cannot be selected. Increase " +
"ServerHistorian:MaxTieClusterOverfetch."), "ServerHistorian:MaxTieClusterOverfetch."),
ct) ct)
@@ -54,8 +54,9 @@ internal readonly record struct ForwardSegmentRead<T>(IReadOnlyList<T> Rows, boo
/// with room for them (at most twice the budget + 4); otherwise the segment is narrowed — to the density the truncated /// with room for them (at most twice the budget + 4); otherwise the segment is narrowed — to the density the truncated
/// read showed, and never again below the newest row it saw (<c>known</c>: rows exist there, so walking past it before /// read showed, and never again below the newest row it saw (<c>known</c>: rows exist there, so walking past it before
/// finding what lies above it would re-read the same dense stretch over and over; the gap above it is bisected /// finding what lies above it would re-read the same dense stretch over and over; the gap above it is bisected
/// instead). Only a single timestamp holding more than <c>M</c> rows — a one-resolution segment that still does not /// instead). Only more than <c>M</c> rows within one millisecond — a segment no wider than the 1 ms request
/// fit at the tie budget — is refused. /// granularity that still does not fit at the tie budget — is refused (per millisecond, not per timestamp: narrower
/// requests are the same request, and the Gateway stores milliseconds).
/// </para> /// </para>
/// <para> /// <para>
/// <b>Proving the rest empty (#691).</b> Walking down to the floor through empty history used to take one read per ×4 /// <b>Proving the rest empty (#691).</b> Walking down to the floor through empty history used to take one read per ×4
@@ -194,12 +195,13 @@ internal static class ReverseWalk
var gap = segHi - known; var gap = segHi - known;
lowered = lowerBy > 0 && lowerBy < gap; lowered = lowerBy > 0 && lowerBy < gap;
bisected = !lowered && (bisecting || lowerBy > 0) && gap > 2 * Math.Max(granularity, knownWidth); bisected = !lowered && (bisecting || lowerBy > 0) && gap > 2 * Math.Max(granularity, knownWidth);
// Never narrower than the request granularity when halving: requests are rounded outward to it, so a // Never narrower than the request granularity: requests are rounded outward to it, so a narrower segment
// narrower half re-sends the same request (#691.4 review). Keep the Min: a truncated read crowded by rows // re-sends the same request — and a width left at one tick (a tie cluster's) stepped segHi down a tick per
// before the segment must still shrink the segment. // read, ~10,000 identical reads (#691.4 re-review). Keep the Min: a truncated read crowded by rows before
// the segment must still shrink the segment.
w = lowered ? gap - lowerBy w = lowered ? gap - lowerBy
: bisected ? gap / 2 : bisected ? gap / 2
: Math.Min(w, gap <= 2 * granularity ? gap : gap / 2); : Math.Min(Math.Max(w, granularity), gap <= 2 * granularity ? gap : gap / 2);
} }
var segLo = segHi - stopTicks <= w ? stopTicks : segHi - w; var segLo = segHi - stopTicks <= w ? stopTicks : segHi - w;
var start = Utc(AlignDown(segLo, granularity)); var start = Utc(AlignDown(segLo, granularity));
@@ -248,10 +250,12 @@ internal static class ReverseWalk
continue; continue;
} }
if (segHi - segLo <= resolution) if (segHi - segLo <= granularity)
{ {
// A one-resolution segment still over its budget: one timestamp's tie cluster. Escalate to the tie // A segment no wider than the request granularity (1 ms — narrower requests are the same request) still
// budget M before refusing — a cluster of up to M rows is served (#686). // over its budget: a tie cluster, or a burst inside one millisecond. Escalate to the tie budget M before
// refusing — up to M rows within one millisecond are served (#686); more are refused, per millisecond
// (#691.4 re-review — exact for the millisecond-resolution Gateway).
if (segBudget < budget) if (segBudget < budget)
{ {
segBudget = budget; segBudget = budget;
@@ -446,20 +446,38 @@ public sealed class NodeManagerHistoryReadEventsPagingTests : IDisposable
var expected = events.OrderByDescending(e => e.EventTimeUtc).ThenByDescending(e => e.EventId, StringComparer.Ordinal) var expected = events.OrderByDescending(e => e.EventTimeUtc).ThenByDescending(e => e.EventId, StringComparer.Ordinal)
.Select(e => e.EventId).ToList(); .Select(e => e.EventId).ToList();
// The walk refuses more than MaxTieClusterOverfetch events within ONE MILLISECOND (#691.4 re-review): with the
// small budget a dense sub-millisecond burst is a legitimate BadTooManyOperations — and nothing else is.
var times = events.Select(e => e.EventTimeUtc.Ticks).Order().ToList();
var msMax = 0;
for (int i = 0, j = 0; i < times.Count; i++)
{
while (times[i] - times[j] >= TimeSpan.TicksPerMillisecond) j++;
msMax = Math.Max(msMax, i - j + 1);
}
var all = new List<string>(); var all = new List<string>();
byte[]? cp = null; byte[]? cp = null;
var pages = 0; var pages = 0;
var refused = false;
do do
{ {
var r = ReadEvents(nm, node, max, cp, end, DateTime.MinValue); var r = ReadEvents(nm, node, max, cp, end, DateTime.MinValue);
r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good, $"seed {seed}"); if (r.Error.StatusCode.Code == StatusCodes.BadTooManyOperations && msMax > nm.MaxTieClusterOverfetch)
{
refused = true;
break;
}
r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good, $"seed {seed}, ms max {msMax}");
all.AddRange(r.Ids); all.AddRange(r.Ids);
cp = r.Cp; cp = r.Cp;
(++pages).ShouldBeLessThan(2000); (++pages).ShouldBeLessThan(2000);
} }
while (cp is not null); while (cp is not null);
all.ShouldBe(expected, $"seed {seed}, N = {max}"); if (refused) all.ShouldBe(expected.Take(all.Count), $"seed {seed}: what came before the refusal");
else all.ShouldBe(expected, $"seed {seed}, N = {max}");
} }
await host.DisposeAsync(); await host.DisposeAsync();
@@ -731,9 +731,38 @@ public sealed class NodeManagerHistoryReadPagingTests : IDisposable
var max = (uint)rng.Next(1, 12); var max = (uint)rng.Next(1, 12);
var end = new DateTime(t + rng.Next(0, 3) * TimeSpan.TicksPerSecond, DateTimeKind.Utc); var end = new DateTime(t + rng.Next(0, 3) * TimeSpan.TicksPerSecond, DateTimeKind.Utc);
var collected = await DrainAsync(nm, nodeId, DateTime.MinValue, end, max, maxPages: 2000); // The walk refuses more than MaxTieClusterOverfetch samples within ONE MILLISECOND (#691.4 re-review), so with
// the small budget a dense sub-millisecond burst is a legitimate BadTooManyOperations — and nothing else is.
var msMax = 0;
for (int i = 0, j = 0; i < series.Count; i++)
{
while (series[i].SourceTimestampUtc!.Value.Ticks - series[j].SourceTimestampUtc!.Value.Ticks >= TimeSpan.TicksPerMillisecond) j++;
msMax = Math.Max(msMax, i - j + 1);
}
collected.ShouldBe(Enumerable.Range(0, series.Count).Reverse().Select(i => (double)i), $"seed {seed}, N = {max}"); var collected = new List<double>();
byte[]? cp = null;
var pages = 0;
var refused = false;
do
{
var (values, error, next) = ReadRaw(nm, nodeId, DateTime.MinValue, end, max, inboundCp: cp);
if (error.StatusCode.Code == StatusCodes.BadTooManyOperations && msMax > nm.MaxTieClusterOverfetch)
{
refused = true;
break;
}
error.StatusCode.Code.ShouldBe(StatusCodes.Good, $"seed {seed}, N = {max}, ms max {msMax}");
collected.AddRange(values);
cp = next;
(++pages).ShouldBeLessThan(2000, "paging must terminate, not loop");
}
while (cp is not null);
var expected = Enumerable.Range(0, series.Count).Reverse().Select(i => (double)i).ToList();
if (refused) collected.ShouldBe(expected.Take(collected.Count), $"seed {seed}: what came before the refusal");
else collected.ShouldBe(expected, $"seed {seed}, N = {max}");
} }
await host.DisposeAsync(); await host.DisposeAsync();
@@ -32,7 +32,14 @@ public sealed class ReverseWalkTests
for (var seed = 0; seed < 400; seed++) for (var seed = 0; seed < 400; seed++)
{ {
var rng = new Random(seed); var rng = new Random(seed);
var series = RandomSeries(rng, out var maxCluster); var series = RandomSeries(rng, out _);
// The refusal unit is one millisecond (#691.4 re-review): the most rows inside any 1 ms window.
var maxCluster = 0;
for (int i = 0, j = 0; i < series.Count; i++)
{
while (Ts(series[i]) - Ts(series[j]) >= TimeSpan.TicksPerMillisecond) j++;
maxCluster = Math.Max(maxCluster, i - j + 1);
}
var source = new AvevaShapedSource(series, roundUp: rng.Next(2) == 0); var source = new AvevaShapedSource(series, roundUp: rng.Next(2) == 0);
var tieBudget = rng.Next(4) switch { 0 => 1, 1 => 3, 2 => 8, _ => 65536 }; var tieBudget = rng.Next(4) switch { 0 => 1, 1 => 3, 2 => 8, _ => 65536 };
var want = (uint)rng.Next(1, 40); var want = (uint)rng.Next(1, 40);
@@ -81,7 +88,7 @@ public sealed class ReverseWalkTests
} }
catch (HistoryReadLimitExceededException) catch (HistoryReadLimitExceededException)
{ {
if (maxCluster <= tieBudget) failures.Add($"seed {seed}: refused with no cluster over {tieBudget} (max {maxCluster})"); if (maxCluster <= tieBudget) failures.Add($"seed {seed}: refused with no millisecond over {tieBudget} rows (max {maxCluster})");
break; break;
} }
} }
@@ -206,6 +213,33 @@ public sealed class ReverseWalkTests
source.Reads.ShouldBeLessThanOrEqualTo(22, string.Join("\n", source.Log)); source.Reads.ShouldBeLessThanOrEqualTo(22, string.Join("\n", source.Log));
} }
/// <summary>
/// #691.4 re-review (S8): a tie cluster under a carried width. 30 rows at one whole-millisecond timestamp 0.3 s below
/// <c>hi</c>, nothing above, N = 1, a carried 1 s width. The truncated read of the cluster left the width at one tick;
/// once the gap above it was under two milliseconds nothing floored it, so segHi stepped down one tick per read —
/// 10,006 reads of the same millisecond (28 before the gallop).
/// </summary>
[Theory]
[InlineData(30, 1, 0.3, 1.0)]
[InlineData(34, 1, 0.52, 0.48)]
[InlineData(25, 1, 1.2, 1.0)]
[InlineData(40, 3, 3.5, 2.0)]
public async Task A_tie_cluster_under_a_carried_width_is_read_in_a_bounded_number_of_reads(
int ties, int want, double secondsBelow, double hintSeconds)
{
var at = T0.Ticks - (long)(secondsBelow * TimeSpan.TicksPerSecond);
at -= at % TimeSpan.TicksPerMillisecond;
var series = Enumerable.Range(0, ties).Select(i => At(at, i)).ToList();
var source = new AvevaShapedSource(series);
var page = await ReverseRawReader.ReadAsync(source, "T", T0, DateTime.MinValue, (uint)want, 65536, T0,
new ReverseWalkHint((long)(hintSeconds * TimeSpan.TicksPerSecond)), Ct);
page.Samples.Select(s => (double)s.Value!).ShouldBe(Enumerable.Range(ties - want, want).Reverse().Select(i => (double)i));
source.Reads.ShouldBeLessThanOrEqualTo(30, string.Join("\n", source.Log.Take(40)));
source.RepeatedRequests.ShouldBe(0);
}
private static long Ts(DataValueSnapshot s) => s.SourceTimestampUtc!.Value.Ticks; private static long Ts(DataValueSnapshot s) => s.SourceTimestampUtc!.Value.Ticks;
/// <summary>Random ascending series: bursts, gaps, sub-millisecond stamps and tie clusters.</summary> /// <summary>Random ascending series: bursts, gaps, sub-millisecond stamps and tie clusters.</summary>