diff --git a/CLAUDE.md b/CLAUDE.md index a2c106fd..eb39bbc5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1353,7 +1353,7 @@ historian tagname, which is the tag's driver `FullName`. - **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; per-sample aggregates (10, simple definitions) are pushed down to T-SQL with bucket math in **100 ns ticks**; the Part 13 time-weighted seven (TimeAverage/TimeAverage2/Total/Total2/Interpolative, #671 — ⚠️ `Total` is now the time-weighted integral, no longer `SUM` — plus the **duration-based** PercentGood/PercentBad, #680) stream window + 4 bound rows into the shared `TimeWeightedAggregator` (both SQL backends, identical results; reverse `Interpolative` anchored at the later time; window capped by `ServerHistorian:MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations`, percentages included). ⚠️ **#680 status rules (client-visible):** TimeAverage/Total are Uncertain if any part of the interval uses an Uncertain value, is bridged across a Bad sample, is extrapolated past the last sample (⇒ a live tag's latest interval is Uncertain) or is not covered; TimeAverage2/Total2 keep the SDK time-based rule (Uncertain time counts as Good) but never extrapolate past the last sample — that interval is Partial, later ones `BadNoData` (a report-by-exception tag with nothing after its last sample reads `BadNoData` until its next sample arrives). `docs/Historian.md` §"Time-weighted aggregates". ⚠️ **#680 per-sample (client-visible):** `Count` counts **Good** samples only (value-based status; an all-Bad interval is generic Bad, value null) and an empty interval **inside the tag's data range** is `0`/Good (outside it still `BadNoData`); `Start`/`End` return the **raw** first/last sample of any quality — its own timestamp and status, value null when Bad; `Delta` is Uncertain only when the interval's first or last sample is unusable; ⚠️ #684: `Average`/`Minimum`/`Maximum`/`Range`/`*ActualTime` use **Good samples only** (was Good + Uncertain; an Uncertain-only interval is `BadNoData`), Min/Max-family status Uncertain on a Bad sample, an Uncertain value beyond the Good extreme (Part 13 — the SDK ignores that) or an Uncertain non-numeric sample, and only a **Good** String/DateTime sample refuses them (`BadAggregateInvalidInputs`; Delta still refuses any usable one); and every per-sample aggregate but `Average` sets **end-of-data Partial** on the interval holding the last sample (forward incl. its earlier edge, reverse strictly inside), never on a Bad result. The data-range lookup is a key seek on SQL Server and a `tag` symbol-index seek on QuestDB (#683 — `{p}sample.tag` is `SYMBOL INDEX`; the bootstrapper ADDs it to a pre-#683 table, idempotent), read BEFORE the bucket statement so a racing append can never yield a Good `Count` of 0 (a concurrent HistoryUpdate delete is the mirror case and still can, transiently). `docs/Historian.md` §"Aggregates". Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. **Retention (#667):** opt-in `SqlServer:RetentionDays` (+ `PurgeInterval` 1h, `PurgeBatchSize` 4000) -> `HistorianRetentionService` purges old `hist.Sample` (per tag) / `hist.Event` in `DELETE TOP(n)` batches, Primary-gated on `IHistorizationRoleView`, one purger per DB via `sp_getapplock`; counter `otopcua.historian.retention.purged{table}`. Optional `IHistorianBackendProvider.CreateRetention` (default null). `docs/Historian.md` §Retention. - **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend". - ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); on forward reads Partial also marks the interval the data starts inside (not Average — the SDK's rule, forward-only); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates. -- ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` walks `(lo, hi]` back in bounded FORWARD segment reads (1 min ×4 while sparse, each ≤ `MaxTieClusterOverfetch + 1` rows, a full read is re-read narrower — never answered truncated; a one-timestamp cluster over the bound ⇒ `BadTooManyOperations`; stops at the Unix epoch). Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events still differ (end-only forward / explicit reverse refused on Gateway, #675). `docs/Historian.md` §"Open-ended and reverse raw reads". +- ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` serves it from FORWARD reads: uncapped ⇒ ONE forward read reversed in memory; capped ⇒ a bounded walk back in segments (ms-aligned request bounds, width aimed from observed density and carried page to page in the continuation point, ≤ `MaxTieClusterOverfetch + 2` rows per read — a full read is re-read, never answered truncated; only one timestamp holding more than max(1, `MaxTieClusterOverfetch`) samples ⇒ `BadTooManyOperations`; walks no later than now + 1 day and never before the Unix epoch). The raw/events tie-path oversized-cluster refusal is now `BadTooManyOperations` too (was `BadHistoryOperationUnsupported`). Uncapped Gateway raw reads (forward too) used to send `max_values = 0`, which the client/gateway reject — now unbounded (`int.MaxValue`; a ceiling refusal ⇒ `BadTooManyOperations`), and a cancelled read is no longer a backend failure. Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events still differ (end-only forward / explicit reverse refused on Gateway, #675). `docs/Historian.md` §"Open-ended and reverse raw reads". - **Advertised aggregates (#682):** `IHistorianDataSource.SupportedAggregates` (Gateway 4 via `GatewayHistorianDataSource.ServedAggregates`, SqlServer/QuestDB 17 via `AggregateFamilies.Served`, Null **none**; a null set ⇒ all) is the one authority for `HistoryServerCapabilities/AggregateFunctions` (i=11201 — only its folder references are trimmed) and the up-front `BadAggregateNotSupported` in HistoryRead-Processed. ⚠️ `ServerCapabilities/AggregateFunctions` (i=2997) is deliberately NOT trimmed: it keeps the SDK's 37 because live `AggregateFilter` subscriptions still compute them all. ⚠️ Historian disabled ⇒ Processed is now `BadAggregateNotSupported`, not `GoodNoData`. `docs/Historian.md` §Advertised aggregates. - **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good. - **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node. diff --git a/docs/Historian.md b/docs/Historian.md index 8f1ece70..c780be2e 100644 --- a/docs/Historian.md +++ b/docs/Historian.md @@ -143,7 +143,7 @@ Processed read is `BadAggregateNotSupported` and no history aggregate functions | `AllowUntrustedServerCertificate` | bool | `false` | **Gateway backend only.** Accept a self-signed / untrusted server certificate (dev / on-prem only). | | `CaCertificatePath` | string\|null | `null` | **Gateway backend only.** PEM CA file pinning the gateway's TLS chain. Null/empty uses the OS trust store. | | `CallTimeout` | TimeSpan | `00:00:30` | **Gateway backend only.** Per-call deadline applied to each unary gateway read. A read for a tag the historian does not yet know about — e.g. a freshly-historized tag that has **not** been provisioned (see [Tag auto-provisioning](#tag-auto-provisioning-ensuretags)) — can run to this **full** deadline before the gateway errors, so an unprovisioned-tag HistoryRead can block for ~30 s. Lower this to fail faster, at the cost of truncating legitimately long reads. | -| `MaxTieClusterOverfetch` | int | `65536` | (Node-manager paging; all backends — a `SqlServer` PK rules out tie clusters, so it is moot there.) Maximum samples the server will fetch in one shot to page through a tie cluster (multiple samples sharing one `SourceTimestamp`). A cluster larger than this ceiling fails `BadHistoryOperationUnsupported`. Raise to handle abnormally large tie clusters; the default covers all normal-data cases. | +| `MaxTieClusterOverfetch` | int | `65536` | (Node-manager paging; all backends — a `SqlServer` PK rules out tie clusters, so it is moot there.) Maximum samples the server will fetch in one shot to page through a tie cluster (multiple samples sharing one `SourceTimestamp`). A cluster larger than this ceiling fails `BadTooManyOperations` (#686; was `BadHistoryOperationUnsupported`). Also the per-read row budget of a Gateway reverse raw read (walked back from forward reads, #686): one timestamp holding more samples is refused `BadTooManyOperations`. Values below 1 count as 1. Raise to handle abnormally large tie clusters; the default covers all normal-data cases. | | `HistoryReadBatchParallelism` | int | `4` | Max historized nodes served **concurrently** within one HistoryRead batch (arch-review 03/S3). The SDK's `HistoryRead*` override surface is synchronous, so each arm bridges once at its boundary; this turns per-node serving from sequential (`N × CallTimeout` worst case) into `⌈N / P⌉ × CallTimeout`. Also caps the gateway-load multiplication a single client can command. `≤ 1` serves sequentially. | | `MaxConcurrentHistoryReadBatches` | int | `16` | Process-wide limit on HistoryRead (and HistoryUpdate) batches served **concurrently** (arch-review 03/S3). A flood of history clients degrades to `BadTooManyOperations` rather than exhausting the SDK request-thread pool. With `HistoryReadBatchParallelism` this caps total in-flight gateway reads at `MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism` (default 16 × 4 = 64). A batch that cannot get a slot within `min(HistoryReadDeadline, 5 s)` fails every handle `BadTooManyOperations`. | | `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` (#671), and the `Gateway` per-sample aggregates (#681 — every per-sample read there streams raw samples, so the cap applies to the window's raw rows; see [Aggregate set](#aggregate-set-and-badaggregatenotsupported-285)). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative, and — since #680 — **PercentGood/PercentBad**) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped on the SQL backends (pushed down). 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). | @@ -612,7 +612,8 @@ paging time-based: > > **Cluster larger than `MaxTieClusterOverfetch`.** If the over-fetch itself reaches the ceiling > without spanning the full cluster, the node fails **loudly** with -> `BadHistoryOperationUnsupported` and the tag + timestamp + ceiling are logged. Remedies: raise +> `BadTooManyOperations` (a request limit; `BadHistoryOperationUnsupported` before #686 — events reads changed the same +> way) and the tag + timestamp + ceiling are logged. Remedies: raise > `MaxTieClusterOverfetch` (or `NumValuesPerNode`) to cover the full cluster, or investigate the > data anomaly (raw samples normally carry strictly increasing distinct timestamps). > @@ -630,17 +631,28 @@ backend: (`… > lo AND … <= hi ORDER BY time DESC`). - **Gateway** (the default backend) reports `false` — the gateway's `ReadRaw` streams ascending only and rejects `start > end` with `InvalidArgument`. The server never sends it one (and the source refuses one unsent, not counted as a - health failure): it **walks the window back in bounded forward reads** (`ReverseRawReader`). It reads one segment - `(segLo, segHi]` forward — requested `[segLo, segHi + resolution]` and filtered on the samples' own timestamps, which also - drops the rows AVEVA fabricates at a query's start — keeps it newest-first, and steps to the next older segment until the - page is full or the window is exhausted. Segments start at **1 minute** and grow ×4 while sparse, so "the last N values" - of a live tag is normally one small read and sparse history is found in a few dozen tiny ones. Each segment read asks for - at most `MaxTieClusterOverfetch + 1` rows; a read that comes back full may be truncated (its newest rows are the ones - needed), so it is discarded and the segment re-read at a quarter of the width — **never answered from a truncated - read**. A single-timestamp tie cluster larger than `MaxTieClusterOverfetch` cannot be served newest-first this way and - the node is refused **`BadTooManyOperations`** (a request limit, not a backend fault). The walk stops at the **Unix epoch** - (AVEVA holds nothing earlier): a walked-back read returns no sample before 1970. Paging is unchanged — a reverse cursor - resumes as a fresh walk from its resume point. + health failure); it serves the reverse window from forward reads (`ReverseRawReader`): + - **Uncapped** (`NumValuesPerNode = 0`): **one** forward read of the whole window, filtered and reversed in memory — the + cost and memory of an uncapped forward read. + - **Capped: a bounded walk back.** It reads one segment `(segLo, segHi]` forward, keeps it newest-first, and steps to + the next older segment until the page is full or the window is exhausted. A segment is requested as + `[⌊segLo⌋, ⌈segHi + 1 tick⌉]`, both rounded **outward to the millisecond**, and filtered on the samples' own + timestamps — exact for an inclusive or exclusive forward end, and the row AVEVA fabricates at a query's (possibly + rounded-up) start can never land inside the segment. The first segment is **1 minute** (or the width the previous + page ended on — it rides the continuation point); each next width is aimed at about half the row budget from the + density just observed, growing at most ×4. So "the last N values" of a live tag is normally one read, a dense tag + costs about one read per page, and sparse history is found in a few dozen tiny ones. + - **Never answered from a truncated read.** The row budget is `M = max(1, MaxTieClusterOverfetch)`; a segment read asks + for `M + 2` rows (room for the fabricated row and one to see past the segment). Rows stream ascending, so a read is + complete when it came back short or its last row is past the segment; otherwise it is discarded — re-read once with + room for the rows before the segment that crowded it (a neighbouring tie cluster, at most `2M + 4` rows), or narrowed + to the density it showed. Only **one timestamp holding more than `M` samples** is refused, **`BadTooManyOperations`** + (a request limit, not a backend fault); a cluster of exactly `M`, or neighbouring clusters each within `M`, is served. + - **Bounds.** The walk starts no later than **now + 1 day** (a `StartTime = DateTime.MaxValue` read does not send + year-9999 query starts) and never reads before the **Unix epoch** (AVEVA holds nothing earlier): a walked-back read + returns only samples strictly after 1970-01-01T00:00:00Z and no later than now + 1 day. + + Paging is unchanged — a reverse cursor resumes as a fresh walk from its resume point. - **Null** reports `true` (empty in either direction). > ⚠️ **Client-visible (#686).** Since #669 an end-only capped raw read, and any explicit reverse raw read, on the Gateway @@ -648,6 +660,12 @@ backend: > refused `start > end`). Before #669 an end-only read on Gateway returned the **oldest** N values ascending. Both now return > the Part 11 answer — the latest values at or before `EndTime`, newest first. **Events differ:** an end-only events read on > Gateway still keeps its forward meaning and an explicit reverse events read is refused (see below). +> +> ⚠️ **Also fixed with #686: uncapped raw reads on Gateway.** `NumValuesPerNode = 0` ("all values") used to be forwarded as +> `max_values = 0`, which the client package and the gateway reject — every uncapped raw read (forward too) failed the node +> and counted as a backend failure. It is now sent unbounded (`int.MaxValue`); a gateway configured with +> `AllowStreamingBeyondCeiling=false` refuses that over its `MaxValuesCeiling`, which is answered **`BadTooManyOperations`** +> (not a backend fault). A cancelled read (client cancel / `HistoryReadDeadline`) is no longer counted as a backend failure. Continuation points are bound to the OPC UA session (the SDK's `ServerConfiguration.MaxHistoryContinuationPoints` cap, default 100, with oldest-eviction; points