docs(historian): #686 open-ended and reverse raw reads on every backend

2026-10-12 14:42:27 -04:00
parent 410c8d8976
commit be2f0cc14d
2 changed files with 30 additions and 0 deletions
@@ -1353,6 +1353,7 @@ historian tagname, which is the tag's driver `FullName`.
 - **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; per-sample aggregates (10, simple definitions) are pushed down to T-SQL with bucket math in **100 ns ticks**; the Part 13 time-weighted seven (TimeAverage/TimeAverage2/Total/Total2/Interpolative, #671 — ⚠️ `Total` is now the time-weighted integral, no longer `SUM` — plus the **duration-based** PercentGood/PercentBad, #680) stream window + 4 bound rows into the shared `TimeWeightedAggregator` (both SQL backends, identical results; reverse `Interpolative` anchored at the later time; window capped by `ServerHistorian:MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations`, percentages included). ⚠️ **#680 status rules (client-visible):** TimeAverage/Total are Uncertain if any part of the interval uses an Uncertain value, is bridged across a Bad sample, is extrapolated past the last sample (⇒ a live tag's latest interval is Uncertain) or is not covered; TimeAverage2/Total2 keep the SDK time-based rule (Uncertain time counts as Good) but never extrapolate past the last sample — that interval is Partial, later ones `BadNoData` (a report-by-exception tag with nothing after its last sample reads `BadNoData` until its next sample arrives). `docs/Historian.md` §"Time-weighted aggregates". ⚠️ **#680 per-sample (client-visible):** `Count` counts **Good** samples only (value-based status; an all-Bad interval is generic Bad, value null) and an empty interval **inside the tag's data range** is `0`/Good (outside it still `BadNoData`); `Start`/`End` return the **raw** first/last sample of any quality — its own timestamp and status, value null when Bad; `Delta` is Uncertain only when the interval's first or last sample is unusable; ⚠️ #684: `Average`/`Minimum`/`Maximum`/`Range`/`*ActualTime` use **Good samples only** (was Good + Uncertain; an Uncertain-only interval is `BadNoData`), Min/Max-family status Uncertain on a Bad sample, an Uncertain value beyond the Good extreme (Part 13 — the SDK ignores that) or an Uncertain non-numeric sample, and only a **Good** String/DateTime sample refuses them (`BadAggregateInvalidInputs`; Delta still refuses any usable one); and every per-sample aggregate but `Average` sets **end-of-data Partial** on the interval holding the last sample (forward incl. its earlier edge, reverse strictly inside), never on a Bad result. The data-range lookup is a key seek on SQL Server and a `tag` symbol-index seek on QuestDB (#683 — `{p}sample.tag` is `SYMBOL INDEX`; the bootstrapper ADDs it to a pre-#683 table, idempotent), read BEFORE the bucket statement so a racing append can never yield a Good `Count` of 0 (a concurrent HistoryUpdate delete is the mirror case and still can, transiently). `docs/Historian.md` §"Aggregates". Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. **Retention (#667):** opt-in `SqlServer:RetentionDays` (+ `PurgeInterval` 1h, `PurgeBatchSize` 4000) -> `HistorianRetentionService` purges old `hist.Sample` (per tag) / `hist.Event` in `DELETE TOP(n)` batches, Primary-gated on `IHistorizationRoleView`, one purger per DB via `sp_getapplock`; counter `otopcua.historian.retention.purged{table}`. Optional `IHistorianBackendProvider.CreateRetention` (default null). `docs/Historian.md` §Retention.
 - **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend".
 - ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); on forward reads Partial also marks the interval the data starts inside (not Average — the SDK's rule, forward-only); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates.
 - ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` walks `(lo, hi]` back in bounded FORWARD segment reads (1 min ×4 while sparse, each ≤ `MaxTieClusterOverfetch + 1` rows, a full read is re-read narrower — never answered truncated; a one-timestamp cluster over the bound ⇒ `BadTooManyOperations`; stops at the Unix epoch). Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events still differ (end-only forward / explicit reverse refused on Gateway, #675). `docs/Historian.md` §"Open-ended and reverse raw reads".
 - **Advertised aggregates (#682):** `IHistorianDataSource.SupportedAggregates` (Gateway 4 via `GatewayHistorianDataSource.ServedAggregates`, SqlServer/QuestDB 17 via `AggregateFamilies.Served`, Null **none**; a null set ⇒ all) is the one authority for `HistoryServerCapabilities/AggregateFunctions` (i=11201 — only its folder references are trimmed) and the up-front `BadAggregateNotSupported` in HistoryRead-Processed. ⚠️ `ServerCapabilities/AggregateFunctions` (i=2997) is deliberately NOT trimmed: it keeps the SDK's 37 because live `AggregateFilter` subscriptions still compute them all. ⚠️ Historian disabled ⇒ Processed is now `BadAggregateNotSupported`, not `GoodNoData`. `docs/Historian.md` §Advertised aggregates.
 - **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good.
 - **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node.
@@ -620,6 +620,35 @@ paging time-based:
 > anomaly. The ceiling exists to bound server-side memory on pathological data, not to cap normal
 > operation.
 **Open-ended and reverse raw reads (#669, #686).** With `NumValuesPerNode > 0` one bound may be unset (Part 11): a
 start-only read pages **forward** to an open upper bound; an end-only read pages **backward** from `EndTime` — the reverse
 window `(MinValue, EndTime]`, newest first, so the first page is the `NumValuesPerNode` values at or before `EndTime`. An
 explicit `StartTime > EndTime` is the reverse window `(EndTime, StartTime]`, newest first. These semantics hold on **every**
 backend:
 - **SqlServer / QuestDB** report `IHistorianDataSource.SupportsReverseRaw = true` and are sent the reverse window directly
  (`… > lo AND … <= hi ORDER BY time DESC`).
 - **Gateway** (the default backend) reports `false` — the gateway's `ReadRaw` streams ascending only and rejects
  `start > end` with `InvalidArgument`. The server never sends it one (and the source refuses one unsent, not counted as a
  health failure): it **walks the window back in bounded forward reads** (`ReverseRawReader`). It reads one segment
  `(segLo, segHi]` forward — requested `[segLo, segHi + resolution]` and filtered on the samples' own timestamps, which also
  drops the rows AVEVA fabricates at a query's start — keeps it newest-first, and steps to the next older segment until the
  page is full or the window is exhausted. Segments start at **1 minute** and grow ×4 while sparse, so "the last N values"
  of a live tag is normally one small read and sparse history is found in a few dozen tiny ones. Each segment read asks for
  at most `MaxTieClusterOverfetch + 1` rows; a read that comes back full may be truncated (its newest rows are the ones
  needed), so it is discarded and the segment re-read at a quarter of the width — **never answered from a truncated
  read**. A single-timestamp tie cluster larger than `MaxTieClusterOverfetch` cannot be served newest-first this way and
  the node is refused **`BadTooManyOperations`** (a request limit, not a backend fault). The walk stops at the **Unix epoch**
  (AVEVA holds nothing earlier): a walked-back read returns no sample before 1970. Paging is unchanged — a reverse cursor
  resumes as a fresh walk from its resume point.
 - **Null** reports `true` (empty in either direction).
 > ⚠️ **Client-visible (#686).** Since #669 an end-only capped raw read, and any explicit reverse raw read, on the Gateway
 > backend failed the node (`BadHistoryOperationUnsupported`) and was counted as a historian backend failure (the gateway
 > refused `start > end`). Before #669 an end-only read on Gateway returned the **oldest** N values ascending. Both now return
 > the Part 11 answer — the latest values at or before `EndTime`, newest first. **Events differ:** an end-only events read on
 > Gateway still keeps its forward meaning and an explicit reverse events read is refused (see below).
 Continuation points are bound to the OPC UA session (the SDK's
 `ServerConfiguration.MaxHistoryContinuationPoints` cap, default 100, with oldest-eviction; points
 are disposed when the session closes). Resuming an unknown / evicted / released point returns