From f6681544e784f2f5052ad424d26976f9911d523d Mon Sep 17 00:00:00 2001 From: Joseph Doherty Date: Mon, 12 Oct 2026 16:27:27 -0400 Subject: [PATCH] =?UTF-8?q?docs(historian):=20#691=20bounding=20values=20(?= =?UTF-8?q?ReturnBounds)=20=E2=80=94=20Historian.md=20+=20CLAUDE.md?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- CLAUDE.md | 1 + docs/Historian.md | 45 +++++++++++++++++++++++++++++++++++++++------ 2 files changed, 40 insertions(+), 6 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 1b49cc92..2ddce37c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1355,6 +1355,7 @@ historian tagname, which is the tag's driver `FullName`. - ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); on forward reads Partial also marks the interval the data starts inside (not Average — the SDK's rule, forward-only); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates. - ⚠️ **Reverse / end-only RAW reads work on Gateway (#686, client-visible):** `IHistorianDataSource.SupportsReverseRaw` (SqlServer/QuestDB/Null `true`, Gateway `false`, interface default `false`, guard-tested) — the node manager sends `start > end` only to a backend that declares it; otherwise `ReverseRawReader` serves it from FORWARD reads: uncapped ⇒ ONE forward read reversed in memory; capped ⇒ a bounded walk back in segments (ms-aligned request bounds, width aimed from observed density at ≈ N + 8 rows and carried page to page in the continuation point — #689: it used to aim at half `MaxTieClusterOverfetch`, ~32k rows per page on a dense tag; a read asks for `2N + 18` rows (a discovery segment ≤ 1,026), `MaxTieClusterOverfetch` is only the tie-cluster ceiling — a full read is re-read, never answered truncated; only more than max(1, `MaxTieClusterOverfetch`) samples within ONE MILLISECOND ⇒ `BadTooManyOperations` (per ms, not per timestamp — requests are ms-rounded, and letting a segment narrow below 1 ms stepped a page through one ms a tick per read, ~10k reads); walks no later than now + 1 day and never before the Unix epoch; #691: after two empty segments it asks once for the whole remainder, and the stretch a page proved empty rides the continuation point — the last pages of a deep page-through no longer each walk ~14 reads to 1970). The raw/events tie-path oversized-cluster refusal is now `BadTooManyOperations` too (was `BadHistoryOperationUnsupported`). Uncapped Gateway raw reads (forward too) used to send `max_values = 0`, which the client/gateway reject — now unbounded (`int.MaxValue`; a ceiling refusal ⇒ `BadTooManyOperations`), and a cancelled read is no longer a backend failure. Since #669 such reads failed the node on Gateway and counted as a backend fault (the gateway rejects start > end — `GatewayHistorianDataSource.ReadRawAsync` now refuses one unsent); before #669 an end-only Gateway read returned the OLDEST N ascending — now the latest N at/before EndTime, newest first. Events follow since #691.4: a reverse/end-only capped events read on Gateway (`SupportsReverseEvents` false; Null now reports `true` and answers empty) is walked back the same way (ties sorted by EventId — the gateway orders by time only; one extra event found ⇒ continuation point) — it used to be refused `BadHistoryOperationUnsupported` (explicit reverse) or answered with the OLDEST N ascending (end-only); needs the gateway's `RuntimeDb:EventReadMaxRows` 0/unlimited (it truncates silently). Uncapped Gateway events reads (forward too) used to throw "Array dimensions exceeded" (result list pre-sized to `int.MaxValue`) — fixed. Above a truncated read the walk gallops up from the newest row seen, then bisects (never below 1 ms): a burst that ended 100 days ago ≈ 18 reads, a dense tag that stopped days ago ≈ 26. `docs/Historian.md` §"Open-ended and reverse raw reads". - **Advertised aggregates (#682):** `IHistorianDataSource.SupportedAggregates` (Gateway 4 via `GatewayHistorianDataSource.ServedAggregates`, SqlServer/QuestDB 17 via `AggregateFamilies.Served`, Null **none**; a null set ⇒ all) is the one authority for `HistoryServerCapabilities/AggregateFunctions` (i=11201 — only its folder references are trimmed) and the up-front `BadAggregateNotSupported` in HistoryRead-Processed. ⚠️ `ServerCapabilities/AggregateFunctions` (i=2997) is deliberately NOT trimmed: it keeps the SDK's 37 because live `AggregateFilter` subscriptions still compute them all. ⚠️ Historian disabled ⇒ Processed is now `BadAggregateNotSupported`, not `GoodNoData`. `docs/Historian.md` §Advertised aggregates. +- **Raw `ReturnBounds` is honoured on every backend (#691, client-visible):** Part 11 bounding values (§4.6 Table 1 is the test oracle) are composed in the server by `HistoryBounds` from each backend's raw reads — "at or before" = capped reverse read (native on SQL backends, the #686 `ReverseRawReader` walk-back on Gateway, never AVEVA's synthetic boundary row), "at or after" = capped forward read. Bounds are stored samples of any quality, count toward `NumValuesPerNode`, start bound on the first page / end bound on the last (pending end bound rides the continuation point); missing ⇒ `Bad_BoundNotFound` (null, at the bound time; a one-sided read's open side ±1 s from the previous value). ⚠️ The .NET SDK's `new ReadRawModifiedDetails()` defaults `ReturnBounds = true`, so a client that never set it now gets bounds — set it `false` in any test/client building details (Client.Shared already does). `IsReadModified` + `ReturnBounds` ⇒ `Bad_InvalidArgument` (§6.5.3.3). `docs/Historian.md` §Bounding values. - **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good. - **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node. - **HistoryUpdate** (SqlServer + QuestDb; Gateway ⇒ `BadHistoryOperationUnsupported`): Insert/Replace/Update/DeleteRaw/DeleteAtTime, **Primary-only** since #674 (off-Primary ⇒ `BadInvalidState`, after authorization AND after planning — an unservable request (Remove / `IsDeleteModified` / not historized) stays `BadHistoryOperationUnsupported` off-Primary; the verdict is `IHistoryUpdateRoleView` = the data-plane verdict but **closed until this node's first redundancy snapshot**, so a pair cannot both accept in the boot window; admin-only nodes register no view ⇒ no gate; reads ungated), items with a `SourceTimestamp` past now + `ServerHistorian:HistoryUpdateMaxFutureSkew` (default 1 day, ≤0 disables) ⇒ `BadOutOfRange` in the node manager (every backend), gated in both ACL modes — flag off needs the `HistoryUpdate` role from a **Cluster-scoped** grant (+ key re-check), flag on uses `DataPlaneAuthorizer`'s HistoryUpdate operation; `WriteOperate` never implies it. The `HistoryWrite` AccessLevel bit is set only when an updater is wired at materialise time. `docs/Historian.md` §HistoryUpdate, `docs/security.md`. ⚠️ **SqlServer schema v1 is accepted (#674)** with `AutoCreateSchema=false`: writes/reads/events/retention work; modified reads + HistoryUpdate ⇒ node-level `BadHistoryOperationUnsupported` on every arm (the updater throws `NotSupportedException`, nothing sent; the `HistoryWrite` bit is still advertised — not knowable at materialise time), retention skips `SampleModified`, one Warning per boot (`SqlServerSchemaBootstrapper.SupportsModifiedTrail`; `RequiredVersion` = 1, `ModifiedTrailVersion` = 2). QuestDB: keep `cairo.ttl.use.wall.clock=true`; a QuestDB trail INSERT retried after a client timeout duplicates (no DEDUP key). diff --git a/docs/Historian.md b/docs/Historian.md index 40aae51a..f05a7998 100644 --- a/docs/Historian.md +++ b/docs/Historian.md @@ -752,10 +752,43 @@ data. asking for `NumValuesPerNode + m + 16` (m = the pre-window rows the first read saw, unclamped). Only if that re-read is still short because of pre-window rows does the node fail `BadTooManyOperations` (not a backend fault) rather than answer a short page that would end the client's paging early. Every gateway stream is still drained to completion. -- **`ReturnBounds` is not honoured** — on any backend: the server does not pass it to the data source, so no bounding values are - returned (see [Known limitations](#known-limitations)). The AVEVA boundary row is not used as a bound either: it is stamped at the - query start rather than carrying the time of the real prior sample, so it could only ever be a Part 11 *interpolated* bound, which - this backend does not compute. +- **`ReturnBounds` is served by the server, not by this row** (#691, [Bounding values](#bounding-values-returnbounds-691)). The AVEVA + boundary row is never used as a bound: it is stamped at the query start rather than carrying the time of the real prior sample, so + it could only ever be a Part 11 *interpolated* bound. The start bound on the Gateway is the real prior sample, found by the + #686 walk-back. + +### Bounding values (`ReturnBounds`, #691) + +⚠️ **Client-visible.** Until #691 `ReadRawModifiedDetails.ReturnBounds` was ignored on every backend. It is now honoured on +**every** backend (SqlServer, QuestDB, Gateway, and none), following OPC UA Part 11 §4.6 / §6.5.3.2 and checked against every row +of Part 11 Table 1. **Note that the OPC Foundation .NET SDK's `new ReadRawModifiedDetails()` defaults `ReturnBounds` to `true`**: +a .NET client that never set it used to get stored values only and now gets bounds too. `Client.Shared` (Client.CLI / Client.UI) +sets it `false` explicitly, and ScadaBridge does not issue HistoryRead. + +- **What a bound is.** Forward (`StartTime < EndTime`): the start bound is the stored sample **at or immediately before** + StartTime, the end bound the one **at or immediately after** EndTime. Reverse mirrors it (start bound at or after StartTime, end + bound at or before EndTime), newest first. A sample exactly **on** StartTime is the bound and is already the data's first value + (nothing extra is added); a sample exactly on EndTime is added as the end bound (the data window excludes EndTime). + `StartTime == EndTime` reads forward: the data is the samples at that instant, the end bound the first sample strictly after it. +- **A bound is a stored sample of any quality** — a Bad sample or a comms-loss outage marker comes back as stored. Inside a tie + cluster the bound is the cluster's first sample in the read's direction. +- **Missing bound:** `Bad_BoundNotFound`, null value, stamped with StartTime / EndTime. A one-sided read's open side (only a + StartTime, or only an EndTime) is never looked up; it ends the read as `Bad_BoundNotFound` stamped one second after (forward) or + before (reverse) the previous returned value (Table 1 footnotes a/b). `Bad_BoundNotSupported` is never returned — every backend + can find a bound. +- **Counting and paging.** Bounds **count toward `NumValuesPerNode`** (§6.5.3.2: "any Bounding Values returned are included in the + numValuesPerNode count"; `NumValuesPerNode = 1` returns the start bound alone). The start bound leads the **first** page and the + end bound ends the **last**: the pending end bound rides the continuation point, and when a page is exactly full the end bound is + the whole next page. Paging a bounded read at any page size concatenates to the unpaged answer. +- **How it is served — one authority, `HistoryBounds` in the OPC UA server.** "At or before T" is a newest-first read of `(0, T]` + capped at one — native on SqlServer/QuestDB, the #686 walk-back of bounded forward reads on the Gateway (so a tag with no sample + before StartTime walks back to 1970: the same ~14–16 small reads as a deep reverse page-through). "At or after T" is a forward read + of `[T, MaxValue)` capped at one; "strictly after" starts one storage resolution later. On QuestDB, whose window bounds truncate to + the microsecond, "on the bound" compares at that resolution, so a sample at ⌊StartTime⌋ is never returned twice. A first page + therefore costs up to three backend reads (start bound, data, end bound). The `BoundingValueConformance` suite pins those exact + lookup reads on SqlServer and QuestDB. +- **Modified reads:** `IsReadModified = true` with `ReturnBounds = true` is `Bad_InvalidArgument` per node (Part 11 §6.5.3.3). +- **Not affected:** Processed, AtTime and Events reads (they have no `ReturnBounds`). ### Continuation-point paging (Events) (#669) @@ -1095,8 +1128,8 @@ whatever the SDK sets — #682 touches only the i=11201 folder (follow-up candid sample per requested timestamp, so the single-shot backend returns the complete result in one read and there is no "full page ⇒ maybe more" signal to page on. Returning the full result with no continuation point is spec-conformant. -- **`ReturnBounds` is ignored** (every backend): Raw reads return stored values only; no bounding values are added at the window - edges. The node manager does not pass the flag to `IHistorianDataSource`. +- **Bounds are raw only:** `ReturnBounds` (#691, [Bounding values](#bounding-values-returnbounds-691)) never interpolates a bound — + it is always a stored sample or `Bad_BoundNotFound`. - **No modified-value history on the `Gateway` backend.** Modified reads (`IsReadModified = true`) are served by `SqlServer` and `QuestDb` ([Modified-value history](#modified-value-history-part-11), #668); the HistorianGateway exposes no modified/edited-history surface, so on `Gateway` (and with no historian) they return `BadHistoryOperationUnsupported`.