diff --git a/docs/Historian.md b/docs/Historian.md index e237d115..194833ed 100644 --- a/docs/Historian.md +++ b/docs/Historian.md @@ -144,7 +144,7 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an | `MaxTieClusterOverfetch` | int | `65536` | (Node-manager paging; all backends — a `SqlServer` PK rules out tie clusters, so it is moot there.) Maximum samples the server will fetch in one shot to page through a tie cluster (multiple samples sharing one `SourceTimestamp`). A cluster larger than this ceiling fails `BadHistoryOperationUnsupported`. Raise to handle abnormally large tie clusters; the default covers all normal-data cases. | | `HistoryReadBatchParallelism` | int | `4` | Max historized nodes served **concurrently** within one HistoryRead batch (arch-review 03/S3). The SDK's `HistoryRead*` override surface is synchronous, so each arm bridges once at its boundary; this turns per-node serving from sequential (`N × CallTimeout` worst case) into `⌈N / P⌉ × CallTimeout`. Also caps the gateway-load multiplication a single client can command. `≤ 1` serves sequentially. | | `MaxConcurrentHistoryReadBatches` | int | `16` | Process-wide limit on HistoryRead (and HistoryUpdate) batches served **concurrently** (arch-review 03/S3). A flood of history clients degrades to `BadTooManyOperations` rather than exhausting the SDK request-thread pool. With `HistoryReadBatchParallelism` this caps total in-flight gateway reads at `MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism` (default 16 × 4 = 64). A batch that cannot get a slot within `min(HistoryReadDeadline, 5 s)` fails every handle `BadTooManyOperations`. | -| `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` only (#671). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped. 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). | +| `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` only (#671). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative, and — since #680 — **PercentGood/PercentBad**) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped. 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). | | `HistoryReadDeadline` | TimeSpan | `00:01:00` | Per-request wall-clock deadline for a HistoryRead (and HistoryUpdate) batch (arch-review 03/S3). A node still in flight at the deadline surfaces `BadTimeout`, so one request can never hold an SDK request thread longer than this regardless of node count. Non-positive = unbounded (a `Validate()` warning). | | `HistoryUpdateMaxFutureSkew` | TimeSpan | `1.00:00:00` | #674. How far past the server clock a HistoryUpdate (Insert / Replace / Update) item's `SourceTimestamp` may lie; an item beyond `now + skew` is refused **`BadOutOfRange`** by the node manager before any backend sees it (every backend). Guards against a far-future typo (a 2062 timestamp). The live recorder is not bounded. Non-positive disables the bound (no warning — a deliberate choice). See [HistoryUpdate](#historyupdate). | @@ -908,10 +908,10 @@ point (see [Continuation-point paging (Events)](#continuation-point-paging-event ### Aggregates (Processed reads) -> **Two kinds.** The *per-sample* aggregates (Average, Minimum, Maximum, Count, Range, Start/End/Delta, Min/MaxActualTime, -> PercentGood/Bad) are simple per-sample definitions — no time-weighting, no interval-boundary values — pushed down to SQL. The -> *time-weighted* aggregates (TimeAverage, TimeAverage2, Total, Total2, Interpolative; #671) follow OPC UA Part 13 with bounding values -> — see [Time-weighted aggregates](#time-weighted-aggregates-part-13-671). Any `HistoryAggregateType` not in the tables throws +> **Two kinds.** The *per-sample* aggregates (Average, Minimum, Maximum, Count, Range, Start/End/Delta, Min/MaxActualTime) are +> simple per-sample definitions — no time-weighting, no interval-boundary values — pushed down to SQL. The *time-weighted* aggregates +> (TimeAverage, TimeAverage2, Total, Total2, Interpolative; #671 — and the duration-based PercentGood/PercentBad since #680) follow +> OPC UA Part 13 with bounding values — see [Time-weighted aggregates](#time-weighted-aggregates-part-13-671). Any `HistoryAggregateType` not in the tables throws > `HistoryAggregateNotSupportedException` ⇒ `BadAggregateNotSupported` for that node (e.g. `StartBound`, `DurationGood`, > `StandardDeviationPopulation`). > @@ -951,7 +951,9 @@ returned status carries the historian *Calculated* info bit, `BadNoData` include result for the interval the tag's data **starts inside** — its earliest stored sample (any quality) lies strictly between the interval's edges, so there is no bounding value before it and the data does not cover the whole interval. This follows the SDK's `m_startOfData` rule, which the SDK turns off for `Average` and in effect applies to forward reads only (a reverse read is fed latest-first, so its -"start of data" is the latest sample, past the window). A reverse read therefore sets no start-of-data Partial — matching the SDK. Only a +"start of data" is the latest sample, past the window). A reverse read therefore sets no start-of-data Partial — matching the SDK +(the time-weighted family does set it, on a reverse read, on the interval holding the *latest* sample when nothing follows it — #680, +see [Time-weighted aggregates](#time-weighted-aggregates-part-13-671)). Only a forward per-sample read of an aggregate other than `Average` fetches the tag's earliest sample (one seek, bounded by the window end); every other read skips that query. An earliest sample exactly on an edge, or before the interval, sets nothing. The time-weighted aggregates also keep setting Partial on the truncated interval (below). @@ -967,23 +969,39 @@ aggregates also keep setting Partial on the truncated interval (below). | `Start`, `End` | first / last usable sample by time (keeps the sample's type) | bucket start | | `Delta` | `End − Start` (numeric samples) | bucket start | | `MinimumActualTime`, `MaximumActualTime` | the usable sample with the smallest / largest `NumValue`; ties ⇒ earliest | **the sample's own time** | -| `PercentGood` | `100 × (Good samples) / (all samples)` | bucket start | -| `PercentBad` | `100 × (Bad samples) / (all samples)` | bucket start | Status and edge rules: - An **empty interval** (no samples) ⇒ `BadNoData`. **`Count` — and every value aggregate — over an interval whose samples are all Bad ⇒ `BadNoData`**, not `0`. - Value aggregates are `Good` when every sample in the bucket is Good and `UncertainDataSubNormal` when the bucket also holds any Bad **or Uncertain** sample. -- `Count`, `Start`, `End`, `PercentGood`, `PercentBad` accept any value kind; the other aggregates (and `Delta`) on a bucket that holds a usable `String`/`DateTime` sample ⇒ `BadAggregateInvalidInputs` for that interval. -- **`PercentGood` / `PercentBad` and Uncertain samples:** Uncertain samples are counted in the denominator but are **neither Good nor Bad**, so `PercentGood + PercentBad < 100` whenever a bucket holds Uncertain samples. Both report status `Good` for any non-empty bucket. +- `Count`, `Start`, `End` accept any value kind; the other aggregates (and `Delta`) on a bucket that holds a usable `String`/`DateTime` sample ⇒ `BadAggregateInvalidInputs` for that interval. +- `PercentGood` / `PercentBad` are **duration-based since #680** — see [Time-weighted aggregates](#time-weighted-aggregates-part-13-671). - Everything except `Count` and `Start`/`End` is a `double`; `Int64` samples aggregate as doubles and `Boolean` as 0/1. ### Time-weighted aggregates (Part 13, #671) -`TimeAverage`, `TimeAverage2`, `Total`, `Total2` and `Interpolative`, evaluated by the shared -`Driver.Historian.Sql.Common/TimeWeightedAggregator.cs` for **both** SQL backends (so `SqlServer` and `QuestDb` agree exactly). -**This whole family — `Total` included — is evaluated in C#, not pushed down to SQL** (see *Pushdown trade-off* and the raw-row cap -below). Bucketing (edges, truncated interval, reverse anchoring, the 100 000-interval cap) is the same as above. +`TimeAverage`, `TimeAverage2`, `Total`, `Total2`, `Interpolative` and (since #680) `PercentGood` / `PercentBad`, evaluated by the +shared `Driver.Historian.Sql.Common/TimeWeightedAggregator.cs` for **both** SQL backends (so `SqlServer` and `QuestDb` agree exactly). +**This whole family — `Total` and the percentages included — is evaluated in C#, not pushed down to SQL** (see *Pushdown trade-off* +and the raw-row cap below). Bucketing (edges, truncated interval, reverse anchoring, the 100 000-interval cap) is the same as above. + +> ⚠️ **#680 (client-visible): time-weighted results now follow Part 13 more closely** (`SqlServer` / `QuestDb`; the SDK's own +> calculators agree except where noted under *SDK parity*): +> - **`TimeAverage` / `Total` status** is `UncertainDataSubNormal` whenever the interval uses an Uncertain value, a segment bridged +> across a Bad value, the stepped extrapolation past the last sample, or does not cover the interval — no longer the time-based +> rule. **The latest interval of a live tag now reads Uncertain, not Good**, and so does a fully-covered interval bridged across an +> outage. +> - **`TimeAverage` / `Total` over an interval wholly inside an outage** (data on both sides) now return the bridged value, +> Uncertain — they used to return generic `Bad`, value null. +> - **`Total` over an interval the data starts inside** integrates only the covered part (it used to be `TimeAverage × interval`). +> - **`TimeAverage2` / `Total2` no longer extrapolate past the end of the data** (the last sample, when nothing at or after the window +> end exists and it is usable): the interval holding it covers only up to it and is **Partial**; later intervals are **`BadNoData`** +> (they used to hold the last value, Good). A one-interval read over the whole history therefore averages the stored data only. +> - **`PercentGood` / `PercentBad` are duration-based** (they counted samples), moved to this family — so they are now subject to +> `ServerHistorian:MaxTimeWeightedRawRows` (`BadTooManyOperations` past the cap) — and an interval with no sample of its own between +> two samples now has a value instead of `BadNoData`. +> - **Partial:** set on the forward end-of-data interval (`TimeAverage2`/`Total2`/percentages) and, on a reverse read, on the interval +> holding the latest sample when nothing follows it (all of this family but `Interpolative`). **Server defaults, fixed:** `TreatUncertainAsBad = false`, `PercentDataBad = PercentDataGood = 100`, `UseSlopedExtrapolation = false` (`ServerAggregateConfiguration` in the OPC UA server project — the SDK's own `AggregateManager` defaults, which a test pins). **A @@ -1001,40 +1019,71 @@ still gets a value. **The signal.** Between a usable sample `p` and the next sample `s`: both `Double`/`Int64` ⇒ **sloped** (linear), segment quality the worse of the two; either `Boolean` ⇒ **stepped** (`p` held), quality `p`'s; `s` Bad ⇒ `p` held up to the outage (never interpolated into -it — the AtTime rule), Uncertain; no later sample at all ⇒ `p` held to the window end (stepped extrapolation), Uncertain. From a Bad -sample (incl. an `Empty` outage marker) to the next sample the signal is **Bad** (outage); before the first sample there is **no -data**. That is the -*simple-bounding* signal. The *interpolated-bounding* signal skips Bad samples: it joins consecutive usable samples (sloped / stepped) -straight across an outage. +it — the AtTime rule), Uncertain. From a Bad sample (incl. an `Empty` outage marker) to the next sample the signal is **Bad** +(outage) — an outage marker with nothing after it runs to the window end; before the first sample there is **no data**; after a +*usable* last sample with nothing at or after the window end the data has **ended** (#680, Part 13 "After End of Data": no +extrapolation). That is the *simple-bounding* signal. The *interpolated-bounding* signal skips Bad samples: it joins consecutive usable +samples (sloped / stepped) straight across an outage, and holds the last usable value to the window end (stepped extrapolation). | Aggregate | Value | Bad intervals | |---|---|---| | `TimeAverage` | ∫ interpolated-bounding signal ÷ its defined duration | interpolated across (Part 13) | -| `Total` | `TimeAverage × interval seconds` (value·s; the truncated last bucket uses its own length) | interpolated across | +| `Total` | ∫ interpolated-bounding signal over its defined part (value·s) — `TimeAverage × interval seconds` on a covered interval; only the covered part when the data starts inside (#680) | interpolated across | | `TimeAverage2` | ∫ simple-bounding signal over non-Bad time ÷ non-Bad duration | excluded | | `Total2` | ∫ simple-bounding signal over non-Bad time (value·s) | excluded | | `Interpolative` | interpolated-bounding value **at the interval start** (reverse read: see below) | interpolated across | +| `PercentGood` / `PercentBad` | `100 ×` Good / Bad time ÷ interval duration (#680, Part 13 §5.4.3.31-34) — see below | counted as Bad time | -**Status, the four time-weighted values** — aligned with the OPC Foundation SDK's `AggregateCalculator.GetTimeBasedStatusCode` -(inspected in `Opc.Ua.Server` 1.5.378.106) under the fixed defaults. With `TreatUncertainAsBad = false`, **Uncertain time counts as -Good time**: Uncertain samples, a value held up to an outage and the stepped extrapolation past the last sample all count toward -PercentDataGood. +**Status, `TimeAverage` / `Total`** (interpolated bounding, #680 — Part 13 "Uncertain if Bad values are skipped or Uncertain values +are used", and the SDK's `AverageAggregateCalculator`): -- Every instant of the interval non-Bad ⇒ `Good`. +- No defined value anywhere in the interval (before the first usable sample) ⇒ `BadNoData`. +- Any part of the interval uses an Uncertain value (either end of a sloped segment, the start of a stepped one), lies on a segment + bridged across a Bad sample, is the stepped extrapolation past the last usable sample, or is not covered (the data starts inside it) + ⇒ `UncertainDataSubNormal`. A segment bridged across an outage marks **every** interval it touches, not just the one holding the Bad + sample. +- Otherwise ⇒ `Good`. + +**Status, `TimeAverage2` / `Total2`** — the SDK's `AggregateCalculator.GetTimeBasedStatusCode` (inspected in `Opc.Ua.Server` +1.5.378.106) under the fixed defaults. With `TreatUncertainAsBad = false`, **Uncertain time counts as Good time** (Uncertain samples, a +value held up to an outage). The duration is the interval's, cut at the end of the data: + +- After the end of the data ⇒ `BadNoData` (Part 13 "After End of Data"); the interval the data ends inside covers only up to the last + sample and is **Partial** (forward); when the data ends exactly on its start there is nothing to cover and the value is the last + sample's (`Total2` 0), with its status. +- Every instant of the (covered) interval non-Bad ⇒ `Good`. - Some non-Bad time but not all (an outage, or time before the first sample) ⇒ `UncertainDataSubNormal`. -- No non-Bad time but some outage time ⇒ generic **`Bad`** (`0x80000000`) — the SDK's all-Bad result, value null. +- No non-Bad time but some outage time (an interval wholly inside an outage) ⇒ generic **`Bad`** (`0x80000000`), value null — Part 13's + PercentDataBad rule. (The SDK answers `BadNoData` there; see *SDK parity*.) - No data at all ⇒ `BadNoData`. -- A usable `String`/`DateTime` sample feeding the interval ⇒ `BadAggregateInvalidInputs`. -All carry the Calculated bit; a non-Bad result for the **truncated interval** (shorter than the processing interval — the latest one -forward, the earliest one reverse) or for the interval the tag's **data starts inside** (#677, see *Partial bit* above) also carries the -**Partial** bit (`0x4`, `AggregateBits.Partial`). Timestamp = interval start (reverse: its later edge); value `double` (Boolean 0/1). +For all four, a usable `String`/`DateTime` sample feeding the interval ⇒ `BadAggregateInvalidInputs`. -> **A live tag's latest interval.** Past the tag's last stored sample the value is held (stepped extrapolation, Uncertain). Under -> the SDK rule that Uncertain time is Good time, so the latest interval of a live tag normally reads **`Good`** — not Uncertain — -> with the last value held to the interval end. It reads Uncertain only if the interval also contains outage time or starts -> before the tag's first sample. Do not take a Good latest interval as proof that data arrived up to its end. If it was truncated -> by the request's end time it also carries **Partial**. +**`PercentGood` / `PercentBad`** (#680, Part 13 §5.4.3.31-34 — they used to count samples). The interval is cut into **stepped +regions**, each with the status of the sample that starts it: Good regions count toward `PercentGood`, Bad ones (incl. outage markers) +toward `PercentBad`, Uncertain ones toward neither (so the two sum to < 100 when Uncertain data is present). The region that starts an +interval before its first own sample takes the status of the *simple bounding value* there (the SDK's `GetSimpleBound`): Bad after a +Bad sample, Uncertain on a segment that runs into an outage or slopes to/from an Uncertain sample, else the earlier sample's. Time +before the tag's first sample is **Bad**. When the data ends (a usable last sample, nothing at or after the window end) its region runs +to the **end of its own interval** (Part 13 — the SDK gives it 1 ms) and later intervals are `BadNoData`; an outage marker with nothing +after it is Bad to the window end. An interval that ends before the first sample ⇒ `BadNoData`. Any value kind is accepted; the status +is `Good` (plus Calculated / Partial). Value = `100 × region time ÷ interval duration`. + +All carry the Calculated bit. A non-Bad result also carries the **Partial** bit (`0x4`, `AggregateBits.Partial`) — never on a Bad +result — for: the **truncated interval** (shorter than the processing interval — the latest one forward, the earliest one reverse; +Part 13 §5.3.3.2 Case 3 requires it — the SDK does not set it, a deliberate difference); the interval the tag's **data starts inside** +(#677, forward, see *Partial bit* above); on a **reverse** read, the interval the **latest** sample lies strictly inside when nothing +follows it (where a reverse read's data starts — the SDK's `m_startOfData`, keyed on the first value it is fed, #680); and, forward, +the **end-of-data** interval of `TimeAverage2` / `Total2` / the percentages (the SDK's `m_endOfData` rule, which it skips for an +extrapolated `TimeAverage` / `Total`). Timestamp = interval start (reverse: its later edge); value `double` (Boolean 0/1). + +> **A live tag's latest interval (#680 — changed, client-visible).** Past the tag's last stored sample `TimeAverage` / `Total` hold the +> last value to the interval end (stepped extrapolation), and that interval now reads **`UncertainDataSubNormal`** — the extrapolated +> part is not data. (It used to read `Good`, because the time-based rule counted the extrapolated time as Good time.) `TimeAverage2` / +> `Total2` do not extrapolate: they average / integrate only up to the last sample and set **Partial**; any later interval is +> `BadNoData`. A `Good` latest interval therefore now does mean the data covered it — for these aggregates, the latest interval of a +> tag that has not yet received a sample at or after the request's end time is never `Good` (`TimeAverage`/`Total`) or always +> `Partial` (`TimeAverage2`/`Total2`, forward). If it was truncated by the request's end time it also carries **Partial**. **Status, `Interpolative`** (point-based, not the PercentDataGood rule): - A usable sample exactly at the point ⇒ that value with **its own stored status** (no historian bit). @@ -1051,19 +1100,25 @@ value at **10:00**, not 09:00, and no point ever falls before the window. The in **SDK parity (#677).** `SdkAggregateParityTests` (unit, `Sql.Common.Tests`) and `ProcessedConformance.Processed_reads_match_the_SDK_aggregate_calculators` (both SQL backends, through the real SQL) run the pinned SDK's own calculators (`AverageAggregateCalculator`, `MinMaxAggregateCalculator`, `CountAggregateCalculator`, the base `AggregateCalculator` -for Interpolative) over the same samples, forward and reverse, and compare every interval's timestamp, value, status code and Partial -bit. The per-sample comparison covers Average, Minimum, Maximum, Count, Range, Start, End, Delta, Minimum/MaximumActualTime and -PercentGood/Bad; reverse reads over data that starts inside the window are compared too. Known differences that remain (tracked as +for Interpolative, `StatusAggregateCalculator` for PercentGood/Bad) over the same samples, forward and reverse, and compare every +interval's timestamp, value, status code and Partial bit. The per-sample comparison covers Average, Minimum, Maximum, Count, Range, +Start, End, Delta and Minimum/MaximumActualTime; the time-weighted one TimeAverage, TimeAverage2, Total, Total2, Interpolative and +PercentGood/Bad (#680), including datasets whose data ends inside the window and with an interval wholly inside an outage; reverse +reads over data that starts inside the window are compared too. Known differences that remain (tracked as #680) — documented and masked in the tests, deliberately not changed here because forward results were to stay as they are — are: `Start`/`End` (the SDK returns the raw first/last sample, of any quality, with its own timestamp and stored status; ours returns the first/last usable value stamped with the interval, under the per-sample status rule — the values agree wherever the SDK's is not Bad); -`Delta` status (the SDK's stays Good with a Bad sample in the interval, ours is Uncertain); `PercentGood`/`PercentBad` (ours count -samples, the SDK measures duration — different definitions, so only timestamp and status are compared); the SDK reports `TimeAverage`/`Total` Uncertain whenever any region of the interval is non-Good, where ours applies the -time-based rule (a fully-covered interval bridged across a Bad sample is Good here); `Total` over an interval the data starts inside is -`TimeAverage × interval` here but the integral of the covered part in the SDK; ours flags a truncated interval Partial, the SDK does -not; the SDK's forward end-of-data Partial rule is not implemented; the SDK's per-sample Count answers 0/Good for an empty interval -inside the data and its per-sample aggregates use Good samples only; and on a reverse `Interpolative` read the SDK marks a sample exactly -at a point Interpolated (Uncertain if a Bad sample follows) where ours — like the SDK forward — returns the raw sample. +`Delta` status (the SDK's stays Good with a Bad sample in the interval, ours is Uncertain); the SDK's per-sample Count answers +0/Good for an empty interval inside the data and its per-sample aggregates use Good samples only. Time-weighted differences (#680 closed +the TimeAverage/Total status, `Total` coverage, end-of-data and PercentGood/Bad ones), each deliberate: ours flags a truncated interval +Partial (Part 13 §5.3.3.2 Case 3), the SDK does not; ours carries the Calculated bit on a `BadNoData`; `TimeAverage2`/`Total2` over an +interval wholly inside an outage are generic `Bad` here (Part 13's PercentDataBad rule), `BadNoData` in the SDK; in the end-of-data +interval the SDK gives the region after the last sample 1 ms (so `Total2` there differs by ~1 ms × value and the percentages by a lot — +ours follow Part 13: `TimeAverage2`/`Total2` stop at the last sample, the percentages' region runs to the interval end); and on a reverse +read the SDK's backward slice files a sample exactly on an interval's later edge as its early bound — so a reverse `Interpolative` at +such a point comes back Interpolated (Uncertain if a Bad sample follows) and a reverse `TimeAverage`/`Total` interval ending on it takes +the status of the segment beyond it — where ours, like the SDK forward, returns the raw sample and keeps the interval's own segments +(an SDK defect: its forward and reverse answers differ). **Pushdown trade-off.** Unlike the per-sample aggregates, the time-weighted family is **not** pushed down to SQL: the backend streams the window's rows plus the four bounds (one batch of five result sets on SQL Server; a bounds `UNION ALL` + a window select on QuestDB) @@ -1077,7 +1132,8 @@ judged the larger risk. For very long, densely sampled windows prefer the pushed SQL backends; `TOP (@maxRows)` / `LIMIT $4`). When that extra row arrives the read stops and the node fails with **`BadTooManyOperations`** — for that node only; it is not counted in the backend's health failures. Without the cap a one-year `Total` on a 1 Hz tag (≈31 M rows) would hold a pooled connection until `HistoryReadDeadline` and then return `BadTimeout`. The four -bound rows do not count toward the cap, and the per-sample aggregates are never capped. Raise the cap, or read in shorter windows. +bound rows do not count toward the cap, and the per-sample aggregates are never capped. Since #680 `PercentGood`/`PercentBad` are in +this family and **are** capped (they used to be pushed down). Raise the cap, or read in shorter windows. ### Health diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AggregateFamily.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AggregateFamily.cs index 4b94d3d8..18066b09 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AggregateFamily.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/AggregateFamily.cs @@ -5,7 +5,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; /// The query shape that serves an aggregate. One query per family. public enum AggregateFamily { - /// Average / Minimum / Maximum / Count / Range / PercentGood / PercentBad: one GROUP BY. + /// Average / Minimum / Maximum / Count / Range: one GROUP BY. Simple, /// Start / End / Delta: first + last usable sample per bucket. @@ -15,7 +15,8 @@ public enum AggregateFamily ActualTime, /// - /// TimeAverage / TimeAverage2 / Total / Total2 / Interpolative: Part 13 time-weighted, with bounding values. + /// TimeAverage / TimeAverage2 / Total / Total2 / Interpolative, and the duration-based PercentGood / PercentBad + /// (#680): Part 13 time-weighted, with bounding values. /// The backend streams the window's samples plus the bounding rows and /// evaluates them (identically for every backend). /// @@ -31,12 +32,12 @@ public static class AggregateFamilies public static AggregateFamily? FamilyOf(HistoryAggregateType aggregate) => aggregate switch { HistoryAggregateType.Average or HistoryAggregateType.Minimum or HistoryAggregateType.Maximum - or HistoryAggregateType.Count or HistoryAggregateType.Range - or HistoryAggregateType.PercentGood or HistoryAggregateType.PercentBad => AggregateFamily.Simple, + or HistoryAggregateType.Count or HistoryAggregateType.Range => AggregateFamily.Simple, HistoryAggregateType.Start or HistoryAggregateType.End or HistoryAggregateType.Delta => AggregateFamily.StartEnd, HistoryAggregateType.MinimumActualTime or HistoryAggregateType.MaximumActualTime => AggregateFamily.ActualTime, HistoryAggregateType.TimeAverage or HistoryAggregateType.TimeAverage2 or HistoryAggregateType.Total - or HistoryAggregateType.Total2 or HistoryAggregateType.Interpolative => AggregateFamily.TimeWeighted, + or HistoryAggregateType.Total2 or HistoryAggregateType.Interpolative + or HistoryAggregateType.PercentGood or HistoryAggregateType.PercentBad => AggregateFamily.TimeWeighted, _ => null, }; } diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/TimeWeightedAggregator.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/TimeWeightedAggregator.cs index d7a78a98..5f0d6446 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/TimeWeightedAggregator.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/TimeWeightedAggregator.cs @@ -4,8 +4,8 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; /// /// The OPC UA Part 13 time-weighted aggregates (): TimeAverage, -/// TimeAverage2, Total, Total2 and Interpolative, evaluated over a time-ordered stream of -/// stored samples. Both SQL backends fetch the same rows and feed them here, so the two produce identical results +/// TimeAverage2, Total, Total2, Interpolative and the duration-based PercentGood / +/// PercentBad (#680), evaluated over a time-ordered stream of stored samples. Both SQL backends fetch the same rows and feed them here, so the two produce identical results /// by construction. /// /// @@ -24,10 +24,12 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; /// s usable, both Double/Int64 ⇒ sloped (linear p → s); quality = worse of the two. /// s usable, either Boolean ⇒ stepped (p held); quality = p's. /// s Bad ⇒ p held up to the outage (never interpolated into it, as AtTime does); Uncertain. -/// no later sample at all ⇒ p held to the window end (stepped extrapolation); Uncertain. +/// no later sample at all (nothing at or after the window end) ⇒ the data has ENDED at p: no +/// extrapolation (Part 13 "After End of Data", #680). /// -/// From a Bad sample (incl. an Empty outage marker) to the next sample, and before the first sample, the -/// signal is Bad (no data). This is the simple bounding signal. +/// From a Bad sample (incl. an Empty outage marker) to the next sample the signal is Bad (outage) — to the +/// window end when nothing follows it; before the first sample there is no data. This is the simple bounding +/// signal. /// /// /// The interpolated bounding signal (Part 13 "interpolated bounding values") skips Bad samples: it joins @@ -36,32 +38,58 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; /// /// /// TimeAverage∫ interpolated-bounding signal ÷ its defined duration in the interval. -/// TotalTimeAverage × interval seconds (value·s) — Part 13 §5.4.3.8. +/// Total∫ interpolated-bounding signal over its defined part, in value·s — Part 13 +/// §5.4.3.8 (= TimeAverage × interval seconds on a covered interval; only the covered part when the data +/// starts inside it, #680). /// TimeAverage2∫ simple-bounding signal over its non-Bad parts ÷ non-Bad duration (Bad intervals excluded). /// Total2∫ simple-bounding signal over its non-Bad parts, in value·s (= TimeAverage2 × non-Bad seconds). +/// PercentGood / PercentBad100 × Good / Bad region time ÷ interval duration +/// (Part 13 §5.4.3.31-34): stepped regions, each with the status of the sample that starts it; the region before an +/// interval's first own sample has the simple bounding value's status (Bad after a Bad sample; Uncertain on a +/// segment running into an outage or sloping to/from an Uncertain sample; else the earlier sample's — the SDK's +/// GetSimpleBound); time before the first sample is Bad; at the end of the data the last region runs to the +/// end of its own interval (Part 13; the SDK gives it 1 ms) and later intervals are BadNoData; an interval +/// ending before the first sample ⇒ BadNoData. Status Good; any value kind. /// InterpolativeThe interpolated-bounding signal's value at each interval start — /// for a reverse read, at End − i·interval (Part 13 anchors a reverse read at its later time), emitted /// latest first. /// /// -/// Status (time-weighted four) — the SDK AggregateCalculator.GetTimeBasedStatusCode rule with -/// TreatUncertainAsBad = false, so Uncertain time (Uncertain samples, a value held up to an outage, stepped -/// extrapolation past the last sample) counts as Good time: every instant of the interval non-Bad ⇒ -/// Good; no non-Bad time but some Bad (outage) time ⇒ Bad; no data at all ⇒ BadNoData; -/// otherwise (PercentDataGood = PercentDataBad = 100) UncertainDataSubNormal. Time before the first sample -/// counts as Bad (no data) time. A usable String/DateTime sample contributing to the interval ⇒ -/// BadAggregateInvalidInputs. Every status carries the Calculated bit, plus the Partial bit on a non-Bad -/// result of the truncated interval or of the interval the data starts inside (the first sample fed lies strictly -/// within it — forward reads only, as in the SDK, #677); the timestamp is the interval's (its start forward, its -/// later edge reverse); the value is a -/// (Boolean = 0/1, Int64 as double). +/// Status (TimeAverage / Total, #680) — Part 13 ("Uncertain if Bad values are skipped or Uncertain values are +/// used") and the SDK's AverageAggregateCalculator: no defined value in the interval ⇒ BadNoData; any +/// part of it on a bridged segment with an Uncertain end (or Uncertain start when stepped), bridged across a Bad +/// sample, extrapolated past the last usable sample, or not covered at all ⇒ UncertainDataSubNormal (so the +/// latest interval of a live tag is Uncertain, and an interval wholly inside an outage has the bridged value, +/// Uncertain); else Good. +/// +/// +/// Status (TimeAverage2 / Total2) — the SDK AggregateCalculator.GetTimeBasedStatusCode rule with +/// TreatUncertainAsBad = false, so Uncertain time (Uncertain samples, a value held up to an outage) counts as Good +/// time, over the interval cut at the end of the data: after the end of the data ⇒ BadNoData; the data +/// ending exactly on the interval start ⇒ the last sample's value (Total2 0) with its status; every covered instant +/// non-Bad ⇒ Good; no non-Bad time but some Bad (outage) time ⇒ Bad (Part 13's PercentDataBad rule; +/// the SDK answers BadNoData); no data at all ⇒ BadNoData; otherwise (PercentDataGood = PercentDataBad +/// = 100) UncertainDataSubNormal. Time before the first sample counts as Bad (no data) time. +/// +/// +/// For the four, a usable String/DateTime sample contributing to the interval ⇒ BadAggregateInvalidInputs. +/// Every status of this family but Interpolative's carries the Calculated bit, plus — on a non-Bad result only — the +/// Partial bit on: the truncated interval (Part 13 §5.3.3.2 Case 3; the SDK does not set it — deliberate); the +/// interval the data starts inside in the read direction (forward: the first sample fed lies strictly within it, +/// #677; reverse: the latest sample fed lies strictly within it, nothing after — the SDK's m_startOfData, +/// keyed on the first value it is fed, #680); and, forward, the end-of-data interval of TimeAverage2 / Total2 / +/// PercentGood / PercentBad (the SDK's m_endOfData, which it skips for an extrapolated TimeAverage / Total). +/// The timestamp is the interval's (its start forward, its later edge +/// reverse); the value is a (Boolean = 0/1, Int64 as double). /// /// /// Status (Interpolative). A usable sample exactly at the point ⇒ its value with its own status /// (raw, no historian bit). Otherwise no usable sample at or before ⇒ BadNoData; else the interpolated /// value with the Interpolated bit and the status of the simple-bounding signal at that instant: Good on /// a Good segment, UncertainDataSubNormal on an Uncertain one, inside an outage (interpolated across it) -/// or past the last sample (extrapolated). +/// or past the last sample (extrapolated). A reverse read returns the raw sample at a point exactly on a sample +/// too — the SDK's reverse answer (Interpolated, from the segment beyond) differs from its own forward one; that +/// asymmetry is the SDK's defect, deliberately not copied. /// /// public sealed class TimeWeightedAggregator @@ -108,13 +136,19 @@ public sealed class TimeWeightedAggregator // Interpolated-bounding (bridged) signal, per bucket. private readonly double[] _bridgedArea; private readonly long[] _bridgedTicks; + // A bridged segment touching the bucket was non-Good: an Uncertain end, a Bad sample skipped, or extrapolated (#680). + private readonly bool[] _bridgedNonGood; private readonly bool[] _invalid; + // PercentGood / PercentBad (#680): stepped status regions, per bucket. + private readonly long[] _goodRegionTicks; + private readonly long[] _badRegionTicks; // Interpolative, per point (index = ascending time; see PointTicks). private readonly DataValueSnapshot?[] _points; private StoredSample? _prevAny; private StoredSample? _prevUsable; private DateTime? _firstSampleUtc; // the earliest sample fed (any quality): where the data starts + private long? _dataEndTicks; // the last sample fed, when usable and nothing at/after the end followed private long _nextPoint; // next point (ascending time) not yet classified private readonly List _pending = []; // points awaiting the next usable sample private bool _finished; @@ -146,7 +180,10 @@ public sealed class TimeWeightedAggregator _badTicks = new long[n]; _bridgedArea = new double[n]; _bridgedTicks = new long[n]; + _bridgedNonGood = new bool[n]; _invalid = new bool[n]; + _goodRegionTicks = new long[n]; + _badRegionTicks = new long[n]; _points = new DataValueSnapshot?[n]; } @@ -172,7 +209,11 @@ public sealed class TimeWeightedAggregator if (_finished) throw new InvalidOperationException("The aggregation is already finished."); var t = s.TimeUtc.Ticks; if (_prevAny is { } last && t <= last.TimeUtc.Ticks) return; - _firstSampleUtc ??= s.TimeUtc; + if (_firstSampleUtc is null) + { + _firstSampleUtc = s.TimeUtc; + AddRegion(_badRegionTicks, _start, t); // PercentBad: the time before the first sample is Bad (no data) + } // Interpolative: classify every bucket start before this sample by the simple-bounding signal there. ClassifyPointsBefore(t); @@ -188,6 +229,7 @@ public sealed class TimeWeightedAggregator { AddBad(bad, t); } + if (_prevAny is { } r) AddStatusRegion(r, s, t); if (!s.IsUsable) DegradePending(); // the segment the pending points sit on runs into an outage @@ -196,7 +238,10 @@ public sealed class TimeWeightedAggregator if (_prevUsable is { } pu) { var sloped = IsSloped(pu, s); - AddBridged(pu, t, sloped ? s.Value.Number : pu.Value.Number); + // Part 13: Uncertain when a Bad value was skipped or an Uncertain value is used (the end sample only + // contributes its value to a sloped segment). + var nonGood = !pu.IsGood || (sloped && !s.IsGood) || _prevAny is { IsUsable: false }; + AddBridged(pu, t, sloped ? s.Value.Number : pu.Value.Number, nonGood); } ResolvePending(_prevUsable, s); _prevUsable = s; @@ -215,11 +260,21 @@ public sealed class TimeWeightedAggregator _finished = true; ClassifyPointsBefore(long.MaxValue); if (_prevAny is { IsUsable: true } p && p.TimeUtc.Ticks < _end) - AddRaw(p, _end, p.Value.Number); // stepped extrapolation + { + // No sample at or after the window end: the data ENDS here (Part 13 "After End of Data"). The simple- + // bounding signal is not extrapolated (TimeAverage2/Total2 cover up to it; later intervals BadNoData), and + // the status region it starts runs to the end of its own interval only (PercentGood/PercentBad). + _dataEndTicks = p.TimeUtc.Ticks; + if (p.TimeUtc.Ticks >= _start && _plan.BucketCount > 0) + AddRegion(RegionTicks(p), p.TimeUtc.Ticks, BucketEnd(FirstBucketAt(p.TimeUtc.Ticks))); + } else if (_prevAny is { } bad && bad.TimeUtc.Ticks < _end) - AddBad(bad, _end); // the outage runs to the window end + { + AddBad(bad, _end); // an outage marker last: the outage runs to the window end + AddRegion(_badRegionTicks, bad.TimeUtc.Ticks, _end); + } if (_prevUsable is { } pu && pu.TimeUtc.Ticks < _end) - AddBridged(pu, _end, pu.Value.Number); + AddBridged(pu, _end, pu.Value.Number, nonGood: true); // stepped extrapolation ⇒ Uncertain ResolvePending(_prevUsable, next: null); } @@ -239,30 +294,77 @@ public sealed class TimeWeightedAggregator : new DataValueSnapshot(null, HistorianStatusCodes.BadNoData, PointTime(i), now); var stamp = _plan.BucketTimestamp(i); - var duration = BucketEnd(i) - BucketStartTicks(i); - // Partial: the truncated interval (the latest forward, the earliest reverse), or the one the data starts inside - // (no bounding value before it — the SDK's m_startOfData rule, #677), which the SDK applies to forward reads only. - var partial = _plan.IsTruncated(i) || (!_plan.Reversed && _plan.DataStartsInside(i, _firstSampleUtc)); - DataValueSnapshot Calc(object? value, uint status) + var lo = BucketStartTicks(i); + var hi = BucketEnd(i); + var duration = hi - lo; + // Partial: the truncated interval (the latest forward, the earliest reverse — Part 13 §5.3.3.2 Case 3, which the + // SDK does not implement), or the one the data starts inside IN THE READ DIRECTION (no bounding value on that + // side — the SDK's m_startOfData rule, #677): forward, the earliest sample strictly inside; reverse, the latest + // sample strictly inside (nothing after it — the SDK keys m_startOfData on the first value it is fed, #680). + // Forward, the end-of-data interval (the latest sample inside it, nothing after) is Partial too — never on an + // extrapolated TimeAverage/Total (the SDK's m_endOfData rule, #680). Never on a Bad result. + var latestInside = _plan.Reversed && _prevAny is { } latest + && latest.TimeUtc.Ticks > lo && latest.TimeUtc.Ticks < hi; + var partial = _plan.IsTruncated(i) || latestInside + || (!_plan.Reversed && _plan.DataStartsInside(i, _firstSampleUtc)); + var endOfData = !_plan.Reversed && _dataEndTicks is { } de && de >= lo && de < hi; + DataValueSnapshot Calc(object? value, uint status, bool endPartial = false) { var code = HistorianInfoBits.WithCalculated(status); - if (partial && !StatusBits.IsBad(status)) code |= HistorianInfoBits.HistorianPartial; + if ((partial || endPartial) && !StatusBits.IsBad(status)) code |= HistorianInfoBits.HistorianPartial; return new DataValueSnapshot(value, code, stamp, now); } - if (_invalid[i]) return Calc(null, HistorianStatusCodes.BadAggregateInvalidInputs); - if (_usableTicks[i] == 0) - return _badTicks[i] > 0 ? Calc(null, HistorianStatusCodes.Bad) : AggregateEvaluator.NoData(stamp, now); - - var status = _usableTicks[i] == duration ? HistorianStatusCodes.Good : HistorianStatusCodes.UncertainDataSubNormal; - return _aggregate switch + switch (_aggregate) { - HistoryAggregateType.TimeAverage => Calc(_bridgedArea[i] / _bridgedTicks[i], status), - HistoryAggregateType.Total => Calc(_bridgedArea[i] / _bridgedTicks[i] * (duration / TicksPerSecond), status), - HistoryAggregateType.TimeAverage2 => Calc(_rawArea[i] / _usableTicks[i], status), - HistoryAggregateType.Total2 => Calc(_rawArea[i] / TicksPerSecond, status), - _ => throw new HistoryAggregateNotSupportedException(_aggregate), - }; + case HistoryAggregateType.TimeAverage or HistoryAggregateType.Total: + { + // Interpolated bounding: Uncertain when any part of the interval was bridged across a Bad value, used an + // Uncertain value, was extrapolated, or is not covered at all (Part 13; the SDK's AverageAggregateCalculator). + if (_invalid[i]) return Calc(null, HistorianStatusCodes.BadAggregateInvalidInputs); + if (_bridgedTicks[i] == 0) return AggregateEvaluator.NoData(stamp, now); + var status = _bridgedNonGood[i] || _bridgedTicks[i] < duration + ? HistorianStatusCodes.UncertainDataSubNormal + : HistorianStatusCodes.Good; + return _aggregate == HistoryAggregateType.TimeAverage + ? Calc(_bridgedArea[i] / _bridgedTicks[i], status) + : Calc(_bridgedArea[i] / TicksPerSecond, status); // ∫ over the covered part only (Part 13 partial width) + } + case HistoryAggregateType.TimeAverage2 or HistoryAggregateType.Total2: + { + if (_invalid[i]) return Calc(null, HistorianStatusCodes.BadAggregateInvalidInputs); + if (_dataEndTicks is { } dataEnd) + { + // After the end of the data: nothing to aggregate (Part 13 "After End of Data: Bad_NoData"). + if (lo > dataEnd) return AggregateEvaluator.NoData(stamp, now); + if (lo == dataEnd && _prevAny is { } last) + { + // The data ends exactly on this interval's start: nothing covered; the value at that instant. + if (!IsNumeric(last)) return Calc(null, HistorianStatusCodes.BadAggregateInvalidInputs); + var st = last.IsGood ? HistorianStatusCodes.Good : HistorianStatusCodes.UncertainDataSubNormal; + return Calc(_aggregate == HistoryAggregateType.TimeAverage2 ? last.Value.Number : 0d, st, endOfData); + } + duration = Math.Min(hi, dataEnd) - lo; + } + if (_usableTicks[i] == 0) + return _badTicks[i] > 0 ? Calc(null, HistorianStatusCodes.Bad) : AggregateEvaluator.NoData(stamp, now); + var status = _usableTicks[i] == duration ? HistorianStatusCodes.Good : HistorianStatusCodes.UncertainDataSubNormal; + return _aggregate == HistoryAggregateType.TimeAverage2 + ? Calc(_rawArea[i] / _usableTicks[i], status, endOfData) + : Calc(_rawArea[i] / TicksPerSecond, status, endOfData); + } + case HistoryAggregateType.PercentGood or HistoryAggregateType.PercentBad: + { + // No data in or around the interval (it ends before the first sample — the SDK's out-of-range test — or + // starts after the end of the data) ⇒ BadNoData; otherwise the share of the interval's duration. + if (_firstSampleUtc is not { } first || hi < first.Ticks || (_dataEndTicks is { } de2 && lo > de2)) + return AggregateEvaluator.NoData(stamp, now); + var ticks = _aggregate == HistoryAggregateType.PercentGood ? _goodRegionTicks[i] : _badRegionTicks[i]; + return Calc(100d * ticks / duration, HistorianStatusCodes.Good, endOfData); + } + default: + throw new HistoryAggregateNotSupportedException(_aggregate); + } } // ------------------------------------------------------------------------------------------ Segments @@ -305,13 +407,68 @@ public sealed class TimeWeightedAggregator } } - private void AddBridged(StoredSample from, long toTicks, double toValue) => + private void AddBridged(StoredSample from, long toTicks, double toValue, bool nonGood) => Distribute(from, toTicks, toValue, (i, area, ticks) => { _bridgedArea[i] += area; _bridgedTicks[i] += ticks; + if (nonGood) _bridgedNonGood[i] = true; }); + // ------------------------------------------------------------------------------------------ PercentGood / PercentBad + + /// The region array a stepped region of 's status counts toward (null: Uncertain, neither). + private long[]? RegionTicks(StoredSample s) => + s.IsGood ? _goodRegionTicks : !s.IsUsable ? _badRegionTicks : null; + + /// + /// The status regions between sample and the next sample (at + /// ). Inside the interval lies in, the region has 's + /// status (stepped, Part 13 §5.4.3.31-34). In a later interval the region starts at that interval's simple bounding + /// value, whose status is the simple-bounding signal's: Bad after a Bad ; Uncertain when the + /// segment runs into an outage or a sloped one has a non-Good end; else 's (the SDK's + /// GetSimpleBound). + /// + private void AddStatusRegion(StoredSample p, StoredSample s, long t) + { + var lo = Math.Max(p.TimeUtc.Ticks, _start); + var hi = Math.Min(t, _end); + if (lo >= hi || _plan.BucketCount == 0) return; + long[]? bound; + if (!p.IsUsable) bound = _badRegionTicks; + else if (!s.IsUsable) bound = null; // held into an outage ⇒ Uncertain + else if (IsSloped(p, s)) bound = p.IsGood && s.IsGood ? _goodRegionTicks : null; + else bound = RegionTicks(p); + var own = RegionTicks(p); + for (var i = FirstBucketAt(lo); i < _plan.BucketCount; i++) + { + var bs = BucketStartTicks(i); + if (bs >= hi) break; + var a = Math.Max(lo, bs); + var b = Math.Min(hi, BucketEnd(i)); + if (a >= b) continue; + var target = p.TimeUtc.Ticks >= bs ? own : bound; + if (target is not null) target[i] += b - a; + } + } + + /// Adds [from, to), clipped to the window, to per bucket. + private void AddRegion(long[]? target, long from, long to) + { + if (target is null) return; + var lo = Math.Max(from, _start); + var hi = Math.Min(to, _end); + if (lo >= hi || _plan.BucketCount == 0) return; + for (var i = FirstBucketAt(lo); i < _plan.BucketCount; i++) + { + var bs = BucketStartTicks(i); + if (bs >= hi) break; + var a = Math.Max(lo, bs); + var b = Math.Min(hi, BucketEnd(i)); + if (a < b) target[i] += b - a; + } + } + /// /// Splits the linear segment [from.t, toTicks) (value from.v → toValue) at the bucket edges, clipped to /// the window, and hands each bucket its trapezoid area (value·ticks) and duration. A non-numeric start sample diff --git a/tests/Drivers/Historian.Conformance/ProcessedConformance.cs b/tests/Drivers/Historian.Conformance/ProcessedConformance.cs index ff9731a0..448ad111 100644 --- a/tests/Drivers/Historian.Conformance/ProcessedConformance.cs +++ b/tests/Drivers/Historian.Conformance/ProcessedConformance.cs @@ -68,13 +68,17 @@ public abstract class ProcessedConformance : ConformanceBase [Theory] [InlineData(HistoryAggregateType.PercentGood, 100d, 50d, 100d)] [InlineData(HistoryAggregateType.PercentBad, 0d, 50d, 0d)] - public async Task Percent_aggregates_count_every_sample(HistoryAggregateType agg, double b0, double b1, double b2) + public async Task Percent_aggregates_are_shares_of_the_interval_duration(HistoryAggregateType agg, double b0, double b1, + double b2) { + // #680 (Part 13 §5.4.3.31-34): stepped regions, each with the status of the sample starting it. B1: [20,30) Bad, + // [30,40) Good. B2 holds the end of the data (+40, nothing after) ⇒ its region runs to the interval end, Partial; + // B3 is after the end of the data ⇒ BadNoData. await ConformanceData.SeedAAsync(B); var s = (await ReadA(agg)).Samples; s.Select(x => x.Value).ShouldBe([b0, b1, b2, null]); // A percentage is a statement about quality, so it is not itself degraded by the Bad input. - s.Select(x => x.StatusCode).ShouldBe([GoodCalc, GoodCalc, GoodCalc, NoDataCalc]); + s.Select(x => x.StatusCode).ShouldBe([GoodCalc, GoodCalc, GoodCalc | 0x4u, NoDataCalc]); } [Fact] @@ -169,9 +173,10 @@ public abstract class ProcessedConformance : ConformanceBase // An interval wider than the window (here TimeSpan.MaxValue) is the same single interval, no overflow. (await src.ReadProcessedAsync("A", start, end, TimeSpan.MaxValue, HistoryAggregateType.Count, Ct)) .Samples.Single().Value.ShouldBe(4); - // The time-weighted path over the whole range: ≈ the extrapolated 50 that dominates eight millennia. + // The time-weighted path over the whole range. TimeAverage2 is not extrapolated past the end of the data (+40, #680): + // (150 + 200 + 450) value·s over the 30 non-Bad seconds = 80/3. var tw = (await src.ReadProcessedAsync("A", start, end, TimeSpan.Zero, HistoryAggregateType.TimeAverage2, Ct)).Samples.Single(); - ((double)tw.Value!).ShouldBe(50d, 1e-6); + ((double)tw.Value!).ShouldBe(80d / 3d, 1e-6); // The data starts (+0) strictly inside the one interval ⇒ Partial as well (#677, the SDK's m_startOfData rule). tw.StatusCode.ShouldBe(UncertainCalc | 0x4u); src.GetHealthSnapshot().TotalFailures.ShouldBe(0); @@ -347,15 +352,17 @@ public abstract class ProcessedConformance : ConformanceBase } } var reverseInterpolative = agg == HistoryAggregateType.Interpolative && from > to; + // Documented SDK divergence (#680, see SdkAggregateParityTests): on a reverse read the SDK files a sample exactly + // on an interval's later edge as its early bound, so TimeAverage/Total there take the status of the segment + // beyond it (Dense: +20 next to the Bad +22.5). + var reverseEdgeStatus = from > to && agg is HistoryAggregateType.TimeAverage or HistoryAggregateType.Total; SdkAggregateOracle.ShouldMatch(ours, SdkAggregateOracle.Run(agg, start, end, interval, rows), $"{agg} {from}→{to}/{intervalS}s", compareCalculatedInterpolatedBits: tw && !reverseInterpolative, - // pre-existing, #680: TimeAverage/Total status rule; Start/End raw-sample timestamp + status; Delta status; - // PercentGood/Bad per-sample vs per-duration values (see SdkAggregateParityTests). - compareCodeBits: agg is not (HistoryAggregateType.TimeAverage or HistoryAggregateType.Total - or HistoryAggregateType.Delta) && !reverseInterpolative && !startOrEnd, - compareTimestamps: !startOrEnd, - compareValues: agg is not (HistoryAggregateType.PercentGood or HistoryAggregateType.PercentBad)); + // pre-existing, #680: Start/End raw-sample timestamp + status; Delta status (see SdkAggregateParityTests). + compareCodeBits: agg != HistoryAggregateType.Delta && !reverseInterpolative && !startOrEnd + && !reverseEdgeStatus, + compareTimestamps: !startOrEnd); } } @@ -383,7 +390,8 @@ public abstract class ProcessedConformance : ConformanceBase avg.StatusCode.ShouldBe(UncertainCalc); (await One(HistoryAggregateType.Count)).Value.ShouldBe(2); (await One(HistoryAggregateType.Start)).Value.ShouldBe(10d); - ((double)(await One(HistoryAggregateType.PercentBad)).Value!).ShouldBe(33.333333333333336, 1e-9); // 100 × 1/3 + // #680: duration-based — [0,5) Good, [5,10) Bad, [10,20) Good (no end bound: the last region runs to the interval end). + ((double)(await One(HistoryAggregateType.PercentBad)).Value!).ShouldBe(25d, 1e-9); } [Fact] @@ -611,6 +619,9 @@ public abstract class ProcessedConformance : ConformanceBase var ex = await Should.ThrowAsync( () => under.ReadProcessedAsync("A", At(0), At(80), Twenty, HistoryAggregateType.Total, Ct)); ex.Limit.ShouldBe(4); + // #680: PercentGood/PercentBad are duration-based, served by the same window stream ⇒ capped too. + (await Should.ThrowAsync( + () => under.ReadProcessedAsync("A", At(0), At(80), Twenty, HistoryAggregateType.PercentGood, Ct))).Limit.ShouldBe(4); // Pushed-down (per-sample) aggregates are not capped; the bound rows outside the window do not count. (await under.ReadProcessedAsync("A", At(0), At(80), Twenty, HistoryAggregateType.Average, Ct)).Samples.Count.ShouldBe(4); (await B.CreateDataSource(3).ReadProcessedAsync("W", At(0), At(80), Twenty, HistoryAggregateType.TimeAverage, Ct)) diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ProcessedReadTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ProcessedReadTests.cs index a12edf4f..461c0b82 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ProcessedReadTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/ProcessedReadTests.cs @@ -11,10 +11,12 @@ public sealed class ProcessedReadTests public void Every_enum_member_is_either_served_or_refused() { // Seventeen served (#671 added TimeAverage, TimeAverage2, Interpolative and Total2, and moved Total to the - // time-weighted family); anything appended to the enum later falls to the refusal (total default). + // time-weighted family; #680 moved PercentGood/PercentBad there too — they are duration-based); anything appended to + // the enum later falls to the refusal (total default). Enum.GetValues().Count(a => AggregateFamilies.FamilyOf(a) is not null).ShouldBe(17); Enum.GetValues().Where(a => AggregateFamilies.FamilyOf(a) == AggregateFamily.TimeWeighted) .ShouldBe([HistoryAggregateType.Total, HistoryAggregateType.Interpolative, HistoryAggregateType.TimeAverage, - HistoryAggregateType.TimeAverage2, HistoryAggregateType.Total2]); + HistoryAggregateType.PercentGood, HistoryAggregateType.PercentBad, HistoryAggregateType.TimeAverage2, + HistoryAggregateType.Total2]); } } diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/SdkAggregateParityTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/SdkAggregateParityTests.cs index 11ca72fd..a8efc390 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/SdkAggregateParityTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/SdkAggregateParityTests.cs @@ -21,26 +21,27 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests; /// ours every usable (Good + Uncertain) sample (TreatUncertainAsBad = false applied consistently); /// the per-sample Calculated / MultipleValues info bits (the SDK's MinMax sets Calculated only when the /// extreme is not at the interval start, and flags ties) — those comparisons check code bits + Partial only; -/// a truncated interval: ours sets Partial on a non-Bad time-weighted result (the SDK does not), so the -/// non-multiple windows mask that one bucket's Partial bit; -/// the SDK's end-of-data Partial rule (forward, the interval holding the last sample when nothing follows -/// and nothing was extrapolated) is not implemented: every dataset carries data past the window end. -/// -/// Pre-existing forward divergences this parity test surfaced (#671 behaviour, unchanged here — forward results were -/// to stay as they are; recorded for a follow-up rather than fixed in place): -/// -/// TimeAverage / Total status: the SDK (interpolated bounds) reports UncertainDataSubNormal when -/// any region of the interval is non-Good (e.g. a segment bridged across a Bad sample); ours applies the -/// time-based rule (Uncertain time counts as Good) to these two as well, so a bridged-but-fully-covered interval -/// is Good. Values and timestamps agree; the status code bits are not compared for these two. -/// Total over an interval the data starts inside: the SDK integrates only the defined part -/// (value·s over the covered seconds); ours is TimeAverage × interval seconds. The Partial bit and timestamp -/// agree; that value is not compared. +/// a truncated interval: ours sets Partial on a non-Bad time-weighted result (Part 13 §5.3.3.2 Case 3 +/// requires it; the SDK does not), so the non-multiple windows mask that one bucket's Partial bit; +/// a time-weighted BadNoData carries the Calculated bit in ours (every processed status does), not in the +/// SDK's — masked by ; +/// TimeAverage2 / Total2 over an interval wholly inside an outage: ours generic Bad (data, but no +/// non-Bad time — Part 13's PercentData rule), the SDK BadNoData — masked by ; +/// the end-of-data interval (the last sample inside it, nothing after) of TimeAverage2 / Total2 / +/// PercentGood / PercentBad: the SDK gives the region after the last sample a 1 ms duration (an artifact of its +/// GetRegionsInValueSet); Part 13 ends TimeAverage2/Total2 at the last sample and runs the PercentGood/Bad +/// region to the interval end, which ours does (#680; pinned by TimeWeightedAggregatorTests). That +/// interval's value is not compared; its status, Partial bit and timestamp are. +/// TimeAverage / Total, reverse, a sample exactly on an interval's later edge: the SDK's backward +/// UpdateSlice files it as the early bound (below), so the interval takes the status of the segment beyond +/// it (Uncertain when that segment skips a Bad sample); ours keeps the interval's own segments — masked by +/// ; /// Interpolative, reverse, a sample exactly at the point: the SDK's backward UpdateSlice files /// a sample AT the point as the early bound (>= 0), so it interpolates from it to the next Good sample — /// same value, but the Interpolated bit, and Uncertain when a Bad sample sits between. Ours returns the raw /// sample with its own status in both directions (as the SDK itself does forward). Reverse Interpolative -/// therefore compares timestamps and values (the anchoring question) but not status. +/// therefore compares timestamps and values (the anchoring question) but not status. (The SDK's asymmetry — +/// forward it returns the raw sample — is its defect; ours is the same in both directions.) /// /// public sealed class SdkAggregateParityTests @@ -64,6 +65,32 @@ public sealed class SdkAggregateParityTests S(100, HistorianValue.FromDouble(60), Good), ]; + /// + /// #680: data that ENDS inside every window (nothing after +52.5): every 2.5 s from −10, values cycling, a Bad + /// sample at +22.5 and an Uncertain one at +40 — the end-of-data rules (TimeAverage2/Total2 stop at the last + /// sample, Partial, then BadNoData; TimeAverage/Total extrapolate, Uncertain). + /// + private static readonly StoredSample[] EndsInside = Enumerable.Range(0, 26) + .Select(i => (t: -10 + i * 2.5, i)) + .Select(x => S(x.t, HistorianValue.FromDouble(10 + (x.i * 7 % 13)), + x.t == 22.5 ? Bad : x.t == 40 ? Uncertain : Good)) + .ToArray(); + + /// + /// #680: an outage +18 … +45 that wholly contains an interval of every window ([20,40), [20,35), [23,43), (20,40]): + /// TimeAverage/Total bridge it (Uncertain); TimeAverage2/Total2 have no non-Bad time there. + /// + private static readonly StoredSample[] OutageInside = + [ + S(-10, HistorianValue.FromDouble(5), Good), + S(0, HistorianValue.FromDouble(10), Good), + S(15, HistorianValue.FromDouble(20), Good), + S(18, HistorianValue.Empty, BadNoCommunication), + S(45, HistorianValue.FromDouble(30), Good), + S(60, HistorianValue.FromDouble(40), Good), + S(100, HistorianValue.FromDouble(40), Good), + ]; + /// Data that starts inside the window (no sample before +7) and runs past its end. private static readonly StoredSample[] LateStart = Enumerable.Range(0, 20) .Select(i => S(7 + i * 5, HistorianValue.FromDouble(i % 2 == 0 ? 20 : 40), Good)) @@ -110,21 +137,74 @@ public sealed class SdkAggregateParityTests return list; } + /// + /// Applies the documented divergences listed on the class (other than truncation): strips the Calculated bit from our + /// BadNoData; maps our generic-Bad TimeAverage2/Total2 outage interval to the SDK's BadNoData; takes the + /// SDK's value for the end-of-data interval of TimeAverage2/Total2/PercentGood/PercentBad (its 1 ms artifact). + /// + private static IReadOnlyList MaskDocumented(HistoryAggregateType agg, IReadOnlyList ours, + IReadOnlyList sdk, double from, double to, double intervalS, StoredSample[] data) + { + var plan = BucketPlan.Create(At(from), At(to), TimeSpan.FromSeconds(intervalS)); + var list = ours.ToList(); + var last = data.Max(x => x.TimeUtc).Ticks; + var dataEnds = last < Math.Max(At(from).Ticks, At(to).Ticks) && data.Single(x => x.TimeUtc.Ticks == last).IsUsable; + for (long i = 0; i < plan.BucketCount; i++) + { + var k = (int)(plan.Reversed ? plan.BucketCount - 1 - i : i); + if (k >= sdk.Count) continue; // the count mismatch is reported by ShouldMatch + var code = list[k].StatusCode & StatusBits.CodeBitsMask; + if (code == HistorianStatusCodes.BadNoData) + list[k] = list[k] with { StatusCode = list[k].StatusCode & ~StatusBits.HistorianBitsMask }; + var twoVariant = agg is HistoryAggregateType.TimeAverage2 or HistoryAggregateType.Total2; + if (twoVariant && code == HistorianStatusCodes.Bad && (sdk[k].StatusCode & StatusBits.CodeBitsMask) == HistorianStatusCodes.BadNoData) + list[k] = list[k] with { StatusCode = HistorianStatusCodes.BadNoData }; + // Reverse TimeAverage/Total: the SDK's backward UpdateSlice files a sample exactly on the interval's later edge + // as the early bound, so the interval inherits the status of the segment beyond it (the root cause of the + // reverse Interpolative divergence). Ours keeps the interval's own segments. + if (plan.Reversed && agg is HistoryAggregateType.TimeAverage or HistoryAggregateType.Total + && data.Any(x => x.TimeUtc.Ticks == plan.UpperTicks(i)) + && code == HistorianStatusCodes.Good + && (sdk[k].StatusCode & StatusBits.CodeBitsMask) == HistorianStatusCodes.UncertainDataSubNormal) + list[k] = list[k] with { StatusCode = (list[k].StatusCode & ~StatusBits.CodeBitsMask) | HistorianStatusCodes.UncertainDataSubNormal }; + if (dataEnds && (twoVariant || agg is HistoryAggregateType.PercentGood or HistoryAggregateType.PercentBad) + && last >= plan.LowerTicks(i) && last < plan.UpperTicks(i) && sdk[k].Value is not null) + list[k] = list[k] with { Value = sdk[k].Value }; + } + return list; + } + [Theory] [MemberData(nameof(Windows))] public void Time_weighted_aggregates_match_the_SDK(double from, double to, double intervalS) { + var failures = new List(); foreach (var agg in new[] { HistoryAggregateType.TimeAverage, HistoryAggregateType.TimeAverage2, HistoryAggregateType.Total, - HistoryAggregateType.Total2, HistoryAggregateType.Interpolative, + HistoryAggregateType.Total2, HistoryAggregateType.Interpolative, HistoryAggregateType.PercentGood, + HistoryAggregateType.PercentBad, }) - foreach (var (name, data) in new[] { ("Bounded", Bounded), ("Dense", Dense) }) - SdkAggregateOracle.ShouldMatch(MaskTruncation(Ours(agg, from, to, intervalS, data), from, to, intervalS), - Sdk(agg, from, to, intervalS, data), $"{agg} {name} {from}→{to}/{intervalS}s", - compareCalculatedInterpolatedBits: !(agg == HistoryAggregateType.Interpolative && from > to), - compareCodeBits: agg is not (HistoryAggregateType.TimeAverage or HistoryAggregateType.Total) - && !(agg == HistoryAggregateType.Interpolative && from > to)); + foreach (var (name, data) in new[] + { + ("Bounded", Bounded), ("Dense", Dense), ("EndsInside", EndsInside), ("OutageInside", OutageInside), + }) + { + try + { + var sdk = Sdk(agg, from, to, intervalS, data); + var ours = MaskDocumented(agg, MaskTruncation(Ours(agg, from, to, intervalS, data), from, to, intervalS), + sdk, from, to, intervalS, data); + SdkAggregateOracle.ShouldMatch(ours, sdk, $"{agg} {name} {from}→{to}/{intervalS}s", + compareCalculatedInterpolatedBits: !(agg == HistoryAggregateType.Interpolative && from > to), + compareCodeBits: !(agg == HistoryAggregateType.Interpolative && from > to)); + } + catch (Exception ex) + { + failures.Add(ex.Message); + } + } + failures.ShouldBeEmpty(string.Join("\n", failures)); } [Theory] @@ -138,7 +218,6 @@ public sealed class SdkAggregateParityTests { SdkAggregateOracle.ShouldMatch(Ours(agg, from, to, intervalS, Dense), Sdk(agg, from, to, intervalS, Dense), $"{agg} Dense {from}→{to}/{intervalS}s", compareCalculatedInterpolatedBits: false, - compareValues: !PercentAggregate(agg), // pre-existing, #680: the SDK's Start/End return the raw first/last sample (any quality) with its own // timestamp and stored status; ours return the first/last USABLE value stamped with the interval, under the // per-sample status rule. The SDK's Delta is Good even when a Bad sample sits in the interval; ours is @@ -159,20 +238,11 @@ public sealed class SdkAggregateParityTests HistoryAggregateType.Average, HistoryAggregateType.Minimum, HistoryAggregateType.Maximum, HistoryAggregateType.Count, HistoryAggregateType.Range, HistoryAggregateType.Start, HistoryAggregateType.End, HistoryAggregateType.Delta, HistoryAggregateType.MinimumActualTime, HistoryAggregateType.MaximumActualTime, - HistoryAggregateType.PercentGood, HistoryAggregateType.PercentBad, ]; - /// - /// pre-existing, #680: ours is the per-SAMPLE percentage (100 × Good samples / all samples); the SDK's - /// StatusAggregateCalculator is the per-DURATION one (Good time / interval time). Different definitions, so the - /// values are not compared; timestamp, status and Partial still are. - /// private static bool StartOrEnd(HistoryAggregateType agg) => agg is HistoryAggregateType.Start or HistoryAggregateType.End; - private static bool PercentAggregate(HistoryAggregateType agg) => - agg is HistoryAggregateType.PercentGood or HistoryAggregateType.PercentBad; - [Theory] [InlineData(HistoryAggregateType.TimeAverage)] [InlineData(HistoryAggregateType.TimeAverage2)] @@ -194,9 +264,7 @@ public sealed class SdkAggregateParityTests var ours = Ours(agg, 40, 0, 10, LateStart); SdkAggregateOracle.ShouldMatch(ours, Sdk(agg, 40, 0, 10, LateStart), $"{agg} reverse LateStart", compareCalculatedInterpolatedBits: tw, - // pre-existing, #680: Total over the interval the data starts inside (TimeAverage × interval vs the SDK's - // integral of the covered part); Start's raw-sample timestamp/status (see Per_sample_aggregates_match_the_SDK). - compareValues: agg != HistoryAggregateType.Total, + // pre-existing, #680: Start's raw-sample timestamp/status (see Per_sample_aggregates_match_the_SDK). compareTimestamps: !StartOrEnd(agg), compareCodeBits: !StartOrEnd(agg) && agg != HistoryAggregateType.Delta); ours.ShouldAllBe(x => (x.StatusCode & HistorianInfoBits.HistorianPartial) == 0); @@ -218,7 +286,7 @@ public sealed class SdkAggregateParityTests var tw = AggregateFamilies.FamilyOf(agg) == AggregateFamily.TimeWeighted; var ours = Ours(agg, 0, 40, 10, LateStart); SdkAggregateOracle.ShouldMatch(ours, Sdk(agg, 0, 40, 10, LateStart), $"{agg} LateStart", - compareCalculatedInterpolatedBits: tw, compareValues: agg != HistoryAggregateType.Total); + compareCalculatedInterpolatedBits: tw); (ours[0].StatusCode & HistorianInfoBits.HistorianPartial).ShouldBe( agg == HistoryAggregateType.Average ? 0u : HistorianInfoBits.HistorianPartial); ours.Skip(1).ShouldAllBe(x => (x.StatusCode & HistorianInfoBits.HistorianPartial) == 0); diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/TimeWeightedAggregatorTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/TimeWeightedAggregatorTests.cs index d9432df5..b9351326 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/TimeWeightedAggregatorTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/TimeWeightedAggregatorTests.cs @@ -17,6 +17,7 @@ public sealed class TimeWeightedAggregatorTests private const uint InvalidInputs = 0x80D60000u; private const uint BadCalc = 0x80000401u; // generic Bad | Calculated (an all-outage interval) private const uint GoodCalcPartial = 0x00000405u; // Good | Calculated | Partial (truncated last interval) + private const uint UncertainCalcPartial = 0x40A40405u; // UncertainDataSubNormal | Calculated | Partial private static IReadOnlyList Run(HistoryAggregateType agg, double from, double to, double intervalS, params StoredSample[] samples) => @@ -33,43 +34,208 @@ public sealed class TimeWeightedAggregatorTests } // ---------------------------------------------------------------- tag A: +0=10 G, +10=20 G, +20=Bad, +30=40 G, +40=50 G - // Window [0,80), 20 s buckets. Raw: [0,10) 10→20 Good (150); [10,20) 20 held into the outage, Uncertain (200); - // [20,30) Bad; [30,40) 40→50 Good (450); [40,80) 50 extrapolated, Uncertain (2000). - // Bridged: [0,10) 150; [10,30) 20→40 = 250 in B0 + 350 in B1; [30,40) 450; [40,80) 50 held. + // Window [0,80), 20 s buckets; nothing after +40 (no end bound: the data ENDS at +40). Raw: [0,10) 10→20 Good (150); + // [10,20) 20 held into the outage, Uncertain (200); [20,30) Bad; [30,40) 40→50 Good (450); after +40 nothing (#680: + // Part 13 "After End of Data" — the simple-bounding signal is not extrapolated). + // Bridged: [0,10) 150; [10,30) 20→40 = 250 in B0 + 350 in B1; [30,40) 450; [40,80) 50 held (extrapolated). [Fact] public void TimeAverage_bridges_the_outage_on_tag_A() { - // B0 (150+250)/20 = 20; B1 (350+450)/20 = 40; B2, B3 = 50. Status (SDK rule, Uncertain time counts as Good): - // B0 is non-Bad throughout (held-into-outage time is Uncertain) ⇒ Good; B1 has the Bad [20,30) ⇒ Uncertain; - // B2, B3 are extrapolated (Uncertain) throughout ⇒ Good. + // B0 (150+250)/20 = 20; B1 (350+450)/20 = 40; B2, B3 = 50. Status (#680, Part 13 + the SDK's interpolated-bounds + // rule): Uncertain when a Bad value was skipped or the value was extrapolated — B0 and B1 hold the segment bridged + // across the Bad +20; B2 and B3 are extrapolated past the last sample. var s = Run(HistoryAggregateType.TimeAverage, 0, 80, 20, A); Values(s, 20, 40, 50, 50); - s.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalc, GoodCalc]); + s.Select(x => x.StatusCode).ShouldBe([UncertainCalc, UncertainCalc, UncertainCalc, UncertainCalc]); s.Select(x => x.SourceTimestampUtc).ShouldBe([At(0), At(20), At(40), At(60)]); } [Fact] public void Total_is_TimeAverage_times_interval_seconds_on_tag_A() { - // 20×20, 40×20, 50×20, 50×20. + // Fully covered intervals: 20×20, 40×20, 50×20, 50×20 (the integral of the interpolated-bounding signal). Values(Run(HistoryAggregateType.Total, 0, 80, 20, A), 400, 800, 1000, 1000); } [Fact] - public void TimeAverage2_excludes_the_bad_interval_on_tag_A() + public void TimeAverage2_excludes_the_bad_interval_and_stops_at_the_end_of_data_on_tag_A() { - // B0 350/20 = 17.5; B1 450/10 = 45 (the Bad [20,30) excluded); B2, B3 = 50. + // B0 350/20 = 17.5; B1 450/10 = 45 (the Bad [20,30) excluded). B2 [40,60) holds the end of the data (+40, on its + // start): no covered time, so the value is the last sample's (50), Good, Partial (#680; the SDK answers the same). + // B3 is after the end of the data ⇒ BadNoData (Part 13 "After End of Data"). var s = Run(HistoryAggregateType.TimeAverage2, 0, 80, 20, A); - Values(s, 17.5, 45, 50, 50); - s.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalc, GoodCalc]); + Values(s, 17.5, 45, 50, null); + s.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalcPartial, NoDataCalc]); } [Fact] public void Total2_integrates_only_the_non_bad_data_on_tag_A() { - // B0 150+200 = 350; B1 450; B2, B3 1000. - Values(Run(HistoryAggregateType.Total2, 0, 80, 20, A), 350, 450, 1000, 1000); + // B0 150+200 = 350; B1 450; B2 nothing covered after the last sample ⇒ 0, Partial; B3 BadNoData. + var s = Run(HistoryAggregateType.Total2, 0, 80, 20, A); + Values(s, 350, 450, 0, null); + s.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalcPartial, NoDataCalc]); + } + + // ---------------------------------------------------------------- #680: end of data, live tags, outages, Total coverage + + /// −5 = 1, +3 = 2, +25 = 3 (all Good), nothing later: the data ends at +25. + private static readonly StoredSample[] EndsInside = + [ + S(-5, HistorianValue.FromDouble(1), Good), S(3, HistorianValue.FromDouble(2), Good), + S(25, HistorianValue.FromDouble(3), Good), + ]; + + [Fact] + public void TimeAverage2_and_Total2_stop_at_the_end_of_data_with_Partial_then_BadNoData() + { + // Window [0,40), 10 s. Segment +3..+25 slopes 2→3 (1/22 per s). [20,30): covered only [20,25): 2+17/22 → 3, area + // 5 × (2.772727… + 3)/2 = 14.431818…; TimeAverage2 = 2.886363…, Good + Partial (the SDK: 14.4348 for Total2 — + // its last region carries a 1 ms artifact). [30,40) is after the end of the data ⇒ BadNoData. + var t2 = Run(HistoryAggregateType.Total2, 0, 40, 10, EndsInside); + ((double)t2[2].Value!).ShouldBe(5 * ((2 + 17d / 22) + 3) / 2, 1e-9); + t2[2].StatusCode.ShouldBe(GoodCalcPartial); + t2[3].StatusCode.ShouldBe(NoDataCalc); + t2[3].Value.ShouldBeNull(); + var ta2 = Run(HistoryAggregateType.TimeAverage2, 0, 40, 10, EndsInside); + ((double)ta2[2].Value!).ShouldBe(((2 + 17d / 22) + 3) / 2, 1e-9); + ta2.Select(x => x.StatusCode).ShouldBe([GoodCalc, GoodCalc, GoodCalcPartial, NoDataCalc]); + } + + [Fact] + public void TimeAverage_and_Total_extrapolate_past_the_end_of_data_as_Uncertain_without_Partial() + { + // The interpolated-bounding signal holds the last value to the window end (stepped extrapolation) — Part 13 and + // the SDK make that Uncertain; the SDK sets no end-of-data Partial on an extrapolated result. + // [20,30): [20,25) 2.772727…→3 + [25,30) 3 ⇒ (14.431818… + 15) / 10; [30,40): 3. + var ta = Run(HistoryAggregateType.TimeAverage, 0, 40, 10, EndsInside); + ((double)ta[2].Value!).ShouldBe((5 * ((2 + 17d / 22) + 3) / 2 + 15) / 10, 1e-9); + ((double)ta[3].Value!).ShouldBe(3d); + ta.Select(x => x.StatusCode).ShouldBe([GoodCalc, GoodCalc, UncertainCalc, UncertainCalc]); + Values(Run(HistoryAggregateType.Total, 30, 40, 0, EndsInside), 30); + } + + [Fact] + public void The_latest_interval_of_a_live_tag_is_Uncertain() + { + // A live tag: −10 = 10, +5 = 20, +12 = 30, nothing after yet. [10,20) is extrapolated from +12 ⇒ Uncertain + // (#680 — it used to read Good). [0,10) is bridged Good→Good with no skip ⇒ Good. Values as the SDK: + // [0,10) 20.952380…, [10,20) 29.714285…. + StoredSample[] live = + [ + S(-10, HistorianValue.FromDouble(10), Good), S(5, HistorianValue.FromDouble(20), Good), + S(12, HistorianValue.FromDouble(30), Good), + ]; + var s = Run(HistoryAggregateType.TimeAverage, 0, 20, 10, live); + Values(s, 20.952380952380953, 29.71428571428571); + s.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc]); + Run(HistoryAggregateType.Total, 0, 20, 10, live)[1].StatusCode.ShouldBe(UncertainCalc); + } + + [Fact] + public void An_uncertain_sample_makes_TimeAverage_and_Total_uncertain() + { + // Part 13: "Uncertain if ... Uncertain values are used". −10 = 10 G, +5 = 20 U, +15 = 30 G, +40 = 30 G; [0,20): + // the value 24.583333… (SDK), Uncertain. + StoredSample[] unc = + [ + S(-10, HistorianValue.FromDouble(10), Good), S(5, HistorianValue.FromDouble(20), Uncertain), + S(15, HistorianValue.FromDouble(30), Good), S(40, HistorianValue.FromDouble(30), Good), + ]; + var ta = Run(HistoryAggregateType.TimeAverage, 0, 20, 0, unc).Single(); + ((double)ta.Value!).ShouldBe(24.583333333333332, 1e-9); + ta.StatusCode.ShouldBe(UncertainCalc); + Run(HistoryAggregateType.Total, 0, 20, 0, unc).Single().StatusCode.ShouldBe(UncertainCalc); + } + + [Fact] + public void A_segment_bridged_across_a_bad_sample_is_uncertain_in_every_interval_it_touches() + { + // −10 = 10 G, +10 Bad, +15 = 20 G, +40 = 20 G: the bridged segment −10..+15 skips the Bad +10, so [0,10) — which + // holds no Bad sample itself — is Uncertain too (the SDK agrees). + StoredSample[] edge = + [ + S(-10, HistorianValue.FromDouble(10), Good), S(10, HistorianValue.FromDouble(30), Bad), + S(15, HistorianValue.FromDouble(20), Good), S(40, HistorianValue.FromDouble(20), Good), + ]; + var s = Run(HistoryAggregateType.TimeAverage, 0, 20, 10, edge); + Values(s, 16, 19.5); + s.Select(x => x.StatusCode).ShouldBe([UncertainCalc, UncertainCalc]); + } + + [Fact] + public void Total_over_an_interval_the_data_starts_inside_integrates_only_the_covered_part() + { + // +5 = 10, +15 = 10, +40 = 10, nothing before: [0,20) is covered from +5 only ⇒ Total = 10 × 15 s = 150 (Part 13 + // partial-interval width; the SDK agrees). TimeAverage is still 10. Uncertain (no data for [0,5)) + Partial. + StoredSample[] late = + [ + S(5, HistorianValue.FromDouble(10), Good), S(15, HistorianValue.FromDouble(10), Good), + S(40, HistorianValue.FromDouble(10), Good), + ]; + var total = Run(HistoryAggregateType.Total, 0, 20, 0, late).Single(); + ((double)total.Value!).ShouldBe(150d); + total.StatusCode.ShouldBe(UncertainCalcPartial); + ((double)Run(HistoryAggregateType.TimeAverage, 0, 20, 0, late).Single().Value!).ShouldBe(10d); + } + + // ---------------------------------------------------------------- #680: PercentGood / PercentBad are duration-based + + [Fact] + public void Percent_aggregates_are_durations_on_tag_A() + { + // Stepped regions, each with the status of the sample that starts it. B0 [0,20) Good throughout; B1 [20,30) Bad + // (the Bad +20), [30,40) Good ⇒ 50 / 50; B2 holds the end of the data (+40, Good) ⇒ its region runs to the interval + // end (Part 13), Partial (forward end-of-data, as the SDK); B3 is after the end of the data ⇒ BadNoData. + var good = Run(HistoryAggregateType.PercentGood, 0, 80, 20, A); + Values(good, 100, 50, 100, null); + good.Select(x => x.StatusCode).ShouldBe([GoodCalc, GoodCalc, GoodCalcPartial, NoDataCalc]); + var bad = Run(HistoryAggregateType.PercentBad, 0, 80, 20, A); + Values(bad, 0, 50, 0, null); + bad.Select(x => x.StatusCode).ShouldBe([GoodCalc, GoodCalc, GoodCalcPartial, NoDataCalc]); + } + + [Fact] + public void With_no_end_bound_the_last_region_runs_to_the_interval_end() + { + // 0 = 10 G, +5 outage, +10 = 30 G, nothing after; [0,20) one interval: [0,5) Good, [5,10) Bad, [10,20) Good + // (Part 13: the last region extends to the interval end — the SDK instead gives it 1 ms, ≈ 49.995 % Bad). + StoredSample[] s = + [ + S(0, HistorianValue.FromDouble(10), Good), S(5, HistorianValue.Empty, BadNoCommunication), + S(10, HistorianValue.FromDouble(30), Good), + ]; + var bad = Run(HistoryAggregateType.PercentBad, 0, 20, 0, s).Single(); + ((double)bad.Value!).ShouldBe(25d, 1e-9); + bad.StatusCode.ShouldBe(GoodCalcPartial); + ((double)Run(HistoryAggregateType.PercentGood, 0, 20, 0, s).Single().Value!).ShouldBe(75d, 1e-9); + } + + [Fact] + public void Percent_regions_use_the_simple_bound_at_the_interval_start_and_count_time_before_the_data_as_Bad() + { + // −3 = 10 G, +4 outage, +6 = 30 U, +12 = 3 G, +30 = 3 G; [0,20), 10 s. The simple bound at +0 lies on a segment + // that runs into the outage ⇒ Uncertain for [0,4) (the SDK's simple bounding value); [4,6) Bad; [6,10) Uncertain. + // B0: Good 0 %, Bad 20 %. B1: [10,12) Uncertain (bound between the Uncertain +6 and +12), [12,20) Good ⇒ 80 / 0. + StoredSample[] s = + [ + S(-3, HistorianValue.FromDouble(10), Good), S(4, HistorianValue.Empty, BadNoCommunication), + S(6, HistorianValue.FromDouble(30), Uncertain), S(12, HistorianValue.FromDouble(3), Good), + S(30, HistorianValue.FromDouble(3), Good), + ]; + Values(Run(HistoryAggregateType.PercentGood, 0, 20, 10, s), 0, 80); + Values(Run(HistoryAggregateType.PercentBad, 0, 20, 10, s), 20, 0); + // The data starts at +10, exactly on the end of [0,10): all of [0,10) is before the data ⇒ 100 % Bad (the SDK + // agrees); data starting at +5 ⇒ [0,5) Bad, Partial. + StoredSample[] late = [S(10, HistorianValue.FromDouble(1), Good), S(30, HistorianValue.FromDouble(2), Good)]; + Values(Run(HistoryAggregateType.PercentBad, 0, 20, 10, late), 100, 0); + StoredSample[] mid = [S(5, HistorianValue.FromDouble(1), Good), S(30, HistorianValue.FromDouble(2), Good)]; + var m = Run(HistoryAggregateType.PercentBad, 0, 20, 0, mid).Single(); + ((double)m.Value!).ShouldBe(25d, 1e-9); + m.StatusCode.ShouldBe(GoodCalcPartial); + // Nothing at all ⇒ BadNoData. + Run(HistoryAggregateType.PercentGood, 0, 20, 10).ShouldAllBe(x => x.StatusCode == NoDataCalc); } [Fact] @@ -146,12 +312,13 @@ public sealed class TimeWeightedAggregatorTests [Fact] public void Past_the_last_sample_the_value_is_extrapolated_stepped() { - // -5 = 7 and nothing later: held at 7 over [0,10). The extrapolated time is Uncertain, which the SDK rule - // (TreatUncertainAsBad = false) counts as Good time ⇒ the interval is Good; the Interpolative point is Uncertain. + // -5 = 7 and nothing later: TimeAverage holds 7 over [0,10), Uncertain (extrapolated, #680). TimeAverage2 does not + // extrapolate: [0,10) is after the end of the data ⇒ BadNoData. The Interpolative point is Uncertain. StoredSample[] one = [S(-5, HistorianValue.FromDouble(7), Good)]; - var s = Run(HistoryAggregateType.TimeAverage2, 0, 10, 0, one).Single(); + var s = Run(HistoryAggregateType.TimeAverage, 0, 10, 0, one).Single(); ((double)s.Value!).ShouldBe(7d); - s.StatusCode.ShouldBe(GoodCalc); + s.StatusCode.ShouldBe(UncertainCalc); + Run(HistoryAggregateType.TimeAverage2, 0, 10, 0, one).Single().StatusCode.ShouldBe(NoDataCalc); var i = Run(HistoryAggregateType.Interpolative, 0, 10, 0, one).Single(); ((double)i.Value!).ShouldBe(7d); i.StatusCode.ShouldBe(UncertainInterpolated); @@ -208,13 +375,17 @@ public sealed class TimeWeightedAggregatorTests } [Fact] - public void An_interval_entirely_inside_an_outage_is_Bad_but_Interpolative_bridges_it() + public void An_interval_entirely_inside_an_outage_bridges_for_TimeAverage_but_is_Bad_for_TimeAverage2() { - // Window [10,15): raw is Bad throughout ⇒ data but no non-Bad time ⇒ generic Bad (the SDK rule; BadNoData is - // reserved for no data at all). Interpolative at +10 = 10 + 20·(10/15). + // Window [10,15): the interpolated-bounding signal (10→30 over [0,15)) bridges it — 23.333… → 30, mean 26.666…, + // Uncertain (a Bad value skipped; #680, Part 13 + the SDK). Total = 26.666… × 5 s. The simple-bounding signal is + // Bad throughout ⇒ TimeAverage2/Total2 generic Bad (Part 13's PercentData rule; the SDK answers BadNoData). + // Interpolative at +10 = 10 + 20·(10/15). var ta = Run(HistoryAggregateType.TimeAverage, 10, 15, 0, Outage).Single(); - ta.StatusCode.ShouldBe(BadCalc); - ta.Value.ShouldBeNull(); + ((double)ta.Value!).ShouldBe((10d + 20d * 10d / 15d + 30d) / 2, 1e-9); + ta.StatusCode.ShouldBe(UncertainCalc); + ((double)Run(HistoryAggregateType.Total, 10, 15, 0, Outage).Single().Value!) + .ShouldBe((10d + 20d * 10d / 15d + 30d) / 2 * 5, 1e-9); Run(HistoryAggregateType.TimeAverage2, 10, 15, 0, Outage).Single().StatusCode.ShouldBe(BadCalc); Run(HistoryAggregateType.Total2, 10, 15, 0, Outage).Single().StatusCode.ShouldBe(BadCalc); var i = Run(HistoryAggregateType.Interpolative, 10, 15, 0, Outage).Single(); @@ -324,11 +495,14 @@ public sealed class TimeWeightedAggregatorTests [Fact] public void A_reverse_window_is_returned_latest_first_stamped_with_later_edges() { - // #677: the four whole intervals are the same spans as forward, but each is stamped with its later edge. + // #677: the four whole intervals are the same spans as forward, but each is stamped with its later edge. The data + // ends at +40 (#680): the latest interval is after it ⇒ BadNoData; the next holds it on its earlier edge ⇒ 50, with + // no end-of-data Partial on a reverse read (the SDK answers the same). var s = TimeWeightedAggregator.Evaluate(HistoryAggregateType.TimeAverage2, BucketPlan.Create(At(80), At(0), TimeSpan.FromSeconds(20)), A, Now); s.Select(x => x.SourceTimestampUtc).ShouldBe([At(80), At(60), At(40), At(20)]); - Values(s, 50, 50, 45, 17.5); + Values(s, null, 50, 45, 17.5); + s.Select(x => x.StatusCode).ShouldBe([NoDataCalc, GoodCalc, UncertainCalc, GoodCalc]); } [Fact] @@ -336,13 +510,15 @@ public sealed class TimeWeightedAggregatorTests { // Tag A, At(50) → At(0), 20 s ⇒ (30,50], (10,30], (0,10]. Raw signal: [0,10) 10→20 Good (150); [10,20) 20 held // into the outage, Uncertain (200); [20,30) Bad; [30,40) 40→50 Good (450); [40,50) 50 extrapolated (500). - // (30,50]: 950/20 = 47.5 Good; (10,30]: 200/10 = 20, the Bad 10 s ⇒ Uncertain; (0,10]: 150/10 = 15, truncated ⇒ + // (30,50]: the data ends at +40 (#680: no extrapolation) ⇒ 450/10 = 45, Good; the latest sample lies strictly + // inside it, where the reverse read's data starts ⇒ Partial (the SDK's m_startOfData keyed on the first value it + // is fed); (10,30]: 200/10 = 20, the Bad 10 s ⇒ Uncertain; (0,10]: 150/10 = 15, truncated ⇒ // Partial. The old layout would have answered [0,20), [20,40), [40,50) stamped 0/20/40, partial at the LATEST end. var s = TimeWeightedAggregator.Evaluate(HistoryAggregateType.TimeAverage2, BucketPlan.Create(At(50), At(0), TimeSpan.FromSeconds(20)), A, Now); s.Select(x => x.SourceTimestampUtc).ShouldBe([At(50), At(30), At(10)]); - Values(s, 47.5, 20, 15); - s.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalcPartial]); + Values(s, 45, 20, 15); + s.Select(x => x.StatusCode).ShouldBe([GoodCalcPartial, UncertainCalc, GoodCalcPartial]); } [Fact] @@ -368,8 +544,9 @@ public sealed class TimeWeightedAggregatorTests // Window [0,50), 20 s: B2 = [40,50) is 10 s of the extrapolated 50 ⇒ TimeAverage 50, Total 500. var s = Run(HistoryAggregateType.Total, 0, 50, 20, A); Values(s, 400, 800, 500); - // Only the truncated interval carries the Partial bit. - s.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalcPartial]); + // Only the truncated interval carries the Partial bit (Part 13 §5.3.3.2 Case 3 — the SDK does not set it; ours + // does, deliberately). Every interval is Uncertain (bridged across the Bad +20, or extrapolated). + s.Select(x => x.StatusCode).ShouldBe([UncertainCalc, UncertainCalc, UncertainCalcPartial]); } [Fact]