Merge feat/677-aggregates: reverse processed reads SDK-anchored, start-of-data Partial (forward), AggregateConfiguration accept-defaults, IntegralDivisor=1, SDK parity tests (#677)

This commit is contained in:
Joseph Doherty
2026-10-11 11:47:52 -04:00
24 changed files with 1291 additions and 135 deletions
+1
View File
@@ -1351,6 +1351,7 @@ historian tagname, which is the tag's driver `FullName`.
- **Selection:** `ServerHistorian:Backend` (`Gateway` default | `SqlServer` | `QuestDb`; unknown fails host start). `HistorianBackendRegistry` → `IHistorianBackendProvider` builds all five seams; the gateway keys are only required for `Gateway`. `SqlServer` uses `ServerHistorian:SqlServer:{ConnectionString (required), Schema=hist, CommandTimeout=30s, AutoCreateSchema=true}`.
- **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; per-sample aggregates (12, simple definitions) are pushed down to T-SQL with bucket math in **100 ns ticks**; the Part 13 time-weighted five (TimeAverage/TimeAverage2/Total/Total2/Interpolative, #671 — ⚠️ `Total` is now the time-weighted integral, no longer `SUM`) stream window + 4 bound rows into the shared `TimeWeightedAggregator` (both SQL backends, identical results; client `AggregateConfiguration` ignored; status = the SDK `AggregateCalculator` time-based rule, Uncertain time counts as Good; reverse `Interpolative` anchored at the later time; window capped by `ServerHistorian:MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations`). Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. **Retention (#667):** opt-in `SqlServer:RetentionDays` (+ `PurgeInterval` 1h, `PurgeBatchSize` 4000) -> `HistorianRetentionService` purges old `hist.Sample` (per tag) / `hist.Event` in `DELETE TOP(n)` batches, Primary-gated on `IHistorizationRoleView`, one purger per DB via `sp_getapplock`; counter `otopcua.historian.retention.purged{table}`. Optional `IHistorianBackendProvider.CreateRetention` (default null). `docs/Historian.md` §Retention.
- **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend".
- ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); on forward reads Partial also marks the interval the data starts inside (not Average — the SDK's rule, forward-only); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates.
- **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good.
- **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node.
- **HistoryUpdate** (SqlServer + QuestDb; Gateway ⇒ `BadHistoryOperationUnsupported`): Insert/Replace/Update/DeleteRaw/DeleteAtTime, **Primary-only** since #674 (off-Primary ⇒ `BadInvalidState`, after authorization AND after planning — an unservable request (Remove / `IsDeleteModified` / not historized) stays `BadHistoryOperationUnsupported` off-Primary; the verdict is `IHistoryUpdateRoleView` = the data-plane verdict but **closed until this node's first redundancy snapshot**, so a pair cannot both accept in the boot window; admin-only nodes register no view ⇒ no gate; reads ungated), items with a `SourceTimestamp` past now + `ServerHistorian:HistoryUpdateMaxFutureSkew` (default 1 day, ≤0 disables) ⇒ `BadOutOfRange` in the node manager (every backend), gated in both ACL modes — flag off needs the `HistoryUpdate` role from a **Cluster-scoped** grant (+ key re-check), flag on uses `DataPlaneAuthorizer`'s HistoryUpdate operation; `WriteOperate` never implies it. The `HistoryWrite` AccessLevel bit is set only when an updater is wired at materialise time. `docs/Historian.md` §HistoryUpdate, `docs/security.md`. ⚠️ **SqlServer schema v1 is accepted (#674)** with `AutoCreateSchema=false`: writes/reads/events/retention work; modified reads + HistoryUpdate ⇒ node-level `BadHistoryOperationUnsupported` on every arm (the updater throws `NotSupportedException`, nothing sent; the `HistoryWrite` bit is still advertised — not knowable at materialise time), retention skips `SampleModified`, one Warning per boot (`SqlServerSchemaBootstrapper.SupportsModifiedTrail`; `RequiredVersion` = 1, `ModifiedTrailVersion` = 2). QuestDB: keep `cairo.ttl.use.wall.clock=true`; a QuestDB trail INSERT retried after a client timeout duplicates (no DEDUP key).
+73 -21
View File
@@ -749,6 +749,12 @@ RetrievalMode.Integral`) — no client-side scaling. (This replaces the retired
likewise a native gateway mode. Bucket status codes and timestamps are preserved unchanged; empty / null
buckets surface as `BadNoData`.
**Integral unit (#677).** AVEVA's integral is `∫ value dt` (seconds) ÷ the tag's `IntegralDivisor`. The proto3 `double` default for
`HistorianTagDefinition.IntegralDivisor` is `0`, so `GatewayTagProvisioner` now sends **`IntegralDivisor = 1`** explicitly on every
`EnsureTags` — the value·seconds unit the SQL backends' `Total` uses. A tag provisioned earlier keeps its divisor until its next
`EnsureTags` (an upsert). The live magnitude check against AVEVA (a constant 1 integrated over 1 s reads 1) is still open — it needs the
VPN. Reverse processed reads on the `Gateway` backend are answered by AVEVA and are not re-anchored by #677.
### Aggregate set and `BadAggregateNotSupported` (#285)
`HistoryAggregateType` carries the wider Part 13 set (Interpolative, TimeAverage, Start, End, Delta, Range,
@@ -913,21 +919,44 @@ point (see [Continuation-point paging (Events)](#continuation-point-paging-event
> that relied on the old sum gets a different number for the same data (e.g. two samples 10 and 20 ten seconds apart no longer total
> 30). There is no switch back; `Count` × `Average` reproduces the old sum.
**Bucketing.** One result per interval of `[start, end)`; bucket *i* = `[start + i·interval, start + (i+1)·interval)`, start-inclusive /
end-exclusive to the tick, the last bucket truncated at `end`. `interval <= 0` ⇒ a single interval `[start, end)`; `start == end` ⇒ no
> ⚠️ **Release note (#677): reverse processed reads changed — client-visible.** A reverse read (`end < start`) is now anchored at the
> request's start (its *later* time) for **every** aggregate, as OPC UA Part 13 and the SDK's own `AggregateCalculator` do: intervals
> step back from `start`, the window is `(end, start]`, the truncated interval is the **earliest** one, and each result is stamped with its
> interval's **later** edge. Before #677 only `Interpolative` did this; every other aggregate swapped the window, anchored it at the
> earlier time (`[end, start)`, the partial interval at the latest end) and stamped each interval with its earlier edge. So a reverse
> 80 s → 0 s read at 20 s used to answer `[60,80) [40,60) [20,40) [0,20)` stamped 60/40/20/0; it now answers
> `(60,80] (40,60] (20,40] (0,20]` stamped 80/60/40/20 — a sample exactly on an interval edge now belongs to the interval it *ends*, a
> sample at `start` is included and one at `end` is not (the same window a reverse raw read uses). Forward reads are unchanged apart
> from the new (forward-only) start-of-data Partial bit below. A client that paired reverse results with forward ones by timestamp must shift by one
> interval.
**Bucketing.** Forward: one result per interval of `[start, end)`; bucket *i* = `[start + i·interval, start + (i+1)·interval)`,
start-inclusive / end-exclusive to the tick, the last bucket truncated at `end`, each stamped with its start. Reverse (`end < start`,
#677): the window is `(end, start]`; result *k* = `(start − (k+1)·interval, start − k·interval]`, clipped at `end` (the **earliest**
interval is the truncated one), stamped with its **later** edge `start − k·interval`, returned latest first. `interval <= 0` ⇒ a single interval `[start, end)`; `start == end` ⇒ no
intervals. The bucket index is computed in **100 ns ticks** (not milliseconds): `DATEDIFF_BIG(microsecond, @start, t) × 10` plus a
sub-microsecond correction from the two tick digits, divided by the interval in ticks. (`DATEDIFF` counts unit *boundaries crossed*, so a
millisecond formulation misplaces samples near a bucket edge, and a nanosecond one overflows past ≈292 years.) A window of more than
**100 000 intervals** is refused before any I/O and reported `BadHistoryOperationUnsupported` for that node. **Reverse windows**
(`end < start`) are swapped, computed forward **anchored at the earlier time**, and returned in **descending** bucket order. Every returned
status carries the historian *Calculated* info bit, `BadNoData` included.
millisecond formulation misplaces samples near a bucket edge, and a nanosecond one overflows past ≈292 years.) A reverse read adds one
constant to that elapsed-ticks dividend (`BucketPlan.BucketIndexAdjustTicks`; QuestDB folds it into its base) that re-anchors the grid at
the later time and makes the buckets later-edge-inclusive — `BucketPlan.IndexOf` is the C# statement of the math both dialects mirror. A
window of more than **100 000 intervals** is refused before any I/O and reported `BadHistoryOperationUnsupported` for that node. Every
returned status carries the historian *Calculated* info bit, `BadNoData` included.
**Partial bit (#677).** On a **forward** read, every aggregate except plain `Average` sets the **Partial** bit (`0x4`) on a non-Bad
result for the interval the tag's data **starts inside** — its earliest stored sample (any quality) lies strictly between the interval's
edges, so there is no bounding value before it and the data does not cover the whole interval. This follows the SDK's `m_startOfData`
rule, which the SDK turns off for `Average` and in effect applies to forward reads only (a reverse read is fed latest-first, so its
"start of data" is the latest sample, past the window). A reverse read therefore sets no start-of-data Partial — matching the SDK. Only a
forward per-sample read of an aggregate other than `Average` fetches the tag's earliest sample (one seek, bounded by the window end);
every other read skips that query. An earliest sample exactly on an edge, or before the interval, sets nothing. The time-weighted
aggregates also keep setting Partial on the truncated interval (below).
**Sample classes.** *Usable* = severity Good or Uncertain **and** a stored value (not `Empty`). *Good* = usable with severity Good.
*Bad* = not usable.
| Aggregate | Computed over | Value timestamp |
|---|---|---|
| `Average`, `Minimum`, `Maximum` | `AVG` / `MIN` / `MAX` of `NumValue` over usable samples | bucket start |
| `Average`, `Minimum`, `Maximum` | `AVG` / `MIN` / `MAX` of `NumValue` over usable samples | bucket start (reverse read: its later edge, as every row marked "bucket start" here) |
| `Count` | number of usable samples (`Int32`) | bucket start |
| `Range` | `Maximum − Minimum` | bucket start |
| `Start`, `End` | first / last usable sample by time (keeps the sample's type) | bucket start |
@@ -949,11 +978,16 @@ Status and edge rules:
`TimeAverage`, `TimeAverage2`, `Total`, `Total2` and `Interpolative`, evaluated by the shared
`Driver.Historian.Sql.Common/TimeWeightedAggregator.cs` for **both** SQL backends (so `SqlServer` and `QuestDb` agree exactly).
**This whole family — `Total` included — is evaluated in C#, not pushed down to SQL** (see *Pushdown trade-off* and the raw-row cap
below). Bucketing (edges, truncated last bucket, the 100 000-interval cap) is the same as above; reverse windows are covered below.
below). Bucketing (edges, truncated interval, reverse anchoring, the 100 000-interval cap) is the same as above.
**Server defaults, fixed:** `TreatUncertainAsBad = false`, `PercentDataBad = PercentDataGood = 100`, `UseSlopedExtrapolation = false`.
**A client's `AggregateConfiguration` in the HistoryRead request is ignored** (the node manager does not forward it); every read uses
these defaults.
**Server defaults, fixed:** `TreatUncertainAsBad = false`, `PercentDataBad = PercentDataGood = 100`, `UseSlopedExtrapolation = false`
(`ServerAggregateConfiguration` in the OPC UA server project — the SDK's own `AggregateManager` defaults, which a test pins). **A
client's `AggregateConfiguration` (#677):** `UseServerCapabilitiesDefaults = true`, or the four values spelled out equal to these defaults
⇒ served; anything else ⇒ **`BadAggregateConfigurationRejected`** for each node of the request (the source is never called). It used to
be silently ignored, so a client asking for, say, `TreatUncertainAsBad = true` got answers computed without it. The check is server-wide
(it applies on the `Gateway` backend too). A client sending an all-zero structure without `UseServerCapabilitiesDefaults` is asking for
`PercentDataBad = PercentDataGood = 0` and is refused; the SDK's default-constructed `AggregateConfiguration` sets
`UseServerCapabilitiesDefaults = true` and is served.
**Bounding values.** Besides the window `[start, end)`, each read fetches four single-row seeks: the latest *usable* sample before
`start`, the latest sample of any quality before `start` (they differ when an outage straddles the start), the first sample of any
@@ -987,9 +1021,9 @@ PercentDataGood.
- No data at all ⇒ `BadNoData`.
- A usable `String`/`DateTime` sample feeding the interval ⇒ `BadAggregateInvalidInputs`.
All carry the Calculated bit; a non-Bad result for the **truncated last interval** (shorter than the processing interval) also carries
the **Partial** bit (`0x4`, `AggregateBits.Partial`). Timestamp = interval start; value `double` (Boolean 0/1). (The per-sample
aggregates do not set Partial — a follow-up.)
All carry the Calculated bit; a non-Bad result for the **truncated interval** (shorter than the processing interval — the latest one
forward, the earliest one reverse) or for the interval the tag's **data starts inside** (#677, see *Partial bit* above) also carries the
**Partial** bit (`0x4`, `AggregateBits.Partial`). Timestamp = interval start (reverse: its later edge); value `double` (Boolean 0/1).
> **A live tag's latest interval.** Past the tag's last stored sample the value is held (stepped extrapolation, Uncertain). Under
> the SDK rule that Uncertain time is Good time, so the latest interval of a live tag normally reads **`Good`** — not Uncertain —
@@ -1004,11 +1038,27 @@ aggregates do not set Partial — a follow-up.)
- `Good` on a segment whose defining samples are Good;
- `UncertainDataSubNormal` inside or up to an outage (bridged) and past the last sample (extrapolated).
**Reverse reads (`end < start`).** Part 13 anchors a reverse read at the request's start, which is the later time. `Interpolative`
follows that: its points are `start, start − interval, …` (in the forward window's terms `End − i·interval`), latest first. A
reverse read 10:00 → 09:00 at 1 h returns the value at **10:00**, not 09:00, and no point ever falls before the window. The interval
aggregates (TimeAverage, Total, …) keep the existing backend-wide reverse rule: computed forward with intervals anchored at the
earlier time, returned in descending order. Re-anchoring those at the later time for every aggregate is a follow-up.
**Reverse reads (`end < start`).** Part 13 anchors a reverse read at the request's start, which is the later time — for every aggregate
since #677. `Interpolative`'s points are `start, start − interval, …`, latest first: a reverse read 10:00 → 09:00 at 1 h returns the
value at **10:00**, not 09:00, and no point ever falls before the window. The interval aggregates use the intervals
`(start − (k+1)·interval, start − k·interval]`, the earliest one truncated, each stamped with its later edge.
**SDK parity (#677).** `SdkAggregateParityTests` (unit, `Sql.Common.Tests`) and
`ProcessedConformance.Processed_reads_match_the_SDK_aggregate_calculators` (both SQL backends, through the real SQL) run the pinned SDK's
own calculators (`AverageAggregateCalculator`, `MinMaxAggregateCalculator`, `CountAggregateCalculator`, the base `AggregateCalculator`
for Interpolative) over the same samples, forward and reverse, and compare every interval's timestamp, value, status code and Partial
bit. The per-sample comparison covers Average, Minimum, Maximum, Count, Range, Start, End, Delta, Minimum/MaximumActualTime and
PercentGood/Bad; reverse reads over data that starts inside the window are compared too. Known differences that remain (tracked as
#680) — documented and masked in the tests, deliberately not changed here because forward results were to stay as they are — are:
`Start`/`End` (the SDK returns the raw first/last sample, of any quality, with its own timestamp and stored status; ours returns the
first/last usable value stamped with the interval, under the per-sample status rule — the values agree wherever the SDK's is not Bad);
`Delta` status (the SDK's stays Good with a Bad sample in the interval, ours is Uncertain); `PercentGood`/`PercentBad` (ours count
samples, the SDK measures duration — different definitions, so only timestamp and status are compared); the SDK reports `TimeAverage`/`Total` Uncertain whenever any region of the interval is non-Good, where ours applies the
time-based rule (a fully-covered interval bridged across a Bad sample is Good here); `Total` over an interval the data starts inside is
`TimeAverage × interval` here but the integral of the covered part in the SDK; ours flags a truncated interval Partial, the SDK does
not; the SDK's forward end-of-data Partial rule is not implemented; the SDK's per-sample Count answers 0/Good for an empty interval
inside the data and its per-sample aggregates use Good samples only; and on a reverse `Interpolative` read the SDK marks a sample exactly
at a point Interpolated (Uncertain if a Bad sample follows) where ours — like the SDK forward — returns the raw sample.
**Pushdown trade-off.** Unlike the per-sample aggregates, the time-weighted family is **not** pushed down to SQL: the backend streams
the window's rows plus the four bounds (one batch of five result sets on SQL Server; a bounds `UNION ALL` + a window select on QuestDB)
@@ -1236,8 +1286,10 @@ Every query filters `NOT deleted`; every value goes through `StoredSample`, so t
more than 126 branches). `AtTimeEvaluator` decides each value, as on `SqlServer`.
- **Events:** `[start, end)` by `ts`, ordered `(ts, event_id)`, `maxEvents + 1` fetched to produce the continuation point; a
reverse read (#675) is `(lo, hi]` ordered `(ts, event_id)` descending.
- **Processed:** one query per node; bucket index `(cast(ts AS LONG) * 10 - startUnixTicks) / intervalTicks` (never `SAMPLE BY`, whose alignment
differs), so bucket edges are identical to `SqlServer` (start-inclusive, end-exclusive, partial last bucket, interval 0 ⇒ one bucket, 100,000-bucket cap).
- **Processed:** one query per node; bucket index `(cast(ts AS LONG) * 10 - bucketBaseUnixTicks) / intervalTicks` (never `SAMPLE BY`, whose alignment
differs), so bucket edges are identical to `SqlServer` (forward start-inclusive / end-exclusive with the partial bucket last; reverse anchored at the
later time, later-edge-inclusive, partial bucket earliest — #677; interval 0 ⇒ one bucket, 100,000-bucket cap). On a forward read of any aggregate
but Average, a second `LIMIT 1` statement (capped at the window end) fetches the tag's earliest live sample for the start-of-data Partial bit.
Min/max times use `row_number()` windows with the earliest-timestamp tie-break.
### Retention (TTL)
@@ -29,6 +29,14 @@ public sealed class GatewayTagProvisioner : IHistorianProvisioning, IAsyncDispos
/// </summary>
private const uint DefaultStorageRateMs = 1000;
/// <summary>
/// Integral divisor stamped on every provisioned analog tag (#677). AVEVA's <c>Total</c> integral is
/// <c>∫ value dt (seconds) ÷ IntegralDivisor</c>; the proto's <c>double</c> default of 0 would reach the gateway as
/// "unset" (or a divide-by-zero), so 1 is sent explicitly — the value·seconds unit the SQL backends' <c>Total</c>
/// uses. (The live magnitude check against AVEVA is still open: it needs the VPN.)
/// </summary>
private const double DefaultIntegralDivisor = 1.0;
private readonly IHistorianGatewayClient _client;
private readonly ILogger<GatewayTagProvisioner> _logger;
@@ -70,6 +78,8 @@ public sealed class GatewayTagProvisioner : IHistorianProvisioning, IAsyncDispos
Description = request.Description ?? string.Empty,
// Required: a 0 storage rate makes the gateway's EnsureTags throw (see the constant).
StorageRateMs = DefaultStorageRateMs,
// Explicit: the proto3 double default (0) is not "1" (see the constant).
IntegralDivisor = DefaultIntegralDivisor,
});
}
@@ -9,21 +9,23 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb;
/// per <see cref="AggregateFamily"/>, mirroring the SQL Server backend's <c>SqlAggregateQueries</c> family split and
/// column order. Every query takes five positional parameters:
/// <list type="number">
/// <item><c>$1</c> — the (rounded) window start in 100 ns ticks since the Unix epoch
/// (<see cref="BucketPlan.StartUnixTicks"/>, <c>Bigint</c>);</item>
/// <item><c>$1</c> — the bucket base in 100 ns ticks since the Unix epoch
/// (<see cref="BucketPlan.BucketBaseUnixTicks"/>, <c>Bigint</c>: the rounded window start forward; re-anchored at
/// the later time for a reverse read, #677);</item>
/// <item><c>$2</c> — the bucket width in ticks (<see cref="BucketPlan.IntervalTicks"/>, <c>Bigint</c>);</item>
/// <item><c>$3</c> — the tag (<c>Varchar</c>);</item>
/// <item><c>$4</c>/<c>$5</c> — the forward window <c>[start, end)</c> (<c>TimestampTz</c> via
/// <see cref="QuestDbConnection.BindTimestamp"/>).</item>
/// <item><c>$4</c>/<c>$5</c> — the earlier / later window edges (<c>TimestampTz</c> via
/// <see cref="QuestDbConnection.BindTimestamp"/>): <c>[$4, $5)</c> forward, <c>($4, $5]</c> reverse.</item>
/// </list>
/// Each returns one row per <b>non-empty</b> bucket; the caller fills the empty ones.
/// </summary>
/// <remarks>
/// <para>
/// <b>Bucketing.</b> <c>b = (cast(ts AS LONG) * 10 − $1) / $2</c>. <c>cast(ts AS LONG)</c> is µs since 1970, so
/// <c>× 10</c> is ticks since 1970; both it and <c>$1</c> are µs-aligned (R6), so the integer division is
/// start-inclusive / end-exclusive to the tick, exactly like SQL Server's. Every row is at or after the start,
/// so the dividend is never negative (no truncation-toward-zero surprise), and nothing overflows before 9999.
/// <c>× 10</c> is ticks since 1970, so the integer division is exact to the tick, exactly like SQL Server's
/// (<see cref="BucketPlan.IndexOf"/> is the C# statement of the math). Every in-window row is at or after the base
/// (forward) or strictly after the window start with the base one tick past the grid origin (reverse), so the
/// dividend is never negative (no truncation-toward-zero surprise), and nothing overflows before 9999.
/// <c>SAMPLE BY</c> is deliberately not used: its alignment rules differ from ours.
/// </para>
/// <para>
@@ -60,13 +62,14 @@ internal static class QuestDbAggregateQueries
/// </summary>
/// <param name="sample">Prefixed sample table name.</param>
/// <returns>The SQL text.</returns>
public static string Simple(string sample) =>
/// <param name="reverse">True for a reverse read (window <c>($4, $5]</c>).</param>
public static string Simple(string sample, bool reverse = false) =>
"SELECT b, " + CommonCounts() + ", " +
"avg(CASE WHEN u = 1 THEN num_value END) AS avg_value, " +
"min(CASE WHEN u = 1 THEN num_value END) AS min_value, " +
"max(CASE WHEN u = 1 THEN num_value END) AS max_value, " +
"sum(CASE WHEN u = 1 THEN num_value END) AS sum_value " +
"FROM (" + Classified(sample) + ") GROUP BY b ORDER BY b";
"FROM (" + Classified(sample, reverse) + ") GROUP BY b ORDER BY b";
/// <summary>
/// Start / End / Delta: the first and last usable sample of each bucket by <c>ts</c>.
@@ -76,8 +79,9 @@ internal static class QuestDbAggregateQueries
/// </summary>
/// <param name="sample">Prefixed sample table name.</param>
/// <returns>The SQL text.</returns>
public static string StartEnd(string sample) =>
"WITH s AS (" + Classified(sample) + "), " +
/// <param name="reverse">True for a reverse read (window <c>($4, $5]</c>).</param>
public static string StartEnd(string sample, bool reverse = false) =>
"WITH s AS (" + Classified(sample, reverse) + "), " +
"st AS (SELECT b, " + CommonCounts() + " FROM s GROUP BY b), " +
"f AS (SELECT b, kind, num_value, int_value, str_value, ts, " +
"row_number() OVER (PARTITION BY b ORDER BY ts ASC) AS rf, " +
@@ -96,8 +100,9 @@ internal static class QuestDbAggregateQueries
/// <param name="sample">Prefixed sample table name.</param>
/// <param name="maximum">True for MaximumActualTime.</param>
/// <returns>The SQL text.</returns>
public static string ActualTime(string sample, bool maximum) =>
"WITH s AS (" + Classified(sample) + "), " +
/// <param name="reverse">True for a reverse read (window <c>($4, $5]</c>).</param>
public static string ActualTime(string sample, bool maximum, bool reverse = false) =>
"WITH s AS (" + Classified(sample, reverse) + "), " +
"st AS (SELECT b, " + CommonCounts() + " FROM s GROUP BY b), " +
"r AS (SELECT b, num_value, ts, " +
"row_number() OVER (PARTITION BY b ORDER BY num_value " + (maximum ? "DESC" : "ASC") + ", ts ASC) AS rn " +
@@ -144,12 +149,22 @@ internal static class QuestDbAggregateQueries
/// <summary>The bucket-index expression (exposed for tests).</summary>
internal const string BucketExpression = "(cast(ts AS LONG) * 10 - $1) / $2";
/// <summary>
/// The tag's earliest live sample (any quality) before the window end: the start-of-data Partial rule (#677; forward
/// reads only, so the forward window's exclusive end bounds the scan). Parameters <c>$1</c> tag, <c>$2</c> the window
/// end. Column: <c>ts</c>; no row when the tag has none before the end.
/// </summary>
/// <param name="sample">Prefixed sample table name.</param>
/// <returns>The SQL text.</returns>
public static string EarliestSample(string sample) =>
$"SELECT ts FROM {sample} WHERE tag = $1 AND ts < $2 AND NOT deleted ORDER BY ts ASC LIMIT 1";
/// <summary>The classified subquery <c>s</c>: the window's live samples with bucket index and the usable/good flags.</summary>
private static string Classified(string sample) =>
private static string Classified(string sample, bool reverse) =>
"SELECT " + BucketExpression + " AS b, ts, kind, num_value, int_value, str_value, " +
$"CASE WHEN status < {NotBadLimit} AND kind <> {EmptyKind} THEN 1 ELSE 0 END AS u, " +
$"CASE WHEN status < {GoodLimit} AND kind <> {EmptyKind} THEN 1 ELSE 0 END AS g " +
$"FROM {sample} WHERE tag = $3 AND ts >= $4 AND ts < $5 AND NOT deleted";
$"FROM {sample} WHERE tag = $3 AND " + (reverse ? "ts > $4 AND ts <= $5" : "ts >= $4 AND ts < $5") + " AND NOT deleted";
/// <summary>The five common count columns (after <c>b</c>).</summary>
private static string CommonCounts() =>
@@ -440,12 +440,13 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource
/// <remarks>
/// The (rounded) window is planned by <see cref="BucketPlan.Create"/>: bucket <c>i</c> =
/// <c>[start + i·interval, start + (i+1)·interval)</c>, start-inclusive / end-exclusive, the last bucket truncated
/// at <c>end</c>, interval &lt;= 0 ⇒ one bucket, <c>start == end</c> ⇒ none, a reversed window computed forward
/// and returned descending, at most <see cref="BucketPlan.MaxBuckets"/> buckets. One statement per node returns
/// the <see cref="BucketRow"/> statistics of each non-empty bucket (bucket index
/// <c>(cast(ts AS LONG)*10 − StartUnixTicks) / IntervalTicks</c>, never <c>SAMPLE BY</c>);
/// <see cref="AggregateEvaluator.Evaluate"/> produces each value and <see cref="BucketPlan.Assemble"/> fills the
/// empty buckets with <c>BadNoData</c>. The time-weighted family (#671) instead streams the window plus four bounding
/// at <c>end</c>, interval &lt;= 0 ⇒ one bucket, <c>start == end</c> ⇒ none, at most
/// <see cref="BucketPlan.MaxBuckets"/> buckets. A reversed window (#677) is anchored at its later time per Part 13 /
/// the SDK: window <c>(end, start]</c>, buckets stepping back from <c>start</c>, the partial one earliest, each stamped
/// with its later edge, returned latest first. One statement per node returns the <see cref="BucketRow"/> statistics
/// of each non-empty bucket (bucket index <c>(cast(ts AS LONG)*10 − BucketBaseUnixTicks) / IntervalTicks</c>, never
/// <c>SAMPLE BY</c>); a second fetches the tag's earliest sample for the start-of-data Partial bit; and
/// <see cref="AggregateEvaluator.EvaluateAll"/> produces the values and fills the empty buckets with <c>BadNoData</c>. The time-weighted family (#671) instead streams the window plus four bounding
/// rows into <see cref="TimeWeightedAggregator"/>, exactly as the SQL Server backend does.
/// </remarks>
/// <exception cref="HistoryAggregateNotSupportedException">The aggregate is not one of the seventeen served.</exception>
@@ -461,7 +462,6 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource
try
{
var now = _time.GetUtcNow().UtcDateTime;
var evaluated = new Dictionary<long, DataValueSnapshot>();
if (family == AggregateFamily.TimeWeighted)
{
var tw = new TimeWeightedAggregator(aggregate, plan);
@@ -471,31 +471,43 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource
RecordSuccess();
return new HistoryReadResult(twResult, ContinuationPoint: null);
}
var rows = new List<BucketRow>();
DateTime? earliest = null;
if (plan.BucketCount > 0 && await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false))
{
var sample = _conn.SampleTable;
var sql = family switch
{
AggregateFamily.Simple => QuestDbAggregateQueries.Simple(sample),
AggregateFamily.StartEnd => QuestDbAggregateQueries.StartEnd(sample),
_ => QuestDbAggregateQueries.ActualTime(sample, aggregate == HistoryAggregateType.MaximumActualTime),
AggregateFamily.Simple => QuestDbAggregateQueries.Simple(sample, plan.Reversed),
AggregateFamily.StartEnd => QuestDbAggregateQueries.StartEnd(sample, plan.Reversed),
_ => QuestDbAggregateQueries.ActualTime(sample, aggregate == HistoryAggregateType.MaximumActualTime, plan.Reversed),
};
await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false);
await using var cmd = _conn.CreateCommand(conn, sql);
QuestDbConnection.BindLong(cmd, plan.StartUnixTicks);
QuestDbConnection.BindLong(cmd, plan.IntervalTicks);
QuestDbConnection.BindText(cmd, fullReference);
QuestDbConnection.BindTimestamp(cmd, plan.StartUtc);
QuestDbConnection.BindTimestamp(cmd, plan.EndUtc);
await using var reader = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false))
await using (var cmd = _conn.CreateCommand(conn, sql))
{
var row = ReadBucket(reader, family);
evaluated[row.Bucket] = AggregateEvaluator.Evaluate(aggregate, row, plan.BucketStart(row.Bucket), now);
QuestDbConnection.BindLong(cmd, plan.BucketBaseUnixTicks);
QuestDbConnection.BindLong(cmd, plan.IntervalTicks);
QuestDbConnection.BindText(cmd, fullReference);
QuestDbConnection.BindTimestamp(cmd, plan.StartUtc);
QuestDbConnection.BindTimestamp(cmd, plan.EndUtc);
await using var reader = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false))
rows.Add(ReadBucket(reader, family));
}
// The tag's earliest live sample before the window end: the start-of-data Partial rule (#677) — forward
// reads of an aggregate other than Average only; skipped otherwise.
if (AggregateEvaluator.NeedsEarliestSample(aggregate, plan))
{
await using var cmd = _conn.CreateCommand(conn, QuestDbAggregateQueries.EarliestSample(sample));
QuestDbConnection.BindText(cmd, fullReference);
QuestDbConnection.BindTimestamp(cmd, plan.EndUtc);
await using var reader = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
if (await reader.ReadAsync(cancellationToken).ConfigureAwait(false) && !reader.IsDBNull(0))
earliest = QuestDbConnection.ReadUtc(reader, 0);
}
}
var result = plan.Assemble(evaluated, bucketStart => AggregateEvaluator.NoData(bucketStart, now));
var result = AggregateEvaluator.EvaluateAll(aggregate, plan, rows, earliest, now);
RecordSuccess();
return new HistoryReadResult(result, ContinuationPoint: null);
@@ -11,8 +11,10 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// every sample, status Good. Otherwise no usable sample ⇒ <c>BadNoData</c>; a numeric aggregate with a usable
/// String/DateTime sample ⇒ <c>BadAggregateInvalidInputs</c>; else the value, Good when every sample is Good,
/// <c>UncertainDataSubNormal</c> when any is Bad or Uncertain. Every status — including <c>BadNoData</c> —
/// carries the Calculated info bits. Timestamps are the bucket start, except Minimum/MaximumActualTime
/// (the sample's time; ties ⇒ earliest).
/// carries the Calculated info bits. Timestamps are the bucket's <see cref="BucketPlan.BucketTimestamp"/> (its
/// start forward, its later edge reverse), except Minimum/MaximumActualTime (the sample's time; ties ⇒ earliest).
/// Every aggregate but <c>Average</c> also carries the Partial bit on the bucket the data starts inside
/// (<see cref="EvaluateAll"/>).
/// </para>
/// <para>
/// Values: Count is <see cref="int"/>; Start/End keep the sample's type; every other aggregate is
@@ -26,7 +28,7 @@ public static class AggregateEvaluator
/// <summary>Evaluates <paramref name="aggregate"/> over one non-empty bucket.</summary>
/// <param name="aggregate">Requested aggregate.</param>
/// <param name="r">The bucket's statistics.</param>
/// <param name="bucketStart">The bucket's start time (the result timestamp).</param>
/// <param name="bucketStart">The bucket's result timestamp (<see cref="BucketPlan.BucketTimestamp"/>).</param>
/// <param name="now">Server timestamp.</param>
/// <returns>The calculated value.</returns>
/// <exception cref="HistoryAggregateNotSupportedException">The aggregate is not one of the per-sample (Simple /
@@ -70,6 +72,58 @@ public static class AggregateEvaluator
}
}
/// <summary>
/// True for the per-sample aggregates the SDK sets the start-of-data Partial bit on (every calculator but
/// <c>AverageAggregateCalculator</c>'s plain <c>Average</c>, which turns <c>SetPartialBit</c> off).
/// </summary>
/// <param name="aggregate">Requested aggregate.</param>
/// <returns>Whether the start-of-data Partial rule applies.</returns>
public static bool SetsStartOfDataPartial(HistoryAggregateType aggregate) => aggregate != HistoryAggregateType.Average;
/// <summary>
/// True when a per-sample read must fetch the tag's earliest sample: the start-of-data Partial rule applies only to a
/// <b>forward</b> read (as in the SDK, whose reverse read keys <c>m_startOfData</c> on the first value it is fed — the
/// latest) of an aggregate other than <c>Average</c>. Otherwise the backend skips that query.
/// </summary>
/// <param name="aggregate">Requested aggregate.</param>
/// <param name="plan">The bucket plan.</param>
/// <returns>Whether the earliest sample is needed.</returns>
public static bool NeedsEarliestSample(HistoryAggregateType aggregate, BucketPlan plan)
{
ArgumentNullException.ThrowIfNull(plan);
return !plan.Reversed && plan.BucketCount > 0 && SetsStartOfDataPartial(aggregate);
}
/// <summary>
/// Evaluates every non-empty bucket row and assembles one result per interval in the plan's order (descending for a
/// reverse read): each value stamped with <see cref="BucketPlan.BucketTimestamp"/> (the earlier edge forward, the
/// later edge reverse); empty buckets <c>BadNoData</c>; and — on a forward read, for every aggregate but
/// <c>Average</c> — the Partial bit on a non-Bad result of the bucket the tag's data starts inside (#677, the SDK's
/// <c>m_startOfData</c> rule; the SDK applies it to forward reads only, <see cref="NeedsEarliestSample"/>).
/// </summary>
/// <param name="aggregate">A per-sample (Simple / StartEnd / ActualTime) aggregate.</param>
/// <param name="plan">The bucket plan.</param>
/// <param name="rows">The backend's non-empty bucket rows (ascending bucket index).</param>
/// <param name="earliestSampleUtc">The tag's earliest stored sample of any quality, or null when none.</param>
/// <param name="now">Server timestamp.</param>
/// <returns><see cref="BucketPlan.BucketCount"/> results.</returns>
public static IReadOnlyList<DataValueSnapshot> EvaluateAll(HistoryAggregateType aggregate, BucketPlan plan,
IEnumerable<BucketRow> rows, DateTime? earliestSampleUtc, DateTime now)
{
ArgumentNullException.ThrowIfNull(plan);
ArgumentNullException.ThrowIfNull(rows);
var evaluated = new Dictionary<long, DataValueSnapshot>();
var partialRule = NeedsEarliestSample(aggregate, plan);
foreach (var row in rows)
{
var v = Evaluate(aggregate, row, plan.BucketTimestamp(row.Bucket), now);
if (partialRule && !StatusBits.IsBad(v.StatusCode) && plan.DataStartsInside(row.Bucket, earliestSampleUtc))
v = v with { StatusCode = v.StatusCode | HistorianInfoBits.HistorianPartial };
evaluated[row.Bucket] = v;
}
return plan.Assemble(evaluated, t => NoData(t, now));
}
/// <summary>The result for a bucket with no samples (or no usable sample): <c>BadNoData</c> + Calculated.</summary>
/// <param name="t">The bucket's start time.</param>
/// <param name="now">Server timestamp.</param>
@@ -3,12 +3,29 @@ using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <summary>
/// The bucket layout of a processed (aggregate) read. Bucket <c>i</c> = <c>[start + i·interval,
/// start + (i+1)·interval)</c> (start-inclusive, end-exclusive to the tick; the last bucket is truncated at
/// <c>end</c>). An interval &lt;= 0 ⇒ one interval <c>[start, end)</c>. <c>start == end</c> ⇒ no intervals.
/// <c>end &lt; start</c> (reverse) ⇒ the window is swapped, computed forward, and assembled in descending bucket
/// order (buckets stay anchored at the earlier time).
/// The bucket layout of a processed (aggregate) read, per OPC UA Part 13 / the SDK's <c>AggregateCalculator</c>.
/// </summary>
/// <remarks>
/// <para>
/// <b>Forward</b> (<c>start &lt; end</c>): the window is <c>[start, end)</c>; bucket <c>i</c> =
/// <c>[start + i·interval, start + (i+1)·interval)</c>, the last bucket truncated at <c>end</c> (the partial one is
/// the <i>latest</i>); each bucket's timestamp is its start (earlier edge); results ascend.
/// </para>
/// <para>
/// <b>Reverse</b> (<c>end &lt; start</c>, #677): the read is anchored at the request's start — its <i>later</i>
/// time — and steps back. The window is <c>(end, start]</c> (Part 11: the request's start is inclusive, its end
/// exclusive, so a reverse read includes its later time and excludes its earlier one — the same window as a reverse
/// raw read); result <c>k</c> = <c>(start − (k+1)·interval, start − k·interval]</c>, clipped at <c>end</c> (the
/// partial bucket is the <i>earliest</i>); each bucket's timestamp is its later edge <c>start − k·interval</c>;
/// results descend. This is exactly the SDK's <c>CreateSlice</c> / <c>GetTimestamp</c> / backward
/// <c>GetValues</c>.
/// </para>
/// <para>
/// Internally buckets are indexed in ascending time (<c>0</c> = earliest) for both directions; <see cref="Assemble"/>
/// reverses the list for a reverse read. <see cref="StartUtc"/>/<see cref="EndUtc"/> are always the earlier / later
/// window edges. An interval &lt;= 0 ⇒ one interval over the whole window. <c>start == end</c> ⇒ no intervals.
/// </para>
/// </remarks>
public sealed class BucketPlan
{
/// <summary>Spec §9.7: a processed read covering more intervals than this is refused.</summary>
@@ -21,12 +38,15 @@ public sealed class BucketPlan
Reversed = reversed;
IntervalTicks = intervalTicks;
BucketCount = bucketCount;
// Forward buckets are anchored at the earlier edge; reverse ones at the later edge, so the grid origin sits a
// whole number of intervals before it (possibly before the window start — the earliest bucket is clipped).
OriginTicks = reversed ? endUtc.Ticks - bucketCount * intervalTicks : startUtc.Ticks;
}
/// <summary>The earlier end of the window (the forward start).</summary>
/// <summary>The earlier edge of the window.</summary>
public DateTime StartUtc { get; }
/// <summary>The later end of the window (the forward, exclusive end).</summary>
/// <summary>The later edge of the window.</summary>
public DateTime EndUtc { get; }
/// <summary>True when the request came in reversed (<c>end &lt; start</c>).</summary>
@@ -38,9 +58,30 @@ public sealed class BucketPlan
/// <summary>Number of buckets (0 for an empty window).</summary>
public long BucketCount { get; }
/// <summary>
/// The tick at which the (unclipped) bucket grid starts: <see cref="StartUtc"/> forward, <c>EndUtc − n·interval</c>
/// reverse. May precede <see cref="StartUtc"/> (and even <see cref="DateTime.MinValue"/>), so it is a raw tick count.
/// </summary>
public long OriginTicks { get; }
/// <summary><see cref="StartUtc"/> as 100 ns ticks since the Unix epoch.</summary>
public long StartUnixTicks => (StartUtc - DateTime.UnixEpoch).Ticks;
/// <summary>
/// The constant a backend adds to <c>elapsed = t − StartUtc</c> (ticks) before the integer division by
/// <see cref="IntervalTicks"/> to get a sample's ascending bucket index: <c>0</c> forward (<c>[lo, hi)</c> buckets);
/// <c>(StartUtc − OriginTicks) − 1</c> reverse, where buckets are <c>(lo, hi]</c> — index <c>⌈(t − origin)/I⌉ − 1
/// = (t − origin − 1) / I</c>. Never makes the dividend negative for an in-window sample (reverse excludes
/// <c>t == StartUtc</c>).
/// </summary>
public long BucketIndexAdjustTicks => Reversed ? StartUtc.Ticks - OriginTicks - 1 : 0;
/// <summary>
/// <see cref="StartUnixTicks"/> minus <see cref="BucketIndexAdjustTicks"/>: the base a backend subtracts from a
/// sample's Unix-epoch ticks to get the same dividend (<c>(t − base) / IntervalTicks</c>).
/// </summary>
public long BucketBaseUnixTicks => StartUnixTicks - BucketIndexAdjustTicks;
/// <summary>Plans the buckets of a processed read.</summary>
/// <param name="startUtc">Requested start.</param>
/// <param name="endUtc">Requested end (earlier than start for a reverse read).</param>
@@ -64,15 +105,59 @@ public sealed class BucketPlan
return new BucketPlan(start, end, reverse, intervalTicks, bucketCount);
}
/// <summary>The start time of bucket <paramref name="index"/> (UTC kind).</summary>
/// <param name="index">Zero-based bucket index.</param>
/// <returns>The bucket start.</returns>
public DateTime BucketStart(long index) =>
DateTime.SpecifyKind(new DateTime(StartUtc.Ticks + index * IntervalTicks), DateTimeKind.Utc);
/// <summary>The earlier edge of bucket <paramref name="index"/> (ascending index), in ticks, clipped to the window.</summary>
/// <param name="index">Zero-based ascending bucket index.</param>
/// <returns>The earlier edge.</returns>
public long LowerTicks(long index) => Math.Max(OriginTicks + index * IntervalTicks, StartUtc.Ticks);
/// <summary>One snapshot per bucket in window order; missing buckets get <paramref name="empty"/>; reversed if the window was.</summary>
/// <param name="byBucket">Evaluated snapshots of the non-empty buckets, by bucket index.</param>
/// <param name="empty">Builds the snapshot of a bucket with no row, from its start time.</param>
/// <summary>The later edge of bucket <paramref name="index"/> (ascending index), in ticks, clipped to the window.</summary>
/// <param name="index">Zero-based ascending bucket index.</param>
/// <returns>The later edge.</returns>
public long UpperTicks(long index) => Math.Min(OriginTicks + (index + 1) * IntervalTicks, EndUtc.Ticks);
/// <summary>
/// The result timestamp of bucket <paramref name="index"/> (UTC kind): its earlier edge forward, its later edge
/// reverse (the SDK's <c>GetTimestamp</c>).
/// </summary>
/// <param name="index">Zero-based ascending bucket index.</param>
/// <returns>The bucket's timestamp.</returns>
public DateTime BucketTimestamp(long index) =>
new(Reversed ? UpperTicks(index) : LowerTicks(index), DateTimeKind.Utc);
/// <summary>True when bucket <paramref name="index"/> is shorter than the interval (the clipped one).</summary>
/// <param name="index">Zero-based ascending bucket index.</param>
/// <returns>Whether the bucket is truncated.</returns>
public bool IsTruncated(long index) => UpperTicks(index) - LowerTicks(index) < IntervalTicks;
/// <summary>
/// The ascending bucket index holding a sample at <paramref name="t"/>, or null when it is outside the window
/// (<c>[start, end)</c> forward, <c>(start, end]</c> reverse). The C# statement of the bucket math every backend's
/// SQL mirrors.
/// </summary>
/// <param name="t">Sample time.</param>
/// <returns>The bucket index, or null.</returns>
public long? IndexOf(DateTime t)
{
var x = t.Ticks;
if (BucketCount == 0) return null;
var inside = Reversed ? x > StartUtc.Ticks && x <= EndUtc.Ticks : x >= StartUtc.Ticks && x < EndUtc.Ticks;
if (!inside) return null;
return (x - StartUtc.Ticks + BucketIndexAdjustTicks) / IntervalTicks;
}
/// <summary>
/// True when the earliest stored sample of the tag lies strictly inside bucket <paramref name="index"/> — the data
/// starts part-way through the interval (no bounding value before it), so Part 13 / the SDK set the Partial bit.
/// </summary>
/// <param name="index">Zero-based ascending bucket index.</param>
/// <param name="earliestSampleUtc">The tag's earliest stored sample (any quality), or null when it has none.</param>
/// <returns>Whether the data starts inside the bucket.</returns>
public bool DataStartsInside(long index, DateTime? earliestSampleUtc) =>
earliestSampleUtc is { } e && e.Ticks > LowerTicks(index) && e.Ticks < UpperTicks(index);
/// <summary>One snapshot per bucket in result order; missing buckets get <paramref name="empty"/>; descending if reversed.</summary>
/// <param name="byBucket">Evaluated snapshots of the non-empty buckets, by ascending bucket index.</param>
/// <param name="empty">Builds the snapshot of a bucket with no row, from its <see cref="BucketTimestamp"/>.</param>
/// <returns><see cref="BucketCount"/> snapshots.</returns>
public IReadOnlyList<DataValueSnapshot> Assemble(IReadOnlyDictionary<long, DataValueSnapshot> byBucket,
Func<DateTime, DataValueSnapshot> empty)
@@ -81,10 +166,7 @@ public sealed class BucketPlan
ArgumentNullException.ThrowIfNull(empty);
var result = new List<DataValueSnapshot>((int)BucketCount);
for (long i = 0; i < BucketCount; i++)
{
var bucketStart = BucketStart(i);
result.Add(byBucket.TryGetValue(i, out var v) ? v : empty(bucketStart));
}
result.Add(byBucket.TryGetValue(i, out var v) ? v : empty(BucketTimestamp(i)));
if (Reversed) result.Reverse();
return result;
}
@@ -18,7 +18,7 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// </para>
/// <para>
/// <b>The signal (server defaults: TreatUncertainAsBad = false, PercentDataBad = PercentDataGood = 100,
/// UseSlopedExtrapolation = false; a client <c>AggregateConfiguration</c> is not honoured).</b> Between a usable
/// UseSlopedExtrapolation = false; the node manager refuses any other client <c>AggregateConfiguration</c>).</b> Between a usable
/// sample <c>p</c> and the next sample <c>s</c>:
/// <list type="bullet">
/// <item><c>s</c> usable, both Double/Int64 ⇒ <b>sloped</b> (linear <c>p → s</c>); quality = worse of the two.</item>
@@ -51,7 +51,9 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// otherwise (PercentDataGood = PercentDataBad = 100) <c>UncertainDataSubNormal</c>. Time before the first sample
/// counts as Bad (no data) time. A usable String/DateTime sample contributing to the interval ⇒
/// <c>BadAggregateInvalidInputs</c>. Every status carries the Calculated bit, plus the Partial bit on a non-Bad
/// result of a truncated last interval; the timestamp is the interval start; the value is a
/// result of the truncated interval or of the interval the data starts inside (the first sample fed lies strictly
/// within it — forward reads only, as in the SDK, #677); the timestamp is the interval's <see cref="BucketPlan.BucketTimestamp"/> (its start forward, its
/// later edge reverse); the value is a
/// <see cref="double"/> (Boolean = 0/1, Int64 as double).
/// </para>
/// <para>
@@ -112,6 +114,7 @@ public sealed class TimeWeightedAggregator
private StoredSample? _prevAny;
private StoredSample? _prevUsable;
private DateTime? _firstSampleUtc; // the earliest sample fed (any quality): where the data starts
private long _nextPoint; // next point (ascending time) not yet classified
private readonly List<PendingPoint> _pending = []; // points awaiting the next usable sample
private bool _finished;
@@ -169,6 +172,7 @@ public sealed class TimeWeightedAggregator
if (_finished) throw new InvalidOperationException("The aggregation is already finished.");
var t = s.TimeUtc.Ticks;
if (_prevAny is { } last && t <= last.TimeUtc.Ticks) return;
_firstSampleUtc ??= s.TimeUtc;
// Interpolative: classify every bucket start before this sample by the simple-bounding signal there.
ClassifyPointsBefore(t);
@@ -234,19 +238,21 @@ public sealed class TimeWeightedAggregator
? point with { ServerTimestampUtc = now }
: new DataValueSnapshot(null, HistorianStatusCodes.BadNoData, PointTime(i), now);
var bucketStart = _plan.BucketStart(i);
var stamp = _plan.BucketTimestamp(i);
var duration = BucketEnd(i) - BucketStartTicks(i);
var partial = duration < _plan.IntervalTicks; // only the truncated last interval
// Partial: the truncated interval (the latest forward, the earliest reverse), or the one the data starts inside
// (no bounding value before it — the SDK's m_startOfData rule, #677), which the SDK applies to forward reads only.
var partial = _plan.IsTruncated(i) || (!_plan.Reversed && _plan.DataStartsInside(i, _firstSampleUtc));
DataValueSnapshot Calc(object? value, uint status)
{
var code = HistorianInfoBits.WithCalculated(status);
if (partial && !StatusBits.IsBad(status)) code |= HistorianInfoBits.HistorianPartial;
return new DataValueSnapshot(value, code, bucketStart, now);
return new DataValueSnapshot(value, code, stamp, now);
}
if (_invalid[i]) return Calc(null, HistorianStatusCodes.BadAggregateInvalidInputs);
if (_usableTicks[i] == 0)
return _badTicks[i] > 0 ? Calc(null, HistorianStatusCodes.Bad) : AggregateEvaluator.NoData(bucketStart, now);
return _badTicks[i] > 0 ? Calc(null, HistorianStatusCodes.Bad) : AggregateEvaluator.NoData(stamp, now);
var status = _usableTicks[i] == duration ? HistorianStatusCodes.Good : HistorianStatusCodes.UncertainDataSubNormal;
return _aggregate switch
@@ -269,9 +275,12 @@ public sealed class TimeWeightedAggregator
a.Value.Kind is HistorianValueKind.Double or HistorianValueKind.Int64
&& b.Value.Kind is HistorianValueKind.Double or HistorianValueKind.Int64;
private long BucketStartTicks(long i) => _start + i * _plan.IntervalTicks;
private long BucketStartTicks(long i) => _plan.LowerTicks(i);
private long BucketEnd(long i) => Math.Min(BucketStartTicks(i) + _plan.IntervalTicks, _end);
private long BucketEnd(long i) => _plan.UpperTicks(i);
/// <summary>The ascending index of the bucket whose (half-open) span holds tick <paramref name="t"/> (t ≥ window start).</summary>
private long FirstBucketAt(long t) => Math.Max(0, (t - _plan.OriginTicks) / _plan.IntervalTicks);
private void AddRaw(StoredSample from, long toTicks, double toValue) =>
Distribute(from, toTicks, toValue, (i, area, ticks) =>
@@ -286,7 +295,7 @@ public sealed class TimeWeightedAggregator
var lo = Math.Max(from.TimeUtc.Ticks, _start);
var hi = Math.Min(toTicks, _end);
if (lo >= hi || _plan.BucketCount == 0) return;
for (var i = (lo - _start) / _plan.IntervalTicks; i < _plan.BucketCount; i++)
for (var i = FirstBucketAt(lo); i < _plan.BucketCount; i++)
{
var bs = BucketStartTicks(i);
if (bs >= hi) break;
@@ -320,7 +329,7 @@ public sealed class TimeWeightedAggregator
var span = (double)(toTicks - t0);
double ValueAt(long x) => toValue == v0 ? v0 : v0 + (toValue - v0) * ((x - t0) / span);
for (var i = (lo - _start) / _plan.IntervalTicks; i < _plan.BucketCount; i++)
for (var i = FirstBucketAt(lo); i < _plan.BucketCount; i++)
{
var bs = BucketStartTicks(i);
if (bs >= hi) break;
@@ -335,13 +344,12 @@ public sealed class TimeWeightedAggregator
// ------------------------------------------------------------------------------------------ Interpolative
/// <summary>
/// The time of Interpolative point <paramref name="j"/> (ascending time). Forward: the bucket starts
/// <c>start + j·interval</c>. Reverse (Part 13 anchors a reverse read at its later time, the request's start):
/// <c>End − (n−1−j)·interval</c>, i.e. <c>End, End − interval, …</c> once <see cref="BucketPlan.Assemble"/> has
/// The time of Interpolative point <paramref name="j"/> (ascending time): the bucket's timestamp. Forward: the
/// bucket starts <c>start + j·interval</c>. Reverse (Part 13 anchors a reverse read at its later time, the
/// request's start): the later edges <c>End, End − interval, …</c> once <see cref="BucketPlan.Assemble"/> has
/// reversed the order — so a reverse read 10:00 → 09:00 at 1 h yields the value at 10:00.
/// </summary>
private long PointTicks(long j) =>
_plan.Reversed ? _end - (_plan.BucketCount - 1 - j) * _plan.IntervalTicks : BucketStartTicks(j);
private long PointTicks(long j) => _plan.BucketTimestamp(j).Ticks;
private DateTime PointTime(long j) => new(PointTicks(j), DateTimeKind.Utc);
@@ -5,13 +5,18 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer;
/// <summary>
/// Push-down T-SQL for processed (aggregate) reads (spec §4 "Processed"), one query per
/// <see cref="AggregateFamily"/> (<see cref="AggregateFamilies.FamilyOf"/> picks it). Every query takes
/// <c>@id</c> (int TagId), <c>@start</c>/<c>@end</c> (<c>datetime2(7)</c>, the forward window
/// <c>[start, end)</c>) and <c>@intervalTicks</c> (bigint bucket width in 100 ns ticks), and returns one row
/// per <b>non-empty</b> bucket. The caller generates every interval and fills the empty ones.
/// <c>@id</c> (int TagId), <c>@start</c>/<c>@end</c> (<c>datetime2(7)</c>, the earlier / later window edges —
/// <see cref="BucketPlan.StartUtc"/>/<see cref="BucketPlan.EndUtc"/>), <c>@intervalTicks</c> (bigint bucket width in
/// 100 ns ticks) and <c>@bucketAdjustTicks</c> (bigint, <see cref="BucketPlan.BucketIndexAdjustTicks"/>), and returns
/// one row per <b>non-empty</b> bucket (ascending bucket index). The caller generates every interval and fills the
/// empty ones.
/// </summary>
/// <remarks>
/// <para>
/// <b>Bucketing.</b> <c>b = elapsedTicks / @intervalTicks</c>, where
/// <b>Bucketing.</b> <c>b = (elapsedTicks + @bucketAdjustTicks) / @intervalTicks</c> (#677: the adjust is 0 for a
/// forward read and re-anchors a reverse read's grid at its later time — <see cref="BucketPlan.IndexOf"/> is the C#
/// statement of the same math). The window is <c>[@start, @end)</c> forward and <c>(@start, @end]</c> reverse (a
/// reverse read includes its later time), selected by the <c>reverse</c> argument. <c>elapsedTicks</c> is
/// <c>elapsedTicks = DATEDIFF_BIG(microsecond, @start, t) * 10 + tick-of-microsecond(t) − tick-of-microsecond(@start)</c>
/// and <c>tick-of-microsecond(x) = (DATEPART(nanosecond, x) / 100) % 10</c>. The spec sketched
/// <c>millisecond</c>, but <c>DATEDIFF</c> counts unit <i>boundaries crossed</i>, not elapsed time, so with a
@@ -47,8 +52,9 @@ internal static class SqlAggregateQueries
/// </summary>
/// <param name="sample">Schema-qualified <c>Sample</c> table name.</param>
/// <returns>The SQL text.</returns>
public static string Simple(string sample) =>
Classified(sample) +
/// <param name="reverse">True for a reverse read (window <c>(@start, @end]</c>).</param>
public static string Simple(string sample, bool reverse = false) =>
Classified(sample, reverse) +
"SELECT b, " + CommonCounts() + ", " +
"AVG(CASE WHEN u = 1 THEN NumValue END) AS AvgValue, " +
"MIN(CASE WHEN u = 1 THEN NumValue END) AS MinValue, " +
@@ -63,8 +69,9 @@ internal static class SqlAggregateQueries
/// </summary>
/// <param name="sample">Schema-qualified <c>Sample</c> table name.</param>
/// <returns>The SQL text.</returns>
public static string StartEnd(string sample) =>
Classified(sample) +
/// <param name="reverse">True for a reverse read (window <c>(@start, @end]</c>).</param>
public static string StartEnd(string sample, bool reverse = false) =>
Classified(sample, reverse) +
", st AS (SELECT b, " + CommonCounts() + " FROM s GROUP BY b)" +
", f AS (SELECT b, ValueKind, NumValue, IntValue, StrValue, SourceTimeUtc, " +
"ROW_NUMBER() OVER (PARTITION BY b ORDER BY SourceTimeUtc ASC) AS rf, " +
@@ -82,8 +89,9 @@ internal static class SqlAggregateQueries
/// <param name="sample">Schema-qualified <c>Sample</c> table name.</param>
/// <param name="maximum">True for MaximumActualTime.</param>
/// <returns>The SQL text.</returns>
public static string ActualTime(string sample, bool maximum) =>
Classified(sample) +
/// <param name="reverse">True for a reverse read (window <c>(@start, @end]</c>).</param>
public static string ActualTime(string sample, bool maximum, bool reverse = false) =>
Classified(sample, reverse) +
", st AS (SELECT b, " + CommonCounts() + " FROM s GROUP BY b)" +
", r AS (SELECT b, NumValue, SourceTimeUtc, " +
"ROW_NUMBER() OVER (PARTITION BY b ORDER BY NumValue " + (maximum ? "DESC" : "ASC") + ", SourceTimeUtc ASC) AS rn " +
@@ -121,14 +129,26 @@ internal static class SqlAggregateQueries
$"SELECT TOP 1 {cols} {key}SourceTimeUtc >= @end{usable} ORDER BY SourceTimeUtc ASC;";
}
/// <summary>
/// The earliest stored sample of the tag (any quality), one row (<c>NULL</c> when none): the start-of-data Partial
/// rule (#677; forward reads only). A <c>MIN</c> over the <c>(TagId, SourceTimeUtc)</c> key — a single seek — bounded
/// by the window end (a sample at or past it cannot lie inside an interval).
/// </summary>
/// <param name="sample">Schema-qualified <c>Sample</c> table name.</param>
/// <returns>The SQL text.</returns>
public static string EarliestSample(string sample) =>
$"SELECT MIN(SourceTimeUtc) FROM {sample} WHERE TagId = @id AND SourceTimeUtc < @end;";
/// <summary>The <c>s</c> CTE: the window's samples with bucket index and the usable/good flags.</summary>
private static string Classified(string sample) =>
private static string Classified(string sample, bool reverse) =>
"WITH s AS (SELECT (DATEDIFF_BIG(microsecond, @start, SourceTimeUtc) * 10 " +
"+ (DATEPART(nanosecond, SourceTimeUtc) / 100) % 10 - (DATEPART(nanosecond, @start) / 100) % 10) / @intervalTicks AS b, " +
"+ (DATEPART(nanosecond, SourceTimeUtc) / 100) % 10 - (DATEPART(nanosecond, @start) / 100) % 10 " +
"+ @bucketAdjustTicks) / @intervalTicks AS b, " +
"SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, " +
"CASE WHEN StatusCode >= 0 AND ValueKind <> 0 THEN 1 ELSE 0 END AS u, " +
"CASE WHEN StatusCode >= 0 AND StatusCode < 1073741824 AND ValueKind <> 0 THEN 1 ELSE 0 END AS g " +
$"FROM {sample} WHERE TagId = @id AND SourceTimeUtc >= @start AND SourceTimeUtc < @end) ";
$"FROM {sample} WHERE TagId = @id AND " +
(reverse ? "SourceTimeUtc > @start AND SourceTimeUtc <= @end) " : "SourceTimeUtc >= @start AND SourceTimeUtc < @end) ");
/// <summary>The five common count columns (after <c>b</c>).</summary>
private static string CommonCounts() =>
@@ -400,16 +400,18 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
/// One result per interval of <c>[start, end)</c>, bucket <c>i</c> = <c>[start + i·interval,
/// start + (i+1)·interval)</c> (start-inclusive, end-exclusive to the tick; the last bucket is truncated at
/// <c>end</c>). <paramref name="interval"/> &lt;= 0 ⇒ one interval <c>[start, end)</c>. <c>start == end</c>
/// ⇒ no intervals. <c>end &lt; start</c> (reverse) ⇒ the window is swapped, computed forward, and returned
/// in descending bucket order (buckets stay anchored at the earlier time).
/// ⇒ no intervals. <c>end &lt; start</c> (reverse, #677) ⇒ Part 13 / SDK anchoring: window <c>(end, start]</c>,
/// buckets stepping back from <c>start</c> (the partial one is the earliest), each stamped with its later edge,
/// returned latest first (<see cref="BucketPlan"/>).
/// </para>
/// <para>
/// Per interval: no samples ⇒ <c>BadNoData</c>. PercentGood / PercentBad ⇒ <c>100 × good|bad / all</c> over
/// every sample, status Good. Otherwise no usable sample ⇒ <c>BadNoData</c>; a numeric aggregate with a usable
/// String/DateTime sample ⇒ <c>BadAggregateInvalidInputs</c>; else the value, Good when every sample is Good,
/// <c>UncertainDataSubNormal</c> when any is Bad or Uncertain. Every status — including <c>BadNoData</c> —
/// carries the Calculated info bits. Timestamps are the bucket start, except Minimum/MaximumActualTime
/// (the sample's time; ties ⇒ earliest).
/// carries the Calculated info bits. Timestamps are the bucket start (its later edge on a reverse read), except
/// Minimum/MaximumActualTime (the sample's time; ties ⇒ earliest). Every aggregate but Average sets the Partial
/// bit on the bucket the tag's earliest sample lies strictly inside (#677).
/// </para>
/// <para>
/// Values: Count is <see cref="int"/>; Start/End keep the sample's type; every other aggregate is
@@ -432,7 +434,6 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
try
{
var now = _time.GetUtcNow().UtcDateTime;
var evaluated = new Dictionary<long, DataValueSnapshot>();
var tagId = plan.BucketCount == 0 ? null : await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false);
if (family == AggregateFamily.TimeWeighted)
{
@@ -442,30 +443,37 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource
RecordSuccess();
return new HistoryReadResult(twResult, ContinuationPoint: null);
}
var rows = new List<BucketRow>();
DateTime? earliest = null;
if (tagId is { } id)
{
var sample = _conn.Q("Sample");
var sql = family switch
{
AggregateFamily.Simple => SqlAggregateQueries.Simple(sample),
AggregateFamily.StartEnd => SqlAggregateQueries.StartEnd(sample),
_ => SqlAggregateQueries.ActualTime(sample, aggregate == HistoryAggregateType.MaximumActualTime),
AggregateFamily.Simple => SqlAggregateQueries.Simple(sample, plan.Reversed),
AggregateFamily.StartEnd => SqlAggregateQueries.StartEnd(sample, plan.Reversed),
_ => SqlAggregateQueries.ActualTime(sample, aggregate == HistoryAggregateType.MaximumActualTime, plan.Reversed),
};
// Second result set: the tag's earliest sample (the start-of-data Partial rule, #677) — forward reads of an
// aggregate other than Average only.
var needsEarliest = AggregateEvaluator.NeedsEarliestSample(aggregate, plan);
if (needsEarliest) sql += " " + SqlAggregateQueries.EarliestSample(sample);
await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false);
await using var cmd = _conn.CreateCommand(conn, sql);
cmd.Parameters.Add(new SqlParameter("@id", SqlDbType.Int) { Value = id });
SqlHistorianConnection.AddDateTime2(cmd, "@start", DateTime.SpecifyKind(plan.StartUtc, DateTimeKind.Utc));
SqlHistorianConnection.AddDateTime2(cmd, "@end", DateTime.SpecifyKind(plan.EndUtc, DateTimeKind.Utc));
cmd.Parameters.Add(new SqlParameter("@intervalTicks", SqlDbType.BigInt) { Value = plan.IntervalTicks });
cmd.Parameters.Add(new SqlParameter("@bucketAdjustTicks", SqlDbType.BigInt) { Value = plan.BucketIndexAdjustTicks });
await using var reader = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false);
while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false))
{
var row = ReadBucket(reader, family);
evaluated[row.Bucket] = AggregateEvaluator.Evaluate(aggregate, row, plan.BucketStart(row.Bucket), now);
}
rows.Add(ReadBucket(reader, family));
if (needsEarliest && await reader.NextResultAsync(cancellationToken).ConfigureAwait(false)
&& await reader.ReadAsync(cancellationToken).ConfigureAwait(false) && !reader.IsDBNull(0))
earliest = DateTime.SpecifyKind(reader.GetDateTime(0), DateTimeKind.Utc);
}
var result = plan.Assemble(evaluated, bucketStart => AggregateEvaluator.NoData(bucketStart, now));
var result = AggregateEvaluator.EvaluateAll(aggregate, plan, rows, earliest, now);
RecordSuccess();
return new HistoryReadResult(result, ContinuationPoint: null);
@@ -5051,6 +5051,8 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
{
// OPC UA ProcessingInterval is a Duration in milliseconds — convert once per batch.
var interval = TimeSpan.FromMilliseconds(details.ProcessingInterval);
// #677: a client AggregateConfiguration other than the server defaults is refused per node, never silently ignored.
var configurationServable = ServerAggregateConfiguration.IsServable(details.AggregateConfiguration);
nodesToProcess = AuthorizeHistoryRead(context, nodesToProcess, results, errors);
WithHistoryReadBatchLimiter(nodesToProcess, results, errors, () =>
{
@@ -5069,6 +5071,12 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2
continue;
}
if (!configurationServable)
{
errors[handle.Index] = StatusCodes.BadAggregateConfigurationRejected;
continue;
}
// Processed is SINGLE-SHOT (no continuation point). Unlike Raw, ReadProcessedDetails carries
// NO client count cap (NumValuesPerNode) — the bucket count is deterministic (window / interval)
// and the single-shot backend returns every bucket in one read, so there is no "full page ⇒
@@ -0,0 +1,62 @@
using Opc.Ua;
namespace ZB.MOM.WW.OtOpcUa.OpcUaServer;
/// <summary>
/// The Part 13 <see cref="AggregateConfiguration"/> processed reads are evaluated under (#677), and the rule for a
/// client-supplied one in <c>ReadProcessedDetails</c>.
/// </summary>
/// <remarks>
/// <para>
/// The aggregate evaluators (<c>AggregateEvaluator</c> / <c>TimeWeightedAggregator</c> in
/// <c>Driver.Historian.Sql.Common</c>) are written against fixed settings, not parameterised: Uncertain samples and
/// Uncertain time count as Good (<see cref="TreatUncertainAsBad"/> = false); an interval is Good only when all of
/// it is (<see cref="PercentDataGood"/> = 100) and Bad only when none of it is non-Bad
/// (<see cref="PercentDataBad"/> = 100), otherwise <c>UncertainDataSubNormal</c>; the value past the last sample
/// is held (stepped, <see cref="UseSlopedExtrapolation"/> = false). These are also the SDK's own server defaults
/// (<c>AggregateManager.GetDefaultConfiguration</c>), which a test pins.
/// </para>
/// <para>
/// A request asking for <c>UseServerCapabilitiesDefaults</c>, or spelling out exactly these values, is served;
/// any other configuration is refused per node with <c>BadAggregateConfigurationRejected</c> rather than silently
/// answered under settings the client did not ask for (it used to be ignored).
/// </para>
/// </remarks>
public static class ServerAggregateConfiguration
{
/// <summary>Uncertain data is not treated as Bad.</summary>
public const bool TreatUncertainAsBad = false;
/// <summary>An interval is Bad only when 100 % of it is Bad.</summary>
public const byte PercentDataBad = 100;
/// <summary>An interval is Good only when 100 % of it is Good.</summary>
public const byte PercentDataGood = 100;
/// <summary>The value past the last sample is held (stepped), not sloped.</summary>
public const bool UseSlopedExtrapolation = false;
/// <summary>The server defaults as an <see cref="AggregateConfiguration"/> (explicit values, not "use defaults").</summary>
public static AggregateConfiguration Defaults => new()
{
UseServerCapabilitiesDefaults = false,
TreatUncertainAsBad = TreatUncertainAsBad,
PercentDataBad = PercentDataBad,
PercentDataGood = PercentDataGood,
UseSlopedExtrapolation = UseSlopedExtrapolation,
};
/// <summary>
/// True when a processed read with <paramref name="requested"/> can be served: none supplied,
/// <c>UseServerCapabilitiesDefaults</c>, or every setting equal to the server's.
/// </summary>
/// <param name="requested">The request's configuration (may be null).</param>
/// <returns>Whether to serve; false ⇒ <c>BadAggregateConfigurationRejected</c>.</returns>
public static bool IsServable(AggregateConfiguration? requested) =>
requested is null
|| requested.UseServerCapabilitiesDefaults
|| (requested.TreatUncertainAsBad == TreatUncertainAsBad
&& requested.PercentDataBad == PercentDataBad
&& requested.PercentDataGood == PercentDataGood
&& requested.UseSlopedExtrapolation == UseSlopedExtrapolation);
}
@@ -1,6 +1,7 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
@@ -171,7 +172,8 @@ public abstract class ProcessedConformance : ConformanceBase
// The time-weighted path over the whole range: ≈ the extrapolated 50 that dominates eight millennia.
var tw = (await src.ReadProcessedAsync("A", start, end, TimeSpan.Zero, HistoryAggregateType.TimeAverage2, Ct)).Samples.Single();
((double)tw.Value!).ShouldBe(50d, 1e-6);
tw.StatusCode.ShouldBe(UncertainCalc);
// The data starts (+0) strictly inside the one interval ⇒ Partial as well (#677, the SDK's m_startOfData rule).
tw.StatusCode.ShouldBe(UncertainCalc | 0x4u);
src.GetHealthSnapshot().TotalFailures.ShouldBe(0);
}
@@ -196,12 +198,165 @@ public abstract class ProcessedConformance : ConformanceBase
}
[Fact]
public async Task A_reverse_window_is_computed_forward_and_returned_descending()
public async Task A_reverse_window_is_anchored_at_its_later_time()
{
// #677 (Part 13 / SDK): At(80) → At(0) is the window (0,80] in intervals (60,80], (40,60], (20,40], (0,20], each
// stamped with its LATER edge, latest first. A sample on an edge belongs to the interval it ends: +40 is in
// (20,40], +20 (Bad) in (0,20], and +0 is outside the window. (The pre-#677 layout answered [60,80) … [0,20)
// stamped 60/40/20/0 with values null, 50, 40, 15.)
await ConformanceData.SeedAAsync(B);
var s = (await B.CreateDataSource().ReadProcessedAsync("A", At(80), At(0), Twenty, HistoryAggregateType.Average, Ct)).Samples;
s.Select(x => x.SourceTimestampUtc).ShouldBe([B.Round(At(60)), B.Round(At(40)), B.Round(At(20)), B.Round(At(0))]);
s.Select(x => x.Value).ShouldBe([null, 50d, 40d, 15d]);
s.Select(x => x.SourceTimestampUtc).ShouldBe([B.Round(At(80)), B.Round(At(60)), B.Round(At(40)), B.Round(At(20))]);
s.Select(x => x.Value).ShouldBe([null, null, 45d, 20d]);
s.Select(x => x.StatusCode).ShouldBe([NoDataCalc, NoDataCalc, GoodCalc, UncertainCalc]);
}
[Fact]
public async Task A_reverse_window_truncates_its_earliest_interval()
{
// At(50) → At(0), 20 s over tag A (+0=10, +10=20, +20=Bad, +30=40, +40=50): (30,50] = {+40} → 50 (+30 is its
// earlier edge, excluded); (10,30] = {+20 Bad, +30} → 40, Uncertain; (0,10] = {+10} → 20 — the truncated interval
// is the EARLIEST one. (Pre-#677: [0,20), [20,40), [40,50) stamped 40/20/0 with values 50, 40, 15.)
await ConformanceData.SeedAAsync(B);
var s = (await B.CreateDataSource().ReadProcessedAsync("A", At(50), At(0), Twenty, HistoryAggregateType.Average, Ct)).Samples;
s.Select(x => x.SourceTimestampUtc).ShouldBe([B.Round(At(50)), B.Round(At(30)), B.Round(At(10))]);
s.Select(x => x.Value).ShouldBe([50d, 40d, 20d]);
s.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalc]);
}
[Fact]
public async Task Reverse_bucket_edges_are_later_edge_inclusive_to_the_tick()
{
// The mirror of the forward edge test: At(40) → At(0), 20 s ⇒ (20,40], (0,20]. The bucket math is pushed down,
// so this pins the SQL, not just the C# plan.
var r = B.ResolutionTicks;
await B.SeedAsync("ER",
(At(0), HistorianValue.FromDouble(1000), Good), // == earlier edge: excluded
(At(0).AddTicks(r), HistorianValue.FromDouble(1), Good), // first tick of (0,20]
(At(20), HistorianValue.FromDouble(2), Good), // == edge: the LATER edge of (0,20]
(At(20).AddTicks(r), HistorianValue.FromDouble(4), Good), // first tick of (20,40]
(At(40), HistorianValue.FromDouble(8), Good), // == request start: included, in (20,40]
(At(40).AddTicks(r), HistorianValue.FromDouble(2000), Good)); // after the window: excluded
var s = (await B.CreateDataSource().ReadProcessedAsync("ER", At(40), At(0), Twenty, HistoryAggregateType.Average, Ct)).Samples;
s.Select(x => x.SourceTimestampUtc).ShouldBe([B.Round(At(40)), B.Round(At(20))]);
s.Select(x => x.Value).ShouldBe([6d, 1.5d]);
}
[Fact]
public async Task The_interval_the_data_starts_inside_carries_the_Partial_bit()
{
// #677 (SDK m_startOfData): the tag's first sample is +5, inside [0,20) ⇒ Partial there for every aggregate but
// Average; [20,40) is fully covered. With an earlier sample stored (even outside the window) nothing is Partial.
await B.SeedAsync("LS",
(At(5), HistorianValue.FromDouble(10), Good),
(At(15), HistorianValue.FromDouble(30), Good),
(At(25), HistorianValue.FromDouble(20), Good),
(At(60), HistorianValue.FromDouble(20), Good));
var src = B.CreateDataSource();
foreach (var agg in new[] { HistoryAggregateType.Count, HistoryAggregateType.Minimum, HistoryAggregateType.TimeAverage2 })
{
var s = (await src.ReadProcessedAsync("LS", At(0), At(40), Twenty, agg, Ct)).Samples;
(s[0].StatusCode & 0x4u).ShouldBe(0x4u, agg.ToString());
(s[1].StatusCode & 0x4u).ShouldBe(0u, agg.ToString());
}
(await src.ReadProcessedAsync("LS", At(0), At(40), Twenty, HistoryAggregateType.Average, Ct)).Samples
.ShouldAllBe(x => (x.StatusCode & 0x4u) == 0);
await B.SeedAsync("LS2",
(At(-100), HistorianValue.FromDouble(10), Good),
(At(5), HistorianValue.FromDouble(10), Good),
(At(15), HistorianValue.FromDouble(30), Good),
(At(25), HistorianValue.FromDouble(20), Good));
(await src.ReadProcessedAsync("LS2", At(0), At(40), Twenty, HistoryAggregateType.Count, Ct)).Samples
.ShouldAllBe(x => (x.StatusCode & 0x4u) == 0);
}
[Fact]
public async Task A_reverse_read_sets_no_start_of_data_Partial_and_matches_the_SDK()
{
// Review F1: the SDK applies the start-of-data rule to FORWARD reads only. Reverse 40 → 0 at 10 s over data that
// starts at +7: (0,10] holds the data's start but is not Partial — checked through the real SQL and the SDK oracle.
var rows = Enumerable.Range(0, 20)
.Select(i => (At(7 + i * 5), HistorianValue.FromDouble(i % 2 == 0 ? 20 : 40), Good)).ToArray();
await B.SeedAsync("RLS", rows);
var sdkRows = rows.Select(r => (B.Round(r.Item1), r.Item2, r.Item3)).ToArray();
var src = B.CreateDataSource();
var ten = TimeSpan.FromSeconds(10);
foreach (var agg in new[]
{
HistoryAggregateType.Count, HistoryAggregateType.Minimum, HistoryAggregateType.Range,
HistoryAggregateType.Delta, HistoryAggregateType.TimeAverage, HistoryAggregateType.TimeAverage2,
HistoryAggregateType.Total2,
})
{
var ours = (await src.ReadProcessedAsync("RLS", At(40), At(0), ten, agg, Ct)).Samples;
ours.ShouldAllBe(x => (x.StatusCode & 0x4u) == 0, agg.ToString());
var tw = AggregateFamilies.FamilyOf(agg) == AggregateFamily.TimeWeighted;
SdkAggregateOracle.ShouldMatch(ours, SdkAggregateOracle.Run(agg, B.Round(At(40)), B.Round(At(0)), ten, sdkRows),
$"{agg} reverse late start", compareCalculatedInterpolatedBits: tw,
// pre-existing, #680: the SDK's Delta is Good with a Bad sample present; ours Uncertain (none here).
compareCodeBits: agg != HistoryAggregateType.Delta);
}
}
/// <summary>Dense: every 2.5 s from −10 to +100, values cycling, one Bad sample at +22.5 (no empty interval).</summary>
private Task SeedDenseAsync() => B.SeedAsync("P", DenseRows());
private static (DateTime T, HistorianValue V, uint Status)[] DenseRows() => Enumerable.Range(0, 45)
.Select(i => (At(-10 + i * 2.5), HistorianValue.FromDouble(10 + (i * 7 % 13)), i == 13 ? Bad : Good))
.ToArray();
[Theory]
[InlineData(0, 80, 20)]
[InlineData(80, 0, 20)]
[InlineData(77, 3, 20)]
[InlineData(3, 77, 20)]
public async Task Processed_reads_match_the_SDK_aggregate_calculators(double from, double to, double intervalS)
{
// #677: the real backend (pushed-down SQL bucket math + the shared evaluators) against the pinned SDK's own Part 13
// calculators over the same stored rows — timestamps, values, status code bits and the Partial bit, forward and
// reverse. See SdkAggregateParityTests (Sql.Common.Tests) for the documented divergences this avoids/masks.
await SeedDenseAsync();
var rows = DenseRows().Select(r => (B.Round(r.T), r.V, r.Status)).ToArray();
var src = B.CreateDataSource();
var interval = TimeSpan.FromSeconds(intervalS);
// The SDK sees the window the backend actually serves (QuestDB rounds the bounds to µs).
var (start, end) = (B.Round(At(from)), B.Round(At(to)));
foreach (var agg in new[]
{
HistoryAggregateType.Average, HistoryAggregateType.Minimum, HistoryAggregateType.Maximum,
HistoryAggregateType.Count, HistoryAggregateType.TimeAverage, HistoryAggregateType.TimeAverage2,
HistoryAggregateType.Total, HistoryAggregateType.Total2, HistoryAggregateType.Interpolative,
HistoryAggregateType.Range, HistoryAggregateType.Start, HistoryAggregateType.End,
HistoryAggregateType.Delta, HistoryAggregateType.MinimumActualTime, HistoryAggregateType.MaximumActualTime,
HistoryAggregateType.PercentGood, HistoryAggregateType.PercentBad,
})
{
var ours = (await src.ReadProcessedAsync("P", At(from), At(to), interval, agg, Ct)).Samples.ToList();
var startOrEnd = agg is HistoryAggregateType.Start or HistoryAggregateType.End;
var tw = AggregateFamilies.FamilyOf(agg) == AggregateFamily.TimeWeighted;
if (tw && agg != HistoryAggregateType.Interpolative)
{
// Documented divergence: ours flags the truncated interval Partial, the SDK does not.
var plan = BucketPlan.Create(start, end, interval);
for (long i = 0; i < plan.BucketCount; i++)
if (plan.IsTruncated(i))
{
var k = (int)(plan.Reversed ? plan.BucketCount - 1 - i : i);
ours[k] = ours[k] with { StatusCode = ours[k].StatusCode & ~0x4u };
}
}
var reverseInterpolative = agg == HistoryAggregateType.Interpolative && from > to;
SdkAggregateOracle.ShouldMatch(ours, SdkAggregateOracle.Run(agg, start, end, interval, rows),
$"{agg} {from}→{to}/{intervalS}s",
compareCalculatedInterpolatedBits: tw && !reverseInterpolative,
// pre-existing, #680: TimeAverage/Total status rule; Start/End raw-sample timestamp + status; Delta status;
// PercentGood/Bad per-sample vs per-duration values (see SdkAggregateParityTests).
compareCodeBits: agg is not (HistoryAggregateType.TimeAverage or HistoryAggregateType.Total
or HistoryAggregateType.Delta) && !reverseInterpolative && !startOrEnd,
compareTimestamps: !startOrEnd,
compareValues: agg is not (HistoryAggregateType.PercentGood or HistoryAggregateType.PercentBad));
}
}
[Fact]
@@ -245,7 +400,8 @@ public abstract class ProcessedConformance : ConformanceBase
(await One(HistoryAggregateType.Count)).StatusCode.ShouldBe(NoDataCalc);
var pb = await One(HistoryAggregateType.PercentBad);
pb.Value.ShouldBe(100d);
pb.StatusCode.ShouldBe(GoodCalc);
// The tag's data starts (+1) inside the interval ⇒ Partial on the non-Bad percentage (#677).
pb.StatusCode.ShouldBe(GoodCalc | 0x4u);
}
[Fact]
@@ -0,0 +1,162 @@
using Opc.Ua;
using Opc.Ua.Server;
using Shouldly;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
/// <summary>
/// Drives the pinned OPC UA SDK's own Part 13 aggregate calculators (<c>Opc.Ua.Server</c> — the classes the SDK's
/// <c>AggregateManager</c> registers) in-process over a sample set, so our processed reads can be compared with the
/// reference implementation bucket by bucket (#677). The calculators run under the server's default
/// <see cref="AggregateConfiguration"/> (TreatUncertainAsBad = false, PercentDataBad = PercentDataGood = 100,
/// UseSlopedExtrapolation = false), sloped interpolation, source timestamps.
/// </summary>
/// <remarks>
/// Shared (source-linked) by <c>Sql.Common.Tests</c> and both SQL backends' integration suites.
/// </remarks>
public static class SdkAggregateOracle
{
/// <summary>One processed value: value (double / int / null), full status code, timestamp.</summary>
public readonly record struct Result(object? Value, uint StatusCode, DateTime Timestamp);
/// <summary>The server-default aggregate configuration (what <c>AggregateManager.GetDefaultConfiguration</c> returns).</summary>
public static AggregateConfiguration DefaultConfiguration => new()
{
TreatUncertainAsBad = false,
PercentDataBad = 100,
PercentDataGood = 100,
UseSlopedExtrapolation = false,
UseServerCapabilitiesDefaults = false,
};
/// <summary>Runs the SDK calculator for <paramref name="aggregate"/> over <paramref name="samples"/>.</summary>
/// <param name="aggregate">The aggregate.</param>
/// <param name="start">Request start (later than <paramref name="end"/> for a reverse read).</param>
/// <param name="end">Request end.</param>
/// <param name="interval">Processing interval.</param>
/// <param name="samples">Every stored sample of the tag, any order (sorted here).</param>
/// <returns>The SDK's processed values, in the order it returns them.</returns>
public static IReadOnlyList<Result> Run(HistoryAggregateType aggregate, DateTime start, DateTime end, TimeSpan interval,
IEnumerable<(DateTime Time, HistorianValue Value, uint Status)> samples)
{
var id = AggregateId(aggregate);
var ms = interval.TotalMilliseconds;
var cfg = DefaultConfiguration;
IAggregateCalculator calc = aggregate switch
{
HistoryAggregateType.Average or HistoryAggregateType.TimeAverage or HistoryAggregateType.TimeAverage2
or HistoryAggregateType.Total or HistoryAggregateType.Total2 =>
new AverageAggregateCalculator(id, start, end, ms, false, cfg, null),
HistoryAggregateType.Minimum or HistoryAggregateType.Maximum or HistoryAggregateType.Range
or HistoryAggregateType.MinimumActualTime or HistoryAggregateType.MaximumActualTime =>
new MinMaxAggregateCalculator(id, start, end, ms, false, cfg, null),
HistoryAggregateType.Count => new CountAggregateCalculator(id, start, end, ms, false, cfg, null),
HistoryAggregateType.PercentGood or HistoryAggregateType.PercentBad =>
new StatusAggregateCalculator(id, start, end, ms, false, cfg, null),
HistoryAggregateType.Start or HistoryAggregateType.End or HistoryAggregateType.Delta =>
new StartEndAggregateCalculator(id, start, end, ms, false, cfg, null),
HistoryAggregateType.Interpolative => new AggregateCalculator(id, start, end, ms, false, cfg, null),
_ => throw new ArgumentOutOfRangeException(nameof(aggregate), aggregate, "no SDK calculator mapped"),
};
// The SDK expects values in the direction of the read: ascending forward, descending reverse.
var ordered = samples.OrderBy(s => s.Time).ToList();
if (end < start) ordered.Reverse();
foreach (var s in ordered) calc.QueueRawValue(ToDataValue(s.Time, s.Value, s.Status));
var results = new List<Result>();
for (var guard = 0; guard < 100_000; guard++)
{
var dv = calc.GetProcessedValue(true);
if (dv is null) break;
results.Add(new Result(dv.Value, dv.StatusCode.Code, DateTime.SpecifyKind(dv.SourceTimestamp, DateTimeKind.Utc)));
}
return results;
}
/// <summary>
/// Asserts our snapshots equal the SDK's: same count, same timestamps, same status code bits (severity + sub-code)
/// and Partial bit (and optionally the Calculated / Interpolated bits), same value (doubles to 1e-9; ints exactly;
/// null on a Bad status). A failure lists every differing interval side by side.
/// </summary>
/// <param name="ours">Our processed values.</param>
/// <param name="sdk">The SDK's.</param>
/// <param name="label">Context for failure messages.</param>
/// <param name="compareCalculatedInterpolatedBits">Also compare the Calculated / Interpolated aggregate bits.</param>
/// <param name="compareCodeBits">Compare the severity + sub-code (false only for a documented status divergence).</param>
/// <param name="compareValues">Compare the values (false only for a documented value divergence).</param>
/// <param name="compareTimestamps">Compare the timestamps (false only for a documented timestamp divergence).</param>
/// <param name="comparePartialBit">Compare the Partial bit (false only for a documented divergence).</param>
public static void ShouldMatch(IReadOnlyList<DataValueSnapshot> ours, IReadOnlyList<Result> sdk, string label,
bool compareCalculatedInterpolatedBits = true, bool compareCodeBits = true, bool compareValues = true,
bool compareTimestamps = true, bool comparePartialBit = true)
{
var problems = new List<string>();
if (ours.Count != sdk.Count) problems.Add($"interval count: ours {ours.Count}, sdk {sdk.Count}");
for (var i = 0; i < Math.Min(ours.Count, sdk.Count); i++)
{
var o = ours[i];
var s = sdk[i];
var bitMask = (comparePartialBit ? 0x4u : 0u) | (compareCodeBits ? 0xFFFF0000u : 0u)
| (compareCalculatedInterpolatedBits ? 0x3u : 0u);
var same = (!compareTimestamps || o.SourceTimestampUtc == s.Timestamp)
&& (o.StatusCode & bitMask) == (s.StatusCode & bitMask);
if (!compareValues || (!compareCodeBits && StatusCode.IsBad(s.StatusCode)))
{
// Values not compared (or the SDK answered Bad under a status this comparison does not check).
}
else if (same && !StatusCode.IsBad(s.StatusCode))
{
same = s.Value switch
{
null => o.Value is null,
int n => o.Value is not null && Convert.ToInt32(o.Value) == n,
_ => o.Value is not null && Math.Abs(Convert.ToDouble(o.Value) - Convert.ToDouble(s.Value)) <= 1e-9,
};
}
else if (same && compareCodeBits) same = o.Value is null;
if (!same)
problems.Add($"[{i}] ours {o.SourceTimestampUtc:HH:mm:ss.fffffff} {o.Value ?? "null"} 0x{o.StatusCode:X8}" +
$" | sdk {s.Timestamp:HH:mm:ss.fffffff} {s.Value ?? "null"} 0x{s.StatusCode:X8}");
}
problems.ShouldBeEmpty($"{label} differs from the SDK:\n" + string.Join("\n", problems));
}
private static DataValue ToDataValue(DateTime t, HistorianValue v, uint status) => new()
{
WrappedValue = v.Kind switch
{
HistorianValueKind.Empty => Variant.Null,
HistorianValueKind.Int64 => new Variant(v.Integer),
HistorianValueKind.Boolean => new Variant(v.Number != 0),
HistorianValueKind.String => new Variant(v.Text),
_ => new Variant(v.Number),
},
StatusCode = status,
SourceTimestamp = t,
ServerTimestamp = t,
};
private static NodeId AggregateId(HistoryAggregateType a) => a switch
{
HistoryAggregateType.Average => ObjectIds.AggregateFunction_Average,
HistoryAggregateType.Minimum => ObjectIds.AggregateFunction_Minimum,
HistoryAggregateType.Maximum => ObjectIds.AggregateFunction_Maximum,
HistoryAggregateType.Range => ObjectIds.AggregateFunction_Range,
HistoryAggregateType.MinimumActualTime => ObjectIds.AggregateFunction_MinimumActualTime,
HistoryAggregateType.MaximumActualTime => ObjectIds.AggregateFunction_MaximumActualTime,
HistoryAggregateType.Count => ObjectIds.AggregateFunction_Count,
HistoryAggregateType.PercentGood => ObjectIds.AggregateFunction_PercentGood,
HistoryAggregateType.PercentBad => ObjectIds.AggregateFunction_PercentBad,
HistoryAggregateType.Start => ObjectIds.AggregateFunction_Start,
HistoryAggregateType.End => ObjectIds.AggregateFunction_End,
HistoryAggregateType.Delta => ObjectIds.AggregateFunction_Delta,
HistoryAggregateType.TimeAverage => ObjectIds.AggregateFunction_TimeAverage,
HistoryAggregateType.TimeAverage2 => ObjectIds.AggregateFunction_TimeAverage2,
HistoryAggregateType.Total => ObjectIds.AggregateFunction_Total,
HistoryAggregateType.Total2 => ObjectIds.AggregateFunction_Total2,
HistoryAggregateType.Interpolative => ObjectIds.AggregateFunction_Interpolative,
_ => throw new ArgumentOutOfRangeException(nameof(a), a, "no SDK aggregate id mapped"),
};
}
@@ -78,6 +78,27 @@ public sealed class GatewayTagProvisionerTests
Assert.True(defs[0].StorageRateMs > 0, "provisioned tags must carry a positive storage rate (0 fails EnsureTags).");
}
[Fact]
public async Task Definition_carries_an_integral_divisor_of_one()
{
// #677: AVEVA's Total integral is value·seconds ÷ IntegralDivisor. The proto3 double default is 0, which the
// gateway passes through, so every provisioned tag must say 1 explicitly — the unit the SQL backends' Total uses.
var fake = new FakeHistorianGatewayClient { EnsureTagsResult = new TagOperationResults() };
var p = Provisioner(fake);
await p.EnsureTagsAsync(
new[]
{
new HistorianTagProvisionRequest("Pump1.Temp", DriverDataType.Float32, null, null),
new HistorianTagProvisionRequest("Pump1.Run", DriverDataType.Boolean, null, null),
},
TestContext.Current.CancellationToken);
var defs = fake.LastEnsureDefinitions!;
Assert.Equal(2, defs.Count);
Assert.All(defs, d => Assert.Equal(1.0, d.IntegralDivisor));
}
[Fact]
public async Task Maps_metadata_and_coalesces_null_metadata_to_empty()
{
@@ -82,7 +82,8 @@ public sealed class ReadTests(QuestDbFixture fx) : IAsyncLifetime
foreach (var agg in new[] { HistoryAggregateType.Average, HistoryAggregateType.Start, HistoryAggregateType.MaximumActualTime })
{
var s = (await src.ReadProcessedAsync("A", start, end, TimeSpan.Zero, agg, Ct)).Samples.Single();
s.StatusCode.ShouldBe(UncertainCalc, agg.ToString());
// The data starts (+0) strictly inside the one interval ⇒ Partial on every aggregate but Average (#677).
s.StatusCode.ShouldBe(agg == HistoryAggregateType.Average ? UncertainCalc : UncertainCalc | 0x4u, agg.ToString());
}
(await src.ReadProcessedAsync("A", start, end, TimeSpan.Zero, HistoryAggregateType.Average, Ct)).Samples.Single().Value.ShouldBe(30d);
(await src.ReadProcessedAsync("A", start, end, TimeSpan.Zero, HistoryAggregateType.Start, Ct)).Samples.Single().Value.ShouldBe(10d);
@@ -14,6 +14,8 @@
<PackageReference Include="xunit.v3"/>
<PackageReference Include="Shouldly"/>
<PackageReference Include="Microsoft.NET.Test.Sdk"/>
<!-- SDK Part 13 aggregate calculators: the conformance suite's processed-read parity oracle (#677). -->
<PackageReference Include="OPCFoundation.NetStandard.Opc.Ua.Server"/>
<PackageReference Include="xunit.runner.visualstudio">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
@@ -37,9 +37,7 @@ public sealed class AggregateEvaluatorTests
private static IReadOnlyList<DataValueSnapshot> ReadA(HistoryAggregateType agg, bool maximum = false)
{
var plan = BucketPlan.Create(At(0), At(80), TimeSpan.FromSeconds(20));
var evaluated = RowsA(maximum).ToDictionary(kv => kv.Key,
kv => AggregateEvaluator.Evaluate(agg, kv.Value, plan.BucketStart(kv.Key), Now));
return plan.Assemble(evaluated, t => AggregateEvaluator.NoData(t, Now));
return AggregateEvaluator.EvaluateAll(agg, plan, RowsA(maximum).Values, At(0), Now);
}
[Theory]
@@ -28,8 +28,10 @@ public sealed class BucketPlanTests
{
var p = BucketPlan.Create(At(0), At(50), TimeSpan.FromSeconds(20));
p.BucketCount.ShouldBe(3);
p.BucketStart(2).ShouldBe(At(40));
p.BucketStart(2).Kind.ShouldBe(DateTimeKind.Utc);
p.BucketTimestamp(2).ShouldBe(At(40));
p.BucketTimestamp(2).Kind.ShouldBe(DateTimeKind.Utc);
p.IsTruncated(2).ShouldBeTrue();
p.IsTruncated(1).ShouldBeFalse();
}
[Fact]
@@ -43,14 +45,104 @@ public sealed class BucketPlanTests
}
[Fact]
public void A_reverse_window_is_planned_forward_and_assembled_descending()
public void A_reverse_window_is_anchored_at_its_later_time_and_stamped_with_later_edges()
{
// #677 (Part 13 / SDK CreateSlice + GetTimestamp): At(80) → At(0), 20 s ⇒ (60,80], (40,60], (20,40], (0,20],
// each stamped with its later edge, latest first.
var p = BucketPlan.Create(At(80), At(0), TimeSpan.FromSeconds(20));
p.Reversed.ShouldBeTrue();
p.StartUtc.ShouldBe(At(0));
p.EndUtc.ShouldBe(At(80));
var s = p.Assemble(new Dictionary<long, DataValueSnapshot>(), t => new DataValueSnapshot(null, 0, t, t));
s.Select(x => x.SourceTimestampUtc).ShouldBe([At(60), At(40), At(20), At(0)]);
s.Select(x => x.SourceTimestampUtc).ShouldBe([At(80), At(60), At(40), At(20)]);
}
[Fact]
public void A_reverse_window_puts_the_partial_interval_at_its_earliest_end()
{
// At(50) → At(0), 20 s: (30,50], (10,30], (0,10] — the truncated bucket is the EARLIEST (the old layout truncated
// the latest: [0,20), [20,40), [40,50)).
var p = BucketPlan.Create(At(50), At(0), TimeSpan.FromSeconds(20));
p.BucketCount.ShouldBe(3);
p.LowerTicks(0).ShouldBe(At(0).Ticks);
p.UpperTicks(0).ShouldBe(At(10).Ticks);
p.IsTruncated(0).ShouldBeTrue();
p.LowerTicks(2).ShouldBe(At(30).Ticks);
p.UpperTicks(2).ShouldBe(At(50).Ticks);
p.IsTruncated(2).ShouldBeFalse();
var s = p.Assemble(new Dictionary<long, DataValueSnapshot>(), t => new DataValueSnapshot(null, 0, t, t));
s.Select(x => x.SourceTimestampUtc).ShouldBe([At(50), At(30), At(10)]);
}
[Fact]
public void Forward_buckets_are_start_inclusive_and_reverse_buckets_later_edge_inclusive()
{
var fwd = BucketPlan.Create(At(0), At(50), TimeSpan.FromSeconds(20));
fwd.IndexOf(At(0)).ShouldBe(0);
fwd.IndexOf(At(20)).ShouldBe(1);
fwd.IndexOf(At(20).AddTicks(-1)).ShouldBe(0);
fwd.IndexOf(At(49.9999999)).ShouldBe(2);
fwd.IndexOf(At(50)).ShouldBeNull();
fwd.IndexOf(At(0).AddTicks(-1)).ShouldBeNull();
// (0,10] (10,30] (30,50]: the later time is in the window, the earlier one is not (as a reverse raw read).
var rev = BucketPlan.Create(At(50), At(0), TimeSpan.FromSeconds(20));
rev.IndexOf(At(50)).ShouldBe(2);
rev.IndexOf(At(30)).ShouldBe(1);
rev.IndexOf(At(30).AddTicks(1)).ShouldBe(2);
rev.IndexOf(At(10)).ShouldBe(0);
rev.IndexOf(At(10).AddTicks(1)).ShouldBe(1);
rev.IndexOf(At(0).AddTicks(1)).ShouldBe(0);
rev.IndexOf(At(0)).ShouldBeNull();
rev.IndexOf(At(50).AddTicks(1)).ShouldBeNull();
}
[Fact]
public void The_backend_bucket_formula_matches_IndexOf_in_both_directions()
{
// b = (t − start + adjust) / interval and b = (unixTicks(t) − base) / interval — what the SQL computes.
foreach (var p in new[]
{
BucketPlan.Create(At(0), At(50), TimeSpan.FromSeconds(20)),
BucketPlan.Create(At(50), At(0), TimeSpan.FromSeconds(20)),
BucketPlan.Create(At(60), At(0), TimeSpan.FromSeconds(20)),
BucketPlan.Create(At(0).AddTicks(70), At(0), TimeSpan.FromTicks(3)),
})
{
for (var t = p.StartUtc.Ticks - 2; t <= p.EndUtc.Ticks + 2; t += Math.Max(1, (p.EndUtc.Ticks - p.StartUtc.Ticks) / 997))
{
var at = new DateTime(t, DateTimeKind.Utc);
if (p.IndexOf(at) is not { } idx) continue;
((t - p.StartUtc.Ticks + p.BucketIndexAdjustTicks) / p.IntervalTicks).ShouldBe(idx);
(((at - DateTime.UnixEpoch).Ticks - p.BucketBaseUnixTicks) / p.IntervalTicks).ShouldBe(idx);
p.LowerTicks(idx).ShouldBeLessThanOrEqualTo(t);
p.UpperTicks(idx).ShouldBeGreaterThanOrEqualTo(t);
}
}
}
[Fact]
public void Data_starts_inside_only_strictly_between_the_bucket_edges()
{
var p = BucketPlan.Create(At(0), At(40), TimeSpan.FromSeconds(20));
p.DataStartsInside(0, At(5)).ShouldBeTrue();
p.DataStartsInside(1, At(5)).ShouldBeFalse();
p.DataStartsInside(1, At(20)).ShouldBeFalse(); // on the edge: data covers the whole interval
p.DataStartsInside(0, null).ShouldBeFalse();
p.DataStartsInside(0, At(-5)).ShouldBeFalse();
}
[Fact]
public void The_earliest_sample_is_needed_only_for_forward_reads_of_aggregates_other_than_Average()
{
// Review F2: backends skip the earliest-sample query when its answer cannot change a result.
var fwd = BucketPlan.Create(At(0), At(40), TimeSpan.FromSeconds(10));
var rev = BucketPlan.Create(At(40), At(0), TimeSpan.FromSeconds(10));
AggregateEvaluator.NeedsEarliestSample(HistoryAggregateType.Count, fwd).ShouldBeTrue();
AggregateEvaluator.NeedsEarliestSample(HistoryAggregateType.Average, fwd).ShouldBeFalse();
AggregateEvaluator.NeedsEarliestSample(HistoryAggregateType.Count, rev).ShouldBeFalse();
AggregateEvaluator.NeedsEarliestSample(HistoryAggregateType.Count, BucketPlan.Create(At(0), At(0), TimeSpan.Zero))
.ShouldBeFalse();
}
[Fact]
@@ -64,6 +156,11 @@ public sealed class BucketPlanTests
s[2].SourceTimestampUtc.ShouldBe(At(40));
}
[Fact]
public void BucketBaseUnixTicks_is_the_start_forward()
=> BucketPlan.Create(DateTime.UnixEpoch.AddTicks(1234567), DateTime.UnixEpoch.AddSeconds(10), TimeSpan.FromSeconds(1))
.BucketBaseUnixTicks.ShouldBe(1234567L);
[Fact]
public void StartUnixTicks_is_ticks_since_the_unix_epoch()
=> BucketPlan.Create(DateTime.UnixEpoch.AddTicks(1234567), DateTime.UnixEpoch.AddSeconds(10), TimeSpan.Zero)
@@ -0,0 +1,259 @@
using Shouldly;
using Xunit;
using ZB.MOM.WW.OtOpcUa.Core.Abstractions;
using ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance;
using static ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests.Fixture;
namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests;
/// <summary>
/// #677: our processed reads against the pinned SDK's own Part 13 aggregate calculators (<see cref="SdkAggregateOracle"/>),
/// forward AND reverse, bucket by bucket — timestamps, values and status. The per-sample aggregates are bucketed with
/// <see cref="BucketPlan.IndexOf"/> (the C# statement of the backends' SQL bucket math; the integration suites run the
/// same comparison through the real SQL) and evaluated by <see cref="AggregateEvaluator.EvaluateAll"/>; the
/// time-weighted ones by <see cref="TimeWeightedAggregator"/>.
/// </summary>
/// <remarks>
/// Deliberate, documented divergences the datasets avoid (they are not anchoring questions):
/// <list type="bullet">
/// <item>an <b>empty</b> interval inside the data range: the SDK's Count answers 0/Good, ours <c>BadNoData</c>;</item>
/// <item><b>Uncertain</b> samples in a per-sample aggregate: the SDK's Average/Count/Min/Max use Good samples only,
/// ours every usable (Good + Uncertain) sample (TreatUncertainAsBad = false applied consistently);</item>
/// <item>the per-sample <b>Calculated / MultipleValues</b> info bits (the SDK's MinMax sets Calculated only when the
/// extreme is not at the interval start, and flags ties) — those comparisons check code bits + Partial only;</item>
/// <item>a <b>truncated</b> interval: ours sets Partial on a non-Bad time-weighted result (the SDK does not), so the
/// non-multiple windows mask that one bucket's Partial bit;</item>
/// <item>the SDK's <b>end-of-data</b> Partial rule (forward, the interval holding the last sample when nothing follows
/// and nothing was extrapolated) is not implemented: every dataset carries data past the window end.</item>
/// </list>
/// Pre-existing forward divergences this parity test surfaced (#671 behaviour, unchanged here — forward results were
/// to stay as they are; recorded for a follow-up rather than fixed in place):
/// <list type="bullet">
/// <item><b>TimeAverage / Total status:</b> the SDK (interpolated bounds) reports <c>UncertainDataSubNormal</c> when
/// <i>any</i> region of the interval is non-Good (e.g. a segment bridged across a Bad sample); ours applies the
/// time-based rule (Uncertain time counts as Good) to these two as well, so a bridged-but-fully-covered interval
/// is Good. Values and timestamps agree; the status code bits are not compared for these two.</item>
/// <item><b>Total over an interval the data starts inside:</b> the SDK integrates only the defined part
/// (value·s over the covered seconds); ours is <c>TimeAverage × interval seconds</c>. The Partial bit and timestamp
/// agree; that value is not compared.</item>
/// <item><b>Interpolative, reverse, a sample exactly at the point:</b> the SDK's backward <c>UpdateSlice</c> files
/// a sample AT the point as the early bound (<c>&gt;= 0</c>), so it interpolates from it to the next Good sample —
/// same value, but the Interpolated bit, and Uncertain when a Bad sample sits between. Ours returns the raw
/// sample with its own status in both directions (as the SDK itself does forward). Reverse Interpolative
/// therefore compares timestamps and values (the anchoring question) but not status.</item>
/// </list>
/// </remarks>
public sealed class SdkAggregateParityTests
{
private static readonly DateTime Now = At(3600);
/// <summary>Dense: every 2.5 s from −10 to +100, values cycling, one Bad sample at +22.5 — no empty interval.</summary>
private static readonly StoredSample[] Dense = Enumerable.Range(0, 45)
.Select(i => (t: -10 + i * 2.5, i))
.Select(x => S(x.t, HistorianValue.FromDouble(10 + (x.i * 7 % 13)), x.t == 22.5 ? Bad : Good))
.ToArray();
/// <summary>Tag A with Good bounds on both sides, plus an Uncertain sample and an outage marker further on.</summary>
private static readonly StoredSample[] Bounded =
[
S(-10, HistorianValue.FromDouble(5), Good),
.. A,
S(55, HistorianValue.FromDouble(70), Uncertain),
S(62, HistorianValue.Empty, BadNoCommunication),
S(71, HistorianValue.FromDouble(30), Good),
S(100, HistorianValue.FromDouble(60), Good),
];
/// <summary>Data that starts inside the window (no sample before +7) and runs past its end.</summary>
private static readonly StoredSample[] LateStart = Enumerable.Range(0, 20)
.Select(i => S(7 + i * 5, HistorianValue.FromDouble(i % 2 == 0 ? 20 : 40), Good))
.ToArray();
public static TheoryData<double, double, double> Windows => new()
{
{ 0, 80, 20 }, { 80, 0, 20 }, // whole intervals (samples sit on the edges: the inclusivity question)
{ 0, 50, 20 }, { 50, 0, 20 }, // a truncated interval: latest forward, EARLIEST reverse
{ 5, 80, 15 }, { 80, 5, 15 }, // a misaligned window
{ 3, 77, 20 }, { 77, 3, 20 }, // misaligned and truncated
{ 0, 60, 0 }, { 60, 0, 0 }, // one interval
};
private static IReadOnlyList<DataValueSnapshot> Ours(HistoryAggregateType agg, double from, double to, double intervalS,
StoredSample[] data)
{
var plan = BucketPlan.Create(At(from), At(to), TimeSpan.FromSeconds(intervalS));
if (AggregateFamilies.FamilyOf(agg) == AggregateFamily.TimeWeighted)
return TimeWeightedAggregator.Evaluate(agg, plan, data, Now);
return AggregateEvaluator.EvaluateAll(agg, plan, InMemoryRows(plan, data, agg == HistoryAggregateType.MaximumActualTime),
data.Min(s => s.TimeUtc), Now);
}
private static IReadOnlyList<SdkAggregateOracle.Result> Sdk(HistoryAggregateType agg, double from, double to,
double intervalS, StoredSample[] data)
{
var interval = intervalS > 0 ? TimeSpan.FromSeconds(intervalS) : (At(to) - At(from)).Duration();
return SdkAggregateOracle.Run(agg, At(from), At(to), interval, data.Select(s => (s.TimeUtc, s.Value, s.Status)));
}
/// <summary>Masks the Partial bit of our one truncated interval (a documented divergence: ours flags it, the SDK does not).</summary>
private static IReadOnlyList<DataValueSnapshot> MaskTruncation(IReadOnlyList<DataValueSnapshot> ours, double from,
double to, double intervalS)
{
var plan = BucketPlan.Create(At(from), At(to), TimeSpan.FromSeconds(intervalS));
var list = ours.ToList();
for (long i = 0; i < plan.BucketCount; i++)
{
if (!plan.IsTruncated(i)) continue;
var k = (int)(plan.Reversed ? plan.BucketCount - 1 - i : i);
list[k] = list[k] with { StatusCode = list[k].StatusCode & ~HistorianInfoBits.HistorianPartial };
}
return list;
}
[Theory]
[MemberData(nameof(Windows))]
public void Time_weighted_aggregates_match_the_SDK(double from, double to, double intervalS)
{
foreach (var agg in new[]
{
HistoryAggregateType.TimeAverage, HistoryAggregateType.TimeAverage2, HistoryAggregateType.Total,
HistoryAggregateType.Total2, HistoryAggregateType.Interpolative,
})
foreach (var (name, data) in new[] { ("Bounded", Bounded), ("Dense", Dense) })
SdkAggregateOracle.ShouldMatch(MaskTruncation(Ours(agg, from, to, intervalS, data), from, to, intervalS),
Sdk(agg, from, to, intervalS, data), $"{agg} {name} {from}→{to}/{intervalS}s",
compareCalculatedInterpolatedBits: !(agg == HistoryAggregateType.Interpolative && from > to),
compareCodeBits: agg is not (HistoryAggregateType.TimeAverage or HistoryAggregateType.Total)
&& !(agg == HistoryAggregateType.Interpolative && from > to));
}
[Theory]
[MemberData(nameof(Windows))]
public void Per_sample_aggregates_match_the_SDK(double from, double to, double intervalS)
{
var failures = new List<string>();
foreach (var agg in PerSample)
{
try
{
SdkAggregateOracle.ShouldMatch(Ours(agg, from, to, intervalS, Dense), Sdk(agg, from, to, intervalS, Dense),
$"{agg} Dense {from}→{to}/{intervalS}s", compareCalculatedInterpolatedBits: false,
compareValues: !PercentAggregate(agg),
// pre-existing, #680: the SDK's Start/End return the raw first/last sample (any quality) with its own
// timestamp and stored status; ours return the first/last USABLE value stamped with the interval, under the
// per-sample status rule. The SDK's Delta is Good even when a Bad sample sits in the interval; ours is
// Uncertain. Values are compared wherever the SDK's answer is not Bad.
compareTimestamps: !StartOrEnd(agg),
compareCodeBits: !StartOrEnd(agg) && agg != HistoryAggregateType.Delta);
}
catch (Exception ex)
{
failures.Add(ex.Message);
}
}
failures.ShouldBeEmpty(string.Join("\n", failures));
}
private static readonly HistoryAggregateType[] PerSample =
[
HistoryAggregateType.Average, HistoryAggregateType.Minimum, HistoryAggregateType.Maximum,
HistoryAggregateType.Count, HistoryAggregateType.Range, HistoryAggregateType.Start, HistoryAggregateType.End,
HistoryAggregateType.Delta, HistoryAggregateType.MinimumActualTime, HistoryAggregateType.MaximumActualTime,
HistoryAggregateType.PercentGood, HistoryAggregateType.PercentBad,
];
/// <summary>
/// pre-existing, #680: ours is the per-SAMPLE percentage (100 × Good samples / all samples); the SDK's
/// <c>StatusAggregateCalculator</c> is the per-DURATION one (Good time / interval time). Different definitions, so the
/// values are not compared; timestamp, status and Partial still are.
/// </summary>
private static bool StartOrEnd(HistoryAggregateType agg) =>
agg is HistoryAggregateType.Start or HistoryAggregateType.End;
private static bool PercentAggregate(HistoryAggregateType agg) =>
agg is HistoryAggregateType.PercentGood or HistoryAggregateType.PercentBad;
[Theory]
[InlineData(HistoryAggregateType.TimeAverage)]
[InlineData(HistoryAggregateType.TimeAverage2)]
[InlineData(HistoryAggregateType.Total)]
[InlineData(HistoryAggregateType.Total2)]
[InlineData(HistoryAggregateType.Minimum)]
[InlineData(HistoryAggregateType.Maximum)]
[InlineData(HistoryAggregateType.Count)]
[InlineData(HistoryAggregateType.Range)]
[InlineData(HistoryAggregateType.Delta)]
[InlineData(HistoryAggregateType.Start)]
[InlineData(HistoryAggregateType.Average)]
public void A_reverse_read_over_data_starting_inside_the_window_sets_no_start_of_data_Partial(HistoryAggregateType agg)
{
// Review F1: the SDK applies its start-of-data rule only on a FORWARD read (a reverse read keys m_startOfData on the
// first value it is fed — the latest — which is past the window here). Reverse 40 → 0 at 10 s, first sample +7:
// (0,10] holds the start of the data but is NOT Partial.
var tw = AggregateFamilies.FamilyOf(agg) == AggregateFamily.TimeWeighted;
var ours = Ours(agg, 40, 0, 10, LateStart);
SdkAggregateOracle.ShouldMatch(ours, Sdk(agg, 40, 0, 10, LateStart), $"{agg} reverse LateStart",
compareCalculatedInterpolatedBits: tw,
// pre-existing, #680: Total over the interval the data starts inside (TimeAverage × interval vs the SDK's
// integral of the covered part); Start's raw-sample timestamp/status (see Per_sample_aggregates_match_the_SDK).
compareValues: agg != HistoryAggregateType.Total,
compareTimestamps: !StartOrEnd(agg),
compareCodeBits: !StartOrEnd(agg) && agg != HistoryAggregateType.Delta);
ours.ShouldAllBe(x => (x.StatusCode & HistorianInfoBits.HistorianPartial) == 0);
}
[Theory]
[InlineData(HistoryAggregateType.TimeAverage)]
[InlineData(HistoryAggregateType.TimeAverage2)]
[InlineData(HistoryAggregateType.Total)]
[InlineData(HistoryAggregateType.Total2)]
[InlineData(HistoryAggregateType.Minimum)]
[InlineData(HistoryAggregateType.Maximum)]
[InlineData(HistoryAggregateType.Count)]
[InlineData(HistoryAggregateType.Average)]
public void The_interval_the_data_starts_inside_matches_the_SDK_Partial_bit(HistoryAggregateType agg)
{
// Forward [0,40) at 10 s: the data starts at +7, inside [0,10) — the SDK sets Partial there (m_startOfData) for
// every calculator but plain Average; [10,20) … are fully covered.
var tw = AggregateFamilies.FamilyOf(agg) == AggregateFamily.TimeWeighted;
var ours = Ours(agg, 0, 40, 10, LateStart);
SdkAggregateOracle.ShouldMatch(ours, Sdk(agg, 0, 40, 10, LateStart), $"{agg} LateStart",
compareCalculatedInterpolatedBits: tw, compareValues: agg != HistoryAggregateType.Total);
(ours[0].StatusCode & HistorianInfoBits.HistorianPartial).ShouldBe(
agg == HistoryAggregateType.Average ? 0u : HistorianInfoBits.HistorianPartial);
ours.Skip(1).ShouldAllBe(x => (x.StatusCode & HistorianInfoBits.HistorianPartial) == 0);
}
/// <summary>The per-bucket statistics the backends' SQL computes, built in memory with <see cref="BucketPlan.IndexOf"/>.</summary>
internal static IEnumerable<BucketRow> InMemoryRows(BucketPlan plan, IEnumerable<StoredSample> samples, bool maximum = false)
{
foreach (var g in samples.Select(s => (s, b: plan.IndexOf(s.TimeUtc))).Where(x => x.b is not null)
.GroupBy(x => x.b!.Value).OrderBy(g => g.Key))
{
var all = g.Select(x => x.s).OrderBy(s => s.TimeUtc).ToList();
var usable = all.Where(s => s.IsUsable).ToList();
var numeric = usable.Where(s => s.Value.Kind is HistorianValueKind.Double or HistorianValueKind.Int64
or HistorianValueKind.Boolean).ToList();
// ActualTime: the extreme usable value, ties ⇒ earliest (the SQL's ROW_NUMBER order).
var extreme = numeric.OrderBy(s => maximum ? -s.Value.Number : s.Value.Number).ThenBy(s => s.TimeUtc)
.Cast<StoredSample?>().FirstOrDefault();
yield return new BucketRow
{
Bucket = g.Key,
Total = all.Count,
Usable = usable.Count,
Good = all.Count(s => s.IsGood),
Bad = all.Count - usable.Count,
NonNumeric = usable.Count - numeric.Count,
Avg = numeric.Count == 0 ? null : numeric.Average(s => s.Value.Number),
Min = numeric.Count == 0 ? null : numeric.Min(s => s.Value.Number),
Max = numeric.Count == 0 ? null : numeric.Max(s => s.Value.Number),
Sum = numeric.Count == 0 ? null : numeric.Sum(s => s.Value.Number),
First = usable.Count == 0 ? HistorianValue.Empty : usable[0].Value,
Last = usable.Count == 0 ? HistorianValue.Empty : usable[^1].Value,
Extreme = extreme?.Value.Number,
ExtremeTime = extreme?.TimeUtc,
};
}
}
}
@@ -322,14 +322,46 @@ public sealed class TimeWeightedAggregatorTests
// ---------------------------------------------------------------- plan shape
[Fact]
public void A_reverse_window_is_computed_forward_and_returned_descending()
public void A_reverse_window_is_returned_latest_first_stamped_with_later_edges()
{
// #677: the four whole intervals are the same spans as forward, but each is stamped with its later edge.
var s = TimeWeightedAggregator.Evaluate(HistoryAggregateType.TimeAverage2,
BucketPlan.Create(At(80), At(0), TimeSpan.FromSeconds(20)), A, Now);
s.Select(x => x.SourceTimestampUtc).ShouldBe([At(60), At(40), At(20), At(0)]);
s.Select(x => x.SourceTimestampUtc).ShouldBe([At(80), At(60), At(40), At(20)]);
Values(s, 50, 50, 45, 17.5);
}
[Fact]
public void A_reverse_window_anchors_at_its_later_time_and_truncates_the_earliest_interval()
{
// Tag A, At(50) → At(0), 20 s ⇒ (30,50], (10,30], (0,10]. Raw signal: [0,10) 10→20 Good (150); [10,20) 20 held
// into the outage, Uncertain (200); [20,30) Bad; [30,40) 40→50 Good (450); [40,50) 50 extrapolated (500).
// (30,50]: 950/20 = 47.5 Good; (10,30]: 200/10 = 20, the Bad 10 s ⇒ Uncertain; (0,10]: 150/10 = 15, truncated ⇒
// Partial. The old layout would have answered [0,20), [20,40), [40,50) stamped 0/20/40, partial at the LATEST end.
var s = TimeWeightedAggregator.Evaluate(HistoryAggregateType.TimeAverage2,
BucketPlan.Create(At(50), At(0), TimeSpan.FromSeconds(20)), A, Now);
s.Select(x => x.SourceTimestampUtc).ShouldBe([At(50), At(30), At(10)]);
Values(s, 47.5, 20, 15);
s.Select(x => x.StatusCode).ShouldBe([GoodCalc, UncertainCalc, GoodCalcPartial]);
}
[Fact]
public void The_interval_the_data_starts_inside_is_Partial()
{
// #677 (SDK m_startOfData): no sample before +5 at all ⇒ [0,10) holds the start of the data ⇒ Partial (its first
// 5 s are no-data time ⇒ Uncertain); [10,20) is fully covered ⇒ no Partial.
StoredSample[] late = [S(5, HistorianValue.FromDouble(10), Good), S(30, HistorianValue.FromDouble(10), Good)];
var s = Run(HistoryAggregateType.TimeAverage, 0, 20, 10, late);
Values(s, 10, 10);
s.Select(x => x.StatusCode).ShouldBe([UncertainCalc | 0x4u, GoodCalc]);
// A bound before the window (the data started earlier) ⇒ no Partial.
StoredSample[] early = [S(-5, HistorianValue.FromDouble(10), Good), S(30, HistorianValue.FromDouble(10), Good)];
Run(HistoryAggregateType.TimeAverage, 0, 20, 10, early).ShouldAllBe(x => x.StatusCode == GoodCalc);
// Data starting exactly on an interval edge covers that interval ⇒ no Partial.
StoredSample[] edge = [S(10, HistorianValue.FromDouble(10), Good), S(30, HistorianValue.FromDouble(10), Good)];
Run(HistoryAggregateType.TimeAverage, 0, 20, 10, edge)[1].StatusCode.ShouldBe(GoodCalc);
}
[Fact]
public void A_truncated_last_bucket_uses_its_own_duration_for_Total_and_is_Partial()
{
@@ -15,12 +15,18 @@
<PackageReference Include="Microsoft.NET.Test.Sdk"/>
<!-- SDK oracle for the status-code / aggregate-bit tests. -->
<PackageReference Include="OPCFoundation.NetStandard.Opc.Ua.Core"/>
<!-- SDK Part 13 aggregate calculators: the parity oracle for processed reads (#677). -->
<PackageReference Include="OPCFoundation.NetStandard.Opc.Ua.Server"/>
<PackageReference Include="xunit.runner.visualstudio">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
</PackageReference>
</ItemGroup>
<ItemGroup>
<Compile Include="..\Historian.Conformance\SdkAggregateOracle.cs" Link="SdkAggregateOracle.cs"/>
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\..\src\Drivers\ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common\ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.csproj"/>
</ItemGroup>
@@ -13,6 +13,8 @@
<PackageReference Include="xunit.v3"/>
<PackageReference Include="Shouldly"/>
<PackageReference Include="Microsoft.NET.Test.Sdk"/>
<!-- SDK Part 13 aggregate calculators: the conformance suite's processed-read parity oracle (#677). -->
<PackageReference Include="OPCFoundation.NetStandard.Opc.Ua.Server"/>
<PackageReference Include="xunit.runner.visualstudio">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
@@ -278,6 +278,94 @@ public sealed class NodeManagerHistoryReadTests : IDisposable
await host.DisposeAsync();
}
/// <summary>
/// #677: a client <c>AggregateConfiguration</c> that differs from the server defaults is refused per node with
/// <c>BadAggregateConfigurationRejected</c> (it used to be silently ignored); the source is never invoked.
/// </summary>
[Theory]
[InlineData(true, (byte)100, (byte)100, false)] // TreatUncertainAsBad
[InlineData(false, (byte)50, (byte)100, false)] // PercentDataBad
[InlineData(false, (byte)100, (byte)80, false)] // PercentDataGood
[InlineData(false, (byte)100, (byte)100, true)] // UseSlopedExtrapolation
[InlineData(false, (byte)0, (byte)0, false)] // an all-zero structure without UseServerCapabilitiesDefaults
public async Task Processed_with_a_non_default_aggregate_configuration_is_rejected(
bool treatUncertainAsBad, byte percentBad, byte percentGood, bool sloped)
{
var (host, server) = await BootAsync();
var nm = server.NodeManager!;
var fake = new RecordingHistorianDataSource { ProcessedResult = Empty() };
nm.HistorianDataSource = fake;
nm.EnsureVariable("eq-1/cfg", parentFolderNodeId: null, displayName: "Cfg", dataType: "Float",
writable: false, historianTagname: "WW.Cfg", realm: AddressSpaceRealm.Uns);
var nodeId = nm.TryGetVariable("eq-1/cfg")!.NodeId;
var (_, errors) = InvokeHistoryRead(server, nm, new ReadProcessedDetails
{
StartTime = DateTime.UtcNow.AddHours(-1), EndTime = DateTime.UtcNow, ProcessingInterval = 10_000.0,
AggregateType = new NodeIdCollection { ObjectIds.AggregateFunction_TimeAverage },
AggregateConfiguration = new AggregateConfiguration
{
UseServerCapabilitiesDefaults = false, TreatUncertainAsBad = treatUncertainAsBad,
PercentDataBad = percentBad, PercentDataGood = percentGood, UseSlopedExtrapolation = sloped,
},
}, nodeId);
errors[0].StatusCode.Code.ShouldBe(StatusCodes.BadAggregateConfigurationRejected);
fake.LastCall.ShouldBeNull();
await host.DisposeAsync();
}
/// <summary>
/// #677: <c>UseServerCapabilitiesDefaults</c> (whatever the other fields say), or a configuration spelling out exactly
/// the server defaults, is served.
/// </summary>
[Theory]
[InlineData(true, true, (byte)0, (byte)0, true)]
[InlineData(false, false, (byte)100, (byte)100, false)]
public async Task Processed_with_the_server_default_aggregate_configuration_is_served(
bool useServerDefaults, bool treatUncertainAsBad, byte percentBad, byte percentGood, bool sloped)
{
var (host, server) = await BootAsync();
var nm = server.NodeManager!;
var fake = new RecordingHistorianDataSource { ProcessedResult = Empty() };
nm.HistorianDataSource = fake;
nm.EnsureVariable("eq-1/cfg-ok", parentFolderNodeId: null, displayName: "CfgOk", dataType: "Float",
writable: false, historianTagname: "WW.CfgOk", realm: AddressSpaceRealm.Uns);
var nodeId = nm.TryGetVariable("eq-1/cfg-ok")!.NodeId;
var (_, errors) = InvokeHistoryRead(server, nm, new ReadProcessedDetails
{
StartTime = DateTime.UtcNow.AddHours(-1), EndTime = DateTime.UtcNow, ProcessingInterval = 10_000.0,
AggregateType = new NodeIdCollection { ObjectIds.AggregateFunction_TimeAverage },
AggregateConfiguration = new AggregateConfiguration
{
UseServerCapabilitiesDefaults = useServerDefaults, TreatUncertainAsBad = treatUncertainAsBad,
PercentDataBad = percentBad, PercentDataGood = percentGood, UseSlopedExtrapolation = sloped,
},
}, nodeId);
StatusCode.IsGood(errors[0].StatusCode).ShouldBeTrue();
fake.LastCall.ShouldBe("Processed");
fake.LastAggregate.ShouldBe(HistoryAggregateType.TimeAverage);
await host.DisposeAsync();
}
/// <summary>#677: the server's declared aggregate defaults are the SDK's own server defaults.</summary>
[Fact]
public void The_server_aggregate_defaults_agree_with_the_SDK_AggregateManager()
{
var sdk = new Opc.Ua.Server.AggregateManager(null!).GetDefaultConfiguration(null);
var ours = ServerAggregateConfiguration.Defaults;
ours.TreatUncertainAsBad.ShouldBe(sdk.TreatUncertainAsBad);
ours.PercentDataBad.ShouldBe(sdk.PercentDataBad);
ours.PercentDataGood.ShouldBe(sdk.PercentDataGood);
ours.UseSlopedExtrapolation.ShouldBe(sdk.UseSlopedExtrapolation);
ServerAggregateConfiguration.IsServable(sdk).ShouldBeTrue();
ServerAggregateConfiguration.IsServable(null).ShouldBeTrue();
}
/// <summary>#285: a Part 13 aggregate beyond the original five (Interpolative) is forwarded to the
/// backend as its own <see cref="HistoryAggregateType"/> member.</summary>
[Fact]