diff --git a/CLAUDE.md b/CLAUDE.md index 935c55cf..57b72912 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1350,7 +1350,7 @@ historian tagname, which is the tag's driver `FullName`. ### Pluggable backends, SQL Server and HistoryUpdate (2026-10) - **Selection:** `ServerHistorian:Backend` (`Gateway` default | `SqlServer` | `QuestDb`; unknown fails host start). `HistorianBackendRegistry` → `IHistorianBackendProvider` builds all five seams; the gateway keys are only required for `Gateway`. `SqlServer` uses `ServerHistorian:SqlServer:{ConnectionString (required), Schema=hist, CommandTimeout=30s, AutoCreateSchema=true}`. -- **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; per-sample aggregates (12, simple definitions) are pushed down to T-SQL with bucket math in **100 ns ticks**; the Part 13 time-weighted five (TimeAverage/TimeAverage2/Total/Total2/Interpolative, #671 — ⚠️ `Total` is now the time-weighted integral, no longer `SUM`) stream window + 4 bound rows into the shared `TimeWeightedAggregator` (both SQL backends, identical results; client `AggregateConfiguration` ignored; status = the SDK `AggregateCalculator` time-based rule, Uncertain time counts as Good; reverse `Interpolative` anchored at the later time; window capped by `ServerHistorian:MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations`). Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. **Retention (#667):** opt-in `SqlServer:RetentionDays` (+ `PurgeInterval` 1h, `PurgeBatchSize` 4000) -> `HistorianRetentionService` purges old `hist.Sample` (per tag) / `hist.Event` in `DELETE TOP(n)` batches, Primary-gated on `IHistorizationRoleView`, one purger per DB via `sp_getapplock`; counter `otopcua.historian.retention.purged{table}`. Optional `IHistorianBackendProvider.CreateRetention` (default null). `docs/Historian.md` §Retention. +- **SQL Server** (`Driver.Historian.SqlServer`): `hist.Sample` PK `(TagId, SourceTimeUtc)` — writes are insert-if-absent, so outbox redelivery is idempotent. Stores every value kind with its **real status**; per-sample aggregates (10, simple definitions) are pushed down to T-SQL with bucket math in **100 ns ticks**; the Part 13 time-weighted seven (TimeAverage/TimeAverage2/Total/Total2/Interpolative, #671 — ⚠️ `Total` is now the time-weighted integral, no longer `SUM` — plus the **duration-based** PercentGood/PercentBad, #680) stream window + 4 bound rows into the shared `TimeWeightedAggregator` (both SQL backends, identical results; reverse `Interpolative` anchored at the later time; window capped by `ServerHistorian:MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations`, percentages included). ⚠️ **#680 status rules (client-visible):** TimeAverage/Total are Uncertain if any part of the interval uses an Uncertain value, is bridged across a Bad sample, is extrapolated past the last sample (⇒ a live tag's latest interval is Uncertain) or is not covered; TimeAverage2/Total2 keep the SDK time-based rule (Uncertain time counts as Good) but never extrapolate past the last sample — that interval is Partial, later ones `BadNoData` (a report-by-exception tag with nothing after its last sample reads `BadNoData` until its next sample arrives). `docs/Historian.md` §"Time-weighted aggregates". Docker-dev recipe, schema and aggregate semantics: `docs/Historian.md` §"SQL Server backend". Integration tests need `HISTORIAN_SQL_TEST_CONNECTION`. **Retention (#667):** opt-in `SqlServer:RetentionDays` (+ `PurgeInterval` 1h, `PurgeBatchSize` 4000) -> `HistorianRetentionService` purges old `hist.Sample` (per tag) / `hist.Event` in `DELETE TOP(n)` batches, Primary-gated on `IHistorizationRoleView`, one purger per DB via `sp_getapplock`; counter `otopcua.historian.retention.purged{table}`. Optional `IHistorianBackendProvider.CreateRetention` (default null). `docs/Historian.md` §Retention. - **QuestDB** (`Driver.Historian.QuestDb`, `ServerHistorian:QuestDb:*`, Npgsql over PG wire; docker-dev overlay `docker-compose.questdb.yml`): same kinds/status/aggregates/HistoryUpdate as SqlServer, soft delete by tombstone (INSERT only — never `UPDATE`/`DELETE`; every read filters `NOT deleted`). ⚠️ **Timestamps are µs** (`QuestDbConnection.Round`/`BindTimestamp`; `IHistorianDataSource.TimestampResolutionTicks` = 10 steps HistoryRead paging), **pre-1970 cannot be stored** (writer drops, HistoryUpdate `BadOutOfRange`), every write waits for WAL visibility, TTL is enforced on write commits only, set Npgsql `Maximum Pool Size` < QuestDB's 64. `docs/Historian.md` §"QuestDB backend". - ⚠️ **Reverse processed reads are Part 13/SDK-anchored (#677, client-visible):** every aggregate on SqlServer/QuestDB now anchors a reverse read at its later time — window `(end, start]`, partial interval earliest, each stamped with its later edge (was: earlier-anchored, stamped earlier edge); on forward reads Partial also marks the interval the data starts inside (not Average — the SDK's rule, forward-only); a non-default client `AggregateConfiguration` ⇒ `BadAggregateConfigurationRejected`; Gateway provisioning sends `IntegralDivisor = 1`. SDK-parity tests: `SdkAggregateParityTests`. `docs/Historian.md` §Aggregates. - **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good. @@ -1381,7 +1381,7 @@ authorization uses the standard `AccessLevels.HistoryRead` bit set at materializ | `MaxTieClusterOverfetch` | `65536` | Bounded over-fetch the HistoryRead-Raw paging uses to page within an oversized same-timestamp tie cluster (retained from the prior backend) | | `HistoryReadBatchParallelism` | `4` | Max nodes served concurrently within one HistoryRead batch (03/S3); `⌈N/P⌉ × CallTimeout` worst case instead of sequential | | `MaxConcurrentHistoryReadBatches` | `16` | Process-wide concurrent HistoryRead batch cap (03/S3); flood degrades to `BadTooManyOperations`. Total in-flight reads capped at 16 × 4 = 64 | -| `MaxTimeWeightedRawRows` | `1000000` | `SqlServer`/`QuestDb` only (#671): raw in-window rows one time-weighted aggregate read (incl. `Total`) may stream into the C# evaluator; window query capped at this + 1, past it ⇒ `BadTooManyOperations` for that node. ≤ 0 = unbounded | +| `MaxTimeWeightedRawRows` | `1000000` | `SqlServer`/`QuestDb` only (#671): raw in-window rows one time-weighted aggregate read (incl. `Total` and, since #680, PercentGood/PercentBad) may stream into the C# evaluator; window query capped at this + 1, past it ⇒ `BadTooManyOperations` for that node. ≤ 0 = unbounded | | `HistoryReadDeadline` | `00:01:00` | Per-request HistoryRead wall-clock deadline (03/S3); a node still in flight surfaces `BadTimeout`. Non-positive = unbounded | ### Alarm-history path (`AlarmHistorian` section) diff --git a/docs/Historian.md b/docs/Historian.md index 194833ed..9d2a5d2a 100644 --- a/docs/Historian.md +++ b/docs/Historian.md @@ -1002,6 +1002,16 @@ and the raw-row cap below). Bucketing (edges, truncated interval, reverse anchor > two samples now has a value instead of `BadNoData`. > - **Partial:** set on the forward end-of-data interval (`TimeAverage2`/`Total2`/percentages) and, on a reverse read, on the interval > holding the latest sample when nothing follows it (all of this family but `Interpolative`). +> - ⚠️ **Report-by-exception tags (a slowly changing or deadband-filtered value) are the visible case.** A tag whose last stored +> sample lies *before* the window and that has nothing after it has, by Part 13, **ended its data**: every interval of +> `TimeAverage2` / `Total2` / `PercentGood` / `PercentBad` is **`BadNoData`**, and every interval of `TimeAverage` / `Total` is the +> held value, **Uncertain**. When the next sample arrives, a re-read of the same window returns values / `Good` for those intervals +> (the held value is then interpolated between real samples, not extrapolated). So a dashboard, a ScadaBridge consumer or any +> client that caches processed results for "closed" intervals can see an interval go `BadNoData` → value, or Uncertain → Good, on a +> later read — re-read recent intervals, or use `TimeAverage`/`Total` (which always carry a value) for a report-by-exception tag. The +> SDK's own calculators behave the same way. Likewise the interval holding a tag's latest sample is Partial and, until a later +> sample arrives, `TimeAverage2`/`Total2` there cover only up to it — the comms-loss recovery shape (a single Good sample after an +> outage marker, nothing after) answers that sample's value (`Total2` 0), Uncertain + Partial, never Bad/null. **Server defaults, fixed:** `TreatUncertainAsBad = false`, `PercentDataBad = PercentDataGood = 100`, `UseSlopedExtrapolation = false` (`ServerAggregateConfiguration` in the OPC UA server project — the SDK's own `AggregateManager` defaults, which a test pins). **A @@ -1049,8 +1059,10 @@ are used", and the SDK's `AverageAggregateCalculator`): value held up to an outage). The duration is the interval's, cut at the end of the data: - After the end of the data ⇒ `BadNoData` (Part 13 "After End of Data"); the interval the data ends inside covers only up to the last - sample and is **Partial** (forward); when the data ends exactly on its start there is nothing to cover and the value is the last - sample's (`Total2` 0), with its status. + sample and is **Partial** (forward). When it has no non-Bad time before the last sample — the data ends exactly on its start, or the + last sample is the first usable one after an outage (comms-loss recovery), or the tag's only sample — there is nothing to cover and + the value is the last sample's (`Total2` 0): `Good` only when the data ends exactly on the interval start with a Good sample, + otherwise `UncertainDataSubNormal` (the SDK answers the same, modulo its 1 ms artifact). - Every instant of the (covered) interval non-Bad ⇒ `Good`. - Some non-Bad time but not all (an outage, or time before the first sample) ⇒ `UncertainDataSubNormal`. - No non-Bad time but some outage time (an interval wholly inside an outage) ⇒ generic **`Bad`** (`0x80000000`), value null — Part 13's @@ -1118,7 +1130,15 @@ ours follow Part 13: `TimeAverage2`/`Total2` stop at the last sample, the percen read the SDK's backward slice files a sample exactly on an interval's later edge as its early bound — so a reverse `Interpolative` at such a point comes back Interpolated (Uncertain if a Bad sample follows) and a reverse `TimeAverage`/`Total` interval ending on it takes the status of the segment beyond it — where ours, like the SDK forward, returns the raw sample and keeps the interval's own segments -(an SDK defect: its forward and reverse answers differ). +(an SDK defect: its forward and reverse answers differ). Three more, kept deliberately (#680 review): an **outage marker as the last +sample** (nothing after it) runs Bad to the window end here — `TimeAverage2`/`Total2` generic `Bad` and the percentages 100 % Bad +after it, no end-of-data Partial on the interval holding it, and `TimeAverage`/`Total` Uncertain over the span the last usable value is +held up to the marker (no Good value follows ⇒ extrapolated) — where the SDK treats the data as ended at the marker (`BadNoData` after +it, Partial on that interval, the held span Good): the tag is in an outage, not out of data. **Booleans** are stepped here (a +non-analog value is held; the simple bound next to an Uncertain sample takes the earlier sample's status), while the SDK interpolates +them as numbers — e.g. an interval that is `false` for its first half and `true` for its second averages 0.5 here, 1 in the SDK, and a +PercentGood region ahead of an Uncertain Boolean is Good here, Uncertain in the SDK. Each is pinned by `TimeWeightedAggregatorTests` and +by a parity dataset (`OutageLast`, `BoolSwitch`) whose masks are documented in `SdkAggregateParityTests`. **Pushdown trade-off.** Unlike the per-sample aggregates, the time-weighted family is **not** pushed down to SQL: the backend streams the window's rows plus the four bounds (one batch of five result sets on SQL Server; a bounds `UNION ALL` + a window select on QuestDB) diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/TimeWeightedAggregator.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/TimeWeightedAggregator.cs index 5f0d6446..f2b328ca 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/TimeWeightedAggregator.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/TimeWeightedAggregator.cs @@ -337,11 +337,16 @@ public sealed class TimeWeightedAggregator { // After the end of the data: nothing to aggregate (Part 13 "After End of Data: Bad_NoData"). if (lo > dataEnd) return AggregateEvaluator.NoData(stamp, now); - if (lo == dataEnd && _prevAny is { } last) + if (_usableTicks[i] == 0 && dataEnd < hi && _prevAny is { } last) { - // The data ends exactly on this interval's start: nothing covered; the value at that instant. + // The data ends inside this interval and no non-Bad time precedes its last sample here (the data + // ends on the interval's start; or the last sample is the first usable one after an outage — the + // comms-loss recovery shape — or the tag's only sample): nothing covered, so the value at that + // instant (Total2 0). Good only when the data ends exactly on the interval start with a Good + // sample (nothing before it in the interval); otherwise the interval also holds Bad / no-data time + // ⇒ Uncertain (the SDK answers the same, modulo its 1 ms artifact). if (!IsNumeric(last)) return Calc(null, HistorianStatusCodes.BadAggregateInvalidInputs); - var st = last.IsGood ? HistorianStatusCodes.Good : HistorianStatusCodes.UncertainDataSubNormal; + var st = lo == dataEnd && last.IsGood ? HistorianStatusCodes.Good : HistorianStatusCodes.UncertainDataSubNormal; return Calc(_aggregate == HistoryAggregateType.TimeAverage2 ? last.Value.Number : 0d, st, endOfData); } duration = Math.Min(hi, dataEnd) - lo; diff --git a/tests/Drivers/Historian.Conformance/ProcessedConformance.cs b/tests/Drivers/Historian.Conformance/ProcessedConformance.cs index 448ad111..8958b19f 100644 --- a/tests/Drivers/Historian.Conformance/ProcessedConformance.cs +++ b/tests/Drivers/Historian.Conformance/ProcessedConformance.cs @@ -352,16 +352,28 @@ public abstract class ProcessedConformance : ConformanceBase } } var reverseInterpolative = agg == HistoryAggregateType.Interpolative && from > to; - // Documented SDK divergence (#680, see SdkAggregateParityTests): on a reverse read the SDK files a sample exactly - // on an interval's later edge as its early bound, so TimeAverage/Total there take the status of the segment - // beyond it (Dense: +20 next to the Bad +22.5). - var reverseEdgeStatus = from > to && agg is HistoryAggregateType.TimeAverage or HistoryAggregateType.Total; - SdkAggregateOracle.ShouldMatch(ours, SdkAggregateOracle.Run(agg, start, end, interval, rows), + var sdk = SdkAggregateOracle.Run(agg, start, end, interval, rows); + if (from > to && agg is HistoryAggregateType.TimeAverage or HistoryAggregateType.Total) + { + // Documented SDK divergence (#680, see SdkAggregateParityTests.MaskDocumented): on a reverse read the SDK files + // a sample exactly on an interval's later edge as its early bound, so a TimeAverage/Total interval ending on a + // sample takes the status of the segment beyond it (Dense: +20 next to the Bad +22.5). Only that interval, + // only Good (ours) vs Uncertain (SDK), is aligned. + var plan = BucketPlan.Create(start, end, interval); + for (long i = 0; i < plan.BucketCount; i++) + { + var k = (int)(plan.BucketCount - 1 - i); + if (k < sdk.Count && rows.Any(r => r.Item1.Ticks == plan.UpperTicks(i)) + && (ours[k].StatusCode & 0xFFFF0000u) == 0 + && (sdk[k].StatusCode & 0xFFFF0000u) == 0x40A40000u) + ours[k] = ours[k] with { StatusCode = ours[k].StatusCode | 0x40A40000u }; + } + } + SdkAggregateOracle.ShouldMatch(ours, sdk, $"{agg} {from}→{to}/{intervalS}s", compareCalculatedInterpolatedBits: tw && !reverseInterpolative, // pre-existing, #680: Start/End raw-sample timestamp + status; Delta status (see SdkAggregateParityTests). - compareCodeBits: agg != HistoryAggregateType.Delta && !reverseInterpolative && !startOrEnd - && !reverseEdgeStatus, + compareCodeBits: agg != HistoryAggregateType.Delta && !reverseInterpolative && !startOrEnd, compareTimestamps: !startOrEnd); } } diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/SdkAggregateParityTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/SdkAggregateParityTests.cs index a8efc390..f7d03091 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/SdkAggregateParityTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/SdkAggregateParityTests.cs @@ -32,6 +32,16 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests; /// GetRegionsInValueSet); Part 13 ends TimeAverage2/Total2 at the last sample and runs the PercentGood/Bad /// region to the interval end, which ours does (#680; pinned by TimeWeightedAggregatorTests). That /// interval's value is not compared; its status, Partial bit and timestamp are. +/// an outage marker as the last sample (): ours runs the outage to the window +/// end — TimeAverage2/Total2 Bad, the percentages 100 % Bad, no end-of-data Partial on the interval holding +/// it; the SDK treats the data as ended (BadNoData after it, Partial on that interval); and TimeAverage/Total +/// over the span the last usable value is held up to the marker are Uncertain here (no Good value follows ⇒ +/// extrapolated, Part 13), Good in the SDK — masked by +/// ; +/// Booleans (): ours steps them (Part 13: a non-analog value is held, and the +/// bound next to an Uncertain sample takes the earlier sample's status); the SDK interpolates them as numbers — its +/// values (e.g. 1 vs our 0.5 over an interval switching half-way) and the Good/Uncertain choice are not compared, +/// Bad vs non-Bad, timestamps and Partial are; /// TimeAverage / Total, reverse, a sample exactly on an interval's later edge: the SDK's backward /// UpdateSlice files it as the early bound (below), so the interval takes the status of the segment beyond /// it (Uncertain when that segment skips a Bad sample); ours keeps the interval's own segments — masked by @@ -91,6 +101,48 @@ public sealed class SdkAggregateParityTests S(100, HistorianValue.FromDouble(40), Good), ]; + /// + /// #680 review: the comms-loss recovery shape — a Good sample, an outage marker, then ONE recovery sample and nothing + /// after (+27.5, inside an interval of every window, never on an edge): the interval holding the recovery sample has + /// no non-Bad time before it, so TimeAverage2/Total2 answer the sample's value (Total2 0), Uncertain + Partial. + /// + private static readonly StoredSample[] RecoveryEnds = + [ + S(-10, HistorianValue.FromDouble(10), Good), + S(5, HistorianValue.Empty, BadNoCommunication), + S(27.5, HistorianValue.FromDouble(20), Good), + ]; + + /// + /// #680 review: an outage marker as the LAST sample (+31, nothing after) — ours: the outage runs to the window end + /// (TimeAverage2/Total2 Bad, the percentages 100 % Bad); the SDK: the data ended (BadNoData, Partial on the interval + /// holding the marker). Also TimeAverage/Total over the span the last usable value is held up to the marker: ours + /// Uncertain (Part 13 — no Good value after it, so extrapolated), the SDK Good. Documented, deliberate differences. + /// + private static readonly StoredSample[] OutageLast = + [ + S(-10, HistorianValue.FromDouble(10), Good), + S(25, HistorianValue.FromDouble(20), Good), + S(31, HistorianValue.Empty, BadNoCommunication), + ]; + + /// + /// #680 review: Boolean (stepped) data with an Uncertain and a Bad sample. Ours steps Booleans (Part 13: a + /// non-analog value is held); the SDK interpolates them as numbers and bounds them sloped — so only timestamps, + /// status and Partial are compared for the values it computes that way (see ). + /// + private static readonly StoredSample[] BoolSwitch = + [ + S(-10, HistorianValue.FromBoolean(true), Good), + S(10, HistorianValue.FromBoolean(false), Good), + S(27.5, HistorianValue.FromBoolean(true), Uncertain), + S(32, HistorianValue.FromBoolean(false), Good), + S(47.5, HistorianValue.FromBoolean(true), Good), + S(62, HistorianValue.FromBoolean(true), Bad), + S(71, HistorianValue.FromBoolean(false), Good), + S(100, HistorianValue.FromBoolean(true), Good), + ]; + /// Data that starts inside the window (no sample before +7) and runs past its end. private static readonly StoredSample[] LateStart = Enumerable.Range(0, 20) .Select(i => S(7 + i * 5, HistorianValue.FromDouble(i % 2 == 0 ? 20 : 40), Good)) @@ -148,7 +200,11 @@ public sealed class SdkAggregateParityTests var plan = BucketPlan.Create(At(from), At(to), TimeSpan.FromSeconds(intervalS)); var list = ours.ToList(); var last = data.Max(x => x.TimeUtc).Ticks; - var dataEnds = last < Math.Max(At(from).Ticks, At(to).Ticks) && data.Single(x => x.TimeUtc.Ticks == last).IsUsable; + var endsInWindow = last < Math.Max(At(from).Ticks, At(to).Ticks); + var dataEnds = endsInWindow && data.Single(x => x.TimeUtc.Ticks == last).IsUsable; + var outageEnds = endsInWindow && !dataEnds; + var boolean = data.Any(x => x.Value.Kind == HistorianValueKind.Boolean); + var percent = agg is HistoryAggregateType.PercentGood or HistoryAggregateType.PercentBad; for (long i = 0; i < plan.BucketCount; i++) { var k = (int)(plan.Reversed ? plan.BucketCount - 1 - i : i); @@ -170,6 +226,36 @@ public sealed class SdkAggregateParityTests if (dataEnds && (twoVariant || agg is HistoryAggregateType.PercentGood or HistoryAggregateType.PercentBad) && last >= plan.LowerTicks(i) && last < plan.UpperTicks(i) && sdk[k].Value is not null) list[k] = list[k] with { Value = sdk[k].Value }; + // An outage marker as the last sample: ours runs the outage to the window end (no end-of-data Partial on the + // interval holding it; Bad / 100 % Bad after it); the SDK treats the data as ended there. + if (outageEnds && (twoVariant || percent)) + { + if (last >= plan.LowerTicks(i) && last < plan.UpperTicks(i)) + list[k] = list[k] with + { + StatusCode = (list[k].StatusCode & ~HistorianInfoBits.HistorianPartial) + | (sdk[k].StatusCode & HistorianInfoBits.HistorianPartial), + }; + else if (plan.LowerTicks(i) > last && (sdk[k].StatusCode & StatusBits.CodeBitsMask) == HistorianStatusCodes.BadNoData) + list[k] = list[k] with { Value = null, StatusCode = HistorianStatusCodes.BadNoData }; + } + // ...and TimeAverage/Total hold the last usable value up to that marker: Part 13 calls that extrapolation (no + // Good value after it) ⇒ Uncertain; the SDK, which sees a later (Bad) sample, reports the held span Good. + var lastUsable = data.Where(x => x.IsUsable).Max(x => x.TimeUtc).Ticks; + if (outageEnds && agg is HistoryAggregateType.TimeAverage or HistoryAggregateType.Total + && plan.LowerTicks(i) < last && plan.UpperTicks(i) > lastUsable + && (list[k].StatusCode & StatusBits.CodeBitsMask) == HistorianStatusCodes.UncertainDataSubNormal + && (sdk[k].StatusCode & StatusBits.CodeBitsMask) == HistorianStatusCodes.Good) + list[k] = list[k] with { StatusCode = list[k].StatusCode & ~StatusBits.CodeBitsMask }; + // Booleans: the SDK slopes them (values, and the status of a bound next to an Uncertain sample); ours steps. + if (boolean && sdk[k].Value is not null && list[k].Value is not null) + list[k] = list[k] with { Value = sdk[k].Value }; + if (boolean && (StatusBits.IsGood(list[k].StatusCode) || StatusBits.IsGood(sdk[k].StatusCode)) + && !StatusBits.IsBad(list[k].StatusCode) && !StatusBits.IsBad(sdk[k].StatusCode)) + list[k] = list[k] with + { + StatusCode = (list[k].StatusCode & ~StatusBits.CodeBitsMask) | (sdk[k].StatusCode & StatusBits.CodeBitsMask), + }; } return list; } @@ -188,6 +274,7 @@ public sealed class SdkAggregateParityTests foreach (var (name, data) in new[] { ("Bounded", Bounded), ("Dense", Dense), ("EndsInside", EndsInside), ("OutageInside", OutageInside), + ("RecoveryEnds", RecoveryEnds), ("OutageLast", OutageLast), ("BoolSwitch", BoolSwitch), }) { try diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/TimeWeightedAggregatorTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/TimeWeightedAggregatorTests.cs index b9351326..6965e13d 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/TimeWeightedAggregatorTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common.Tests/TimeWeightedAggregatorTests.cs @@ -180,6 +180,84 @@ public sealed class TimeWeightedAggregatorTests ((double)Run(HistoryAggregateType.TimeAverage, 0, 20, 0, late).Single().Value!).ShouldBe(10d); } + [Fact] + public void A_recovery_sample_after_an_outage_that_ends_the_data_is_its_own_value_Uncertain_and_Partial() + { + // #680 review (the comms-loss recovery shape): −10 = 10 G, +5 outage, +25 = 20 G (the recovery), nothing after. + // [0,20): 10 held into the outage over [0,5) ⇒ TimeAverage2 10 (Total2 50), Uncertain. [20,40) holds the end of the + // data but no non-Bad time before it ⇒ the sample's value 20 (Total2 0), Uncertain + Partial — never Bad/null (the + // SDK answers the same). [40,60) is after the end of the data ⇒ BadNoData. + StoredSample[] s = + [ + S(-10, HistorianValue.FromDouble(10), Good), S(5, HistorianValue.Empty, BadNoCommunication), + S(25, HistorianValue.FromDouble(20), Good), + ]; + var ta2 = Run(HistoryAggregateType.TimeAverage2, 0, 60, 20, s); + Values(ta2, 10, 20, null); + ta2.Select(x => x.StatusCode).ShouldBe([UncertainCalc, UncertainCalcPartial, NoDataCalc]); + var t2 = Run(HistoryAggregateType.Total2, 0, 60, 20, s); + Values(t2, 50, 0, null); + t2.Select(x => x.StatusCode).ShouldBe([UncertainCalc, UncertainCalcPartial, NoDataCalc]); + } + + [Fact] + public void A_single_sample_is_its_own_value_Uncertain_and_Partial() + { + // #680 review: the tag's only sample, +25 = 20. One interval [0,60) ⇒ 20 (Total2 0), Uncertain + Partial (time + // before the data is Bad). At 20 s: [0,20) before the data ⇒ BadNoData; [20,40) the sample ⇒ 20, Uncertain + + // Partial; [40,60) after the end of the data ⇒ BadNoData. + StoredSample[] one = [S(25, HistorianValue.FromDouble(20), Good)]; + var whole = Run(HistoryAggregateType.TimeAverage2, 0, 60, 0, one).Single(); + ((double)whole.Value!).ShouldBe(20d); + whole.StatusCode.ShouldBe(UncertainCalcPartial); + var total = Run(HistoryAggregateType.Total2, 0, 60, 0, one).Single(); + ((double)total.Value!).ShouldBe(0d); + total.StatusCode.ShouldBe(UncertainCalcPartial); + var split = Run(HistoryAggregateType.TimeAverage2, 0, 60, 20, one); + Values(split, null, 20, null); + split.Select(x => x.StatusCode).ShouldBe([NoDataCalc, UncertainCalcPartial, NoDataCalc]); + } + + [Fact] + public void An_outage_marker_as_the_last_sample_runs_Bad_to_the_window_end() + { + // #680 review, a deliberate difference from the SDK (which treats the data as ended at the marker): −10 = 10 G, + // +25 = 20 G, +30 outage, nothing after. TimeAverage2 [20,40): [20,25) 18.5714…→20 (96.43) + [25,30) 20 held into + // the outage (100) over 10 s = 19.642857…, Uncertain, NOT Partial (the data did not end — the tag is in an + // outage); [40,60) inside the still-open outage ⇒ generic Bad. PercentBad [40,60) = 100, Good. TimeAverage holds + // 20 to the window end — extrapolated (no Good value follows) ⇒ Uncertain. + StoredSample[] s = + [ + S(-10, HistorianValue.FromDouble(10), Good), S(25, HistorianValue.FromDouble(20), Good), + S(30, HistorianValue.Empty, BadNoCommunication), + ]; + var ta2 = Run(HistoryAggregateType.TimeAverage2, 0, 60, 20, s); + ((double)ta2[1].Value!).ShouldBe(19.642857142857142, 1e-9); + ta2.Select(x => x.StatusCode).Skip(1).ShouldBe([UncertainCalc, BadCalc]); + var pb = Run(HistoryAggregateType.PercentBad, 0, 60, 20, s); + Values(pb, 0, 50, 100); + pb.ShouldAllBe(x => x.StatusCode == GoodCalc); + Run(HistoryAggregateType.TimeAverage, 0, 60, 20, s).Select(x => x.StatusCode).Skip(1) + .ShouldBe([UncertainCalc, UncertainCalc]); + } + + [Fact] + public void Boolean_percent_regions_take_the_stepped_bound_status() + { + // #680 review: −10 = true G, +15 = true U, +30 = false G, +50 = false G. [0,20): the bound at +0 is stepped from the + // Good −10 (a Boolean is held, not sloped toward the Uncertain +15) ⇒ [0,15) Good, [15,20) Uncertain ⇒ PercentGood + // 75. (The SDK slopes Booleans, so it calls the bound Uncertain and answers 0.) [20,40): [20,30) Uncertain (bound + // stepped from the Uncertain +15), [30,40) Good ⇒ 50. + StoredSample[] s = + [ + S(-10, HistorianValue.FromBoolean(true), Good), S(15, HistorianValue.FromBoolean(true), Uncertain), + S(30, HistorianValue.FromBoolean(false), Good), S(50, HistorianValue.FromBoolean(false), Good), + ]; + Values(Run(HistoryAggregateType.PercentGood, 0, 40, 20, s), 75, 50); + // TimeAverage over the same Booleans is stepped: [20,40) = 1 over [20,30), 0 over [30,40) ⇒ 0.5 (the SDK: 1). + Values(Run(HistoryAggregateType.TimeAverage, 0, 40, 20, s), 1, 0.5); + } + // ---------------------------------------------------------------- #680: PercentGood / PercentBad are duration-based [Fact]