- Average/Minimum/Maximum/Range/*ActualTime refuse (BadAggregateInvalidInputs) only a GOOD
String/DateTime sample (new GoodNonNumericCount / good_non_numeric_count column); an Uncertain
one is skipped and makes the min/max family Uncertain, matching the SDK (Average skips it,
ComputeMinMax flags the uncastable value). Delta keeps its usable-sample refusal.
- ProcessedConformance: an Uncertain value above the Good maximum (pins UncMax/unc_max through
the real SQL) and an Uncertain String through the real SQL.
- The_uncertain_dataset_is_not_vacuous also proves the min/max STATUS discrimination.
Co-Authored-By: Claude <noreply@anthropic.com>
Part 13 defines Average/Minimum/Maximum/Range/*ActualTime over Good raw values
(independent of TreatUncertainAsBad) and the SDK's calculators skip Uncertain values
under the server's TreatUncertainAsBad=false. SqlServer/QuestDB value columns now use
the good flag (g) instead of usable (u); new UncMin/UncMax columns feed the Part 13
Minimum/Maximum/Range status rule (Uncertain on a Bad sample or an Uncertain value
beyond the Good extreme -- the SDK applies only the Bad half). An Uncertain-only
interval is BadNoData. Uncertain-bearing parity datasets (in-memory + real SQL) match
the SDK unmasked; the beyond-extreme divergence is pinned explicitly.
Co-Authored-By: Claude <noreply@anthropic.com>
Count: Good samples only, value-based status (all Bad => generic Bad), 0/Good on an
empty interval inside the tag's data range [first, last] (both directions).
Start/End: the raw first/last sample of any quality (own value, status, timestamp).
Delta: Uncertain only when the interval's first or last sample is not usable.
Forward end-of-data Partial (SDK m_endOfData, earlier edge inclusive) for every
per-sample aggregate but Average (PercentGood/Bad left to lane A).
Backends push down first/last-any-quality columns and fetch the tag's data range
(earliest <= window end, latest >= window start) only when NeedsDataRange.
SDK parity masks for Start/End/Delta/Count removed; new parity datasets.
TimeAverage2/Total2: the interval the data ends inside with no non-Bad time
before its last sample (comms-loss recovery, a single sample) answers that
sample's value (Total2 0), Uncertain + Partial, instead of Bad/BadNoData —
as the SDK. Parity datasets RecoveryEnds, OutageLast, BoolSwitch with
documented masks; ProcessedConformance's reverse TA/Total mask narrowed to
the later-edge-sample interval. Docs: report-by-exception consequence,
outage-last and Boolean differences; CLAUDE.md historian bullet.
TimeAverage/Total: Uncertain when a Bad value is skipped, an Uncertain value
is used, the value is extrapolated or the interval is not covered; an
interval wholly inside an outage bridges (Uncertain); Total integrates only
the covered part. TimeAverage2/Total2 stop at the end of the data (Partial,
then BadNoData). PercentGood/PercentBad become duration-based and move to
the time-weighted family (now subject to MaxTimeWeightedRawRows). Reverse
reads set Partial on the interval holding the latest sample when nothing
follows (SDK m_startOfData). Parity tests gain end-of-data and outage
datasets; pre-existing #680 masks removed.
F1: the SDK applies its m_startOfData Partial rule to forward reads only; ours
now does too (AggregateEvaluator.EvaluateAll + TimeWeightedAggregator). New
reverse late-start parity tests through the SDK oracle (unit) and real SQL
(conformance, both backends).
F2: AggregateEvaluator.NeedsEarliestSample — the earliest-sample query runs
only for forward reads of aggregates other than Average, and is bounded by
the window end (SqlServer < @end, QuestDB ts < $2).
F3: per-sample SDK parity extended to Range, Start, End, Delta, Min/Max
ActualTime and PercentGood/Bad (oracle maps StatusAggregateCalculator);
pre-existing definitional differences masked and marked #680.
Docs: Historian.md Partial/SDK-parity sections, CLAUDE.md bullet.
- BucketPlan: reverse processed reads anchored at the later time for every
aggregate: window (end, start], partial interval earliest, results stamped
with the later edge, latest first (SDK CreateSlice/GetTimestamp/backward
GetValues). SqlServer adds @bucketAdjustTicks + flips the window predicate;
QuestDB binds BucketBaseUnixTicks. IndexOf is the C# statement of the math.
- Partial bit on the interval the tag's data starts inside (SDK m_startOfData),
every aggregate but Average; per-sample reads fetch the earliest sample.
- Node manager: non-default client AggregateConfiguration =>
BadAggregateConfigurationRejected per node (ServerAggregateConfiguration,
pinned to the SDK AggregateManager defaults).
- GatewayTagProvisioner sends IntegralDivisor = 1.
- SDK parity oracle (Opc.Ua.Server calculators) in unit + both backends'
conformance suites, forward and reverse; remaining divergences documented.
- docs/Historian.md + CLAUDE.md release note (client-visible).
M1: the time-weighted family (incl. Total, formerly a pushed-down SUM) streams
raw window rows into the C# evaluator. New shared knob
ServerHistorian:MaxTimeWeightedRawRows (default 1,000,000; <=0 unbounded with a
Validate warning) flows through HistorianBackendRegistry into both providers.
The window query is capped at cap+1 (SQL Server TOP (@maxRows), QuestDB LIMIT $4);
one row past the cap throws the new Core.Abstractions
HistoryReadLimitExceededException, which the node manager maps to
BadTooManyOperations for that node. Not counted as a backend health failure;
bound rows are not counted; per-sample aggregates are not capped.
M2: reverse Interpolative is anchored at the later time (Part 13): points at
End - i*interval, emitted latest first. 10:00->09:00 @1h yields 10:00.
L1: Partial bit (0x4, = Opc.Ua.AggregateBits.Partial) on non-Bad results of a
truncated last interval (time-weighted family; per-sample family is a follow-up).
L2/L3: aligned with the SDK AggregateCalculator.GetTimeBasedStatusCode
(decompiled from Opc.Ua.Server 1.5.378.106): TreatUncertainAsBad=false counts
Uncertain time (Uncertain samples, held-into-outage, extrapolation) as Good
time; data but no non-Bad time => generic Bad (0x80000000); BadNoData only
when there is no data at all. Consequence: a live tag's latest interval reads
Good (documented).
L5: release note no longer claims Total matches the Gateway Integral exactly
(AVEVA's divisor is unverified); it notes Total is now evaluated in C# and
subject to the cap.
Tests: unit (reverse Interpolative x3, Partial, Bad vs BadNoData incl. an
open-ended outage, cap helpers, SDK bit oracles); conformance (reverse
Interpolative, raw-row cap at/over/0, bounds not counted, health unaffected);
node manager BadTooManyOperations; options default + warning. Each new
behaviour falsified by mutation (aggregator x9, data-source cap checks on
both backends against the real DBs, node-manager catch).
New AggregateFamily.TimeWeighted serving TimeAverage, TimeAverage2, Total,
Total2 and Interpolative, evaluated by the shared Sql.Common
TimeWeightedAggregator for both SQL backends (identical results by
construction). HistoryAggregateType gains TimeAverage2 + Total2 (appended);
OtOpcUaNodeManager.MapAggregate maps their NodeIds. Gateway's mapper already
refuses unknown defined members => BadAggregateNotSupported per node.
Semantics (server defaults, client AggregateConfiguration ignored):
TreatUncertainAsBad=false, PercentDataBad=PercentDataGood=100,
UseSlopedExtrapolation=false. Sloped for Double/Int64, stepped for Boolean;
a usable sample followed by a Bad one is held up to the outage (AtTime rule);
stepped Uncertain extrapolation past the last sample. TimeAverage/Total use
interpolated bounding (bridge outages); TimeAverage2/Total2 exclude Bad time.
Bounding values: latest usable + latest any before start, first any + first
usable at/after end (four TOP 1 / LIMIT 1 seeks).
CLIENT-VISIBLE: Total on SqlServer/QuestDb was SUM of samples; it is now the
time-weighted integral (TimeAverage x interval seconds), matching Gateway's
Integral. Release note in docs/Historian.md.
Trade-off (deviation from the preferred hybrid pushdown, allowed by the
design): the time-weighted family is NOT pushed down. Each read streams the
window's rows plus the four bounds (SQL Server: one five-result-set batch;
QuestDB: bounds UNION ALL + window select) into the C# evaluator. Network is
O(samples in window), memory O(buckets). Writing the per-segment
interpolation/edge splitting twice in two SQL dialects and keeping them
bit-identical was judged the bigger risk; this also makes the QuestDB
lag/lead spike moot.
Tests: 35 hand-computed TimeWeightedAggregator unit cases (each of 10
implementation mutations goes red); served-aggregate pin 13 -> 17;
conformance sparse fixture (intervals wider and narrower than sample gaps)
+ an outage straddling both window edges (each of the four SQL seeks
disabled in turn goes red); node-manager NodeId mapping test.
Joseph Doherty