dohertj2/lmxopcua
1
0
Fork 0
Code Issues 32 Pull Requests Actions Packages Projects Releases Wiki Activity

fix(historian): #671 review — raw-row cap, reverse Interpolative, SDK status rule, Partial bit

Browse Source 
M1: the time-weighted family (incl. Total, formerly a pushed-down SUM) streams
raw window rows into the C# evaluator. New shared knob
ServerHistorian:MaxTimeWeightedRawRows (default 1,000,000; <=0 unbounded with a
Validate warning) flows through HistorianBackendRegistry into both providers.
The window query is capped at cap+1 (SQL Server TOP (@maxRows), QuestDB LIMIT $4);
one row past the cap throws the new Core.Abstractions
HistoryReadLimitExceededException, which the node manager maps to
BadTooManyOperations for that node. Not counted as a backend health failure;
bound rows are not counted; per-sample aggregates are not capped.

M2: reverse Interpolative is anchored at the later time (Part 13): points at
End - i*interval, emitted latest first. 10:00->09:00 @1h yields 10:00.

L1: Partial bit (0x4, = Opc.Ua.AggregateBits.Partial) on non-Bad results of a
truncated last interval (time-weighted family; per-sample family is a follow-up).

L2/L3: aligned with the SDK AggregateCalculator.GetTimeBasedStatusCode
(decompiled from Opc.Ua.Server 1.5.378.106): TreatUncertainAsBad=false counts
Uncertain time (Uncertain samples, held-into-outage, extrapolation) as Good
time; data but no non-Bad time => generic Bad (0x80000000); BadNoData only
when there is no data at all. Consequence: a live tag's latest interval reads
Good (documented).

L5: release note no longer claims Total matches the Gateway Integral exactly
(AVEVA's divisor is unverified); it notes Total is now evaluated in C# and
subject to the cap.

Tests: unit (reverse Interpolative x3, Partial, Bad vs BadNoData incl. an
open-ended outage, cap helpers, SDK bit oracles); conformance (reverse
Interpolative, raw-row cap at/over/0, bounds not counted, health unaffected);
node manager BadTooManyOperations; options default + warning. Each new
behaviour falsified by mutation (aggregator x9, data-source cap checks on
both backends against the real DBs, node-manager catch).
This commit is contained in:
Joseph Doherty
2026-10-11 09:10:26 -04:00
parent 1f88669304
commit 56bb4855bb
24 changed files with 460 additions and 86 deletions
Download Patch File Download Diff File Expand all files Collapse all files
src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common/TimeWeightedAggregator.cs
+103 -37
View File
@@ -39,17 +39,23 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common;
/// <item><term>Total</term><description><c>TimeAverage × interval seconds</c> (value·s) — Part 13 §5.4.3.8.</description></item>
/// <item><term>TimeAverage2</term><description>∫ simple-bounding signal over its non-Bad parts ÷ non-Bad duration (Bad intervals excluded).</description></item>
/// <item><term>Total2</term><description>∫ simple-bounding signal over its non-Bad parts, in value·s (= TimeAverage2 × non-Bad seconds).</description></item>
/// <item><term>Interpolative</term><description>The interpolated-bounding signal's value at each interval start.</description></item>
/// <item><term>Interpolative</term><description>The interpolated-bounding signal's value at each interval start —
/// for a reverse read, at <c>End − i·interval</c> (Part 13 anchors a reverse read at its later time), emitted
/// latest first.</description></item>
/// </list>
/// <para>
/// <b>Status (time-weighted four).</b> No non-Bad time in the interval ⇒ <c>BadNoData</c>; the whole interval
/// Good ⇒ <c>Good</c>; otherwise <c>UncertainDataSubNormal</c> (PercentDataGood = PercentDataBad = 100). A
/// usable String/DateTime sample contributing to the interval ⇒ <c>BadAggregateInvalidInputs</c>. Every status
/// carries the Calculated bit; the timestamp is the interval start; the value is a <see cref="double"/>
/// (Boolean = 0/1, Int64 as double).
/// <b>Status (time-weighted four)</b> — the SDK <c>AggregateCalculator.GetTimeBasedStatusCode</c> rule with
/// TreatUncertainAsBad = false, so Uncertain time (Uncertain samples, a value held up to an outage, stepped
/// extrapolation past the last sample) counts as Good time: every instant of the interval non-Bad ⇒
/// <c>Good</c>; no non-Bad time but some Bad (outage) time ⇒ <c>Bad</c>; no data at all ⇒ <c>BadNoData</c>;
/// otherwise (PercentDataGood = PercentDataBad = 100) <c>UncertainDataSubNormal</c>. Time before the first sample
/// counts as Bad (no data) time. A usable String/DateTime sample contributing to the interval ⇒
/// <c>BadAggregateInvalidInputs</c>. Every status carries the Calculated bit, plus the Partial bit on a non-Bad
/// result of a truncated last interval; the timestamp is the interval start; the value is a
/// <see cref="double"/> (Boolean = 0/1, Int64 as double).
/// </para>
/// <para>
/// <b>Status (Interpolative).</b> A usable sample exactly at the interval start ⇒ its value with its own status
/// <b>Status (Interpolative).</b> A usable sample exactly at the point ⇒ its value with its own status
/// (raw, no historian bit). Otherwise no usable sample at or before ⇒ <c>BadNoData</c>; else the interpolated
/// value with the Interpolated bit and the status of the simple-bounding signal at that instant: <c>Good</c> on
/// a Good segment, <c>UncertainDataSubNormal</c> on an Uncertain one, inside an outage (interpolated across it)
@@ -60,26 +66,54 @@ public sealed class TimeWeightedAggregator
{
private const double TicksPerSecond = TimeSpan.TicksPerSecond;
/// <summary>
/// Default cap on the raw in-window rows one time-weighted read may stream (<c>ServerHistorian:MaxTimeWeightedRawRows</c>).
/// </summary>
public const int DefaultMaxWindowRows = 1_000_000;
/// <summary>
/// The <c>TOP</c>/<c>LIMIT</c> a backend puts on its window query for a configured cap: <c>cap + 1</c> (so one
/// row past the cap proves it was exceeded without reading on), or <see cref="int.MaxValue"/> when the cap is
/// &lt;= 0 (unbounded).
/// </summary>
/// <param name="maxWindowRows">The configured cap.</param>
/// <returns>The row limit to put on the window query.</returns>
public static int WindowFetchLimit(int maxWindowRows) =>
maxWindowRows <= 0 || maxWindowRows == int.MaxValue ? int.MaxValue : maxWindowRows + 1;
/// <summary>Throws when <paramref name="rowsRead"/> has passed a positive <paramref name="maxWindowRows"/>.</summary>
/// <param name="rowsRead">Window rows read so far.</param>
/// <param name="maxWindowRows">The configured cap (&lt;= 0 = unbounded).</param>
/// <param name="tag">The tag, for the message.</param>
/// <exception cref="HistoryReadLimitExceededException">The cap is exceeded.</exception>
public static void CheckWindowRows(long rowsRead, int maxWindowRows, string tag)
{
if (maxWindowRows > 0 && rowsRead > maxWindowRows)
throw new HistoryReadLimitExceededException(maxWindowRows,
$"The time-weighted aggregate read of '{tag}' covers more than {maxWindowRows} raw samples " +
"(ServerHistorian:MaxTimeWeightedRawRows); narrow the window or use a pushed-down aggregate such as Average.");
}
private readonly HistoryAggregateType _aggregate;
private readonly BucketPlan _plan;
private readonly long _start;
private readonly long _end;
// Simple-bounding (raw) signal, per bucket: ∫ over non-Bad time, non-Bad ticks, Good ticks.
// Simple-bounding (raw) signal, per bucket: ∫ over non-Bad time, non-Bad ticks, ticks inside an outage.
private readonly double[] _rawArea;
private readonly long[] _usableTicks;
private readonly long[] _goodTicks;
private readonly long[] _badTicks;
// Interpolated-bounding (bridged) signal, per bucket.
private readonly double[] _bridgedArea;
private readonly long[] _bridgedTicks;
private readonly bool[] _invalid;
// Interpolative, per bucket start.
// Interpolative, per point (index = ascending time; see PointTicks).
private readonly DataValueSnapshot?[] _points;
private StoredSample? _prevAny;
private StoredSample? _prevUsable;
private long _nextPoint; // next bucket start not yet classified
private readonly List<PendingPoint> _pending = []; // bucket starts awaiting the next usable sample
private long _nextPoint; // next point (ascending time) not yet classified
private readonly List<PendingPoint> _pending = []; // points awaiting the next usable sample
private bool _finished;
/// <summary>
@@ -106,7 +140,7 @@ public sealed class TimeWeightedAggregator
var n = (int)plan.BucketCount;
_rawArea = new double[n];
_usableTicks = new long[n];
_goodTicks = new long[n];
_badTicks = new long[n];
_bridgedArea = new double[n];
_bridgedTicks = new long[n];
_invalid = new bool[n];
@@ -142,15 +176,13 @@ public sealed class TimeWeightedAggregator
// Simple-bounding segment [prevAny, s).
if (_prevAny is { IsUsable: true } p)
{
if (s.IsUsable)
{
var sloped = IsSloped(p, s);
AddRaw(p, t, sloped ? s.Value.Number : p.Value.Number, sloped ? p.IsGood && s.IsGood : p.IsGood);
}
else
{
AddRaw(p, t, p.Value.Number, good: false); // held up to the outage
}
// Toward a usable sample: sloped or stepped; toward a Bad one: held up to the outage.
var sloped = s.IsUsable && IsSloped(p, s);
AddRaw(p, t, sloped ? s.Value.Number : p.Value.Number);
}
else if (_prevAny is { } bad)
{
AddBad(bad, t);
}
if (!s.IsUsable) DegradePending(); // the segment the pending points sit on runs into an outage
@@ -179,7 +211,9 @@ public sealed class TimeWeightedAggregator
_finished = true;
ClassifyPointsBefore(long.MaxValue);
if (_prevAny is { IsUsable: true } p && p.TimeUtc.Ticks < _end)
AddRaw(p, _end, p.Value.Number, good: false); // stepped extrapolation
AddRaw(p, _end, p.Value.Number); // stepped extrapolation
else if (_prevAny is { } bad && bad.TimeUtc.Ticks < _end)
AddBad(bad, _end); // the outage runs to the window end
if (_prevUsable is { } pu && pu.TimeUtc.Ticks < _end)
AddBridged(pu, _end, pu.Value.Number);
ResolvePending(_prevUsable, next: null);
@@ -195,20 +229,26 @@ public sealed class TimeWeightedAggregator
private DataValueSnapshot Result(long i, DateTime now)
{
var bucketStart = _plan.BucketStart(i);
if (_aggregate == HistoryAggregateType.Interpolative)
return _points[i] is { } point
? point with { ServerTimestampUtc = now }
: new DataValueSnapshot(null, HistorianStatusCodes.BadNoData, bucketStart, now);
: new DataValueSnapshot(null, HistorianStatusCodes.BadNoData, PointTime(i), now);
DataValueSnapshot Calc(object? value, uint status) =>
new(value, HistorianInfoBits.WithCalculated(status), bucketStart, now);
var bucketStart = _plan.BucketStart(i);
var duration = BucketEnd(i) - BucketStartTicks(i);
var partial = duration < _plan.IntervalTicks; // only the truncated last interval
DataValueSnapshot Calc(object? value, uint status)
{
var code = HistorianInfoBits.WithCalculated(status);
if (partial && !StatusBits.IsBad(status)) code |= HistorianInfoBits.HistorianPartial;
return new DataValueSnapshot(value, code, bucketStart, now);
}
if (_invalid[i]) return Calc(null, HistorianStatusCodes.BadAggregateInvalidInputs);
if (_usableTicks[i] == 0) return AggregateEvaluator.NoData(bucketStart, now);
if (_usableTicks[i] == 0)
return _badTicks[i] > 0 ? Calc(null, HistorianStatusCodes.Bad) : AggregateEvaluator.NoData(bucketStart, now);
var duration = BucketEnd(i) - BucketStartTicks(i);
var status = _goodTicks[i] == duration ? HistorianStatusCodes.Good : HistorianStatusCodes.UncertainDataSubNormal;
var status = _usableTicks[i] == duration ? HistorianStatusCodes.Good : HistorianStatusCodes.UncertainDataSubNormal;
return _aggregate switch
{
HistoryAggregateType.TimeAverage => Calc(_bridgedArea[i] / _bridgedTicks[i], status),
@@ -233,14 +273,29 @@ public sealed class TimeWeightedAggregator
private long BucketEnd(long i) => Math.Min(BucketStartTicks(i) + _plan.IntervalTicks, _end);
private void AddRaw(StoredSample from, long toTicks, double toValue, bool good) =>
private void AddRaw(StoredSample from, long toTicks, double toValue) =>
Distribute(from, toTicks, toValue, (i, area, ticks) =>
{
_rawArea[i] += area;
_usableTicks[i] += ticks;
if (good) _goodTicks[i] += ticks;
});
/// <summary>Counts <c>[from.t, toTicks)</c> as outage (Bad) time; no value, never marks a bucket invalid.</summary>
private void AddBad(StoredSample from, long toTicks)
{
var lo = Math.Max(from.TimeUtc.Ticks, _start);
var hi = Math.Min(toTicks, _end);
if (lo >= hi || _plan.BucketCount == 0) return;
for (var i = (lo - _start) / _plan.IntervalTicks; i < _plan.BucketCount; i++)
{
var bs = BucketStartTicks(i);
if (bs >= hi) break;
var a = Math.Max(lo, bs);
var b = Math.Min(hi, BucketEnd(i));
if (a < b) _badTicks[i] += b - a;
}
}
private void AddBridged(StoredSample from, long toTicks, double toValue) =>
Distribute(from, toTicks, toValue, (i, area, ticks) =>
{
@@ -279,13 +334,24 @@ public sealed class TimeWeightedAggregator
// ------------------------------------------------------------------------------------------ Interpolative
/// <summary>Classifies every unclassified bucket start strictly before <paramref name="beforeTicks"/>.</summary>
/// <summary>
/// The time of Interpolative point <paramref name="j"/> (ascending time). Forward: the bucket starts
/// <c>start + j·interval</c>. Reverse (Part 13 anchors a reverse read at its later time, the request's start):
/// <c>End − (n−1−j)·interval</c>, i.e. <c>End, End − interval, …</c> once <see cref="BucketPlan.Assemble"/> has
/// reversed the order — so a reverse read 10:00 → 09:00 at 1 h yields the value at 10:00.
/// </summary>
private long PointTicks(long j) =>
_plan.Reversed ? _end - (_plan.BucketCount - 1 - j) * _plan.IntervalTicks : BucketStartTicks(j);
private DateTime PointTime(long j) => new(PointTicks(j), DateTimeKind.Utc);
/// <summary>Classifies every unclassified point strictly before <paramref name="beforeTicks"/>.</summary>
private void ClassifyPointsBefore(long beforeTicks)
{
if (_aggregate != HistoryAggregateType.Interpolative) return;
for (; _nextPoint < _plan.BucketCount; _nextPoint++)
{
var a = BucketStartTicks(_nextPoint);
var a = PointTicks(_nextPoint);
if (a >= beforeTicks) break;
if (_prevAny is not { } p)
@@ -296,7 +362,7 @@ public sealed class TimeWeightedAggregator
{
// A usable sample exactly at the interval start: its own value and status.
_points[_nextPoint] = IsNumeric(p)
? new DataValueSnapshot(p.Value.Number, p.Status, _plan.BucketStart(_nextPoint), default)
? new DataValueSnapshot(p.Value.Number, p.Status, PointTime(_nextPoint), default)
: InvalidPoint(_nextPoint);
}
else if (!p.IsUsable)
@@ -318,7 +384,7 @@ public sealed class TimeWeightedAggregator
if (_pending.Count == 0) return;
foreach (var pt in _pending)
{
var bucketStart = _plan.BucketStart(pt.Bucket);
var bucketStart = PointTime(pt.Bucket);
if (p is not { } prev || pt.Quality == PointQuality.NoPrior)
{
_points[pt.Bucket] = new DataValueSnapshot(null, HistorianStatusCodes.BadNoData, bucketStart, default);
@@ -360,5 +426,5 @@ public sealed class TimeWeightedAggregator
}
private DataValueSnapshot InvalidPoint(long bucket) =>
new(null, HistorianStatusCodes.BadAggregateInvalidInputs, _plan.BucketStart(bucket), default);
new(null, HistorianStatusCodes.BadAggregateInvalidInputs, PointTime(bucket), default);
}