dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

Time-weighted aggregates: forward differences from the SDK AggregateCalculator (TimeAverage/Total status, Total at start of data, Partial on truncated) #680

New Issue
Closed
opened 2026-10-11 11:26:40 -04:00 by dohertj2 · 1 comment
dohertj2 commented 2026-10-11 11:26:40 -04:00
Owner
Copy Link
Copy Source

Found by the in-process SDK parity test added for #677 (SdkAggregateParityTests, which drives the OPC UA .NET Standard SDK's own AggregateCalculators against the SQL backends' results). Reverse anchoring now matches the SDK; these forward differences pre-date #677 and are masked in the parity test and documented in docs/Historian.md:

  1. TimeAverage / Total status. The SDK reports Uncertain whenever any part of the interval is not Good; the SQL backends apply the time-based rule (Uncertain/Bad time vs Good time), so an interval bridged across a Bad sample reads Good. The #671 docs claimed this matched the SDK — it only does for TimeAverage2 / Total2.
  2. Total when the data starts inside the interval. The SDK integrates only the covered part; the SQL backends compute TimeAverage × interval.
  3. Partial on truncated intervals. The SQL backends set it; the SDK does not.

Smaller, also masked: reverse Interpolative exactly at a sample (SDK flags Interpolated, we return the raw sample); per-sample Count on an empty interval (SDK 0/Good, counts Good samples only); the SDK's forward end-of-data Partial rule is not implemented.

Decide per item: align with the SDK (client-visible status/value change on SqlServer/QuestDB) or keep and document as an intentional deviation.

Found by the in-process SDK parity test added for #677 (`SdkAggregateParityTests`, which drives the OPC UA .NET Standard SDK's own `AggregateCalculator`s against the SQL backends' results). Reverse anchoring now matches the SDK; these **forward** differences pre-date #677 and are masked in the parity test and documented in `docs/Historian.md`: 1. **TimeAverage / Total status.** The SDK reports Uncertain whenever any part of the interval is not Good; the SQL backends apply the time-based rule (Uncertain/Bad time vs Good time), so an interval bridged across a Bad sample reads Good. The #671 docs claimed this matched the SDK — it only does for TimeAverage2 / Total2. 2. **Total when the data starts inside the interval.** The SDK integrates only the covered part; the SQL backends compute TimeAverage × interval. 3. **Partial on truncated intervals.** The SQL backends set it; the SDK does not. Smaller, also masked: reverse Interpolative exactly at a sample (SDK flags Interpolated, we return the raw sample); per-sample Count on an empty interval (SDK 0/Good, counts Good samples only); the SDK's forward end-of-data Partial rule is not implemented. Decide per item: align with the SDK (client-visible status/value change on SqlServer/QuestDB) or keep and document as an intentional deviation.
dohertj2 commented 2026-10-12 08:38:37 -04:00
Author
Owner
Copy Link
Copy Source

Fixed — merged f6a26286 (time-weighted lane), 11f79253 (per-sample lane), d266ff83 (live-gate fix). Part 13 is treated as authoritative; the SDK AggregateCalculator (1.5.378, decompiled) is the parity reference where it agrees.

Aligned (client-visible on SqlServer / QuestDB):

  1. TimeAverage/Total status: Uncertain if any part of the interval uses an Uncertain value, is bridged across a Bad sample, is extrapolated past the last sample (⇒ a live tag's latest interval is Uncertain) or is not fully covered. The #671 docs' parity claim is corrected.
  2. Total with data starting inside the interval integrates only the covered part.
  3. (new) TimeAverage/Total over an interval wholly inside an outage: bridged value, Uncertain (was Bad/null).
  4. PercentGood/PercentBad: duration-based (stepped regions), moved to the time-weighted family ⇒ now subject to MaxTimeWeightedRawRows.
  5. (new) TimeAverage2/Total2 never extrapolate past the last sample: that interval Partial, later ones BadNoData; a recovery value after an outage marker / a tag's only sample ⇒ its value, Uncertain + Partial.
  6. Count: Good samples only, value-based status; an empty interval inside the tag's data range is 0/Good.
  7. Start/End: the raw first/last sample of any quality (own timestamp, status, null value if Bad). Delta: Uncertain only if the first/last sample is unusable.
  8. Forward/reverse end-of-data Partial on the interval holding the last sample (every per-sample aggregate except Average; the time-weighted family per the SDK).

Kept, documented as intentional: Partial on truncated intervals (spec §5.3.3.2 Case 3 requires it; SDK non-conformant); reverse Interpolative exactly at a sample (SDK asymmetry); Partial never on Bad results; TimeAverage2/Total2 all-outage interval generic Bad (the docs' "SDK all-Bad result" wording was wrong); outage marker as last sample runs Bad to window end; Boolean stepped values; Min/Max-at-interval-start Raw vs Calculated bit; integer tags keep Double interpolation.

⚠️ Report-by-exception consequence: a tag with nothing after its last sample reads BadNoData for TA2/Total2/PercentGood/PercentBad and Uncertain for TA/Total in its trailing intervals; they change on re-read once a new sample arrives. Caching consumers should re-read recent intervals.

Verification: SdkAggregateParityTests masks removed (new datasets: EndsInside, OutageInside, OutageLast, RecoveryEnds, BoolSwitch, Gappy, Bracketed, EarlyEnd, EndOnEdge, OutageEnds); every production change falsified; full suite green. Live gate 8 (docker-dev, both backends, central-1/2): D1 seeded edge cases vs the SDK calculator — every mismatch is a documented deviation; D2 live tag — SqlServer/QuestDB identical on 434 result rows; D3 percent row cap ⇒ BadTooManyOperations; D4 #679.

Gate defect F1 (fixed 2ceed561): on QuestDB the new Start/End/Delta query referenced its CTE 4 times; a row committed between readers left a null first/last ⇒ InvalidCastException ⇒ the node's read failed (~1 in 17k–90k reads of a tag being written). Now one GROUP BY scan (QuestDB) / one windowed pass (SqlServer), and the bucket readers treat a missing pick as "no sample". Opt-in stress test (HISTORIAN_QUESTDB_STRESS_SECONDS): old code 6 faults / ~105k reads, new 0 / 106k.

Follow-ups: #683 (QuestDB data-range lookup is a partition scan; SqlServer StartEnd cost), #684 (Average/Min/Max include Uncertain samples; spec/SDK use Good only).

**Fixed — merged `f6a26286` (time-weighted lane), `11f79253` (per-sample lane), `d266ff83` (live-gate fix).** Part 13 is treated as authoritative; the SDK `AggregateCalculator` (1.5.378, decompiled) is the parity reference where it agrees. **Aligned (client-visible on SqlServer / QuestDB):** 1. TimeAverage/Total status: Uncertain if any part of the interval uses an Uncertain value, is bridged across a Bad sample, is extrapolated past the last sample (⇒ a live tag's latest interval is Uncertain) or is not fully covered. The #671 docs' parity claim is corrected. 2. Total with data starting inside the interval integrates only the covered part. 3. (new) TimeAverage/Total over an interval wholly inside an outage: bridged value, Uncertain (was Bad/null). 4. PercentGood/PercentBad: duration-based (stepped regions), moved to the time-weighted family ⇒ now subject to `MaxTimeWeightedRawRows`. 5. (new) TimeAverage2/Total2 never extrapolate past the last sample: that interval Partial, later ones `BadNoData`; a recovery value after an outage marker / a tag's only sample ⇒ its value, Uncertain + Partial. 6. Count: Good samples only, value-based status; an empty interval inside the tag's data range is `0`/Good. 7. Start/End: the raw first/last sample of any quality (own timestamp, status, null value if Bad). Delta: Uncertain only if the first/last sample is unusable. 8. Forward/reverse end-of-data Partial on the interval holding the last sample (every per-sample aggregate except Average; the time-weighted family per the SDK). **Kept, documented as intentional:** Partial on truncated intervals (spec §5.3.3.2 Case 3 requires it; SDK non-conformant); reverse Interpolative exactly at a sample (SDK asymmetry); Partial never on Bad results; TimeAverage2/Total2 all-outage interval generic Bad (the docs' "SDK all-Bad result" wording was wrong); outage marker as last sample runs Bad to window end; Boolean stepped values; Min/Max-at-interval-start Raw vs Calculated bit; integer tags keep Double interpolation. ⚠️ Report-by-exception consequence: a tag with nothing after its last sample reads `BadNoData` for TA2/Total2/PercentGood/PercentBad and Uncertain for TA/Total in its trailing intervals; they change on re-read once a new sample arrives. Caching consumers should re-read recent intervals. **Verification:** `SdkAggregateParityTests` masks removed (new datasets: EndsInside, OutageInside, OutageLast, RecoveryEnds, BoolSwitch, Gappy, Bracketed, EarlyEnd, EndOnEdge, OutageEnds); every production change falsified; full suite green. **Live gate 8** (docker-dev, both backends, central-1/2): D1 seeded edge cases vs the SDK calculator — every mismatch is a documented deviation; D2 live tag — SqlServer/QuestDB identical on 434 result rows; D3 percent row cap ⇒ BadTooManyOperations; D4 #679. **Gate defect F1 (fixed `2ceed561`):** on QuestDB the new Start/End/Delta query referenced its CTE 4 times; a row committed between readers left a null first/last ⇒ `InvalidCastException` ⇒ the node's read failed (~1 in 17k–90k reads of a tag being written). Now one GROUP BY scan (QuestDB) / one windowed pass (SqlServer), and the bucket readers treat a missing pick as "no sample". Opt-in stress test (`HISTORIAN_QUESTDB_STRESS_SECONDS`): old code 6 faults / ~105k reads, new 0 / 106k. Follow-ups: #683 (QuestDB data-range lookup is a partition scan; SqlServer StartEnd cost), #684 (Average/Min/Max include Uncertain samples; spec/SDK use Good only).
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 cross-driver
queue management
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/mqtt
MQTT / Sparkplug B broker-subscribe driver
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#680
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?