dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

Gateway historian: compute per-sample aggregates (Average, Count, Start/End, Delta, PercentGood...) from raw reads #681

New Issue
Closed
opened 2026-10-11 15:23:24 -04:00 by dohertj2 · 2 comments
dohertj2 commented 2026-10-11 15:23:24 -04:00
Owner
Copy Link
Copy Source

Follow-up to #679. AVEVA's 15 retrieval modes include no arithmetic mean and no sample count, so since #679 the Gateway backend answers Average and Count with BadAggregateNotSupported (TimeAverage → TimeWeightedAverage is served instead).

Proposal: serve the per-sample aggregates (Average, Count, Minimum/Maximum incl. *ActualTime, Range, Start, End, Delta, PercentGood/PercentBad) on Gateway by reading raw samples (ReadRawAsync, capped like MaxTimeWeightedRawRows ⇒ BadTooManyOperations) and evaluating them with the shared Driver.Historian.Sql.Common evaluator (AggregateEvaluator, BucketPlan, AggregateFamilies) — not SQL-specific; Gateway already references Core.AlarmHistorian, Sql.Common's only extra dependency.

Gains: exact parity with SqlServer/QuestDB (incl. the #680 SDK-parity fixes), no reliance on unverified AVEVA mode semantics. Cost: raw-read volume per processed read.

Related Gateway differences to settle at the same time:

  • SampleMapper.ToAggregateSnapshot stamps buckets with the interval END (EndTime ?? StartTime); Part 13 / SQL backends stamp the interval start on forward reads.
  • Gateway status is plain Good/BadNoData — no Calculated bit, no Uncertain.
  • MinimumWithTime/MaximumWithTime rows lose the actual time of the extreme.

Needs the VPN to wonder-sql-vd03 for a live leg.

Follow-up to #679. AVEVA's 15 retrieval modes include no arithmetic mean and no sample count, so since #679 the Gateway backend answers `Average` and `Count` with `BadAggregateNotSupported` (TimeAverage → TimeWeightedAverage is served instead). Proposal: serve the per-sample aggregates (Average, Count, Minimum/Maximum incl. *ActualTime, Range, Start, End, Delta, PercentGood/PercentBad) on Gateway by reading raw samples (`ReadRawAsync`, capped like `MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations`) and evaluating them with the shared `Driver.Historian.Sql.Common` evaluator (`AggregateEvaluator`, `BucketPlan`, `AggregateFamilies`) — not SQL-specific; Gateway already references `Core.AlarmHistorian`, Sql.Common's only extra dependency. Gains: exact parity with SqlServer/QuestDB (incl. the #680 SDK-parity fixes), no reliance on unverified AVEVA mode semantics. Cost: raw-read volume per processed read. Related Gateway differences to settle at the same time: - `SampleMapper.ToAggregateSnapshot` stamps buckets with the interval END (`EndTime ?? StartTime`); Part 13 / SQL backends stamp the interval start on forward reads. - Gateway status is plain Good/BadNoData — no Calculated bit, no Uncertain. - `MinimumWithTime`/`MaximumWithTime` rows lose the actual time of the extreme. Needs the VPN to wonder-sql-vd03 for a live leg.
dohertj2 commented 2026-10-12 13:59:25 -04:00
Author
Owner
Copy Link
Copy Source

Implemented — merged 197246c3. The Gateway backend now serves the ten per-sample aggregates (Average, Count, Minimum, Maximum, Range, Start, End, Delta, MinimumActualTime, MaximumActualTime) by reading raw samples and evaluating them with the shared Driver.Historian.Sql.Common evaluator — the same code (and #680/#684 semantics) as SqlServer/QuestDB, incl. interval-start stamps, Calculated/Uncertain/Partial bits and #677 reverse reads. TimeAverage/Total stay native AVEVA modes (exact raw evaluation needs the real prior sample, which AVEVA replaces with an initial-value row); TimeAverage2/Total2/Interpolative/PercentGood/PercentBad stay BadAggregateNotSupported. Served set = GatewayHistorianDataSource.ServedAggregates (12), advertised via #682.

⚠️ Client-visible: Average/Count work again on Gateway (true mean / Good-sample count); Minimum/Maximum are now computed from raw instead of AVEVA MinimumWithTime/MaximumWithTime; per-sample reads stream raw rows, capped by MaxTimeWeightedRawRows ⇒ BadTooManyOperations (a month of 1 Hz data exceeds the default 1,000,000).

Design notes (two review rounds): earlier-data evidence = a pre-window row with a non-Bad OPC quality (AVEVA's no-data boundary row arrives WITH a value but Bad quality); every gateway stream is drained to completion (no early break — an abandoned stream evicts the gateway's shared historian session, filed upstream as historiangw#26); separate 2-row look-ahead and 1970 probe reads; gateway ceiling rejection ⇒ BadTooManyOperations; caller cancellation not counted as a backend fault.

Verification: parity suite (10 aggregates × 13 datasets × 21 forward/reverse windows, with and without the real AVEVA boundary-row shape) against an independent oracle; falsified 9 ways; live gate 9 G4 (unreachable gateway): all 12 served aggregates reach the backend, 25 unserved refused without a call, deadline cancel not counted.

Live AVEVA leg still open (VPN to wonder-sql-vd03): boundary-row quality with/without earlier data; no terminal row at the query end / past now; cheap 2-row reads from 1970 and to now+1 d; ceiling-rejection text; cap drain without session eviction; Min/Max vs the old native modes; TimeAverage/Total magnitude and the tail+43 interval-start stamp. Leaving open for that leg.

**Implemented — merged `197246c3`.** The Gateway backend now serves the ten per-sample aggregates (Average, Count, Minimum, Maximum, Range, Start, End, Delta, MinimumActualTime, MaximumActualTime) by reading raw samples and evaluating them with the shared `Driver.Historian.Sql.Common` evaluator — the same code (and #680/#684 semantics) as SqlServer/QuestDB, incl. interval-start stamps, Calculated/Uncertain/Partial bits and #677 reverse reads. TimeAverage/Total stay native AVEVA modes (exact raw evaluation needs the real prior sample, which AVEVA replaces with an initial-value row); TimeAverage2/Total2/Interpolative/PercentGood/PercentBad stay `BadAggregateNotSupported`. Served set = `GatewayHistorianDataSource.ServedAggregates` (12), advertised via #682. ⚠️ Client-visible: Average/Count work again on Gateway (true mean / Good-sample count); Minimum/Maximum are now computed from raw instead of AVEVA `MinimumWithTime`/`MaximumWithTime`; per-sample reads stream raw rows, capped by `MaxTimeWeightedRawRows` ⇒ `BadTooManyOperations` (a month of 1 Hz data exceeds the default 1,000,000). Design notes (two review rounds): earlier-data evidence = a pre-window row with a non-Bad OPC quality (AVEVA's no-data boundary row arrives WITH a value but Bad quality); every gateway stream is drained to completion (no early break — an abandoned stream evicts the gateway's shared historian session, filed upstream as historiangw#26); separate 2-row look-ahead and 1970 probe reads; gateway ceiling rejection ⇒ `BadTooManyOperations`; caller cancellation not counted as a backend fault. Verification: parity suite (10 aggregates × 13 datasets × 21 forward/reverse windows, with and without the real AVEVA boundary-row shape) against an independent oracle; falsified 9 ways; live gate 9 G4 (unreachable gateway): all 12 served aggregates reach the backend, 25 unserved refused without a call, deadline cancel not counted. **Live AVEVA leg still open (VPN to wonder-sql-vd03):** boundary-row quality with/without earlier data; no terminal row at the query end / past now; cheap 2-row reads from 1970 and to now+1 d; ceiling-rejection text; cap drain without session eviction; Min/Max vs the old native modes; TimeAverage/Total magnitude and the `tail+43` interval-start stamp. Leaving open for that leg.
dohertj2 commented 2026-10-12 14:21:07 -04:00
Author
Owner
Copy Link
Copy Source

Live AVEVA leg PASSED (2026-10-12, VPN, local HistorianGateway → wonder-sql-vd03, new known-value tag HistGW.LiveTest.Gate681).

Check Result
(1) window before the first sample: boundary row opc 0 / value 0 ⇒ not evidence; 1970 probe finds nothing; BadNoData before, start-of-data Partial PASS
(2) window after a Bad sample: boundary row copies the Bad quality; probe still finds earlier data PASS (safe)
(3) no terminal row at the query end or at now PASS
(4) 2-row reads from 1970 / to now+1 d PASS (121–171 ms)
(5) ceiling-rejection text "max_values exceeds the server ceiling" PASS (default mismatch filed as #688)
(6) over-cap window drained then refused; no session eviction, warm sessions steady PASS
(7) Minimum/Maximum from raw == AVEVA MinimumWithTime/MaximumWithTime incl. times PASS
(7) tail+43 = interval start PASS
Category=LiveIntegration suite 6/6 PASS

The raw-evaluated per-sample path is verified. Native TimeAverage/Total end to end FAIL for reasons outside this issue: the gateway's aggregate interval is 10,000× too large (historiangw#27) and OtOpcUa's native-row handling (#685). Other findings: #686 (reverse/end-only raw reads on Gateway), #687 (raw reads return AVEVA's synthetic boundary row). Closing #681.

**Live AVEVA leg PASSED (2026-10-12, VPN, local HistorianGateway → wonder-sql-vd03, new known-value tag `HistGW.LiveTest.Gate681`).** | Check | Result | |---|---| | (1) window before the first sample: boundary row opc 0 / value 0 ⇒ not evidence; 1970 probe finds nothing; BadNoData before, start-of-data Partial | PASS | | (2) window after a Bad sample: boundary row copies the Bad quality; probe still finds earlier data | PASS (safe) | | (3) no terminal row at the query end or at now | PASS | | (4) 2-row reads from 1970 / to now+1 d | PASS (121–171 ms) | | (5) ceiling-rejection text "max_values exceeds the server ceiling" | PASS (default mismatch filed as #688) | | (6) over-cap window drained then refused; no session eviction, warm sessions steady | PASS | | (7) Minimum/Maximum from raw == AVEVA MinimumWithTime/MaximumWithTime incl. times | PASS | | (7) `tail+43` = interval start | PASS | | `Category=LiveIntegration` suite | 6/6 PASS | The raw-evaluated per-sample path is verified. Native `TimeAverage`/`Total` end to end FAIL for reasons outside this issue: the gateway's aggregate interval is 10,000× too large (historiangw#27) and OtOpcUa's native-row handling (#685). Other findings: #686 (reverse/end-only raw reads on Gateway), #687 (raw reads return AVEVA's synthetic boundary row). Closing #681.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 cross-driver
queue management
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/mqtt
MQTT / Sparkplug B broker-subscribe driver
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#681
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?