dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

Gateway historian: strict lowered ceiling refuses ordinary pages; deep event reads slow (follow-ups from #689–#691 gate) #694

New Issue
Open
opened 2026-10-13 00:11:06 -04:00 by dohertj2 · 3 comments
dohertj2 commented 2026-10-13 00:11:06 -04:00
Owner
Copy Link
Copy Source

Current status (2026-10-13): items 1 and 2 are implemented and merged in PR #699 (618d98e8), with strict-ceiling paging and bounded deep event probes covered by regressions. Item 3 remains open: live native TimeAverage/Total forward/reverse, all-NULL and sub-ms clipped intervals require historiangw #27 verification after #29 retrieval recovery. Actual deployed OPC UA validation is tracked separately in #700. The event identity/received-time observation remains under historiangw #28 and SQL projection fault #39; sender identity fix #705 is merged, but native persistence is unverified.

Original findings:

Found by the #689–#691 live AVEVA gate (2026-10-13, master 942ca7bc). Low priority.

  1. Strict gateway with a lowered ceiling refuses ordinary page sizes (F3). With MaxValuesCeiling=1000, AllowStreamingBeyondCeiling=false, a capped raw read with N=5000 returns BadTooManyOperations. It could serve up to the learned ceiling and return a continuation point instead (a page smaller than N is legal in Part 11).
  2. Deep event reads (F4). The #691.5 "rest is empty" probe for events down to 1970 takes ~3.8 s live, and a deep events read without a source filter hit the gateway's 30 s SQL timeout. Consider bounding the events floor probe (e.g. the tag's/source's earliest event, or a configurable horizon) and checking the gateway's Runtime.dbo.Events index use.
  3. Native TimeAverage/Total re-check after historiangw#27 (carried over from #691 item 6): re-run forward and reverse native legs incl. the all-NULL interval and the sub-ms clipped interval once the gateway sends the interval in ms.

Gateway-side observation for historiangw (not OtOpcUa): 21 of 23 OtOpcUaLiveValidation events read back with an all-zero EventId, and receivedTime is ~4 h before eventTime.

Current status (2026-10-13): items 1 and 2 are implemented and merged in PR #699 (618d98e8), with strict-ceiling paging and bounded deep event probes covered by regressions. Item 3 remains open: live native TimeAverage/Total forward/reverse, all-NULL and sub-ms clipped intervals require historiangw #27 verification after #29 retrieval recovery. Actual deployed OPC UA validation is tracked separately in #700. The event identity/received-time observation remains under historiangw #28 and SQL projection fault #39; sender identity fix #705 is merged, but native persistence is unverified. Original findings: Found by the #689–#691 live AVEVA gate (2026-10-13, master `942ca7bc`). Low priority. 1. **Strict gateway with a lowered ceiling refuses ordinary page sizes (F3).** With `MaxValuesCeiling=1000`, `AllowStreamingBeyondCeiling=false`, a capped raw read with N=5000 returns `BadTooManyOperations`. It could serve up to the learned ceiling and return a continuation point instead (a page smaller than N is legal in Part 11). 2. **Deep event reads (F4).** The #691.5 "rest is empty" probe for events down to 1970 takes ~3.8 s live, and a deep events read without a source filter hit the gateway's 30 s SQL timeout. Consider bounding the events floor probe (e.g. the tag's/source's earliest event, or a configurable horizon) and checking the gateway's `Runtime.dbo.Events` index use. 3. **Native TimeAverage/Total re-check after historiangw#27** (carried over from #691 item 6): re-run forward and reverse native legs incl. the all-NULL interval and the sub-ms clipped interval once the gateway sends the interval in ms. Gateway-side observation for historiangw (not OtOpcUa): 21 of 23 `OtOpcUaLiveValidation` events read back with an all-zero EventId, and `receivedTime` is ~4 h before `eventTime`.
dohertj2 commented 2026-10-13 02:12:15 -04:00
Author
Owner
Copy Link
Copy Source

Items 1 and 2 fixed on master 66a2b0c3 (commits 3c3513ea, de8bc7a1). Live AVEVA gate PASS (2026-10-13). Item 3 stays open, blocked on historiangw#27 (still open), so this issue stays open for it.

1. Strict lowered ceiling.

  • IHistorianDataSource.MaxRawValuesPerRead (Gateway: learned ceiling − 16; 0 elsewhere). The node manager's RawPageLimit sizes every raw page so page + boundary ties + lookahead stay within it.
  • Capped reads above the ceiling, and uncapped N = 0 reads, now get a ceiling-sized page plus a continuation point instead of BadTooManyOperations.
  • The read that teaches the ceiling is re-planned once internally.
  • Live (ceiling 1,000, AllowStreamingBeyondCeiling=false): N=5000 and N=0, forward / reverse / end-only / from 1970, page at 983/982 rows with continuation points. Every result matched the streaming gateway's exactly. One learning refusal, then no raw request above 1,000.

2. Deep event reads. New opt-in, lossy ServerHistorian:EventReadHorizon (default 00:00:00 = off). When set, events older than now − horizon are never read; a window wholly older answers GoodNoData without contacting the gateway. Live: a 30-day horizon cut an end-only events read from 17.6 s to 1.57 s. No per-source earliest-event cache, by design: late store-and-forward drains would make it stale. Gateway-side Runtime.dbo.Events index use is for historiangw.

Found and fixed along the way (review fuzz):

  • Critical, pre-existing. A Gateway window starting at or before the Unix epoch counted AVEVA's synthetic row as "reached end", so a ceiling-truncated read came back Good with no continuation point: forward from 1970 = first page only; end-only past the data = the oldest rows presented as the latest.
  • Rows crowding the 1 ms before a page start no longer fail the page: it shrinks via PreWindowRows and retries once.
  • The #687 crowding re-read now doubles its room (up to 8 re-reads, capped at MaxTieClusterOverfetch), so a 200-row margin is served.
  • A cold reverse walk no longer re-runs at the full page size (was 200–3,000 gateway reads).

Tests: 30 GatewayCeilingPagingTests on a real SDK server plus the real Gateway source. Reviewer fuzz: ~24k drains (cold and warm, random ceilings, ceiling changes, ReturnBounds), 0 wrong results, 0 non-terminating.

Follow-ups: #695 (warm reads refuse a cluster that fits under the ceiling), #696 (AtTime future-time cost; strict-ceiling stopped-tag cost), #697 (backend outage answered BadHistoryOperationUnsupported), #698 (value types across backends).

Items 1 and 2 fixed on master `66a2b0c3` (commits `3c3513ea`, `de8bc7a1`). Live AVEVA gate PASS (2026-10-13). **Item 3 stays open, blocked on historiangw#27** (still open), so this issue stays open for it. **1. Strict lowered ceiling.** - `IHistorianDataSource.MaxRawValuesPerRead` (Gateway: learned ceiling − 16; 0 elsewhere). The node manager's `RawPageLimit` sizes every raw page so page + boundary ties + lookahead stay within it. - Capped reads above the ceiling, and uncapped `N = 0` reads, now get a ceiling-sized page plus a continuation point instead of `BadTooManyOperations`. - The read that teaches the ceiling is re-planned once internally. - Live (ceiling 1,000, `AllowStreamingBeyondCeiling=false`): N=5000 and N=0, forward / reverse / end-only / from 1970, page at 983/982 rows with continuation points. Every result matched the streaming gateway's exactly. One learning refusal, then no raw request above 1,000. **2. Deep event reads.** New opt-in, **lossy** `ServerHistorian:EventReadHorizon` (default `00:00:00` = off). When set, events older than now − horizon are never read; a window wholly older answers `GoodNoData` without contacting the gateway. Live: a 30-day horizon cut an end-only events read from 17.6 s to 1.57 s. No per-source earliest-event cache, by design: late store-and-forward drains would make it stale. Gateway-side `Runtime.dbo.Events` index use is for historiangw. **Found and fixed along the way (review fuzz):** - **Critical, pre-existing.** A Gateway window starting at or before the Unix epoch counted AVEVA's synthetic row as "reached end", so a ceiling-truncated read came back **Good with no continuation point**: forward from 1970 = first page only; end-only past the data = the *oldest* rows presented as the latest. - Rows crowding the 1 ms before a page start no longer fail the page: it shrinks via `PreWindowRows` and retries once. - The #687 crowding re-read now doubles its room (up to 8 re-reads, capped at `MaxTieClusterOverfetch`), so a 200-row margin is served. - A cold reverse walk no longer re-runs at the full page size (was 200–3,000 gateway reads). **Tests:** 30 `GatewayCeilingPagingTests` on a real SDK server plus the real Gateway source. Reviewer fuzz: ~24k drains (cold and warm, random ceilings, ceiling changes, ReturnBounds), 0 wrong results, 0 non-terminating. Follow-ups: #695 (warm reads refuse a cluster that fits under the ceiling), #696 (AtTime future-time cost; strict-ceiling stopped-tag cost), #697 (backend outage answered BadHistoryOperationUnsupported), #698 (value types across backends).
dohertj2 commented 2026-10-13 03:52:22 -04:00
Author
Owner
Copy Link
Copy Source

Rechecked remaining item 3: historiangw#27 is still open. The native TimeAverage/Total live rerun remains blocked on that upstream interval fix. Items 1/2 remain handled by base 66a2b0c3. Leaving this issue open; no upstream code changed.

Pushed commit 97668d26; PR: #699.

Full solution verified: 14,581 passed, 609 skipped, 0 failed across 62 suites (dotnet test ZB.MOM.WW.OtOpcUa.slnx --verbosity quiet, exit 0). Independent final review found no important defects. New fixes have not been deployed or live-tested end to end through OPC UA.

Rechecked remaining item 3: historiangw#27 is still open. The native TimeAverage/Total live rerun remains blocked on that upstream interval fix. Items 1/2 remain handled by base 66a2b0c3. Leaving this issue open; no upstream code changed. Pushed commit `97668d26`; PR: https://gitea.dohertylan.com/dohertj2/lmxopcua/pulls/699. Full solution verified: 14,581 passed, 609 skipped, 0 failed across 62 suites (`dotnet test ZB.MOM.WW.OtOpcUa.slnx --verbosity quiet`, exit 0). Independent final review found no important defects. New fixes have not been deployed or live-tested end to end through OPC UA.
dohertj2 commented 2026-10-13 04:49:11 -04:00
Author
Owner
Copy Link
Copy Source

PR #699 merged as 618d98e8; #653 and #695–#698 are closed. The native TimeAverage/Total live rerun remains open here, blocked on historiangw#27. Broader deployed endpoint verification is tracked by #700, with the existing-tag Raw control outage tracked by historiangw#29.

PR #699 merged as 618d98e8; #653 and #695–#698 are closed. The native TimeAverage/Total live rerun remains open here, blocked on historiangw#27. Broader deployed endpoint verification is tracked by #700, with the existing-tag Raw control outage tracked by historiangw#29.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
auto-managed
queue management
 cross-driver
queue management
 driver/abcip
queue management
 driver/ablegacy
queue management
 driver/focas
queue management
 driver/mqtt
MQTT / Sparkplug B broker-subscribe driver
 driver/opcuaclient
queue management
 driver/s7
queue management
 driver/twincat
queue management
 phase/1
queue management
 phase/2
queue management
 phase/3
queue management
 phase/4
queue management
 phase/5
queue management
 phase/6
queue management
 queue/blocked
queue management
 queue/done
queue management
 queue/failed
queue management
 queue/in-progress
queue management
 queue/queued
queue management
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
dohertj2 (dohertj2)
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: dohertj2/lmxopcua#694
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?