F1 (pre-existing): SliceTieCluster continued a forward read while next <= endUtc, but the
forward window is end-exclusive; an oversized cluster one step below EndTime resumed AT
EndTime, the resume became a start == end point read and returned the EndTime samples as
data (microsecond: EndTime 8.1 us truncated to 8.0). Continue only while next is strictly
before EndTime truncated to the backend resolution.
Tests: A_tie_cluster_just_below_the_end_time_never_resumes_onto_the_end_time (native +
microsecond; both fail on the old rule) and
A_cursor_inside_a_tie_cluster_over_fetches_without_a_normal_read_first (pins the
WithinTieCluster shortcut by per-page read count; fails with the shortcut disabled).
Docs: the oversized-cluster exception to 'every page full / no lookahead => last' (F2),
and the N + skip + 1 strict-ceiling edge note referencing #694 (F4).
A resumed raw page asked the backend for only N rows from the boundary timestamp
(inclusive), then trimmed the boundary ties already emitted: it netted N - skip values,
and at N = 1 nothing — the empty "full" page was taken for a stalled tie cluster and a
MaxTieClusterOverfetch + 1 (65,537-row) over-fetch stepped one tick, returning an EMPTY
page with a continuation point between every value (8 values = 17 pages / 25 reads),
forward and reverse, with or without ReturnBounds, on every backend.
Each page's read now asks for N + boundarySkip + 1 rows: the ties it trims, the page and
one lookahead row. The lookahead row is the 'more data' signal, so an exact multiple ends
on its last full page (no trailing empty page). The over-fetch fires only when every row
past the trimmed ties is still on the boundary (a genuine cluster larger than the page);
a cursor left inside a cluster is flagged WithinTieCluster and over-fetches directly.
A page that never leaves the boundary adds the prior skip to its cursor.
Low items: BoundedReadHasAnythingAsync carries its walk hint and, when there is no data,
the end bound it found (HistoryEndBound.Resolved + DataExhausted), so the next page reads
nothing; HistoryBounds.IsPlaceholder is an explicit marker (ConditionalWeakTable over the
NotFound instances) instead of status+null shape, so a stored Bad_BoundNotFound row is data.
Tests: NodeManagerHistoryReadPageFillTests (N=1 every shape/direction/bounds, tie clusters
straddling boundaries at N=1/2/3, exact-multiple, carried end bound + walk hint, stored
placeholder-shaped row). Existing paging tests updated where they encoded the short pages.
The node manager composes the start/end bounds from each backend's existing raw
reads (one authority, HistoryBounds): at-or-before = a capped reverse read
(native, or ReverseRawReader's walk-back on a forward-only backend — never
AVEVA's synthetic boundary row), at-or-after = a capped forward read. Bounds
count toward NumValuesPerNode, the start bound leads the first page, the end
bound ends the last (carried in the continuation point). Missing bound =
Bad_BoundNotFound (null, at the bound time; one-sided reads ±1 s from the
previous value, Table 1 footnotes a/b). A modified read with ReturnBounds is
Bad_InvalidArgument (Part 11 6.5.3.3).
The .NET SDK's ReadRawModifiedDetails defaults ReturnBounds to TRUE; the
existing tests built details without setting it and now set it false.
Oracle: every row of Part 11 Table 1 over three backend shapes, plus paging
concatenation, ties, Bad bounds, microsecond truncation; conformance pins the
lookup primitives on SqlServer + QuestDB.
The last pages of a deep reverse page-through on a forward-only backend
each walked down to 1970 one x4 growth step at a time (~14-16 reads,
~2.5 s live). After two empty segments the walk now asks once per page
for the whole remainder at the page budget (a truncated answer is
discarded and the walk carries on), and the stretch the page's last
complete read proved empty rides the continuation point
(HistoryContinuationState.EmptyFromTicks/EmptyToTicks, session-held, so
no versioning) so the next page steps over it. Fuzz now pages with the
walk's own hints and with true random empty stretches.
1. Uncapped reverse raw read on a forward-only backend = ONE forward read of the
window, filtered and reversed in memory (was a segment-by-segment walk).
2. Segment width is aimed at ~half the row budget from the density just observed
(overflow: from the truncated read's own sample span) and carried page to page
in HistoryContinuationState.SegmentWidthTicks.
3. The walk starts no later than now + 1 day and never reads before the epoch.
4. Request bounds are rounded outward to the millisecond, so a row AVEVA
fabricates at a rounded-up query start cannot land inside the segment.
5. Budget M = max(1, MaxTieClusterOverfetch); reads ask M + 2 rows, completeness is
'short OR last row past the segment', and a read crowded by rows before the
segment is re-read once (<= 2M + 4). A cluster of exactly M, neighbouring
clusters each within M, and M <= 1 are all served; only one timestamp over M
is refused. ServerHistorianOptions text updated.
6. The raw and events tie-path oversized-cluster refusal is BadTooManyOperations
(was BadHistoryOperationUnsupported), consistent with the walk.
7. GatewayHistorianDataSource.ReadRawAsync: a caller cancellation is not a backend
failure; NumValuesPerNode = 0 is sent as int.MaxValue (the client package and
the gateway reject max_values <= 0, so every uncapped Gateway raw read failed);
a MaxValuesCeiling refusal is a HistoryReadLimitExceededException.
8. Tests: upper-edge filter, epoch floor, look-ahead, ms alignment, exact/adjacent
clusters, minimal budget, carried density, and the walk over the real Gateway
source + FakeHistorianGatewayClient (OpcUaServer internals visible to it).
- IHistorianDataSource.ReadEventsAsync(..., bool reverse, ...) overload: reverse = Part 11 window (lo, hi]
newest-first by (time, id) DESC. Default forwards forward reads and throws NotSupportedException for
reverse (Null); Gateway refuses explicitly (ascending-only stream). Server answers
BadHistoryOperationUnsupported instead of silently returning the earliest N ascending.
- SqlServer / QuestDB serve reverse natively (ORDER BY time DESC, id DESC + TOP/LIMIT).
- Node manager: reverse event reads page with continuation points; direction stored in the cursor
(HistoryContinuationState.Reverse) so a resume continues in the cursor's direction; reverse tie clusters
over-fetched forward and sliced from the newest end (SliceTieCluster(reverse:)); end-only reads with a
cap page backward from EndTime (mirrors raw).
- Resumed event pages over-fetch by the boundary skip, so every non-final page is a full cap page.
- Tests: node-manager reverse paging (ties, clusters, coarse resolution, overfetch bound, resume
direction both ways, end-only, unsupported backend); shared EventReadConformance on SqlServer +
QuestDB; over-the-wire reverse paging; offline Gateway pin of the UTC DateTime.MaxValue bound through
the published client's proto conversion (in-process gRPC capture) and FILETIME.
- docs/Historian.md event paging section.
Every HistoryUpdate a SQL backend applies now leaves a modification trail
(prior value, principal, Part 11 update type, modification time), and
HistoryReadRaw with IsReadModified=true serves it as HistoryModifiedData.
- Core: HistoryUpdateContext threaded into all three IHistorianDataUpdater
methods; IHistorianDataSource.ReadModifiedAsync (default NotSupported =>
BadHistoryOperationUnsupported on Gateway/Null); HistoryModified* contracts.
- SqlServer: schema v2 (002_sample_modified.sql, RequiredVersion=2, docs copy);
trail written by OUTPUT ... INTO on the same MERGE/DELETE statement, so a
failed update leaves no audit row; retention purges SampleModified per tag on
the source-time cutoff; ReadModifiedAsync keyed on (SourceTimeUtc, ModifiedId).
- QuestDB: append-only {prefix}sample_modified (TTL like samples), trail row
written BEFORE the change (orphan-on-crash, never an untracked change);
existence reads now carry the prior value; ReadModifiedAsync.
- Node manager: modified reads dispatch + page via the #669 continuation store
(HistoryContinuationState gains HistoryCursorKind + LastModifiedId), gated by
the existing HistoryRead decision; session principal / key id passed to the
updater. IsDeleteModified stays unsupported.
- Client.CLI: historyread --modified.
- Tests: node-manager dispatch/paging/cursor binding/principal threading,
over-the-wire HistoryModifiedData, shared ModifiedValueConformance on both
backends, SqlServer atomicity (forced trail failure => no change) + v1->v2
upgrade + retention trail purge.
- Docs: Historian.md (Modified-value history section, schema v2 upgrade note),
security.md, Client.CLI.md, CLAUDE.md.
A capped events read now pages: the source's truncation flag triggers a
session-bound cursor (time inclusive + tie skip), oversized tie clusters
are paged within via a bounded half-open over-fetch, and release/unknown/
cross-kind points behave as for Raw.
IHistorianDataSource.TimestampResolutionTicks (default 1; QuestDB 10). A drained tie
cluster now steps off T by the resolution, so a forward one-value-per-page read on a
microsecond backend no longer re-reads the same sample forever (C1). QuestDB AtTime
results carry the client's original request time (M3); regression for the NEXT seek
skipping a tombstone (M1). Conformance backend wires the real writer and updater.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
A reverse read (start > end) with a small NumValuesPerNode resumed through
the tie-cluster path, which drained the cluster and stepped FORWARD to
T + 1 tick - outside a reverse window - so the read ended with GoodNoData
after one sample. SliceTieCluster now steps down to T - 1 tick for a
reverse read (terminal at the excluded end), and the node manager slices
the over-fetched cluster in the read's own direction so the skip count is
not applied from the wrong end. Forward paging is unchanged.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
HistoryRead is served server-side by IHistorianDataSource; nothing ever dispatched
to a driver's IHistoryProvider. Removes IHistoryProvider, the OpcUaClient history
passthrough, DriverCapability.HistoryRead (+ resilience tier rows and AdminUI row),
HistoryReadStatusException and its node-manager catches, and the tests that only
covered them. Shared result types and HistoryAggregateNotSupportedException stay
(moved to Historian/HistoryReadContracts.cs). A stored ResilienceConfig still
carrying a HistoryRead key parses (diagnostic only; new test).
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
HistorianGateway is now the sole historian backend (read + alarm SendEvent +
continuous WriteLiveValues). Document the final state and retire the Wonderware
sidecar from the docs/config/labels:
- CLAUDE.md: rewrite the Historian section — ServerHistorian /
ContinuousHistorization / AlarmHistorian config keys, the IHistorianProvisioning
EnsureTags hook, the GatewayAlarmHistorianWriter SendEvent path + ReadEvents
dependency on gateway RuntimeDb:EventReadsEnabled=true, gateway-side
prerequisites (RuntimeDb flags + historian:read/write/tags:write scopes),
migration note, and two KNOWN-LIMITATION callouts (live-validation gate +
empty historized-ref-set recorder follow-on).
- appsettings.json: fix the stale ServerHistorian block (Host/Port/SharedSecret/
ServerCertThumbprint -> Endpoint/ApiKey/UseTls/AllowUntrustedServerCertificate/
CaCertificatePath/CallTimeout, keep MaxTieClusterOverfetch); add a disabled
ContinuousHistorization block; prune the orphaned Wonderware keys from
AlarmHistorian (keep the SQLite knobs). ApiKey env-supplied via
ServerHistorian__ApiKey (commented; valid strict JSON via _comment keys).
- README.md + docs (Historian.md, AlarmHistorian.md, Configuration.md,
ServiceHosting.md, DriverLifecycle.md, drivers/README.md, Uns.md, VirtualTags.md,
AlarmTracking.md, Client.UI.md, README.md, TestConnectProbes.md): retire the
Wonderware historian backend from current-backend descriptions; fix the stale
ServerHistorian/AlarmHistorian config tables (now gateway shape); convert
drivers/Historian.Wonderware.md to a retired stub pointing at the gateway.
- Source/UI labels (descriptive text only, no behavior change):
OtOpcUaServerHostedService.cs, HistoryPaging.cs, OtOpcUaSdkServer.cs,
HistorianAdapterActor.cs, VirtualTagModal.razor, ScriptedAlarmModal.razor,
AlarmsHistorian.razor now name the HistorianGateway backend.
Build clean (0 errors); AdminUI.Tests green (514 passed).
Claude-Session: https://claude.ai/code/session_012SDSQ3AcaXqPcBtDESBRii
C1 (critical): a boundary tie cluster larger than NumValuesPerNode could
silently truncate a resumed read to GoodNoData, permanently dropping the
un-emitted ties — the (timestamp, skip) cursor cannot advance past a single
timestamp the fixed-(start,end,cap) backend keeps re-returning. Now detected
and failed LOUDLY per node with BadHistoryOperationUnsupported + a log naming
the tag/timestamp/cap; documented in Historian.md with the larger-cap remedy.
Regression test Raw_tie_cluster_larger_than_page_fails_loudly_not_silently.
I3: build HistoryData before Save() so a projection failure can never orphan a
stored continuation cursor.
N1 (YAGNI): drop the never-produced HistoryReadKind enum + Processed-only
Aggregate/IntervalTicks fields from HistoryContinuationState — only Raw pages.
N3: ComputeResumeCursor guards its documented non-empty precondition.
I1: document InMemoryHistoryContinuationStore's eventual-consistency (test double).
Build clean, 182/182 OpcUaServer tests pass.
The Wonderware historian backend is single-shot — it returns up to
NumValuesPerNode samples with a null continuation point — so paging is
synthesised server-side, time-based, for the only count-capped arm (Raw):
- A full page (count == NumValuesPerNode, NumValuesPerNode > 0) emits an
opaque 16-byte continuation point and stores a resume cursor; a short page
(or NumValuesPerNode == 0 "all values") emits none.
- A resume read takes the stored cursor, reads the next page from the boundary
forward, and emits a fresh CP only if that page is also full.
- The resume cursor is tie-safe (HistoryPaging.ComputeResumeCursor /
TrimBoundaryDuplicates): the next page resumes from the boundary timestamp
INCLUSIVE and drops the head ties already returned, so samples sharing the
boundary SourceTimestamp are neither duplicated nor skipped.
Continuation points are bound to the OPC UA session via the SDK's
ISession.SaveHistoryContinuationPoint / RestoreHistoryContinuationPoint store
(SessionHistoryContinuationStore) — capped by ServerConfiguration.
MaxHistoryContinuationPoints (default 100, oldest-evicted) and disposed on
session close. releaseContinuationPoints is honoured via an override of
HistoryReleaseContinuationPoints (the base dispatcher routes release-only reads
there, never to the per-details arms). An unknown / evicted / released point
resumes to BadContinuationPointInvalid.
Processed and AtTime stay single-shot: neither details type carries a client
count cap, so the single-shot backend returns the complete result in one read
and there is no "full page" signal to page on (spec-conformant). Modified-value
history remains out of scope.
The pure paging decisions + CP store contract are unit-tested via HistoryPaging
+ InMemoryHistoryContinuationStore; the full multi-page round trip is driven
end-to-end through the node manager with an in-memory store + a series-backed
fake historian (the in-process harness is session-less).
Joseph Doherty