ReverseRawReader aimed every segment at half of MaxTieClusterOverfetch
(~32k rows by default) and carried that width page to page, so a paged
'latest N' read on a dense Gateway tag fetched ~32,770 rows per page
(2-5 s live) to return N. The walk now lives in a generic ReverseWalk
core: segments are aimed at ~N+8 rows (page budget 2N+16), a discovery
segment (first without a hint / after an empty one) may read 1,024, and
the tie budget is only the ceiling a single-timestamp cluster escalates
to. A truncated read never re-enters the stretch below its newest row
until the gap above it is bisected. Tests: per-page row counts on a
dense tag (node manager + walk), randomized exactness fuzz (walk and
node-manager paging), dense-tag-below-a-long-gap read bound.
1. Uncapped reverse raw read on a forward-only backend = ONE forward read of the
window, filtered and reversed in memory (was a segment-by-segment walk).
2. Segment width is aimed at ~half the row budget from the density just observed
(overflow: from the truncated read's own sample span) and carried page to page
in HistoryContinuationState.SegmentWidthTicks.
3. The walk starts no later than now + 1 day and never reads before the epoch.
4. Request bounds are rounded outward to the millisecond, so a row AVEVA
fabricates at a rounded-up query start cannot land inside the segment.
5. Budget M = max(1, MaxTieClusterOverfetch); reads ask M + 2 rows, completeness is
'short OR last row past the segment', and a read crowded by rows before the
segment is re-read once (<= 2M + 4). A cluster of exactly M, neighbouring
clusters each within M, and M <= 1 are all served; only one timestamp over M
is refused. ServerHistorianOptions text updated.
6. The raw and events tie-path oversized-cluster refusal is BadTooManyOperations
(was BadHistoryOperationUnsupported), consistent with the walk.
7. GatewayHistorianDataSource.ReadRawAsync: a caller cancellation is not a backend
failure; NumValuesPerNode = 0 is sent as int.MaxValue (the client package and
the gateway reject max_values <= 0, so every uncapped Gateway raw read failed);
a MaxValuesCeiling refusal is a HistoryReadLimitExceededException.
8. Tests: upper-edge filter, epoch floor, look-ahead, ms alignment, exact/adjacent
clusters, minimal budget, carried density, and the walk over the real Gateway
source + FakeHistorianGatewayClient (OpcUaServer internals visible to it).
- HistoryUpdate items with SourceTimestamp > now + ServerHistorian:HistoryUpdateMaxFutureSkew
(default 1 day, <=0 disables) => BadOutOfRange in the node manager (every backend, never sent);
clock injected via OtOpcUaNodeManager.Clock (host TimeProvider).
- HistoryUpdate is Primary-only: off-Primary => BadInvalidState per node, after authorization,
same IDriverPrimaryRoleSource verdict as the native-ack gate; HistoryRead ungated.
- SqlServer: base schema requirement back to v1; SupportsModifiedTrail (v>=2). On v1 with
AutoCreateSchema=false: writes/reads/events/retention work, modified reads + HistoryUpdate =>
BadHistoryOperationUnsupported with nothing sent, retention skips SampleModified, one Warning per boot.
- QuestDB integration test pinning wall-clock TTL (far-future row evicts only partitions older
than the TTL by the wall clock), with a positive control.
- Docs: Historian.md, Configuration.md, security.md, CLAUDE.md (incl. QuestDB retry duplicates,
#674.4 anonymous-readable trail with ACL off, #674.5 not implemented).
Co-Authored-By: Claude <noreply@anthropic.com>
M1: the time-weighted family (incl. Total, formerly a pushed-down SUM) streams
raw window rows into the C# evaluator. New shared knob
ServerHistorian:MaxTimeWeightedRawRows (default 1,000,000; <=0 unbounded with a
Validate warning) flows through HistorianBackendRegistry into both providers.
The window query is capped at cap+1 (SQL Server TOP (@maxRows), QuestDB LIMIT $4);
one row past the cap throws the new Core.Abstractions
HistoryReadLimitExceededException, which the node manager maps to
BadTooManyOperations for that node. Not counted as a backend health failure;
bound rows are not counted; per-sample aggregates are not capped.
M2: reverse Interpolative is anchored at the later time (Part 13): points at
End - i*interval, emitted latest first. 10:00->09:00 @1h yields 10:00.
L1: Partial bit (0x4, = Opc.Ua.AggregateBits.Partial) on non-Bad results of a
truncated last interval (time-weighted family; per-sample family is a follow-up).
L2/L3: aligned with the SDK AggregateCalculator.GetTimeBasedStatusCode
(decompiled from Opc.Ua.Server 1.5.378.106): TreatUncertainAsBad=false counts
Uncertain time (Uncertain samples, held-into-outage, extrapolation) as Good
time; data but no non-Bad time => generic Bad (0x80000000); BadNoData only
when there is no data at all. Consequence: a live tag's latest interval reads
Good (documented).
L5: release note no longer claims Total matches the Gateway Integral exactly
(AVEVA's divisor is unverified); it notes Total is now evaluated in C# and
subject to the cap.
Tests: unit (reverse Interpolative x3, Partial, Bad vs BadNoData incl. an
open-ended outage, cap helpers, SDK bit oracles); conformance (reverse
Interpolative, raw-row cap at/over/0, bounds not counted, health unaffected);
node manager BadTooManyOperations; options default + warning. Each new
behaviour falsified by mutation (aggregator x9, data-source cap checks on
both backends against the real DBs, node-manager catch).
- Historian.md: Empty samples always Bad (writer/updater/read clamp),
long-name refusal, 5 s bootstrap failure cache, AtTime after now, info-bit
masking, HistoryUpdate does NOT type-check (M-1 decision), DeleteRaw
needs both bounds, reverse Raw paging, the historian/historization metric
prefixes, and provisioning numeric-only lines qualified as Gateway.
- CLAUDE.md numeric-only lines qualified as Gateway backend.
- Stale 'sole historian' lines in DriverLifecycle, ServiceHosting and the
retired Wonderware stub; Client.CLI exit-code contract.
- ServerHistorianOptions: Endpoint/Backend XML docs untangled, Validate()
indented; Program.cs comments are backend-neutral; T10 week-test day count.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
OtOpcUa commits no plaintext secrets — the only secret-shaped committed value is
ServerHistorian:ApiKey='' (empty, disabled by default). Because the pre-host expander
is fail-closed AND section-agnostic, hard-committing a ${secret:} token would break
every dev/CI/integration boot for zero at-rest benefit. Instead document the token
delivery convention for all five config secrets (jwt/ldap/deploy/configdb/historian)
in docs/security.md and note the token option in the ServerHistorian comments.
Mechanism proven live: unseeded token boot fails with SecretNotFoundException at
pre-host expansion; seeded token resolves and boot proceeds. Behavior-neutral
(comment + doc only).
Joseph Doherty