Every HistoryUpdate a SQL backend applies now leaves a modification trail
(prior value, principal, Part 11 update type, modification time), and
HistoryReadRaw with IsReadModified=true serves it as HistoryModifiedData.
- Core: HistoryUpdateContext threaded into all three IHistorianDataUpdater
methods; IHistorianDataSource.ReadModifiedAsync (default NotSupported =>
BadHistoryOperationUnsupported on Gateway/Null); HistoryModified* contracts.
- SqlServer: schema v2 (002_sample_modified.sql, RequiredVersion=2, docs copy);
trail written by OUTPUT ... INTO on the same MERGE/DELETE statement, so a
failed update leaves no audit row; retention purges SampleModified per tag on
the source-time cutoff; ReadModifiedAsync keyed on (SourceTimeUtc, ModifiedId).
- QuestDB: append-only {prefix}sample_modified (TTL like samples), trail row
written BEFORE the change (orphan-on-crash, never an untracked change);
existence reads now carry the prior value; ReadModifiedAsync.
- Node manager: modified reads dispatch + page via the #669 continuation store
(HistoryContinuationState gains HistoryCursorKind + LastModifiedId), gated by
the existing HistoryRead decision; session principal / key id passed to the
updater. IsDeleteModified stays unsupported.
- Client.CLI: historyread --modified.
- Tests: node-manager dispatch/paging/cursor binding/principal threading,
over-the-wire HistoryModifiedData, shared ModifiedValueConformance on both
backends, SqlServer atomicity (forced trail failure => no change) + v1->v2
upgrade + retention trail purge.
- Docs: Historian.md (Modified-value history section, schema v2 upgrade note),
security.md, Client.CLI.md, CLAUDE.md.
M1: the time-weighted family (incl. Total, formerly a pushed-down SUM) streams
raw window rows into the C# evaluator. New shared knob
ServerHistorian:MaxTimeWeightedRawRows (default 1,000,000; <=0 unbounded with a
Validate warning) flows through HistorianBackendRegistry into both providers.
The window query is capped at cap+1 (SQL Server TOP (@maxRows), QuestDB LIMIT $4);
one row past the cap throws the new Core.Abstractions
HistoryReadLimitExceededException, which the node manager maps to
BadTooManyOperations for that node. Not counted as a backend health failure;
bound rows are not counted; per-sample aggregates are not capped.
M2: reverse Interpolative is anchored at the later time (Part 13): points at
End - i*interval, emitted latest first. 10:00->09:00 @1h yields 10:00.
L1: Partial bit (0x4, = Opc.Ua.AggregateBits.Partial) on non-Bad results of a
truncated last interval (time-weighted family; per-sample family is a follow-up).
L2/L3: aligned with the SDK AggregateCalculator.GetTimeBasedStatusCode
(decompiled from Opc.Ua.Server 1.5.378.106): TreatUncertainAsBad=false counts
Uncertain time (Uncertain samples, held-into-outage, extrapolation) as Good
time; data but no non-Bad time => generic Bad (0x80000000); BadNoData only
when there is no data at all. Consequence: a live tag's latest interval reads
Good (documented).
L5: release note no longer claims Total matches the Gateway Integral exactly
(AVEVA's divisor is unverified); it notes Total is now evaluated in C# and
subject to the cap.
Tests: unit (reverse Interpolative x3, Partial, Bad vs BadNoData incl. an
open-ended outage, cap helpers, SDK bit oracles); conformance (reverse
Interpolative, raw-row cap at/over/0, bounds not counted, health unaffected);
node manager BadTooManyOperations; options default + warning. Each new
behaviour falsified by mutation (aggregator x9, data-source cap checks on
both backends against the real DBs, node-manager catch).
New AggregateFamily.TimeWeighted serving TimeAverage, TimeAverage2, Total,
Total2 and Interpolative, evaluated by the shared Sql.Common
TimeWeightedAggregator for both SQL backends (identical results by
construction). HistoryAggregateType gains TimeAverage2 + Total2 (appended);
OtOpcUaNodeManager.MapAggregate maps their NodeIds. Gateway's mapper already
refuses unknown defined members => BadAggregateNotSupported per node.
Semantics (server defaults, client AggregateConfiguration ignored):
TreatUncertainAsBad=false, PercentDataBad=PercentDataGood=100,
UseSlopedExtrapolation=false. Sloped for Double/Int64, stepped for Boolean;
a usable sample followed by a Bad one is held up to the outage (AtTime rule);
stepped Uncertain extrapolation past the last sample. TimeAverage/Total use
interpolated bounding (bridge outages); TimeAverage2/Total2 exclude Bad time.
Bounding values: latest usable + latest any before start, first any + first
usable at/after end (four TOP 1 / LIMIT 1 seeks).
CLIENT-VISIBLE: Total on SqlServer/QuestDb was SUM of samples; it is now the
time-weighted integral (TimeAverage x interval seconds), matching Gateway's
Integral. Release note in docs/Historian.md.
Trade-off (deviation from the preferred hybrid pushdown, allowed by the
design): the time-weighted family is NOT pushed down. Each read streams the
window's rows plus the four bounds (SQL Server: one five-result-set batch;
QuestDB: bounds UNION ALL + window select) into the C# evaluator. Network is
O(samples in window), memory O(buckets). Writing the per-segment
interpolation/edge splitting twice in two SQL dialects and keeping them
bit-identical was judged the bigger risk; this also makes the QuestDB
lag/lead spike moot.
Tests: 35 hand-computed TimeWeightedAggregator unit cases (each of 10
implementation mutations goes red); served-aggregate pin 13 -> 17;
conformance sparse fixture (intervals wider and narrower than sample gaps)
+ an outage straddling both window edges (each of the four SQL seeks
disabled in turn goes red); node-manager NodeId mapping test.
- I-1: QuestDbConnection caps Npgsql Maximum Pool Size at 24 (DefaultMaxPoolSize)
when the connection string does not set it; an explicit value is kept. The
effective value is logged once at Information.
- M-1: QuestDbTagCatalog.EnsureAsync waits for the tag table's sequenced commits
and re-checks before inserting, so it no longer overwrites a provisioner row
(unit/description) that is committed but not yet applied. Integration test
reproduces the clobber against real QuestDB (falsified).
- M-3: the pre-1970 drop Warning is rate-limited per tag (once per 10 min);
DroppedPre1970 still counts every sample.
- M-5: QuestDbHistorianProvider is IAsyncDisposable/IDisposable and disposes the
shared Npgsql pool; Program.cs registers the provider through a factory and
every seam resolves it from the container, so DI disposes it at shutdown.
- M-8, M-9: docs fixes (HistoryUpdate transaction wording; folded duplicate
HistoryUpdate paragraph). Pool-size docs updated for the new default cap.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
AtTime conformance stamps are the requested time (M3; identical on SQL Server). The
QuestDB test fixture prunes idle pooled connections after 1 s: undisposed per-test data
sources exhausted QuestDB's 64-connection PG-wire limit in a full run.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
IHistorianDataSource.TimestampResolutionTicks (default 1; QuestDB 10). A drained tie
cluster now steps off T by the resolution, so a forward one-value-per-page read on a
microsecond backend no longer re-reads the same sample forever (C1). QuestDB AtTime
results carry the client's original request time (M3); regression for the NEXT seek
skipping a tombstone (M1). Conformance backend wires the real writer and updater.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
QuestDbHistorianDataSource serves Raw (LIMIT $n, no continuation, R8), AtTime
(chunked LIMIT 1 neighbour seeks, <=50 times per statement), Events and
Processed (one statement per node, bucket = (cast(ts AS LONG)*10 - start)/interval,
shared BucketPlan/AggregateEvaluator/AtTimeEvaluator). Every sample read filters
NOT deleted; every row goes through StoredSample.FromStored; RC2 long.MinValue
read repair. QuestDbConformanceBackend (seed/read-back) for the conformance suite;
writer/updater factories are T7 placeholders.
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
Joseph Doherty