Commit Graph

1068 Commits

Author SHA1 Message Date
Joseph Doherty cf1af55720 fix(historian): #686 review — bounded, density-aware reverse raw walk; uncapped Gateway raw reads
1. Uncapped reverse raw read on a forward-only backend = ONE forward read of the
   window, filtered and reversed in memory (was a segment-by-segment walk).
2. Segment width is aimed at ~half the row budget from the density just observed
   (overflow: from the truncated read's own sample span) and carried page to page
   in HistoryContinuationState.SegmentWidthTicks.
3. The walk starts no later than now + 1 day and never reads before the epoch.
4. Request bounds are rounded outward to the millisecond, so a row AVEVA
   fabricates at a rounded-up query start cannot land inside the segment.
5. Budget M = max(1, MaxTieClusterOverfetch); reads ask M + 2 rows, completeness is
   'short OR last row past the segment', and a read crowded by rows before the
   segment is re-read once (<= 2M + 4). A cluster of exactly M, neighbouring
   clusters each within M, and M <= 1 are all served; only one timestamp over M
   is refused. ServerHistorianOptions text updated.
6. The raw and events tie-path oversized-cluster refusal is BadTooManyOperations
   (was BadHistoryOperationUnsupported), consistent with the walk.
7. GatewayHistorianDataSource.ReadRawAsync: a caller cancellation is not a backend
   failure; NumValuesPerNode = 0 is sent as int.MaxValue (the client package and
   the gateway reject max_values <= 0, so every uncapped Gateway raw read failed);
   a MaxValuesCeiling refusal is a HistoryReadLimitExceededException.
8. Tests: upper-edge filter, epoch floor, look-ahead, ms alignment, exact/adjacent
   clusters, minimal budget, carried density, and the walk over the real Gateway
   source + FakeHistorianGatewayClient (OpcUaServer internals visible to it).
2026-10-12 15:11:21 -04:00
Joseph Doherty 410c8d8976 fix(historian): #686 serve reverse/end-only raw reads on forward-only backends (Gateway)
#669 turned an end-only capped HistoryRead-Raw into a reverse read (start > end)
for every backend; the gateway rejects start > end (InvalidArgument), so on the
default Gateway backend every end-only capped and every explicit reverse raw read
failed the node and was counted as a backend health failure.

- IHistorianDataSource.SupportsReverseRaw (default false; SqlServer/QuestDB/Null
  true, Gateway false; guard test pins that every production backend declares it).
- OtOpcUaNodeManager sends a reverse window only to a backend that declares it;
  otherwise ReverseRawReader walks (lo, hi] back in bounded forward segment reads
  (1 min, x4 while sparse; each <= MaxTieClusterOverfetch + 1 rows; a full read is
  re-read narrower, never answered truncated; filtered on sample timestamps, which
  also drops AVEVA's fabricated query-start rows; stops at the Unix epoch). A
  one-timestamp tie cluster over the bound => BadTooManyOperations.
- ServeRawPagedAsync maps HistoryReadLimitExceededException to BadTooManyOperations.
- GatewayHistorianDataSource.ReadRawAsync refuses start > end unsent, without
  recording a health failure.
2026-10-12 14:42:27 -04:00
Joseph Doherty 5f225f0b44 fix(historian): #681 on #682/#684 — Gateway served set is the 12-member ServedAggregates; raw buckets use Good-only values
- #682: ServedAggregates (ten raw-evaluated per-sample + TimeAverage/Total) replaces the placeholder
  AggregateModeMapper.Served; NativeAggregates is derived from the mapper's switch. The Gateway
  expectations in AggregateModeMapperTests, HistorianBackendRegistryTests and
  AdvertisedAggregateFunctionsWireTests move to the 12-member set (the wire tests' 'not served by
  Gateway' example is now TimeAverage2).
- #684: RawBucketAccumulator applies the SQL backends' Good-only value columns (Avg/Min/Max/Sum and
  the ActualTime extreme over Good numeric samples), UncertainMin/UncertainMax over Uncertain numeric
  samples, and GoodNonNumeric. The parity oracle carries the #684 InMemoryRows and the DenseUncertain /
  UncertainBeyond / UncertainText datasets.
2026-10-12 10:00:30 -04:00
Joseph Doherty 78f604e0cd feat(historian): Gateway serves the per-sample aggregates from raw reads with the shared Sql.Common evaluator (#681)
Average, Count, Minimum, Maximum, Range, Start, End, Delta and Minimum/MaximumActualTime are evaluated
on the Gateway backend from forward ReadRaw reads with BucketPlan + AggregateEvaluator, so Gateway
matches SqlServer/QuestDB exactly (values, status bits, interval-start stamps, #677 reverse).

- Window read [start - 1 ms, end] so AVEVA's initial-value row never lands in a bucket; a pre-window
  row proves earlier data only with a non-Bad OPC quality (histsdk puts a value on every NULL row).
- Earlier data otherwise from a 2-row [1970, start) probe; later data from a 2-row [end, now+1d]
  look-ahead; both ignore rows stamped at their query start/end.
- No gateway stream is ever abandoned (a client-side cancel makes the gateway evict its shared
  historian session); past ServerHistorian:MaxTimeWeightedRawRows the bounded stream is drained, then
  BadTooManyOperations. The gateway's max_values-ceiling InvalidArgument maps to the same limit; a
  caller cancellation is not a backend fault.
- TimeAverage/Total stay AVEVA-native (no prior-sample read for exact Part 13 bounds), now stamped with
  the interval start (PENDING the live leg: histsdk's tail+43 start_time field is unverified) and
  carrying the Calculated bits.
- Served set: GatewayHistorianDataSource.ServedAggregates (12). Live AVEVA leg open (VPN).
2026-10-12 09:56:54 -04:00
Joseph Doherty c019d0fc2b fix(historian): trim only HistoryServerCapabilities/AggregateFunctions; null set = all (#682 review)
ServerCapabilities/AggregateFunctions (i=2997) is Part 5's list of every
aggregate the server supports, and live AggregateFilter subscriptions still
compute all 37 SDK aggregates, so it keeps the SDK's full set; only i=11201
follows the backend. A null SupportedAggregates is treated as the interface
default (all) by both the advertisement and HistoryRead-Processed (it used to
NRE inside the batch). Docs: i=2997/i=11201 split and the boot window.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-12 09:18:53 -04:00
Joseph Doherty 6ea3ba44da feat(historian): advertise only the backend's aggregates under AggregateFunctions (#682)
IHistorianDataSource.SupportedAggregates (default: every HistoryAggregateType)
is the one authority for both ServerCapabilities/AggregateFunctions (i=2997) and
HistoryServerCapabilities/AggregateFunctions (i=11201) and for the up-front
BadAggregateNotSupported in HistoryRead-Processed. Gateway declares
AggregateModeMapper.Served (4), SqlServer/QuestDB AggregateFamilies.Served (17),
Null none. Only the folder references are trimmed; the SDK's calculator
factories stay so live AggregateFilter subscriptions keep working.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-12 08:52:47 -04:00
Joseph Doherty beb1c583b1 fix(opcuaclient): #676 live gate A1 — a sustained closed connection is a definitive loss
Live gate A1 (docker-dev, opc-plc restart / SIGINT / kill): the fast path never fired —
a real process stop delivers no session status; the keep-alives report
BadSecureChannelClosed (0x80860000) once, then BadConnectionClosed (0x80AE0000)
interleaved with BadNoCommunication for the whole outage.

Reproduced in-process with TcpStallProxy.HardClose (reset every connection, refuse new
ones; server and session alive behind it): the identical sequence, and the SDK 1.5.378
session never opens a new connection on its own (48 s with the proxy accepting again,
zero attempts) — a closed channel cannot recover in place.

The driver now treats BadSecureChannelClosed/BadConnectionClosed as definitive on the
SECOND closed-channel keep-alive failure in one outage (a good keep-alive resets the
count), raising ConnectionLost ~1 keep-alive after the drop. BadNoCommunication /
BadRequestTimeout / BadNotConnected (stall/pause) stay with the watchdog; the stall test
still sees no ConnectionLost. New wire test (hard close + reopen ⇒ raised once, SDK never
reconnects, reinit brings the identical value back Good) and unit tests (sustained raises
once; a single blip that recovers does not) — all red on the previous code. Docs updated.
2026-10-11 12:34:52 -04:00
Joseph Doherty 945ec2c069 Merge feat/677-aggregates: reverse processed reads SDK-anchored, start-of-data Partial (forward), AggregateConfiguration accept-defaults, IntegralDivisor=1, SDK parity tests (#677) 2026-10-11 11:47:52 -04:00
Joseph Doherty 50c09e21d5 Merge feat/676-opcuaclient-followups: endpoint host+port rewrite, TCP-stall wire test, ReconnectPeriod removed, IDriverConnectionLossSource fast recovery, trailing re-read, replay summary (#676) 2026-10-11 11:45:26 -04:00
Joseph Doherty 029a916288 Merge feat/674-historyupdate-posture follow-ups: HistoryUpdate closed until first redundancy snapshot, node-level v1 refusal, host wiring tests, plan-before-gate, UTC skew (#674 review) 2026-10-11 11:41:36 -04:00
Joseph Doherty 232c03ed1b fix(historian): #674 review follow-ups — HistoryUpdate boot-window gate, node-level v1 refusal, plan-before-gate, UTC skew compare
- HistoryUpdate Primary gate now reads IHistoryUpdateRoleView (PrimaryGatePolicy.ShouldServeHistoryUpdate):
  the data-plane verdict but CLOSED until this node's first redundancy snapshot, so both pair nodes cannot
  accept in the pre-join boot window. Single nodes open on their own singleton's first snapshot; admin-only
  nodes register no view => no gate. DriverHostActor publishes it at every Primary-verdict publish point.
- SqlServer v1: the updater throws NotSupportedException on every arm; the node manager maps it to a
  node-level BadHistoryOperationUnsupported (was per-item under a Good node for UpdateData/DeleteAtTime).
  HistoryWrite bit still advertised on v1 (not knowable at materialise time) — documented.
- The update plan is evaluated before the Primary gate: an unservable request (Remove, IsDeleteModified,
  not historized, no updater) is BadHistoryOperationUnsupported off-Primary, not BadInvalidState.
- Local-kind SourceTimestamp converted to UTC before the future-skew compare and sent as UTC.
- Hosted-service wiring extracted to BuildHistoryUpdatePosture and tested (gate source, skew, clock).
- Docs: Historian.md (boot window, ordering, node-level v1, HistoryWrite bit), CLAUDE.md.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-11 11:41:11 -04:00
Joseph Doherty 93d6c1da67 fix(historian): #675 review follow-ups — SupportsReverseEvents capability, no catch-as-unsupported
- IHistorianDataSource.SupportsReverseEvents (default false; SqlServer/QuestDB true; Gateway/Null declare false).
  The node manager checks it BEFORE calling the source: an end-only capped events read becomes reverse only on
  a supporting backend (Gateway, the default, keeps the pre-#675 forward [MinValue, EndTime) paging), and an
  explicit reverse read on a non-supporting backend is BadHistoryOperationUnsupported without a call.
- Removed the broad catch (NotSupportedException) in the events path: a NotSupportedException from inside a
  backend is now logged as a backend fault (Error), not an 'unsupported' Warning.
- Null declares the reverse overload itself; guard test: every production IHistorianDataSource declares both
  the reverse overload and the capability (positive-control scan of all four backends).
- Conformance: capability reported + oversized reverse tie cluster (forward over-fetch reversed == reverse
  read order; capped reverse read stopping inside the cluster flags truncation) on SqlServer + QuestDB.
- docs/Historian.md: end-only capped events read is backward on SqlServer/QuestDB, forward on Gateway.
2026-10-11 11:36:12 -04:00
Joseph Doherty 30a9d5f094 fix(opcuaclient): #676 review fix-ups (port-less rewrite, full reconcile, delivery lock, timers)
1. RewriteEndpointUrl keeps the advertised port when the configured URL names none
   (Uri.Port -1 would have sent the session to 4840); port-less + IPv6 tests.
2. The trailing re-read repairs any (value, status) difference that is not older than
   the last delivered value — Good->Bad/Uncertain included; a read without a source
   timestamp counts as newer; an equal source time with different content is published
   (repairs the replay guard's by-design equal-timestamp drop).
3. One driver-wide delivery lock makes decide+publish atomic across live notifications,
   the recovery read and the trailing read, so an older value can never land after a
   newer one (deterministic race test via a seam).
4. Known gap documented: a ConnectionLost raised while the host is not Connected is
   swallowed and that session falls back to the watchdog.
5. Only a SERVER-reported subscription loss raises ConnectionLost; the publishing-stall
   heuristic stays with the #527 watchdog.
6. Post-recovery timers run on Task.Delay(…, Clock, ct) with a session-work CTS
   cancelled on session release and dispose.
7. Replay summary at Information only when something was dropped (Debug otherwise);
   a re-arm reports the superseded window's drops and counts afresh.
8. DriverInstanceActor: AttachHostStatusSource doc comment restored; Reconnecting
   swallows a stray DisconnectObserved (test).
9. Stall wire test: 120 s session timeout, 3000-cycle subscription lifetime, one further
   bad keep-alive tick, 60 s headroom on the post-recovery waits (5/5 green).
2026-10-11 11:35:53 -04:00
Joseph Doherty fccffd0a0f feat(opcuaclient): #676 follow-ups — endpoint rewrite, stall wire test, fast definitive-loss recovery
- Endpoint rewrite: the driver now connects to the configured host AND port (scheme
  match, advertised path kept) instead of the server-advertised URL — fixes NAT/port-map
  upstreams and lets an in-test proxy carry the session (SDK SelectEndpoint rule).
- TcpStallProxy + OpcUaClientDriverStallWireTests: the in-place blip (#670) is now
  wire-tested on loopback — one Bad, frozen read stamp, same session, identical value
  republished Good, a post-recovery change arrives, no source-time regression after
  the SDK's late backlog flush.
- Defect found by that test, fixed: a change published right after a stall rode a
  Publish response the SDK timed out and never republished, leaving the node Good on
  the pre-change value indefinitely. New trailing reconcile read (Timeout +
  KeepAliveInterval after recovery) publishes any Good value newer than the last
  accepted one, through the replay guard's mark.
- IDriverConnectionLossSource (Core.Abstractions): opt-in definitive-loss signal.
  DriverInstanceActor attaches in PreStart / detaches in PostStop and turns a raise
  into a disconnect only while Connected. OpcUaClient raises once per session on
  keep-alive BadSessionIdInvalid/BadSessionClosed/BadSessionNotActivated/
  BadServerHalted (measured: the SDK never re-opens a halted channel) or a dropped
  subscription; ambiguous codes stay with the #527 watchdog.
- Replay guard: one Information summary per recovery when the window closes
  ("dropped N late notifications across M refs"), with a timer so quiet
  subscriptions log it too.
- ReconnectPeriod deleted (options, AdminUI form, docs, tests); stored configs that
  carry reconnectPeriod still parse in the factory, probe and browser.
- Docs: OpcUaClient.md, DriverLifecycle.md, driver-specs.md, CLAUDE.md.
2026-10-11 11:35:52 -04:00
Joseph Doherty 9120182314 Merge master into feat/677-aggregates
# Conflicts:
#	docs/Historian.md
2026-10-11 11:30:46 -04:00
Joseph Doherty 27ca6cd791 feat(historian): #677 Part 13/SDK reverse anchoring, start-of-data Partial, AggregateConfiguration, IntegralDivisor
- BucketPlan: reverse processed reads anchored at the later time for every
  aggregate: window (end, start], partial interval earliest, results stamped
  with the later edge, latest first (SDK CreateSlice/GetTimestamp/backward
  GetValues). SqlServer adds @bucketAdjustTicks + flips the window predicate;
  QuestDB binds BucketBaseUnixTicks. IndexOf is the C# statement of the math.
- Partial bit on the interval the tag's data starts inside (SDK m_startOfData),
  every aggregate but Average; per-sample reads fetch the earliest sample.
- Node manager: non-default client AggregateConfiguration =>
  BadAggregateConfigurationRejected per node (ServerAggregateConfiguration,
  pinned to the SDK AggregateManager defaults).
- GatewayTagProvisioner sends IntegralDivisor = 1.
- SDK parity oracle (Opc.Ua.Server calculators) in unit + both backends'
  conformance suites, forward and reverse; remaining divergences documented.
- docs/Historian.md + CLAUDE.md release note (client-visible).
2026-10-11 11:26:00 -04:00
Joseph Doherty 8e6ac4b344 Merge feat/675-reverse-event-paging: reverse event HistoryRead paging, end-only backward, Gateway MaxValue pin (#675) 2026-10-11 11:23:51 -04:00
Joseph Doherty e59ab69398 Merge feat/674-historyupdate-posture: future-skew bound, Primary-only HistoryUpdate, SqlServer v1 accepted with modified trail disabled, QuestDB TTL pin (#674) 2026-10-11 11:21:30 -04:00
Joseph Doherty 8d154c6a81 feat(historian): #675 reverse HistoryRead-Events paging + end-only backward reads
- IHistorianDataSource.ReadEventsAsync(..., bool reverse, ...) overload: reverse = Part 11 window (lo, hi]
  newest-first by (time, id) DESC. Default forwards forward reads and throws NotSupportedException for
  reverse (Null); Gateway refuses explicitly (ascending-only stream). Server answers
  BadHistoryOperationUnsupported instead of silently returning the earliest N ascending.
- SqlServer / QuestDB serve reverse natively (ORDER BY time DESC, id DESC + TOP/LIMIT).
- Node manager: reverse event reads page with continuation points; direction stored in the cursor
  (HistoryContinuationState.Reverse) so a resume continues in the cursor's direction; reverse tie clusters
  over-fetched forward and sliced from the newest end (SliceTieCluster(reverse:)); end-only reads with a
  cap page backward from EndTime (mirrors raw).
- Resumed event pages over-fetch by the boundary skip, so every non-final page is a full cap page.
- Tests: node-manager reverse paging (ties, clusters, coarse resolution, overfetch bound, resume
  direction both ways, end-only, unsupported backend); shared EventReadConformance on SqlServer +
  QuestDB; over-the-wire reverse paging; offline Gateway pin of the UTC DateTime.MaxValue bound through
  the published client's proto conversion (in-process gRPC capture) and FILETIME.
- docs/Historian.md event paging section.
2026-10-11 11:19:54 -04:00
Joseph Doherty 0658d244d0 feat(historian): #674 HistoryUpdate posture — future-skew bound, Primary gate, SqlServer v1 accepted, QuestDB TTL pin
- HistoryUpdate items with SourceTimestamp > now + ServerHistorian:HistoryUpdateMaxFutureSkew
  (default 1 day, <=0 disables) => BadOutOfRange in the node manager (every backend, never sent);
  clock injected via OtOpcUaNodeManager.Clock (host TimeProvider).
- HistoryUpdate is Primary-only: off-Primary => BadInvalidState per node, after authorization,
  same IDriverPrimaryRoleSource verdict as the native-ack gate; HistoryRead ungated.
- SqlServer: base schema requirement back to v1; SupportsModifiedTrail (v>=2). On v1 with
  AutoCreateSchema=false: writes/reads/events/retention work, modified reads + HistoryUpdate =>
  BadHistoryOperationUnsupported with nothing sent, retention skips SampleModified, one Warning per boot.
- QuestDB integration test pinning wall-clock TTL (far-future row evicts only partitions older
  than the TTL by the wall clock), with a positive control.
- Docs: Historian.md, Configuration.md, security.md, CLAUDE.md (incl. QuestDB retry duplicates,
  #674.4 anonymous-readable trail with ACL off, #674.5 not implemented).

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-11 11:18:44 -04:00
Joseph Doherty 1375f9349e Merge fix/678-small: retention cancel/fallback tests, #672 tests, FOCAS race, TTL log wording, debug overlay, role-overlay suffix fix (#678) 2026-10-11 10:55:19 -04:00
Joseph Doherty f292cf3beb fix(host): #678.6 build the role-overlay suffix from plain roles only; audit overlays; driver overlay Opc.Ua Debug -> Information 2026-10-11 10:47:01 -04:00
Joseph Doherty fe919f8b08 fix(client): endpoint port rewrite, Part 13 time-weighted aggregates, clean history-refusal error (#678.5, #677.5, #674.6, #678.6)
- EndpointSelector rewrites host AND port to the user-supplied URL (same scheme only)
- AggregateType/mapper/CLI/UI gain TimeAverage, TimeAverage2, Total, Total2, Interpolative
- HistoryReadCommand turns ServiceResultException into a one-line CommandException
- NodeManagerHistoryReadTests non-historized case sets IsReadModified=false
2026-10-11 10:45:18 -04:00
Joseph Doherty 42eb96af9e test(runtime): #678.2 #672 initial-state unreachable peer + strengthen Self_unreachable_is_ignored 2026-10-11 10:43:15 -04:00
Joseph Doherty 7c80d36eb6 fix(historian): #668 review follow-ups (explicit modified source time, paging tests, ACL test, QuestDB orphan warning, docs)
- HistoryModifiedValue gains explicit SourceTimeUtc (SqlServer + QuestDB populate it); the node manager's resume cursor uses it instead of the prior value's nullable source timestamp (MinValue for inserts).
- ACL test now uses a trail-returning data source and asserts Good + data on the allow leg.
- Node-manager tests: end-only backward open-ended paging, reverse resume cursor, null-prior cursor.
- QuestDB updater logs a Warning (tag + time range) when a sample write fails after its trail rows landed.
- docs/Historian.md: rolling-upgrade window, IDENTITY-vs-commit ordering, QuestDB cross-node ModifiedId clock ordering.
2026-10-11 09:37:17 -04:00
Joseph Doherty 062086f9a6 Merge feat/668-modified-values: Part 11 modified-value history (IsReadModified) on SqlServer + QuestDB (#668) 2026-10-11 09:32:13 -04:00
Joseph Doherty cfd403710e feat(historian): Part 11 modified-value history on SqlServer + QuestDB (#668)
Every HistoryUpdate a SQL backend applies now leaves a modification trail
(prior value, principal, Part 11 update type, modification time), and
HistoryReadRaw with IsReadModified=true serves it as HistoryModifiedData.

- Core: HistoryUpdateContext threaded into all three IHistorianDataUpdater
  methods; IHistorianDataSource.ReadModifiedAsync (default NotSupported =>
  BadHistoryOperationUnsupported on Gateway/Null); HistoryModified* contracts.
- SqlServer: schema v2 (002_sample_modified.sql, RequiredVersion=2, docs copy);
  trail written by OUTPUT ... INTO on the same MERGE/DELETE statement, so a
  failed update leaves no audit row; retention purges SampleModified per tag on
  the source-time cutoff; ReadModifiedAsync keyed on (SourceTimeUtc, ModifiedId).
- QuestDB: append-only {prefix}sample_modified (TTL like samples), trail row
  written BEFORE the change (orphan-on-crash, never an untracked change);
  existence reads now carry the prior value; ReadModifiedAsync.
- Node manager: modified reads dispatch + page via the #669 continuation store
  (HistoryContinuationState gains HistoryCursorKind + LastModifiedId), gated by
  the existing HistoryRead decision; session principal / key id passed to the
  updater. IsDeleteModified stays unsupported.
- Client.CLI: historyread --modified.
- Tests: node-manager dispatch/paging/cursor binding/principal threading,
  over-the-wire HistoryModifiedData, shared ModifiedValueConformance on both
  backends, SqlServer atomicity (forced trail failure => no change) + v1->v2
  upgrade + retention trail purge.
- Docs: Historian.md (Modified-value history section, schema v2 upgrade note),
  security.md, Client.CLI.md, CLAUDE.md.
2026-10-11 09:26:36 -04:00
Joseph Doherty 3b739b982b Merge feat/671-time-weighted-aggregates: Part 13 time-weighted aggregates (#671) 2026-10-11 09:10:52 -04:00
Joseph Doherty 56bb4855bb fix(historian): #671 review — raw-row cap, reverse Interpolative, SDK status rule, Partial bit
M1: the time-weighted family (incl. Total, formerly a pushed-down SUM) streams
raw window rows into the C# evaluator. New shared knob
ServerHistorian:MaxTimeWeightedRawRows (default 1,000,000; <=0 unbounded with a
Validate warning) flows through HistorianBackendRegistry into both providers.
The window query is capped at cap+1 (SQL Server TOP (@maxRows), QuestDB LIMIT $4);
one row past the cap throws the new Core.Abstractions
HistoryReadLimitExceededException, which the node manager maps to
BadTooManyOperations for that node. Not counted as a backend health failure;
bound rows are not counted; per-sample aggregates are not capped.

M2: reverse Interpolative is anchored at the later time (Part 13): points at
End - i*interval, emitted latest first. 10:00->09:00 @1h yields 10:00.

L1: Partial bit (0x4, = Opc.Ua.AggregateBits.Partial) on non-Bad results of a
truncated last interval (time-weighted family; per-sample family is a follow-up).

L2/L3: aligned with the SDK AggregateCalculator.GetTimeBasedStatusCode
(decompiled from Opc.Ua.Server 1.5.378.106): TreatUncertainAsBad=false counts
Uncertain time (Uncertain samples, held-into-outage, extrapolation) as Good
time; data but no non-Bad time => generic Bad (0x80000000); BadNoData only
when there is no data at all. Consequence: a live tag's latest interval reads
Good (documented).

L5: release note no longer claims Total matches the Gateway Integral exactly
(AVEVA's divisor is unverified); it notes Total is now evaluated in C# and
subject to the cap.

Tests: unit (reverse Interpolative x3, Partial, Bad vs BadNoData incl. an
open-ended outage, cap helpers, SDK bit oracles); conformance (reverse
Interpolative, raw-row cap at/over/0, bounds not counted, health unaffected);
node manager BadTooManyOperations; options default + warning. Each new
behaviour falsified by mutation (aggregator x9, data-source cap checks on
both backends against the real DBs, node-manager catch).
2026-10-11 09:10:26 -04:00
Joseph Doherty 1f88669304 feat(historian): Part 13 time-weighted aggregates on SqlServer + QuestDb (#671)
New AggregateFamily.TimeWeighted serving TimeAverage, TimeAverage2, Total,
Total2 and Interpolative, evaluated by the shared Sql.Common
TimeWeightedAggregator for both SQL backends (identical results by
construction). HistoryAggregateType gains TimeAverage2 + Total2 (appended);
OtOpcUaNodeManager.MapAggregate maps their NodeIds. Gateway's mapper already
refuses unknown defined members => BadAggregateNotSupported per node.

Semantics (server defaults, client AggregateConfiguration ignored):
TreatUncertainAsBad=false, PercentDataBad=PercentDataGood=100,
UseSlopedExtrapolation=false. Sloped for Double/Int64, stepped for Boolean;
a usable sample followed by a Bad one is held up to the outage (AtTime rule);
stepped Uncertain extrapolation past the last sample. TimeAverage/Total use
interpolated bounding (bridge outages); TimeAverage2/Total2 exclude Bad time.
Bounding values: latest usable + latest any before start, first any + first
usable at/after end (four TOP 1 / LIMIT 1 seeks).

CLIENT-VISIBLE: Total on SqlServer/QuestDb was SUM of samples; it is now the
time-weighted integral (TimeAverage x interval seconds), matching Gateway's
Integral. Release note in docs/Historian.md.

Trade-off (deviation from the preferred hybrid pushdown, allowed by the
design): the time-weighted family is NOT pushed down. Each read streams the
window's rows plus the four bounds (SQL Server: one five-result-set batch;
QuestDB: bounds UNION ALL + window select) into the C# evaluator. Network is
O(samples in window), memory O(buckets). Writing the per-segment
interpolation/edge splitting twice in two SQL dialects and keeping them
bit-identical was judged the bigger risk; this also makes the QuestDB
lag/lead spike moot.

Tests: 35 hand-computed TimeWeightedAggregator unit cases (each of 10
implementation mutations goes red); served-aggregate pin 13 -> 17;
conformance sparse fixture (intervals wider and narrower than sample gaps)
+ an outage straddling both window edges (each of the four SQL seeks
disabled in turn goes red); node-manager NodeId mapping test.
2026-10-11 08:44:44 -04:00
Joseph Doherty db6446ce73 Merge fix/670-opcuaclient-session-loss: OpcUaClient upstream session loss goes Bad and recovers (#670) 2026-10-11 08:41:48 -04:00
Joseph Doherty b028322568 fix(opcuaclient): #670 review — subscription loss with a live session, stale recovery latch
M1: a subscription that dies while the session lives (outage longer than
the subscription lifetime but shorter than the session timeout) used to
end Healthy with fresh keep-alive stamps and frozen values. PublishError
BadNoSubscription / BadSubscriptionIdInvalid naming a tracked
subscription, or a tracked subscription whose SDK PublishingStopped
persists one more keep-alive window + 5 s across good keep-alives, now
declares it lost: Degraded, outage published once, and nothing advances
LastSuccessfulRead until the session is released, so the #527 watchdog
re-initialises and re-subscribes. Both detectors wire-tested.

M2: in-place recovery is bound to the session whose keep-alive recovered;
one overtaken by a re-initialise no longer re-arms the new session's
outage latch or degrades its health (ReadCoreAsync expectedSession).

L1 CAS health updates; L2 re-check before each recovery publish; L3 skip
CloseSession on a lost session; L4 SDK log forwarding rate-limited
(20/min); L5 per-driver SDK LoggerFactory disposed with the driver.
Wire test asserts the read stamp stays frozen while the server is down.
2026-10-11 08:41:27 -04:00
Joseph Doherty d803e0eab4 Merge fix/669-event-continuation: event HistoryRead continuation points + open-ended reads (#669) 2026-10-11 08:37:39 -04:00
Joseph Doherty 50b504d293 fix(opcua): raw open-ended/end-only reads per Part 11; extreme-bound conformance (#669) 2026-10-11 08:37:24 -04:00
Joseph Doherty acce99de9a fix(opcua): event paging review fixes (#669): open-ended end, node-bound cursors, save after projection 2026-10-11 08:30:36 -04:00
Joseph Doherty 91434e7f61 Merge fix/672-capture-gap: capture while the recording Primary is unreachable (#672) 2026-10-11 08:29:59 -04:00
Joseph Doherty 2a8f9af5e4 fix(opcuaclient): upstream session loss goes Bad and recovers (#670)
A bad keep-alive now degrades health (LastSuccessfulRead preserved) and,
once per outage, publishes BadNoCommunication (value retained) for every
subscribed ref plus a quality-only annotation on subscribed alarm
conditions. Good keep-alives and data/event notifications advance
LastSuccessfulRead, so an idle-but-alive subscription never looks dead
to the #527 watchdog. A keep-alive that recovers on the same session
re-reads the Bad refs and republishes them Good; a session that does not
come back is recovered by the watchdog's ShutdownAsync + InitializeAsync
+ re-subscribe.

Deletes the SessionReconnectHandler wiring (it retried forever in SDK
1.5.378, cloned subscriptions the driver never re-tracked and kept health
Healthy), OnReconnectComplete, TryRearmReconnect and the reconnect_*
counters (new keepalive_recovered). Teardown of a lost session skips
wire deletes and bounds CloseSession. SDK warnings reach the driver log.
Wire test reworked: restart => Bad, then watchdog-style reinit => Good.
2026-10-11 08:27:04 -04:00
Joseph Doherty 1b2c36eb87 fix(historian): capture while the recording Primary is unreachable (#672)
A SIGKILLed Primary stays an unreachable-but-not-downed member until
auto-down, and the redundancy singleton usually dies with it, so the
standby's cached snapshot kept naming it Primary and nobody captured for
~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp)
subscription, caches the last snapshot, and excludes unreachable peers
from the historization peer-is-Primary predicate only, republishing just
the capture verdict on a reachability change. Alarm drain and the Primary
data-plane verdict are unchanged. Gated by
ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true).

Tests: message-driven cases with real ReachabilityEvents (incl. a
falsification control on the drain/data-plane verdicts) plus a real
two-node cluster crash test with downing disabled.
2026-10-11 08:26:39 -04:00
Joseph Doherty 924eff8d3c feat(historian): SQL Server retention purge (RetentionDays, Primary-gated, applock) (#667) 2026-10-11 08:21:30 -04:00
Joseph Doherty f9fd495292 fix(opcua): issue and resume continuation points for event HistoryRead (#669)
A capped events read now pages: the source's truncation flag triggers a
session-bound cursor (time inclusive + tie skip), oversized tie clusters
are paged within via a bounded half-open over-fetch, and release/unknown/
cross-kind points behave as for Raw.
2026-10-11 08:21:02 -04:00
Joseph Doherty d2be213e86 fix(historian): page by the backend's timestamp resolution; QuestDB AtTime stamps the requested time
IHistorianDataSource.TimestampResolutionTicks (default 1; QuestDB 10). A drained tie
cluster now steps off T by the resolution, so a forward one-value-per-page read on a
microsecond backend no longer re-reads the same sample forever (C1). QuestDB AtTime
results carry the client's original request time (M3); regression for the NEXT seek
skipping a tombstone (M1). Conformance backend wires the real writer and updater.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:30:25 -04:00
Joseph Doherty 13d628a07f feat(historian): QuestDb provider, registry + consumer-gated validator, compose env
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 02:23:21 -04:00
Joseph Doherty 7fafeb433c test(opcua): flag-OFF HistoryUpdate over the wire - Cluster HistoryUpdate grant allowed, WriteOperate alone denied
The rig's key roles are now settable; the test derives them with the
production ApiKeyInterimRoleMapper from the same grants it installs.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 14:31:01 -04:00
Joseph Doherty f8ad74dc19 fix(opcua): DeleteRawModified with an unspecified bound is BadInvalidArgument; HistoryUpdate node-manager tests
- A DateTime.MinValue start or end on DeleteRawModified is refused
  BadInvalidArgument before the backend.
- Tests: anonymous without the HistoryUpdate role denied (with it allowed),
  UpdateTagAttributes enabling historization sets HistoryWrite, a throwing
  updater is BadInternalError, the MinValue bounds case; corrected the
  test helper's comment on how the SDK base seeds unadmitted slots.
- GatewayHistorianValueWriter remarks describe the recorder's real status
  capture (and that it does not persist on the gateway).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 14:27:33 -04:00
Joseph Doherty 96ec751a0b fix(historian): latch the recorder's dropped-kind Debug log per run; pin the outbox.unreadable gauge
The DroppedUnsupportedKind counter still counts every value; only the Debug
line is latched (cleared by the next captured value), so a Gateway-backend
comms loss no longer logs one line per tag.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 14:23:42 -04:00
Joseph Doherty 7c787b8b64 test(host): an invalid SqlServer:Schema is consumer-gated, not a Resolve-time throw
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 14:22:41 -04:00
Joseph Doherty 8d51380ff1 fix(opcua): reverse HistoryRead-Raw paging no longer stops after the first page
A reverse read (start > end) with a small NumValuesPerNode resumed through
the tie-cluster path, which drained the cluster and stepped FORWARD to
T + 1 tick - outside a reverse window - so the read ended with GoodNoData
after one sample. SliceTieCluster now steps down to T - 1 tick for a
reverse read (terminal at the excluded end), and the node manager slices
the over-fetched cluster in the read's own direction so the skip count is
not applied from the wrong end. Forward paging is unchanged.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 14:20:00 -04:00
Joseph Doherty 7500106c2f feat(historian): SqlServerHistorianProvider + SqlServer backend registration
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:42:38 -04:00
Joseph Doherty a611cc358f Merge ph-T13 (pluggable historian)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:35:22 -04:00
Joseph Doherty 32852382cc test(historian): over-the-wire HistoryUpdate integration tests
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:07:19 -04:00