Commit Graph

202 Commits

Author SHA1 Message Date
Joseph Doherty dbffb6bdb4 fix(historian): #691.3 strict-ceiling gateway: re-ask once at the named ceiling, never answer a truncated read 2026-10-12 16:13:14 -04:00
Joseph Doherty 39552e15b9 fix(historian): #690 Gateway raw/AtTime Bad rows carry no value (a stored NULL was value 0) 2026-10-12 16:08:54 -04:00
Joseph Doherty 8114f13a28 fix(historian): #687 review — one bounded re-read instead of failing, raw budget unclamped, exclusive forward end
- M1/M2: when the 1 ms pre-window margin (boundary row + stored samples, e.g. a
  >16-sample tie cluster at start-1ms / lastTs-1ms) uses up the cap + 16 budget,
  re-read ONCE asking for cap + m + 16 (m = pre-window rows seen); fail
  BadTooManyOperations only if the re-read is still short because of them. The
  raw budget is no longer clamped at the gateway default ceiling (AVEVA always
  emits the boundary row, so page sizes 999,985-1,000,000 always came back
  short); the #688 clamp stays on the per-sample aggregate path only.
- Merge trap: NumValuesPerNode 0 => int.MaxValue, never through the clamp.
- L3: the forward end is exclusive ([start, end), the IHistorianDataSource
  contract); a point read (start == end, the tie-cluster over-fetch) inclusive.
- RequestStartFor never goes before the Unix epoch (#686's walk-back floor).
- L4 (docs): an Uncertain native row for an all-NULL interval is unverified live.
2026-10-12 15:19:34 -04:00
Joseph Doherty 6796ba4126 fix(historian): #688 clamp the Gateway per-sample row budget at the gateway's default ceiling
The default MaxTimeWeightedRawRows (1,000,000) + 18 exceeded the gateway's
default MaxValuesCeiling (1,000,000), so with AllowStreamingBeyondCeiling=false
every Gateway per-sample aggregate read was refused. The allowance is now clamped
at 1,000,000 while the cap is at or under it (RawAggregateReader.
ClampToGatewayCeiling); a cap above it keeps its allowance. Raw reads are not
clamped (rebased onto #686).
2026-10-12 15:15:35 -04:00
Joseph Doherty 9e47532283 fix(historian): #687 Gateway raw reads drop AVEVA's synthetic start-boundary row
An AVEVA Full raw read leads with a row stamped at the query start (the value in
effect, Quality=133, or a no-data NULL row); it was returned as a stored sample.
Raw reads now use the #681 classification (RawAggregateReader.RequestStartFor /
IsWindowSample — one authority, by timestamp, since a stored NULL looks the same
on the wire): request from 1 ms early with NumValuesPerNode + 16 rows, keep rows
at or after StartTime, drain the whole stream. A full page still holds exactly
the cap, so #669 paging/resume/tie-cluster reads are unchanged; a margin that
swallows the allowance fails the node rather than ending paging early.
ReturnBounds is not plumbed on any backend (documented).
2026-10-12 15:15:16 -04:00
Joseph Doherty 44a03bedc0 fix(historian): #685 Gateway native TimeAverage/Total — drop AVEVA's pre-window row, keep Uncertain values
AVEVA answers one extra row for the interval before the window; it was mapped as
a result stamped before StartTime (N+1 values). Rows whose interval lies outside
the requested span (either direction) are now dropped after the drained stream.
An Uncertain (opc 64-191) interval kept no value (opc >= 192 was the only data
test); it now carries its value as UncertainDataSubNormal | Calculated.
Docs: AVEVA's TWA excludes NULL-gap time while the SQL backends bridge it;
end-to-end per-interval values also need historiangw#27.
2026-10-12 15:14:56 -04:00
Joseph Doherty 2d03268975 docs(historian): #686 review — reverse raw walk details, uncapped Gateway raw reads, tie-cluster status 2026-10-12 15:11:21 -04:00
Joseph Doherty be2f0cc14d docs(historian): #686 open-ended and reverse raw reads on every backend 2026-10-12 14:42:27 -04:00
Joseph Doherty cc0cc8cd89 fix(historian): #683 review — a refused tag ADD INDEX only warns; document index trade-offs
- Bootstrapper: an ALTER ... ADD INDEX refused up front (e.g. no DDL rights)
  logs a Warning and lets the bootstrap succeed (the index is optional); a
  WAL-apply failure remains unobserved, like the TTL ALTER — documented.
  Integration test (VARCHAR tag => refused ALTER) fails with the catch removed.
- Historian.md: a dominant tag can read ~20% slower on long windows (single-
  threaded index scan vs parallel JIT filter); O3 writes rewrite index files;
  sample_modified.tag / event.source deliberately not indexed, and why.
- CLAUDE.md: the concurrent-HistoryUpdate-delete caveat on the range-first order.
2026-10-12 10:15:53 -04:00
Joseph Doherty 549d2d68b3 perf(historian): index the QuestDB sample tag column; read the data range before the buckets (#683)
QuestDB: {p}sample.tag is now SYMBOL INDEX. The processed-read data-range
lookups (EarliestSample / LatestSample) were partition scans of the unindexed
symbol; on 53.6M rows / 120 tags / 61 daily partitions the indexed seeks are
1-1.5 ms warm / 2-4 ms cold whatever the tag's history (unindexed: up to 11 ms
warm, 14 ms cold; cost grew with the table). The bootstrapper adds the index to a
pre-#683 table (ALTER ... ADD INDEX, a WAL txn, idempotent, ~3 s for 68M rows);
AutoCreateSchema=false only logs the statement. Disk +~12%; write apply time
within noise; main per-sample/raw/time-weighted queries unchanged or faster.

Rejected by measurement/analysis: bounded existence probes (still full scans
when a side is absent), LATEST ON (slowest unindexed), per-tag first/last side
table (cannot track TTL drops or deletes INSERT-only).

Both backends now read the data range before the bucket statement: an append
racing the read reaches the buckets but not the range, so it can no longer
produce a Good Count of 0 for an interval that has samples.

Tests: TagIndexTests (new-table index, bound-param seek uses the index,
tombstones excluded / outage marker counted, pre-#683 migration with identical
processed results across 10 aggregates x 4 windows x both directions).
2026-10-12 10:15:48 -04:00
Joseph Doherty 78f604e0cd feat(historian): Gateway serves the per-sample aggregates from raw reads with the shared Sql.Common evaluator (#681)
Average, Count, Minimum, Maximum, Range, Start, End, Delta and Minimum/MaximumActualTime are evaluated
on the Gateway backend from forward ReadRaw reads with BucketPlan + AggregateEvaluator, so Gateway
matches SqlServer/QuestDB exactly (values, status bits, interval-start stamps, #677 reverse).

- Window read [start - 1 ms, end] so AVEVA's initial-value row never lands in a bucket; a pre-window
  row proves earlier data only with a non-Bad OPC quality (histsdk puts a value on every NULL row).
- Earlier data otherwise from a 2-row [1970, start) probe; later data from a 2-row [end, now+1d]
  look-ahead; both ignore rows stamped at their query start/end.
- No gateway stream is ever abandoned (a client-side cancel makes the gateway evict its shared
  historian session); past ServerHistorian:MaxTimeWeightedRawRows the bounded stream is drained, then
  BadTooManyOperations. The gateway's max_values-ceiling InvalidArgument maps to the same limit; a
  caller cancellation is not a backend fault.
- TimeAverage/Total stay AVEVA-native (no prior-sample read for exact Part 13 bounds), now stamped with
  the interval start (PENDING the live leg: histsdk's tail+43 start_time field is unverified) and
  carrying the Calculated bits.
- Served set: GatewayHistorianDataSource.ServedAggregates (12). Live AVEVA leg open (VPN).
2026-10-12 09:56:54 -04:00
Joseph Doherty c0d6bacd9c fix(historian): #684 review — Uncertain non-numeric samples do not refuse the Good-value aggregates; real-SQL Uncertain-above-max test
- Average/Minimum/Maximum/Range/*ActualTime refuse (BadAggregateInvalidInputs) only a GOOD
  String/DateTime sample (new GoodNonNumericCount / good_non_numeric_count column); an Uncertain
  one is skipped and makes the min/max family Uncertain, matching the SDK (Average skips it,
  ComputeMinMax flags the uncastable value). Delta keeps its usable-sample refusal.
- ProcessedConformance: an Uncertain value above the Good maximum (pins UncMax/unc_max through
  the real SQL) and an Uncertain String through the real SQL.
- The_uncertain_dataset_is_not_vacuous also proves the min/max STATUS discrimination.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-12 09:54:51 -04:00
Joseph Doherty a11ab0dc08 fix(historian): per-sample Average/Min/Max/Range/ActualTime use Good samples only, Part 13 min/max status (#684)
Part 13 defines Average/Minimum/Maximum/Range/*ActualTime over Good raw values
(independent of TreatUncertainAsBad) and the SDK's calculators skip Uncertain values
under the server's TreatUncertainAsBad=false. SqlServer/QuestDB value columns now use
the good flag (g) instead of usable (u); new UncMin/UncMax columns feed the Part 13
Minimum/Maximum/Range status rule (Uncertain on a Bad sample or an Uncertain value
beyond the Good extreme -- the SDK applies only the Bad half). An Uncertain-only
interval is BadNoData. Uncertain-bearing parity datasets (in-memory + real SQL) match
the SDK unmasked; the beyond-extreme divergence is pinned explicitly.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-12 09:24:09 -04:00
Joseph Doherty c019d0fc2b fix(historian): trim only HistoryServerCapabilities/AggregateFunctions; null set = all (#682 review)
ServerCapabilities/AggregateFunctions (i=2997) is Part 5's list of every
aggregate the server supports, and live AggregateFilter subscriptions still
compute all 37 SDK aggregates, so it keeps the SDK's full set; only i=11201
follows the backend. A null SupportedAggregates is treated as the interface
default (all) by both the advertisement and HistoryRead-Processed (it used to
NRE inside the batch). Docs: i=2997/i=11201 split and the boot window.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-12 09:18:53 -04:00
Joseph Doherty ede8610804 refactor(historian): Gateway served set behind GatewayHistorianDataSource.ServedAggregates (#682, merge seam for #681)
Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-12 08:57:19 -04:00
Joseph Doherty 6ea3ba44da feat(historian): advertise only the backend's aggregates under AggregateFunctions (#682)
IHistorianDataSource.SupportedAggregates (default: every HistoryAggregateType)
is the one authority for both ServerCapabilities/AggregateFunctions (i=2997) and
HistoryServerCapabilities/AggregateFunctions (i=11201) and for the up-front
BadAggregateNotSupported in HistoryRead-Processed. Gateway declares
AggregateModeMapper.Served (4), SqlServer/QuestDB AggregateFamilies.Served (17),
Null none. Only the folder references are trimmed; the SDK's calculator
factories stay so live AggregateFilter subscriptions keep working.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-12 08:52:47 -04:00
Joseph Doherty b5e3e5108c docs(historian): #680 lane B review — QuestDB data-range lookups are partition scans, both-direction gate, all-Bad Count 0x80000401, outage-last parity test, CLAUDE.md per-sample bullet 2026-10-11 16:39:39 -04:00
Joseph Doherty 0d206a0884 fix(historian): #680 review — recovery/only-sample TA2/Total2 value, outage-last + Boolean parity, docs
TimeAverage2/Total2: the interval the data ends inside with no non-Bad time
before its last sample (comms-loss recovery, a single sample) answers that
sample's value (Total2 0), Uncertain + Partial, instead of Bad/BadNoData —
as the SDK. Parity datasets RecoveryEnds, OutageLast, BoolSwitch with
documented masks; ProcessedConformance's reverse TA/Total mask narrowed to
the later-edge-sample interval. Docs: report-by-exception consequence,
outage-last and Boolean differences; CLAUDE.md historian bullet.
2026-10-11 16:02:22 -04:00
Joseph Doherty fb5ec80fe6 fix(historian): #679 Gateway — Average/Count not supported (no AVEVA equivalent), TimeAverage → TimeWeightedAverage 2026-10-11 15:19:21 -04:00
Joseph Doherty beb1c583b1 fix(opcuaclient): #676 live gate A1 — a sustained closed connection is a definitive loss
Live gate A1 (docker-dev, opc-plc restart / SIGINT / kill): the fast path never fired —
a real process stop delivers no session status; the keep-alives report
BadSecureChannelClosed (0x80860000) once, then BadConnectionClosed (0x80AE0000)
interleaved with BadNoCommunication for the whole outage.

Reproduced in-process with TcpStallProxy.HardClose (reset every connection, refuse new
ones; server and session alive behind it): the identical sequence, and the SDK 1.5.378
session never opens a new connection on its own (48 s with the proxy accepting again,
zero attempts) — a closed channel cannot recover in place.

The driver now treats BadSecureChannelClosed/BadConnectionClosed as definitive on the
SECOND closed-channel keep-alive failure in one outage (a good keep-alive resets the
count), raising ConnectionLost ~1 keep-alive after the drop. BadNoCommunication /
BadRequestTimeout / BadNotConnected (stall/pause) stay with the watchdog; the stall test
still sees no ConnectionLost. New wire test (hard close + reopen ⇒ raised once, SDK never
reconnects, reinit brings the identical value back Good) and unit tests (sustained raises
once; a single blip that recovers does not) — all red on the previous code. Docs updated.
2026-10-11 12:34:52 -04:00
Joseph Doherty 945ec2c069 Merge feat/677-aggregates: reverse processed reads SDK-anchored, start-of-data Partial (forward), AggregateConfiguration accept-defaults, IntegralDivisor=1, SDK parity tests (#677) 2026-10-11 11:47:52 -04:00
Joseph Doherty 7349260231 fix(historian): #677 review — start-of-data Partial forward-only, bounded/skipped earliest-sample query, wider SDK parity
F1: the SDK applies its m_startOfData Partial rule to forward reads only; ours
now does too (AggregateEvaluator.EvaluateAll + TimeWeightedAggregator). New
reverse late-start parity tests through the SDK oracle (unit) and real SQL
(conformance, both backends).
F2: AggregateEvaluator.NeedsEarliestSample — the earliest-sample query runs
only for forward reads of aggregates other than Average, and is bounded by
the window end (SqlServer < @end, QuestDB ts < $2).
F3: per-sample SDK parity extended to Range, Start, End, Delta, Min/Max
ActualTime and PercentGood/Bad (oracle maps StatusAggregateCalculator);
pre-existing definitional differences masked and marked #680.
Docs: Historian.md Partial/SDK-parity sections, CLAUDE.md bullet.
2026-10-11 11:47:22 -04:00
Joseph Doherty 50c09e21d5 Merge feat/676-opcuaclient-followups: endpoint host+port rewrite, TCP-stall wire test, ReconnectPeriod removed, IDriverConnectionLossSource fast recovery, trailing re-read, replay summary (#676) 2026-10-11 11:45:26 -04:00
Joseph Doherty 232c03ed1b fix(historian): #674 review follow-ups — HistoryUpdate boot-window gate, node-level v1 refusal, plan-before-gate, UTC skew compare
- HistoryUpdate Primary gate now reads IHistoryUpdateRoleView (PrimaryGatePolicy.ShouldServeHistoryUpdate):
  the data-plane verdict but CLOSED until this node's first redundancy snapshot, so both pair nodes cannot
  accept in the pre-join boot window. Single nodes open on their own singleton's first snapshot; admin-only
  nodes register no view => no gate. DriverHostActor publishes it at every Primary-verdict publish point.
- SqlServer v1: the updater throws NotSupportedException on every arm; the node manager maps it to a
  node-level BadHistoryOperationUnsupported (was per-item under a Good node for UpdateData/DeleteAtTime).
  HistoryWrite bit still advertised on v1 (not knowable at materialise time) — documented.
- The update plan is evaluated before the Primary gate: an unservable request (Remove, IsDeleteModified,
  not historized, no updater) is BadHistoryOperationUnsupported off-Primary, not BadInvalidState.
- Local-kind SourceTimestamp converted to UTC before the future-skew compare and sent as UTC.
- Hosted-service wiring extracted to BuildHistoryUpdatePosture and tested (gate source, skew, clock).
- Docs: Historian.md (boot window, ordering, node-level v1, HistoryWrite bit), CLAUDE.md.

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-11 11:41:11 -04:00
Joseph Doherty 30a9d5f094 fix(opcuaclient): #676 review fix-ups (port-less rewrite, full reconcile, delivery lock, timers)
1. RewriteEndpointUrl keeps the advertised port when the configured URL names none
   (Uri.Port -1 would have sent the session to 4840); port-less + IPv6 tests.
2. The trailing re-read repairs any (value, status) difference that is not older than
   the last delivered value — Good->Bad/Uncertain included; a read without a source
   timestamp counts as newer; an equal source time with different content is published
   (repairs the replay guard's by-design equal-timestamp drop).
3. One driver-wide delivery lock makes decide+publish atomic across live notifications,
   the recovery read and the trailing read, so an older value can never land after a
   newer one (deterministic race test via a seam).
4. Known gap documented: a ConnectionLost raised while the host is not Connected is
   swallowed and that session falls back to the watchdog.
5. Only a SERVER-reported subscription loss raises ConnectionLost; the publishing-stall
   heuristic stays with the #527 watchdog.
6. Post-recovery timers run on Task.Delay(…, Clock, ct) with a session-work CTS
   cancelled on session release and dispose.
7. Replay summary at Information only when something was dropped (Debug otherwise);
   a re-arm reports the superseded window's drops and counts afresh.
8. DriverInstanceActor: AttachHostStatusSource doc comment restored; Reconnecting
   swallows a stray DisconnectObserved (test).
9. Stall wire test: 120 s session timeout, 3000-cycle subscription lifetime, one further
   bad keep-alive tick, 60 s headroom on the post-recovery waits (5/5 green).
2026-10-11 11:35:53 -04:00
Joseph Doherty fccffd0a0f feat(opcuaclient): #676 follow-ups — endpoint rewrite, stall wire test, fast definitive-loss recovery
- Endpoint rewrite: the driver now connects to the configured host AND port (scheme
  match, advertised path kept) instead of the server-advertised URL — fixes NAT/port-map
  upstreams and lets an in-test proxy carry the session (SDK SelectEndpoint rule).
- TcpStallProxy + OpcUaClientDriverStallWireTests: the in-place blip (#670) is now
  wire-tested on loopback — one Bad, frozen read stamp, same session, identical value
  republished Good, a post-recovery change arrives, no source-time regression after
  the SDK's late backlog flush.
- Defect found by that test, fixed: a change published right after a stall rode a
  Publish response the SDK timed out and never republished, leaving the node Good on
  the pre-change value indefinitely. New trailing reconcile read (Timeout +
  KeepAliveInterval after recovery) publishes any Good value newer than the last
  accepted one, through the replay guard's mark.
- IDriverConnectionLossSource (Core.Abstractions): opt-in definitive-loss signal.
  DriverInstanceActor attaches in PreStart / detaches in PostStop and turns a raise
  into a disconnect only while Connected. OpcUaClient raises once per session on
  keep-alive BadSessionIdInvalid/BadSessionClosed/BadSessionNotActivated/
  BadServerHalted (measured: the SDK never re-opens a halted channel) or a dropped
  subscription; ambiguous codes stay with the #527 watchdog.
- Replay guard: one Information summary per recovery when the window closes
  ("dropped N late notifications across M refs"), with a timer so quiet
  subscriptions log it too.
- ReconnectPeriod deleted (options, AdminUI form, docs, tests); stored configs that
  carry reconnectPeriod still parse in the factory, probe and browser.
- Docs: OpcUaClient.md, DriverLifecycle.md, driver-specs.md, CLAUDE.md.
2026-10-11 11:35:52 -04:00
Joseph Doherty 9120182314 Merge master into feat/677-aggregates
# Conflicts:
#	docs/Historian.md
2026-10-11 11:30:46 -04:00
Joseph Doherty 27ca6cd791 feat(historian): #677 Part 13/SDK reverse anchoring, start-of-data Partial, AggregateConfiguration, IntegralDivisor
- BucketPlan: reverse processed reads anchored at the later time for every
  aggregate: window (end, start], partial interval earliest, results stamped
  with the later edge, latest first (SDK CreateSlice/GetTimestamp/backward
  GetValues). SqlServer adds @bucketAdjustTicks + flips the window predicate;
  QuestDB binds BucketBaseUnixTicks. IndexOf is the C# statement of the math.
- Partial bit on the interval the tag's data starts inside (SDK m_startOfData),
  every aggregate but Average; per-sample reads fetch the earliest sample.
- Node manager: non-default client AggregateConfiguration =>
  BadAggregateConfigurationRejected per node (ServerAggregateConfiguration,
  pinned to the SDK AggregateManager defaults).
- GatewayTagProvisioner sends IntegralDivisor = 1.
- SDK parity oracle (Opc.Ua.Server calculators) in unit + both backends'
  conformance suites, forward and reverse; remaining divergences documented.
- docs/Historian.md + CLAUDE.md release note (client-visible).
2026-10-11 11:26:00 -04:00
Joseph Doherty 0658d244d0 feat(historian): #674 HistoryUpdate posture — future-skew bound, Primary gate, SqlServer v1 accepted, QuestDB TTL pin
- HistoryUpdate items with SourceTimestamp > now + ServerHistorian:HistoryUpdateMaxFutureSkew
  (default 1 day, <=0 disables) => BadOutOfRange in the node manager (every backend, never sent);
  clock injected via OtOpcUaNodeManager.Clock (host TimeProvider).
- HistoryUpdate is Primary-only: off-Primary => BadInvalidState per node, after authorization,
  same IDriverPrimaryRoleSource verdict as the native-ack gate; HistoryRead ungated.
- SqlServer: base schema requirement back to v1; SupportsModifiedTrail (v>=2). On v1 with
  AutoCreateSchema=false: writes/reads/events/retention work, modified reads + HistoryUpdate =>
  BadHistoryOperationUnsupported with nothing sent, retention skips SampleModified, one Warning per boot.
- QuestDB integration test pinning wall-clock TTL (far-future row evicts only partitions older
  than the TTL by the wall clock), with a positive control.
- Docs: Historian.md, Configuration.md, security.md, CLAUDE.md (incl. QuestDB retry duplicates,
  #674.4 anonymous-readable trail with ACL off, #674.5 not implemented).

Co-Authored-By: Claude <noreply@anthropic.com>
2026-10-11 11:18:44 -04:00
Joseph Doherty 062086f9a6 Merge feat/668-modified-values: Part 11 modified-value history (IsReadModified) on SqlServer + QuestDB (#668) 2026-10-11 09:32:13 -04:00
Joseph Doherty cfd403710e feat(historian): Part 11 modified-value history on SqlServer + QuestDB (#668)
Every HistoryUpdate a SQL backend applies now leaves a modification trail
(prior value, principal, Part 11 update type, modification time), and
HistoryReadRaw with IsReadModified=true serves it as HistoryModifiedData.

- Core: HistoryUpdateContext threaded into all three IHistorianDataUpdater
  methods; IHistorianDataSource.ReadModifiedAsync (default NotSupported =>
  BadHistoryOperationUnsupported on Gateway/Null); HistoryModified* contracts.
- SqlServer: schema v2 (002_sample_modified.sql, RequiredVersion=2, docs copy);
  trail written by OUTPUT ... INTO on the same MERGE/DELETE statement, so a
  failed update leaves no audit row; retention purges SampleModified per tag on
  the source-time cutoff; ReadModifiedAsync keyed on (SourceTimeUtc, ModifiedId).
- QuestDB: append-only {prefix}sample_modified (TTL like samples), trail row
  written BEFORE the change (orphan-on-crash, never an untracked change);
  existence reads now carry the prior value; ReadModifiedAsync.
- Node manager: modified reads dispatch + page via the #669 continuation store
  (HistoryContinuationState gains HistoryCursorKind + LastModifiedId), gated by
  the existing HistoryRead decision; session principal / key id passed to the
  updater. IsDeleteModified stays unsupported.
- Client.CLI: historyread --modified.
- Tests: node-manager dispatch/paging/cursor binding/principal threading,
  over-the-wire HistoryModifiedData, shared ModifiedValueConformance on both
  backends, SqlServer atomicity (forced trail failure => no change) + v1->v2
  upgrade + retention trail purge.
- Docs: Historian.md (Modified-value history section, schema v2 upgrade note),
  security.md, Client.CLI.md, CLAUDE.md.
2026-10-11 09:26:36 -04:00
Joseph Doherty 3b739b982b Merge feat/671-time-weighted-aggregates: Part 13 time-weighted aggregates (#671) 2026-10-11 09:10:52 -04:00
Joseph Doherty 56bb4855bb fix(historian): #671 review — raw-row cap, reverse Interpolative, SDK status rule, Partial bit
M1: the time-weighted family (incl. Total, formerly a pushed-down SUM) streams
raw window rows into the C# evaluator. New shared knob
ServerHistorian:MaxTimeWeightedRawRows (default 1,000,000; <=0 unbounded with a
Validate warning) flows through HistorianBackendRegistry into both providers.
The window query is capped at cap+1 (SQL Server TOP (@maxRows), QuestDB LIMIT $4);
one row past the cap throws the new Core.Abstractions
HistoryReadLimitExceededException, which the node manager maps to
BadTooManyOperations for that node. Not counted as a backend health failure;
bound rows are not counted; per-sample aggregates are not capped.

M2: reverse Interpolative is anchored at the later time (Part 13): points at
End - i*interval, emitted latest first. 10:00->09:00 @1h yields 10:00.

L1: Partial bit (0x4, = Opc.Ua.AggregateBits.Partial) on non-Bad results of a
truncated last interval (time-weighted family; per-sample family is a follow-up).

L2/L3: aligned with the SDK AggregateCalculator.GetTimeBasedStatusCode
(decompiled from Opc.Ua.Server 1.5.378.106): TreatUncertainAsBad=false counts
Uncertain time (Uncertain samples, held-into-outage, extrapolation) as Good
time; data but no non-Bad time => generic Bad (0x80000000); BadNoData only
when there is no data at all. Consequence: a live tag's latest interval reads
Good (documented).

L5: release note no longer claims Total matches the Gateway Integral exactly
(AVEVA's divisor is unverified); it notes Total is now evaluated in C# and
subject to the cap.

Tests: unit (reverse Interpolative x3, Partial, Bad vs BadNoData incl. an
open-ended outage, cap helpers, SDK bit oracles); conformance (reverse
Interpolative, raw-row cap at/over/0, bounds not counted, health unaffected);
node manager BadTooManyOperations; options default + warning. Each new
behaviour falsified by mutation (aggregator x9, data-source cap checks on
both backends against the real DBs, node-manager catch).
2026-10-11 09:10:26 -04:00
Joseph Doherty 1f88669304 feat(historian): Part 13 time-weighted aggregates on SqlServer + QuestDb (#671)
New AggregateFamily.TimeWeighted serving TimeAverage, TimeAverage2, Total,
Total2 and Interpolative, evaluated by the shared Sql.Common
TimeWeightedAggregator for both SQL backends (identical results by
construction). HistoryAggregateType gains TimeAverage2 + Total2 (appended);
OtOpcUaNodeManager.MapAggregate maps their NodeIds. Gateway's mapper already
refuses unknown defined members => BadAggregateNotSupported per node.

Semantics (server defaults, client AggregateConfiguration ignored):
TreatUncertainAsBad=false, PercentDataBad=PercentDataGood=100,
UseSlopedExtrapolation=false. Sloped for Double/Int64, stepped for Boolean;
a usable sample followed by a Bad one is held up to the outage (AtTime rule);
stepped Uncertain extrapolation past the last sample. TimeAverage/Total use
interpolated bounding (bridge outages); TimeAverage2/Total2 exclude Bad time.
Bounding values: latest usable + latest any before start, first any + first
usable at/after end (four TOP 1 / LIMIT 1 seeks).

CLIENT-VISIBLE: Total on SqlServer/QuestDb was SUM of samples; it is now the
time-weighted integral (TimeAverage x interval seconds), matching Gateway's
Integral. Release note in docs/Historian.md.

Trade-off (deviation from the preferred hybrid pushdown, allowed by the
design): the time-weighted family is NOT pushed down. Each read streams the
window's rows plus the four bounds (SQL Server: one five-result-set batch;
QuestDB: bounds UNION ALL + window select) into the C# evaluator. Network is
O(samples in window), memory O(buckets). Writing the per-segment
interpolation/edge splitting twice in two SQL dialects and keeping them
bit-identical was judged the bigger risk; this also makes the QuestDB
lag/lead spike moot.

Tests: 35 hand-computed TimeWeightedAggregator unit cases (each of 10
implementation mutations goes red); served-aggregate pin 13 -> 17;
conformance sparse fixture (intervals wider and narrower than sample gaps)
+ an outage straddling both window edges (each of the four SQL seeks
disabled in turn goes red); node-manager NodeId mapping test.
2026-10-11 08:44:44 -04:00
Joseph Doherty db6446ce73 Merge fix/670-opcuaclient-session-loss: OpcUaClient upstream session loss goes Bad and recovers (#670) 2026-10-11 08:41:48 -04:00
Joseph Doherty 91434e7f61 Merge fix/672-capture-gap: capture while the recording Primary is unreachable (#672) 2026-10-11 08:29:59 -04:00
Joseph Doherty 2a8f9af5e4 fix(opcuaclient): upstream session loss goes Bad and recovers (#670)
A bad keep-alive now degrades health (LastSuccessfulRead preserved) and,
once per outage, publishes BadNoCommunication (value retained) for every
subscribed ref plus a quality-only annotation on subscribed alarm
conditions. Good keep-alives and data/event notifications advance
LastSuccessfulRead, so an idle-but-alive subscription never looks dead
to the #527 watchdog. A keep-alive that recovers on the same session
re-reads the Bad refs and republishes them Good; a session that does not
come back is recovered by the watchdog's ShutdownAsync + InitializeAsync
+ re-subscribe.

Deletes the SessionReconnectHandler wiring (it retried forever in SDK
1.5.378, cloned subscriptions the driver never re-tracked and kept health
Healthy), OnReconnectComplete, TryRearmReconnect and the reconnect_*
counters (new keepalive_recovered). Teardown of a lost session skips
wire deletes and bounds CloseSession. SDK warnings reach the driver log.
Wire test reworked: restart => Bad, then watchdog-style reinit => Good.
2026-10-11 08:27:04 -04:00
Joseph Doherty 1b2c36eb87 fix(historian): capture while the recording Primary is unreachable (#672)
A SIGKILLed Primary stays an unreachable-but-not-downed member until
auto-down, and the redundancy singleton usually dies with it, so the
standby's cached snapshot kept naming it Primary and nobody captured for
~30 s. DriverHostActor now holds its own ReachabilityEvent (+MemberUp)
subscription, caches the last snapshot, and excludes unreachable peers
from the historization peer-is-Primary predicate only, republishing just
the capture verdict on a reachability change. Alarm drain and the Primary
data-plane verdict are unchanged. Gated by
ContinuousHistorization:CaptureWhilePrimaryUnreachable (default true).

Tests: message-driven cases with real ReachabilityEvents (incl. a
falsification control on the drain/data-plane verdicts) plus a real
two-node cluster crash test with downing disabled.
2026-10-11 08:26:39 -04:00
Joseph Doherty 924eff8d3c feat(historian): SQL Server retention purge (RetentionDays, Primary-gated, applock) (#667) 2026-10-11 08:21:30 -04:00
Joseph Doherty 94691ec50f docs(historian): QuestDB backend documentation (as built)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-11 03:09:11 -04:00
Joseph Doherty 60a1e7e669 docs(historian): fix-wave docs and stale comments
- Historian.md: Empty samples always Bad (writer/updater/read clamp),
  long-name refusal, 5 s bootstrap failure cache, AtTime after now, info-bit
  masking, HistoryUpdate does NOT type-check (M-1 decision), DeleteRaw
  needs both bounds, reverse Raw paging, the historian/historization metric
  prefixes, and provisioning numeric-only lines qualified as Gateway.
- CLAUDE.md numeric-only lines qualified as Gateway backend.
- Stale 'sole historian' lines in DriverLifecycle, ServiceHosting and the
  retired Wonderware stub; Client.CLI exit-code contract.
- ServerHistorianOptions: Endpoint/Backend XML docs untangled, Validate()
  indented; Program.cs comments are backend-neutral; T10 week-test day count.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 14:33:30 -04:00
Joseph Doherty cc34daf181 docs(historian): pluggable backends, SQL Server backend, HistoryUpdate, outbox v2, spec corrections
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 13:51:56 -04:00
Joseph Doherty 1ea1da8236 refactor(runtime): delete the dead in-place ApplyDelta seam; config change = respawn only (#665)
Since #516 a DriverConfig change is stop + respawn, so DriverSpawnPlan.ToApplyDelta
was always empty and the in-place path had no producer. #663 and #664 were both
latent bugs in it. This deletes it:
- host: ApplyChildDelta, _pendingDelta, HandleApplyResult and HandleDeltaApplied
- plan: DriverSpawnPlan.ToApplyDelta
- actor: the ApplyDelta, ApplyResult and DeltaApplied messages, the Connected
  reinit handler and the mid-connect AdoptConfigDuringInit
- tests: the actor tests that drove the seam

IDriver.ReinitializeAsync and each driver's implementation and unit tests stay.
Comments and docs that cited AdoptConfigDuringInit as the reason a driver must
release prior state at init now cite the IDriver contract and overlapping
retries, which still apply.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 11:23:16 -04:00
Joseph Doherty 97fcf624ae fix(drivers): mid-connect ApplyDelta adopts config in Calculation, refuses it in Galaxy (#664)
AdoptConfigDuringInit re-runs InitializeAsync(newConfig) and acks success, but
Calculation and Galaxy ignored the JSON there: Calculation kept its old tag
set, and Galaxy connected on its old endpoint. Calculation now rebuilds its tag
table from a config with a body (sharing AdoptConfig with ReinitializeAsync and
stopping any running timer scheduler first). Galaxy, which cannot hot-swap its
options, refuses a non-equivalent one the way ReinitializeAsync does. The actor
now sends DeltaApplied once the adopted init succeeds, so a dependency
consumer's mux interest is re-registered. Unreachable from deploys since #516.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 11:05:58 -04:00
Joseph Doherty 39c5af3d7f fix(runtime): in-place ApplyDelta releases and re-subscribes data + alarms (#663)
A Connected ApplyDelta called ReinitializeAsync and kept the actor's data and
native-alarm handles, though every driver's reinit drops the driver-side
subscriptions behind them (#630). The reconcile only re-subscribes a missing
handle, so the driver went silent until a reconnect. Release both at the
driver before the reinit and re-subscribe both in a finally. Unreachable from
deploys since #516; pinned by a two-case test, each case failing against a
different wrong fix.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 07:49:56 -04:00
Joseph Doherty 5067509dd4 fix(runtime): keep native-alarm feed across data re-subscribes; reconcile retries alarms (#662)
A data re-subscribe (every deploy's bulk SetDesiredSubscriptions push to a
kept driver) dropped the prior data subscription through the full
DetachSubscription, which also unhooked OnAlarmEvent and forgot the alarm
handle. On any deploy that did not respawn the driver, the alarm
subscription kept logging "established" while every event reached nobody
until a reconnect or respawn. The data path now uses DetachDataSubscription;
only a lost connection and PostStop detach both.

The #488 health-poll reconcile only looked at data, so an alarm subscribe
that failed on (re)connect gated the feed for the life of the connection.
ReconcileAlarmSubscription now re-attaches a missing handler and
re-subscribes a stale/missing alarm handle every tick.

OpcUaClient removes a half-built alarm subscription from its session when
SubscribeAlarmsAsync throws, so the new retries cannot accumulate.

Tests: actor-level data-resubscribe + reconcile-retry tests and an
end-to-end NativeAlarmFeedSurvivesDeployTests through DriverHostActor; all
three fail against the unfixed actor.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 07:30:25 -04:00
Joseph Doherty ea220363f3 Merge hw5-657 (#657)
v2-ci / build (push) Successful in 5m44s
v2-ci / unit-tests (push) Failing after 24m4s
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 07:15:30 -04:00
Joseph Doherty a7c27e978f docs(abcip): #592 native array-element write on Z35162 (_SEQ_EP[31], user-approved, restored)
Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 06:41:14 -04:00
Joseph Doherty 1f983d610f test(abcip): hw5 hardware gate — #566 idle close drops TCP; native STRING write on Z-35162 (#592)
A. #566 on a real CompactLogix (Z-35162, read-only): a class-3 connection idled 60 s past its
16 s timeout is closed at the TCP level, not with CIP 0x0107/0x0203; the next request fails on
its write (Broken pipe -> BadConnection) in 9 ms and the one after reopens. The transport already
handles this, so no production behaviour changed; the shape is pinned offline by
CipConnectionTests.A_target_that_resets_an_idle_session_... (FakeEipServer.ResetAllConnections),
proven red with a reconnect-skipping mutation. The diagnostics-only exchange tap now also sees
Forward Open replies (granted RPIs).

B. Native STRING write through the driver on Program:ES03_TemperatureControl._StrTmp: write,
read back, restored byte-exact (user-approved). Array write NOT run: the census's unreferenced
DINT[32] _SEQ_EP arrays are aliased sequencer step registers (raw L5X grep), so none qualifies.
#567 still impossible (a 128 B array / 88 B STRING never fragments).

New env-gated harness: AbCip.IntegrationTests/HardwareGate (IdleConnectionEvidenceTests,
StringWriteGateTests). Docs: gate results (hw5 section + raw output), AbCip.md, CLAUDE.md.

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 06:16:16 -04:00
Joseph Doherty 48a80b5962 fix(alarms): atomic InactiveUnacknowledged + flagged Clear/Acknowledge first sight for OpcUaClient (#657)
After an OpcUaClient respawn an upstream inactive+unacked condition went out as
Retrigger+Clear (duplicate Activated+Cleared rows, Active flicker), and a
first-sight inactive+acked one as Unspecified (an Activated row for an inactive
alarm; a retained-Active host condition never cleared).

- New AlarmTransitionKind.InactiveUnacknowledged: one projection, rows only for
  what changed (Activated+Cleared from acked, Cleared from active+unacked, none
  when already inactive+unacked).
- OpcUaClient uses it for moves into inactive+unacked; first sight of an
  inactive condition is InactiveUnacknowledged or Clear+Acknowledge, flagged.
- Projector: a flagged Acknowledge on an already-acked condition writes no row.
- E2E tests through DriverHostActor.ForwardNativeAlarm (red-proven).

Claude-Session: https://claude.ai/code/session_01Ef2ozxS7QPanGT8uoATbTe
2026-10-09 06:08:39 -04:00