dohertj2/lmxopcua
1
0
Fork 0
Code Issues 32 Pull Requests Actions Packages Projects Releases Wiki Activity

Merge master into feat/677-aggregates

Browse Source 
# Conflicts:
#	docs/Historian.md
This commit is contained in:
Joseph Doherty
2026-10-11 11:30:46 -04:00
parent 27ca6cd791 8e6ac4b344
commit 9120182314
66 changed files with 2054 additions and 154 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Historian.md
+86 -19
View File
@@ -85,7 +85,8 @@ consumes the historian. The existing enable gates (`ServerHistorian:Enabled`, `A
| Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` | `ServerHistorian:QuestDb:*` |
Recording stays **Primary-only** (#537) and **at-least-once** with the #538 per-tag quarantine on every backend;
HistoryRead and HistoryUpdate are served from any node (the database is shared). The historian connection string is
HistoryRead is served from any node (the database is shared). HistoryUpdate is **Primary-only** since #674
(`BadInvalidState` on any other node — see [HistoryUpdate](#primary-only-674)). The historian connection string is
**not** a ConfigDb string, and a driver-only node may carry it without breaking the Phase 4 "no ConfigDb" posture.
---
@@ -145,6 +146,7 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an
| `MaxConcurrentHistoryReadBatches` | int | `16` | Process-wide limit on HistoryRead (and HistoryUpdate) batches served **concurrently** (arch-review 03/S3). A flood of history clients degrades to `BadTooManyOperations` rather than exhausting the SDK request-thread pool. With `HistoryReadBatchParallelism` this caps total in-flight gateway reads at `MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism` (default 16 × 4 = 64). A batch that cannot get a slot within `min(HistoryReadDeadline, 5 s)` fails every handle `BadTooManyOperations`. |
| `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` only (#671). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped. 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). |
| `HistoryReadDeadline` | TimeSpan | `00:01:00` | Per-request wall-clock deadline for a HistoryRead (and HistoryUpdate) batch (arch-review 03/S3). A node still in flight at the deadline surfaces `BadTimeout`, so one request can never hold an SDK request thread longer than this regardless of node count. Non-positive = unbounded (a `Validate()` warning). |
| `HistoryUpdateMaxFutureSkew` | TimeSpan | `1.00:00:00` | #674. How far past the server clock a HistoryUpdate (Insert / Replace / Update) item's `SourceTimestamp` may lie; an item beyond `now + skew` is refused **`BadOutOfRange`** by the node manager before any backend sees it (every backend). Guards against a far-future typo (a 2062 timestamp). The live recorder is not bounded. Non-positive disables the bound (no warning — a deliberate choice). See [HistoryUpdate](#historyupdate). |
**`ServerHistorian:SqlServer`** (read only when `Backend=SqlServer`; bound by `HistorianBackendRegistry.BindSqlServer`):
@@ -630,13 +632,31 @@ says is complete carries no continuation point. `NumValuesPerNode == 0` stays un
- The cursor is `(source, next start, end, boundary skip, cap)` in the same session-bound store as Raw (flagged as an events
cursor — a Raw point presented to an events read, or the reverse, is `BadContinuationPointInvalid`). The next page reads from
the last event's `EventTimeUtc` **inclusive** and drops the events at that timestamp already emitted.
the last event's `EventTimeUtc` **inclusive** and drops the events at that timestamp already emitted — and asks the source for
that many more rows than the cap, so every page short of the last is a FULL `NumValuesPerNode` page (#675; resumed pages used
to come back short by the skip).
- A tie cluster larger than the cap is paged within by over-fetching the half-open window `[T, T + resolution)` up to
`MaxTieClusterOverfetch`; beyond that bound the node answers `BadHistoryOperationUnsupported` (raise the bound).
- **Tie exactness depends on the backend.** SqlServer and QuestDB order by `(EventTimeUtc, EventId)`, so the skip count is exact.
The Gateway backend has no stable intra-timestamp order (and only a truncation flag), so ties at a page boundary are
best-effort there.
- Reverse event reads (`start > end`) are not paged server-side; no continuation point is issued for them.
- **Reverse event reads page newest-first (#675).** `StartTime > EndTime` reads the Part 11 reverse window `(EndTime, StartTime]`
with events in DESCENDING `(EventTimeUtc, EventId)` order, so a cap keeps the **latest** N and a continuation point is issued
when the source flags more. The direction is stored **in the cursor**: a reverse point resumes reverse (and a forward one
forward) whatever the resume request's details say. Reverse tie clusters are over-fetched forward and sliced from the newest
end, exactly as Raw does. Before #675 a capped reverse read silently returned the EARLIEST N, ascending, Good, with no
continuation point.
- **End-only events reads read backward (#675).** `StartTime` unset (`DateTime.MinValue`) with an `EndTime` and
`NumValuesPerNode > 0` pages backward from `EndTime` over `(MinValue, EndTime]` — the same rule as Raw. Start-only reads still
page forward to an open upper bound (`DateTime.MaxValue`, UTC). Without a cap an end-only read keeps its forward
`[MinValue, EndTime)` meaning.
- **Backends.** The server calls `IHistorianDataSource.ReadEventsAsync(…, bool reverse, …)`. SqlServer and QuestDB serve reverse
natively (`… > lo AND … <= hi ORDER BY time DESC, id DESC`). **Gateway and Null refuse a reverse read** (the interface default
throws `NotSupportedException`) and the node answers `BadHistoryOperationUnsupported` — the gateway streams ascending only and
rejects `end < start`, and draining the whole window to keep its tail would be exactly the unbounded read the cap prevents.
- **Gateway open upper bound.** The `DateTime.MaxValue` (UTC) bound of a start-only forward read is pinned offline through the
published client's proto `Timestamp` conversion (captured by an in-process gRPC server) and the FILETIME encoding
(`GatewayEventBoundsTests`). What a live AVEVA Historian does with a year-9999 bound is still unverified (needs the VPN).
- No session on the request (in-process callers) degrades to single-shot, as for Raw.
### Modified-value history (Part 11)
@@ -686,6 +706,25 @@ modification time and the Part 11 `HistoryUpdateType`. A HistoryReadRaw with `Is
within one node the order is strict.
- **Retention.** The trail ages out on the same **source-time** cutoff as the samples it describes: `SqlServer` purges `hist.SampleModified`
per tag alongside `hist.Sample`; QuestDB puts the same TTL on `{p}sample_modified`.
- **SqlServer schema v1 (#674).** With `AutoCreateSchema=false` a database still at schema **version 1** (script `002_sample_modified.sql` not
applied) keeps historizing — live writes, raw/processed/at-time reads, alarm events and retention all work — but has no trail: modified reads
and every HistoryUpdate item answer **`BadHistoryOperationUnsupported`** with nothing sent to SQL, retention skips `SampleModified`, and the
node logs **one Warning per boot**. Apply `002_sample_modified.sql` and restart the node to enable them (the verdict is taken once per
process). `AutoCreateSchema=true` (the default) still upgrades v1 to v2 in place. See [Schema](#schema).
- **QuestDB retry duplicates (#674).** `{p}sample_modified` is append-only with **no DEDUP key** (a key would collapse genuinely distinct
modifications of one sample), and each trail row gets a fresh `modified_id`. So a HistoryUpdate a client **retries** after a timeout — when the
first attempt had in fact landed — records a second trail row for the same change (and, for Insert, the retry is `BadEntryExists`). The
sample itself is idempotent (dedup upsert); only the audit trail can show the duplicate. `SqlServer` has the same exposure only if the client
retries a committed request; there is no request id to dedupe on in either backend.
- **Cross-node ordering (QuestDB).** The per-tag lock is per process; the [Primary-only](#primary-only-674) HistoryUpdate gate (#674) is what keeps
two nodes of a pair from recording concurrent updates of one tag (and so a wrong prior value). Two **separate** pairs sharing one QuestDB can
still race; that is unsupported.
- ⚠️ **Who can read the trail (#674.4).** Modified reads are ordinary HistoryReads: with `Security:Acl:Enabled=false` (the default) **any admitted
session, including Anonymous, can read who rewrote history** (`UserName` is the principal). Turn the ACL flag on, or set
`OpcUa:AllowAnonymous=false`, if the principal names are sensitive (see [security.md](security.md)).
- **Not implemented (#674.5):** `IsDeleteModified` (deleting trail entries — `BadHistoryOperationUnsupported`); recording an **Update** (upsert) as
a split `Insert` / `Replace` — it is recorded as `Update` either way; and **`AuditHistoryUpdateEvent`** (no audit event is emitted for a history
change — the trail is the only record).
```bash
otopcua-cli historyread -u opc.tcp://localhost:4840 -U apikey -P otua_<keyId>_<secret> \
@@ -758,7 +797,7 @@ The bootstrapper applies every script above the recorded version, in order, unde
| Object | Purpose |
|---|---|
| `SchemaVersion(Version, AppliedUtc)` | Applied-script ledger. Required version: **2** (since #668). |
| `SchemaVersion(Version, AppliedUtc)` | Applied-script ledger. Required version: **1**; version **2** adds the modification trail (#668). A v1 database is served with the trail features off (#674 — see below). |
| `Tag(TagId int identity, Name nvarchar(400) unique, DataType tinyint, EngineeringUnit, Description, CreatedUtc, UpdatedUtc)` | One row per historian tagname. `DataType` is a `DriverDataType`. |
| `Sample(TagId, SourceTimeUtc datetime2(7), ValueKind tinyint, NumValue float, IntValue bigint, StrValue nvarchar(max), StatusCode int, ServerTimeUtc)` | **PK `(TagId, SourceTimeUtc)`** — one value per timestamp (Part 11), which makes redelivery idempotent and rules out tie clusters. `StatusCode` is the uint status stored bit-for-bit. |
| `Event(EventId nvarchar(64) PK nonclustered, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, Message, UserName, Comment, EventTimeUtc, ReceivedTimeUtc)` | Alarm history. Clustered on `(EventTimeUtc, EventId)`; extra index `(SourceName, EventTimeUtc)`. |
@@ -774,11 +813,14 @@ everywhere (full 100 ns precision — never a `DataTable`, which would round to
`sqlcmd -S <server> -U <user> -P <pw> -d <db> -I -v Schema=hist -i docs/historian/sqlserver/001_initial.sql`, then the same with
`002_sample_modified.sql` (`-v Schema=` supplies the `$(Schema)` placeholder; verified against the docker-dev SQL container). Re-running is a no-op.
⚠️ **Upgrading an existing v1 historian (#668).** A build with this change requires schema **version 2**. With `AutoCreateSchema=true`
(the default) the first historian call after the upgrade applies `002_sample_modified.sql` in place (it only creates `SampleModified`; existing
samples are untouched). With `AutoCreateSchema=false` **every historian call fails** (`missing version 2 (found 1)`, in health `LastError`)
until an operator runs `002_sample_modified.sql` by hand as above — do that before rolling the new build out. Modifications made before the
upgrade have no trail; the trail starts with the first HistoryUpdate after it.
⚠️ **Upgrading an existing v1 historian (#668, #674).** With `AutoCreateSchema=true` (the default) the first historian call after the
upgrade applies `002_sample_modified.sql` in place (it only creates `SampleModified`; existing samples are untouched). With
`AutoCreateSchema=false` a **v1 database is accepted** (#674 — between #668 and #674 it refused every historian call, continuous-historization
drains included): writes, reads, events and retention work, while modified-value reads and HistoryUpdate report
`BadHistoryOperationUnsupported` without touching SQL, retention skips `SampleModified`, and one Warning per boot names the missing script.
Run `002_sample_modified.sql` by hand as above and **restart the node** to enable them (the version is read once per process). Only a schema
below version 1 (`missing version 1 (found 0)`) still fails every call. Modifications made before the upgrade have no trail; the trail starts
with the first HistoryUpdate after it.
### Writes
@@ -843,8 +885,10 @@ result never has both.
### Events
`[start, end)` by `EventTimeUtc`; **a reversed window (`end < start`) is swapped and returned ascending**. A null `sourceName` means all
sources, otherwise `SourceName = @source`. Ordered `(EventTimeUtc, EventId)` ascending. A bounded read fetches `maxEvents + 1` rows and
`[start, end)` by `EventTimeUtc`; through the five-argument read **a reversed window (`end < start`) is swapped and returned
ascending**. The direction-aware overload with `reverse: true` (what the node manager calls for a reverse read, #675) reads
`(lo, hi]` ordered `(EventTimeUtc, EventId)` **descending** — the clustered index scanned backwards. A null `sourceName` means all
sources, otherwise `SourceName = @source`. Forward is ordered `(EventTimeUtc, EventId)` ascending. A bounded read fetches `maxEvents + 1` rows and
returns a non-null continuation point when the extra row exists; the node manager turns that flag into an OPC UA continuation
point (see [Continuation-point paging (Events)](#continuation-point-paging-events-669)).
@@ -1042,6 +1086,8 @@ equivalent TTL (`QuestDb:RetentionDays`); the Gateway backend's retention is the
timeout 0) lets one purger run and the others skip that tick (Debug log).
- **Observability.** Counter `otopcua.historian.retention.purged{table=Sample|SampleModified|Event}`; a failed pass logs a Warning and retries next
interval — it never affects capture or reads.
- **Schema v1 (#674).** On a v1 schema (no `SampleModified`, `AutoCreateSchema=false`) the trail purge is skipped and omitted from the
pass results; samples and events purge normally.
- **Not done: partition switching.** For very large installs (billions of rows) a partitioned `hist.Sample` with sliding-window
`SWITCH`/drop is cheaper than row deletes; the purger is deliberately simple batched deletes. Another table is added by one more
`DeleteLoopAsync` call in `SqlServerHistorianRetention.PurgeAsync` (as `hist.SampleModified` was, #668).
@@ -1222,11 +1268,12 @@ Every query filters `NOT deleted`; every value goes through `StoredSample`, so t
from the last timestamp ± `TimestampResolutionTicks`.
- **AtTime:** previous-or-equal and next neighbour of each time by `LIMIT 1` seeks, `UNION ALL`-ed, **50 times per statement** (QuestDB 10.0.1 refuses
more than 126 branches). `AtTimeEvaluator` decides each value, as on `SqlServer`.
- **Events:** `[start, end)` by `ts`, ordered `(ts, event_id)`, `maxEvents + 1` fetched to produce the continuation point.
- **Events:** `[start, end)` by `ts`, ordered `(ts, event_id)`, `maxEvents + 1` fetched to produce the continuation point; a
reverse read (#675) is `(lo, hi]` ordered `(ts, event_id)` descending.
- **Processed:** one query per node; bucket index `(cast(ts AS LONG) * 10 - bucketBaseUnixTicks) / intervalTicks` (never `SAMPLE BY`, whose alignment
differs), so bucket edges are identical to `SqlServer` (forward start-inclusive / end-exclusive with the partial bucket last; reverse anchored at the
later time, later-edge-inclusive, partial bucket earliest — #677; interval 0 ⇒ one bucket, 100,000-bucket cap). A second `LIMIT 1` statement fetches
the tag's earliest live sample for the start-of-data Partial bit.
later time, later-edge-inclusive, partial bucket earliest — #677; interval 0 ⇒ one bucket, 100,000-bucket cap). On a forward read of any aggregate
but Average, a second `LIMIT 1` statement (capped at the window end) fetches the tag's earliest live sample for the start-of-data Partial bit.
Min/max times use `row_number()` windows with the earliest-timestamp tie-break.
### Retention (TTL)
@@ -1240,7 +1287,14 @@ expired day partitions itself, tombstones included.
re-read to verify; a very large `RetentionDays` that QuestDB normalises into MONTH/YEAR is re-`ALTER`ed on every boot (harmless).
- ⚠️ **TTL is enforced only on write commits.** QuestDB checks for expired partitions when a write commits to the table, so **a quiet table never
drops anything**: partitions older than the TTL linger until the next write lands. Do not read a stale-looking partition as a broken TTL.
- Expiry is judged against the wall clock, so a future-dated sample does not expire current data.
- **Expiry is judged against the wall clock** (`min(max timestamp, now)`), so a future-dated sample does not expire current data.
⚠️ **Keep `cairo.ttl.use.wall.clock` at its default, `true`** (#674). It is what QuestDB documents as protecting a TTL table from a
future-dated row; with it `false` the TTL would be measured from the table's newest timestamp and one far-future row (a 2062 typo) could
drop every older partition — including `{p}sample_modified`, the record of who changed history. Evidence (QuestDB 10.0.1, 2026-10-11): the
default is `true`, and a 2066 row inserted into a 30-day-TTL table evicted nothing inside the TTL, while a partition genuinely older than
30 days was evicted by that same commit — pinned by `TtlWallClockTests`. (The same run with the flag `false` also kept the in-TTL
partitions, so the test pins the outcome rather than isolating the flag; do not rely on that.) The server-side
`ServerHistorian:HistoryUpdateMaxFutureSkew` bound (default 1 day) is the primary guard: a HistoryUpdate cannot write a far-future row.
- The TTL is visible in table metadata: `SELECT table_name, ttlValue, ttlUnit FROM tables() WHERE starts_with(table_name, 'hist_')`.
### Docker-dev recipe
@@ -1281,8 +1335,8 @@ dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Integration
OPC UA **HistoryUpdate** lets a client insert, replace, update and delete stored raw history. It is available only when the selected
backend provides an updater — **`SqlServer` and `QuestDb` do; `Gateway` does not**, and an updater is registered only when `ServerHistorian:Enabled=true`.
Updates go straight to the backend (not through the outbox), are **not Primary-gated** (the database is shared), and share the HistoryRead
batch limiter and `HistoryReadDeadline`.
Updates go straight to the backend (not through the outbox), are **Primary-only** (#674, [below](#primary-only-674)), and share the
HistoryRead batch limiter and `HistoryReadDeadline`.
### Operations
@@ -1300,7 +1354,8 @@ caller's principal, in the [modification trail](#modified-value-history-part-11)
mixed (Part 11 reports item outcomes in `OperationResults`); it is Bad only for whole-node failures.
**Per-item rejections** (answered without touching SQL; other items still proceed): a null item ⇒ `BadInvalidArgument`;
`SourceTimestamp == DateTime.MinValue` ⇒ `BadInvalidTimestamp`; a value the historian cannot type (array, unsupported CLR type) ⇒
`SourceTimestamp == DateTime.MinValue` ⇒ `BadInvalidTimestamp`; a **`SourceTimestamp` later than `now + ServerHistorian:HistoryUpdateMaxFutureSkew`**
(default 1 day, server clock; #674) ⇒ **`BadOutOfRange`** — on every backend, never sent; a value the historian cannot type (array, unsupported CLR type) ⇒
`BadTypeMismatch`; a **`NaN` / `±Infinity` `Double` ⇒ `BadInvalidArgument`** — it cannot live in a SQL `float`, and unlike the live-value
writer (which stores it as `Empty`) an operator's explicit update is rejected rather than silently rewritten; and a **null value with a
Good or Uncertain status ⇒ `BadInvalidArgument`** — an `Empty` sample is always Bad, so a null value is accepted only with a Bad status
@@ -1323,7 +1378,19 @@ outcome, later ones `BadNoEntryExists`. **Unknown tag:** Insert / Update create
`ServerHistorian:Enabled=false`); `PerformUpdateType.Remove`; `DeleteRawModifiedDetails.IsDeleteModified=true` (the
[modification trail](#modified-value-history-part-11) is an audit record, aged out only by retention); and the out-of-scope detail types — **event** HistoryUpdate (`UpdateEventDetails` / `DeleteEventDetails`) and `UpdateStructureData` —
which are not overridden, so the SDK base rejects them. A backend exception is `BadInternalError` for that node and a deadline is `BadTimeout`.
No `AuditHistoryUpdateEvent` is emitted.
On a `SqlServer` database still at schema v1 every item is `BadHistoryOperationUnsupported` ([#674](#modified-value-history-part-11)).
No `AuditHistoryUpdateEvent` is emitted (#674.5, not implemented).
### Primary-only (#674)
HistoryUpdate is served only by the node holding the redundancy **Primary** data-plane role — the same verdict
(`IDriverPrimaryRoleSource`, published by `DriverHostActor`) that gates native-alarm acknowledges (#597). On any other node every node of
the request that **passed authorization** is refused **`BadInvalidState`** (node-level code, with a message pointing at the Primary) and
nothing reaches the backend. Authorization runs first, so an unauthorized caller learns `BadUserAccessDenied`, not the node's role.
HistoryRead is **not** gated. Why: rewriting history is a single-writer operation — both backends capture the prior value for the
[modification trail](#modified-value-history-part-11) under a per-tag, **per-process** lock (QuestDB) or statement (SqlServer), so two pair
nodes updating one tag concurrently could record a wrong prior value. A single-node deployment (no redundancy manager) is unaffected. Clients
should send HistoryUpdate to the server advertising the higher `ServiceLevel`.
### The `HistoryWrite` AccessLevel bit