feat(historian): #674 HistoryUpdate posture — future-skew bound, Primary gate, SqlServer v1 accepted, QuestDB TTL pin
- HistoryUpdate items with SourceTimestamp > now + ServerHistorian:HistoryUpdateMaxFutureSkew (default 1 day, <=0 disables) => BadOutOfRange in the node manager (every backend, never sent); clock injected via OtOpcUaNodeManager.Clock (host TimeProvider). - HistoryUpdate is Primary-only: off-Primary => BadInvalidState per node, after authorization, same IDriverPrimaryRoleSource verdict as the native-ack gate; HistoryRead ungated. - SqlServer: base schema requirement back to v1; SupportsModifiedTrail (v>=2). On v1 with AutoCreateSchema=false: writes/reads/events/retention work, modified reads + HistoryUpdate => BadHistoryOperationUnsupported with nothing sent, retention skips SampleModified, one Warning per boot. - QuestDB integration test pinning wall-clock TTL (far-future row evicts only partitions older than the TTL by the wall clock), with a positive control. - Docs: Historian.md, Configuration.md, security.md, CLAUDE.md (incl. QuestDB retry duplicates, #674.4 anonymous-readable trail with ACL off, #674.5 not implemented). Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Joseph Doherty
+58
-12
@@ -85,7 +85,8 @@ consumes the historian. The existing enable gates (`ServerHistorian:Enabled`, `A
|
||||
| Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` | `ServerHistorian:QuestDb:*` |
|
||||
|
||||
Recording stays **Primary-only** (#537) and **at-least-once** with the #538 per-tag quarantine on every backend;
|
||||
HistoryRead and HistoryUpdate are served from any node (the database is shared). The historian connection string is
|
||||
HistoryRead is served from any node (the database is shared). HistoryUpdate is **Primary-only** since #674
|
||||
(`BadInvalidState` on any other node — see [HistoryUpdate](#primary-only-674)). The historian connection string is
|
||||
**not** a ConfigDb string, and a driver-only node may carry it without breaking the Phase 4 "no ConfigDb" posture.
|
||||
|
||||
---
|
||||
@@ -145,6 +146,7 @@ and all HistoryRead calls on historized nodes return `GoodNoData` (empty, not an
|
||||
| `MaxConcurrentHistoryReadBatches` | int | `16` | Process-wide limit on HistoryRead (and HistoryUpdate) batches served **concurrently** (arch-review 03/S3). A flood of history clients degrades to `BadTooManyOperations` rather than exhausting the SDK request-thread pool. With `HistoryReadBatchParallelism` this caps total in-flight gateway reads at `MaxConcurrentHistoryReadBatches × HistoryReadBatchParallelism` (default 16 × 4 = 64). A batch that cannot get a slot within `min(HistoryReadDeadline, 5 s)` fails every handle `BadTooManyOperations`. |
|
||||
| `MaxTimeWeightedRawRows` | int | `1000000` | `SqlServer` / `QuestDb` only (#671). The most raw in-window samples one [time-weighted aggregate](#time-weighted-aggregates-part-13-671) read (TimeAverage, TimeAverage2, **Total**, Total2, Interpolative) may stream into the server-side evaluator — that family is **not** pushed down to SQL. The window query is capped at this + 1 rows; one row past the cap fails **that node** with `BadTooManyOperations` (not counted as a backend fault) instead of holding a pooled connection until `HistoryReadDeadline` and returning `BadTimeout`. Bound rows outside the window do not count; per-sample aggregates are not capped. 1 000 000 ≈ 11.5 days of a 1 Hz tag. Non-positive = unbounded (a `Validate()` warning). |
|
||||
| `HistoryReadDeadline` | TimeSpan | `00:01:00` | Per-request wall-clock deadline for a HistoryRead (and HistoryUpdate) batch (arch-review 03/S3). A node still in flight at the deadline surfaces `BadTimeout`, so one request can never hold an SDK request thread longer than this regardless of node count. Non-positive = unbounded (a `Validate()` warning). |
|
||||
| `HistoryUpdateMaxFutureSkew` | TimeSpan | `1.00:00:00` | #674. How far past the server clock a HistoryUpdate (Insert / Replace / Update) item's `SourceTimestamp` may lie; an item beyond `now + skew` is refused **`BadOutOfRange`** by the node manager before any backend sees it (every backend). Guards against a far-future typo (a 2062 timestamp). The live recorder is not bounded. Non-positive disables the bound (no warning — a deliberate choice). See [HistoryUpdate](#historyupdate). |
|
||||
|
||||
**`ServerHistorian:SqlServer`** (read only when `Backend=SqlServer`; bound by `HistorianBackendRegistry.BindSqlServer`):
|
||||
|
||||
@@ -686,6 +688,25 @@ modification time and the Part 11 `HistoryUpdateType`. A HistoryReadRaw with `Is
|
||||
within one node the order is strict.
|
||||
- **Retention.** The trail ages out on the same **source-time** cutoff as the samples it describes: `SqlServer` purges `hist.SampleModified`
|
||||
per tag alongside `hist.Sample`; QuestDB puts the same TTL on `{p}sample_modified`.
|
||||
- **SqlServer schema v1 (#674).** With `AutoCreateSchema=false` a database still at schema **version 1** (script `002_sample_modified.sql` not
|
||||
applied) keeps historizing — live writes, raw/processed/at-time reads, alarm events and retention all work — but has no trail: modified reads
|
||||
and every HistoryUpdate item answer **`BadHistoryOperationUnsupported`** with nothing sent to SQL, retention skips `SampleModified`, and the
|
||||
node logs **one Warning per boot**. Apply `002_sample_modified.sql` and restart the node to enable them (the verdict is taken once per
|
||||
process). `AutoCreateSchema=true` (the default) still upgrades v1 to v2 in place. See [Schema](#schema).
|
||||
- **QuestDB retry duplicates (#674).** `{p}sample_modified` is append-only with **no DEDUP key** (a key would collapse genuinely distinct
|
||||
modifications of one sample), and each trail row gets a fresh `modified_id`. So a HistoryUpdate a client **retries** after a timeout — when the
|
||||
first attempt had in fact landed — records a second trail row for the same change (and, for Insert, the retry is `BadEntryExists`). The
|
||||
sample itself is idempotent (dedup upsert); only the audit trail can show the duplicate. `SqlServer` has the same exposure only if the client
|
||||
retries a committed request; there is no request id to dedupe on in either backend.
|
||||
- **Cross-node ordering (QuestDB).** The per-tag lock is per process; the [Primary-only](#primary-only-674) HistoryUpdate gate (#674) is what keeps
|
||||
two nodes of a pair from recording concurrent updates of one tag (and so a wrong prior value). Two **separate** pairs sharing one QuestDB can
|
||||
still race; that is unsupported.
|
||||
- ⚠️ **Who can read the trail (#674.4).** Modified reads are ordinary HistoryReads: with `Security:Acl:Enabled=false` (the default) **any admitted
|
||||
session, including Anonymous, can read who rewrote history** (`UserName` is the principal). Turn the ACL flag on, or set
|
||||
`OpcUa:AllowAnonymous=false`, if the principal names are sensitive (see [security.md](security.md)).
|
||||
- **Not implemented (#674.5):** `IsDeleteModified` (deleting trail entries — `BadHistoryOperationUnsupported`); recording an **Update** (upsert) as
|
||||
a split `Insert` / `Replace` — it is recorded as `Update` either way; and **`AuditHistoryUpdateEvent`** (no audit event is emitted for a history
|
||||
change — the trail is the only record).
|
||||
|
||||
```bash
|
||||
otopcua-cli historyread -u opc.tcp://localhost:4840 -U apikey -P otua_<keyId>_<secret> \
|
||||
@@ -752,7 +773,7 @@ The bootstrapper applies every script above the recorded version, in order, unde
|
||||
|
||||
| Object | Purpose |
|
||||
|---|---|
|
||||
| `SchemaVersion(Version, AppliedUtc)` | Applied-script ledger. Required version: **2** (since #668). |
|
||||
| `SchemaVersion(Version, AppliedUtc)` | Applied-script ledger. Required version: **1**; version **2** adds the modification trail (#668). A v1 database is served with the trail features off (#674 — see below). |
|
||||
| `Tag(TagId int identity, Name nvarchar(400) unique, DataType tinyint, EngineeringUnit, Description, CreatedUtc, UpdatedUtc)` | One row per historian tagname. `DataType` is a `DriverDataType`. |
|
||||
| `Sample(TagId, SourceTimeUtc datetime2(7), ValueKind tinyint, NumValue float, IntValue bigint, StrValue nvarchar(max), StatusCode int, ServerTimeUtc)` | **PK `(TagId, SourceTimeUtc)`** — one value per timestamp (Part 11), which makes redelivery idempotent and rules out tie clusters. `StatusCode` is the uint status stored bit-for-bit. |
|
||||
| `Event(EventId nvarchar(64) PK nonclustered, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, Message, UserName, Comment, EventTimeUtc, ReceivedTimeUtc)` | Alarm history. Clustered on `(EventTimeUtc, EventId)`; extra index `(SourceName, EventTimeUtc)`. |
|
||||
@@ -768,11 +789,14 @@ everywhere (full 100 ns precision — never a `DataTable`, which would round to
|
||||
`sqlcmd -S <server> -U <user> -P <pw> -d <db> -I -v Schema=hist -i docs/historian/sqlserver/001_initial.sql`, then the same with
|
||||
`002_sample_modified.sql` (`-v Schema=` supplies the `$(Schema)` placeholder; verified against the docker-dev SQL container). Re-running is a no-op.
|
||||
|
||||
⚠️ **Upgrading an existing v1 historian (#668).** A build with this change requires schema **version 2**. With `AutoCreateSchema=true`
|
||||
(the default) the first historian call after the upgrade applies `002_sample_modified.sql` in place (it only creates `SampleModified`; existing
|
||||
samples are untouched). With `AutoCreateSchema=false` **every historian call fails** (`missing version 2 (found 1)`, in health `LastError`)
|
||||
until an operator runs `002_sample_modified.sql` by hand as above — do that before rolling the new build out. Modifications made before the
|
||||
upgrade have no trail; the trail starts with the first HistoryUpdate after it.
|
||||
⚠️ **Upgrading an existing v1 historian (#668, #674).** With `AutoCreateSchema=true` (the default) the first historian call after the
|
||||
upgrade applies `002_sample_modified.sql` in place (it only creates `SampleModified`; existing samples are untouched). With
|
||||
`AutoCreateSchema=false` a **v1 database is accepted** (#674 — between #668 and #674 it refused every historian call, continuous-historization
|
||||
drains included): writes, reads, events and retention work, while modified-value reads and HistoryUpdate report
|
||||
`BadHistoryOperationUnsupported` without touching SQL, retention skips `SampleModified`, and one Warning per boot names the missing script.
|
||||
Run `002_sample_modified.sql` by hand as above and **restart the node** to enable them (the version is read once per process). Only a schema
|
||||
below version 1 (`missing version 1 (found 0)`) still fails every call. Modifications made before the upgrade have no trail; the trail starts
|
||||
with the first HistoryUpdate after it.
|
||||
|
||||
### Writes
|
||||
|
||||
@@ -999,6 +1023,8 @@ equivalent TTL (`QuestDb:RetentionDays`); the Gateway backend's retention is the
|
||||
timeout 0) lets one purger run and the others skip that tick (Debug log).
|
||||
- **Observability.** Counter `otopcua.historian.retention.purged{table=Sample|SampleModified|Event}`; a failed pass logs a Warning and retries next
|
||||
interval — it never affects capture or reads.
|
||||
- **Schema v1 (#674).** On a v1 schema (no `SampleModified`, `AutoCreateSchema=false`) the trail purge is skipped and omitted from the
|
||||
pass results; samples and events purge normally.
|
||||
- **Not done: partition switching.** For very large installs (billions of rows) a partitioned `hist.Sample` with sliding-window
|
||||
`SWITCH`/drop is cheaper than row deletes; the purger is deliberately simple batched deletes. Another table is added by one more
|
||||
`DeleteLoopAsync` call in `SqlServerHistorianRetention.PurgeAsync` (as `hist.SampleModified` was, #668).
|
||||
@@ -1195,7 +1221,14 @@ expired day partitions itself, tombstones included.
|
||||
re-read to verify; a very large `RetentionDays` that QuestDB normalises into MONTH/YEAR is re-`ALTER`ed on every boot (harmless).
|
||||
- ⚠️ **TTL is enforced only on write commits.** QuestDB checks for expired partitions when a write commits to the table, so **a quiet table never
|
||||
drops anything**: partitions older than the TTL linger until the next write lands. Do not read a stale-looking partition as a broken TTL.
|
||||
- Expiry is judged against the wall clock, so a future-dated sample does not expire current data.
|
||||
- **Expiry is judged against the wall clock** (`min(max timestamp, now)`), so a future-dated sample does not expire current data.
|
||||
⚠️ **Keep `cairo.ttl.use.wall.clock` at its default, `true`** (#674). It is what QuestDB documents as protecting a TTL table from a
|
||||
future-dated row; with it `false` the TTL would be measured from the table's newest timestamp and one far-future row (a 2062 typo) could
|
||||
drop every older partition — including `{p}sample_modified`, the record of who changed history. Evidence (QuestDB 10.0.1, 2026-10-11): the
|
||||
default is `true`, and a 2066 row inserted into a 30-day-TTL table evicted nothing inside the TTL, while a partition genuinely older than
|
||||
30 days was evicted by that same commit — pinned by `TtlWallClockTests`. (The same run with the flag `false` also kept the in-TTL
|
||||
partitions, so the test pins the outcome rather than isolating the flag; do not rely on that.) The server-side
|
||||
`ServerHistorian:HistoryUpdateMaxFutureSkew` bound (default 1 day) is the primary guard: a HistoryUpdate cannot write a far-future row.
|
||||
- The TTL is visible in table metadata: `SELECT table_name, ttlValue, ttlUnit FROM tables() WHERE starts_with(table_name, 'hist_')`.
|
||||
|
||||
### Docker-dev recipe
|
||||
@@ -1236,8 +1269,8 @@ dotnet test tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Integration
|
||||
|
||||
OPC UA **HistoryUpdate** lets a client insert, replace, update and delete stored raw history. It is available only when the selected
|
||||
backend provides an updater — **`SqlServer` and `QuestDb` do; `Gateway` does not**, and an updater is registered only when `ServerHistorian:Enabled=true`.
|
||||
Updates go straight to the backend (not through the outbox), are **not Primary-gated** (the database is shared), and share the HistoryRead
|
||||
batch limiter and `HistoryReadDeadline`.
|
||||
Updates go straight to the backend (not through the outbox), are **Primary-only** (#674, [below](#primary-only-674)), and share the
|
||||
HistoryRead batch limiter and `HistoryReadDeadline`.
|
||||
|
||||
### Operations
|
||||
|
||||
@@ -1255,7 +1288,8 @@ caller's principal, in the [modification trail](#modified-value-history-part-11)
|
||||
mixed (Part 11 reports item outcomes in `OperationResults`); it is Bad only for whole-node failures.
|
||||
|
||||
**Per-item rejections** (answered without touching SQL; other items still proceed): a null item ⇒ `BadInvalidArgument`;
|
||||
`SourceTimestamp == DateTime.MinValue` ⇒ `BadInvalidTimestamp`; a value the historian cannot type (array, unsupported CLR type) ⇒
|
||||
`SourceTimestamp == DateTime.MinValue` ⇒ `BadInvalidTimestamp`; a **`SourceTimestamp` later than `now + ServerHistorian:HistoryUpdateMaxFutureSkew`**
|
||||
(default 1 day, server clock; #674) ⇒ **`BadOutOfRange`** — on every backend, never sent; a value the historian cannot type (array, unsupported CLR type) ⇒
|
||||
`BadTypeMismatch`; a **`NaN` / `±Infinity` `Double` ⇒ `BadInvalidArgument`** — it cannot live in a SQL `float`, and unlike the live-value
|
||||
writer (which stores it as `Empty`) an operator's explicit update is rejected rather than silently rewritten; and a **null value with a
|
||||
Good or Uncertain status ⇒ `BadInvalidArgument`** — an `Empty` sample is always Bad, so a null value is accepted only with a Bad status
|
||||
@@ -1278,7 +1312,19 @@ outcome, later ones `BadNoEntryExists`. **Unknown tag:** Insert / Update create
|
||||
`ServerHistorian:Enabled=false`); `PerformUpdateType.Remove`; `DeleteRawModifiedDetails.IsDeleteModified=true` (the
|
||||
[modification trail](#modified-value-history-part-11) is an audit record, aged out only by retention); and the out-of-scope detail types — **event** HistoryUpdate (`UpdateEventDetails` / `DeleteEventDetails`) and `UpdateStructureData` —
|
||||
which are not overridden, so the SDK base rejects them. A backend exception is `BadInternalError` for that node and a deadline is `BadTimeout`.
|
||||
No `AuditHistoryUpdateEvent` is emitted.
|
||||
On a `SqlServer` database still at schema v1 every item is `BadHistoryOperationUnsupported` ([#674](#modified-value-history-part-11)).
|
||||
No `AuditHistoryUpdateEvent` is emitted (#674.5, not implemented).
|
||||
|
||||
### Primary-only (#674)
|
||||
|
||||
HistoryUpdate is served only by the node holding the redundancy **Primary** data-plane role — the same verdict
|
||||
(`IDriverPrimaryRoleSource`, published by `DriverHostActor`) that gates native-alarm acknowledges (#597). On any other node every node of
|
||||
the request that **passed authorization** is refused **`BadInvalidState`** (node-level code, with a message pointing at the Primary) and
|
||||
nothing reaches the backend. Authorization runs first, so an unauthorized caller learns `BadUserAccessDenied`, not the node's role.
|
||||
HistoryRead is **not** gated. Why: rewriting history is a single-writer operation — both backends capture the prior value for the
|
||||
[modification trail](#modified-value-history-part-11) under a per-tag, **per-process** lock (QuestDB) or statement (SqlServer), so two pair
|
||||
nodes updating one tag concurrently could record a wrong prior value. A single-node deployment (no redundancy manager) is unaffected. Clients
|
||||
should send HistoryUpdate to the server advertising the higher `ServiceLevel`.
|
||||
|
||||
### The `HistoryWrite` AccessLevel bit
|
||||
|
||||
|
||||
Reference in New Issue
Block a user