dohertj2/lmxopcua
1
0
Fork 0
Code Issues 14 Pull Requests Actions Packages Projects Releases Wiki Activity

Merge feat/668-modified-values: Part 11 modified-value history (IsReadModified) on SqlServer + QuestDB (#668)

Browse Source
This commit is contained in:
Joseph Doherty
2026-10-11 09:32:13 -04:00
parent 699871040a cfd403710e
commit 062086f9a6
51 changed files with 2173 additions and 195 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/Client.CLI.md
+12
View File
@@ -210,8 +210,19 @@ otopcua-cli historyread -u opc.tcp://localhost:4840/OtOpcUa \
-n "ns=1;s=TestMachine_001.TestHistoryValue" \
--start "2026-03-25T00:00:00Z" --end "2026-03-30T00:00:00Z" \
--aggregate Average --interval 3600000
# Modified-value history (what HistoryUpdate changed): prior value, update type, user, modification time
otopcua-cli historyread -u opc.tcp://localhost:4840/OtOpcUa -U apikey -P otua_<keyId>_<secret> \
-n "ns=2;s=Plant/Modbus/dev1/Speed" \
--start "2026-01-15T00:00:00Z" --end "2026-01-16T00:00:00Z" --modified
```
`--modified` sends `ReadRawModifiedDetails` with `IsReadModified = true` (OPC UA Part 11, Gitea #668) and prints one line per
recorded modification: the source timestamp, the **prior** value (`(none)` for an insert, which had none), its status, the update type
(`Insert` / `Replace` / `Update` / `Delete`), the user (the API key's principal, or `anonymous`) and the modification time. It follows
continuation points up to `--max`. A server on the `Gateway` backend (no modification trail) answers `BadHistoryOperationUnsupported`.
`--modified` cannot be combined with `--aggregate`. See [Historian.md §Modified-value history](Historian.md#modified-value-history-part-11).
| Flag | Description |
|------|-------------|
| `-n` / `--node` | Node ID to read history for (required) |
@@ -220,6 +231,7 @@ otopcua-cli historyread -u opc.tcp://localhost:4840/OtOpcUa \
| `--max` | Maximum number of values (default: 1000) |
| `--aggregate` | Aggregate function: Average, Minimum, Maximum, Count, Start, End, StandardDeviation |
| `--interval` | Processing interval in milliseconds for aggregates (default: 3600000) |
| `--modified` | Read modified-value history (Part 11 `IsReadModified`) instead of raw samples |
### historyupdate
docs/Historian.md
+86 -28
View File
@@ -73,12 +73,13 @@ consumes the historian. The existing enable gates (`ServerHistorian:Enabled`, `A
| | `Gateway` (default) | `SqlServer` | `QuestDb` |
|---|---|---|---|
| Storage | AVEVA Historian via the HistorianGateway sidecar | `{Schema}` tables in a SQL Server database you point it at | `{prefix}sample` / `event` / `tag` WAL tables in a QuestDB instance |
| Storage | AVEVA Historian via the HistorianGateway sidecar | `{Schema}` tables in a SQL Server database you point it at | `{prefix}sample` / `event` / `tag` / `sample_modified` WAL tables in a QuestDB instance |
| Value kinds recorded | `Double`, `Int64`, `Boolean` (numeric analog only) | all: `Double`, `Int64`, `Boolean`, `String`, `DateTime`, plus `Empty` (a Bad sample with no value) | same as `SqlServer` |
| Status code stored | Sent as DA quality but **discarded** by the gateway (C-002); AVEVA stores every sample Good | Stored bit-for-bit (`StatusCode` int column) | Stored bit-for-bit (unsigned, in a `LONG`) |
| Comms-loss / driver-Bad publishes | Dropped, counted in `DroppedUnsupportedKind` | Stored as an `Empty` outage-marker sample carrying the Bad code (see [below](#typed-values-real-status-and-outage-markers)) | Stored as an `Empty` outage-marker sample, as on `SqlServer` |
| Processed aggregates | Average, Minimum, Maximum, Total, Count | The seventeen in the [aggregate table](#aggregates-processed-reads) (incl. Part 13 time-weighted, #671) | The same seventeen, identical results |
| HistoryUpdate | No (`BadHistoryOperationUnsupported`) | Yes (Insert / Replace / Update / DeleteRaw / DeleteAtTime) | Yes — soft delete (tombstones), see [below](#historyupdate-soft-delete) |
| Modified-value history (`IsReadModified`, #668) | No (`BadHistoryOperationUnsupported`) | Yes — `SampleModified`, atomic with the change | Yes — `{prefix}sample_modified`, written before the change (orphan-on-crash) |
| Alarm history | gateway `SendEvent` | `Event` table (insert-if-absent) | `{prefix}event` table (dedup upsert on `(ts, event_id)`) |
| Provisioning (`EnsureTags`) | numeric types only; others skipped | `MERGE` upsert of every request, all types | Upsert into `{prefix}tag`, all types |
| Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` | `ServerHistorian:QuestDb:*` |
@@ -520,6 +521,7 @@ The server supports all four OPC UA HistoryRead variants:
| **Processed** | Historized variable | `Average`, `Minimum`, `Maximum`, `Count`, `Start`, `End`, `StandardDeviation` (the CLI's own list; `SqlServer` / `QuestDb` serve seventeen aggregates — see the [aggregate table](#aggregates-processed-reads)) |
| **AtTime** | Historized variable | n/a (client supplies exact timestamps) |
| **Events** | Equipment folder (event notifier) | n/a |
| **Modified** (Raw with `IsReadModified = true`, #668) | Historized variable | `--modified` (see [Modified-value history](#modified-value-history-part-11)) |
**Variable nodes** (historized tags) serve Raw, Processed, and AtTime history.
`Historizing=true` and `AccessLevels.HistoryRead` are set at materialization so any compliant
@@ -637,6 +639,48 @@ says is complete carries no continuation point. `NumValuesPerNode == 0` stays un
- Reverse event reads (`start > end`) are not paged server-side; no continuation point is issued for them.
- No session on the request (in-process callers) degrades to single-shot, as for Raw.
### Modified-value history (Part 11)
**Gitea #668.** On the `SqlServer` and `QuestDb` backends every HistoryUpdate leaves a **modification trail**: one row per sample it
inserted, replaced or deleted, carrying the value **as it was before** the change (none for an insert), the caller's principal, the
modification time and the Part 11 `HistoryUpdateType`. A HistoryReadRaw with `IsReadModified = true` returns that trail as a
`HistoryModifiedData` — one `DataValue` (the prior value) plus one parallel `ModificationInfo` (`ModificationTime`, `UpdateType`,
`UserName`) per entry.
| Operation | Recorded `UpdateType` | Prior value |
|---|---|---|
| Insert | `Insert` | none — served `null` / `BadNoData` |
| Replace | `Replace` | the replaced value and its status |
| Update (upsert) | `Update` | the replaced value, or none when it inserted |
| DeleteAtTime / DeleteRaw | `Delete` | the deleted value and its status |
- **Only applied items are recorded.** A refused item (`BadEntryExists`, `BadNoEntryExists`, `BadInvalidTimestamp`, …), a DeleteRaw over an
empty range, the **live recorder's writes** and the **retention purge** leave nothing.
- **`UserName` is the session's principal** — the API key's `NodeAcl` principal, or `anonymous` for an admitted anonymous session (the
key id is stored too, not served). Longer than 200 characters is truncated, never refused.
- **Window and order.** Same windows as Raw: forward `[start, end)` ascending by `(SourceTime, ModifiedId)`, reverse (`start > end`)
`(end, start]` descending, `start == end` the modifications at exactly that time; an unset bound with a cap pages open-ended like Raw.
Several modifications of one timestamp appear in the order they were made.
- **Paging.** `NumValuesPerNode > 0` pages with session-bound continuation points keyed on the last entry's `(SourceTime, ModifiedId)` —
an exact, exclusive key, so there are no ties to over-fetch and no trailing empty page (the backend reads `cap + 1` rows to know).
The cursor is bound to the node and to the modified read kind (a raw or events point is `BadContinuationPointInvalid`).
`NumValuesPerNode == 0` is unbounded.
- **Authorization** is the ordinary **HistoryRead** decision — no new operation or permission (ungated with `Security:Acl:Enabled` off).
- **Unsupported** ⇒ `BadHistoryOperationUnsupported`: the `Gateway` backend (the AVEVA Historian exposes no modification trail) and the
`Null` default. **Deleting the trail** (`DeleteRawModifiedDetails.IsDeleteModified = true`) is not supported either — the trail is an
audit record and is aged out only by retention (below).
- **Atomicity differs by backend.** `SqlServer` writes the trail row with `OUTPUT … INTO` in the **same statement** that changes `hist.Sample`,
so a change and its trail row commit or roll back together. QuestDB has no multi-table transaction, so the trail row is written **before**
the change: a crash (or failed sample INSERT) in between leaves an **orphan trail row** describing a change that never landed — never a
change without its trail row. Treat a QuestDB trail entry whose sample still shows the prior value as such an orphan.
- **Retention.** The trail ages out on the same **source-time** cutoff as the samples it describes: `SqlServer` purges `hist.SampleModified`
per tag alongside `hist.Sample`; QuestDB puts the same TTL on `{p}sample_modified`.
```bash
otopcua-cli historyread -u opc.tcp://localhost:4840 -U apikey -P otua_<keyId>_<secret> \
-n "ns=2;s=Plant/Modbus/dev1/Speed" --start 2026-01-15T00:00:00Z --end 2026-01-16T00:00:00Z --modified
```
### Total aggregate
The OPC UA `Total` aggregate is **supported** over the HistorianGateway backend. The gateway exposes a
@@ -667,13 +711,9 @@ member upstream. A NodeId outside the mapped set (e.g. `StandardDeviationSample`
sample per requested timestamp, so the single-shot backend returns the complete result in one
read and there is no "full page ⇒ maybe more" signal to page on. Returning the full result with
no continuation point is spec-conformant.
- **No modified-value history** (`HistoryReadModified`). Requests for modified values return
`BadHistoryOperationUnsupported`. This is **infra-gated, not a server-code gap**: the HistorianGateway
backend (`GatewayHistorianDataSource`) exposes only a current-value read path — there is no
modified/edited-history surface to source the data from. The server-side override is in place (it cleanly
rejects modified reads per node) and `IsReadModified` is honoured; serving real modified-value history is
unblocked only once the gateway grows a modified-read RPC. Until then, rejecting is the correct,
spec-conformant behaviour.
- **No modified-value history on the `Gateway` backend.** Modified reads (`IsReadModified = true`) are served by
`SqlServer` and `QuestDb` ([Modified-value history](#modified-value-history-part-11), #668); the HistorianGateway exposes no
modified/edited-history surface, so on `Gateway` (and with no historian) they return `BadHistoryOperationUnsupported`.
### Redundancy and authorization
@@ -694,15 +734,18 @@ bootstrapper and one tag catalog, so the schema is ensured once per process and
### Schema
Created by the embedded idempotent script `Schema/001_initial.sql` (mirrored at
[`docs/historian/sqlserver/001_initial.sql`](historian/sqlserver/001_initial.sql)); `$(Schema)` is the configured schema.
Created by the embedded idempotent scripts `Schema/001_initial.sql` and `Schema/002_sample_modified.sql` (mirrored at
[`docs/historian/sqlserver/001_initial.sql`](historian/sqlserver/001_initial.sql) and
[`docs/historian/sqlserver/002_sample_modified.sql`](historian/sqlserver/002_sample_modified.sql)); `$(Schema)` is the configured schema.
The bootstrapper applies every script above the recorded version, in order, under the schema applock.
| Object | Purpose |
|---|---|
| `SchemaVersion(Version, AppliedUtc)` | Applied-script ledger. Required version: **1**. |
| `SchemaVersion(Version, AppliedUtc)` | Applied-script ledger. Required version: **2** (since #668). |
| `Tag(TagId int identity, Name nvarchar(400) unique, DataType tinyint, EngineeringUnit, Description, CreatedUtc, UpdatedUtc)` | One row per historian tagname. `DataType` is a `DriverDataType`. |
| `Sample(TagId, SourceTimeUtc datetime2(7), ValueKind tinyint, NumValue float, IntValue bigint, StrValue nvarchar(max), StatusCode int, ServerTimeUtc)` | **PK `(TagId, SourceTimeUtc)`** — one value per timestamp (Part 11), which makes redelivery idempotent and rules out tie clusters. `StatusCode` is the uint status stored bit-for-bit. |
| `Event(EventId nvarchar(64) PK nonclustered, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, Message, UserName, Comment, EventTimeUtc, ReceivedTimeUtc)` | Alarm history. Clustered on `(EventTimeUtc, EventId)`; extra index `(SourceName, EventTimeUtc)`. |
| `SampleModified(TagId, SourceTimeUtc, ModifiedId bigint identity, ModificationTimeUtc, UpdateType tinyint, Principal nvarchar(200), ApiKeyId, ValueKind, NumValue, IntValue, StrValue, StatusCode, ServerTimeUtc)` (v2) | The Part 11 [modification trail](#modified-value-history-part-11): one row per sample a HistoryUpdate changed, carrying the **prior** value columns (all NULL for an insert). Clustered PK `(TagId, SourceTimeUtc, ModifiedId)`. ⚠️ No FOREIGN KEY, CHECK or trigger — it is an `OUTPUT … INTO` target, which forbids them; adding one fails every HistoryUpdate. |
| `SampleRow`, `TimeRow` table types | TVPs for batched sample writes/merges, AtTime and delete-at-time. |
Column mapping per `ValueKind`: `Double` → `NumValue`; `Int64` → `IntValue` + `NumValue` (as a double, for aggregates);
@@ -710,9 +753,15 @@ Column mapping per `ValueKind`: `Double` → `NumValue`; `Int64` → `IntValue`
`ValueKind` codes: Empty 0, Double 1, Int64 2, Boolean 3, String 4, DateTime 5. Timestamps are bound as `datetime2(7)`
everywhere (full 100 ns precision — never a `DataTable`, which would round to 3.33 ms).
**Applying the script by hand** (needed when `AutoCreateSchema=false`): create the database, then
`sqlcmd -S <server> -U <user> -P <pw> -d <db> -I -v Schema=hist -i docs/historian/sqlserver/001_initial.sql`
(`-v Schema=` supplies the `$(Schema)` placeholder; verified against the docker-dev SQL container). Re-running is a no-op.
**Applying the scripts by hand** (needed when `AutoCreateSchema=false`): create the database, then run each script in order —
`sqlcmd -S <server> -U <user> -P <pw> -d <db> -I -v Schema=hist -i docs/historian/sqlserver/001_initial.sql`, then the same with
`002_sample_modified.sql` (`-v Schema=` supplies the `$(Schema)` placeholder; verified against the docker-dev SQL container). Re-running is a no-op.
⚠️ **Upgrading an existing v1 historian (#668).** A build with this change requires schema **version 2**. With `AutoCreateSchema=true`
(the default) the first historian call after the upgrade applies `002_sample_modified.sql` in place (it only creates `SampleModified`; existing
samples are untouched). With `AutoCreateSchema=false` **every historian call fails** (`missing version 2 (found 1)`, in health `LastError`)
until an operator runs `002_sample_modified.sql` by hand as above — do that before rolling the new build out. Modifications made before the
upgrade have no trail; the trail starts with the first HistoryUpdate after it.
### Writes
@@ -919,7 +968,7 @@ The read client reports `GetHealthSnapshot()` (query/failure counters, consecuti
### Retention (`RetentionDays`)
`hist.Sample` and `hist.Event` grow without bound unless `ServerHistorian:SqlServer:RetentionDays` is set. QuestDB already has an
`hist.Sample`, `hist.SampleModified` and `hist.Event` grow without bound unless `ServerHistorian:SqlServer:RetentionDays` is set. QuestDB already has an
equivalent TTL (`QuestDb:RetentionDays`); the Gateway backend's retention is the AVEVA Historian's own.
- **What runs.** `HistorianRetentionService` (a hosted service, driver-role nodes, Backend=`SqlServer` with `RetentionDays` set; inert
@@ -931,15 +980,17 @@ equivalent TTL (`QuestDb:RetentionDays`); the Gateway backend's retention is the
database** — if they differ, the smallest window wins.
- **How it deletes.** Cutoff = `UtcNow - RetentionDays` on source time. Samples are purged **per tag** with
`DELETE TOP (@n) ... WHERE TagId = @id AND SourceTimeUtc < @cutoff` (a seek on the clustered `(TagId, SourceTimeUtc)` key; a global
time predicate would scan), then events with `DELETE TOP (@n) ... WHERE EventTimeUtc < @cutoff` (seek on `CX_Event_Time`), each looped
time predicate would scan) — and, in the same per-tag pass, the modification trail with
`DELETE TOP (@n) FROM SampleModified WHERE TagId = @id AND SourceTimeUtc < @cutoff` (#668; same **source-time** cutoff, clustered seek; the
purge itself never writes a trail row) — then events with `DELETE TOP (@n) ... WHERE EventTimeUtc < @cutoff` (seek on `CX_Event_Time`), each looped
until a batch comes back short. Cancellation is honoured between batches. The `hist.Tag` catalogue is never purged.
- **Shared databases.** Several pairs may point at one database; a session `sp_getapplock` (`OtOpcUa.Historian.Retention.<schema>`,
timeout 0) lets one purger run and the others skip that tick (Debug log).
- **Observability.** Counter `otopcua.historian.retention.purged{table=Sample|Event}`; a failed pass logs a Warning and retries next
- **Observability.** Counter `otopcua.historian.retention.purged{table=Sample|SampleModified|Event}`; a failed pass logs a Warning and retries next
interval — it never affects capture or reads.
- **Not done: partition switching.** For very large installs (billions of rows) a partitioned `hist.Sample` with sliding-window
`SWITCH`/drop is cheaper than row deletes; the purger is deliberately simple batched deletes. Another table (e.g. a future
`hist.SampleModified`, #668) is added by one more `DeleteLoopAsync` call in `SqlServerHistorianRetention.PurgeAsync`.
`SWITCH`/drop is cheaper than row deletes; the purger is deliberately simple batched deletes. Another table is added by one more
`DeleteLoopAsync` call in `SqlServerHistorianRetention.PurgeAsync` (as `hist.SampleModified` was, #668).
### Docker-dev recipe
@@ -1004,10 +1055,10 @@ Read only when `Backend=QuestDb`; bound by `HistorianBackendRegistry.BindQuestDb
| Key | Default | Validation | Meaning |
|---|---|---|---|
| `QuestDb:ConnectionString` | `""` | **Required** (non-blank) | Npgsql connection string for QuestDB's PG-wire port, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading`. `ServerCompatibilityMode=NoTypeLoading` skips the `pg_type` load; every query form was verified with it. Supply via env `ServerHistorian__QuestDb__ConnectionString` / `${secret:}` for anything real. |
| `QuestDb:TablePrefix` | `hist_` | `^[A-Za-z0-9_]+$` | QuestDB has no schemas, so the three tables are `{prefix}sample`, `{prefix}event`, `{prefix}tag`. Spliced into DDL/DML, hence the regex. |
| `QuestDb:RetentionDays` | unset | `>= 1` when set | Applied as table TTL to `{prefix}sample` and `{prefix}event` (see [Retention](#retention-ttl)). Unset keeps data forever. |
| `QuestDb:TablePrefix` | `hist_` | `^[A-Za-z0-9_]+$` | QuestDB has no schemas, so the four tables are `{prefix}sample`, `{prefix}event`, `{prefix}tag` and `{prefix}sample_modified` (#668). Spliced into DDL/DML, hence the regex. |
| `QuestDb:RetentionDays` | unset | `>= 1` when set | Applied as table TTL to `{prefix}sample`, `{prefix}event` and `{prefix}sample_modified` (see [Retention](#retention-ttl)). Unset keeps data forever. |
| `QuestDb:CommandTimeout` | `00:00:30` | `> 0` | Per-command timeout, rounded up to whole seconds (minimum 1 s). |
| `QuestDb:AutoCreateSchema` | `true` | — | `true`: create the tables (and reconcile the TTL) on first use. `false`: only verify the three tables exist; if one is missing every historian call fails with `HistorianSchemaMissingException` (health `LastError`, no crash) and no TTL is touched. A failed bootstrap is not latched: it is re-thrown for 5 s without touching QuestDB, then retried. |
| `QuestDb:AutoCreateSchema` | `true` | — | `true`: create the tables (and reconcile the TTL) on first use. `false`: only verify the four tables exist (⚠️ an install from before #668 lacks `{prefix}sample_modified` — create it with the DDL in `QuestDbTables.CreateSampleModified` before upgrading, or every call fails); if one is missing every historian call fails with `HistorianSchemaMissingException` (health `LastError`, no crash) and no TTL is touched. A failed bootstrap is not latched: it is re-thrown for 5 s without touching QuestDB, then retried. |
| `QuestDb:VisibilityTimeout` | `00:00:05` | `>= 0` | How long a write waits for its rows to become visible to queries ([WAL visibility](#wal-visibility-and-the-read-your-writes-wait)). `0` disables the wait. |
⚠️ **Connection pool size.** QuestDB's default limit is **64** concurrent PG-wire connections (`pg.net.connection.limit`), while Npgsql's
@@ -1019,14 +1070,15 @@ limit is server-configurable. The effective value is logged once at Information
### Tables
All three are **WAL tables with `DEDUP UPSERT KEYS`**, so re-sending a row with the same key replaces it (the upsert is what makes
HistoryUpdate Replace work; the writer's own insert-if-absent semantics are built on top, below).
All are **WAL tables**; the first three carry **`DEDUP UPSERT KEYS`**, so re-sending a row with the same key replaces it (the upsert is what
makes HistoryUpdate Replace work; the writer's own insert-if-absent semantics are built on top, below). The modification trail is append-only.
| Table | Key | Partitioning |
|---|---|---|
| `{p}sample` (`ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts`) | `(ts, tag)` | `PARTITION BY DAY` (TTL when configured) |
| `{p}event` (one-for-one with `hist.Event`) | `(ts, event_id)` | `PARTITION BY DAY` (TTL when configured) |
| `{p}tag` (`ts, name, data_type, eng_unit, description, updated_utc`) | `(ts, name)` with `ts` fixed at `1970-01-01` — an upsert by name | `PARTITION BY YEAR`, never expires |
| `{p}sample_modified` (`ts` = the sample's source time, `tag, modified_id, modification_ts, update_type, principal, api_key_id, kind, num_value, int_value, str_value, status, server_ts`) (#668) | none — append-only, no DEDUP; `modified_id` (process-monotonic ≈ UTC ticks) orders a timestamp's trail | `PARTITION BY DAY` (TTL when configured). A NULL `status` marks "no prior value" (QuestDB `BYTE` has no NULL, so `kind` is 0) |
`status` holds the uint OPC UA status code **unsigned** in a `LONG` (so "Bad" is `status >= 2147483648`, not `< 0`); `int_value` also holds
Booleans as 0/1; `kind` is the `HistorianValueKind` in a `BYTE`. Tag names are limited to 400 characters (longer is a per-tag failure,
@@ -1093,6 +1145,11 @@ present **and not deleted**.
| DeleteAtTime | write a tombstone (`deleted = true`, value NULL, `BadNoData`) where a live row exists | `Good` / `BadNoEntryExists` |
| DeleteRaw | tombstone every live row in `[start, end)`, 1,000 per page, waiting for visibility between pages | the same codes as `SqlServer` |
**Modification trail (#668).** Each operation first appends one `{p}sample_modified` row per sample it will change, carrying the live row's
current value (read by the same existence query), then writes the change. No multi-table transaction exists, so this order is the guarantee:
a crash between the two leaves an **orphan trail row**, never an untracked change — the documented deviation from `SqlServer`'s atomic trail.
See [Modified-value history](#modified-value-history-part-11).
The per-item codes come from the shared `HistoryUpdatePlanner`, so they equal `SqlServer`'s (including the duplicate-timestamp-in-one-request rules
above). **Every read and aggregate filters `NOT deleted`.** A client Insert at a deleted timestamp re-populates it (the OPC UA intent); only the
*live writer's* redelivery is barred from doing so.
@@ -1118,7 +1175,7 @@ Every query filters `NOT deleted`; every value goes through `StoredSample`, so t
### Retention (TTL)
`RetentionDays` becomes `TTL n DAYS` on `{p}sample` and `{p}event` (never on `{p}tag`: TTL must be a multiple of the partition size). QuestDB drops
`RetentionDays` becomes `TTL n DAYS` on `{p}sample`, `{p}event` and `{p}sample_modified` (never on `{p}tag`: TTL must be a multiple of the partition size). QuestDB drops
expired day partitions itself, tombstones included.
- At bootstrap the current TTL is read from `tables()` (`ttlValue`, `ttlUnit`). QuestDB **normalises the unit** (7 days reads `1 WEEK`), so the
@@ -1182,7 +1239,8 @@ batch limiter and `HistoryReadDeadline`.
| `DeleteAtTimeDetails` | `HistoryDeleteAtTime` | per time: `Good`; `BadNoEntryExists` |
Each backend call is applied as a unit (SqlServer: one transaction, `MERGE … OUTPUT $action` against the TVP; QuestDb: an INSERT per
chunk plus the WAL-visibility wait, no transaction — see [QuestDB backend](#questdb-backend)). The node-level `StatusCode` stays `Good` when item results are
chunk plus the WAL-visibility wait, no transaction — see [QuestDB backend](#questdb-backend)). Every applied item is also recorded, with the
caller's principal, in the [modification trail](#modified-value-history-part-11) (#668). The node-level `StatusCode` stays `Good` when item results are
mixed (Part 11 reports item outcomes in `OperationResults`); it is Bad only for whole-node failures.
**Per-item rejections** (answered without touching SQL; other items still proceed): a null item ⇒ `BadInvalidArgument`;
@@ -1206,8 +1264,8 @@ outcome, later ones `BadNoEntryExists`. **Unknown tag:** Insert / Update create
(DeleteRaw: `BadNoData`) without creating anything.
**Unsupported ⇒ `BadHistoryOperationUnsupported`** (per node): the node is not historized; the backend has no updater (`Gateway`, or
`ServerHistorian:Enabled=false`); `PerformUpdateType.Remove`; `DeleteRawModifiedDetails.IsDeleteModified=true` (no modified-value history is
kept); and the out-of-scope detail types — **event** HistoryUpdate (`UpdateEventDetails` / `DeleteEventDetails`) and `UpdateStructureData` —
`ServerHistorian:Enabled=false`); `PerformUpdateType.Remove`; `DeleteRawModifiedDetails.IsDeleteModified=true` (the
[modification trail](#modified-value-history-part-11) is an audit record, aged out only by retention); and the out-of-scope detail types — **event** HistoryUpdate (`UpdateEventDetails` / `DeleteEventDetails`) and `UpdateStructureData` —
which are not overridden, so the SDK base rejects them. A backend exception is `BadInternalError` for that node and a deadline is `BadTimeout`.
No `AuditHistoryUpdateEvent` is emitted.
docs/historian/sqlserver/002_sample_modified.sql
+29
View File
@@ -0,0 +1,29 @@
-- OtOpcUa SQL Server historian backend - schema version 2: OPC UA Part 11 modified-value history (Gitea #668).
-- $(Schema) is substituted by SqlServerSchemaBootstrapper (identifier-validated) BEFORE the script is split on GO lines.
-- Every statement is guarded so a re-run is a no-op.
--
-- One row per sample a HistoryUpdate inserted, replaced or deleted, carrying the PRIOR value (all NULL for an
-- insert). Rows are written by the SAME statement that changes hist.Sample (OUTPUT ... INTO), so a failed update
-- leaves no audit row and a committed one always has its row.
-- WARNING: OUTPUT ... INTO forbids, on its target table: enabled triggers, either side of a FOREIGN KEY, and CHECK
-- constraints. Do not add any of those to SampleModified, or every HistoryUpdate fails.
IF OBJECT_ID(N'[$(Schema)].[SampleModified]', N'U') IS NULL
CREATE TABLE [$(Schema)].[SampleModified] (
TagId int NOT NULL,
SourceTimeUtc datetime2(7) NOT NULL,
ModifiedId bigint IDENTITY(1,1) NOT NULL,
ModificationTimeUtc datetime2(7) NOT NULL,
UpdateType tinyint NOT NULL,
Principal nvarchar(200) NOT NULL,
ApiKeyId nvarchar(200) NULL,
ValueKind tinyint NULL,
NumValue float NULL,
IntValue bigint NULL,
StrValue nvarchar(max) NULL,
StatusCode int NULL,
ServerTimeUtc datetime2(7) NULL,
CONSTRAINT PK_SampleModified PRIMARY KEY CLUSTERED (TagId, SourceTimeUtc, ModifiedId))
GO
IF NOT EXISTS (SELECT 1 FROM [$(Schema)].[SchemaVersion] WHERE Version = 2)
INSERT [$(Schema)].[SchemaVersion] (Version, AppliedUtc) VALUES (2, SYSUTCDATETIME())
GO
docs/security.md
+6
View File
@@ -884,6 +884,12 @@ HistoryUpdate rewrites recorded history, so unlike HistoryRead it is gated **wha
- **Flag off (default):** the session's API key is re-checked on the call (`DataPlaneAuthorizer.AuthorizeKeyUsable`, Gitea #563 — an expired or disabled key loses its activation-time roles), then the session must carry the **`HistoryUpdate` role** (`OpcUaDataPlaneRoles.HistoryUpdate`). `ApiKeyInterimRoleMapper` confers it **only from a `Cluster`-scoped grant** carrying `NodePermissions.HistoryUpdate` — narrower-scoped grants confer nothing, for the same reason as `WriteOperate` (honouring a flag without honouring its scope would widen it). A key holding only `WriteOperate` is denied `BadUserAccessDenied`.
- **Flag on:** `DataPlaneAuthorizer.Authorize(identity, OpcUaOperation.HistoryUpdate, nodeId, realm)` decides per node and **replaces** the role check (never chained) — a session with no roles succeeds on the strength of a scoped `HistoryUpdate` grant, and a `WriteOperate` grant does not help.
- **Availability is separate from authorization.** A node accepts HistoryUpdate only if it is historized **and** the selected historian backend has an updater when the node materialises (the `HistoryWrite` AccessLevel bit). `Backend=Gateway` has none, so it answers `BadHistoryOperationUnsupported` for every node whatever the grant. Operations, result codes and unsupported cases: [`docs/Historian.md` §HistoryUpdate](Historian.md#historyupdate).
- **Every applied change is attributed (Part 11 modified-value history, Gitea #668).** The session's principal (the API key's
`NodeAcl` principal, or the reserved `anonymous`) and key id are passed to the backend with every HistoryUpdate and recorded in the
modification trail next to the prior value; a modified-value HistoryRead (`IsReadModified = true`) returns it as
`ModificationInfo.UserName`. Reading the trail is gated by the **same `HistoryRead` decision** as a raw read — there is no separate
permission — so with `Security:Acl:Enabled` off it is as ungated as any HistoryRead, and anyone who may read a node's history can see
who rewrote it and what it held before. See [`docs/Historian.md` §Modified-value history](Historian.md#modified-value-history-part-11).
### Evaluator — `PermissionTrie`