diff --git a/CLAUDE.md b/CLAUDE.md index 96a304f6..89e33b34 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1348,6 +1348,7 @@ historian tagname, which is the tag's driver `FullName`. - **Real status fix:** the recorder used to stamp every sample Good (`192`); it now carries the driver's status. ⚠️ On the **Gateway** backend this changes nothing stored: the gateway discards quality (C-002), so AVEVA still stores Good. - **Comms loss is historized as an outage marker** (`Empty` + `BadNoCommunication`, one per tag per outage) on SqlServer; dropped (`DroppedUnsupportedKind`) on Gateway. Outbox records are **format v2** — drain or delete `ContinuousHistorization:OutboxPath` before downgrading a node. - **HistoryUpdate** (SqlServer only; Gateway ⇒ `BadHistoryOperationUnsupported`): Insert/Replace/Update/DeleteRaw/DeleteAtTime, gated in both ACL modes — flag off needs the `HistoryUpdate` role from a **Cluster-scoped** grant (+ key re-check), flag on uses `DataPlaneAuthorizer`'s HistoryUpdate operation; `WriteOperate` never implies it. The `HistoryWrite` AccessLevel bit is set only when an updater is wired at materialise time. `docs/Historian.md` §HistoryUpdate, `docs/security.md`. +- **Modified-value history (#668, Part 11):** every applied HistoryUpdate item leaves a trail row (prior value, principal from the session's API key / `anonymous`, `HistoryModificationType`, time) — SqlServer `hist.SampleModified` (**schema v2**, written by `OUTPUT … INTO` in the same statement ⇒ atomic; ⚠️ no FK/CHECK/trigger on it, and `AutoCreateSchema=false` installs must run `002_sample_modified.sql` before upgrading), QuestDB `{p}sample_modified` (append-only, written **before** the change ⇒ a crash leaves an orphan trail row, never an untracked change). `IsReadModified=true` HistoryRead serves it as `HistoryModifiedData` via `IHistorianDataSource.ReadModifiedAsync` (default ⇒ `NotSupported` ⇒ `BadHistoryOperationUnsupported` on Gateway/Null), paged on `(SourceTime, ModifiedId)`, gated by plain HistoryRead. Live writes and retention never write it; retention/TTL purge it on source time. `IHistorianDataUpdater` methods take a `HistoryUpdateContext`. CLI `historyread --modified`. `docs/Historian.md` §Modified-value history. ### Read path (`ServerHistorian` section) diff --git a/docs/Client.CLI.md b/docs/Client.CLI.md index 15d43f7c..3729023e 100644 --- a/docs/Client.CLI.md +++ b/docs/Client.CLI.md @@ -210,8 +210,19 @@ otopcua-cli historyread -u opc.tcp://localhost:4840/OtOpcUa \ -n "ns=1;s=TestMachine_001.TestHistoryValue" \ --start "2026-03-25T00:00:00Z" --end "2026-03-30T00:00:00Z" \ --aggregate Average --interval 3600000 + +# Modified-value history (what HistoryUpdate changed): prior value, update type, user, modification time +otopcua-cli historyread -u opc.tcp://localhost:4840/OtOpcUa -U apikey -P otua__ \ + -n "ns=2;s=Plant/Modbus/dev1/Speed" \ + --start "2026-01-15T00:00:00Z" --end "2026-01-16T00:00:00Z" --modified ``` +`--modified` sends `ReadRawModifiedDetails` with `IsReadModified = true` (OPC UA Part 11, Gitea #668) and prints one line per +recorded modification: the source timestamp, the **prior** value (`(none)` for an insert, which had none), its status, the update type +(`Insert` / `Replace` / `Update` / `Delete`), the user (the API key's principal, or `anonymous`) and the modification time. It follows +continuation points up to `--max`. A server on the `Gateway` backend (no modification trail) answers `BadHistoryOperationUnsupported`. +`--modified` cannot be combined with `--aggregate`. See [Historian.md §Modified-value history](Historian.md#modified-value-history-part-11). + | Flag | Description | |------|-------------| | `-n` / `--node` | Node ID to read history for (required) | @@ -220,6 +231,7 @@ otopcua-cli historyread -u opc.tcp://localhost:4840/OtOpcUa \ | `--max` | Maximum number of values (default: 1000) | | `--aggregate` | Aggregate function: Average, Minimum, Maximum, Count, Start, End, StandardDeviation | | `--interval` | Processing interval in milliseconds for aggregates (default: 3600000) | +| `--modified` | Read modified-value history (Part 11 `IsReadModified`) instead of raw samples | ### historyupdate diff --git a/docs/Historian.md b/docs/Historian.md index 6a93b4df..0e66252c 100644 --- a/docs/Historian.md +++ b/docs/Historian.md @@ -73,12 +73,13 @@ consumes the historian. The existing enable gates (`ServerHistorian:Enabled`, `A | | `Gateway` (default) | `SqlServer` | `QuestDb` | |---|---|---|---| -| Storage | AVEVA Historian via the HistorianGateway sidecar | `{Schema}` tables in a SQL Server database you point it at | `{prefix}sample` / `event` / `tag` WAL tables in a QuestDB instance | +| Storage | AVEVA Historian via the HistorianGateway sidecar | `{Schema}` tables in a SQL Server database you point it at | `{prefix}sample` / `event` / `tag` / `sample_modified` WAL tables in a QuestDB instance | | Value kinds recorded | `Double`, `Int64`, `Boolean` (numeric analog only) | all: `Double`, `Int64`, `Boolean`, `String`, `DateTime`, plus `Empty` (a Bad sample with no value) | same as `SqlServer` | | Status code stored | Sent as DA quality but **discarded** by the gateway (C-002); AVEVA stores every sample Good | Stored bit-for-bit (`StatusCode` int column) | Stored bit-for-bit (unsigned, in a `LONG`) | | Comms-loss / driver-Bad publishes | Dropped, counted in `DroppedUnsupportedKind` | Stored as an `Empty` outage-marker sample carrying the Bad code (see [below](#typed-values-real-status-and-outage-markers)) | Stored as an `Empty` outage-marker sample, as on `SqlServer` | | Processed aggregates | Average, Minimum, Maximum, Total, Count | The seventeen in the [aggregate table](#aggregates-processed-reads) (incl. Part 13 time-weighted, #671) | The same seventeen, identical results | | HistoryUpdate | No (`BadHistoryOperationUnsupported`) | Yes (Insert / Replace / Update / DeleteRaw / DeleteAtTime) | Yes — soft delete (tombstones), see [below](#historyupdate-soft-delete) | +| Modified-value history (`IsReadModified`, #668) | No (`BadHistoryOperationUnsupported`) | Yes — `SampleModified`, atomic with the change | Yes — `{prefix}sample_modified`, written before the change (orphan-on-crash) | | Alarm history | gateway `SendEvent` | `Event` table (insert-if-absent) | `{prefix}event` table (dedup upsert on `(ts, event_id)`) | | Provisioning (`EnsureTags`) | numeric types only; others skipped | `MERGE` upsert of every request, all types | Upsert into `{prefix}tag`, all types | | Connection | `Endpoint` + `ApiKey` + TLS keys (top-level `ServerHistorian:*`) | `ServerHistorian:SqlServer:*` | `ServerHistorian:QuestDb:*` | @@ -520,6 +521,7 @@ The server supports all four OPC UA HistoryRead variants: | **Processed** | Historized variable | `Average`, `Minimum`, `Maximum`, `Count`, `Start`, `End`, `StandardDeviation` (the CLI's own list; `SqlServer` / `QuestDb` serve seventeen aggregates — see the [aggregate table](#aggregates-processed-reads)) | | **AtTime** | Historized variable | n/a (client supplies exact timestamps) | | **Events** | Equipment folder (event notifier) | n/a | +| **Modified** (Raw with `IsReadModified = true`, #668) | Historized variable | `--modified` (see [Modified-value history](#modified-value-history-part-11)) | **Variable nodes** (historized tags) serve Raw, Processed, and AtTime history. `Historizing=true` and `AccessLevels.HistoryRead` are set at materialization so any compliant @@ -637,6 +639,48 @@ says is complete carries no continuation point. `NumValuesPerNode == 0` stays un - Reverse event reads (`start > end`) are not paged server-side; no continuation point is issued for them. - No session on the request (in-process callers) degrades to single-shot, as for Raw. +### Modified-value history (Part 11) + +**Gitea #668.** On the `SqlServer` and `QuestDb` backends every HistoryUpdate leaves a **modification trail**: one row per sample it +inserted, replaced or deleted, carrying the value **as it was before** the change (none for an insert), the caller's principal, the +modification time and the Part 11 `HistoryUpdateType`. A HistoryReadRaw with `IsReadModified = true` returns that trail as a +`HistoryModifiedData` — one `DataValue` (the prior value) plus one parallel `ModificationInfo` (`ModificationTime`, `UpdateType`, +`UserName`) per entry. + +| Operation | Recorded `UpdateType` | Prior value | +|---|---|---| +| Insert | `Insert` | none — served `null` / `BadNoData` | +| Replace | `Replace` | the replaced value and its status | +| Update (upsert) | `Update` | the replaced value, or none when it inserted | +| DeleteAtTime / DeleteRaw | `Delete` | the deleted value and its status | + +- **Only applied items are recorded.** A refused item (`BadEntryExists`, `BadNoEntryExists`, `BadInvalidTimestamp`, …), a DeleteRaw over an + empty range, the **live recorder's writes** and the **retention purge** leave nothing. +- **`UserName` is the session's principal** — the API key's `NodeAcl` principal, or `anonymous` for an admitted anonymous session (the + key id is stored too, not served). Longer than 200 characters is truncated, never refused. +- **Window and order.** Same windows as Raw: forward `[start, end)` ascending by `(SourceTime, ModifiedId)`, reverse (`start > end`) + `(end, start]` descending, `start == end` the modifications at exactly that time; an unset bound with a cap pages open-ended like Raw. + Several modifications of one timestamp appear in the order they were made. +- **Paging.** `NumValuesPerNode > 0` pages with session-bound continuation points keyed on the last entry's `(SourceTime, ModifiedId)` — + an exact, exclusive key, so there are no ties to over-fetch and no trailing empty page (the backend reads `cap + 1` rows to know). + The cursor is bound to the node and to the modified read kind (a raw or events point is `BadContinuationPointInvalid`). + `NumValuesPerNode == 0` is unbounded. +- **Authorization** is the ordinary **HistoryRead** decision — no new operation or permission (ungated with `Security:Acl:Enabled` off). +- **Unsupported** ⇒ `BadHistoryOperationUnsupported`: the `Gateway` backend (the AVEVA Historian exposes no modification trail) and the + `Null` default. **Deleting the trail** (`DeleteRawModifiedDetails.IsDeleteModified = true`) is not supported either — the trail is an + audit record and is aged out only by retention (below). +- **Atomicity differs by backend.** `SqlServer` writes the trail row with `OUTPUT … INTO` in the **same statement** that changes `hist.Sample`, + so a change and its trail row commit or roll back together. QuestDB has no multi-table transaction, so the trail row is written **before** + the change: a crash (or failed sample INSERT) in between leaves an **orphan trail row** describing a change that never landed — never a + change without its trail row. Treat a QuestDB trail entry whose sample still shows the prior value as such an orphan. +- **Retention.** The trail ages out on the same **source-time** cutoff as the samples it describes: `SqlServer` purges `hist.SampleModified` + per tag alongside `hist.Sample`; QuestDB puts the same TTL on `{p}sample_modified`. + +```bash +otopcua-cli historyread -u opc.tcp://localhost:4840 -U apikey -P otua__ \ + -n "ns=2;s=Plant/Modbus/dev1/Speed" --start 2026-01-15T00:00:00Z --end 2026-01-16T00:00:00Z --modified +``` + ### Total aggregate The OPC UA `Total` aggregate is **supported** over the HistorianGateway backend. The gateway exposes a @@ -667,13 +711,9 @@ member upstream. A NodeId outside the mapped set (e.g. `StandardDeviationSample` sample per requested timestamp, so the single-shot backend returns the complete result in one read and there is no "full page ⇒ maybe more" signal to page on. Returning the full result with no continuation point is spec-conformant. -- **No modified-value history** (`HistoryReadModified`). Requests for modified values return - `BadHistoryOperationUnsupported`. This is **infra-gated, not a server-code gap**: the HistorianGateway - backend (`GatewayHistorianDataSource`) exposes only a current-value read path — there is no - modified/edited-history surface to source the data from. The server-side override is in place (it cleanly - rejects modified reads per node) and `IsReadModified` is honoured; serving real modified-value history is - unblocked only once the gateway grows a modified-read RPC. Until then, rejecting is the correct, - spec-conformant behaviour. +- **No modified-value history on the `Gateway` backend.** Modified reads (`IsReadModified = true`) are served by + `SqlServer` and `QuestDb` ([Modified-value history](#modified-value-history-part-11), #668); the HistorianGateway exposes no + modified/edited-history surface, so on `Gateway` (and with no historian) they return `BadHistoryOperationUnsupported`. ### Redundancy and authorization @@ -694,15 +734,18 @@ bootstrapper and one tag catalog, so the schema is ensured once per process and ### Schema -Created by the embedded idempotent script `Schema/001_initial.sql` (mirrored at -[`docs/historian/sqlserver/001_initial.sql`](historian/sqlserver/001_initial.sql)); `$(Schema)` is the configured schema. +Created by the embedded idempotent scripts `Schema/001_initial.sql` and `Schema/002_sample_modified.sql` (mirrored at +[`docs/historian/sqlserver/001_initial.sql`](historian/sqlserver/001_initial.sql) and +[`docs/historian/sqlserver/002_sample_modified.sql`](historian/sqlserver/002_sample_modified.sql)); `$(Schema)` is the configured schema. +The bootstrapper applies every script above the recorded version, in order, under the schema applock. | Object | Purpose | |---|---| -| `SchemaVersion(Version, AppliedUtc)` | Applied-script ledger. Required version: **1**. | +| `SchemaVersion(Version, AppliedUtc)` | Applied-script ledger. Required version: **2** (since #668). | | `Tag(TagId int identity, Name nvarchar(400) unique, DataType tinyint, EngineeringUnit, Description, CreatedUtc, UpdatedUtc)` | One row per historian tagname. `DataType` is a `DriverDataType`. | | `Sample(TagId, SourceTimeUtc datetime2(7), ValueKind tinyint, NumValue float, IntValue bigint, StrValue nvarchar(max), StatusCode int, ServerTimeUtc)` | **PK `(TagId, SourceTimeUtc)`** — one value per timestamp (Part 11), which makes redelivery idempotent and rules out tie clusters. `StatusCode` is the uint status stored bit-for-bit. | | `Event(EventId nvarchar(64) PK nonclustered, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, Message, UserName, Comment, EventTimeUtc, ReceivedTimeUtc)` | Alarm history. Clustered on `(EventTimeUtc, EventId)`; extra index `(SourceName, EventTimeUtc)`. | +| `SampleModified(TagId, SourceTimeUtc, ModifiedId bigint identity, ModificationTimeUtc, UpdateType tinyint, Principal nvarchar(200), ApiKeyId, ValueKind, NumValue, IntValue, StrValue, StatusCode, ServerTimeUtc)` (v2) | The Part 11 [modification trail](#modified-value-history-part-11): one row per sample a HistoryUpdate changed, carrying the **prior** value columns (all NULL for an insert). Clustered PK `(TagId, SourceTimeUtc, ModifiedId)`. ⚠️ No FOREIGN KEY, CHECK or trigger — it is an `OUTPUT … INTO` target, which forbids them; adding one fails every HistoryUpdate. | | `SampleRow`, `TimeRow` table types | TVPs for batched sample writes/merges, AtTime and delete-at-time. | Column mapping per `ValueKind`: `Double` → `NumValue`; `Int64` → `IntValue` + `NumValue` (as a double, for aggregates); @@ -710,9 +753,15 @@ Column mapping per `ValueKind`: `Double` → `NumValue`; `Int64` → `IntValue` `ValueKind` codes: Empty 0, Double 1, Int64 2, Boolean 3, String 4, DateTime 5. Timestamps are bound as `datetime2(7)` everywhere (full 100 ns precision — never a `DataTable`, which would round to 3.33 ms). -**Applying the script by hand** (needed when `AutoCreateSchema=false`): create the database, then -`sqlcmd -S -U -P -d -I -v Schema=hist -i docs/historian/sqlserver/001_initial.sql` -(`-v Schema=` supplies the `$(Schema)` placeholder; verified against the docker-dev SQL container). Re-running is a no-op. +**Applying the scripts by hand** (needed when `AutoCreateSchema=false`): create the database, then run each script in order — +`sqlcmd -S -U -P -d -I -v Schema=hist -i docs/historian/sqlserver/001_initial.sql`, then the same with +`002_sample_modified.sql` (`-v Schema=` supplies the `$(Schema)` placeholder; verified against the docker-dev SQL container). Re-running is a no-op. + +⚠️ **Upgrading an existing v1 historian (#668).** A build with this change requires schema **version 2**. With `AutoCreateSchema=true` +(the default) the first historian call after the upgrade applies `002_sample_modified.sql` in place (it only creates `SampleModified`; existing +samples are untouched). With `AutoCreateSchema=false` **every historian call fails** (`missing version 2 (found 1)`, in health `LastError`) +until an operator runs `002_sample_modified.sql` by hand as above — do that before rolling the new build out. Modifications made before the +upgrade have no trail; the trail starts with the first HistoryUpdate after it. ### Writes @@ -919,7 +968,7 @@ The read client reports `GetHealthSnapshot()` (query/failure counters, consecuti ### Retention (`RetentionDays`) -`hist.Sample` and `hist.Event` grow without bound unless `ServerHistorian:SqlServer:RetentionDays` is set. QuestDB already has an +`hist.Sample`, `hist.SampleModified` and `hist.Event` grow without bound unless `ServerHistorian:SqlServer:RetentionDays` is set. QuestDB already has an equivalent TTL (`QuestDb:RetentionDays`); the Gateway backend's retention is the AVEVA Historian's own. - **What runs.** `HistorianRetentionService` (a hosted service, driver-role nodes, Backend=`SqlServer` with `RetentionDays` set; inert @@ -931,15 +980,17 @@ equivalent TTL (`QuestDb:RetentionDays`); the Gateway backend's retention is the database** — if they differ, the smallest window wins. - **How it deletes.** Cutoff = `UtcNow - RetentionDays` on source time. Samples are purged **per tag** with `DELETE TOP (@n) ... WHERE TagId = @id AND SourceTimeUtc < @cutoff` (a seek on the clustered `(TagId, SourceTimeUtc)` key; a global - time predicate would scan), then events with `DELETE TOP (@n) ... WHERE EventTimeUtc < @cutoff` (seek on `CX_Event_Time`), each looped + time predicate would scan) — and, in the same per-tag pass, the modification trail with + `DELETE TOP (@n) FROM SampleModified WHERE TagId = @id AND SourceTimeUtc < @cutoff` (#668; same **source-time** cutoff, clustered seek; the + purge itself never writes a trail row) — then events with `DELETE TOP (@n) ... WHERE EventTimeUtc < @cutoff` (seek on `CX_Event_Time`), each looped until a batch comes back short. Cancellation is honoured between batches. The `hist.Tag` catalogue is never purged. - **Shared databases.** Several pairs may point at one database; a session `sp_getapplock` (`OtOpcUa.Historian.Retention.`, timeout 0) lets one purger run and the others skip that tick (Debug log). -- **Observability.** Counter `otopcua.historian.retention.purged{table=Sample|Event}`; a failed pass logs a Warning and retries next +- **Observability.** Counter `otopcua.historian.retention.purged{table=Sample|SampleModified|Event}`; a failed pass logs a Warning and retries next interval — it never affects capture or reads. - **Not done: partition switching.** For very large installs (billions of rows) a partitioned `hist.Sample` with sliding-window - `SWITCH`/drop is cheaper than row deletes; the purger is deliberately simple batched deletes. Another table (e.g. a future - `hist.SampleModified`, #668) is added by one more `DeleteLoopAsync` call in `SqlServerHistorianRetention.PurgeAsync`. + `SWITCH`/drop is cheaper than row deletes; the purger is deliberately simple batched deletes. Another table is added by one more + `DeleteLoopAsync` call in `SqlServerHistorianRetention.PurgeAsync` (as `hist.SampleModified` was, #668). ### Docker-dev recipe @@ -1004,10 +1055,10 @@ Read only when `Backend=QuestDb`; bound by `HistorianBackendRegistry.BindQuestDb | Key | Default | Validation | Meaning | |---|---|---|---| | `QuestDb:ConnectionString` | `""` | **Required** (non-blank) | Npgsql connection string for QuestDB's PG-wire port, e.g. `Host=questdb;Port=8812;Username=admin;Password=quest;Database=qdb;ServerCompatibilityMode=NoTypeLoading`. `ServerCompatibilityMode=NoTypeLoading` skips the `pg_type` load; every query form was verified with it. Supply via env `ServerHistorian__QuestDb__ConnectionString` / `${secret:}` for anything real. | -| `QuestDb:TablePrefix` | `hist_` | `^[A-Za-z0-9_]+$` | QuestDB has no schemas, so the three tables are `{prefix}sample`, `{prefix}event`, `{prefix}tag`. Spliced into DDL/DML, hence the regex. | -| `QuestDb:RetentionDays` | unset | `>= 1` when set | Applied as table TTL to `{prefix}sample` and `{prefix}event` (see [Retention](#retention-ttl)). Unset keeps data forever. | +| `QuestDb:TablePrefix` | `hist_` | `^[A-Za-z0-9_]+$` | QuestDB has no schemas, so the four tables are `{prefix}sample`, `{prefix}event`, `{prefix}tag` and `{prefix}sample_modified` (#668). Spliced into DDL/DML, hence the regex. | +| `QuestDb:RetentionDays` | unset | `>= 1` when set | Applied as table TTL to `{prefix}sample`, `{prefix}event` and `{prefix}sample_modified` (see [Retention](#retention-ttl)). Unset keeps data forever. | | `QuestDb:CommandTimeout` | `00:00:30` | `> 0` | Per-command timeout, rounded up to whole seconds (minimum 1 s). | -| `QuestDb:AutoCreateSchema` | `true` | — | `true`: create the tables (and reconcile the TTL) on first use. `false`: only verify the three tables exist; if one is missing every historian call fails with `HistorianSchemaMissingException` (health `LastError`, no crash) and no TTL is touched. A failed bootstrap is not latched: it is re-thrown for 5 s without touching QuestDB, then retried. | +| `QuestDb:AutoCreateSchema` | `true` | — | `true`: create the tables (and reconcile the TTL) on first use. `false`: only verify the four tables exist (⚠️ an install from before #668 lacks `{prefix}sample_modified` — create it with the DDL in `QuestDbTables.CreateSampleModified` before upgrading, or every call fails); if one is missing every historian call fails with `HistorianSchemaMissingException` (health `LastError`, no crash) and no TTL is touched. A failed bootstrap is not latched: it is re-thrown for 5 s without touching QuestDB, then retried. | | `QuestDb:VisibilityTimeout` | `00:00:05` | `>= 0` | How long a write waits for its rows to become visible to queries ([WAL visibility](#wal-visibility-and-the-read-your-writes-wait)). `0` disables the wait. | ⚠️ **Connection pool size.** QuestDB's default limit is **64** concurrent PG-wire connections (`pg.net.connection.limit`), while Npgsql's @@ -1019,14 +1070,15 @@ limit is server-configurable. The effective value is logged once at Information ### Tables -All three are **WAL tables with `DEDUP UPSERT KEYS`**, so re-sending a row with the same key replaces it (the upsert is what makes -HistoryUpdate Replace work; the writer's own insert-if-absent semantics are built on top, below). +All are **WAL tables**; the first three carry **`DEDUP UPSERT KEYS`**, so re-sending a row with the same key replaces it (the upsert is what +makes HistoryUpdate Replace work; the writer's own insert-if-absent semantics are built on top, below). The modification trail is append-only. | Table | Key | Partitioning | |---|---|---| | `{p}sample` (`ts, tag, kind, num_value, int_value, str_value, status, deleted, server_ts`) | `(ts, tag)` | `PARTITION BY DAY` (TTL when configured) | | `{p}event` (one-for-one with `hist.Event`) | `(ts, event_id)` | `PARTITION BY DAY` (TTL when configured) | | `{p}tag` (`ts, name, data_type, eng_unit, description, updated_utc`) | `(ts, name)` with `ts` fixed at `1970-01-01` — an upsert by name | `PARTITION BY YEAR`, never expires | +| `{p}sample_modified` (`ts` = the sample's source time, `tag, modified_id, modification_ts, update_type, principal, api_key_id, kind, num_value, int_value, str_value, status, server_ts`) (#668) | none — append-only, no DEDUP; `modified_id` (process-monotonic ≈ UTC ticks) orders a timestamp's trail | `PARTITION BY DAY` (TTL when configured). A NULL `status` marks "no prior value" (QuestDB `BYTE` has no NULL, so `kind` is 0) | `status` holds the uint OPC UA status code **unsigned** in a `LONG` (so "Bad" is `status >= 2147483648`, not `< 0`); `int_value` also holds Booleans as 0/1; `kind` is the `HistorianValueKind` in a `BYTE`. Tag names are limited to 400 characters (longer is a per-tag failure, @@ -1093,6 +1145,11 @@ present **and not deleted**. | DeleteAtTime | write a tombstone (`deleted = true`, value NULL, `BadNoData`) where a live row exists | `Good` / `BadNoEntryExists` | | DeleteRaw | tombstone every live row in `[start, end)`, 1,000 per page, waiting for visibility between pages | the same codes as `SqlServer` | +**Modification trail (#668).** Each operation first appends one `{p}sample_modified` row per sample it will change, carrying the live row's +current value (read by the same existence query), then writes the change. No multi-table transaction exists, so this order is the guarantee: +a crash between the two leaves an **orphan trail row**, never an untracked change — the documented deviation from `SqlServer`'s atomic trail. +See [Modified-value history](#modified-value-history-part-11). + The per-item codes come from the shared `HistoryUpdatePlanner`, so they equal `SqlServer`'s (including the duplicate-timestamp-in-one-request rules above). **Every read and aggregate filters `NOT deleted`.** A client Insert at a deleted timestamp re-populates it (the OPC UA intent); only the *live writer's* redelivery is barred from doing so. @@ -1118,7 +1175,7 @@ Every query filters `NOT deleted`; every value goes through `StoredSample`, so t ### Retention (TTL) -`RetentionDays` becomes `TTL n DAYS` on `{p}sample` and `{p}event` (never on `{p}tag`: TTL must be a multiple of the partition size). QuestDB drops +`RetentionDays` becomes `TTL n DAYS` on `{p}sample`, `{p}event` and `{p}sample_modified` (never on `{p}tag`: TTL must be a multiple of the partition size). QuestDB drops expired day partitions itself, tombstones included. - At bootstrap the current TTL is read from `tables()` (`ttlValue`, `ttlUnit`). QuestDB **normalises the unit** (7 days reads `1 WEEK`), so the @@ -1182,7 +1239,8 @@ batch limiter and `HistoryReadDeadline`. | `DeleteAtTimeDetails` | `HistoryDeleteAtTime` | per time: `Good`; `BadNoEntryExists` | Each backend call is applied as a unit (SqlServer: one transaction, `MERGE … OUTPUT $action` against the TVP; QuestDb: an INSERT per -chunk plus the WAL-visibility wait, no transaction — see [QuestDB backend](#questdb-backend)). The node-level `StatusCode` stays `Good` when item results are +chunk plus the WAL-visibility wait, no transaction — see [QuestDB backend](#questdb-backend)). Every applied item is also recorded, with the +caller's principal, in the [modification trail](#modified-value-history-part-11) (#668). The node-level `StatusCode` stays `Good` when item results are mixed (Part 11 reports item outcomes in `OperationResults`); it is Bad only for whole-node failures. **Per-item rejections** (answered without touching SQL; other items still proceed): a null item ⇒ `BadInvalidArgument`; @@ -1206,8 +1264,8 @@ outcome, later ones `BadNoEntryExists`. **Unknown tag:** Insert / Update create (DeleteRaw: `BadNoData`) without creating anything. **Unsupported ⇒ `BadHistoryOperationUnsupported`** (per node): the node is not historized; the backend has no updater (`Gateway`, or -`ServerHistorian:Enabled=false`); `PerformUpdateType.Remove`; `DeleteRawModifiedDetails.IsDeleteModified=true` (no modified-value history is -kept); and the out-of-scope detail types — **event** HistoryUpdate (`UpdateEventDetails` / `DeleteEventDetails`) and `UpdateStructureData` — +`ServerHistorian:Enabled=false`); `PerformUpdateType.Remove`; `DeleteRawModifiedDetails.IsDeleteModified=true` (the +[modification trail](#modified-value-history-part-11) is an audit record, aged out only by retention); and the out-of-scope detail types — **event** HistoryUpdate (`UpdateEventDetails` / `DeleteEventDetails`) and `UpdateStructureData` — which are not overridden, so the SDK base rejects them. A backend exception is `BadInternalError` for that node and a deadline is `BadTimeout`. No `AuditHistoryUpdateEvent` is emitted. diff --git a/docs/historian/sqlserver/002_sample_modified.sql b/docs/historian/sqlserver/002_sample_modified.sql new file mode 100644 index 00000000..0d9f1988 --- /dev/null +++ b/docs/historian/sqlserver/002_sample_modified.sql @@ -0,0 +1,29 @@ +-- OtOpcUa SQL Server historian backend - schema version 2: OPC UA Part 11 modified-value history (Gitea #668). +-- $(Schema) is substituted by SqlServerSchemaBootstrapper (identifier-validated) BEFORE the script is split on GO lines. +-- Every statement is guarded so a re-run is a no-op. +-- +-- One row per sample a HistoryUpdate inserted, replaced or deleted, carrying the PRIOR value (all NULL for an +-- insert). Rows are written by the SAME statement that changes hist.Sample (OUTPUT ... INTO), so a failed update +-- leaves no audit row and a committed one always has its row. +-- WARNING: OUTPUT ... INTO forbids, on its target table: enabled triggers, either side of a FOREIGN KEY, and CHECK +-- constraints. Do not add any of those to SampleModified, or every HistoryUpdate fails. +IF OBJECT_ID(N'[$(Schema)].[SampleModified]', N'U') IS NULL +CREATE TABLE [$(Schema)].[SampleModified] ( + TagId int NOT NULL, + SourceTimeUtc datetime2(7) NOT NULL, + ModifiedId bigint IDENTITY(1,1) NOT NULL, + ModificationTimeUtc datetime2(7) NOT NULL, + UpdateType tinyint NOT NULL, + Principal nvarchar(200) NOT NULL, + ApiKeyId nvarchar(200) NULL, + ValueKind tinyint NULL, + NumValue float NULL, + IntValue bigint NULL, + StrValue nvarchar(max) NULL, + StatusCode int NULL, + ServerTimeUtc datetime2(7) NULL, + CONSTRAINT PK_SampleModified PRIMARY KEY CLUSTERED (TagId, SourceTimeUtc, ModifiedId)) +GO +IF NOT EXISTS (SELECT 1 FROM [$(Schema)].[SchemaVersion] WHERE Version = 2) +INSERT [$(Schema)].[SchemaVersion] (Version, AppliedUtc) VALUES (2, SYSUTCDATETIME()) +GO diff --git a/docs/security.md b/docs/security.md index b11835c3..710e3598 100644 --- a/docs/security.md +++ b/docs/security.md @@ -884,6 +884,12 @@ HistoryUpdate rewrites recorded history, so unlike HistoryRead it is gated **wha - **Flag off (default):** the session's API key is re-checked on the call (`DataPlaneAuthorizer.AuthorizeKeyUsable`, Gitea #563 — an expired or disabled key loses its activation-time roles), then the session must carry the **`HistoryUpdate` role** (`OpcUaDataPlaneRoles.HistoryUpdate`). `ApiKeyInterimRoleMapper` confers it **only from a `Cluster`-scoped grant** carrying `NodePermissions.HistoryUpdate` — narrower-scoped grants confer nothing, for the same reason as `WriteOperate` (honouring a flag without honouring its scope would widen it). A key holding only `WriteOperate` is denied `BadUserAccessDenied`. - **Flag on:** `DataPlaneAuthorizer.Authorize(identity, OpcUaOperation.HistoryUpdate, nodeId, realm)` decides per node and **replaces** the role check (never chained) — a session with no roles succeeds on the strength of a scoped `HistoryUpdate` grant, and a `WriteOperate` grant does not help. - **Availability is separate from authorization.** A node accepts HistoryUpdate only if it is historized **and** the selected historian backend has an updater when the node materialises (the `HistoryWrite` AccessLevel bit). `Backend=Gateway` has none, so it answers `BadHistoryOperationUnsupported` for every node whatever the grant. Operations, result codes and unsupported cases: [`docs/Historian.md` §HistoryUpdate](Historian.md#historyupdate). +- **Every applied change is attributed (Part 11 modified-value history, Gitea #668).** The session's principal (the API key's + `NodeAcl` principal, or the reserved `anonymous`) and key id are passed to the backend with every HistoryUpdate and recorded in the + modification trail next to the prior value; a modified-value HistoryRead (`IsReadModified = true`) returns it as + `ModificationInfo.UserName`. Reading the trail is gated by the **same `HistoryRead` decision** as a raw read — there is no separate + permission — so with `Security:Acl:Enabled` off it is as ungated as any HistoryRead, and anyone who may read a node's history can see + who rewrote it and what it held before. See [`docs/Historian.md` §Modified-value history](Historian.md#modified-value-history-part-11). ### Evaluator — `PermissionTrie` diff --git a/src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/Commands/HistoryReadCommand.cs b/src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/Commands/HistoryReadCommand.cs index 8ea809c4..573c758f 100644 --- a/src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/Commands/HistoryReadCommand.cs +++ b/src/Client/ZB.MOM.WW.OtOpcUa.Client.CLI/Commands/HistoryReadCommand.cs @@ -56,6 +56,13 @@ public class HistoryReadCommand : CommandBase [CommandOption("interval", Description = "Processing interval in milliseconds for aggregates")] public double IntervalMs { get; init; } = 3600000; + /// + /// Gets a value indicating whether to read modified-value history (Part 11 IsReadModified) instead of raw samples: + /// the prior value each HistoryUpdate replaced or deleted, with its update type, user and modification time. + /// + [CommandOption("modified", Description = "Read modified-value history (what HistoryUpdate changed: prior value, update type, user, time) instead of raw samples")] + public bool Modified { get; init; } + /// public override async ValueTask ExecuteAsync(IConsole console) { @@ -64,6 +71,8 @@ public class HistoryReadCommand : CommandBase throw new CommandException($"--max must be greater than 0 (was {MaxValues})."); if (!string.IsNullOrEmpty(Aggregate) && IntervalMs <= 0) throw new CommandException($"--interval must be greater than 0 (was {IntervalMs})."); + if (Modified && !string.IsNullOrEmpty(Aggregate)) + throw new CommandException("--modified and --aggregate cannot be combined."); NodeId nodeId; try @@ -119,6 +128,30 @@ public class HistoryReadCommand : CommandBase var ct = console.RegisterCancellationHandler(); (service, _) = await CreateServiceAndConnectAsync(ct); + if (Modified) + { + await console.Output.WriteLineAsync( + $"Modified history for {NodeId} ({start:yyyy-MM-dd HH:mm} -> {end:yyyy-MM-dd HH:mm})"); + var entries = await service.HistoryReadModifiedAsync(nodeId, start, end, MaxValues, ct); + await console.Output.WriteLineAsync(); + await console.Output.WriteLineAsync( + $"{"Timestamp",-35} {"Prior value",-15} {"Status",-10} {"Update",-8} {"User",-20} {"Modified at"}"); + foreach (var e in entries) + { + var dv = e.Value; + var status = StatusCode.IsGood(dv.StatusCode) ? "Good" + : StatusCode.IsBad(dv.StatusCode) ? "Bad" + : "Uncertain"; + await console.Output.WriteLineAsync( + $"{dv.SourceTimestamp.ToString("O"),-35} {dv.Value?.ToString() ?? "(none)",-15} {status,-10} " + + $"{e.UpdateType,-8} {e.UserName ?? "",-20} {e.ModificationTime:O}"); + } + + await console.Output.WriteLineAsync(); + await console.Output.WriteLineAsync($"{entries.Count} modifications returned."); + return; + } + IReadOnlyList values; if (string.IsNullOrEmpty(Aggregate)) diff --git a/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Adapters/DefaultSessionAdapter.cs b/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Adapters/DefaultSessionAdapter.cs index 40a2205e..dd871594 100644 --- a/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Adapters/DefaultSessionAdapter.cs +++ b/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Adapters/DefaultSessionAdapter.cs @@ -206,6 +206,74 @@ internal sealed class DefaultSessionAdapter : ISessionAdapter return allValues; } + /// + public async Task> HistoryReadModifiedAsync( + NodeId nodeId, DateTime startTime, DateTime endTime, int maxValues, CancellationToken ct) + { + var details = new ReadRawModifiedDetails + { + StartTime = startTime, + EndTime = endTime, + NumValuesPerNode = (uint)maxValues, + IsReadModified = true, + ReturnBounds = false + }; + + var nodesToRead = new HistoryReadValueIdCollection + { + new HistoryReadValueId { NodeId = nodeId } + }; + + var all = new List(); + byte[]? continuationPoint = null; + + do + { + if (continuationPoint != null) + nodesToRead[0].ContinuationPoint = continuationPoint; + + var response = await _session.HistoryReadAsync( + null, + new ExtensionObject(details), + TimestampsToReturn.Both, + false, + nodesToRead, + ct).ConfigureAwait(false); + + var results = response.Results; + if (results == null || results.Count == 0) + break; + + var result = results[0]; + HistoryReadStatus.ThrowIfRefused(result.StatusCode, nodeId); + + if (result.HistoryData is ExtensionObject ext && ext.Body is HistoryModifiedData modified) + { + for (var i = 0; i < modified.DataValues.Count; i++) + { + var info = i < modified.ModificationInfos.Count ? modified.ModificationInfos[i] : null; + all.Add(new Models.HistoryModifiedEntry( + modified.DataValues[i], + info?.ModificationTime ?? DateTime.MinValue, + info?.UpdateType ?? default, + info?.UserName)); + } + } + + continuationPoint = result.ContinuationPoint; + } while (continuationPoint != null && continuationPoint.Length > 0 && all.Count < maxValues); + + // A read stopped by the client's own cap leaves a server cursor behind; release it. + if (continuationPoint is { Length: > 0 }) + { + nodesToRead[0].ContinuationPoint = continuationPoint; + await _session.HistoryReadAsync(null, new ExtensionObject(details), TimestampsToReturn.Both, true, + nodesToRead, ct).ConfigureAwait(false); + } + + return all; + } + /// public async Task> HistoryReadAggregateAsync( NodeId nodeId, DateTime startTime, DateTime endTime, NodeId aggregateId, double intervalMs, diff --git a/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Adapters/ISessionAdapter.cs b/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Adapters/ISessionAdapter.cs index 7fc340fe..f3d81e64 100644 --- a/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Adapters/ISessionAdapter.cs +++ b/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Adapters/ISessionAdapter.cs @@ -110,6 +110,20 @@ internal interface ISessionAdapter : IDisposable Task> HistoryReadRawAsync(NodeId nodeId, DateTime startTime, DateTime endTime, int maxValues, CancellationToken ct = default); + /// + /// Reads modified-value history (Part 11, IsReadModified = true): the prior values and modification info + /// the server recorded for every HistoryUpdate in the window, following continuation points. + /// + /// The historized node. + /// Window start. + /// Window end. + /// The maximum number of entries to return. + /// The cancellation token. + /// The modification entries in server order. + Task> HistoryReadModifiedAsync(NodeId nodeId, DateTime startTime, + DateTime endTime, int maxValues, CancellationToken ct = default) => + throw new NotSupportedException("This session adapter does not support modified-value history reads."); + /// /// Sends a HistoryUpdate request for the supplied details. /// diff --git a/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/IOpcUaClientService.cs b/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/IOpcUaClientService.cs index c29ea0a3..207197ea 100644 --- a/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/IOpcUaClientService.cs +++ b/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/IOpcUaClientService.cs @@ -193,6 +193,20 @@ public interface IOpcUaClientService : IDisposable Task> HistoryReadRawAsync(NodeId nodeId, DateTime startTime, DateTime endTime, int maxValues = 1000, CancellationToken ct = default); + /// + /// Reads modified-value history (Part 11, IsReadModified = true): what each HistoryUpdate in the window + /// replaced or deleted, with the server's modification time, update type and user name. + /// + /// The historized node. + /// Window start. + /// Window end. + /// The maximum number of entries to return. + /// The cancellation token. + /// The modification entries in server order. + Task> HistoryReadModifiedAsync(NodeId nodeId, DateTime startTime, + DateTime endTime, int maxValues = 1000, CancellationToken ct = default) => + throw new NotSupportedException("This client service does not support modified-value history reads."); + /// /// Sends a HistoryUpdate request (insert/replace/update or delete) for the supplied details. /// diff --git a/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Models/HistoryModifiedEntry.cs b/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Models/HistoryModifiedEntry.cs new file mode 100644 index 00000000..6662f42e --- /dev/null +++ b/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/Models/HistoryModifiedEntry.cs @@ -0,0 +1,17 @@ +using Opc.Ua; + +namespace ZB.MOM.WW.OtOpcUa.Client.Shared.Models; + +/// +/// One entry of an OPC UA Part 11 modified-value history read (HistoryReadRaw with IsReadModified = true): the +/// value as it was BEFORE the modification plus the server's . +/// +/// The prior value at its source timestamp (an insert had none: null value, Bad status). +/// When the modification was applied. +/// The Part 11 update type (Insert / Replace / Update / Delete). +/// Who made the modification (the server's recorded principal). +public sealed record HistoryModifiedEntry( + DataValue Value, + DateTime ModificationTime, + HistoryUpdateType UpdateType, + string? UserName); diff --git a/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/OpcUaClientService.cs b/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/OpcUaClientService.cs index 0c8b4e49..a2350a89 100644 --- a/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/OpcUaClientService.cs +++ b/src/Client/ZB.MOM.WW.OtOpcUa.Client.Shared/OpcUaClientService.cs @@ -559,6 +559,15 @@ public sealed class OpcUaClientService : IOpcUaClientService return await _session!.HistoryReadRawAsync(nodeId, startTime, endTime, maxValues, ct); } + /// + public async Task> HistoryReadModifiedAsync( + NodeId nodeId, DateTime startTime, DateTime endTime, int maxValues = 1000, CancellationToken ct = default) + { + ThrowIfDisposed(); + ThrowIfNotConnected(); + return await _session!.HistoryReadModifiedAsync(nodeId, startTime, endTime, maxValues, ct); + } + /// public async Task HistoryUpdateAsync(HistoryUpdateDetails details, CancellationToken ct = default) { diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/HistoryModifiedContracts.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/HistoryModifiedContracts.cs new file mode 100644 index 00000000..aa1c03f7 --- /dev/null +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/HistoryModifiedContracts.cs @@ -0,0 +1,69 @@ +namespace ZB.MOM.WW.OtOpcUa.Core.Abstractions; + +// OPC UA Part 11 modified-value history (Gitea #668): every HistoryUpdate a backend applies leaves an audit row +// carrying the PRIOR value, who made the change and when; HistoryRead with IsReadModified=true serves them. + +/// Who is performing a HistoryUpdate. Threaded into every call. +/// +/// The session's authenticated principal (an API key's principal, or the reserved anonymous). Recorded as +/// the modification's user name. +/// +/// The API key id the session authenticated with, or null (anonymous / non-key). +public sealed record HistoryUpdateContext(string Principal, string? ApiKeyId) +{ + /// The longest principal / key id a backend stores; longer values are truncated (never refused). + public const int MaxIdentityLength = 200; + + /// The principal clamped to characters. + public string StoredPrincipal => Clamp(Principal) ?? string.Empty; + + /// The key id clamped to characters. + public string? StoredApiKeyId => Clamp(ApiKeyId); + + private static string? Clamp(string? s) => s is { Length: > MaxIdentityLength } ? s[..MaxIdentityLength] : s; +} + +/// +/// The OPC UA Part 11 HistoryUpdateType of a recorded modification. The numeric values are the spec's and +/// are persisted (SQL tinyint, QuestDB BYTE) — never renumber. +/// +public enum HistoryModificationType : byte +{ + /// A sample was inserted where none existed (no prior value). + Insert = 1, + + /// An existing sample was replaced. + Replace = 2, + + /// An upsert (PerformUpdateType.Update): replaced a sample, or inserted one (then no prior value). + Update = 3, + + /// A sample was deleted (DeleteAtTime / DeleteRaw). + Delete = 4, +} + +/// One recorded modification: the value as it was BEFORE the change, plus who changed it and when. +/// +/// The prior value at its source timestamp. An insert had no prior value: Value = null, +/// StatusCode = BadNoData. +/// +/// When the modification was applied. +/// The kind of modification. +/// The principal that made it. +/// The backend's ordering key within one source timestamp (paging cursor). +public sealed record HistoryModifiedValue( + DataValueSnapshot Value, + DateTime ModificationTimeUtc, + HistoryModificationType UpdateType, + string UserName, + long ModifiedId); + +/// Resume position for a paged modified-value read: strictly after (or before, reversed) this key. +/// The last returned modification's source time. +/// The last returned modification's id. +public sealed record HistoryModifiedCursor(DateTime SourceTimeUtc, long ModifiedId); + +/// Result of . +/// Modifications ordered by (SourceTimeUtc, ModifiedId) (descending for a reverse read). +/// true when the cap truncated the result and more modifications remain in the window. +public sealed record HistoryModifiedReadResult(IReadOnlyList Values, bool HasMore); diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataSource.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataSource.cs index beb0dea2..a135d765 100644 --- a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataSource.cs +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataSource.cs @@ -122,4 +122,32 @@ public interface IHistorianDataSource : IDisposable /// Same-timestamp ties are unaffected (they are paged within the timestamp, never stepped over). /// long TimestampResolutionTicks => 1; + + /// + /// Read modified-value history — OPC UA HistoryReadRaw with IsReadModified = true (Part 11, Gitea #668): + /// every modification a HistoryUpdate applied to the tag inside the window, each carrying the PRIOR value. + /// + /// + /// Window: forward (start < end) [start, end) ascending by (SourceTimeUtc, ModifiedId); + /// reverse (end < start) (end, start] descending; start == end the modifications at + /// exactly that time. continues strictly past that key in the read's direction. + /// The default throws — a backend that keeps no modification trail + /// (Gateway, Null) — which the server reports as BadHistoryOperationUnsupported. + /// + /// The historian tagname. + /// Window start (UTC). + /// Window end (UTC). + /// Cap on returned modifications; 0 is unbounded. + /// Resume cursor from a previous page, or null for the first page. + /// Cancellation. + /// The modifications, with set when truncated. + /// The backend keeps no modified-value history. + Task ReadModifiedAsync( + string fullReference, + DateTime startUtc, + DateTime endUtc, + uint maxValues, + HistoryModifiedCursor? resumeAfter, + CancellationToken cancellationToken) => + throw new NotSupportedException("This historian backend keeps no modified-value history."); } diff --git a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataUpdater.cs b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataUpdater.cs index b1968c74..7890fbd9 100644 --- a/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataUpdater.cs +++ b/src/Core/ZB.MOM.WW.OtOpcUa.Core.Abstractions/Historian/IHistorianDataUpdater.cs @@ -17,12 +17,20 @@ public enum HistoryUpdateMode /// Optional historian capability behind OPC UA HistoryUpdate. A backend that cannot modify history /// supplies no updater (the server then reports HistoryUpdate as unsupported). /// +/// +/// Modified-value history (Part 11, Gitea #668). Every sample an implementation inserts, replaces or deletes +/// MUST leave one modification record carrying the prior value (none for an insert), the +/// , the modification time and the , +/// readable through . A refused item (any non-Good code) leaves +/// no record. Live recorder writes and retention purges never leave one. +/// public interface IHistorianDataUpdater { /// Inserts, replaces or upserts samples for one tag. /// Historian tag name. /// Update mode. /// Samples to apply. + /// Who is making the change (recorded in the modification trail). /// Cancellation token. /// /// One status code per value, in order: Insert gives GoodEntryInserted or BadEntryExists; @@ -30,20 +38,24 @@ public interface IHistorianDataUpdater /// GoodEntryInserted or GoodEntryReplaced; a null timestamp gives BadInvalidTimestamp. /// Task> UpdateDataAsync(string tag, HistoryUpdateMode mode, - IReadOnlyList values, CancellationToken ct); + IReadOnlyList values, HistoryUpdateContext context, CancellationToken ct); /// Deletes raw samples in a time range. /// Historian tag name. /// Range start (UTC). /// Range end (UTC). + /// Who is making the change (recorded in the modification trail). /// Cancellation token. /// Good, or BadNoData when the range held nothing. - Task DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, CancellationToken ct); + Task DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, HistoryUpdateContext context, + CancellationToken ct); /// Deletes the samples at exact timestamps. /// Historian tag name. /// Timestamps to delete (UTC). + /// Who is making the change (recorded in the modification trail). /// Cancellation token. /// One status code per time, in order: Good or BadNoEntryExists. - Task> DeleteAtTimeAsync(string tag, IReadOnlyList timesUtc, CancellationToken ct); + Task> DeleteAtTimeAsync(string tag, IReadOnlyList timesUtc, HistoryUpdateContext context, + CancellationToken ct); } diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbConnection.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbConnection.cs index 5d965006..d3c7db2d 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbConnection.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbConnection.cs @@ -83,6 +83,8 @@ internal sealed class QuestDbConnection : IAsyncDisposable, IDisposable public string EventTable => T(QuestDbTables.EventBase); /// The prefixed tag table. public string TagTable => T(QuestDbTables.TagBase); + /// The prefixed Part 11 modification-trail table (#668). + public string SampleModifiedTable => T(QuestDbTables.SampleModifiedBase); /// Opens a pooled connection (no bootstrap; callers run the bootstrapper first). /// Cancellation. diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs index 28042318..95eb33fb 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataSource.cs @@ -140,6 +140,101 @@ public sealed class QuestDbHistorianDataSource : IHistorianDataSource } } + // ------------------------------------------------------------------------------------------ Modified (Part 11, #668) + + /// + /// + /// Reads {prefix}sample_modified, the append-only trail writes + /// BEFORE each change. Windows follow (bounds rounded to µs); rows are ordered by + /// (ts, modified_id); maxValues + 1 rows are fetched to report + /// exactly. A row whose status is NULL had no prior value + /// (an insert) and is served null / BadNoData. Because the trail row lands first, a crash between it and + /// the sample write leaves an orphan row describing a change that never applied (documented). + /// + public async Task ReadModifiedAsync( + string fullReference, DateTime startUtc, DateTime endUtc, uint maxValues, HistoryModifiedCursor? resumeAfter, + CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(fullReference); + var bounded = maxValues > 0 && maxValues < int.MaxValue; + try + { + var values = new List(); + var hasMore = false; + if (await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false)) + { + var start = QuestDbConnection.Round(startUtc); + var end = QuestDbConnection.Round(endUtc); + var reverse = start > end; + string where; + if (start == end) where = "ts = $2 AND ts = $3"; + else if (!reverse) where = "ts >= $2 AND ts < $3"; + else where = "ts > $2 AND ts <= $3"; + var next = 4; + if (resumeAfter is not null) + { + where += reverse + ? " AND (ts < $4 OR (ts = $4 AND modified_id < $5))" + : " AND (ts > $4 OR (ts = $4 AND modified_id > $5))"; + next = 6; + } + var order = reverse ? "DESC" : "ASC"; + var (lo, hi) = start <= end ? (start, end) : (end, start); + + await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false); + await using var cmd = _conn.CreateCommand(conn, + $"SELECT {SampleColumns}, modified_id, modification_ts, update_type, principal FROM {_conn.SampleModifiedTable} " + + $"WHERE tag = $1 AND {where} ORDER BY ts {order}, modified_id {order}" + (bounded ? $" LIMIT ${next}" : "")); + QuestDbConnection.BindText(cmd, fullReference); + QuestDbConnection.BindTimestamp(cmd, lo); + QuestDbConnection.BindTimestamp(cmd, hi); + if (resumeAfter is not null) + { + QuestDbConnection.BindTimestamp(cmd, resumeAfter.SourceTimeUtc); + QuestDbConnection.BindLong(cmd, resumeAfter.ModifiedId); + } + if (bounded) + cmd.Parameters.Add(new NpgsqlParameter { NpgsqlDbType = NpgsqlDbType.Integer, Value = (int)maxValues + 1 }); + + await using var reader = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false); + while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false)) + { + if (bounded && values.Count == maxValues) + { + hasMore = true; + break; + } + + var sourceTime = QuestDbConnection.ReadUtc(reader, 0); + var modifiedAt = QuestDbConnection.ReadUtc(reader, 8); + DataValueSnapshot prior; + if (reader.IsDBNull(5)) + { + prior = new DataValueSnapshot(null, HistorianStatusCodes.BadNoData, sourceTime, modifiedAt); + } + else + { + var s = ReadStored(reader, 0); + prior = new DataValueSnapshot(s.Value.ToObject(), s.Status, s.TimeUtc, ReadServerTime(reader, 6, modifiedAt)); + } + + values.Add(new HistoryModifiedValue(prior, modifiedAt, + (HistoryModificationType)QuestDbConnection.ReadKind(reader, 9), + reader.IsDBNull(10) ? string.Empty : reader.GetString(10), + reader.GetInt64(7))); + } + } + + RecordSuccess(); + return new HistoryModifiedReadResult(values, hasMore); + } + catch (Exception ex) + { + RecordFailure(ex); + throw; + } + } + // ------------------------------------------------------------------------------------------ AtTime /// diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs index f4e00820..91cdf513 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbHistorianDataUpdater.cs @@ -25,6 +25,13 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; /// 1970 (read bounds are not clamped). /// Read-your-writes: after writing, waits for the rows to become /// visible, bounded by VisibilityTimeout; a timeout only logs a Warning, never changes a code. +/// Modified-value history (Part 11, #668). Every sample this updater inserts, replaces or tombstones first +/// gets a row in {prefix}sample_modified carrying its PRIOR live value (the existence read already fetches it; +/// none for an insert — a tombstone is "absent", so an insert over one records no prior value), the caller's +/// , the and the modification time. QuestDB has +/// no multi-table transaction, so the trail row is written BEFORE the change: a crash (or a failed sample INSERT) +/// between the two leaves an orphan trail row describing a change that never landed — never a change without its +/// trail row. This is the documented deviation from SQL Server, whose trail commits atomically with the change. /// internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater { @@ -35,6 +42,7 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater private const int LockStripes = 64; private readonly SemaphoreSlim[] _locks = Enumerable.Range(0, LockStripes).Select(_ => new SemaphoreSlim(1, 1)).ToArray(); private long _liveRowsRead; + private static long _lastModifiedId; private readonly QuestDbConnection _conn; private readonly QuestDbTagCatalog _tags; private readonly QuestDbWalVisibility _wal; @@ -71,10 +79,11 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater /// public async Task> UpdateDataAsync(string tag, HistoryUpdateMode mode, - IReadOnlyList values, CancellationToken ct) + IReadOnlyList values, HistoryUpdateContext context, CancellationToken ct) { ArgumentNullException.ThrowIfNull(tag); ArgumentNullException.ThrowIfNull(values); + ArgumentNullException.ThrowIfNull(context); // RC1: a valid-timestamp item before 1970 can never be stored; refuse it and plan the rest. var final = new uint[values.Count]; @@ -100,7 +109,7 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater ? await _tags.ExistsAsync(tag, ct).ConfigureAwait(false) : await EnsureAsync(tag, plan.Winners[0].Row.Value.Kind, ct).ConfigureAwait(false); var outcome = known - ? await ApplyAsync(tag, mode, plan.Winners.Select(w => w.Row).ToList(), ct).ConfigureAwait(false) + ? await ApplyAsync(tag, mode, plan.Winners.Select(w => w.Row).ToList(), context, ct).ConfigureAwait(false) : new Dictionary(); planned = HistoryUpdatePlanner.Complete(plan, mode, outcome); } @@ -109,9 +118,11 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater } /// - public async Task DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, CancellationToken ct) + public async Task DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, HistoryUpdateContext context, + CancellationToken ct) { ArgumentNullException.ThrowIfNull(tag); + ArgumentNullException.ThrowIfNull(context); if (!HistorianTagNames.IsValid(tag)) return HistorianStatusCodes.BadInvalidArgument; if (!await _tags.ExistsAsync(tag, ct).ConfigureAwait(false)) return HistoryUpdatePlanner.DeleteRawResult(tagKnown: false, 0); @@ -123,17 +134,27 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater while (cursor < end) { var page = new List(BatchSize); + var priors = new List(BatchSize); await using (var conn = await _conn.OpenAsync(ct).ConfigureAwait(false)) await using (var cmd = _conn.CreateCommand(conn, - $"SELECT ts FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts LIMIT {BatchSize}")) + $"SELECT {LiveColumns} FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts < $3 AND NOT deleted ORDER BY ts LIMIT {BatchSize}")) { QuestDbConnection.BindText(cmd, tag); QuestDbConnection.BindTimestamp(cmd, cursor); QuestDbConnection.BindTimestamp(cmd, end); await using var rd = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false); - while (await rd.ReadAsync(ct).ConfigureAwait(false)) page.Add(QuestDbConnection.ReadUtc(rd, 0)); + while (await rd.ReadAsync(ct).ConfigureAwait(false)) + { + var row = ReadLiveRow(rd); + page.Add(row.TimeUtc); + priors.Add(row); + } } if (page.Count == 0) break; + // Trail BEFORE the change (no multi-table transaction): a crash here leaves an orphan trail row, never a + // tombstone without its trail row. + await WriteTrailAsync(tag, HistoryModificationType.Delete, + priors.Select(r => (r.TimeUtc, (LiveRow?)r)).ToList(), context, ct).ConfigureAwait(false); await WriteTombstonesAsync(tag, page, ct).ConfigureAwait(false); // The wait between pages is required: without it the next page could re-read rows just tombstoned. await _wal.WaitAsync(_conn.SampleTable, ct).ConfigureAwait(false); @@ -145,10 +166,12 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater } /// - public async Task> DeleteAtTimeAsync(string tag, IReadOnlyList timesUtc, CancellationToken ct) + public async Task> DeleteAtTimeAsync(string tag, IReadOnlyList timesUtc, + HistoryUpdateContext context, CancellationToken ct) { ArgumentNullException.ThrowIfNull(tag); ArgumentNullException.ThrowIfNull(timesUtc); + ArgumentNullException.ThrowIfNull(context); if (!HistorianTagNames.IsValid(tag)) { var refused = new uint[timesUtc.Count]; @@ -168,9 +191,14 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater using var _ = await LockAsync(tag, ct).ConfigureAwait(false); var live = await ReadLiveAsync(tag, candidates.Select(t => new DateTime(t, DateTimeKind.Utc)).ToList(), ct).ConfigureAwait(false); - var toDelete = candidates.Where(live.Contains).OrderBy(x => x).ToList(); + var toDelete = candidates.Where(live.ContainsKey).OrderBy(x => x).ToList(); foreach (var chunk in toDelete.Chunk(BatchSize)) + { + // Trail BEFORE the change (see the class remarks). + await WriteTrailAsync(tag, HistoryModificationType.Delete, + chunk.Select(t => (live[t].TimeUtc, (LiveRow?)live[t])).ToList(), context, ct).ConfigureAwait(false); await WriteTombstonesAsync(tag, chunk.Select(t => new DateTime(t, DateTimeKind.Utc)).ToList(), ct).ConfigureAwait(false); + } if (toDelete.Count > 0) await _wal.WaitAsync(_conn.SampleTable, ct).ConfigureAwait(false); deleted.UnionWith(toDelete); return HistoryUpdatePlanner.DeleteAtTimeResults(rounded, deleted, tagKnown: true); @@ -183,48 +211,162 @@ internal sealed class QuestDbHistorianDataUpdater : IHistorianDataUpdater } private async Task> ApplyAsync(string tag, HistoryUpdateMode mode, - List rows, CancellationToken ct) + List rows, HistoryUpdateContext context, CancellationToken ct) { // Key = the planner's tick (what Complete looks up); the stored time is Round(of the same row). var live = await ReadLiveAsync(tag, rows.Select(r => QuestDbConnection.Round(r.TimeUtc)).ToList(), ct).ConfigureAwait(false); var outcome = new Dictionary(); var toWrite = new List(); + var trail = new List<(DateTime, LiveRow?)>(); foreach (var r in rows) { - var present = live.Contains(QuestDbConnection.Round(r.TimeUtc).Ticks); + var stored = QuestDbConnection.Round(r.TimeUtc); + var present = live.TryGetValue(stored.Ticks, out var prior); if (mode == HistoryUpdateMode.Insert && present) continue; if (mode == HistoryUpdateMode.Replace && !present) continue; outcome[r.TimeUtc.Ticks] = present ? UpsertOutcome.Replaced : UpsertOutcome.Inserted; toWrite.Add(r); + trail.Add((stored, present ? prior : null)); } + // Trail BEFORE the change (see the class remarks). + await WriteTrailAsync(tag, ToModificationType(mode), trail, context, ct).ConfigureAwait(false); foreach (var chunk in toWrite.Chunk(BatchSize)) await WriteRowsAsync(tag, chunk, ct).ConfigureAwait(false); if (toWrite.Count > 0) await _wal.WaitAsync(_conn.SampleTable, ct).ConfigureAwait(false); return outcome; } - /// Ticks of the live (present, not deleted) rows near the given times: one range query per cluster (I-2). - private async Task> ReadLiveAsync(string tag, IReadOnlyList times, CancellationToken ct) + /// The live (present, not deleted) rows near the given times, keyed by stored ticks: one range query per cluster (I-2). + private async Task> ReadLiveAsync(string tag, IReadOnlyList times, CancellationToken ct) { - var live = new HashSet(); + var live = new Dictionary(); await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); foreach (var (lo, hi) in QuestDbTimeClusters.Cluster(times, QuestDbTimeClusters.ExistenceGap)) { await using var cmd = _conn.CreateCommand(conn, - $"SELECT ts FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts <= $3 AND NOT deleted"); + $"SELECT {LiveColumns} FROM {_conn.SampleTable} WHERE tag = $1 AND ts >= $2 AND ts <= $3 AND NOT deleted"); QuestDbConnection.BindText(cmd, tag); QuestDbConnection.BindTimestamp(cmd, lo); QuestDbConnection.BindTimestamp(cmd, hi); await using var rd = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false); while (await rd.ReadAsync(ct).ConfigureAwait(false)) { - live.Add(QuestDbConnection.ReadUtc(rd, 0).Ticks); + var row = ReadLiveRow(rd); + live[row.TimeUtc.Ticks] = row; Interlocked.Increment(ref _liveRowsRead); } } return live; } + /// The columns of a live sample the trail needs (prior value + status + server time). + private const string LiveColumns = "ts, kind, num_value, int_value, str_value, status, server_ts"; + + /// A live sample as it was before a modification. + private readonly record struct LiveRow(DateTime TimeUtc, HistorianValue Value, uint Status, DateTime? ServerTimeUtc); + + private static LiveRow ReadLiveRow(NpgsqlDataReader rd) + { + var value = QuestDbHistorianDataSource.ToValue( + QuestDbConnection.ReadKind(rd, 1), + rd.IsDBNull(2) ? null : rd.GetDouble(2), + rd.IsDBNull(3) ? null : rd.GetInt64(3), + rd.IsDBNull(4) ? null : rd.GetString(4)); + var status = rd.IsDBNull(5) ? HistorianStatusCodes.BadNoData : SampleColumnMapper.FromLongStatus(rd.GetInt64(5)); + return new LiveRow(QuestDbConnection.ReadUtc(rd, 0), value, status, + rd.IsDBNull(6) ? null : QuestDbConnection.ReadUtc(rd, 6)); + } + + /// The Part 11 modification type recorded for an update mode. + private static HistoryModificationType ToModificationType(HistoryUpdateMode mode) => mode switch + { + HistoryUpdateMode.Insert => HistoryModificationType.Insert, + HistoryUpdateMode.Replace => HistoryModificationType.Replace, + _ => HistoryModificationType.Update, + }; + + /// + /// A process-wide, strictly increasing modification id (≈ UTC ticks, bumped past the last issued value). It orders a + /// tag's trail within one source time; ids from two nodes interleave by wall clock, which is the modification order. + /// + internal static long NextModifiedId() + { + while (true) + { + var last = Interlocked.Read(ref _lastModifiedId); + var next = Math.Max(DateTime.UtcNow.Ticks, last + 1); + if (Interlocked.CompareExchange(ref _lastModifiedId, next, last) == last) return next; + } + } + + /// + /// Appends one modification-trail row per entry (prior = null ⇒ an insert: kind Empty, NULL value and + /// status), then waits for the trail's WAL visibility so a modified read right after the update sees it. + /// + private async Task WriteTrailAsync(string tag, HistoryModificationType type, IReadOnlyList<(DateTime TimeUtc, LiveRow? Prior)> entries, + HistoryUpdateContext context, CancellationToken ct) + { + if (entries.Count == 0) return; + var now = DateTime.UtcNow; + foreach (var chunk in entries.Chunk(BatchSize)) + { + await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); + await using var cmd = _conn.CreateCommand(conn, ""); + var sb = new System.Text.StringBuilder( + $"INSERT INTO {_conn.SampleModifiedTable} (ts, tag, modified_id, modification_ts, update_type, principal, " + + "api_key_id, kind, num_value, int_value, str_value, status, server_ts) VALUES "); + for (var i = 0; i < chunk.Length; i++) + { + var (time, prior) = chunk[i]; + QuestDbConnection.BindTimestamp(cmd, time, clampToEpoch: true); + QuestDbConnection.BindText(cmd, tag); + QuestDbConnection.BindLong(cmd, NextModifiedId()); + QuestDbConnection.BindTimestamp(cmd, now); + QuestDbConnection.BindByte(cmd, (byte)type); + QuestDbConnection.BindText(cmd, context.StoredPrincipal); + QuestDbConnection.BindText(cmd, context.StoredApiKeyId); + if (prior is { } p) + { + var c = SampleColumnMapper.ToColumns(p.Value); + QuestDbConnection.BindByte(cmd, c.ValueKind); + Typed(cmd, NpgsqlDbType.Double, c.NumValue); + Typed(cmd, NpgsqlDbType.Bigint, c.IntValue); + Typed(cmd, NpgsqlDbType.Varchar, c.StrValue); + QuestDbConnection.BindLong(cmd, SampleColumnMapper.ToLongStatus(p.Status)); + Typed(cmd, NpgsqlDbType.TimestampTz, p.ServerTimeUtc); + } + else + { + // No prior value: QuestDB BYTE has no NULL, so kind is Empty (0) and the NULL status marks "none". + QuestDbConnection.BindByte(cmd, (byte)HistorianValueKind.Empty); + Typed(cmd, NpgsqlDbType.Double, null); + Typed(cmd, NpgsqlDbType.Bigint, null); + Typed(cmd, NpgsqlDbType.Varchar, null); + Typed(cmd, NpgsqlDbType.Bigint, null); + Typed(cmd, NpgsqlDbType.TimestampTz, null); + } + + var b = i * 13; + if (i > 0) sb.Append(','); + sb.Append('('); + for (var q = 1; q <= 13; q++) sb.Append(q > 1 ? "," : "").Append('$').Append(b + q); + sb.Append(')'); + } + cmd.CommandText = sb.ToString(); + try + { + await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false); + } + catch (Exception ex) when (ex is not OperationCanceledException) + { + _log.LogWarning(ex, "QuestDB modification-trail INSERT of {Count} rows into {Table} failed; the update was not applied.", + chunk.Length, _conn.SampleModifiedTable); + throw; + } + } + await _wal.WaitAsync(_conn.SampleModifiedTable, ct).ConfigureAwait(false); + } + private Task WriteRowsAsync(string tag, IReadOnlyList rows, CancellationToken ct) => InsertAsync(rows.Count, ct, (cmd, i) => { diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbSchemaBootstrapper.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbSchemaBootstrapper.cs index 45d753f0..296f787d 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbSchemaBootstrapper.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbSchemaBootstrapper.cs @@ -6,7 +6,7 @@ using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb; /// -/// Lazily creates the three tables once per instance (CREATE TABLE IF NOT EXISTS), then reconciles the TTL +/// Lazily creates the four tables once per instance (CREATE TABLE IF NOT EXISTS), then reconciles the TTL /// of the sample and event tables with . /// /// Concurrency: two nodes starting together are safe. If a CREATE still fails with a server error diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTables.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTables.cs index c7a1ad21..595c10a9 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTables.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb/QuestDbTables.cs @@ -15,6 +15,8 @@ internal static class QuestDbTables public const string EventBase = "event"; /// Base name of the tag catalog table. public const string TagBase = "tag"; + /// Base name of the Part 11 modification-trail table (#668). + public const string SampleModifiedBase = "sample_modified"; /// The fixed designated timestamp of every tag row, which makes the dedup key an upsert by name. public static readonly DateTime TagEpoch = DateTime.UnixEpoch; @@ -25,12 +27,17 @@ internal static class QuestDbTables public static string Event(string prefix) => Name(prefix, EventBase); /// The prefixed tag table name. public static string Tag(string prefix) => Name(prefix, TagBase); + /// The prefixed modification-trail table name. + public static string SampleModified(string prefix) => Name(prefix, SampleModifiedBase); - /// All three table names in creation order. - public static string[] All(string prefix) => [Sample(prefix), Event(prefix), Tag(prefix)]; + /// All four table names in creation order. + public static string[] All(string prefix) => [Sample(prefix), Event(prefix), Tag(prefix), SampleModified(prefix)]; - /// The tables that carry a TTL (sample and event; the tag catalog never expires). - public static string[] Expiring(string prefix) => [Sample(prefix), Event(prefix)]; + /// + /// The tables that carry a TTL (sample, event and the modification trail, which ages out with the samples it + /// describes; the tag catalog never expires). + /// + public static string[] Expiring(string prefix) => [Sample(prefix), Event(prefix), SampleModified(prefix)]; /// CREATE TABLE IF NOT EXISTS for the sample table. public static string CreateSample(string prefix, int? retentionDays) => @@ -50,9 +57,22 @@ internal static class QuestDbTables $"CREATE TABLE IF NOT EXISTS {Tag(prefix)} (ts TIMESTAMP, name SYMBOL, data_type BYTE, eng_unit VARCHAR, " + "description VARCHAR, updated_utc TIMESTAMP) TIMESTAMP(ts) PARTITION BY YEAR WAL DEDUP UPSERT KEYS(ts, name)"; - /// The three CREATE statements in creation order. + /// + /// CREATE TABLE IF NOT EXISTS for the Part 11 modification trail (#668): one row per sample a HistoryUpdate + /// inserted, replaced or deleted, carrying the PRIOR value (NULL kind/value/status for an insert). ts is the + /// sample's source time (so the TTL ages it out with the samples); modified_id orders the trail within one + /// source time. Append-only: no DEDUP, never updated or deleted. + /// + public static string CreateSampleModified(string prefix, int? retentionDays) => + $"CREATE TABLE IF NOT EXISTS {SampleModified(prefix)} (ts TIMESTAMP, tag SYMBOL, modified_id LONG, " + + "modification_ts TIMESTAMP, update_type BYTE, principal VARCHAR, api_key_id VARCHAR, kind BYTE, num_value DOUBLE, " + + "int_value LONG, str_value VARCHAR, status LONG, server_ts TIMESTAMP) " + + $"TIMESTAMP(ts) PARTITION BY DAY{Ttl(retentionDays)} WAL"; + + /// The four CREATE statements in creation order (parallel to ). public static string[] CreateAll(string prefix, int? retentionDays) => - [CreateSample(prefix, retentionDays), CreateEvent(prefix, retentionDays), CreateTag(prefix)]; + [CreateSample(prefix, retentionDays), CreateEvent(prefix, retentionDays), CreateTag(prefix), + CreateSampleModified(prefix, retentionDays)]; /// ALTER TABLE t SET TTL n DAYS; null removes the TTL (SET TTL 0 DAYS). public static string AlterTtl(string table, int? retentionDays) => diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/Schema/002_sample_modified.sql b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/Schema/002_sample_modified.sql new file mode 100644 index 00000000..0d9f1988 --- /dev/null +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/Schema/002_sample_modified.sql @@ -0,0 +1,29 @@ +-- OtOpcUa SQL Server historian backend - schema version 2: OPC UA Part 11 modified-value history (Gitea #668). +-- $(Schema) is substituted by SqlServerSchemaBootstrapper (identifier-validated) BEFORE the script is split on GO lines. +-- Every statement is guarded so a re-run is a no-op. +-- +-- One row per sample a HistoryUpdate inserted, replaced or deleted, carrying the PRIOR value (all NULL for an +-- insert). Rows are written by the SAME statement that changes hist.Sample (OUTPUT ... INTO), so a failed update +-- leaves no audit row and a committed one always has its row. +-- WARNING: OUTPUT ... INTO forbids, on its target table: enabled triggers, either side of a FOREIGN KEY, and CHECK +-- constraints. Do not add any of those to SampleModified, or every HistoryUpdate fails. +IF OBJECT_ID(N'[$(Schema)].[SampleModified]', N'U') IS NULL +CREATE TABLE [$(Schema)].[SampleModified] ( + TagId int NOT NULL, + SourceTimeUtc datetime2(7) NOT NULL, + ModifiedId bigint IDENTITY(1,1) NOT NULL, + ModificationTimeUtc datetime2(7) NOT NULL, + UpdateType tinyint NOT NULL, + Principal nvarchar(200) NOT NULL, + ApiKeyId nvarchar(200) NULL, + ValueKind tinyint NULL, + NumValue float NULL, + IntValue bigint NULL, + StrValue nvarchar(max) NULL, + StatusCode int NULL, + ServerTimeUtc datetime2(7) NULL, + CONSTRAINT PK_SampleModified PRIMARY KEY CLUSTERED (TagId, SourceTimeUtc, ModifiedId)) +GO +IF NOT EXISTS (SELECT 1 FROM [$(Schema)].[SchemaVersion] WHERE Version = 2) +INSERT [$(Schema)].[SchemaVersion] (Version, AppliedUtc) VALUES (2, SYSUTCDATETIME()) +GO diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataSource.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataSource.cs index 0dbb7f05..530192c7 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataSource.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataSource.cs @@ -135,6 +135,97 @@ public sealed class SqlServerHistorianDataSource : IHistorianDataSource } } + // ------------------------------------------------------------------------------------------ Modified (Part 11, #668) + + /// + /// + /// Reads SampleModified (schema v2), the trail writes in the + /// same statement as each change. Windows follow ; rows are ordered by + /// (SourceTimeUtc, ModifiedId) (the clustered key after TagId, so a seek). maxValues + 1 rows + /// are fetched to report exactly. An insert's audit row carries no + /// prior value and is served null / BadNoData; its server timestamp is the modification time. An + /// unknown tag is an empty result. + /// + public async Task ReadModifiedAsync( + string fullReference, DateTime startUtc, DateTime endUtc, uint maxValues, HistoryModifiedCursor? resumeAfter, + CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(fullReference); + var bounded = maxValues > 0 && maxValues < int.MaxValue; + try + { + var values = new List(); + var hasMore = false; + var tagId = await ResolveAsync(fullReference, cancellationToken).ConfigureAwait(false); + if (tagId is { } id) + { + var reverse = startUtc > endUtc; + string where; + if (startUtc == endUtc) where = "SourceTimeUtc = @lo"; + else if (!reverse) where = "SourceTimeUtc >= @lo AND SourceTimeUtc < @hi"; + else where = "SourceTimeUtc > @lo AND SourceTimeUtc <= @hi"; + if (resumeAfter is not null) + where += reverse + ? " AND (SourceTimeUtc < @ct OR (SourceTimeUtc = @ct AND ModifiedId < @cid))" + : " AND (SourceTimeUtc > @ct OR (SourceTimeUtc = @ct AND ModifiedId > @cid))"; + var order = reverse ? "DESC" : "ASC"; + var (lo, hi) = startUtc <= endUtc ? (startUtc, endUtc) : (endUtc, startUtc); + + await using var conn = await _conn.OpenAsync(cancellationToken).ConfigureAwait(false); + await using var cmd = _conn.CreateCommand(conn, + $"SELECT {(bounded ? "TOP (@take) " : "")}SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode, " + + "ServerTimeUtc, ModifiedId, ModificationTimeUtc, UpdateType, Principal " + + $"FROM {_conn.Q("SampleModified")} WHERE TagId = @id AND {where} " + + $"ORDER BY SourceTimeUtc {order}, ModifiedId {order}"); + if (bounded) cmd.Parameters.Add(new SqlParameter("@take", SqlDbType.BigInt) { Value = (long)maxValues + 1 }); + cmd.Parameters.Add(new SqlParameter("@id", SqlDbType.Int) { Value = id }); + SqlHistorianConnection.AddDateTime2(cmd, "@lo", lo); + SqlHistorianConnection.AddDateTime2(cmd, "@hi", hi); + if (resumeAfter is not null) + { + SqlHistorianConnection.AddDateTime2(cmd, "@ct", resumeAfter.SourceTimeUtc); + cmd.Parameters.Add(new SqlParameter("@cid", SqlDbType.BigInt) { Value = resumeAfter.ModifiedId }); + } + + await using var reader = await cmd.ExecuteReaderAsync(cancellationToken).ConfigureAwait(false); + while (await reader.ReadAsync(cancellationToken).ConfigureAwait(false)) + { + if (bounded && values.Count == maxValues) + { + hasMore = true; + break; + } + + var sourceTime = SqlHistorianConnection.GetUtc(reader, 0); + var modifiedAt = SqlHistorianConnection.GetUtc(reader, 8); + DataValueSnapshot prior; + if (reader.IsDBNull(1)) + { + // An insert: there was no prior value. + prior = new DataValueSnapshot(null, HistorianStatusCodes.BadNoData, sourceTime, modifiedAt); + } + else + { + var s = ReadStored(reader, 0); + prior = new DataValueSnapshot(s.Value.ToObject(), s.Status, s.TimeUtc, + reader.IsDBNull(6) ? modifiedAt : SqlHistorianConnection.GetUtc(reader, 6)); + } + + values.Add(new HistoryModifiedValue( + prior, modifiedAt, (HistoryModificationType)reader.GetByte(9), reader.GetString(10), reader.GetInt64(7))); + } + } + + RecordSuccess(); + return new HistoryModifiedReadResult(values, hasMore); + } + catch (Exception ex) + { + RecordFailure(ex); + throw; + } + } + // ------------------------------------------------------------------------------------------ AtTime /// diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataUpdater.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataUpdater.cs index 057dfe45..d21d4ed2 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataUpdater.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianDataUpdater.cs @@ -22,6 +22,11 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; /// is written. /// Unknown tag. Insert/Update create it via ; Replace and the /// deletes report BadNoEntryExists (DeleteRaw: BadNoData) without creating anything. +/// Modified-value history (Part 11, #668). Every row the MERGE inserts or updates and every row a +/// delete removes is copied into SampleModified by an OUTPUT … INTO on the SAME statement — the prior +/// value from deleted.* (NULL for an insert), the caller's , the +/// and SYSUTCDATETIME(). One statement means the sample change and its +/// audit row commit or roll back together: a failed update leaves no audit row. /// internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater { @@ -39,10 +44,11 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater /// public async Task> UpdateDataAsync(string tag, HistoryUpdateMode mode, - IReadOnlyList values, CancellationToken ct) + IReadOnlyList values, HistoryUpdateContext context, CancellationToken ct) { ArgumentNullException.ThrowIfNull(tag); ArgumentNullException.ThrowIfNull(values); + ArgumentNullException.ThrowIfNull(context); var plan = HistoryUpdatePlanner.Plan(tag, mode, values); if (plan.Winners.Count == 0) return HistoryUpdatePlanner.Complete(plan, mode, new Dictionary()); @@ -52,14 +58,16 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater var outcome = tagId is null ? new Dictionary() - : await MergeAsync(tagId.Value, mode, plan.Winners.Select(w => w.Row).ToList(), ct).ConfigureAwait(false); + : await MergeAsync(tagId.Value, mode, plan.Winners.Select(w => w.Row).ToList(), context, ct).ConfigureAwait(false); return HistoryUpdatePlanner.Complete(plan, mode, outcome); } /// - public async Task DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, CancellationToken ct) + public async Task DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, HistoryUpdateContext context, + CancellationToken ct) { ArgumentNullException.ThrowIfNull(tag); + ArgumentNullException.ThrowIfNull(context); if (!HistorianTagNames.IsValid(tag)) return HistorianStatusCodes.BadInvalidArgument; var tagId = await _tags.TryGetTagIdAsync(tag, ct).ConfigureAwait(false); if (tagId is null) return HistoryUpdatePlanner.DeleteRawResult(tagKnown: false, 0); @@ -67,9 +75,11 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater await using var tx = (SqlTransaction)await conn.BeginTransactionAsync(ct).ConfigureAwait(false); int rows; await using (var cmd = _conn.CreateCommand(conn, - $"DELETE FROM {_conn.Q("Sample")} WHERE TagId = @tagId AND SourceTimeUtc >= @s AND SourceTimeUtc < @e", tx)) + $"DELETE FROM {_conn.Q("Sample")} {AuditInto(HistoryModificationType.Delete)} " + + "WHERE TagId = @tagId AND SourceTimeUtc >= @s AND SourceTimeUtc < @e", tx)) { cmd.Parameters.Add(new SqlParameter("@tagId", System.Data.SqlDbType.Int) { Value = tagId.Value }); + AddAuditParameters(cmd, context); SqlHistorianConnection.AddDateTime2(cmd, "@s", startUtc); SqlHistorianConnection.AddDateTime2(cmd, "@e", endUtc); rows = await cmd.ExecuteNonQueryAsync(ct).ConfigureAwait(false); @@ -79,10 +89,12 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater } /// - public async Task> DeleteAtTimeAsync(string tag, IReadOnlyList timesUtc, CancellationToken ct) + public async Task> DeleteAtTimeAsync(string tag, IReadOnlyList timesUtc, + HistoryUpdateContext context, CancellationToken ct) { ArgumentNullException.ThrowIfNull(tag); ArgumentNullException.ThrowIfNull(timesUtc); + ArgumentNullException.ThrowIfNull(context); if (!HistorianTagNames.IsValid(tag)) { var refused = new uint[timesUtc.Count]; @@ -100,9 +112,11 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); await using var tx = (SqlTransaction)await conn.BeginTransactionAsync(ct).ConfigureAwait(false); await using (var cmd = _conn.CreateCommand(conn, - $"DELETE s OUTPUT deleted.SourceTimeUtc FROM {_conn.Q("Sample")} AS s JOIN @times AS t ON s.SourceTimeUtc = t.T WHERE s.TagId = @tagId", tx)) + $"DELETE s {AuditInto(HistoryModificationType.Delete)} OUTPUT deleted.SourceTimeUtc " + + $"FROM {_conn.Q("Sample")} AS s JOIN @times AS t ON s.SourceTimeUtc = t.T WHERE s.TagId = @tagId", tx)) { cmd.Parameters.Add(new SqlParameter("@tagId", System.Data.SqlDbType.Int) { Value = tagId.Value }); + AddAuditParameters(cmd, context); _conn.AddStructured(cmd, "@times", "TimeRow", SampleRowMapper.ToTimeRecords(distinct.Select(t => new DateTime(t, DateTimeKind.Utc)))); await using var rd = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false); @@ -114,7 +128,7 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater } private async Task> MergeAsync(int tagId, HistoryUpdateMode mode, - List rows, CancellationToken ct) + List rows, HistoryUpdateContext context, CancellationToken ct) { var sql = $"MERGE {_conn.Q("Sample")} WITH (HOLDLOCK) AS t USING @rows AS r ON t.TagId = @tagId AND t.SourceTimeUtc = r.SourceTimeUtc "; if (mode != HistoryUpdateMode.Insert) @@ -123,7 +137,9 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater if (mode != HistoryUpdateMode.Replace) sql += "WHEN NOT MATCHED THEN INSERT (TagId, SourceTimeUtc, ValueKind, NumValue, IntValue, StrValue, StatusCode, ServerTimeUtc) " + "VALUES (@tagId, r.SourceTimeUtc, r.ValueKind, r.NumValue, r.IntValue, r.StrValue, r.StatusCode, SYSUTCDATETIME()) "; - sql += "OUTPUT $action, r.SourceTimeUtc;"; + // The MERGE's deleted.* is NULL for an inserted row, so one OUTPUT INTO records both an insert (no prior value) + // and a replace (the prior value). TagId comes from inserted.* — present on both branches. + sql += $"{AuditInto(ToModificationType(mode), "inserted")} OUTPUT $action, r.SourceTimeUtc;"; var outcome = new Dictionary(); await using var conn = await _conn.OpenAsync(ct).ConfigureAwait(false); @@ -131,6 +147,7 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater await using (var cmd = _conn.CreateCommand(conn, sql, tx)) { cmd.Parameters.Add(new SqlParameter("@tagId", System.Data.SqlDbType.Int) { Value = tagId }); + AddAuditParameters(cmd, context); _conn.AddStructured(cmd, "@rows", "SampleRow", SampleRowMapper.ToSampleRecords(rows)); await using var rd = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false); while (await rd.ReadAsync(ct).ConfigureAwait(false)) @@ -141,4 +158,33 @@ internal sealed class SqlServerHistorianDataUpdater : IHistorianDataUpdater await tx.CommitAsync(ct).ConfigureAwait(false); return outcome; } + + /// The Part 11 modification type recorded for an update mode. + /// The update mode. + /// Insert ⇒ Insert, Replace ⇒ Replace, Update ⇒ Update. + internal static HistoryModificationType ToModificationType(HistoryUpdateMode mode) => mode switch + { + HistoryUpdateMode.Insert => HistoryModificationType.Insert, + HistoryUpdateMode.Replace => HistoryModificationType.Replace, + _ => HistoryModificationType.Update, + }; + + /// + /// The OUTPUT … INTO SampleModified clause copying each affected row's prior value (deleted.*) into + /// the modification trail. names the pseudo-table that carries TagId / + /// SourceTimeUtc for every affected row (deleted for a DELETE, inserted for the MERGE). + /// + private string AuditInto(HistoryModificationType type, string keySource = "deleted") => + $"OUTPUT {keySource}.TagId, {keySource}.SourceTimeUtc, SYSUTCDATETIME(), {(byte)type}, @principal, @apiKeyId, " + + "deleted.ValueKind, deleted.NumValue, deleted.IntValue, deleted.StrValue, deleted.StatusCode, deleted.ServerTimeUtc " + + $"INTO {_conn.Q("SampleModified")} (TagId, SourceTimeUtc, ModificationTimeUtc, UpdateType, Principal, ApiKeyId, " + + "ValueKind, NumValue, IntValue, StrValue, StatusCode, ServerTimeUtc)"; + + private static void AddAuditParameters(SqlCommand cmd, HistoryUpdateContext context) + { + cmd.Parameters.Add(new SqlParameter("@principal", System.Data.SqlDbType.NVarChar, HistoryUpdateContext.MaxIdentityLength) + { Value = context.StoredPrincipal }); + cmd.Parameters.Add(new SqlParameter("@apiKeyId", System.Data.SqlDbType.NVarChar, HistoryUpdateContext.MaxIdentityLength) + { Value = (object?)context.StoredApiKeyId ?? DBNull.Value }); + } } diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianOptions.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianOptions.cs index 84a26732..4828e4ac 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianOptions.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianOptions.cs @@ -18,7 +18,8 @@ public sealed partial class SqlServerHistorianOptions /// /// true creates the schema and applies missing scripts on first use; false only verifies - /// that schema version 1 exists and fails every historian call with a clear error otherwise. + /// that the required schema version (SqlServerSchemaBootstrapper.RequiredVersion, currently 2) exists and + /// fails every historian call with a clear error otherwise. /// public bool AutoCreateSchema { get; set; } = true; diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianRetention.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianRetention.cs index 52e96f0f..c8bc8e00 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianRetention.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerHistorianRetention.cs @@ -7,7 +7,7 @@ using ZB.MOM.WW.OtOpcUa.Runtime.Historian; namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; /// -/// Deletes hist.Sample / hist.Event rows older than the retention window, in small +/// Deletes hist.Sample / hist.SampleModified / hist.Event rows older than the retention window, in small /// DELETE TOP(n) batches (under the lock-escalation threshold). Samples are purged per tag so each /// delete is a seek on the clustered key (TagId, SourceTimeUtc); events seek on CX_Event_Time. /// A session sp_getapplock (timeout 0) serialises purgers that share one database; a loser skips. @@ -69,7 +69,7 @@ internal sealed class SqlServerHistorianRetention : IHistorianRetention return []; } - long samples = 0; + long samples = 0, modified = 0; var tagIds = new List(); await using (var cmd = _conn.CreateCommand(conn, $"SELECT TagId FROM {_conn.Q("Tag")}")) await using (var r = await cmd.ExecuteReaderAsync(ct).ConfigureAwait(false)) @@ -80,13 +80,23 @@ internal sealed class SqlServerHistorianRetention : IHistorianRetention samples += await DeleteLoopAsync(conn, $"DELETE TOP (@n) FROM {_conn.Q("Sample")} WHERE TagId = @id AND SourceTimeUtc < @cutoff", cutoff, id, ct).ConfigureAwait(false); + // The Part 11 modification trail (#668) ages out on the SAME source-time cutoff as the samples it + // describes; a purge never writes to it (no trigger, no OUTPUT INTO). + modified += await DeleteLoopAsync(conn, + $"DELETE TOP (@n) FROM {_conn.Q("SampleModified")} WHERE TagId = @id AND SourceTimeUtc < @cutoff", + cutoff, id, ct).ConfigureAwait(false); } var events = await DeleteLoopAsync(conn, $"DELETE TOP (@n) FROM {_conn.Q("Event")} WHERE EventTimeUtc < @cutoff", cutoff, null, ct).ConfigureAwait(false); - return [new RetentionPurgeResult("Sample", samples), new RetentionPurgeResult("Event", events)]; + return + [ + new RetentionPurgeResult("Sample", samples), + new RetentionPurgeResult("SampleModified", modified), + new RetentionPurgeResult("Event", events), + ]; } finally { diff --git a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerSchemaBootstrapper.cs b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerSchemaBootstrapper.cs index 5539e251..c1c1a782 100644 --- a/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerSchemaBootstrapper.cs +++ b/src/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer/SqlServerSchemaBootstrapper.cs @@ -14,12 +14,16 @@ namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer; /// retries (). Within that window the last failure is re-thrown without touching /// SQL, so a missing schema (or a SQL outage) does not serialise every write, read and alarm event behind one /// semaphore and one round-trip each. -/// With AutoCreateSchema=false it only verifies MAX(Version) >= 1. +/// With AutoCreateSchema=false it only verifies MAX(Version) >= . /// internal sealed partial class SqlServerSchemaBootstrapper { - /// The schema version this build requires. - public const int RequiredVersion = 1; + /// + /// The schema version this build requires: 2 = 002_sample_modified.sql (Part 11 modified-value + /// history, Gitea #668). An existing v1 schema is upgraded in place on the next bootstrap when + /// AutoCreateSchema is on; otherwise the script must be applied by hand. + /// + public const int RequiredVersion = 2; private const string LockResource = "OtOpcUaHistorianSchema"; private const int LockTimeoutMs = 30_000; @@ -48,7 +52,7 @@ internal sealed partial class SqlServerSchemaBootstrapper /// Ensures the schema is ready; no-op after the first success. /// Cancellation. /// A task completing when the schema is usable. - /// AutoCreateSchema is false and version 1 is absent. + /// AutoCreateSchema is false and is absent. public async Task EnsureAsync(CancellationToken ct) { if (_ready) return; @@ -94,7 +98,7 @@ internal sealed partial class SqlServerSchemaBootstrapper if (v < RequiredVersion) throw new HistorianSchemaMissingException( $"Historian schema [{_conn.Schema}] is missing version {RequiredVersion} (found {v}) and AutoCreateSchema is false. " + - "Apply docs/historian/sqlserver/001_initial.sql or enable AutoCreateSchema."); + "Apply the docs/historian/sqlserver/NNN_*.sql scripts above version " + v + " in order, or enable AutoCreateSchema."); } private async Task ApplyAsync(SqlConnection conn, CancellationToken ct) diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/HistoryPaging.cs b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/HistoryPaging.cs index c4c4ad53..8526d254 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/HistoryPaging.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/HistoryPaging.cs @@ -31,11 +31,20 @@ namespace ZB.MOM.WW.OtOpcUa.OpcUaServer; /// prior pages and must be dropped from the head of the next page (tie de-dup). /// /// The client's per-page cap; re-applied to every resumed page. -/// -/// true for an EVENT HistoryRead cursor (Gitea #669): then carries the -/// registered historian event-source name and the boundary fields are over EventTimeUtc. A cursor -/// is only ever resumed by the read kind that issued it, so a raw point presented to an events read -/// (or vice versa) is a BadContinuationPointInvalid miss. +/// +/// Which read issued the cursor (Gitea #669 / #668). A cursor is only ever resumed by the read kind that issued it, so a +/// raw point presented to an events or modified read (or any other mismatch) is a BadContinuationPointInvalid +/// miss. : carries the registered historian +/// event-source name and the boundary fields are over EventTimeUtc. +/// +/// +/// only: the ModifiedId of the last returned modification. With +/// (its source time) it is the exclusive resume key; +/// is unused because the key is unique. +/// +/// +/// only: the normalised window start, so a resumed page keeps the original +/// window and direction (the modified read resumes by key, not by moving the window start). /// internal sealed record HistoryContinuationState( string Tagname, @@ -43,7 +52,22 @@ internal sealed record HistoryContinuationState( DateTime EndUtc, int BoundarySkipCount, uint NumValuesPerNode, - bool IsEvents = false); + HistoryCursorKind Kind = HistoryCursorKind.Raw, + long LastModifiedId = 0, + DateTime WindowStartUtc = default); + +/// The HistoryRead kind that issued a . +internal enum HistoryCursorKind +{ + /// HistoryReadRaw (values). + Raw, + + /// HistoryReadEvents (#669). + Events, + + /// HistoryReadRaw with IsReadModified = true — the Part 11 modification trail (#668). + Modified, +} /// /// Pure server-side continuation-point paging decisions for the count-capped variable-history arms diff --git a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs index bb0ea345..33d623f7 100644 --- a/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs +++ b/src/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer/OtOpcUaNodeManager.cs @@ -4977,8 +4977,12 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 { if (details.IsReadModified) { - // We never serve modified-value history; mark this node unsupported and move on. - errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported; + // Part 11 modified-value history (#668): the prior values every HistoryUpdate recorded. Gated by the + // same HistoryRead authorization as a raw read (above); a backend with no trail ⇒ unsupported. + work.Add(() => ServeModifiedPagedAsync( + handle, session, nodesToRead, results, errors, + details.StartTime, details.EndTime, details.NumValuesPerNode, + ct)); continue; } @@ -5156,7 +5160,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 // for THIS node only. var state = _historyContinuationStore.TryTake(session, inboundCp!); // A point is bound to the node that issued it: another node's source name is a miss too. - if (state is null || !state.IsEvents || state.Tagname != (sourceName ?? string.Empty)) + if (state is null || state.Kind != HistoryCursorKind.Events || state.Tagname != (sourceName ?? string.Empty)) { errors[handle.Index] = StatusCodes.BadContinuationPointInvalid; results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadContinuationPointInvalid }; @@ -5237,7 +5241,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 if (nextStartUtc is { } resumeAt) { pendingCursor = new HistoryContinuationState( - sourceName ?? string.Empty, resumeAt, endUtc, nextSkip, numValuesPerNode, IsEvents: true); + sourceName ?? string.Empty, resumeAt, endUtc, nextSkip, numValuesPerNode, HistoryCursorKind.Events); } } else if (truncated && events.Count > 0) @@ -5247,7 +5251,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 var skip = 0; for (var i = events.Count - 1; i >= 0 && events[i].EventTimeUtc == boundary; i--) skip++; pendingCursor = new HistoryContinuationState( - sourceName ?? string.Empty, boundary, endUtc, skip, numValuesPerNode, IsEvents: true); + sourceName ?? string.Empty, boundary, endUtc, skip, numValuesPerNode, HistoryCursorKind.Events); } var historyEvent = ProjectEvents(events, selectClauses); @@ -5374,7 +5378,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 } var items = (IList?)details.UpdateValues ?? Array.Empty(); - return (updater, tagname, ct) => UpdateDataAsync(updater, tagname, mode, items, ct); + return (updater, tagname, who, ct) => UpdateDataAsync(updater, tagname, mode, items, who, ct); }); } @@ -5390,17 +5394,18 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 ServeHistoryUpdate(context, nodesToProcess, results, errors, handle => { var details = nodesToUpdate[handle.Index]; - // We never keep modified-value history, so there is none to delete. + // Deleting the modified-value trail itself (Part 11 IsDeleteModified) is not supported: the trail is the + // audit record of every change and is aged out only by retention (#668). if (details.IsDeleteModified) return null; var start = details.StartTime; var end = details.EndTime; // Part 11 requires both bounds of a raw delete; DateTime.MinValue is the wire's "unspecified". Refuse // it here rather than let a backend read MinValue as "from the beginning of time". if (start == DateTime.MinValue || end == DateTime.MinValue) - return (_, _, _) => Task.FromResult(new HistoryUpdateResult { StatusCode = StatusCodes.BadInvalidArgument }); - return async (updater, tagname, ct) => + return (_, _, _, _) => Task.FromResult(new HistoryUpdateResult { StatusCode = StatusCodes.BadInvalidArgument }); + return async (updater, tagname, who, ct) => { - var code = await updater.DeleteRawAsync(tagname, start, end, ct).ConfigureAwait(false); + var code = await updater.DeleteRawAsync(tagname, start, end, who, ct).ConfigureAwait(false); return new HistoryUpdateResult { StatusCode = code }; }; }); @@ -5418,11 +5423,11 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 ServeHistoryUpdate(context, nodesToProcess, results, errors, handle => { var times = nodesToUpdate[handle.Index].ReqTimes?.ToList() ?? new List(); - return async (updater, tagname, ct) => + return async (updater, tagname, who, ct) => { var codes = times.Count == 0 ? Array.Empty() - : await updater.DeleteAtTimeAsync(tagname, times, ct).ConfigureAwait(false); + : await updater.DeleteAtTimeAsync(tagname, times, who, ct).ConfigureAwait(false); if (codes.Count != times.Count) throw new InvalidOperationException( $"historian updater returned {codes.Count} result(s) for {times.Count} time(s)"); @@ -5445,11 +5450,12 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 /// The node's historian tagname. /// The update mode. /// The request's values, in order. + /// The caller, recorded in the modification trail. /// The request deadline. /// The node's result, carrying one operation result per item. private static async Task UpdateDataAsync( IHistorianDataUpdater updater, string tagname, HistoryUpdateMode mode, IList items, - CancellationToken ct) + HistoryUpdateContext who, CancellationToken ct) { var codes = new uint[items.Count]; var toSend = new List(items.Count); @@ -5478,7 +5484,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 if (toSend.Count > 0) { - var backend = await updater.UpdateDataAsync(tagname, mode, toSend, ct).ConfigureAwait(false); + var backend = await updater.UpdateDataAsync(tagname, mode, toSend, who, ct).ConfigureAwait(false); if (backend.Count != toSend.Count) throw new InvalidOperationException( $"historian updater returned {backend.Count} result(s) for {toSend.Count} value(s)"); @@ -5510,11 +5516,14 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 List nodesToProcess, IList results, IList errors, - Func>?> plan) + Func>?> plan) { nodesToProcess = AuthorizeHistoryUpdate(context, nodesToProcess, results, errors); if (nodesToProcess.Count == 0) return; + // Who is rewriting history — recorded on every modification-trail row (Part 11 ModificationInfo.UserName, #668). + var who = HistoryUpdateCallerOf(context); + WithHistoryReadBatchLimiter(nodesToProcess, results, errors, () => { using var deadline = CreateDeadline(); @@ -5533,13 +5542,30 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 var h = handle; var tag = tagname; - work.Add(() => ServeUpdateNodeAsync(h, results, errors, token => operation(updater!, tag, token), ct)); + work.Add(() => ServeUpdateNodeAsync(h, results, errors, token => operation(updater!, tag, who, token), ct)); } RunBounded(work, HistoryReadBatchParallelism); }); } + /// + /// The of the session behind : its API-key principal + /// (or the reserved anonymous principal) and key id. Only an identity that already passed + /// reaches a backend, so the anonymous fallback for a missing identity is + /// defence in depth, not a path. + /// + /// The SDK context carrying the session identity. + /// The caller recorded in the modification trail. + internal static HistoryUpdateContext HistoryUpdateCallerOf(ServerSystemContext context) + { + var identity = (context as ISessionOperationContext)?.UserIdentity as RoleCarryingUserIdentity + ?? context.OperationContext?.UserIdentity as RoleCarryingUserIdentity; + return identity is null + ? new HistoryUpdateContext(OpcUaPrincipals.Anonymous, null) + : new HistoryUpdateContext(identity.Principal, identity.ApiKeyId); + } + /// /// Runs one node's HistoryUpdate operation and fills its slots. Never throws (the /// contract): a deadline cancellation is BadTimeout, any backend failure @@ -5950,7 +5976,7 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 // Resume read: take the stored cursor. A miss (unknown / evicted / malformed point) ⇒ // BadContinuationPointInvalid for THIS node. var state = _historyContinuationStore.TryTake(session, inboundCp); - if (state is null || state.IsEvents + if (state is null || state.Kind != HistoryCursorKind.Raw || !TryGetHistorizedTagname(handle.NodeId, out var boundTag) || boundTag != state.Tagname) { errors[handle.Index] = StatusCodes.BadContinuationPointInvalid; @@ -6116,6 +6142,138 @@ public sealed class OtOpcUaNodeManager : CustomNodeManager2 } } + /// + /// Serve one historized variable for a HistoryReadRaw with IsReadModified = true (Part 11 modified-value + /// history, Gitea #668), with continuation-point paging. The backend orders the trail by + /// (SourceTimeUtc, ModifiedId) and reports truncation itself (), + /// so the cursor is an exact, exclusive key — there are no ties to trim. Window normalisation matches + /// : with a cap, an unset EndTime pages forward to an open bound and an unset + /// StartTime pages backward. A cursor is bound to the node's tagname and to the modified read kind; any other + /// point is BadContinuationPointInvalid. A backend that keeps no trail (Gateway, Null) throws + /// ⇒ BadHistoryOperationUnsupported for this node. + /// + /// The node handle; handle.Index indexes results/errors. + /// The session (null on the session-less in-process path ⇒ single-shot). + /// The per-node read list (inbound continuation points). + /// The service-level results list. + /// The service-level errors list. + /// Request StartTime. + /// Request EndTime. + /// Per-page cap; 0 = all. + /// Request deadline. + private async Task ServeModifiedPagedAsync( + NodeHandle handle, + ISession? session, + IList nodesToRead, + IList results, + IList errors, + DateTime startTimeUtc, + DateTime endUtc, + uint numValuesPerNode, + CancellationToken ct) + { + var inboundCp = nodesToRead[handle.Index].ContinuationPoint; + try + { + if (!TryGetHistorizedTagname(handle.NodeId, out var tagname) || tagname is null) + { + errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported; + results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadHistoryOperationUnsupported }; + return; + } + + var startUtc = startTimeUtc; + HistoryModifiedCursor? resumeAfter = null; + if (inboundCp is { Length: > 0 }) + { + var state = _historyContinuationStore.TryTake(session, inboundCp); + if (state is null || state.Kind != HistoryCursorKind.Modified || state.Tagname != tagname) + { + errors[handle.Index] = StatusCodes.BadContinuationPointInvalid; + results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadContinuationPointInvalid }; + return; + } + + // The cursor carries the NORMALISED window, so a resumed page never re-derives its direction. + startUtc = state.WindowStartUtc; + endUtc = state.EndUtc; + numValuesPerNode = state.NumValuesPerNode; + resumeAfter = new HistoryModifiedCursor(state.NextStartUtc, state.LastModifiedId); + } + else if (numValuesPerNode > 0) + { + if (endUtc == DateTime.MinValue && startUtc != DateTime.MinValue) + endUtc = new DateTime(DateTime.MaxValue.Ticks, DateTimeKind.Utc); + else if (startUtc == DateTime.MinValue && endUtc != DateTime.MinValue) + (startUtc, endUtc) = (endUtc, new DateTime(0, DateTimeKind.Utc)); + } + + var page = await HistorianDataSource + .ReadModifiedAsync(tagname, startUtc, endUtc, numValuesPerNode, resumeAfter, ct) + .ConfigureAwait(false); + + var data = ToHistoryModifiedData(page.Values); + byte[]? outboundCp = null; + if (page.HasMore && numValuesPerNode > 0 && page.Values.Count > 0) + { + var last = page.Values[^1]; + outboundCp = _historyContinuationStore.Save(session, new HistoryContinuationState( + tagname, last.Value.SourceTimestampUtc ?? DateTime.MinValue, endUtc, 0, numValuesPerNode, + HistoryCursorKind.Modified, last.ModifiedId, startUtc)); + } + + results[handle.Index] = new SdkHistoryReadResult + { + StatusCode = page.Values.Count == 0 ? StatusCodes.GoodNoData : StatusCodes.Good, + HistoryData = new ExtensionObject(data), + ContinuationPoint = outboundCp, + }; + errors[handle.Index] = ServiceResult.Good; + } + catch (OperationCanceledException) + { + errors[handle.Index] = StatusCodes.BadTimeout; + results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadTimeout }; + } + catch (NotSupportedException ex) + { + // The backend keeps no modification trail (Gateway / Null) — the spec status, not a server fault. + _logger.LogDebug(ex, "OtOpcUaNodeManager: backend serves no modified-value history for node {NodeId}", handle.NodeId); + errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported; + results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadHistoryOperationUnsupported }; + } + catch (Exception ex) + { + _logger.LogError(ex, "OtOpcUaNodeManager: HistoryRead (modified) failed for node {NodeId}", handle.NodeId); + errors[handle.Index] = StatusCodes.BadHistoryOperationUnsupported; + results[handle.Index] = new SdkHistoryReadResult { StatusCode = StatusCodes.BadHistoryOperationUnsupported }; + } + } + + /// + /// Project a modification trail into an SDK : one prior + /// per modification plus a parallel (time, Part 11 update type, user name). + /// + /// The modifications, in the order to emit. + /// The populated . + internal static HistoryModifiedData ToHistoryModifiedData(IReadOnlyList values) + { + var dataValues = new DataValueCollection(values.Count); + var infos = new ModificationInfoCollection(values.Count); + foreach (var m in values) + { + dataValues.Add(ToSdkDataValue(m.Value)); + infos.Add(new ModificationInfo + { + ModificationTime = m.ModificationTimeUtc, + UpdateType = (HistoryUpdateType)(int)m.UpdateType, + UserName = m.UserName, + }); + } + + return new HistoryModifiedData { DataValues = dataValues, ModificationInfos = infos }; + } + /// protected override void HistoryReleaseContinuationPoints( ServerSystemContext context, diff --git a/tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/Fakes/FakeOpcUaClientService.cs b/tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/Fakes/FakeOpcUaClientService.cs index 5c950179..51a1b6d6 100644 --- a/tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/Fakes/FakeOpcUaClientService.cs +++ b/tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/Fakes/FakeOpcUaClientService.cs @@ -326,6 +326,20 @@ public sealed class FakeOpcUaClientService : IOpcUaClientService return Task.FromResult(HistoryReadResult); } + /// Gets the list of (NodeId, start, end, maxValues) tuples from HistoryReadModifiedAsync calls. + public List<(NodeId NodeId, DateTime Start, DateTime End, int MaxValues)> HistoryReadModifiedCalls { get; } = []; + + /// Gets or sets the entries returned by HistoryReadModifiedAsync. + public IReadOnlyList HistoryReadModifiedResult { get; set; } = []; + + /// + public Task> HistoryReadModifiedAsync( + NodeId nodeId, DateTime startTime, DateTime endTime, int maxValues = 1000, CancellationToken ct = default) + { + HistoryReadModifiedCalls.Add((nodeId, startTime, endTime, maxValues)); + return Task.FromResult(HistoryReadModifiedResult); + } + /// Gets the details passed to HistoryUpdateAsync. public List HistoryUpdateCalls { get; } = []; diff --git a/tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/HistoryReadCommandTests.cs b/tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/HistoryReadCommandTests.cs index 245bfd07..a5ba8af2 100644 --- a/tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/HistoryReadCommandTests.cs +++ b/tests/Client/ZB.MOM.WW.OtOpcUa.Client.CLI.Tests/HistoryReadCommandTests.cs @@ -146,4 +146,55 @@ public class HistoryReadCommandTests fakeService.DisconnectCalled.ShouldBeTrue(); fakeService.DisposeCalled.ShouldBeTrue(); } -} \ No newline at end of file + + /// #668: --modified reads the modification trail (not raw) and prints update type, user and prior value. + [Fact] + public async Task Execute_ModifiedRead_CallsHistoryReadModified_and_prints_the_trail() + { + var t = new DateTime(2025, 6, 15, 10, 0, 0, DateTimeKind.Utc); + var fakeService = new FakeOpcUaClientService + { + HistoryReadModifiedResult = + [ + new HistoryModifiedEntry(new DataValue(Variant.Null, StatusCodes.BadNoData, t, t), t.AddDays(1), HistoryUpdateType.Insert, "editor"), + new HistoryModifiedEntry(new DataValue(new Variant(1.5), StatusCodes.Good, t, t), t.AddDays(2), HistoryUpdateType.Replace, "editor"), + ], + }; + var command = new HistoryReadCommand(new FakeOpcUaClientServiceFactory(fakeService)) + { + Url = "opc.tcp://localhost:4840", + NodeId = "ns=2;s=HistNode", + Modified = true, + MaxValues = 50, + }; + + using var console = TestConsoleHelper.CreateConsole(); + await command.ExecuteAsync(console); + + fakeService.HistoryReadModifiedCalls.ShouldHaveSingleItem().MaxValues.ShouldBe(50); + fakeService.HistoryReadRawCalls.ShouldBeEmpty(); + var output = TestConsoleHelper.GetOutput(console); + output.ShouldContain("Modified history for ns=2;s=HistNode"); + output.ShouldContain("Replace"); + output.ShouldContain("Insert"); + output.ShouldContain("editor"); + output.ShouldContain("(none)"); + output.ShouldContain("2 modifications returned."); + } + + /// --modified cannot be combined with --aggregate. + [Fact] + public async Task Execute_ModifiedWithAggregate_ThrowsCommandException() + { + var command = new HistoryReadCommand(new FakeOpcUaClientServiceFactory(new FakeOpcUaClientService())) + { + Url = "opc.tcp://localhost:4840", + NodeId = "ns=2;s=HistNode", + Modified = true, + Aggregate = "Average", + }; + + using var console = TestConsoleHelper.CreateConsole(); + await Should.ThrowAsync(async () => await command.ExecuteAsync(console)); + } +} diff --git a/tests/Drivers/Historian.Conformance/ConformanceData.cs b/tests/Drivers/Historian.Conformance/ConformanceData.cs index ac08471d..7906a519 100644 --- a/tests/Drivers/Historian.Conformance/ConformanceData.cs +++ b/tests/Drivers/Historian.Conformance/ConformanceData.cs @@ -8,6 +8,9 @@ public static class ConformanceData /// Base time: 2026-10-09 08:00:00.1234567 UTC (sub-millisecond ticks on purpose). public static readonly DateTime T0 = new DateTime(2026, 10, 9, 8, 0, 0, DateTimeKind.Utc).AddTicks(1_234_567); + /// The HistoryUpdate caller the suite passes to every updater call (recorded in the modification trail). + public static readonly ZB.MOM.WW.OtOpcUa.Core.Abstractions.HistoryUpdateContext Editor = new("conformance-editor", "k-conf"); + /// Good. public const uint Good = 0x00000000u; /// Bad (generic). diff --git a/tests/Drivers/Historian.Conformance/ModifiedValueConformance.cs b/tests/Drivers/Historian.Conformance/ModifiedValueConformance.cs new file mode 100644 index 00000000..f2e7cb7d --- /dev/null +++ b/tests/Drivers/Historian.Conformance/ModifiedValueConformance.cs @@ -0,0 +1,217 @@ +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; + +/// +/// OPC UA Part 11 modified-value history (Gitea #668): every HistoryUpdate a backend applies leaves an ordered +/// modification trail — the PRIOR value, the caller's principal, the Part 11 update type and the modification time — +/// readable through . Refused items, live recorder writes and +/// reads leave nothing. +/// +public abstract class ModifiedValueConformance : ConformanceBase +{ + private static readonly DateTime T0 = new(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc); + private static DateTime T(int n) => T0.AddSeconds(n).AddTicks(1230); // µs-aligned so both backends store it exactly + private static readonly DateTime Lo = T0.AddHours(-1); + private static readonly DateTime Hi = T0.AddHours(1); + private static readonly CancellationToken Ct = CancellationToken.None; + private static readonly HistoryUpdateContext Editor = ConformanceData.Editor; + + private static HistorizationValue V(int n, double d, uint status = Good) => + new(T(n), HistorianValue.FromDouble(d), status); + + private async Task> TrailAsync( + string tag, DateTime? start = null, DateTime? end = null, uint max = 0, HistoryModifiedCursor? after = null) + { + using var src = B.CreateDataSource(); + var r = await src.ReadModifiedAsync(tag, start ?? Lo, end ?? Hi, max, after, Ct); + if (max == 0) r.HasMore.ShouldBeFalse(); + return r.Values; + } + + private static void ShouldBeNoPrior(HistoryModifiedValue m) + { + m.Value.Value.ShouldBeNull(); + m.Value.StatusCode.ShouldBe(HistorianStatusCodes.BadNoData); + } + + [Fact] + public async Task Insert_records_one_Insert_per_sample_with_no_prior_value_and_the_principal() + { + var before = DateTime.UtcNow.AddSeconds(-5); + var u = B.CreateUpdater(); + (await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(2, 2), V(1, 1)], Editor, Ct)) + .ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted]); + + var trail = await TrailAsync("t"); + + trail.Count.ShouldBe(2); + trail.Select(m => m.Value.SourceTimestampUtc).ShouldBe([T(1), T(2)]); // ordered by source time + trail.ShouldAllBe(m => m.UpdateType == HistoryModificationType.Insert); + trail.ShouldAllBe(m => m.UserName == "conformance-editor"); + trail.ShouldAllBe(m => m.ModificationTimeUtc >= before && m.ModificationTimeUtc <= DateTime.UtcNow.AddSeconds(5)); + foreach (var m in trail) ShouldBeNoPrior(m); + } + + [Fact] + public async Task Replace_records_the_prior_value_and_status_after_the_insert() + { + var u = B.CreateUpdater(); + await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(1, 1)], Editor, Ct); + var other = new HistoryUpdateContext("second-editor", null); + (await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(1, 11, Uncertain)], other, Ct)) + .ShouldBe([HistorianStatusCodes.GoodEntryReplaced]); + (await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(1, 111)], Editor, Ct)) + .ShouldBe([HistorianStatusCodes.GoodEntryReplaced]); + + var trail = await TrailAsync("t"); + + trail.Select(m => m.UpdateType).ShouldBe( + [HistoryModificationType.Insert, HistoryModificationType.Replace, HistoryModificationType.Replace]); + trail.Select(m => m.UserName).ShouldBe(["conformance-editor", "second-editor", "conformance-editor"]); + ShouldBeNoPrior(trail[0]); + trail[1].Value.Value.ShouldBe(1d); // what the first Replace overwrote + trail[1].Value.StatusCode.ShouldBe(Good); + trail[2].Value.Value.ShouldBe(11d); // what the second Replace overwrote, with its status + trail[2].Value.StatusCode.ShouldBe(Uncertain); + trail.ShouldAllBe(m => m.Value.SourceTimestampUtc == T(1)); + trail.Select(m => m.ModifiedId).ShouldBeInOrder(SortDirection.Ascending); + trail.Select(m => m.ModifiedId).Distinct().Count().ShouldBe(3); + // The sample itself carries the latest value. + (await B.ReadBackAsync("t")).ShouldHaveSingleItem().V.Number.ShouldBe(111d); + } + + [Fact] + public async Task Update_records_Update_with_the_prior_value_when_it_replaced_and_none_when_it_inserted() + { + var u = B.CreateUpdater(); + await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(1, 1)], Editor, Ct); + (await u.UpdateDataAsync("t", HistoryUpdateMode.Update, [V(2, 2), V(1, 10)], Editor, Ct)) + .ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryReplaced]); + + var trail = await TrailAsync("t"); + + trail.Select(m => (m.Value.SourceTimestampUtc, m.UpdateType)).ShouldBe( + [ + (T(1), HistoryModificationType.Insert), + (T(1), HistoryModificationType.Update), + (T(2), HistoryModificationType.Update), + ]); + trail[1].Value.Value.ShouldBe(1d); + ShouldBeNoPrior(trail[2]); + } + + [Fact] + public async Task DeleteAtTime_records_a_Delete_carrying_each_deleted_value() + { + var u = B.CreateUpdater(); + await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(1, 1), V(2, 2), V(3, 3, Uncertain)], Editor, Ct); + var deleter = new HistoryUpdateContext("deleter", "k-del"); + (await u.DeleteAtTimeAsync("t", [T(3), T(1), T(9)], deleter, Ct)) + .ShouldBe([Good, Good, HistorianStatusCodes.BadNoEntryExists]); + + var deletes = (await TrailAsync("t")).Where(m => m.UpdateType == HistoryModificationType.Delete).ToList(); + + deletes.Select(m => m.Value.SourceTimestampUtc).ShouldBe([T(1), T(3)]); + deletes.Select(m => (double)m.Value.Value!).ShouldBe([1d, 3d]); + deletes[1].Value.StatusCode.ShouldBe(Uncertain); + deletes.ShouldAllBe(m => m.UserName == "deleter"); + } + + [Fact] + public async Task DeleteRaw_records_a_Delete_for_every_sample_it_removed() + { + var u = B.CreateUpdater(); + await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(1, 1), V(2, 2), V(3, 3)], Editor, Ct); + (await u.DeleteRawAsync("t", T(1), T(3), Editor, Ct)).ShouldBe(Good); // [T1, T3) + + var deletes = (await TrailAsync("t")).Where(m => m.UpdateType == HistoryModificationType.Delete).ToList(); + + deletes.Select(m => m.Value.SourceTimestampUtc).ShouldBe([T(1), T(2)]); + deletes.Select(m => (double)m.Value.Value!).ShouldBe([1d, 2d]); + } + + [Fact] + public async Task Refused_items_leave_no_trail() + { + var u = B.CreateUpdater(); + await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(1, 1)], Editor, Ct); + (await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(1, 99)], Editor, Ct)).ShouldBe([HistorianStatusCodes.BadEntryExists]); + (await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(5, 5)], Editor, Ct)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]); + (await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [new HistorizationValue(null, HistorianValue.FromDouble(1), Good)], Editor, Ct)) + .ShouldBe([HistorianStatusCodes.BadInvalidTimestamp]); + (await u.DeleteAtTimeAsync("t", [T(7)], Editor, Ct)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]); + (await u.DeleteRawAsync("t", T(10), T(20), Editor, Ct)).ShouldBe(HistorianStatusCodes.BadNoData); + + var trail = await TrailAsync("t"); + trail.ShouldHaveSingleItem().UpdateType.ShouldBe(HistoryModificationType.Insert); // only the first, applied insert + } + + [Fact] + public async Task Live_recorder_writes_leave_no_trail() + { + var w = B.CreateValueWriter(); + (await w.WriteLiveValuesAsync("live", [new(T(1), HistorianValue.FromDouble(1), Good), new(T(2), HistorianValue.FromDouble(2), Good)], Ct)) + .ShouldBeTrue(); + (await B.ReadBackAsync("live")).Count.ShouldBe(2); // the writes landed … + (await TrailAsync("live")).ShouldBeEmpty(); // … and recorded no modification + } + + [Fact] + public async Task An_unknown_tag_has_an_empty_trail_and_reads_create_no_tag() + { + (await TrailAsync("never")).ShouldBeEmpty(); + (await B.TagExistsAsync("never")).ShouldBeFalse(); + } + + [Fact] + public async Task The_trail_pages_by_key_exactly_once_and_reports_HasMore() + { + var u = B.CreateUpdater(); + await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(1, 1), V(2, 2), V(3, 3)], Editor, Ct); + await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(2, 20)], Editor, Ct); + await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(2, 200)], Editor, Ct); + var all = await TrailAsync("t"); + all.Count.ShouldBe(5); // T1 ins | T2 ins, rep, rep | T3 ins + + var seen = new List(); + HistoryModifiedCursor? cursor = null; + using var src = B.CreateDataSource(); + for (var page = 0; page < 10; page++) + { + var r = await src.ReadModifiedAsync("t", Lo, Hi, 2, cursor, Ct); + seen.AddRange(r.Values.Select(m => m.ModifiedId)); + if (!r.HasMore) break; + r.Values.Count.ShouldBe(2); + var last = r.Values[^1]; + cursor = new HistoryModifiedCursor(last.Value.SourceTimestampUtc!.Value, last.ModifiedId); + } + + seen.ShouldBe(all.Select(m => m.ModifiedId)); // a page boundary inside the T2 run neither repeats nor skips + } + + [Fact] + public async Task A_reverse_window_reads_the_trail_descending() + { + var u = B.CreateUpdater(); + await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(1, 1), V(2, 2)], Editor, Ct); + await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(2, 20)], Editor, Ct); + var forward = await TrailAsync("t"); + + var reverse = await TrailAsync("t", start: Hi, end: Lo); + + reverse.Select(m => m.ModifiedId).ShouldBe(forward.Select(m => m.ModifiedId).Reverse()); + } + + [Fact] + public async Task The_window_is_half_open_on_source_time() + { + var u = B.CreateUpdater(); + await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(1, 1), V(2, 2), V(3, 3)], Editor, Ct); + (await TrailAsync("t", start: T(1), end: T(3))).Select(m => m.Value.SourceTimestampUtc).ShouldBe([T(1), T(2)]); + (await TrailAsync("t", start: T(2), end: T(2))).Select(m => m.Value.SourceTimestampUtc).ShouldBe([T(2)]); + } +} diff --git a/tests/Drivers/Historian.Conformance/QualityContractConformance.cs b/tests/Drivers/Historian.Conformance/QualityContractConformance.cs index 39a42d1c..144ee9da 100644 --- a/tests/Drivers/Historian.Conformance/QualityContractConformance.cs +++ b/tests/Drivers/Historian.Conformance/QualityContractConformance.cs @@ -85,7 +85,7 @@ public abstract class QualityContractConformance : ConformanceBase new(At(1), HistorianValue.Empty, 0x40000000u), new(At(2), HistorianValue.Empty, BadNoCommunication), new(At(3), HistorianValue.FromDouble(1), 0u), - ], Ct); + ], ConformanceData.Editor, Ct); r.ShouldBe([BadInvalidArgument, BadInvalidArgument, HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted]); (await B.CreateDataSource().ReadRawAsync("u", At(0), At(10), 0, Ct)).Samples diff --git a/tests/Drivers/Historian.Conformance/TagNameConformance.cs b/tests/Drivers/Historian.Conformance/TagNameConformance.cs index 46054377..e6c2038b 100644 --- a/tests/Drivers/Historian.Conformance/TagNameConformance.cs +++ b/tests/Drivers/Historian.Conformance/TagNameConformance.cs @@ -39,10 +39,10 @@ public abstract class TagNameConformance : ConformanceBase var u = B.CreateUpdater(); (await u.UpdateDataAsync(Long, HistoryUpdateMode.Update, - [new(At(0), HistorianValue.FromDouble(9), 0u), new(At(1), HistorianValue.FromDouble(9), 0u)], Ct)) + [new(At(0), HistorianValue.FromDouble(9), 0u), new(At(1), HistorianValue.FromDouble(9), 0u)], ConformanceData.Editor, Ct)) .ShouldBe([BadInvalidArgument, BadInvalidArgument]); - (await u.DeleteRawAsync(Long, At(-1), At(10), Ct)).ShouldBe(BadInvalidArgument); - (await u.DeleteAtTimeAsync(Long, [At(0)], Ct)).ShouldBe([BadInvalidArgument]); + (await u.DeleteRawAsync(Long, At(-1), At(10), ConformanceData.Editor, Ct)).ShouldBe(BadInvalidArgument); + (await u.DeleteAtTimeAsync(Long, [At(0)], ConformanceData.Editor, Ct)).ShouldBe([BadInvalidArgument]); // The prefix series is untouched: not replaced, not deleted. var raw = (await B.CreateDataSource().ReadRawAsync(Prefix, At(-1), At(10), 0, Ct)).Samples; diff --git a/tests/Drivers/Historian.Conformance/UpdateConformance.cs b/tests/Drivers/Historian.Conformance/UpdateConformance.cs index 4d60029e..2f67507d 100644 --- a/tests/Drivers/Historian.Conformance/UpdateConformance.cs +++ b/tests/Drivers/Historian.Conformance/UpdateConformance.cs @@ -23,10 +23,10 @@ public abstract class UpdateConformance : ConformanceBase public async Task Insert_new_is_GoodEntryInserted_and_existing_is_BadEntryExists_in_order() { var u = B.CreateUpdater(); - var r1 = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2)], Ct); + var r1 = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2)], ConformanceData.Editor, Ct); r1.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted]); // Mixed: T(2) exists, T(3) new, T(1) exists - out of timestamp order on purpose. - var r2 = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(3), 3), V(T(2), 99), V(T(1), 98)], Ct); + var r2 = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(3), 3), V(T(2), 99), V(T(1), 98)], ConformanceData.Editor, Ct); r2.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadEntryExists, HistorianStatusCodes.BadEntryExists]); var rows = await B.ReadBackAsync("t"); rows.Select(x => (double?)x.V.Number).ShouldBe([1d, 2d, 3d]); // existing values untouched @@ -37,9 +37,9 @@ public abstract class UpdateConformance : ConformanceBase public async Task Replace_existing_is_GoodEntryReplaced_with_new_value_and_status_missing_is_BadNoEntryExists() { var u = B.CreateUpdater(); - await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2)], Ct); + await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2)], ConformanceData.Editor, Ct); var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, - [V(T(2), 22, UncertainStatus), V(T(5), 5), V(T(1), 11)], Ct); + [V(T(2), 22, UncertainStatus), V(T(5), 5), V(T(1), 11)], ConformanceData.Editor, Ct); r.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.BadNoEntryExists, HistorianStatusCodes.GoodEntryReplaced]); var rows = await B.ReadBackAsync("t"); rows.Count.ShouldBe(2); // T(5) was not created @@ -53,8 +53,8 @@ public abstract class UpdateConformance : ConformanceBase public async Task Update_upserts_inserted_or_replaced_per_item() { var u = B.CreateUpdater(); - await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct); - var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Update, [V(T(2), 2), V(T(1), 10)], Ct); + await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], ConformanceData.Editor, Ct); + var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Update, [V(T(2), 2), V(T(1), 10)], ConformanceData.Editor, Ct); r.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryReplaced]); (await B.ReadBackAsync("t")).Select(x => (double?)x.V.Number).ShouldBe([10d, 2d]); } @@ -63,15 +63,15 @@ public abstract class UpdateConformance : ConformanceBase public async Task Unknown_tag_Insert_and_Update_create_it_but_Replace_and_deletes_do_not() { var u = B.CreateUpdater(); - (await u.UpdateDataAsync("rep", HistoryUpdateMode.Replace, [V(T(1), 1)], Ct)) + (await u.UpdateDataAsync("rep", HistoryUpdateMode.Replace, [V(T(1), 1)], ConformanceData.Editor, Ct)) .ShouldBe([HistorianStatusCodes.BadNoEntryExists]); - (await u.DeleteRawAsync("del", T(0), T(9), Ct)).ShouldBe(HistorianStatusCodes.BadNoData); - (await u.DeleteAtTimeAsync("del2", [T(1)], Ct)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]); + (await u.DeleteRawAsync("del", T(0), T(9), ConformanceData.Editor, Ct)).ShouldBe(HistorianStatusCodes.BadNoData); + (await u.DeleteAtTimeAsync("del2", [T(1)], ConformanceData.Editor, Ct)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]); foreach (var name in new[] { "rep", "del", "del2" }) (await B.TagExistsAsync(name)).ShouldBeFalse(name); // none of those created a tag - (await u.UpdateDataAsync("ins", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct)).ShouldBe([HistorianStatusCodes.GoodEntryInserted]); - (await u.UpdateDataAsync("upd", HistoryUpdateMode.Update, [V(T(1), 1)], Ct)).ShouldBe([HistorianStatusCodes.GoodEntryInserted]); + (await u.UpdateDataAsync("ins", HistoryUpdateMode.Insert, [V(T(1), 1)], ConformanceData.Editor, Ct)).ShouldBe([HistorianStatusCodes.GoodEntryInserted]); + (await u.UpdateDataAsync("upd", HistoryUpdateMode.Update, [V(T(1), 1)], ConformanceData.Editor, Ct)).ShouldBe([HistorianStatusCodes.GoodEntryInserted]); (await B.TagExistsAsync("ins")).ShouldBeTrue(); (await B.TagExistsAsync("upd")).ShouldBeTrue(); } @@ -81,14 +81,14 @@ public abstract class UpdateConformance : ConformanceBase { var u = B.CreateUpdater(); var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, - [V(null, 1), V(T(1), 2), V(DateTime.MinValue, 3)], Ct); + [V(null, 1), V(T(1), 2), V(DateTime.MinValue, 3)], ConformanceData.Editor, Ct); r.ShouldBe([HistorianStatusCodes.BadInvalidTimestamp, HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadInvalidTimestamp]); (await B.ReadBackAsync("t")).Count.ShouldBe(1); // All-invalid request: no DB work, no tag created, still one code per item. var u2 = B.CreateUpdater(); - await u2.UpdateDataAsync("seed", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct); // bootstraps the schema - (await u2.UpdateDataAsync("x", HistoryUpdateMode.Insert, [V(null, 1)], Ct)).ShouldBe([HistorianStatusCodes.BadInvalidTimestamp]); + await u2.UpdateDataAsync("seed", HistoryUpdateMode.Insert, [V(T(1), 1)], ConformanceData.Editor, Ct); // bootstraps the schema + (await u2.UpdateDataAsync("x", HistoryUpdateMode.Insert, [V(null, 1)], ConformanceData.Editor, Ct)).ShouldBe([HistorianStatusCodes.BadInvalidTimestamp]); (await B.TagExistsAsync("seed")).ShouldBeTrue(); (await B.TagExistsAsync("x")).ShouldBeFalse(); // "x" was never created } @@ -98,7 +98,7 @@ public abstract class UpdateConformance : ConformanceBase { var u = B.CreateUpdater(); var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, - [V(T(1), double.NaN), V(T(2), 2), V(T(3), double.PositiveInfinity), V(T(4), double.NegativeInfinity)], Ct); + [V(T(1), double.NaN), V(T(2), 2), V(T(3), double.PositiveInfinity), V(T(4), double.NegativeInfinity)], ConformanceData.Editor, Ct); r.ShouldBe([HistorianStatusCodes.BadInvalidArgument, HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadInvalidArgument, HistorianStatusCodes.BadInvalidArgument]); (await B.ReadBackAsync("t")).Select(x => (double?)x.V.Number).ShouldBe([2d]); @@ -108,7 +108,7 @@ public abstract class UpdateConformance : ConformanceBase public async Task Duplicate_timestamp_in_one_Insert_request_second_is_BadEntryExists_first_wins() { var u = B.CreateUpdater(); - var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(1), 99)], Ct); + var r = await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(1), 99)], ConformanceData.Editor, Ct); r.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadEntryExists]); (await B.ReadBackAsync("t")).Select(x => (double?)x.V.Number).ShouldBe([1d, 2d]); } @@ -117,12 +117,12 @@ public abstract class UpdateConformance : ConformanceBase public async Task Duplicate_timestamp_in_one_Replace_or_Update_request_last_wins_and_all_share_the_code() { var u = B.CreateUpdater(); - await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct); - var rep = await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(T(1), 5), V(T(1), 6)], Ct); + await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], ConformanceData.Editor, Ct); + var rep = await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(T(1), 5), V(T(1), 6)], ConformanceData.Editor, Ct); rep.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.GoodEntryReplaced]); (await B.ReadBackAsync("t")).Single().V.Number.ShouldBe(6d); - var upd = await u.UpdateDataAsync("t", HistoryUpdateMode.Update, [V(T(7), 7), V(T(7), 8)], Ct); + var upd = await u.UpdateDataAsync("t", HistoryUpdateMode.Update, [V(T(7), 7), V(T(7), 8)], ConformanceData.Editor, Ct); upd.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted]); (await B.ReadBackAsync("t")).Last().V.Number.ShouldBe(8d); } @@ -131,9 +131,9 @@ public abstract class UpdateConformance : ConformanceBase public async Task Replace_and_Update_can_change_the_value_kind_and_clear_stale_columns() { var u = B.CreateUpdater(); - await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [new HistorizationValue(T(1), HistorianValue.FromDouble(1.5), Good)], Ct); + await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [new HistorizationValue(T(1), HistorianValue.FromDouble(1.5), Good)], ConformanceData.Editor, Ct); (await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, - [new HistorizationValue(T(1), HistorianValue.FromString("hi"), Good)], Ct)).ShouldBe([HistorianStatusCodes.GoodEntryReplaced]); + [new HistorizationValue(T(1), HistorianValue.FromString("hi"), Good)], ConformanceData.Editor, Ct)).ShouldBe([HistorianStatusCodes.GoodEntryReplaced]); var rows = await B.ReadBackAsync("t"); rows.Single().V.ShouldBe(HistorianValue.FromString("hi")); // Number cleared (0), not left at 1.5 } @@ -142,13 +142,13 @@ public abstract class UpdateConformance : ConformanceBase public async Task DeleteRaw_is_half_open_Good_when_rows_removed_BadNoData_when_none() { var u = B.CreateUpdater(); - await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(3), 3)], Ct); + await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(3), 3)], ConformanceData.Editor, Ct); // [T(1), T(3)) removes T(1) and T(2); T(3) (the end) survives. - (await u.DeleteRawAsync("t", T(1), T(3), Ct)).ShouldBe(Good); + (await u.DeleteRawAsync("t", T(1), T(3), ConformanceData.Editor, Ct)).ShouldBe(Good); (await B.ReadBackAsync("t")).Select(x => x.T).ShouldBe([B.Round(T(3))]); - (await u.DeleteRawAsync("t", T(1), T(3), Ct)).ShouldBe(HistorianStatusCodes.BadNoData); + (await u.DeleteRawAsync("t", T(1), T(3), ConformanceData.Editor, Ct)).ShouldBe(HistorianStatusCodes.BadNoData); // sub-ms boundary: a start one resolution unit after T(3) excludes it. - (await u.DeleteRawAsync("t", T(3).AddTicks(B.ResolutionTicks), T(3).AddTicks(100 * B.ResolutionTicks), Ct)).ShouldBe(HistorianStatusCodes.BadNoData); + (await u.DeleteRawAsync("t", T(3).AddTicks(B.ResolutionTicks), T(3).AddTicks(100 * B.ResolutionTicks), ConformanceData.Editor, Ct)).ShouldBe(HistorianStatusCodes.BadNoData); (await B.ReadBackAsync("t")).Count.ShouldBe(1); } @@ -156,8 +156,8 @@ public abstract class UpdateConformance : ConformanceBase public async Task DeleteAtTime_reports_per_time_in_order_with_exact_tick_matching() { var u = B.CreateUpdater(); - await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(3), 3)], Ct); - var r = await u.DeleteAtTimeAsync("t", [T(3), T(2).AddTicks(B.ResolutionTicks) /* near miss */, T(1), DateTime.MinValue], Ct); + await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1), V(T(2), 2), V(T(3), 3)], ConformanceData.Editor, Ct); + var r = await u.DeleteAtTimeAsync("t", [T(3), T(2).AddTicks(B.ResolutionTicks) /* near miss */, T(1), DateTime.MinValue], ConformanceData.Editor, Ct); r.ShouldBe([Good, HistorianStatusCodes.BadNoEntryExists, Good, HistorianStatusCodes.BadInvalidTimestamp]); (await B.ReadBackAsync("t")).Select(x => x.T).ShouldBe([B.Round(T(2))]); } @@ -166,8 +166,8 @@ public abstract class UpdateConformance : ConformanceBase public async Task DeleteAtTime_duplicate_time_first_reports_Good_later_BadNoEntryExists() { var u = B.CreateUpdater(); - await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], Ct); - (await u.DeleteAtTimeAsync("t", [T(1), T(1)], Ct)).ShouldBe([Good, HistorianStatusCodes.BadNoEntryExists]); + await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T(1), 1)], ConformanceData.Editor, Ct); + (await u.DeleteAtTimeAsync("t", [T(1), T(1)], ConformanceData.Editor, Ct)).ShouldBe([Good, HistorianStatusCodes.BadNoEntryExists]); (await B.ReadBackAsync("t")).ShouldBeEmpty(); } } diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/BootstrapTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/BootstrapTests.cs index cae1231e..27981561 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/BootstrapTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/BootstrapTests.cs @@ -52,7 +52,7 @@ public sealed class BootstrapTests(QuestDbFixture fx) var p = fx.UniquePrefix(); var (c, boot, _) = Make(p); await boot.EnsureAsync(default); - (await TablesAsync(c, p)).ShouldBe([p + "event", p + "sample", p + "tag"]); + (await TablesAsync(c, p)).ShouldBe([p + "event", p + "sample", p + "sample_modified", p + "tag"]); } [Fact] @@ -69,7 +69,7 @@ public sealed class BootstrapTests(QuestDbFixture fx) await Task.Delay(300); // an ALTER would have been applied by now (await SequencerTxnAsync(c2, p + "sample")).ShouldBe(before); (await SequencerTxnAsync(c2, p + "event")).ShouldBe(await SequencerTxnAsync(c, p + "event")); - (await TablesAsync(c2, p)).Count.ShouldBe(3); + (await TablesAsync(c2, p)).Count.ShouldBe(4); } [Fact] @@ -84,7 +84,7 @@ public sealed class BootstrapTests(QuestDbFixture fx) await Task.WhenAll( Task.Run(() => a.Boot.EnsureAsync(default)), Task.Run(() => b.Boot.EnsureAsync(default))); - (await TablesAsync(a.Conn, p)).Count.ShouldBe(3); + (await TablesAsync(a.Conn, p)).Count.ShouldBe(4); } } @@ -97,6 +97,8 @@ public sealed class BootstrapTests(QuestDbFixture fx) await b1.EnsureAsync(default); QuestDbTables.TtlHours((await TtlAsync(c1, withTtl + "sample")).Value, (await TtlAsync(c1, withTtl + "sample")).Unit).ShouldBe(168); QuestDbTables.TtlHours((await TtlAsync(c1, withTtl + "event")).Value, (await TtlAsync(c1, withTtl + "event")).Unit).ShouldBe(168); + // #668: the modification trail ages out with the samples it describes. + QuestDbTables.TtlHours((await TtlAsync(c1, withTtl + "sample_modified")).Value, (await TtlAsync(c1, withTtl + "sample_modified")).Unit).ShouldBe(168); (await TtlAsync(c1, withTtl + "tag")).Value.ShouldBe(0); // the tag catalog never expires var noTtl = fx.UniquePrefix(); diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs index 63477891..d0ca0bdd 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/QuestDbConformance.cs @@ -53,3 +53,10 @@ public sealed class QuestDbTagNameConformance(QuestDbFixture fx) : TagNameConfor /// protected override Task CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx); } + +[Collection(QuestDbCollection.Name)] +public sealed class QuestDbModifiedValueConformance(QuestDbFixture fx) : ModifiedValueConformance +{ + /// + protected override Task CreateBackendAsync() => QuestDbConformanceBackend.CreateAsync(fx); +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.Redelivery.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.Redelivery.cs index c3b85295..8cab5cea 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.Redelivery.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.Redelivery.cs @@ -26,7 +26,7 @@ public sealed partial class TombstoneTests ]; (await writer.WriteLiveValuesAsync("r", batch, default)).ShouldBeTrue(); - (await rig.Updater.DeleteAtTimeAsync("r", [T0], default)).ShouldBe([HistorianStatusCodes.Good]); + (await rig.Updater.DeleteAtTimeAsync("r", [T0], UpdateTests.Who, default)).ShouldBe([HistorianStatusCodes.Good]); (await writer.WriteLiveValuesAsync("r", batch, default)).ShouldBeTrue(); // the redelivery (await UpdateTests.RowsAsync(rig, "r", liveOnly: true)).Select(r => r.Num).ShouldBe([2d]); diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.cs index 841b73b7..2116a1fd 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/TombstoneTests.cs @@ -17,8 +17,8 @@ public sealed partial class TombstoneTests(QuestDbFixture fx) fx.SkipIfUnavailable(); var rig = UpdateTests.MakeRig(fx); await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, - [UpdateTests.V(T0, 1), UpdateTests.V(T0.AddSeconds(1), 2)], default); - (await rig.Updater.DeleteAtTimeAsync("t", [T0], default)).ShouldBe([HistorianStatusCodes.Good]); + [UpdateTests.V(T0, 1), UpdateTests.V(T0.AddSeconds(1), 2)], UpdateTests.Who, default); + (await rig.Updater.DeleteAtTimeAsync("t", [T0], UpdateTests.Who, default)).ShouldBe([HistorianStatusCodes.Good]); (await UpdateTests.RowsAsync(rig, "t", liveOnly: true)).Select(r => r.Num).ShouldBe([2d]); var all = await UpdateTests.RowsAsync(rig, "t"); @@ -35,12 +35,12 @@ public sealed partial class TombstoneTests(QuestDbFixture fx) { fx.SkipIfUnavailable(); var rig = UpdateTests.MakeRig(fx); - await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [UpdateTests.V(T0, 1)], default); - await rig.Updater.DeleteAtTimeAsync("t", [T0], default); + await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [UpdateTests.V(T0, 1)], UpdateTests.Who, default); + await rig.Updater.DeleteAtTimeAsync("t", [T0], UpdateTests.Who, default); - (await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Replace, [UpdateTests.V(T0, 5)], default)) + (await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Replace, [UpdateTests.V(T0, 5)], UpdateTests.Who, default)) .ShouldBe([HistorianStatusCodes.BadNoEntryExists]); // a tombstone is no entry for Replace - (await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [UpdateTests.V(T0, 9)], default)) + (await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [UpdateTests.V(T0, 9)], UpdateTests.Who, default)) .ShouldBe([HistorianStatusCodes.GoodEntryInserted]); var live = await UpdateTests.RowsAsync(rig, "t", liveOnly: true); live.Count.ShouldBe(1); diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs index 4d4e52a5..cddee903 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.IntegrationTests/UpdateTests.cs @@ -24,6 +24,9 @@ public sealed class UpdateTests(QuestDbFixture fx) return new Rig(c, new QuestDbHistorianDataUpdater(c, tags, wal), tags); } + /// The HistoryUpdate caller these tests pass to the updater (recorded in the modification trail). + internal static readonly HistoryUpdateContext Who = new("questdb-test", null); + internal static HistorizationValue V(DateTime? t, double v, uint status = Good) => new(t, HistorianValue.FromDouble(v), status); @@ -57,10 +60,10 @@ public sealed class UpdateTests(QuestDbFixture fx) { fx.SkipIfUnavailable(); var rig = MakeRig(fx); - (await rig.Updater.UpdateDataAsync("t1", HistoryUpdateMode.Insert, [V(T0, 1)], default)) + (await rig.Updater.UpdateDataAsync("t1", HistoryUpdateMode.Insert, [V(T0, 1)], UpdateTests.Who, default)) .ShouldBe([HistorianStatusCodes.GoodEntryInserted]); var codes = await rig.Updater.UpdateDataAsync("t1", HistoryUpdateMode.Insert, - [V(T0, 99), V(T0.AddSeconds(1), 2)], default); + [V(T0, 99), V(T0.AddSeconds(1), 2)], UpdateTests.Who, default); codes.ShouldBe([HistorianStatusCodes.BadEntryExists, HistorianStatusCodes.GoodEntryInserted]); var rows = await RowsAsync(rig, "t1"); // read right after return: read-your-writes rows.Select(r => r.Num).ShouldBe([1d, 2d]); @@ -72,14 +75,14 @@ public sealed class UpdateTests(QuestDbFixture fx) { fx.SkipIfUnavailable(); var rig = MakeRig(fx); - (await rig.Updater.UpdateDataAsync("nope", HistoryUpdateMode.Replace, [V(T0, 1)], default)) + (await rig.Updater.UpdateDataAsync("nope", HistoryUpdateMode.Replace, [V(T0, 1)], UpdateTests.Who, default)) .ShouldBe([HistorianStatusCodes.BadNoEntryExists]); (await TagRowsAsync(rig, "nope")).ShouldBe(0); (await RowsAsync(rig, "nope")).ShouldBeEmpty(); - await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1)], default); + await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1)], UpdateTests.Who, default); var codes = await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Replace, - [V(T0, 5), V(T0.AddSeconds(1), 6)], default); + [V(T0, 5), V(T0.AddSeconds(1), 6)], UpdateTests.Who, default); codes.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.BadNoEntryExists]); (await RowsAsync(rig, "t")).Select(r => r.Num).ShouldBe([5d]); } @@ -89,9 +92,9 @@ public sealed class UpdateTests(QuestDbFixture fx) { fx.SkipIfUnavailable(); var rig = MakeRig(fx); - await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1)], default); + await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1)], UpdateTests.Who, default); var codes = await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Update, - [V(T0, 2), V(T0.AddSeconds(1), 3), V(T0.AddSeconds(1).AddTicks(3), 4)], default); + [V(T0, 2), V(T0.AddSeconds(1), 3), V(T0.AddSeconds(1).AddTicks(3), 4)], UpdateTests.Who, default); codes.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.GoodEntryInserted]); (await RowsAsync(rig, "t")).Select(r => r.Num).ShouldBe([2d, 4d]); @@ -104,7 +107,7 @@ public sealed class UpdateTests(QuestDbFixture fx) var rig = MakeRig(fx); // offsets 0 and 3 ticks sit inside one 10-tick window => the same stored µs var codes = await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, - [V(T0, 1), V(T0.AddTicks(3), 2)], default); + [V(T0, 1), V(T0.AddTicks(3), 2)], UpdateTests.Who, default); codes.ShouldBe([HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadEntryExists]); (await RowsAsync(rig, "t")).Select(r => r.Num).ShouldBe([1d]); } @@ -119,7 +122,7 @@ public sealed class UpdateTests(QuestDbFixture fx) V(null, 1), V(DateTime.MinValue, 1), V(T0, double.NaN), new HistorizationValue(T0.AddSeconds(1), HistorianValue.Empty, Good), V(T0.AddSeconds(2), 7), - ], default); + ], UpdateTests.Who, default); codes.ShouldBe([HistorianStatusCodes.BadInvalidTimestamp, HistorianStatusCodes.BadInvalidTimestamp, HistorianStatusCodes.BadInvalidArgument, HistorianStatusCodes.BadInvalidArgument, HistorianStatusCodes.GoodEntryInserted]); @@ -132,10 +135,10 @@ public sealed class UpdateTests(QuestDbFixture fx) fx.SkipIfUnavailable(); var rig = MakeRig(fx); var tag = new string('x', HistorianTagNames.MaxLength + 1); - (await rig.Updater.UpdateDataAsync(tag, HistoryUpdateMode.Insert, [V(T0, 1)], default)) + (await rig.Updater.UpdateDataAsync(tag, HistoryUpdateMode.Insert, [V(T0, 1)], UpdateTests.Who, default)) .ShouldBe([HistorianStatusCodes.BadInvalidArgument]); - (await rig.Updater.DeleteAtTimeAsync(tag, [T0], default)).ShouldBe([HistorianStatusCodes.BadInvalidArgument]); - (await rig.Updater.DeleteRawAsync(tag, T0, T0.AddHours(1), default)).ShouldBe(HistorianStatusCodes.BadInvalidArgument); + (await rig.Updater.DeleteAtTimeAsync(tag, [T0], UpdateTests.Who, default)).ShouldBe([HistorianStatusCodes.BadInvalidArgument]); + (await rig.Updater.DeleteRawAsync(tag, T0, T0.AddHours(1), UpdateTests.Who, default)).ShouldBe(HistorianStatusCodes.BadInvalidArgument); } [Fact] @@ -146,15 +149,15 @@ public sealed class UpdateTests(QuestDbFixture fx) var pre = new DateTime(1969, 12, 31, 23, 59, 59, DateTimeKind.Utc); foreach (var mode in new[] { HistoryUpdateMode.Insert, HistoryUpdateMode.Replace, HistoryUpdateMode.Update }) { - var codes = await rig.Updater.UpdateDataAsync("t", mode, [V(pre, 1), V(DateTime.MinValue, 1), V(T0, 2)], default); + var codes = await rig.Updater.UpdateDataAsync("t", mode, [V(pre, 1), V(DateTime.MinValue, 1), V(T0, 2)], UpdateTests.Who, default); codes[0].ShouldBe(HistorianStatusCodes.BadOutOfRange); codes[1].ShouldBe(HistorianStatusCodes.BadInvalidTimestamp); } // Pre-1970 only: the whole batch is refused, no tag row is created, no INSERT would succeed anyway. - (await rig.Updater.UpdateDataAsync("only-pre", HistoryUpdateMode.Update, [V(pre, 1)], default)) + (await rig.Updater.UpdateDataAsync("only-pre", HistoryUpdateMode.Update, [V(pre, 1)], UpdateTests.Who, default)) .ShouldBe([HistorianStatusCodes.BadOutOfRange]); (await TagRowsAsync(rig, "only-pre")).ShouldBe(0); - (await rig.Updater.DeleteAtTimeAsync("t", [pre], default)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]); + (await rig.Updater.DeleteAtTimeAsync("t", [pre], UpdateTests.Who, default)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]); } [Fact] @@ -162,15 +165,15 @@ public sealed class UpdateTests(QuestDbFixture fx) { fx.SkipIfUnavailable(); var rig = MakeRig(fx); - await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1), V(T0.AddSeconds(1), 2)], default); + await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1), V(T0.AddSeconds(1), 2)], UpdateTests.Who, default); var codes = await rig.Updater.DeleteAtTimeAsync("t", - [T0, T0.AddSeconds(5), T0, DateTime.MinValue], default); + [T0, T0.AddSeconds(5), T0, DateTime.MinValue], UpdateTests.Who, default); codes.ShouldBe([Good, HistorianStatusCodes.BadNoEntryExists, HistorianStatusCodes.BadNoEntryExists, HistorianStatusCodes.BadInvalidTimestamp]); (await RowsAsync(rig, "t", liveOnly: true)).Select(r => r.Num).ShouldBe([2d]); // a tombstone is not a live row: deleting it again finds nothing - (await rig.Updater.DeleteAtTimeAsync("t", [T0], default)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]); - (await rig.Updater.DeleteAtTimeAsync("unknown", [T0], default)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]); + (await rig.Updater.DeleteAtTimeAsync("t", [T0], UpdateTests.Who, default)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]); + (await rig.Updater.DeleteAtTimeAsync("unknown", [T0], UpdateTests.Who, default)).ShouldBe([HistorianStatusCodes.BadNoEntryExists]); } [Fact] @@ -178,9 +181,9 @@ public sealed class UpdateTests(QuestDbFixture fx) { fx.SkipIfUnavailable(); var rig = MakeRig(fx); - await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1)], default); + await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1)], UpdateTests.Who, default); // +7 ticks is inside the same µs; the row lives at T0. - (await rig.Updater.DeleteAtTimeAsync("t", [T0.AddTicks(7)], default)).ShouldBe([Good]); + (await rig.Updater.DeleteAtTimeAsync("t", [T0.AddTicks(7)], UpdateTests.Who, default)).ShouldBe([Good]); (await RowsAsync(rig, "t", liveOnly: true)).ShouldBeEmpty(); } @@ -189,18 +192,18 @@ public sealed class UpdateTests(QuestDbFixture fx) { fx.SkipIfUnavailable(); var rig = MakeRig(fx); - (await rig.Updater.DeleteRawAsync("unknown", T0, T0.AddHours(1), default)).ShouldBe(HistorianStatusCodes.BadNoData); + (await rig.Updater.DeleteRawAsync("unknown", T0, T0.AddHours(1), UpdateTests.Who, default)).ShouldBe(HistorianStatusCodes.BadNoData); // 2,500 rows => 3 pages of at most 1,000; plus one row exactly at the (exclusive) end. var values = Enumerable.Range(0, 2500).Select(i => V(T0.AddSeconds(i), i)).ToList(); values.Add(V(T0.AddSeconds(2500), -1)); - (await rig.Updater.UpdateDataAsync("big", HistoryUpdateMode.Insert, values, default)) + (await rig.Updater.UpdateDataAsync("big", HistoryUpdateMode.Insert, values, UpdateTests.Who, default)) .ShouldAllBe(c => c == HistorianStatusCodes.GoodEntryInserted); - (await rig.Updater.DeleteRawAsync("big", T0, T0.AddSeconds(2500), default)).ShouldBe(Good); + (await rig.Updater.DeleteRawAsync("big", T0, T0.AddSeconds(2500), UpdateTests.Who, default)).ShouldBe(Good); var live = await RowsAsync(rig, "big", liveOnly: true); live.Count.ShouldBe(1); live[0].Num.ShouldBe(-1d); - (await rig.Updater.DeleteRawAsync("big", T0, T0.AddSeconds(2500), default)).ShouldBe(HistorianStatusCodes.BadNoData); + (await rig.Updater.DeleteRawAsync("big", T0, T0.AddSeconds(2500), UpdateTests.Who, default)).ShouldBe(HistorianStatusCodes.BadNoData); } [Fact] @@ -208,12 +211,12 @@ public sealed class UpdateTests(QuestDbFixture fx) { fx.SkipIfUnavailable(); var rig = MakeRig(fx); - await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1), V(T0.AddSeconds(1), 2)], default); - (await rig.Updater.DeleteRawAsync("t", new DateTime(1900, 1, 1, 0, 0, 0, DateTimeKind.Utc), T0.AddSeconds(1), default)) + await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T0, 1), V(T0.AddSeconds(1), 2)], UpdateTests.Who, default); + (await rig.Updater.DeleteRawAsync("t", new DateTime(1900, 1, 1, 0, 0, 0, DateTimeKind.Utc), T0.AddSeconds(1), UpdateTests.Who, default)) .ShouldBe(Good); (await RowsAsync(rig, "t", liveOnly: true)).Select(r => r.Num).ShouldBe([2d]); // DateTime.MinValue as the start is nudged inside, not bound as -infinity - (await rig.Updater.DeleteRawAsync("t", DateTime.MinValue, DateTime.MaxValue, default)).ShouldBe(Good); + (await rig.Updater.DeleteRawAsync("t", DateTime.MinValue, DateTime.MaxValue, UpdateTests.Who, default)).ShouldBe(Good); (await RowsAsync(rig, "t", liveOnly: true)).ShouldBeEmpty(); } @@ -226,9 +229,9 @@ public sealed class UpdateTests(QuestDbFixture fx) for (var i = 0; i < 40; i++) { var t = T0.AddSeconds(i); - (await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(t, i)], default)) + (await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(t, i)], UpdateTests.Who, default)) .ShouldBe([HistorianStatusCodes.GoodEntryInserted]); - (await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(t, i + 1000)], default)) + (await rig.Updater.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(t, i + 1000)], UpdateTests.Who, default)) .ShouldBe([HistorianStatusCodes.GoodEntryReplaced], $"iteration {i}: the insert was not visible"); } } @@ -242,8 +245,8 @@ public sealed class UpdateTests(QuestDbFixture fx) { var t = T0.AddSeconds(i); var codes = await Task.WhenAll( - Task.Run(() => rig.Updater.UpdateDataAsync("race", HistoryUpdateMode.Insert, [V(t, 1)], default)), - Task.Run(() => rig.Updater.UpdateDataAsync("race", HistoryUpdateMode.Insert, [V(t, 2)], default))); + Task.Run(() => rig.Updater.UpdateDataAsync("race", HistoryUpdateMode.Insert, [V(t, 1)], UpdateTests.Who, default)), + Task.Run(() => rig.Updater.UpdateDataAsync("race", HistoryUpdateMode.Insert, [V(t, 2)], UpdateTests.Who, default))); codes.Select(c => c[0]).OrderBy(c => c).ToArray().ShouldBe( new[] { HistorianStatusCodes.GoodEntryInserted, HistorianStatusCodes.BadEntryExists }, $"iteration {i}"); } @@ -256,10 +259,10 @@ public sealed class UpdateTests(QuestDbFixture fx) var rig = MakeRig(fx); // 1 Hz seed across 2 h = 7,200 live rows; the request touches 3 instants 1 h apart. var seed = Enumerable.Range(0, 7200).Select(i => V(T0.AddSeconds(i), i)).ToList(); - await rig.Updater.UpdateDataAsync("sparse", HistoryUpdateMode.Insert, seed, default); + await rig.Updater.UpdateDataAsync("sparse", HistoryUpdateMode.Insert, seed, UpdateTests.Who, default); var before = rig.Updater.LiveRowsRead; var codes = await rig.Updater.UpdateDataAsync("sparse", HistoryUpdateMode.Replace, - [V(T0, 1), V(T0.AddHours(1), 2), V(T0.AddHours(2).AddSeconds(-1), 3), V(T0.AddHours(1).AddMilliseconds(500), 4)], default); + [V(T0, 1), V(T0.AddHours(1), 2), V(T0.AddHours(2).AddSeconds(-1), 3), V(T0.AddHours(1).AddMilliseconds(500), 4)], UpdateTests.Who, default); codes.ShouldBe([HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.GoodEntryReplaced, HistorianStatusCodes.BadNoEntryExists]); // 3 clusters of a single instant each (T0+1h and +1h.5s are one cluster of 2 instants, 1 live row) diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbTablesTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbTablesTests.cs index ef42fb50..748469d8 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbTablesTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.QuestDb.Tests/QuestDbTablesTests.cs @@ -39,10 +39,22 @@ public sealed class QuestDbTablesTests [Fact] public void The_prefix_is_applied_to_every_table_name() { - QuestDbTables.All("t1_").ShouldBe(["t1_sample", "t1_event", "t1_tag"]); + QuestDbTables.All("t1_").ShouldBe(["t1_sample", "t1_event", "t1_tag", "t1_sample_modified"]); + QuestDbTables.Expiring("t1_").ShouldBe(["t1_sample", "t1_event", "t1_sample_modified"]); QuestDbTables.CreateAll("t1_", 3).ShouldAllBe(s => s.Contains("t1_")); } + [Fact] + public void Sample_modified_ddl_is_pinned_append_only_and_carries_the_ttl() + { + QuestDbTables.CreateSampleModified("hist_", null).ShouldBe( + "CREATE TABLE IF NOT EXISTS hist_sample_modified (ts TIMESTAMP, tag SYMBOL, modified_id LONG, " + + "modification_ts TIMESTAMP, update_type BYTE, principal VARCHAR, api_key_id VARCHAR, kind BYTE, num_value DOUBLE, " + + "int_value LONG, str_value VARCHAR, status LONG, server_ts TIMESTAMP) TIMESTAMP(ts) PARTITION BY DAY WAL"); + QuestDbTables.CreateSampleModified("hist_", 30).ShouldEndWith("PARTITION BY DAY TTL 30 DAYS WAL"); + QuestDbTables.CreateSampleModified("hist_", null).ShouldNotContain("DEDUP"); + } + [Fact] public void An_invalid_prefix_is_refused() => Should.Throw(() => QuestDbTables.CreateSample("a b", null)); diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ModifiedHistoryAtomicityTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ModifiedHistoryAtomicityTests.cs new file mode 100644 index 00000000..0a3e98ea --- /dev/null +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/ModifiedHistoryAtomicityTests.cs @@ -0,0 +1,112 @@ +using Microsoft.Data.SqlClient; +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Conformance; +using ZB.MOM.WW.OtOpcUa.Driver.Historian.Sql.Common; + +namespace ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests; + +/// +/// #668, SQL Server only: the modification-trail row is written by the SAME statement that changes +/// hist.Sample (OUTPUT … INTO), so a change and its trail row commit or roll back together. The failure +/// is forced INSIDE the audit insert (a unique index on the trail that the second modification of one timestamp +/// violates), i.e. after the sample row was already changed within the statement — the strongest place to prove it. +/// Each test has a positive control: the same operation succeeds, with its trail row, once the index is dropped. +/// +[Collection(SqlHistorianCollection.Name)] +public sealed class ModifiedHistoryAtomicityTests(SqlHistorianFixture fx) +{ + private static readonly DateTime T1 = new(2026, 1, 1, 0, 0, 1, DateTimeKind.Utc); + private static readonly DateTime T2 = T1.AddSeconds(1); + private static readonly HistoryUpdateContext Who = ConformanceData.Editor; + + private static HistorizationValue V(DateTime t, double d) => new(t, HistorianValue.FromDouble(d), 0u); + + private async Task ExecAsync(ReadTestDb db, string sql) + { + await using var c = await db.Conn.OpenAsync(CancellationToken.None); + await using var cmd = db.Conn.CreateCommand(c, sql); + await cmd.ExecuteNonQueryAsync(); + } + + private async Task CountTrailAsync(ReadTestDb db) + { + await using var c = await db.Conn.OpenAsync(CancellationToken.None); + await using var cmd = db.Conn.CreateCommand(c, $"SELECT COUNT(*) FROM {db.Conn.Q("SampleModified")}"); + return Convert.ToInt64(await cmd.ExecuteScalarAsync()); + } + + private Task BreakTheTrailAsync(ReadTestDb db) => ExecAsync(db, + $"CREATE UNIQUE INDEX UX_Break ON {db.Conn.Q("SampleModified")} (TagId, SourceTimeUtc)"); + + private Task RepairTheTrailAsync(ReadTestDb db) => ExecAsync(db, + $"DROP INDEX UX_Break ON {db.Conn.Q("SampleModified")}"); + + private static async Task ValueAtAsync(ReadTestDb db, string tag, DateTime t) + { + using var src = db.Source(); + var r = await src.ReadRawAsync(tag, t, t, 0, CancellationToken.None); + return (double)r.Samples.ShouldHaveSingleItem().Value!; + } + + [Fact] + public async Task A_replace_whose_trail_insert_fails_changes_nothing() + { + var db = await ReadTestDb.CreateAsync(fx); + var u = new SqlServerHistorianDataUpdater(db.Conn, new SqlServerTagCatalog(db.Conn, db.Boot)); + (await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T1, 1)], Who, CancellationToken.None)) + .ShouldBe([HistorianStatusCodes.GoodEntryInserted]); + (await CountTrailAsync(db)).ShouldBe(1); + + await BreakTheTrailAsync(db); + await Should.ThrowAsync( + () => u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(T1, 99)], Who, CancellationToken.None)); + + (await ValueAtAsync(db, "t", T1)).ShouldBe(1d, "the sample change rolled back with its trail row"); + (await CountTrailAsync(db)).ShouldBe(1, "a failed update leaves no trail row"); + + // Positive control: the identical replace succeeds, with its trail row, once the trail accepts it. + await RepairTheTrailAsync(db); + (await u.UpdateDataAsync("t", HistoryUpdateMode.Replace, [V(T1, 99)], Who, CancellationToken.None)) + .ShouldBe([HistorianStatusCodes.GoodEntryReplaced]); + (await ValueAtAsync(db, "t", T1)).ShouldBe(99d); + (await CountTrailAsync(db)).ShouldBe(2); + } + + [Fact] + public async Task Deletes_whose_trail_insert_fails_delete_nothing() + { + var db = await ReadTestDb.CreateAsync(fx); + var u = new SqlServerHistorianDataUpdater(db.Conn, new SqlServerTagCatalog(db.Conn, db.Boot)); + await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T1, 1), V(T2, 2)], Who, CancellationToken.None); + + await BreakTheTrailAsync(db); + await Should.ThrowAsync(() => u.DeleteAtTimeAsync("t", [T1], Who, CancellationToken.None)); + await Should.ThrowAsync(() => u.DeleteRawAsync("t", T1, T2.AddSeconds(1), Who, CancellationToken.None)); + + (await ValueAtAsync(db, "t", T1)).ShouldBe(1d); + (await ValueAtAsync(db, "t", T2)).ShouldBe(2d); + (await CountTrailAsync(db)).ShouldBe(2, "only the two inserts"); + + await RepairTheTrailAsync(db); + (await u.DeleteAtTimeAsync("t", [T1], Who, CancellationToken.None)).ShouldBe([HistorianStatusCodes.Good]); + (await u.DeleteRawAsync("t", T1, T2.AddSeconds(1), Who, CancellationToken.None)).ShouldBe(HistorianStatusCodes.Good); + (await CountTrailAsync(db)).ShouldBe(4); + } + + [Fact] + public async Task An_over_long_principal_is_truncated_never_refused() + { + var db = await ReadTestDb.CreateAsync(fx); + var u = new SqlServerHistorianDataUpdater(db.Conn, new SqlServerTagCatalog(db.Conn, db.Boot)); + var who = new HistoryUpdateContext(new string('p', 500), new string('k', 500)); + + (await u.UpdateDataAsync("t", HistoryUpdateMode.Insert, [V(T1, 1)], who, CancellationToken.None)) + .ShouldBe([HistorianStatusCodes.GoodEntryInserted]); + + using var src = db.Source(); + var trail = await src.ReadModifiedAsync("t", T1.AddHours(-1), T1.AddHours(1), 0, null, CancellationToken.None); + trail.Values.ShouldHaveSingleItem().UserName.ShouldBe(new string('p', HistoryUpdateContext.MaxIdentityLength)); + } +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RawReadTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RawReadTests.cs index 6a7a213e..b6421b11 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RawReadTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RawReadTests.cs @@ -30,7 +30,7 @@ public sealed class RawReadTests(SqlHistorianFixture fx) var h = src.GetHealthSnapshot(); h.TotalFailures.ShouldBe(2); h.ConsecutiveFailures.ShouldBe(2); - h.LastError!.ShouldContain("missing version 1"); + h.LastError!.ShouldContain($"missing version {SqlServerSchemaBootstrapper.RequiredVersion}"); h.ProcessConnectionOpen.ShouldBeFalse(); // Another node creates the schema; the same data source then succeeds (bootstrap failure is not latched). diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RetentionTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RetentionTests.cs index 05c29da5..046869d6 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RetentionTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/RetentionTests.cs @@ -56,6 +56,9 @@ public sealed class RetentionTests(SqlHistorianFixture fx) sql.Append($"SET @i = 0; WHILE @i < 2 BEGIN INSERT [{s}].Sample (TagId, SourceTimeUtc, ValueKind, NumValue, StatusCode, ServerTimeUtc) SELECT TagId, DATEADD(minute, @i, '{fresh}'), 1, 1.0, 0, SYSUTCDATETIME() FROM [{s}].Tag; SET @i += 1; END;"); sql.Append($"SET @i = 0; WHILE @i < 4 BEGIN INSERT [{s}].Event (EventId, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, EventTimeUtc, ReceivedTimeUtc) VALUES (CONCAT('old', @i), 's', 'a', 'n', 't', 1, 'Activated', DATEADD(minute, @i, '{old}'), SYSUTCDATETIME()); SET @i += 1; END;"); sql.Append($"INSERT [{s}].Event (EventId, SourceName, AlarmId, AlarmName, AlarmTypeName, Severity, EventKind, EventTimeUtc, ReceivedTimeUtc) VALUES ('new0', 's', 'a', 'n', 't', 1, 'Activated', '{fresh}', SYSUTCDATETIME());"); + // #668 modification trail: 2 old + 1 fresh row per tag (by SOURCE time — the modification time is "now" on all). + sql.Append($"SET @i = 0; WHILE @i < 2 BEGIN INSERT [{s}].SampleModified (TagId, SourceTimeUtc, ModificationTimeUtc, UpdateType, Principal, ValueKind, NumValue, StatusCode, ServerTimeUtc) SELECT TagId, DATEADD(minute, @i, '{old}'), SYSUTCDATETIME(), 2, N'p', 1, 1.0, 0, SYSUTCDATETIME() FROM [{s}].Tag; SET @i += 1; END;"); + sql.Append($"INSERT [{s}].SampleModified (TagId, SourceTimeUtc, ModificationTimeUtc, UpdateType, Principal) SELECT TagId, '{fresh}', SYSUTCDATETIME(), 1, N'p' FROM [{s}].Tag;"); await ExecAsync(sql.ToString()); } @@ -67,11 +70,22 @@ public sealed class RetentionTests(SqlHistorianFixture fx) fx.SkipIfUnavailable(); var (c, b, _, s) = Make(); await SeedAsync(b, s); + var identBefore = await CountAsync($"SELECT IDENT_CURRENT('[{s}].SampleModified')"); var purger = new SqlServerHistorianRetention(c, b, TimeSpan.FromHours(1), 30, batch, time: new FixedTime(Now)); var results = await purger.PurgeAsync(CancellationToken.None); - results.ShouldBe([new RetentionPurgeResult("Sample", 15), new RetentionPurgeResult("Event", 4)]); + // #668: a purge never WRITES to the modification trail. The identity seed proves no row was ever inserted — + // a row-count check could not, since the trail purge would delete an old-source-time audit row again. + (await CountAsync($"SELECT IDENT_CURRENT('[{s}].SampleModified')")).ShouldBe(identBefore); + + results.ShouldBe([ + new RetentionPurgeResult("Sample", 15), + new RetentionPurgeResult("SampleModified", 6), + new RetentionPurgeResult("Event", 4), + ]); + (await CountAsync($"SELECT COUNT(*) FROM [{s}].SampleModified")).ShouldBe(3); + (await CountAsync($"SELECT COUNT(*) FROM [{s}].SampleModified WHERE SourceTimeUtc < '{Now.AddDays(-30):yyyy-MM-ddTHH:mm:ss}'")).ShouldBe(0); (await CountAsync($"SELECT COUNT(*) FROM [{s}].Sample")).ShouldBe(6); (await CountAsync($"SELECT COUNT(*) FROM [{s}].Sample WHERE SourceTimeUtc < '{Now.AddDays(-30):yyyy-MM-ddTHH:mm:ss}'")).ShouldBe(0); (await CountAsync($"SELECT COUNT(*) FROM [{s}].Event")).ShouldBe(1); @@ -87,7 +101,11 @@ public sealed class RetentionTests(SqlHistorianFixture fx) var purger = new SqlServerHistorianRetention(c, b, TimeSpan.FromHours(1), 30, 4000, time: new FixedTime(Now)); await purger.PurgeAsync(CancellationToken.None); var again = await purger.PurgeAsync(CancellationToken.None); - again.ShouldBe([new RetentionPurgeResult("Sample", 0), new RetentionPurgeResult("Event", 0)]); + again.ShouldBe([ + new RetentionPurgeResult("Sample", 0), + new RetentionPurgeResult("SampleModified", 0), + new RetentionPurgeResult("Event", 0), + ]); } [Fact] diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SchemaBootstrapIntegrationTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SchemaBootstrapIntegrationTests.cs index 6fd3ca86..f6a1779c 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SchemaBootstrapIntegrationTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SchemaBootstrapIntegrationTests.cs @@ -32,7 +32,7 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx) var s = SqlHistorianFixture.UniqueSchema(); await Boot(Opts(s)).EnsureAsync(CancellationToken.None); await Boot(Opts(s)).EnsureAsync(CancellationToken.None); // fresh instance => re-runs the guarded path - (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(1); + (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.RequiredVersion); (await ScalarAsync($"SELECT COUNT(*) FROM sys.indexes WHERE name='CX_Event_Time' AND object_id=OBJECT_ID('[{s}].Event')")).ShouldBe(1); } @@ -51,7 +51,47 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx) cmd.CommandText = b; await cmd.ExecuteNonQueryAsync(); } - (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(1); + (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.RequiredVersion); + } + + /// + /// #668 upgrade path: a schema left at version 1 (only 001_initial.sql applied, as every pre-#668 install is) + /// is upgraded in place by an auto-creating bootstrap — SampleModified appears, both version rows exist, and + /// existing samples are untouched. With AutoCreateSchema=false the same v1 schema is refused. + /// + [Fact] + public async Task A_version_1_schema_is_upgraded_to_version_2_in_place() + { + fx.SkipIfUnavailable(); + var s = SqlHistorianFixture.UniqueSchema(); + var v1 = new StreamReader(typeof(SqlServerSchemaBootstrapper).Assembly.GetManifestResourceStream("Schema.001_initial.sql")!).ReadToEnd(); + await using (var c = new SqlConnection(fx.ConnectionString)) + { + await c.OpenAsync(); + foreach (var b in SqlServerSchemaBootstrapper.SplitBatches(v1.Replace("$(Schema)", s))) + { + await using var cmd = c.CreateCommand(); + cmd.CommandText = b; + await cmd.ExecuteNonQueryAsync(); + } + await using var seed = c.CreateCommand(); + seed.CommandText = $"INSERT [{s}].Tag (Name, DataType, CreatedUtc, UpdatedUtc) VALUES (N'old', 1, SYSUTCDATETIME(), SYSUTCDATETIME()); " + + $"INSERT [{s}].Sample (TagId, SourceTimeUtc, ValueKind, NumValue, StatusCode, ServerTimeUtc) SELECT TagId, '2026-01-01', 1, 4.0, 0, SYSUTCDATETIME() FROM [{s}].Tag"; + await seed.ExecuteNonQueryAsync(); + } + (await ScalarAsync($"SELECT COUNT(*) FROM sys.tables WHERE name = 'SampleModified' AND schema_id = SCHEMA_ID('{s}')")).ShouldBe(0); + + var refused = await Should.ThrowAsync( + () => Boot(Opts(s, auto: false)).EnsureAsync(CancellationToken.None)); + refused.Message.ShouldContain("missing version 2 (found 1)"); + + await Boot(Opts(s)).EnsureAsync(CancellationToken.None); + + (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(2); + (await ScalarAsync($"SELECT MAX(Version) FROM [{s}].SchemaVersion")).ShouldBe(2); + (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SampleModified")).ShouldBe(0); + (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].Sample")).ShouldBe(1); + await Boot(Opts(s, auto: false)).EnsureAsync(CancellationToken.None); // now satisfies the verify-only path } [Fact] @@ -61,7 +101,7 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx) var s = SqlHistorianFixture.UniqueSchema(); var tasks = Enumerable.Range(0, 4).Select(_ => Task.Run(() => Boot(Opts(s)).EnsureAsync(CancellationToken.None))).ToArray(); await Task.WhenAll(tasks); - (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(1); + (await ScalarAsync($"SELECT COUNT(*) FROM [{s}].SchemaVersion")).ShouldBe(SqlServerSchemaBootstrapper.RequiredVersion); } [Fact] @@ -70,7 +110,7 @@ public sealed class SchemaBootstrapIntegrationTests(SqlHistorianFixture fx) fx.SkipIfUnavailable(); var ex = await Should.ThrowAsync( () => Boot(Opts(SqlHistorianFixture.UniqueSchema(), auto: false)).EnsureAsync(CancellationToken.None)); - ex.Message.ShouldContain("missing version 1"); + ex.Message.ShouldContain("missing version 2"); } [Fact] diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformance.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformance.cs index 5a02a4c5..790beffe 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformance.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.IntegrationTests/SqlServerConformance.cs @@ -53,3 +53,10 @@ public sealed class SqlServerTagNameConformance(SqlHistorianFixture fx) : TagNam /// protected override Task CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx); } + +[Collection(SqlHistorianCollection.Name)] +public sealed class SqlServerModifiedValueConformance(SqlHistorianFixture fx) : ModifiedValueConformance +{ + /// + protected override Task CreateBackendAsync() => SqlServerConformanceBackend.CreateAsync(fx); +} diff --git a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/SchemaScriptTests.cs b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/SchemaScriptTests.cs index 649dd550..a260fc32 100644 --- a/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/SchemaScriptTests.cs +++ b/tests/Drivers/ZB.MOM.WW.OtOpcUa.Driver.Historian.SqlServer.Tests/SchemaScriptTests.cs @@ -29,4 +29,30 @@ public sealed class SchemaScriptTests dir.ShouldNotBeNull(); File.ReadAllText(Path.Combine(dir!.FullName, "docs/historian/sqlserver/001_initial.sql")).ShouldBe(text); } + + /// + /// #668: schema v2 (the Part 11 modification trail) is embedded, matches its docs copy, records version 2 (which + /// must equal ), and carries none of the constructs an + /// OUTPUT … INTO target forbids (FOREIGN KEY / CHECK / trigger) — any of them would fail every HistoryUpdate. + /// + [Fact] + public void Schema_v2_sample_modified_is_embedded_matches_the_docs_copy_and_is_a_legal_output_into_target() + { + var asm = typeof(SqlServerSchemaBootstrapper).Assembly; + using var s = asm.GetManifestResourceStream("Schema.002_sample_modified.sql"); + s.ShouldNotBeNull(); + var text = new StreamReader(s!).ReadToEnd(); + text.ShouldContain("[SampleModified]"); + text.ShouldContain("PRIMARY KEY CLUSTERED (TagId, SourceTimeUtc, ModifiedId)"); + text.ShouldContain("VALUES (2, SYSUTCDATETIME())"); + SqlServerSchemaBootstrapper.RequiredVersion.ShouldBe(2); + var ddl = string.Join('\n', text.Split('\n').Where(l => !l.TrimStart().StartsWith("--", StringComparison.Ordinal))); + ddl.ShouldNotContain("FOREIGN KEY", Case.Insensitive); + ddl.ShouldNotContain("CHECK", Case.Insensitive); + ddl.ShouldNotContain("TRIGGER", Case.Insensitive); + + var dir = new DirectoryInfo(AppContext.BaseDirectory); + while (dir is not null && !File.Exists(Path.Combine(dir.FullName, "ZB.MOM.WW.OtOpcUa.slnx"))) dir = dir.Parent; + File.ReadAllText(Path.Combine(dir!.FullName, "docs/historian/sqlserver/002_sample_modified.sql")).ShouldBe(text); + } } diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/HistoryUpdateOverTheWireTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/HistoryUpdateOverTheWireTests.cs index 65573c1d..76e6883f 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/HistoryUpdateOverTheWireTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.IntegrationTests/HistoryUpdateOverTheWireTests.cs @@ -72,6 +72,50 @@ public sealed class HistoryUpdateOverTheWireTests finally { await rig.DisposeAsync(); } } + /// + /// Part 11 modified-value history over the wire (#668): two inserts and a delete by the key principal, then a + /// HistoryReadRaw with IsReadModified = true decodes as a whose + /// ModificationInfos carry the update type and the principal, and whose values carry the PRIOR value. + /// + [Fact] + public async Task A_modified_read_returns_the_trail_with_principal_and_update_type_over_the_wire() + { + var ct = TestContext.Current.CancellationToken; + var store = new SharedStore(); + var rig = await StartAsync("HuWireModified", encrypted: false, ct); + try + { + rig.Nm.HistorianDataUpdater = store; + rig.Nm.HistorianDataSource = store; + rig.Enforce(true, [Key()], ClusterGrant(KeyPrincipal, "acl-editor", Editor)); + var id = rig.EnsureRawVariable("Speed", "Double", writable: false, Tagname); + + using var session = await rig.ConnectAsync(anonymous: false, ct); + var node = RawNode(session, id); + await UpdateAsync(session, node, PerformUpdateType.Insert, ct, (T1, 1.5), (T2, 2.5)); + await DeleteAtTimeAsync(session, node, ct, T1); + + var details = new ReadRawModifiedDetails + { + StartTime = T1.AddHours(-1), EndTime = T1.AddHours(1), NumValuesPerNode = 100, IsReadModified = true, + }; + var response = await session.HistoryReadAsync( + null, new ExtensionObject(details), TimestampsToReturn.Both, false, + new HistoryReadValueIdCollection { new HistoryReadValueId { NodeId = node } }, ct); + var result = response.Results.ShouldHaveSingleItem(); + result.StatusCode.Code.ShouldBe(StatusCodes.Good); + var data = ExtensionObject.ToEncodeable(result.HistoryData).ShouldBeOfType(); + + data.DataValues.Select(v => v.SourceTimestamp).ShouldBe([T1, T1, T2]); + data.ModificationInfos.Select(m => m.UpdateType) + .ShouldBe([HistoryUpdateType.Insert, HistoryUpdateType.Delete, HistoryUpdateType.Insert]); + data.ModificationInfos.ShouldAllBe(m => m.UserName == KeyPrincipal); + data.DataValues[0].StatusCode.Code.ShouldBe(StatusCodes.BadNoData); // an insert had no prior value + ((double)data.DataValues[1].Value).ShouldBe(1.5); // the deleted sample's prior value + } + finally { await rig.DisposeAsync(); } + } + /// /// ALLOW beside DENY: the same server, same node — the editor session is admitted, the session /// without the HistoryUpdate grant gets BadUserAccessDenied and the updater is never called. @@ -263,14 +307,21 @@ public sealed class HistoryUpdateOverTheWireTests { private readonly object _gate = new(); private readonly SortedDictionary _samples = []; + private readonly List _trail = []; private int _updateCalls; + private void Record(DateTime t, double? prior, HistoryModificationType type, HistoryUpdateContext who) => + _trail.Add(new HistoryModifiedValue( + new DataValueSnapshot(prior, prior is null ? StatusCodes.BadNoData : StatusCodes.Good, t, t), + DateTime.UtcNow, type, who.Principal, _trail.Count + 1)); + public int UpdateCalls => Volatile.Read(ref _updateCalls); public int Count { get { lock (_gate) return _samples.Count; } } public async Task> UpdateDataAsync( - string tag, HistoryUpdateMode mode, IReadOnlyList values, CancellationToken ct) + string tag, HistoryUpdateMode mode, IReadOnlyList values, HistoryUpdateContext context, + CancellationToken ct) { Interlocked.Increment(ref _updateCalls); await Task.Delay(1, ct).ConfigureAwait(false); @@ -285,21 +336,43 @@ public sealed class HistoryUpdateOverTheWireTests if (_samples.ContainsKey(t)) { codes.Add(StatusCodes.BadEntryExists); continue; } v.Value.TryGetDouble(out var d).ShouldBeTrue(); _samples[t] = d; + Record(t, null, HistoryModificationType.Insert, context); codes.Add(StatusCodes.GoodEntryInserted); } } return codes; } - public Task DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, CancellationToken ct) => + public Task DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, HistoryUpdateContext context, + CancellationToken ct) => throw new NotSupportedException(); public async Task> DeleteAtTimeAsync( - string tag, IReadOnlyList timesUtc, CancellationToken ct) + string tag, IReadOnlyList timesUtc, HistoryUpdateContext context, CancellationToken ct) { await Task.Delay(1, ct).ConfigureAwait(false); lock (_gate) - return timesUtc.Select(t => _samples.Remove(t) ? StatusCodes.Good : StatusCodes.BadNoEntryExists).ToList(); + return timesUtc.Select(t => + { + if (!_samples.Remove(t, out var prior)) return StatusCodes.BadNoEntryExists; + Record(t, prior, HistoryModificationType.Delete, context); + return StatusCodes.Good; + }).ToList(); + } + + public async Task ReadModifiedAsync( + string fullReference, DateTime startUtc, DateTime endUtc, uint maxValues, HistoryModifiedCursor? resumeAfter, + CancellationToken cancellationToken) + { + await Task.Delay(1, cancellationToken).ConfigureAwait(false); + fullReference.ShouldBe(Tagname); + lock (_gate) + { + var rows = _trail + .Where(m => m.Value.SourceTimestampUtc >= startUtc && m.Value.SourceTimestampUtc < endUtc) + .OrderBy(m => m.Value.SourceTimestampUtc).ThenBy(m => m.ModifiedId).ToList(); + return new HistoryModifiedReadResult(rows, false); + } } public async Task ReadRawAsync( diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/History/HistoryUpdateTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/History/HistoryUpdateTests.cs index 4a6617cd..4448d22c 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/History/HistoryUpdateTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/History/HistoryUpdateTests.cs @@ -463,6 +463,56 @@ public sealed class HistoryUpdateTests : IDisposable finally { await host.DisposeAsync(); } } + /// + /// #668: the session's API-key principal and key id reach EVERY updater call (they become the Part 11 + /// ModificationInfo.UserName of the trail rows the backend records). + /// + [Fact] + public async Task The_key_principal_and_key_id_are_threaded_to_every_updater_call() + { + var (host, server) = await BootAsync(); + try + { + var nm = server.NodeManager!; + var updater = new FakeUpdater(); + nm.HistorianDataUpdater = updater; + WireAcl(nm, enabled: false, NodePermissions.HistoryUpdate); + var nodeId = Historized(nm, AddressSpaceRealm.Raw, RawPath); + var identity = KeyIdentity(OpcUaDataPlaneRoles.HistoryUpdate); + + Update(nm, identity, nodeId, PerformUpdateType.Update, Value(T1, 1d)); + Call(nm, identity, typeof(DeleteRawModifiedDetails), + new DeleteRawModifiedDetails { NodeId = nodeId, StartTime = T1, EndTime = T2, IsDeleteModified = false }); + Call(nm, identity, typeof(DeleteAtTimeDetails), + new DeleteAtTimeDetails { NodeId = nodeId, ReqTimes = new DateTimeCollection { T1 } }); + + updater.Callers.Count.ShouldBe(3); + updater.Callers.ShouldAllBe(c => c == new HistoryUpdateContext(Principal, KeyId)); + } + finally { await host.DisposeAsync(); } + } + + /// #668: an admitted anonymous session is recorded as the reserved anonymous principal, no key id. + [Fact] + public async Task An_anonymous_session_is_recorded_as_the_anonymous_principal() + { + var (host, server) = await BootAsync(); + try + { + var nm = server.NodeManager!; + var updater = new FakeUpdater(); + nm.HistorianDataUpdater = updater; + WireAcl(nm, enabled: false, NodePermissions.HistoryUpdate); + var nodeId = Historized(nm, AddressSpaceRealm.Raw, RawPath); + + var (_, errors) = Insert(nm, AnonymousIdentity(OpcUaDataPlaneRoles.HistoryUpdate), nodeId, Value(T1, 1d)); + + errors[0].StatusCode.Code.ShouldBe(StatusCodes.Good); + updater.Callers.ShouldHaveSingleItem().ShouldBe(new HistoryUpdateContext(OpcUaPrincipals.Anonymous, null)); + } + finally { await host.DisposeAsync(); } + } + /// DeleteAtTime returns the updater's per-time codes in order. [Fact] public async Task Delete_at_time_passes_per_time_codes_through() @@ -593,6 +643,7 @@ public sealed class HistoryUpdateTests : IDisposable private sealed class FakeUpdater : IHistorianDataUpdater { public List<(string Tag, HistoryUpdateMode Mode, IReadOnlyList Values)> UpdateCalls { get; } = []; + public List Callers { get; } = []; public List<(string Tag, DateTime Start, DateTime End)> DeleteRawCalls { get; } = []; public List> DeleteAtTimeCalls { get; } = []; public uint[]? UpdateCodes { get; init; } @@ -601,26 +652,28 @@ public sealed class HistoryUpdateTests : IDisposable public Exception? Throw { get; init; } public async Task> UpdateDataAsync( - string tag, HistoryUpdateMode mode, IReadOnlyList values, CancellationToken ct) + string tag, HistoryUpdateMode mode, IReadOnlyList values, HistoryUpdateContext context, + CancellationToken ct) { await Task.Delay(1, ct).ConfigureAwait(false); if (Throw is not null) throw Throw; - lock (UpdateCalls) UpdateCalls.Add((tag, mode, values.ToList())); + lock (UpdateCalls) { UpdateCalls.Add((tag, mode, values.ToList())); Callers.Add(context); } return UpdateCodes ?? Enumerable.Repeat(StatusCodes.GoodEntryInserted, values.Count).ToArray(); } - public async Task DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, CancellationToken ct) + public async Task DeleteRawAsync(string tag, DateTime startUtc, DateTime endUtc, HistoryUpdateContext context, + CancellationToken ct) { await Task.Delay(1, ct).ConfigureAwait(false); - lock (DeleteRawCalls) DeleteRawCalls.Add((tag, startUtc, endUtc)); + lock (DeleteRawCalls) { DeleteRawCalls.Add((tag, startUtc, endUtc)); lock (UpdateCalls) Callers.Add(context); } return DeleteRawCode; } public async Task> DeleteAtTimeAsync( - string tag, IReadOnlyList timesUtc, CancellationToken ct) + string tag, IReadOnlyList timesUtc, HistoryUpdateContext context, CancellationToken ct) { await Task.Delay(1, ct).ConfigureAwait(false); - lock (DeleteAtTimeCalls) DeleteAtTimeCalls.Add(timesUtc.ToList()); + lock (DeleteAtTimeCalls) { DeleteAtTimeCalls.Add(timesUtc.ToList()); lock (UpdateCalls) Callers.Add(context); } return DeleteAtTimeCodes ?? Enumerable.Repeat(StatusCodes.Good, timesUtc.Count).ToArray(); } } diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerAclEnforcementTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerAclEnforcementTests.cs index 8198716a..65ed1720 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerAclEnforcementTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerAclEnforcementTests.cs @@ -89,6 +89,33 @@ public sealed class NodeManagerAclEnforcementTests : IDisposable finally { await host.DisposeAsync(); } } + /// + /// #668: a modified-value read (IsReadModified = true) is gated by the SAME HistoryRead decision — no new + /// operation. Denied without the bit; the granted principal reaches the backend (Null ⇒ unsupported, which is + /// "allowed but no trail", a different fact from the denial). + /// + [Fact] + public async Task Modified_history_read_is_gated_by_the_history_read_grant() + { + var (host, server) = await BootAsync(); + try + { + var nm = server.NodeManager!; + nm.HistorianDataSource = NullHistorianDataSource.Instance; + var nodeId = Historized(nm, "Plant/Modbus/dev1/Speed"); + + nm.Authorizer = Authorizer(enabled: true, GrantOn(NodePermissions.Read)); + var (deniedResults, deniedErrors) = ReadRaw(nm, Identity(), modified: true, nodeId); + deniedErrors[0].StatusCode.Code.ShouldBe(StatusCodes.BadUserAccessDenied); + deniedResults[0].StatusCode.Code.ShouldBe(StatusCodes.BadUserAccessDenied); + + nm.Authorizer = Authorizer(enabled: true, GrantOn(NodePermissions.HistoryRead)); + var (_, allowedErrors) = ReadRaw(nm, Identity(), modified: true, nodeId); + allowedErrors[0].StatusCode.Code.ShouldBe(StatusCodes.BadHistoryOperationUnsupported); + } + finally { await host.DisposeAsync(); } + } + /// A granted principal reads normally — the gate is not simply denying everything. [Fact] public async Task History_read_succeeds_for_a_granted_principal() @@ -308,7 +335,11 @@ public sealed class NodeManagerAclEnforcementTests : IDisposable } private static (List Results, List Errors) ReadRaw( - OtOpcUaNodeManager nm, IUserIdentity? identity, params NodeId[] nodeIds) + OtOpcUaNodeManager nm, IUserIdentity? identity, params NodeId[] nodeIds) => + ReadRaw(nm, identity, modified: false, nodeIds); + + private static (List Results, List Errors) ReadRaw( + OtOpcUaNodeManager nm, IUserIdentity? identity, bool modified, params NodeId[] nodeIds) { var context = new OperationContext( new RequestHeader(), secureChannelContext: null, RequestType.HistoryRead, identity); @@ -324,7 +355,7 @@ public sealed class NodeManagerAclEnforcementTests : IDisposable StartTime = DateTime.UtcNow.AddHours(-1), EndTime = DateTime.UtcNow, NumValuesPerNode = 10, - IsReadModified = false, + IsReadModified = modified, }, TimestampsToReturn.Both, releaseContinuationPoints: false, diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadModifiedTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadModifiedTests.cs new file mode 100644 index 00000000..dd2ee07c --- /dev/null +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadModifiedTests.cs @@ -0,0 +1,308 @@ +using Microsoft.Extensions.Logging.Abstractions; +using Opc.Ua; +using Opc.Ua.Server; +using Shouldly; +using Xunit; +using ZB.MOM.WW.OtOpcUa.Commons.OpcUa; +using ZB.MOM.WW.OtOpcUa.Core.Abstractions; +using HistorianRead = ZB.MOM.WW.OtOpcUa.Core.Abstractions.HistoryReadResult; +using SdkHistoryReadResult = Opc.Ua.HistoryReadResult; + +namespace ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests; + +/// +/// Gitea #668 — OPC UA Part 11 modified-value history: HistoryReadRaw with IsReadModified = true dispatches +/// to , projects a with one +/// per value, and pages with continuation points keyed on +/// (SourceTimeUtc, ModifiedId). Same harness as : a real +/// SDK server, a session-less in-process HistoryRead and an , over a +/// fake that behaves like the SQL backends (ordered by key, half-open window, cap + exact truncation flag). +/// +public sealed class NodeManagerHistoryReadModifiedTests : IDisposable +{ + private const string Tag = "WW.Mod"; + private static CancellationToken Ct => TestContext.Current.CancellationToken; + private static readonly DateTime Epoch = new(2026, 1, 1, 0, 0, 0, DateTimeKind.Utc); + private readonly string _pkiRoot = Path.Combine(Path.GetTempPath(), $"otopcua-modified-{Guid.NewGuid():N}"); + + [Fact] + public async Task A_modified_read_dispatches_to_the_trail_and_projects_modification_infos() + { + var modifiedAt = Epoch.AddDays(2); + var trail = new List + { + Mod(1, seconds: 10, prior: null, HistoryModificationType.Insert, "alice", modifiedAt), + Mod(2, seconds: 10, prior: 1.5, HistoryModificationType.Replace, "bob", modifiedAt.AddMinutes(1)), + Mod(3, seconds: 20, prior: 7.0, HistoryModificationType.Delete, "carol", modifiedAt.AddMinutes(2)), + }; + var (host, nm, node, src) = await SetupAsync(trail); + + var r = Read(nm, node, Epoch, Epoch.AddHours(1), max: 100, cp: null); + + r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good); + r.Result.StatusCode.Code.ShouldBe(StatusCodes.Good); + r.Cp.ShouldBeNull(); + src.LastTag.ShouldBe(Tag); + src.LastStart.ShouldBe(Epoch); + src.LastEnd.ShouldBe(Epoch.AddHours(1)); + src.LastMax.ShouldBe(100u); + src.LastCursor.ShouldBeNull(); + + var data = r.Data!; + data.DataValues.Count.ShouldBe(3); + data.ModificationInfos.Count.ShouldBe(3); + data.ModificationInfos.Select(m => m.UpdateType) + .ShouldBe([HistoryUpdateType.Insert, HistoryUpdateType.Replace, HistoryUpdateType.Delete]); + data.ModificationInfos.Select(m => m.UserName).ShouldBe(["alice", "bob", "carol"]); + data.ModificationInfos.Select(m => m.ModificationTime) + .ShouldBe([modifiedAt, modifiedAt.AddMinutes(1), modifiedAt.AddMinutes(2)]); + data.DataValues[0].Value.ShouldBeNull(); + data.DataValues[0].StatusCode.Code.ShouldBe(StatusCodes.BadNoData); + data.DataValues[1].Value.ShouldBe(1.5); + data.DataValues[1].SourceTimestamp.ShouldBe(Epoch.AddSeconds(10)); + data.DataValues[2].Value.ShouldBe(7.0); + await host.DisposeAsync(); + } + + [Fact] + public async Task An_empty_trail_is_GoodNoData() + { + var (host, nm, node, _) = await SetupAsync([]); + var r = Read(nm, node, Epoch, Epoch.AddHours(1), max: 10, cp: null); + r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good); + r.Result.StatusCode.Code.ShouldBe(StatusCodes.GoodNoData); + r.Data!.ModificationInfos.ShouldBeEmpty(); + await host.DisposeAsync(); + } + + [Fact] + public async Task Modified_reads_page_exactly_once_in_order_even_through_same_time_runs() + { + // 30 modifications; ids 5..9 all share one source time, so a page boundary cuts through the run. + var trail = Enumerable.Range(1, 30) + .Select(i => Mod(i, seconds: i is >= 5 and <= 9 ? 5 : i, prior: i, HistoryModificationType.Replace, "u", Epoch)) + .ToList(); + var (host, nm, node, src) = await SetupAsync(trail); + + var ids = Drain(nm, node, Epoch, Epoch.AddHours(1), max: 4); + + ids.ShouldBe(src.Ordered(reverse: false).Select(m => m.ModifiedId)); + ids.Count.ShouldBe(30); + src.Calls.ShouldBe(8); // 7 full pages + a 2-value tail; the exact HasMore flag means no trailing empty page + await host.DisposeAsync(); + } + + [Fact] + public async Task A_reverse_modified_read_pages_descending() + { + var trail = Enumerable.Range(1, 11).Select(i => Mod(i, seconds: i / 2, prior: i, HistoryModificationType.Delete, "u", Epoch)).ToList(); + var (host, nm, node, src) = await SetupAsync(trail); + + var ids = Drain(nm, node, Epoch.AddHours(1), Epoch.AddSeconds(-1), max: 3); + + ids.ShouldBe(src.Ordered(reverse: true).Select(m => m.ModifiedId)); + ids.First().ShouldBe(11); + await host.DisposeAsync(); + } + + [Fact] + public async Task An_open_ended_forward_modified_read_pages_to_the_end() + { + var trail = Enumerable.Range(1, 9).Select(i => Mod(i, seconds: i, prior: i, HistoryModificationType.Update, "u", Epoch)).ToList(); + var (host, nm, node, src) = await SetupAsync(trail); + + Drain(nm, node, Epoch, DateTime.MinValue, max: 4).ShouldBe(src.Ordered(reverse: false).Select(m => m.ModifiedId)); + src.LastEnd.ShouldBe(new DateTime(DateTime.MaxValue.Ticks, DateTimeKind.Utc)); + await host.DisposeAsync(); + } + + [Fact] + public async Task An_unbounded_modified_read_never_pages() + { + var trail = Enumerable.Range(1, 50).Select(i => Mod(i, seconds: i, prior: i, HistoryModificationType.Update, "u", Epoch)).ToList(); + var (host, nm, node, _) = await SetupAsync(trail); + var r = Read(nm, node, Epoch, Epoch.AddHours(1), max: 0, cp: null); + r.Data!.DataValues.Count.ShouldBe(50); + r.Cp.ShouldBeNull(); + await host.DisposeAsync(); + } + + [Fact] + public async Task Cursors_are_bound_to_the_read_kind_and_the_node() + { + var trail = Enumerable.Range(1, 10).Select(i => Mod(i, seconds: i, prior: i, HistoryModificationType.Update, "u", Epoch)).ToList(); + var (host, nm, node, _) = await SetupAsync(trail); + + // A RAW cursor for the same tag is not a modified cursor. + var raw = nm.HistoryContinuationStore.Save(null, new HistoryContinuationState(Tag, Epoch, Epoch.AddDays(1), 0, 4)); + Read(nm, node, Epoch, Epoch.AddHours(1), max: 4, cp: raw).Error.StatusCode.Code + .ShouldBe(StatusCodes.BadContinuationPointInvalid); + + // A MODIFIED cursor presented to a raw read is invalid too. + var p1 = Read(nm, node, Epoch, Epoch.AddHours(1), max: 4, cp: null); + p1.Cp.ShouldNotBeNull(); + Read(nm, node, Epoch, Epoch.AddHours(1), max: 4, cp: p1.Cp, modified: false).Error.StatusCode.Code + .ShouldBe(StatusCodes.BadContinuationPointInvalid); + + // A modified cursor issued for one node is invalid on another node's tag. + var p2 = Read(nm, node, Epoch, Epoch.AddHours(1), max: 4, cp: null); + nm.EnsureVariable("eq-1/other", null, "Other", "Double", false, historianTagname: "WW.Other", realm: AddressSpaceRealm.Uns); + var other = nm.TryGetVariable("eq-1/other")!.NodeId; + Read(nm, other, Epoch, Epoch.AddHours(1), max: 4, cp: p2.Cp).Error.StatusCode.Code + .ShouldBe(StatusCodes.BadContinuationPointInvalid); + await host.DisposeAsync(); + } + + [Fact] + public async Task A_backend_failure_is_isolated_to_its_node() + { + var (host, nm, node, src) = await SetupAsync([Mod(1, 1, 1, HistoryModificationType.Update, "u", Epoch)]); + src.Throw = new InvalidOperationException("boom"); + var r = Read(nm, node, Epoch, Epoch.AddHours(1), max: 4, cp: null); + r.Error.StatusCode.Code.ShouldBe(StatusCodes.BadHistoryOperationUnsupported); + await host.DisposeAsync(); + } + + // --- helpers --------------------------------------------------------------------------------- + + private static HistoryModifiedValue Mod( + long id, int seconds, double? prior, HistoryModificationType type, string user, DateTime modifiedAt) + { + var t = Epoch.AddSeconds(seconds); + var value = prior is null + ? new DataValueSnapshot(null, StatusCodes.BadNoData, t, modifiedAt) + : new DataValueSnapshot(prior.Value, StatusCodes.Good, t, t); + return new HistoryModifiedValue(value, modifiedAt, type, user, id); + } + + private static List Drain(OtOpcUaNodeManager nm, NodeId node, DateTime start, DateTime end, uint max) + { + var ids = new List(); + byte[]? cp = null; + var pages = 0; + do + { + var r = Read(nm, node, start, end, max, cp); + r.Error.StatusCode.Code.ShouldBe(StatusCodes.Good, r.Error.ToString()); + ids.AddRange(r.Ids); + cp = r.Cp; + (++pages).ShouldBeLessThan(60, "paging must terminate"); + } + while (cp is not null); + return ids; + } + + private static (ServiceResult Error, SdkHistoryReadResult Result, HistoryModifiedData? Data, byte[]? Cp, List Ids) Read( + OtOpcUaNodeManager nm, NodeId node, DateTime start, DateTime end, uint max, byte[]? cp, bool modified = true) + { + var context = new OperationContext(new RequestHeader(), secureChannelContext: null, RequestType.HistoryRead, identity: null); + var results = new List { null! }; + var errors = new List { null! }; + nm.HistoryRead(context, + new ReadRawModifiedDetails { StartTime = start, EndTime = end, NumValuesPerNode = max, IsReadModified = modified }, + TimestampsToReturn.Both, false, + new List { new() { NodeId = node, ContinuationPoint = cp } }, results, errors); + + HistoryModifiedData? data = null; + var ids = new List(); + if (results[0]?.HistoryData is { } ho && !ExtensionObject.IsNull(ho) + && ExtensionObject.ToEncodeable(ho) is HistoryModifiedData hm) + { + data = hm; + // The fake stores each modification's id in its (unique) server timestamp offset — see SqlLikeTrail. + ids.AddRange(hm.DataValues.Select(v => SqlLikeTrail.IdOf(v))); + } + + return (errors[0], results[0], data, results[0]?.ContinuationPoint, ids); + } + + private async Task<(OpcUaApplicationHost Host, OtOpcUaNodeManager Nm, NodeId Node, SqlLikeTrail Src)> SetupAsync( + List trail) + { + var started = await OpcUaHostStart.StartAsync( + port => new OpcUaApplicationHost( + new OpcUaApplicationHostOptions + { + ApplicationName = "OtOpcUa.ModifiedHistoryTest", + ApplicationUri = $"urn:OtOpcUa.ModifiedHistoryTest:{Guid.NewGuid():N}", + OpcUaPort = port, + PublicHostname = "localhost", + PkiStoreRoot = _pkiRoot, + }, + NullLogger.Instance), + () => new OtOpcUaSdkServer(), Ct); + var nm = started.Server.NodeManager!; + nm.HistoryContinuationStore = new InMemoryHistoryContinuationStore(); + var src = new SqlLikeTrail(trail); + nm.HistorianDataSource = src; + nm.EnsureVariable("eq-1/mod", null, "Mod", "Double", false, historianTagname: Tag, realm: AddressSpaceRealm.Uns); + return (started.Host, nm, nm.TryGetVariable("eq-1/mod")!.NodeId, src); + } + + /// + /// Behaves like the SqlServer/QuestDB modified reads: ordered by (SourceTimeUtc, ModifiedId) (descending + /// when reversed), half-open window, exclusive cursor, cap + exact . + /// To let the test recover each projected value's id, the server timestamp handed out is + /// Epoch + id ticks (an otherwise unused field in these tests). + /// + private sealed class SqlLikeTrail(List trail) : IHistorianDataSource + { + public string? LastTag { get; private set; } + public DateTime LastStart { get; private set; } + public DateTime LastEnd { get; private set; } + public uint LastMax { get; private set; } + public HistoryModifiedCursor? LastCursor { get; private set; } + public int Calls { get; private set; } + public Exception? Throw { get; set; } + + public static long IdOf(DataValue v) => v.ServerTimestamp.Ticks - Epoch.Ticks; + + public IEnumerable Ordered(bool reverse) => reverse + ? trail.OrderByDescending(m => m.Value.SourceTimestampUtc).ThenByDescending(m => m.ModifiedId) + : trail.OrderBy(m => m.Value.SourceTimestampUtc).ThenBy(m => m.ModifiedId); + + public async Task ReadModifiedAsync( + string fullReference, DateTime startUtc, DateTime endUtc, uint maxValues, HistoryModifiedCursor? resumeAfter, + CancellationToken cancellationToken) + { + await Task.Delay(1, cancellationToken).ConfigureAwait(false); + if (Throw is not null) throw Throw; + Calls++; + (LastTag, LastStart, LastEnd, LastMax, LastCursor) = (fullReference, startUtc, endUtc, maxValues, resumeAfter); + var reverse = startUtc > endUtc; + var (lo, hi) = reverse ? (endUtc, startUtc) : (startUtc, endUtc); + var rows = Ordered(reverse).Where(m => + { + var t = m.Value.SourceTimestampUtc!.Value; + var inWindow = startUtc == endUtc ? t == lo : reverse ? t > lo && t <= hi : t >= lo && t < hi; + if (!inWindow) return false; + if (resumeAfter is null) return true; + var c = t.CompareTo(resumeAfter.SourceTimeUtc); + return reverse ? c < 0 || (c == 0 && m.ModifiedId < resumeAfter.ModifiedId) + : c > 0 || (c == 0 && m.ModifiedId > resumeAfter.ModifiedId); + }).Select(m => m with { Value = m.Value with { ServerTimestampUtc = Epoch.AddTicks(m.ModifiedId) } }).ToList(); + var take = maxValues == 0 ? rows.Count : (int)Math.Min(maxValues, (uint)rows.Count); + return new HistoryModifiedReadResult(rows.Take(take).ToList(), rows.Count > take); + } + + public Task ReadRawAsync(string r, DateTime s, DateTime e, uint m, CancellationToken ct) => + NullHistorianDataSource.Instance.ReadRawAsync(r, s, e, m, ct); + public Task ReadProcessedAsync( + string r, DateTime s, DateTime e, TimeSpan i, HistoryAggregateType a, CancellationToken ct) => + NullHistorianDataSource.Instance.ReadProcessedAsync(r, s, e, i, a, ct); + public Task ReadAtTimeAsync(string r, IReadOnlyList t, CancellationToken ct) => + NullHistorianDataSource.Instance.ReadAtTimeAsync(r, t, ct); + public Task ReadEventsAsync(string? n, DateTime s, DateTime e, int m, CancellationToken ct) => + NullHistorianDataSource.Instance.ReadEventsAsync(n, s, e, m, ct); + public HistorianHealthSnapshot GetHealthSnapshot() => NullHistorianDataSource.Instance.GetHealthSnapshot(); + public void Dispose() { } + } + + public void Dispose() + { + if (Directory.Exists(_pkiRoot)) + { + try { Directory.Delete(_pkiRoot, recursive: true); } catch { /* best effort */ } + } + } +} diff --git a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadTests.cs b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadTests.cs index f9eebd88..f148eed7 100644 --- a/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadTests.cs +++ b/tests/Server/ZB.MOM.WW.OtOpcUa.OpcUaServer.Tests/NodeManagerHistoryReadTests.cs @@ -175,10 +175,13 @@ public sealed class NodeManagerHistoryReadTests : IDisposable await host.DisposeAsync(); } - /// Raw with IsReadModified=true ⇒ BadHistoryOperationUnsupported (we don't serve modified - /// history); the source is never invoked. + /// + /// #668: Raw with IsReadModified=true on a backend that keeps no modification trail (the recording fake has no + /// ReadModifiedAsync override, like Gateway) ⇒ BadHistoryOperationUnsupported, and the RAW read is never + /// invoked in its place. + /// [Fact] - public async Task Raw_read_modified_yields_BadHistoryOperationUnsupported() + public async Task Raw_read_modified_on_a_backend_without_a_trail_yields_BadHistoryOperationUnsupported() { var (host, server) = await BootAsync(); var nm = server.NodeManager!; @@ -197,10 +200,16 @@ public sealed class NodeManagerHistoryReadTests : IDisposable IsReadModified = true, }; - var (_, errors) = InvokeHistoryRead(server, nm, details, nodeId); + var (results, errors) = InvokeHistoryRead(server, nm, details, nodeId); errors[0].StatusCode.Code.ShouldBe(StatusCodes.BadHistoryOperationUnsupported); - fake.LastCall.ShouldBeNull(); + results[0].StatusCode.Code.ShouldBe(StatusCodes.BadHistoryOperationUnsupported); + fake.LastCall.ShouldBeNull(); // the raw read is never served in place of a modified one + + // The Null default (no historian configured) answers the same. + nm.HistorianDataSource = NullHistorianDataSource.Instance; + InvokeHistoryRead(server, nm, details, nodeId).Item2[0].StatusCode.Code + .ShouldBe(StatusCodes.BadHistoryOperationUnsupported); await host.DisposeAsync(); }